Phase 3 Implementation Plan: Ethereum/Tests Suite Integration¶

Status: Largely implemented. All planned spec files exist under src/it/scala/com/chipprbots/ethereum/ethtest/ (GeneralStateTestsSpec, BlockchainTestsSpec, VMTestsSpec, TransactionTestsSpec, ComprehensiveBlockchainTestsSpec, ExecutionSpecsStateTestsSpec), and nightly CI runs the suite via .github/workflows/ethereum-tests-nightly.yml. Retained as a record of the rollout plan and as a reference for ongoing coverage work.

Overview¶

Complete the ethereum/tests adapter by expanding test coverage to the full suite, replacing custom tests, and integrating into CI pipeline.

Duration: 4 weeks Prerequisites: Phase 1 & 2 complete (✅) Goal: 100+ ethereum/tests passing, CI integrated

Week 1: Expand Test Coverage¶

Objective¶

Run comprehensive ethereum/tests across multiple categories and validate against different networks.

Tasks¶

Task 1.1: GeneralStateTests Category (Days 1-2)¶

Goal: Run 25+ GeneralStateTests

Steps: 1. Create GeneralStateTestsSpec.scala extending EthereumTestsSpec 2. Start with basic tests: - add11.json - Simple arithmetic - ValueOverflow.json - Overflow handling - CreateContractWithBalance.json - Contract creation 3. Run tests for multiple networks (Berlin, Istanbul, Constantinople) 4. Validate state transitions and final state roots

Success Criteria: - At least 25 GeneralStateTests passing - Multiple networks validated - Clear error messages for failures

Task 1.2: BlockchainTests Category (Days 3-4)¶

Goal: Run 25+ BlockchainTests

Steps: 1. Create BlockchainTestsSpec.scala extending EthereumTestsSpec 2. Start with ValidBlocks tests: - SimpleTx.json (already validated) - ValueOverflow.json - UncleFromSideChain.json - Uncle handling 3. Test uncle block validation 4. Validate difficulty calculations

Success Criteria: - At least 25 BlockchainTests passing - Uncle handling working - Difficulty calculations validated

Task 1.3: Test Organization (Day 5)¶

Goal: Create organized test structure

Steps: 1. Create test category structure:

src/it/scala/com/chipprbots/ethereum/ethtest/
  ├── EthereumTestsSpec.scala (base class)
  ├── EthereumTestExecutor.scala
  ├── EthereumTestHelper.scala
  ├── categories/
  │   ├── GeneralStateTestsSpec.scala
  │   ├── BlockchainTestsSpec.scala
  │   └── VMTestsSpec.scala (future)
  └── SimpleEthereumTest.scala (keep for validation)

Add test filtering by:
Network (Berlin, Istanbul, etc.)
Category (GeneralStateTests, BlockchainTests)
Test name pattern
Create test runner utilities

Success Criteria: - Organized test structure - Easy to run specific categories - Clear test organization

Deliverables¶

GeneralStateTestsSpec.scala
BlockchainTestsSpec.scala
Test organization structure
50+ tests passing
Test results documented

Week 2: Handle Edge Cases¶

Objective¶

Improve error handling, support missing features, and handle test failures gracefully.

Tasks¶

Task 2.1: Error Reporting Enhancement (Days 1-2)¶

Goal: Clear, actionable error messages

Steps: 1. Add detailed failure reporting: - Expected vs actual state roots - Account balance mismatches - Storage key/value differences - Transaction execution errors

Create failure analysis report:

### Test Failure Report

**Test:** SimpleTx_Berlin
**Category:** BlockchainTests
**Network:** Berlin

**Failure Type:** State root mismatch
**Expected:** 0xcc353bc...
**Actual:** 0x1234567...

**Account Differences:**
- Address 0xa94f53...
  - Expected balance: 1000000000
  - Actual balance: 999999000

Add debug logging for execution:
Transaction processing
Opcode execution
State changes

Success Criteria: - Clear error messages - Easy to diagnose failures - Debug logging available

Task 2.2: Missing EIP Support (Days 3-4)¶

Goal: Identify and document unsupported features

Steps: 1. Run tests and identify EIP-related failures 2. Create EIP support matrix:

| EIP | Name | Status | Priority |
|-----|------|--------|----------|
| 161 | State trie clearing | ✅ Complete | High |
| 1559 | Fee market | ❌ Not applicable (ETC) | N/A |
| 2929 | Gas cost increases | ⏳ Pending | Medium |

Implement high-priority missing EIPs
Document unsupported features with rationale

Success Criteria: - EIP support matrix created - High-priority EIPs implemented - Clear documentation of unsupported features

Task 2.3: Test Filtering (Day 5)¶

Goal: Run specific test subsets

Steps: 1. Implement filtering by:

// Filter by network
runTests(network = "Berlin")

// Filter by category
runTests(category = "GeneralStateTests")

// Filter by name pattern
runTests(namePattern = ".*Create.*")

// Exclude known failures
runTests(exclude = knownFailures)

Create test suite configurations:
Quick smoke tests (5 tests, < 1 minute)
Standard suite (50 tests, < 5 minutes)
Comprehensive suite (all tests, < 30 minutes)
Add command-line options for test filtering

Success Criteria: - Flexible test filtering - Predefined test suites - Easy to run specific tests

Deliverables¶

Enhanced error reporting
EIP support matrix
Missing EIP implementations
Test filtering infrastructure
75+ tests passing

Week 3: Replace Custom Tests¶

Objective¶

Migrate from custom tests to ethereum/tests where applicable.

Tasks¶

Task 3.1: Test Mapping (Days 1-2)¶

Goal: Map custom tests to ethereum/tests equivalents

Steps: 1. Analyze existing tests: - ForksTest.scala - ETC hard fork transitions - ContractTest.scala - Contract deployment and calls - ECIP1017Test.scala - ETC-specific emission schedule

Create mapping document:

| Custom Test | ethereum/tests Equivalent | Status |
|-------------|---------------------------|--------|
| ForksTest - Atlantis transition | BlockchainTests/TransitionTests/bcFrontierToAtlantis | ✅ Available |
| ContractTest - Simple storage | GeneralStateTests/stExample/add11 | ✅ Available |
| ECIP1017Test - Emission | N/A - ETC specific | ⚠️ Keep custom |

Identify gaps where custom tests are still needed

Success Criteria: - Complete mapping of tests - Gaps identified - Migration plan created

Task 3.2: Augment ForksTest (Days 3-4)¶

Goal: Augment ForksTest with ethereum/tests

Steps: 1. Keep existing ForksTest for ETC-specific validations 2. Add ethereum/tests for standard EVM behavior:

class ForksTest extends EthereumTestsSpec {
  // Existing ETC-specific tests
  "Atlantis fork" should "activate at block 8772000" in { ... }

  // New: ethereum/tests for Byzantium-equivalent
  it should "match Byzantium EVM behavior" in {
    runTestFile("/BlockchainTests/TransitionTests/bcFrontierToHomestead/...")
  }
}

Validate that ethereum/tests cover standard cases
Document which tests provide what coverage

Success Criteria: - ForksTest augmented with ethereum/tests - ETC-specific tests retained - Clear documentation of coverage

Task 3.3: Augment ContractTest (Day 5)¶

Goal: Augment ContractTest with ethereum/tests

Steps: 1. Keep custom tests for specific scenarios 2. Add ethereum/tests for standard contract operations:

class ContractTest extends EthereumTestsSpec {
  // Existing custom contract tests
  "Contract" should "deploy with constructor" in { ... }

  // New: ethereum/tests for contract behavior
  it should "match standard CREATE behavior" in {
    runTestFile("/GeneralStateTests/stCreate/...")
  }
}

Ensure comprehensive contract operation coverage

Success Criteria: - ContractTest augmented with ethereum/tests - Custom tests retained where needed - Comprehensive coverage achieved

Deliverables¶

Test mapping document
Augmented ForksTest with ethereum/tests
Augmented ContractTest with ethereum/tests
Migration guide
100+ tests passing

Week 4: CI Integration¶

Objective¶

Integrate ethereum/tests into CI pipeline with automated execution and reporting.

Tasks¶

Task 4.1: GitHub Actions Workflow (Days 1-2)¶

Goal: Automated test execution on PR and merge

Steps: 1. Create .github/workflows/ethereum-tests.yml:

name: Ethereum Tests

on:
  pull_request:
    branches: [ main, develop ]
  push:
    branches: [ main, develop ]

jobs:
  ethereum-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          submodules: true  # Initialize ets/tests

      - name: Setup JDK 17
        uses: actions/setup-java@v3

      - name: Run Quick Suite
        run: sbt "IntegrationTest/testOnly *.SimpleEthereumTest"

      - name: Run Standard Suite
        run: sbt "IntegrationTest/testOnly *.GeneralStateTestsSpec *.BlockchainTestsSpec"

      - name: Generate Report
        run: sbt ethereumTestReport

      - name: Upload Results
        uses: actions/upload-artifact@v3
        with:
          name: ethereum-test-results
          path: target/ethereum-test-report.html

Configure test timeouts and resource limits
Add caching for faster builds

Success Criteria: - Automated test execution - Tests run on every PR - Reasonable execution time (< 10 minutes for standard suite)

Task 4.2: Test Result Reporting (Day 3)¶

Goal: Clear test result visualization

Steps: 1. Generate HTML test reports 2. Create summary statistics:

Ethereum/Tests Results
=====================
Total: 120 tests
Passed: 115 (95.8%)
Failed: 5 (4.2%)

By Category:
- GeneralStateTests: 48/50 (96%)
- BlockchainTests: 47/50 (94%)
- VMTests: 20/20 (100%)

Add failure details with links to test files
Store results as CI artifacts

Success Criteria: - Clear test reports - Easy to see what failed - Historical tracking possible

Task 4.3: Performance Optimization (Day 4)¶

Goal: Fast CI execution

Steps: 1. Implement parallel test execution:

// Run test categories in parallel
testCategories.par.foreach(category => runTests(category))

Add test result caching:
Cache compiled test suite
Cache successful test results
Rerun only failed tests
Optimize storage cleanup between tests

Success Criteria: - Standard suite < 5 minutes - Comprehensive suite < 30 minutes - Efficient resource usage

Task 4.4: Documentation & Rollout (Day 5)¶

Goal: Complete documentation and enable for all developers

Steps: 1. Create comprehensive README: - How to run ethereum/tests locally - How to add new test cases - How to debug failures - CI integration details

Update CONTRIBUTING.md:
Requirement to pass ethereum/tests
How to interpret test results
What to do if tests fail
Add PR template reminders about ethereum/tests
Announce to team and provide training

Success Criteria: - Complete documentation - Team trained on usage - ethereum/tests required for PR approval

Deliverables¶

CI workflow configured
Automated test reports
Performance optimizations
Comprehensive documentation
Full rollout to team

Success Metrics¶

Coverage¶

✅ 100+ ethereum/tests passing
✅ 3+ test categories validated
✅ Multiple networks tested (Berlin, Istanbul, Constantinople)
✅ Both standard and ETC-specific scenarios covered

Quality¶

✅ Zero false positives
✅ Clear error messages for all failures
✅ Documented limitations and unsupported features

Performance¶

✅ Quick suite < 1 minute (smoke tests)
✅ Standard suite < 5 minutes (CI default)
✅ Comprehensive suite < 30 minutes (nightly/manual)

Integration¶

✅ CI pipeline integrated
✅ Automated reporting
✅ PR validation enabled
✅ Team adoption complete

Risk Mitigation¶

Risk: Test Suite Too Slow¶

Mitigation: - Parallel execution - Selective testing (run only affected categories) - Test result caching - Nightly runs for comprehensive suite

Risk: Too Many Test Failures¶

Mitigation: - Start with known-good tests - Incremental expansion - Document expected failures - Focus on high-value tests first

Risk: Network-Specific Behavior Differences¶

Mitigation: - Test each network separately - Document network-specific quirks - Clear error messages indicating network context

Risk: Missing EIP Support¶

Mitigation: - Create EIP support matrix upfront - Prioritize high-impact EIPs - Document unsupported features clearly - Plan EIP implementation sprints

Timeline Summary¶

Week	Focus	Deliverables	Tests Passing
1	Expand Coverage	GeneralStateTests, BlockchainTests	50+
2	Edge Cases	Error handling, EIP support, filtering	75+
3	Replace Custom	Test mapping, augmented tests	100+
4	CI Integration	Automation, reporting, rollout	100+

Next Steps After Phase 3¶

Continuous Expansion: Add more test categories (VMTests, DifficultyTests)
Performance Monitoring: Track test execution time trends
Coverage Reporting: Measure code coverage from ethereum/tests
Cross-Client Validation: Compare results with other ETC clients (Core-Geth)
Test Suite Maintenance: Keep ethereum/tests submodule updated

The foundation is solid. Phase 3 will complete the vision of comprehensive EVM validation against the official ethereum/tests suite.