ADR-015: Ethereum/Tests Adapter Implementation

Status: ✅ Phase 1-2 Complete, Phase 3 Ready

Date: November 2025

Verification Date: November 17, 2025

Deciders: Chippr Robotics LLC Engineering Team

Implementation Status: - Phase 1 (JSON Parsing): ✅ Complete - Phase 2 (Execution): ✅ Complete - ALL TESTS PASSING - Phase 3 (Integration): ⏳ Ready to begin

Latest Update: November 15, 2025 - ✅ All 4 validation tests passing - ✅ Block execution working with SimpleTx_Berlin and SimpleTx_Istanbul - ✅ MPT storage issue resolved (unified storage instance) - ✅ Post-state validation confirmed - ✅ State roots matching expected values - 🚀 Ready for Phase 3: broader test suite execution

Verification Report: See Testing Tags Verification Report

Context

Following ADR-014's recommendation to adopt the ethereum/tests repository for comprehensive EVM validation, we need to implement a test adapter that can:

1. Parse JSON blockchain tests from the official ethereum/tests repository
2. Execute these tests against our EVM implementation
3. Replace brittle custom test fixtures with community-maintained tests
4. Ensure compliance with other execution clients (geth, nethermind, besu)

Problem Discovery

Current State: - Custom test fixtures require manual regeneration after EVM changes - Test fixtures from PR #421 are incomplete/incorrect, blocking 93% of integration tests - No systematic way to validate EVM compliance with Ethereum specification - Maintenance burden of custom fixtures is high

Requirements: 1. Support for ethereum/tests JSON format 2. Execution of blockchain tests with pre/post state validation 3. Fork configuration mapping (Frontier → Berlin) 4. State root validation 5. Compatible with ETC blocks < 19.25M (pre-Spiral fork)

ETC/ETH Compatibility Analysis

Per ADR-014, Ethereum Classic maintains 100% EVM compatibility with Ethereum through block 19.25M (Spiral fork). This means:

• All opcodes, gas costs, and state transitions are identical
• Official ethereum/tests can be used directly for validation
• No ETC-specific modifications needed for pre-Spiral tests
• Post-Spiral tests must be filtered out

Decision

We decided to implement a multi-phase ethereum/tests adapter:

Phase 1: Infrastructure (Initial Implementation)

Components: 1. JSON Parser (EthereumTestsAdapter.scala) - Parse BlockchainTests JSON format - Support for pre-state, blocks, post-state, and network fields - Circe-based JSON decoding with strongly-typed case classes

1. Domain Converter (TestConverter.scala)
2. Convert JSON hex strings to internal domain objects
3. Map network names (e.g., "Byzantium") to fork configurations

4. Handle account state, transactions, and block headers

5. Test Runner (EthereumTestsSpec.scala)

6. Base class for running ethereum/tests
7. Test discovery and execution framework
8. Integration with ScalaTest

Design Decisions:

A. JSON Parsing with Circe - Chose Circe for type-safe JSON parsing - Explicit decoders for each test component - Better error messages than reflection-based approaches

B. Separation of Concerns - Parser (EthereumTestsAdapter) → Converter (TestConverter) → Executor (future) - Each component has single responsibility - Easy to test and maintain independently

C. Hex String Handling - All ethereum/tests use hex strings with "0x" prefix - Centralized parsing in TestConverter - Validation of hex format before conversion

D. Fork Configuration Mapping - Network names map to ForkBlockNumbers configuration - All forks activated at block 0 for test scenarios - Supports Frontier through Berlin (pre-Spiral only)

Phase 2: Execution (Next Sprint)

TODO: 1. Implement EthereumTestExecutor 2. Use existing BlockExecution infrastructure 3. Set up state from JSON pre-state 4. Execute blocks and validate post-state 5. Compare state roots

Phase 3: Integration (Future)

TODO: 1. Add ethereum/tests as git submodule or CI download 2. Create test suites for relevant categories 3. Replace ForksTest with ethereum/tests 4. Replace ContractTest with ethereum/tests 5. Remove custom fixture files

Implementation Details

File Structure

src/it/scala/com/chipprbots/ethereum/ethtest/
  ├── EthereumTestsAdapter.scala   - JSON parsing and loading
  ├── TestConverter.scala           - JSON to domain conversion
  └── EthereumTestsSpec.scala       - Test runner base class

docs/testing/
  └── ETHEREUM_TESTS_ADAPTER.md     - Comprehensive documentation

Data Model

BlockchainTestSuite → Map[String, BlockchainTest] - Each file contains multiple test cases - Test name is the key

BlockchainTest:

case class BlockchainTest(
    pre: Map[String, AccountState],      // Initial state
    blocks: Seq[TestBlock],               // Blocks to execute
    postState: Map[String, AccountState], // Expected final state
    network: String                       // Fork configuration
)

Conversion Flow:

JSON String
  ↓ (parse)
BlockchainTestSuite
  ↓ (map)
List[(String, BlockchainTest)]
  ↓ (convert)
Internal Domain Objects
  ↓ (execute)
State Validation

Network Mapping

ethereum/tests	ETC Fork	Block Number
Frontier	Frontier	0
Homestead	Homestead	1.15M
EIP150	Tangerine Whistle	2.46M
EIP158	Spurious Dragon	3M
Byzantium	Atlantis	8.77M
Constantinople	Agharta	9.57M
Istanbul	Phoenix	10.5M
Berlin	Magneto	13.2M

For test execution, all forks are activated at block 0 to match test expectations.

Consequences

Positive

Test Quality: - ✅ Thousands of validated test cases from ethereum/tests - ✅ Community-maintained and continuously updated - ✅ Used by all major Ethereum clients (standard compliance) - ✅ Comprehensive coverage of edge cases

Maintenance: - ✅ No need to regenerate fixtures after EVM changes - ✅ Automatic updates when ethereum/tests updated - ✅ Reduced maintenance burden vs custom fixtures

Validation: - ✅ Ensures EVM execution matches Ethereum specification - ✅ Validates state root calculations - ✅ Tests all opcodes and gas costs - ✅ Covers fork transitions

Development: - ✅ Clear separation of parser, converter, and executor - ✅ Type-safe JSON parsing with Circe - ✅ Easy to add new test categories - ✅ Scalable architecture

Negative

Implementation Effort: - ⚠️ Multi-phase implementation (3 phases) - ⚠️ Initial phase doesn't execute tests yet (parser only) - ⚠️ Requires additional work to replace existing tests

Dependencies: - ⚠️ Adds Circe JSON library dependency - ⚠️ Requires ethereum/tests repository (git submodule or download) - ⚠️ Test files are large (100s of MB)

Test Execution Time: - ⚠️ Thousands of tests will take longer than current 14 tests - ⚠️ May need test filtering/parallelization - ⚠️ CI time may increase significantly

Neutral

Compatibility: - ℹ️ Only supports pre-Spiral forks (blocks < 19.25M) - ℹ️ Post-Spiral tests must be filtered out - ℹ️ Network names must be mapped to ETC fork names

Alternative Approaches Considered

1. Manual Fixture Regeneration

Approach: Regenerate existing fixtures with corrected EVM config

Pros: - Minimal code changes - Fast implementation (1-2 days)

Cons: - ❌ Doesn't solve long-term maintenance problem - ❌ Still requires manual work after EVM changes - ❌ Limited coverage (14 tests vs thousands) - ❌ No validation against Ethereum spec

Decision: Rejected - doesn't meet long-term goals

2. Custom Test Generator

Approach: Build our own test generator from ETC node

Pros: - Full control over test scenarios - ETC-specific tests possible

Cons: - ❌ High maintenance burden - ❌ Requires synced ETC node - ❌ Doesn't provide standard compliance validation - ❌ Duplication of effort vs ethereum/tests

Decision: Rejected - reinventing the wheel

3. External Test Runner

Approach: Use existing ethereum/tests runner (e.g., retesteth)

Pros: - No implementation needed - Already mature and tested

Cons: - ❌ Doesn't integrate with our test suite - ❌ Different programming language (C++/Go) - ❌ Harder to debug failures - ❌ Not part of sbt test workflow

Decision: Rejected - poor integration

Implementation Plan

Phase 1: Infrastructure ✅ (Completed)

Deliverables: - [x] EthereumTestsAdapter.scala - JSON parsing - [x] TestConverter.scala - Domain conversion - [x] EthereumTestsSpec.scala - Test runner - [x] ETHEREUM_TESTS_ADAPTER.md - Documentation - [x] ADR-015 - This document

Validation: - [x] Code follows CONTRIBUTING.md guidelines - [x] Scalafmt formatting applied - [x] Proper package structure - [x] Comprehensive documentation

Phase 2: Execution ✅ COMPLETE

Status: All tests passing, ready for Phase 3

Completed: - [x] EthereumTestExecutor.scala - Test execution infrastructure - [x] EthereumTestHelper.scala - Block execution with BlockExecution integration - [x] Initial state setup from pre-state - [x] Storage initialization (SerializingMptStorage, EvmCodeStorage) - [x] Account creation with balance, nonce, code, and storage - [x] State root calculation and validation - [x] SimpleEthereumTest.scala - 4 validation tests (ALL PASSING) - [x] Block execution loop using BlockExecution infrastructure - [x] Transaction execution and receipt validation - [x] Post-state validation against expected state - [x] State root comparison - [x] Comprehensive error reporting

Key Achievements: - ✅ Fixed MPT storage persistence issue (unified storage instance) - ✅ Genesis block header parsing and usage - ✅ Chain weight tracking for executed blocks - ✅ End-to-end block execution validated - ✅ SimpleTx_Berlin and SimpleTx_Istanbul tests passing - ✅ State roots matching expected values: - Initial: cafd881ab193703b83816c49ff6c2bf6ba6f464a1be560c42106128c8dbc35e7 - Final: cc353bc3876f143b9dd89c5191e475d3a6caba66834f16d8b287040daea9752c

Critical Fix: Storage persistence issue resolved by using blockchain.getBackingMptStorage(0) to ensure initial state and block execution share the same storage instance.

Timeline: Completed November 15, 2025

Phase 3: Integration (Future)

Tasks: 1. Download ethereum/tests repository 2. Filter tests by network (pre-Spiral only) 3. Create test suites by category 4. Replace ForksTest 5. Replace ContractTest
6. Remove old fixtures

Timeline: 2-3 weeks

Validation

Code Quality

Standards Met: - ✅ Follows Scala 3 best practices - ✅ Type-safe JSON parsing with Circe decoders - ✅ Comprehensive Scaladoc comments - ✅ Clear separation of concerns - ✅ Proper error handling

Formatting: - ✅ Scalafmt configuration (.scalafmt.conf) - ✅ 120 character max line length - ✅ Consistent spacing and indentation - ✅ Scala 3 dialect

Testing: - ✅ Base test class provided (EthereumTestsSpec) - ✅ Example usage documented - ✅ Integration test location (src/it/scala) - ✅ SimpleEthereumTest validates infrastructure (3 tests passing) - ✅ State setup tested and working

Documentation

Comprehensive Docs: - ✅ ADR-015 (this document) - ⏳ ETHEREUM_TESTS_ADAPTER.md (to be created with full execution guide) - ✅ Inline Scaladoc comments - ⏳ Architecture diagrams (to be added) - ✅ Usage examples in SimpleEthereumTest

Success Metrics

Short-term (Phase 1): ✅ COMPLETE - ✅ JSON parser successfully decodes ethereum/tests format - ✅ Converter produces valid domain objects - ✅ Documentation is clear and comprehensive

Medium-term (Phase 2): ✅ COMPLETE - ✅ Test executor runs simple value transfer tests - ✅ Block execution infrastructure integrated - ✅ State root validation passes - ✅ SimpleTx tests pass successfully (Berlin and Istanbul networks) - ✅ End-to-end execution validated

Long-term (Phase 3): ⏳ READY TO BEGIN - [ ] Run comprehensive ethereum/tests suite (100+ tests) - [ ] Multiple test categories passing (GeneralStateTests, BlockchainTests) - [ ] ForksTest augmented with ethereum/tests - [ ] ContractTest augmented with ethereum/tests - [ ] CI integration complete

Long-term (Phase 3): - [ ] All relevant ethereum/tests categories passing - [ ] ForksTest replaced with ethereum/tests - [ ] ContractTest replaced with ethereum/tests - [ ] CI runs ethereum/tests automatically - [ ] 100+ tests passing from official test suite

References

• ethereum/tests Repository
• Test Format Documentation
• VM-007: EIP-161 noEmptyAccounts Fix
• ETC Fork Timeline
• Contributing Guide

Related Files

• src/it/scala/com/chipprbots/ethereum/ethtest/EthereumTestsAdapter.scala
• src/it/scala/com/chipprbots/ethereum/ethtest/TestConverter.scala
• src/it/scala/com/chipprbots/ethereum/ethtest/EthereumTestsSpec.scala
• docs/testing/ETHEREUM_TESTS_ADAPTER.md

Lessons Learned

1. JSON Parsing: Circe provides excellent type safety for complex JSON structures
2. Hex Handling: Centralized hex string parsing prevents errors throughout codebase
3. Fork Mapping: Network names in ethereum/tests don't exactly match ETC fork names - requires translation layer
4. Phased Approach: Breaking implementation into phases allows incremental progress and validation
5. Documentation First: Writing comprehensive docs helps clarify design before implementation

Future Considerations

1. Performance: May need test parallelization for large test suites
2. Test Selection: Implement filtering to run subset of tests (e.g., by category, fork, or difficulty)
3. Continuous Updates: Automate ethereum/tests repository updates in CI
4. Custom Tests: May still need some ETC-specific tests for post-Spiral behavior
5. Test Result Database: Track test results over time to detect regressions