Skip to content

Nightly Test Run Analysis - December 19, 2025

Executive Summary

The nightly comprehensive test suite reported 37 test failures out of 184 total tests (147 succeeded, 37 failed, 1 ignored).

Test Execution Time: 49 minutes 10 seconds (2950 seconds)

Workflow Run: Nightly Comprehensive Test Suite

Test Command: sbt testComprehensive (runs testAll)

Failed Test Suites

The following test suite classes contain failures:

  1. com.chipprbots.ethereum.ethtest.VMTestsSpec - 1 failure
  2. com.chipprbots.ethereum.txExecTest.ForksTest - 1 failure
  3. com.chipprbots.ethereum.sync.E2ESyncSpec - 2 failures
  4. com.chipprbots.ethereum.consensus.mess.MESSIntegrationSpec - 3 failures
  5. com.chipprbots.ethereum.network.E2EHandshakeSpec - 2 failures
  6. com.chipprbots.ethereum.sync.E2EStateTestSpec - 15 failures
  7. com.chipprbots.ethereum.blockchain.sync.snap.SNAPSyncIntegrationSpec - 2 failures
  8. com.chipprbots.ethereum.sync.FastSyncItSpec - 8 failures
  9. com.chipprbots.ethereum.sync.E2EFastSyncSpec - 3 failures

Failure Categories

1. Timeout Issues (28 failures - 76% of all failures)

Most failures are related to test timeouts after retry limits are exceeded.

Error Pattern: 

java.util.concurrent.TimeoutException: Task time out after all retries
at com.chipprbots.ethereum.sync.util.SyncCommonItSpecUtils$.retryUntilWithDelay$$anonfun$1

Affected Tests: - E2ESyncSpec: - should handle blockchain reorganization correctly (1m 36s timeout) - should continue sync after peer disconnection and reconnection (1m 37s timeout)

  • E2EStateTestSpec (15 failures, all timeouts):
  • should synchronize state trie between peers (1m 35s)
  • should maintain state trie consistency across multiple state updates (1m 34s)
  • should handle large state tries efficiently (1m 34s)
  • should validate state roots during synchronization (1m 35s)
  • should maintain state root consistency during chain reorganization (1m 37s)
  • should propagate account state changes between peers (1m 34s)
  • should handle rapid account state updates (1m 34s)
  • should synchronize contract storage between peers (1m 35s)
  • should handle storage updates across multiple blocks (1m 34s)
  • should recover from partial state loss (1m 35s)
  • should handle state synchronization with missing peers (1m 36s)
  • should maintain state integrity during peer disconnection (1m 35s)
  • should maintain state integrity across entire sync process (1m 34s)
  • should verify state consistency after blockchain reorganization with state changes (1m 35s)

  • should persist state correctly across peer restarts (1m 35s)

  • FastSyncItSpec (8 timeout failures):

  • should sync blockchain with state nodes (1m 38s)
  • should sync blockchain with state nodes when peer do not response with full responses (1m 38s)
  • should sync blockchain with state nodes when one of the peers send empty state responses (1m 38s)
  • should update pivot block and sync this new pivot block state (1m 34s)
  • should sync state to peer from partially synced state (1m 35s)
  • should follow the longest chains (1m 39s)
  • should handle state synchronization from multiple peers (1m 36s)

  • should handle state nodes with complex account structures (1m 35s)

  • SNAPSyncIntegrationSpec (2 timeout failures):

  • should initialize and distribute tasks to workers (5s 192ms)

  • should queue healing tasks for missing nodes via coordinator (3s 26ms)

  • E2EHandshakeSpec:

  • should handle bidirectional connection attempts (9s 126ms timeout)
  • should handshake with peers at genesis (7s 155ms timeout)

Root Cause Analysis: - Tests are timing out waiting for synchronization operations to complete - The retry mechanism (retryUntilWithDelay) exhausts all attempts before success condition is met - Suggests either: 1. Network/peer communication issues in test setup 2. State synchronization logic not completing as expected 3. Test timeout values too aggressive for CI environment 4. Missing test data or mock responses causing hangs

Accompanying Log Patterns: 

PEER_REQUEST_TIMEOUT: peer=PeerId(...), reqType=GetNodeData, elapsed=5019ms (timeout=5000ms)

2. MPT/State Trie Missing Node Issues (Multiple occurrences)

Error Pattern: 

com.chipprbots.ethereum.mpt.MerklePatriciaTrie$MissingNodeException: Node not found <hash>, trie is inconsistent
com.chipprbots.ethereum.mpt.MerklePatriciaTrie$MissingRootNodeException: Root node not found <hash>

Observed Errors: - Multiple block import errors with missing MPT nodes - Node hashes not found: 81a660226f57c558268355ddc31cb4305880cf26d5ad25ef4d638d2fbd57d83c, c47df861695e994eb6b19b74a5e693d0d7a279e91deb6617fcb7787b347ab417, and many others - Root nodes not found: 225ce73da683bb17cd073a9c008b73ce25b6474a6fc32bd66836e04336e3d6a8, f337a6ca26ee85e24211ff69f02a1bd71df4639a862c6a4c30adf56251a19106, etc.

Root Cause: - State trie nodes are missing from the database during block import/validation - Indicates incomplete state synchronization or test fixture setup issues - May be related to the timeout issues - operations don't complete, leaving partial state

3. Synchronization State Issues (3 failures - 8%)

Error Pattern: 

None was not defined

Affected Tests: - E2EFastSyncSpec: - should successfully sync blockchain with state nodes (E2EFastSyncSpec.scala:106) - should handle incomplete state downloads gracefully (E2EFastSyncSpec.scala:205) - should recover from peer disconnection during state download (E2EFastSyncSpec.scala:374)

Root Cause: - Test expects an Option or Either to be defined/successful but gets None or Left - Indicates that the sync process is not reaching expected completion state - Likely related to the timeout issues - sync doesn't complete, so state is undefined

4. Fork/Chain Validation Issues (1 failure)

Error Pattern: 

Left(MPTError(com.chipprbots.ethereum.mpt.MerklePatriciaTrie$MissingRootNodeException: Root node not found 225ce73da683bb17cd073a9c008b73ce25b6474a6fc32bd66836e04336e3d6a8)) 
was not an instance of scala.util.Right, but an instance of scala.util.Left

Affected Test: - ForksTest: should execute blocks with respect to forks (ForksTest.scala:76)

Root Cause: - Fork execution expects successful result (Right) but gets error (Left) - The underlying error is again a missing MPT root node - Fork transition logic is failing due to missing state data

5. MESS Consensus Issues (3 failures)

Error Pattern: 

Test names suggest issues with chain selection logic in MESS consensus

Affected Tests: - MESSIntegrationSpec: - should prefer recently seen chain over old chain with same difficulty (243ms) - should handle blocks without first-seen time using block timestamp (1ms) - should correctly handle chain reorganization scenario (3ms)

Root Cause: - Fast failures (< 1 second) suggest assertion failures rather than timeouts - Chain selection logic in MESS consensus not working as expected - May be related to first-seen timestamp tracking or chain scoring algorithm

6. Data Decoding Issues (1 failure)

Error Pattern: 

java.lang.RuntimeException: Failed to load test suite: java.lang.RuntimeException: 
Failed to decode test suite: DecodingFailure at .add.blocks: Missing required field

Affected Test: - VMTestsSpec: should load and parse a sample VM arithmetic test (83ms)

Root Cause: - Test data file has incorrect format or missing required fields - Ethereum test suite JSON format may have changed or is incompatible - Test data files may need updating

7. Network Protocol Issues

Observed Warnings (not direct test failures but related): 

RECV_MSG: peer=127.0.0.1:46506, msg[0] DECODE_ERROR: Unknown snap/1 message type: 29

Root Cause: - Snap sync protocol receiving unexpected message types - May indicate version mismatch or incomplete protocol implementation - Could contribute to sync timeout issues

Critical Findings

Primary Issue: State Synchronization Reliability

Impact: 28 out of 37 failures (76%) are timeout-related, predominantly in state sync tests.

Key Problems: 1. Peer Request Timeouts: GetNodeData requests consistently timing out at 5-second threshold 2. Missing State Nodes: MPT nodes not available when needed for validation 3. Incomplete Synchronization: Tests expecting completed sync state but finding undefined/incomplete state 4. Test Infrastructure: Possible issues with test peer setup or mock data availability

Secondary Issues

  1. MESS Consensus Chain Selection: Logic issues in recently-added MESS consensus mechanism
  2. Protocol Decoding: Ethereum test suite compatibility and unknown SNAP message types
  3. Fork Handling: Fork execution failing due to missing state (cascading from primary issue)

Immediate Actions (Priority 1)

  1. Investigate Peer Request Timeout Issues
  2. Review PeerRequestHandler timeout configuration (currently 5000ms)
  3. Check if test environment needs longer timeouts for CI
  4. Verify mock peer responses in test fixtures

  5. Location: Check com.chipprbots.ethereum.blockchain.sync.PeerRequestHandler

  6. Fix State Trie Node Availability

  7. Review state node storage/retrieval in tests
  8. Ensure test fixtures properly populate MPT nodes before tests run
  9. Check database initialization in test setup

  10. Location: com.chipprbots.ethereum.mpt.MerklePatriciaTrie

  11. Review Test Synchronization Logic

  12. Examine retryUntilWithDelay timeout/retry parameters
  13. Consider environment-specific timeout configuration (CI vs local)
  14. Location: com.chipprbots.ethereum.sync.util.SyncCommonItSpecUtils.scala

Secondary Actions (Priority 2)

  1. Fix MESS Consensus Chain Selection
  2. Review chain selection logic with first-seen timestamps
  3. Test edge cases: missing timestamps, equal difficulty, reorgs

  4. Location: com.chipprbots.ethereum.consensus.mess.MESSIntegrationSpec

  5. Update Ethereum Test Suite Data

  6. Verify test data JSON format matches current schema
  7. Update if ethereum/tests repository has changed

  8. Location: VMTestsSpec test data files

  9. Address SNAP Protocol Message Handling

  10. Add handling for message type 29 or filter it appropriately
  11. Review snap/1 protocol specification
  12. Location: RLPx message decoding logic

Monitoring and Prevention (Priority 3)

  1. Add Diagnostic Logging
  2. Enhance logging around peer request handling
  3. Log state trie access attempts and failures

  4. Track synchronization progress more granularly

  5. Consider Test Categorization

  6. Mark long-running sync tests differently
  7. Run them separately or with extended timeouts

  8. Consider parallel execution limits for resource-heavy tests

  9. CI Environment Optimization

  10. Profile test execution times in CI vs local
  11. Adjust timeout multipliers for CI environment
  12. Consider dedicated test database setup

Test Suite Structure Reference

Command Used

sbt testComprehensive

Configuration

  • Alias points to: testAll
  • Runs: All test suites including integration tests
  • Timeout: 240 minutes for entire job
  • Environment: FUKUII_DEV=true

Test Hierarchy

IntegrationTest scope includes:
├── E2ESyncSpec (End-to-end sync scenarios)
├── E2EStateTestSpec (State synchronization scenarios)
├── E2EHandshakeSpec (Peer handshake scenarios)
├── E2EFastSyncSpec (Fast sync scenarios)
├── FastSyncItSpec (Fast sync integration tests)
├── SNAPSyncIntegrationSpec (SNAP sync integration)
├── MESSIntegrationSpec (MESS consensus integration)
├── VMTestsSpec (VM compliance tests)
└── ForksTest (Fork execution tests)

Files to Review

Based on the failure analysis, these are the key files to investigate:

Core Sync Logic

  • src/main/scala/com/chipprbots/ethereum/blockchain/sync/PeerRequestHandler.scala
  • src/main/scala/com/chipprbots/ethereum/blockchain/sync/regular/BlockImporter.scala
  • src/test/scala/com/chipprbots/ethereum/sync/util/SyncCommonItSpecUtils.scala

State Trie Management

  • src/main/scala/com/chipprbots/ethereum/mpt/MerklePatriciaTrie.scala
  • src/main/scala/com/chipprbots/ethereum/mpt/MptStorage.scala

MESS Consensus

  • src/main/scala/com/chipprbots/ethereum/consensus/mess/ (chain selection logic)
  • src/it/scala/com/chipprbots/ethereum/consensus/mess/MESSIntegrationSpec.scala

Test Specs (to understand test setup issues)

  • src/it/scala/com/chipprbots/ethereum/sync/E2EStateTestSpec.scala
  • src/it/scala/com/chipprbots/ethereum/sync/FastSyncItSpec.scala
  • src/it/scala/com/chipprbots/ethereum/sync/E2EFastSyncSpec.scala
  • src/it/scala/com/chipprbots/ethereum/network/E2EHandshakeSpec.scala

Network Protocol

  • src/main/scala/com/chipprbots/ethereum/network/rlpx/RLPxConnectionHandler.scala

Success Metrics

Before considering this issue resolved:

  1. ✅ All 37 test failures analyzed and categorized
  2. ⬜ Root cause identified for each failure category
  3. ⬜ Fixes implemented and validated locally
  4. ⬜ Nightly test suite re-run with 0 failures
  5. ⬜ No new test failures introduced by fixes
  6. ⬜ Documentation updated with any necessary configuration changes

Next Steps

  1. Create separate issues for each major category:
  2. Issue 1: Fix state synchronization timeout issues (28 tests)
  3. Issue 2: Fix MESS consensus chain selection (3 tests)
  4. Issue 3: Fix test data decoding for VM tests (1 test)

  5. Issue 4: Investigate remaining E2E sync issues (5 tests)

  6. Prioritize based on impact:

  7. Priority 1: State sync timeouts (affects 76% of failures)
  8. Priority 2: MESS consensus issues (affects consensus reliability)

  9. Priority 3: Individual test fixes

  10. Assign to appropriate developers with domain expertise:

  11. State sync/MPT → Blockchain sync team
  12. MESS consensus → Consensus team
  13. Test infrastructure → QA/Testing team

Appendix: Full Test Failure List

  1. ❌ VMTestsSpec: should load and parse a sample VM arithmetic test (83ms) - Decoding failure
  2. ❌ ForksTest: should execute blocks with respect to forks (1s 329ms) - Missing root node
  3. ❌ E2ESyncSpec: should handle blockchain reorganization correctly (1m 36s) - Timeout
  4. ❌ E2ESyncSpec: should continue sync after peer disconnection and reconnection (1m 37s) - Timeout
  5. ❌ MESSIntegrationSpec: should prefer recently seen chain over old chain with same difficulty (243ms) - Assertion
  6. ❌ MESSIntegrationSpec: should handle blocks without first-seen time using block timestamp (1ms) - Assertion
  7. ❌ MESSIntegrationSpec: should correctly handle chain reorganization scenario (3ms) - Assertion
  8. ❌ E2EHandshakeSpec: should handle bidirectional connection attempts (9s 126ms) - Timeout
  9. ❌ E2EHandshakeSpec: should handshake with peers at genesis (7s 155ms) - Timeout
  10. ❌ E2EStateTestSpec: should synchronize state trie between peers (1m 35s) - Timeout
  11. ❌ E2EStateTestSpec: should maintain state trie consistency across multiple state updates (1m 34s) - Timeout
  12. ❌ E2EStateTestSpec: should handle large state tries efficiently (1m 34s) - Timeout
  13. ❌ E2EStateTestSpec: should validate state roots during synchronization (1m 35s) - Timeout
  14. ❌ E2EStateTestSpec: should maintain state root consistency during chain reorganization (1m 37s) - Timeout
  15. ❌ E2EStateTestSpec: should propagate account state changes between peers (1m 34s) - Timeout
  16. ❌ E2EStateTestSpec: should handle rapid account state updates (1m 34s) - Timeout
  17. ❌ E2EStateTestSpec: should synchronize contract storage between peers (1m 35s) - Timeout
  18. ❌ E2EStateTestSpec: should handle storage updates across multiple blocks (1m 34s) - Timeout
  19. ❌ E2EStateTestSpec: should recover from partial state loss (1m 35s) - Timeout
  20. ❌ E2EStateTestSpec: should handle state synchronization with missing peers (1m 36s) - Timeout
  21. ❌ E2EStateTestSpec: should maintain state integrity during peer disconnection (1m 35s) - Timeout
  22. ❌ E2EStateTestSpec: should maintain state integrity across entire sync process (1m 34s) - Timeout
  23. ❌ E2EStateTestSpec: should verify state consistency after blockchain reorganization with state changes (1m 35s) - Timeout
  24. ❌ E2EStateTestSpec: should persist state correctly across peer restarts (1m 35s) - Timeout
  25. ❌ SNAPSyncIntegrationSpec: should initialize and distribute tasks to workers (5s 192ms) - Timeout
  26. ❌ SNAPSyncIntegrationSpec: should queue healing tasks for missing nodes via coordinator (3s 26ms) - Timeout
  27. ❌ FastSyncItSpec: should sync blockchain with state nodes (1m 38s) - Timeout
  28. ❌ FastSyncItSpec: should sync blockchain with state nodes when peer do not response with full responses (1m 38s) - Timeout
  29. ❌ FastSyncItSpec: should sync blockchain with state nodes when one of the peers send empty state responses (1m 38s) - Timeout
  30. ❌ FastSyncItSpec: should update pivot block and sync this new pivot block state (1m 34s) - Timeout
  31. ❌ FastSyncItSpec: should sync state to peer from partially synced state (1m 35s) - Timeout
  32. ❌ FastSyncItSpec: should follow the longest chains (1m 39s) - Timeout
  33. ❌ E2EFastSyncSpec: should successfully sync blockchain with state nodes (6s 719ms) - None was not defined
  34. ❌ E2EFastSyncSpec: should handle state synchronization from multiple peers (1m 36s) - Timeout
  35. ❌ E2EFastSyncSpec: should handle incomplete state downloads gracefully (6s 394ms) - None was not defined
  36. ❌ FastSyncItSpec: should handle state nodes with complex account structures (1m 35s) - Timeout
  37. ❌ E2EFastSyncSpec: should recover from peer disconnection during state download (7s 446ms) - None was not defined

Document Generated: 2025-12-19
Analyzed By: GitHub Copilot
Log Source: Night Run Log
Total Execution Time: 49m 10s
Success Rate: 79.9% (147/184 tests passed)