Root Cause Investigation: Category 1 State Sync Timeout Failures¶

Investigation Date¶

December 19-20, 2025

Issue Summary¶

28 out of 37 test failures (76%) in the nightly run are state synchronization timeouts. Tests consistently fail with:

PEER_REQUEST_TIMEOUT: peer=PeerId(...), reqType=GetNodeData, elapsed=5019ms (timeout=5000ms)
java.util.concurrent.TimeoutException: Task time out after all retries

Root Cause Analysis¶

Primary Issue: Hardcoded 5-Second Timeout in Integration Tests¶

File: src/it/scala/com/chipprbots/ethereum/sync/util/CommonFakePeer.scala
Line: 77
Problem: implicit val akkaTimeout: Timeout = Timeout(5.second)

This 5-second timeout is used for all Pekko actor interactions in integration tests, including peer requests for state data. In CI environments (GitHub Actions), this timeout is insufficient due to:

Shared Resources: CI runners share CPU, memory, and I/O with other jobs
Higher Latency: Network operations and disk I/O are slower in containerized CI environments
State Data Size: GetNodeData requests for MPT nodes can be large and take time to serialize/deserialize
Concurrent Operations: Multiple test peers requesting data simultaneously increase contention

Secondary Issue: Integration Test Configuration Missing Sync Timeout Override¶

File: src/it/resources/application.conf
Problem: Missing sync.peer-response-timeout configuration override

The integration test configuration only overrides the data directory but doesn't adjust sync timeouts for the CI environment. This means tests rely on either: - Base configuration timeout (45 seconds) - but Pekko timeout (5 seconds) triggers first - Test configuration timeout (3 seconds in src/test/resources/application.conf)

Evidence from Logs¶

From the nightly run log analysis: - 28 tests timeout at exactly 5000ms (5 seconds) - Most are in E2EStateTestSpec (15), FastSyncItSpec (8), and related suites - All show PEER_REQUEST_TIMEOUT for GetNodeData requests - Tests that pass locally take 1-3 seconds, suggesting CI is 2-3x slower - Timeout occurs before the retry mechanism in SyncCommonItSpecUtils can complete

Why This Affects State Sync Tests Specifically¶

MPT Node Retrieval: State sync tests require downloading Merkle Patricia Trie nodes via GetNodeData requests
Large Payloads: State nodes can be large, especially for accounts with storage
Sequential Dependencies: Each missing node blocks progress, requiring another request
Test Complexity: E2E state tests create full blockchain scenarios with multiple peers

Solution Implemented¶

Fix 1: Dynamic Timeout Based on Environment¶

File: src/it/scala/com/chipprbots/ethereum/sync/util/CommonFakePeer.scala

// Before (Line 77):
implicit val akkaTimeout: Timeout = Timeout(5.second)

// After (Lines 77-81):
// Use longer timeout in CI environment to accommodate slower I/O and network operations
// CI environments (GitHub Actions, etc.) often have higher latency due to shared resources
private val baseTimeout = 5.seconds
private val ciMultiplier = sys.env.get("CI").map(_ => 6).getOrElse(1) // 30 seconds in CI, 5 seconds locally
implicit val akkaTimeout: Timeout = Timeout(baseTimeout * ciMultiplier)

Rationale: - Detects CI environment via CI environment variable (set by GitHub Actions) - Uses 6x multiplier (30 seconds) in CI, maintaining 5 seconds locally - Preserves fast local test execution while accommodating CI latency - 30 seconds aligns with other sync timeouts and allows time for retries

Fix 2: Integration Test Configuration Override¶

File: src/it/resources/application.conf

sync {
  # Increase timeout for integration tests to accommodate CI environment latency
  # CI environments have slower I/O and shared resources causing longer response times
  peer-response-timeout = 30.seconds
}

Rationale: - Overrides the base configuration peer-response-timeout for integration tests - 30 seconds matches the CI-adjusted Pekko timeout - Consistent with production sync timeout (45 seconds) scaled for test scenarios - Prevents premature timeout in PeerRequestHandler

Expected Impact¶

Tests Expected to Pass After Fix¶

Definite (28 tests - all timeouts): 1. E2EStateTestSpec (15 tests) - All state sync scenarios 2. FastSyncItSpec (8 tests) - All fast sync with state node tests 3. E2ESyncSpec (2 tests) - Blockchain reorganization and reconnection 4. SNAPSyncIntegrationSpec (2 tests) - SNAP sync coordinator tests 5. E2EFastSyncSpec (1 test) - State sync from multiple peers

Probable (3-5 additional tests - cascading failures): 6. E2EFastSyncSpec (2-3 tests) - "None was not defined" errors likely caused by incomplete sync due to timeouts 7. ForksTest (1 test) - Missing MPT root node likely from timeout during state setup 8. E2EHandshakeSpec (0-2 tests) - May still need separate investigation if not timeout-related

Total Expected Resolution: 28-33 of 37 failures (76-89%)

Remaining Work¶

After this fix is validated, the following issues should be investigated separately:

MESS Consensus Issues (3 tests, P1) - Chain selection logic
VM Test Data Decoding (1 test, P2) - JSON format compatibility
E2EHandshakeSpec Issues (0-2 tests, P2) - If not resolved by timeout fix

Validation Plan¶

Local Testing (Optional)¶

# Run single failing test locally
sbt "IntegrationTest/testOnly com.chipprbots.ethereum.sync.E2EStateTestSpec -- -z \"synchronize state trie between peers\""

# Run all state sync tests
sbt "IntegrationTest/testOnly com.chipprbots.ethereum.sync.E2EStateTestSpec"

# Run fast sync tests
sbt "IntegrationTest/testOnly com.chipprbots.ethereum.sync.FastSyncItSpec"

CI Testing (Required)¶

# Run full integration test suite
sbt "IntegrationTest/test"

# Or run comprehensive suite (includes all integration tests)
sbt testComprehensive

Success Criteria¶

E2EStateTestSpec: 15/15 tests pass
FastSyncItSpec: 8/8 tests pass
E2ESyncSpec: 2/2 tests pass (or identify as separate issue)
SNAPSyncIntegrationSpec: 2/2 tests pass
No new test failures introduced
Test execution time remains reasonable (< 2 hours for full suite)
Log shows successful peer requests within new timeout threshold

Technical Details¶

Timeout Flow in Tests¶

Test starts → Creates FakePeer instances
FakePeer initialization → Sets akkaTimeout from CommonFakePeer
Sync request sent → Uses PeerRequestHandler with syncConfig.peerResponseTimeout
Both timeouts active:
Pekko actor timeout (Akka ask pattern): Now 30s in CI
Peer response timeout (application level): Now 30s in CI
Whichever timeout triggers first → Cancels the request

Previously, Pekko timeout (5s) triggered before peer response timeout could complete, causing false failures.

Configuration Precedence¶

Integration Tests Configuration Cascade:
1. Base configuration: conf/base.conf (45s timeout)
2. Test overrides: src/test/resources/application.conf (3s timeout) 
3. Integration test overrides: src/it/resources/application.conf (30s timeout) ✅ NEW
4. System properties: -Dfukuii.sync.peer-response-timeout=X

The integration tests now use 30s from their specific configuration, overriding the base config.

Why 30 Seconds?¶

Based on log analysis and best practices: - Observed CI slowdown: 2-3x slower than local (5s → 15s typical) - Safety margin: 2x additional buffer for variance (15s → 30s) - Retry budget: Allows ~2-3 retries within test timeout window - Production alignment: Scaled from production 45s timeout - Test efficiency: Short enough to fail fast on real issues

Monitoring and Validation¶

After merging, monitor the following metrics in nightly runs:

Test Success Rate: Should increase from 79.9% to ~95%+ (if remaining 6-9 issues persist)
Timeout Logs: Should see fewer PEER_REQUEST_TIMEOUT errors
Test Duration: E2E state tests may take slightly longer but within limits
False Positives: Watch for tests that now timeout at 30s (indicates deeper issues)

References¶

Issue: Review Last Night Run and Troubleshoot 37 Failures
Full Analysis: docs/troubleshooting/NIGHTLY_RUN_ANALYSIS_2025-12-19.md
Tracking Doc: docs/troubleshooting/FAILURE_TRACKING.md
Log Source: Nightly workflow run #47, December 19, 2025

Changes Made¶

Modified Files¶

src/it/scala/com/chipprbots/ethereum/sync/util/CommonFakePeer.scala
Added CI environment detection
Implemented dynamic timeout scaling (5s → 30s in CI)
src/it/resources/application.conf
Added sync.peer-response-timeout = 30.seconds override

No Changes Required¶

Production configuration remains unchanged (45s is appropriate for production)
Unit tests remain unchanged (faster, simpler scenarios don't need longer timeouts)
Test retry logic remains unchanged (90 retries with 1s delay is sufficient)

Prepared by: @copilot
Date: December 19-20, 2025
Status: Fix implemented, awaiting validation
Next Action: Run nightly workflow to validate fix