Skip to content

ADR-016: ETH66+ Protocol-Aware Message Formatting

Historical note: ETH wire protocols 63–67 have been removed from fukuii as of the ETH69 alignment sprint (2026). This ADR documents the original decision and remains valid as historical record. Current live protocols: ETH68, ETH69, ETH70.

Status

Accepted

Context

During investigation of peer connection failures in sync tests (Issue #441), we discovered a critical message format mismatch that prevented peers from recognizing each other as available for synchronization after successful RLPx handshake.

Initial Problem Statement

  • RLPx handshake completing successfully and negotiating to ETH68 protocol
  • Peers entering "FULLY ESTABLISHED" state with proper capability negotiation
  • GetBlockHeaders requests sent immediately after handshake
  • Zero peers available for sync: "Cannot pick pivot block. Need at least 1 peers, but there are only 0 which meet the criteria"
  • Peers having maxBlockNumber = 0 despite successful status exchange
  • Tests timing out after 2+ minutes waiting for sync to start

Investigation Timeline

Phase 1: Initial Hypothesis - Message Decoding Failure

Symptoms from logs (chippr-robotics/fukuii#437): 

[RLPx] Cannot decode message from 127.0.0.1:36185, because of Cannot decode GetBlockHeaders

Initial Fix (Commit 4458be6): - Added backward-compatible fallback decoding in ETH66.scala - Decoders now accept both ETH62 format (4 fields) and ETH66 format (5 fields with requestId) - Example for GetBlockHeaders: 

// ETH66+ format: [requestId, [block, maxHeaders, skip, reverse]]
case RLPList(RLPValue(requestIdBytes), RLPList(...)) => decode with requestId

// Fallback to ETH62 format: [block, maxHeaders, skip, reverse]
case RLPList(RLPValue(blockBytes), RLPValue(maxHeadersBytes), ...) => decode with requestId=0

Result: Eliminated "Cannot decode" errors, but peers still not available for sync

Phase 2: Type Mismatch Discovery

Symptoms: - No decode errors in logs anymore - Handshakes completing successfully - Peers still showing maxBlockNumber = 0 - PivotBlockSelector reporting "0 peers meet criteria"

Root Cause Identified: After protocol negotiation to ETH68: 1. Decoders: ETH68MessageDecoder uses ETH66.BlockHeaders decoders → creates ETH66.BlockHeaders instances 2. Pattern Matches: Code imports ETH62.BlockHeaders and pattern matches fail silently 3. Result: Incoming BlockHeaders responses don't match pattern, get ignored, maxBlockNumber never updated

Key Files Affected: - NetworkPeerManagerActor.scala - Pattern match on BlockHeaders in updateMaxBlock() and updateForkAccepted() - PivotBlockSelector.scala - Pattern match on MessageFromPeer(blockHeaders: BlockHeaders, ...) - FastSync.scala - ResponseReceived with BlockHeaders - HeadersFetcher.scala - AdaptedMessage with BlockHeaders

First Attempt Fix (Commit e8bd068 + mithril agent work): - Updated imports to alias both types: ETH62.{BlockHeaders => ETH62BlockHeaders}, ETH66.{BlockHeaders => ETH66BlockHeaders} - Added pattern matches for both: case ETH62BlockHeaders(headers) and case ETH66BlockHeaders(_, headers) - Updated message sending to use ETH66GetBlockHeaders(0, ...)

Result: Improved, but violated protocol consistency

Phase 3: Core-Geth Analysis - Protocol-Aware Solution

New Requirement: Don't mix message formats - if ETH68 is negotiated, use ETH68 format consistently

Core-Geth Investigation (https://github.com/etclabscore/core-geth): 

// core-geth always uses GetBlockHeadersPacket with RequestId for ETH66+
type GetBlockHeadersPacket struct {
    RequestId uint64
    *GetBlockHeadersRequest
}

// Example usage - no version checking, format is implicit
req := &Request{
    code: GetBlockHeadersMsg,
    want: BlockHeadersMsg,
    data: &GetBlockHeadersPacket{
        RequestId: id,
        GetBlockHeadersRequest: &GetBlockHeadersRequest{...},
    },
}

Key Findings: 1. Core-geth always uses RequestId wrapper when protocol is ETH66+ 2. No explicit version checking - format is implicit from protocol negotiation 3. Consistent format per connection - never mixes ETH62 and ETH66 formats 4. Single message type hierarchy - no separate ETH62 vs ETH66 classes

Fukuii's Architecture Issue: - Separate type hierarchies: ETH62.GetBlockHeaders vs ETH66.GetBlockHeaders are different classes - Import determines type: import ETH62.GetBlockHeaders hardcoded in most files - Decoder mismatch: ETH68MessageDecoder creates ETH66.GetBlockHeaders, but code expects ETH62.GetBlockHeaders

Decision Point

We have PeerInfo.remoteStatus.capability (type: Capability) storing negotiated protocol: - ETH63, ETH64, ETH65 → pre-ETH66 (no RequestId; ETC64 retired) - ETH66, ETH67, ETH68 → ETH66+ (with RequestId)

Options Considered:

  1. Unify type hierarchy (like core-geth) - rejected as too invasive
  2. Always send ETH66 format - rejected as breaks pre-ETH66 peer compatibility
  3. Protocol-aware message creation - selected

Decision

Implemented: Protocol-Aware Message Formatting

We implement a system where message format is determined by the peer's negotiated capability, with defensive pattern matching for robustness.

Component 1: Capability Helper Method

Location: src/main/scala/com/chipprbots/ethereum/network/p2p/messages/Capability.scala

def usesRequestId(capability: Capability): Boolean = capability match {
  case Capability.ETH66 | Capability.ETH67 | Capability.ETH68 => true
  case _ => false // ETH63, ETH64, ETH65
}

Rationale: Centralized capability detection prevents inconsistent checks across codebase

Component 2: Protocol-Aware Message Creation

Pattern Applied: 

// When sending GetBlockHeaders
val message = if (Capability.usesRequestId(peerInfo.remoteStatus.capability)) {
  ETH66GetBlockHeaders(requestId = 0, block, maxHeaders, skip, reverse)
} else {
  ETH62GetBlockHeaders(block, maxHeaders, skip, reverse)
}

Files Updated: 1. NetworkPeerManagerActor.scala - Sends GetBlockHeaders after handshake 2. PivotBlockSelector.scala - Sends GetBlockHeaders for pivot block selection 3. FastSync.scala - Sends GetBlockHeaders during header chain sync 4. FastSyncBranchResolverActor.scala - Sends GetBlockHeaders for branch resolution 5. EtcForkBlockExchangeState.scala - Sends GetBlockHeaders during fork verification 6. PeersClient.scala - Adapts messages based on selected peer capability

Rationale: Each peer connection uses consistent message format based on negotiated protocol

Component 3: Dual-Format Pattern Matching

Pattern Applied: 

// Receiving BlockHeaders - must handle both formats
message match {
  case ETH62BlockHeaders(headers) => 
    // Handle ETH62 format (from pre-ETH66 peers)
    processHeaders(headers)

  case ETH66BlockHeaders(requestId, headers) =>
    // Handle ETH66 format (from ETH66+ peers)
    processHeaders(headers) // requestId often ignored in response handling

  case _ => // other messages
}

Files Updated: 1. NetworkPeerManagerActor.scala - updateForkAccepted(), updateMaxBlock() 2. BlockFetcher.scala - Response handling 3. HeadersFetcher.scala - Response handling 4. FastSync.scala - Response handling 5. PivotBlockSelector.scala - Voting process 6. FastSyncBranchResolverActor.scala - Binary search handling

Rationale: - Nodes connect to peers with different protocol versions simultaneously - Must handle responses in format matching what was sent - Defensive programming for protocol deviations (see ADR-011)

Component 4: Type Adaptation in PeersClient

Special Case: PeersClient handles request/response matching generically

private def adaptMessageForPeer(
    message: MessageSerializable,
    peer: Peer,
    peerInfo: PeerInfo
): MessageSerializable = {
  val usesRequestId = peerInfo.remoteStatus.capability.usesRequestId

  message match {
    case ETH66GetBlockHeaders(requestId, block, maxHeaders, skip, reverse) if !usesRequestId =>
      // Convert to ETH62 for pre-ETH66 peer
      ETH62GetBlockHeaders(block, maxHeaders, skip, reverse)

    case ETH62GetBlockHeaders(block, maxHeaders, skip, reverse) if usesRequestId =>
      // Convert to ETH66 for ETH66+ peer
      ETH66GetBlockHeaders(0, block, maxHeaders, skip, reverse)

    case other => other
  }
}

Rationale: Generic request/response infrastructure needs runtime adaptation

Kept: Backward-Compatible Decoders

Location: src/main/scala/com/chipprbots/ethereum/network/p2p/messages/ETH66.scala

The fallback decoding from Phase 1 (commit 4458be6) is retained for robustness: - Handles protocol deviations by peers (see ADR-011 for precedent) - Provides defensive layer against implementation errors - Minimal performance impact (fast-path check fails quickly)

Consequences

Positive

  1. Protocol Compliance: Matches core-geth behavior - consistent format per peer connection
  2. Backward Compatibility: Works with both pre-ETH66 (ETH63-65) and ETH66+ (ETH66-68) peers
  3. Type Safety: Leverages Scala's type system and pattern matching for correctness
  4. Defensive: Handles both expected format and potential deviations
  5. Peer Recognition: maxBlockNumber correctly updated, peers available for sync
  6. Tests Pass: Expected to fix 18 failing integration tests in FastSyncItSpec and RegularSyncItSpec

Negative

  1. Code Duplication: Pattern matches duplicated for ETH62 and ETH66 variants
  2. Type Complexity: Developers must understand two type hierarchies
  3. Import Management: Must carefully manage aliased imports
  4. Runtime Checks: Protocol version checked at runtime (not compile time)

Neutral

  1. Migration Path: Future Scala versions might allow more elegant type unification
  2. Core-Geth Alignment: Architecture still differs from core-geth but behavior aligns
  3. Maintenance Burden: New message types require both ETH62 and ETH66 variants

Implementation Details

Testing Methodology

Test Environment: Local integration tests with multiple peer instances Test Scenarios: 1. Peers negotiating to ETH68 - should use ETH66 format messages 2. Peers negotiating to ETH64 - should use ETH62 format messages 3. Mixed network - some ETH66+, some pre-ETH66 peers 4. Message format verification through logging

Key Validation Points: - ✅ RLPx handshake completes - ✅ Capability negotiation succeeds - ✅ GetBlockHeaders sent in correct format - ✅ BlockHeaders responses received and decoded - ✅ maxBlockNumber updated in PeerInfo - ✅ PivotBlockSelector finds available peers - ✅ Sync proceeds successfully

Code Locations

Core Infrastructure: - Capability.scala - Protocol version detection helper - ETH66.scala - Backward-compatible decoders (Phase 1) - MessageDecoders.scala - Protocol-specific decoder selection

Message Sending (protocol-aware creation): - NetworkPeerManagerActor.scala:109 - Post-handshake GetBlockHeaders - PivotBlockSelector.scala:230 - Pivot block header request - FastSync.scala:851 - Header chain sync request - FastSyncBranchResolverActor.scala:179 - Branch resolution request - EtcForkBlockExchangeState.scala:25 - Fork verification request - PeersClient.scala - Generic message adaptation

Message Receiving (dual-format pattern matching): - NetworkPeerManagerActor.scala:199-235 - updateForkAccepted - NetworkPeerManagerActor.scala:264-269 - updateMaxBlock - PivotBlockSelector.scala:137 - Voting process - FastSync.scala:219 - Response handling - HeadersFetcher.scala:54,84 - Response handling - BlockFetcher.scala:329 - Response handling - FastSyncBranchResolverActor.scala:77,94 - Response handling

Migration Notes for Developers

When adding new message types: 1. Create both ETH62 and ETH66 variants if request/response pair 2. Add decoders in both ETH62.scala and ETH66.scala 3. Add backward-compatible fallback in ETH66 decoder 4. Update MessageDecoders.scala for all protocol versions 5. Use protocol-aware creation pattern in application code 6. Handle both variants in pattern matches

When debugging message issues: 1. Check peer's remoteStatus.capability - determines expected format 2. Verify decoder selection in MessageDecoders 3. Look for type mismatches in pattern matches 4. Enable RLPx debug logging for wire format inspection

Alternatives Considered

Alternative 1: Unified Message Type Hierarchy

Description: Refactor to single message type hierarchy like core-geth 

case class GetBlockHeaders(
  requestId: Option[BigInt],  // None for pre-ETH66, Some for ETH66+
  block: Either[BigInt, ByteString],
  maxHeaders: BigInt,
  skip: BigInt,
  reverse: Boolean
)

Rejected Because: - Massive refactoring across entire codebase - Risk of introducing consensus bugs - Breaks type safety (optional requestId) - Not minimal change per requirements

Alternative 2: Always Use ETH66 Format

Description: Send ETH66 format to all peers, rely on backward-compatible decoders 

// Always send ETH66
peer.ref ! SendMessage(ETH66GetBlockHeaders(0, block, maxHeaders, skip, reverse))

Rejected Because: - Violates Ethereum protocol specifications - Pre-ETH66 peers expect ETH62 format - Could cause interoperability issues with strict clients - No alignment with core-geth behavior

Alternative 3: Runtime Message Conversion Layer

Description: Add middleware that converts messages based on capability 

trait MessageAdapter {
  def adapt(msg: Message, capability: Capability): Message
}

Rejected Because: - Additional complexity layer - Performance overhead on hot path - Doesn't solve pattern matching issue - Harder to debug than explicit protocol-aware creation

Alternative 4: Implicit Conversion Between Types

Description: Use implicit conversions to automatically convert ETH62 ↔ ETH66 

implicit def eth62ToEth66(msg: ETH62.GetBlockHeaders): ETH66.GetBlockHeaders = ???

Rejected Because: - Hidden behavior (implicit conversions are invisible) - Doesn't solve when to use which type - Scala 3 deprecates some implicit patterns - Makes debugging harder

References

Specifications

  1. Ethereum Wire Protocol (ETH)
  2. ETH/66 Change Log
  3. Ethereum Execution Specs

Implementation References

  1. Core-Geth - ETC reference implementation
  2. eth/protocols/eth/protocol.go - Message type definitions
  3. eth/protocols/eth/peer.go - Message creation
  4. eth/protocols/eth/handlers.go - Message handling
  5. Go Ethereum (Geth) - Upstream reference
  6. Besu - Java-based Ethereum client
  1. CON-001: RLPx Protocol Deviations - Defensive protocol handling precedent
  2. CON-003: Block Sync Improvements - Fast sync architecture
  1. chippr-robotics/fukuii#441 - Peer connection errors
  2. chippr-robotics/fukuii#437 - Previous investigation logs

Future Work

Short Term

  1. Compilation Verification: Ensure all changes compile successfully
  2. Integration Testing: Run full FastSyncItSpec and RegularSyncItSpec test suites
  3. Performance Testing: Measure impact of runtime capability checks
  4. Log Analysis: Verify correct message formats in actual network conditions

Medium Term

  1. Type Unification Study: Evaluate Scala 3 features (union types, opaque types) for cleaner architecture
  2. Message Format Metrics: Add monitoring for ETH62 vs ETH66 message usage
  3. Protocol Version Analytics: Track which protocols are actually used in network
  4. Documentation: Add developer guide for protocol-aware message handling

Long Term

  1. Architecture Evolution: Consider message type redesign if Scala 3 enables better patterns
  2. ETH/69+ Support: Ensure architecture supports future protocol versions
  3. Protocol Negotiation Enhancement: Explore capability-based feature negotiation
  4. Cross-Client Testing: Automated testing against multiple client implementations

Lessons Learned

  1. Type Systems Have Limits: Separate type hierarchies for protocol versions create maintenance burden but provide type safety
  2. Runtime Checks Are Sometimes Necessary: Not everything can be compile-time verified in distributed systems
  3. Defensive Programming Pays Off: Backward-compatible decoders caught issues that perfect protocol compliance wouldn't
  4. Reference Implementations Matter: Core-geth analysis revealed the "right" approach
  5. Pattern Matching Is Powerful: Handling both message formats via pattern matching is elegant and maintainable
  6. Minimal Changes Are Hard: "Just add protocol awareness" touched 10+ files across multiple subsystems
  7. Integration Tests Reveal Truth: Unit tests can't catch peer protocol mismatch issues
  8. Documentation Prevents Repeats: Future developers need clear guidance on protocol-aware patterns

Decision Log

  • 2025-11-16 05:00 UTC: Initial investigation started - "Cannot decode GetBlockHeaders" errors
  • 2025-11-16 05:15 UTC: Added backward-compatible fallback decoders (commit 4458be6)
  • 2025-11-16 05:30 UTC: Identified type mismatch as root cause
  • 2025-11-16 05:45 UTC: Attempted mixed message format approach (commit e8bd068 + mithril work)
  • 2025-11-16 06:00 UTC: Analyzed core-geth for protocol-aware pattern
  • 2025-11-16 06:15 UTC: Implemented protocol-aware message creation (forge agent)
  • 2025-11-16 06:30 UTC: Documented findings in ADR-016
  • 2025-11-16: Next - compilation verification and integration testing

Appendix: Message Format Examples

ETH62 Format (Pre-ETH66)

GetBlockHeaders message:
RLP: [block, maxHeaders, skip, reverse]
Bytes: 0xc4 0x01 0x01 0x00 0x00
       └─ RLPList with 4 items

BlockHeaders response:
RLP: [header1, header2, ...]
Bytes: 0xf8 0x... (list of headers)

ETH66 Format (ETH66+)

GetBlockHeaders message:
RLP: [requestId, [block, maxHeaders, skip, reverse]]

For requestId=0 (empty bytes per RLP spec):
Bytes: 0xc6 0x80 0xc4 0x01 0x01 0x00 0x00
       │    │    └─ Inner RLPList: [block=1, maxHeaders=1, skip=0, reverse=0]
       │    └─ requestId=0 encoded as 0x80 (empty byte string, NOT 0x00)
       └─ Outer RLPList marker

For requestId=42:
Bytes: 0xc6 0x2a 0xc4 0x01 0x01 0x00 0x00
            └─ requestId=42 encoded as 0x2a (single byte < 0x80)

IMPORTANT: Per Ethereum RLP specification, integer 0 MUST be encoded as 
an empty byte string (0x80), not as a single byte 0x00. This is critical
for interoperability with core-geth and other Ethereum clients.

BlockHeaders response:
RLP: [requestId, [header1, header2, ...]]
Bytes: 0x... 0x80 0xf8 0x...
       └─ RLPList with 2 items (requestId + headers list)

Capability Detection

// Example peer capabilities after negotiation
val peer1Capability = Capability.ETH68  // Uses ETH66 format
val peer2Capability = Capability.ETH64  // Uses ETH62 format
val peer3Capability = Capability.ETH65  // Uses ETH62 format

peer1Capability.usesRequestId  // true
peer2Capability.usesRequestId  // false
peer3Capability.usesRequestId  // false