Peering Runbook¶

Audience: Operators managing network connectivity and peer relationships
Estimated Time: 15-30 minutes
Prerequisites: Running Fukuii node

Overview¶

This runbook covers peer discovery, network connectivity troubleshooting, and optimization of peer relationships in Fukuii. A healthy peer network is essential for reliable blockchain synchronization and staying up-to-date with the network.

Understanding Peering¶

Peer Types¶

Fukuii distinguishes between two types of peer connections:

Outgoing Peers: Connections initiated by your node
Default min: 20 peers
Default max: 50 peers
Your node actively seeks these connections
Incoming Peers: Connections from other nodes to yours
Default max: 30 peers
Requires open/forwarded ports
Indicates your node is publicly accessible

Network Protocols¶

Fukuii uses two network protocols:

Discovery Protocol (UDP)
Port: 30303 (default)
Purpose: Find peers on the network
Protocol: Ethereum Node Discovery Protocol v4
Ethereum Protocol (TCP)
Port: 9076 (default)
Purpose: Exchange blockchain data
Protocol: RLPx with ETH/66 capability

Healthy Peer Count¶

Minimum: 5-10 peers for basic operation
Typical: 20-40 peers for stable synchronization
Maximum: 80 total peers (50 outgoing + 30 incoming)

Peer Discovery Process¶

Bootstrap Process¶

When Fukuii starts, it follows this discovery sequence:

Load Known Nodes
Reads previously discovered peers from: ~/.fukuii/<network>/knownNodes.json
Enabled by default with reuse-known-nodes = true
Contact Bootstrap Nodes
Connects to hardcoded bootstrap nodes in network configuration
Bootstrap nodes are maintained by the ETC community
Perform Kademlia Lookup
Uses DHT (Distributed Hash Table) to discover more peers
Gradually builds routing table of network peers
Establish Connections
Attempts TCP connections to discovered peers
Performs RLPx handshake
Exchanges status and capabilities
Persist Known Nodes
Periodically saves discovered peers to disk
Interval: 20 seconds (default)
Max persisted: 200 nodes (default)

Configuration Parameters¶

Key configuration parameters (in base.conf):

fukuii.network {
  discovery {
    discovery-enabled = true
    reuse-known-nodes = true
    scan-interval = 2.minutes        # Reduced network overhead
    request-timeout = 3.seconds      # More tolerant of latency
    kademlia-timeout = 10.seconds    # More time for responses
    kademlia-bucket-size = 16
  }

  peer {
    min-outgoing-peers = 20
    max-outgoing-peers = 50
    max-incoming-peers = 30
    connect-retry-delay = 15.seconds  # Reduced connection churn
    connect-max-retries = 2           # Fail faster, try new peers
    wait-for-handshake-timeout = 10.seconds  # More tolerant of latency
    wait-for-tcp-ack-timeout = 15.seconds    # Prevent premature failures
    update-nodes-interval = 60.seconds       # Reduced reconnection attempts
    short-blacklist-duration = 3.minutes     # Faster retry for TooManyPeers
    long-blacklist-duration = 60.minutes     # Reasonable recovery time
  }

  known-nodes {
    persist-interval = 20.seconds
    max-persisted-nodes = 200
  }
}

fukuii.sync {
  peers-scan-interval = 5.seconds        # Reduced overhead
  blacklist-duration = 120.seconds       # Faster retry for transient issues
  critical-blacklist-duration = 60.minutes  # Still a penalty but allows recovery
  peer-response-timeout = 45.seconds     # More tolerant of peer load
}

Monitoring Peer Health¶

Check Current Peer Count¶

Using JSON-RPC:

curl -X POST --data '{"jsonrpc":"2.0","method":"net_peerCount","params":[],"id":1}' \
  http://localhost:8546

Expected response:

{
  "jsonrpc":"2.0",
  "id":1,
  "result":"0x14"  # Hex number, e.g., 0x14 = 20 peers
}

Get Detailed Peer Information¶

# Check if admin API is enabled (requires special configuration)
curl -X POST --data '{"jsonrpc":"2.0","method":"admin_peers","params":[],"id":1}' \
  http://localhost:8546

Note: admin_peers may not be available in production configurations for security reasons.

Monitor Logs for Peer Activity¶

tail -f ~/.fukuii/etc/logs/fukuii.log | grep -i peer

Key log patterns:

Good signs:

INFO  [PeerManagerActor] - Connected to peer: Peer(...)
INFO  [PeerActor] - Successfully handshaked with peer
INFO  [PeerDiscoveryManager] - Discovered X peers

Warning signs:

WARN  [PeerManagerActor] - Disconnected from peer: reason=...
WARN  [PeerActor] - Handshake timeout with peer
ERROR [ServerActor] - Failed to bind to port 9076

Check Network Connectivity¶

Verify your node is reachable from the internet:

# Check if discovery port is open (requires external tool)
# From another machine or online port checker:
nc -zvu <your-public-ip> 30303

# Check if P2P port is open
nc -zv <your-public-ip> 9076

Online port checkers: - https://canyouseeme.org/ - https://www.yougetsignal.com/tools/open-ports/

Troubleshooting Connectivity¶

Problem: Zero or Very Few Peers¶

Symptoms: - net_peerCount returns 0 or very low number (< 5) - Logs show No peers available - Sync is not progressing

Diagnostic Steps:

Verify network connectivity

ping 8.8.8.8
curl -I https://www.google.com

Check if discovery is enabled

Verify in your configuration or logs:

grep "discovery-enabled" ~/.fukuii/etc/logs/fukuii.log

Check ports are not blocked

# Check locally if ports are listening
sudo netstat -tulpn | grep -E "30303|9076"

Expected output:

udp6       0      0 :::30303              :::*                  <pid>/java
tcp6       0      0 :::9076               :::*                  <pid>/java

Check firewall rules

# Ubuntu/Debian
sudo ufw status

# RHEL/CentOS
sudo firewall-cmd --list-all

Solutions:

A. Enable discovery if disabled

Edit your configuration to ensure:

fukuii.network.discovery.discovery-enabled = true

B. Open firewall ports

# Ubuntu/Debian with ufw
sudo ufw allow 30303/udp
sudo ufw allow 9076/tcp

# RHEL/CentOS with firewalld
sudo firewall-cmd --permanent --add-port=30303/udp
sudo firewall-cmd --permanent --add-port=9076/tcp
sudo firewall-cmd --reload

C. Configure port forwarding

If behind NAT/router:

Log in to your router admin interface
Forward port 30303 (UDP) to your node's internal IP
Forward port 9076 (TCP) to your node's internal IP

Or enable UPnP in Fukuii config:

fukuii.network.automatic-port-forwarding = true

D. Manually add peers

If discovery fails, you can manually specify peers in your config:

fukuii.network.bootstrap-nodes = [
  "enode://pubkey@ip:port",
  "enode://pubkey@ip:port"
]

Find bootstrap nodes from: - Official ETC documentation - Community resources - Other node operators

E. Reset known nodes

If knownNodes.json is corrupted:

# Stop Fukuii
# Backup and remove known nodes
mv ~/.fukuii/etc/knownNodes.json ~/.fukuii/etc/knownNodes.json.bak
# Restart Fukuii

Problem: Peers Connecting but Quickly Disconnecting¶

Symptoms: - Peer count fluctuates rapidly - Logs show many disconnect messages - Synchronization is unstable

Common Causes:

Network incompatibility - Your node is on a different fork/network
Clock skew - System time is incorrect
Resource exhaustion - Node is overloaded
Firewall issues - Intermittent blocking

Diagnostic Steps:

Check system time

date
# Should be accurate to within a few seconds

Sync time if needed:

sudo ntpdate pool.ntp.org
# Or
sudo systemctl restart systemd-timesyncd

Check for network mismatch

Verify you're running the correct network:

# Check logs for network ID
grep "network" ~/.fukuii/etc/logs/fukuii.log | head -5

Monitor resource usage

# Check CPU, memory, disk I/O
top
iostat -x 1

Solutions:

A. Fix system time

# Install NTP
sudo apt-get install ntp  # Ubuntu/Debian
sudo systemctl enable ntp
sudo systemctl start ntp

B. Verify network configuration

Ensure you're running the correct network:

./bin/fukuii etc  # For ETC mainnet

C. Increase timeouts (if network latency is high)

In your configuration (values shown are examples of increased timeouts):

fukuii.network.peer {
  wait-for-hello-timeout = 10.seconds     # increase from default 5s
  wait-for-status-timeout = 45.seconds    # increase from default 30s
  wait-for-handshake-timeout = 15.seconds # increase from default 10s
  wait-for-tcp-ack-timeout = 20.seconds   # increase from default 15s
}

Problem: Only Outgoing Peers (No Incoming)¶

Symptoms: - All peers are outgoing connections - max-incoming-peers is never reached - Node works but is not contributing to network health

Cause: Your node is not publicly accessible (behind NAT without port forwarding)

Impact: - Your node works fine for syncing - Network health suffers if many nodes are not publicly accessible - You don't help other nodes discover the network

Solutions:

See "Configure port forwarding" section above. This is optional for personal nodes but recommended for public infrastructure.

Problem: High Peer Churn¶

Symptoms: - Constant connect/disconnect in logs - Peer count is unstable - Frequent "blacklisted peer" messages

Diagnostic Steps:

# Check for blacklist activity in logs
grep -i blacklist ~/.fukuii/etc/logs/fukuii.log | tail -20

Causes: - Incompatible peers (wrong network, old version) - Misbehaving peers - Network instability

Solutions:

This is usually normal behavior as Fukuii filters incompatible peers. However, if excessive:

Update to latest version - May have better peer filtering
Adjust peer limits - Temporarily increase max peers to compensate:
```
fukuii.network.peer.max-outgoing-peers = 60
```

Advanced Configuration¶

Optimizing for Fast Sync¶

For initial synchronization, maximize peers:

fukuii.network.peer {
  min-outgoing-peers = 30
  max-outgoing-peers = 60
}

After sync completes, reduce to stable values.

Optimizing for Bandwidth Conservation¶

For limited bandwidth scenarios:

fukuii.network.peer {
  min-outgoing-peers = 10
  max-outgoing-peers = 15
  max-incoming-peers = 10
}

Disabling Discovery (Static Peers Only)¶

For private networks or when you have a fixed set of peers:

fukuii.network {
  discovery.discovery-enabled = false
  discovery.reuse-known-nodes = false

  bootstrap-nodes = [
    "enode://pubkey1@ip1:port1",
    "enode://pubkey2@ip2:port2"
  ]
}

Warning: Only use this if you have reliable static peers. Otherwise, your node may become isolated.

Custom Discovery Settings¶

For specialized network environments:

fukuii.network.discovery {
  # Increase scan frequency for faster peer discovery (not recommended for production)
  scan-interval = 1.minute  # default: 2.minutes

  # Adjust Kademlia parameters
  kademlia-bucket-size = 20  # default: 16
  kademlia-alpha = 5  # default: 3 (higher = more aggressive discovery)

  # Adjust timeouts for high-latency networks
  request-timeout = 5.seconds  # default: 3.seconds
  kademlia-timeout = 15.seconds  # default: 10.seconds
}

Setting External Address¶

If your node has a public IP that differs from its local IP:

fukuii.network {
  discovery {
    host = "your.public.ip.address"
  }

  server-address {
    interface = "0.0.0.0"  # Listen on all interfaces
  }
}

Best Practices¶

For Home/Personal Nodes¶

Open ports if possible - Helps network health
Use default peer limits - Balanced for typical home connections
Enable discovery - Automatic peer management
Enable UPnP - Simplifies NAT traversal

For Production/Infrastructure Nodes¶

Allocate sufficient bandwidth - 1-10 Mbps minimum
Open all ports - Be a good network citizen
Monitor peer count - Alert if < 10 peers
Use static IP - Configure external address
Increase peer limits - Handle more connections if resources allow
Regular monitoring - Check peer health daily

For Private/Test Networks¶

Disable public discovery - Use static peers only
Configure bootstrap nodes - Point to your network's nodes
Adjust timeout values - May need tuning for test environments
Document peer topology - Maintain list of all network nodes

General Recommendations¶

Keep system time accurate - Use NTP
Monitor connection quality - Watch for high latency peers
Update regularly - New versions may improve peer management
Log peer activity - Helps diagnose issues
Backup known nodes - Can speed up recovery after restarts

Monitoring and Alerting¶

Metrics to Monitor¶

Set up alerts for:

# Peer count below threshold
net_peerCount < 10

# No peers for extended period
net_peerCount == 0 for > 5 minutes

# Excessive peer churn
peer_disconnect_rate > 10 per minute

Using Prometheus¶

If metrics are enabled, query peer metrics:

curl http://localhost:9095/metrics | grep peer

Example Prometheus alert:

- alert: LowPeerCount
  expr: ethereum_peer_count < 10
  for: 5m
  annotations:
    summary: "Fukuii node has low peer count"
    description: "Node {{ $labels.instance }} has only {{ $value }} peers"

First Start - Initial node setup including network configuration
Log Triage - Analyzing peer-related log messages
Known Issues - Common networking problems

Peering Runbook¶

Overview¶

Table of Contents¶

Understanding Peering¶

Peer Types¶

Network Protocols¶

Healthy Peer Count¶

Peer Discovery Process¶

Bootstrap Process¶

Configuration Parameters¶

Monitoring Peer Health¶

Check Current Peer Count¶

Get Detailed Peer Information¶

Monitor Logs for Peer Activity¶

Check Network Connectivity¶

Troubleshooting Connectivity¶

Problem: Zero or Very Few Peers¶

Problem: Peers Connecting but Quickly Disconnecting¶

Problem: Only Outgoing Peers (No Incoming)¶

Problem: High Peer Churn¶

Advanced Configuration¶

Optimizing for Fast Sync¶

Optimizing for Bandwidth Conservation¶

Disabling Discovery (Static Peers Only)¶

Custom Discovery Settings¶

Setting External Address¶

Best Practices¶

For Home/Personal Nodes¶

For Production/Infrastructure Nodes¶

For Private/Test Networks¶

General Recommendations¶

Monitoring and Alerting¶

Metrics to Monitor¶

Using Prometheus¶

Related Runbooks¶

Further Reading¶