Backup & Restore Runbook¶

Audience: Operators managing data protection and disaster recovery
Estimated Time: 1-3 hours (depending on data size)
Prerequisites: Running Fukuii node, sufficient backup storage

Overview¶

This runbook covers backup strategies, restoration procedures, and disaster recovery planning for Fukuii nodes. Proper backups are essential for protecting against data loss from hardware failures, corruption, or operational errors.

Table of Contents¶

Backup Strategies
What to Backup
Backup Procedures
Restore Procedures
Portable Chain Snapshots (Export / Import)
Disaster Recovery
Testing and Validation

Backup Strategies¶

Strategy Comparison¶

Legend: - RTO = Recovery Time Objective (how long to restore) - RPO = Recovery Point Objective (how much data loss)

Strategy	RTO	RPO	Storage Cost	Complexity	Use Case
Full Backup	Hours	24h	High	Low	Development
Incremental	1-2h	1h	Medium	Medium	Production
Snapshot	Minutes	Minutes	Medium	Medium	Cloud/VM
Live Replication	Seconds	Seconds	High	High	Critical
Hybrid	30m-1h	30m	Medium-High	Medium	Recommended

Recommended Strategy¶

For most production deployments, use a hybrid approach:

Critical data (keys, config): Frequent backups (hourly) to multiple locations
Blockchain database: Periodic backups (daily/weekly) + on-demand before major changes
Known nodes: Daily backups
Logs: Optional (can be retained but not critical for recovery)

What to Backup¶

Essential Files (MUST backup)¶

These are small but critical:

~/.fukuii/etc/
├── node.key                    # ~100 bytes - CRITICAL
├── keystore/                   # ~1 KB per key - CRITICAL
│   └── UTC--2024...
├── app-state.json              # ~1 KB - Important
└── knownNodes.json             # ~50 KB - Helpful

Priority: HIGHEST - These files are small and cannot be recreated.

Database (Optional but recommended)¶

~/.fukuii/etc/rocksdb/          # 300-400 GB - Large but valuable
├── blockchain/
└── state/

Priority: MEDIUM - Can be re-synced from network (takes days) but backup saves time.

Configuration Files¶

/path/to/fukuii/conf/
├── custom.conf                 # Your custom configuration
└── .jvmopts                    # JVM tuning parameters

Priority: HIGH - Small files that define your node's behavior.

Logs (Usually not needed)¶

~/.fukuii/etc/logs/             # ~500 MB - Rotated automatically

Priority: LOW - Useful for debugging but not needed for recovery.

Backup Size Estimates¶

Component	Size	Backup Frequency	Storage (1 month)
Keys + Config	~1 MB	Daily	~30 MB
Known Nodes	~50 KB	Daily	~1.5 MB
Database	~350 GB	Weekly	~1.4 TB
Total	~350 GB	Mixed	~1.4 TB

Backup Procedures¶

Method 1: Essential Files Only (Recommended for All)¶

Backs up critical files that cannot be recreated.

Frequency: Daily (or after any key generation)
Duration: < 1 minute
Storage: < 10 MB

#!/bin/bash
# backup-essentials.sh

DATADIR=~/.fukuii/etc
BACKUP_DIR=/backup/fukuii/essentials
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_PATH="$BACKUP_DIR/fukuii-essentials-$DATE"

mkdir -p "$BACKUP_PATH"

# Backup critical files
cp "$DATADIR/node.key" "$BACKUP_PATH/" 2>/dev/null || echo "No node.key"
cp -r "$DATADIR/keystore" "$BACKUP_PATH/" 2>/dev/null || echo "No keystore"
cp "$DATADIR/app-state.json" "$BACKUP_PATH/" 2>/dev/null || echo "No app-state"
cp "$DATADIR/knownNodes.json" "$BACKUP_PATH/" 2>/dev/null || echo "No knownNodes"

# Create archive
cd "$BACKUP_DIR"
tar -czf "fukuii-essentials-$DATE.tar.gz" "fukuii-essentials-$DATE/"
rm -rf "fukuii-essentials-$DATE/"

# Keep only last 30 backups
ls -t fukuii-essentials-*.tar.gz | tail -n +31 | xargs rm -f

echo "Backup completed: fukuii-essentials-$DATE.tar.gz"

Schedule with cron:

# Daily at 3 AM
0 3 * * * /path/to/backup-essentials.sh

Method 2: Full Database Backup (Offline)¶

Complete backup including blockchain database.

Frequency: Weekly or before major upgrades
Duration: 30-60 minutes (depending on disk speed)
Storage: ~350 GB per backup

Important: Stop the node first for consistent backup.

#!/bin/bash
# backup-full-offline.sh

DATADIR=~/.fukuii/etc
BACKUP_DIR=/backup/fukuii/full
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_PATH="$BACKUP_DIR/fukuii-full-$DATE"

# Stop Fukuii
echo "Stopping Fukuii..."
# For systemd:
# sudo systemctl stop fukuii
# For Docker:
# docker stop fukuii
# For screen/tmux: send stop command or kill process
pkill -f fukuii || echo "Fukuii not running"

sleep 10  # Wait for clean shutdown

# Create backup
echo "Creating backup..."
mkdir -p "$BACKUP_DIR"
rsync -avh --progress "$DATADIR/" "$BACKUP_PATH/"

# Create compressed archive (optional, saves space but takes longer)
# tar -czf "$BACKUP_DIR/fukuii-full-$DATE.tar.gz" -C "$BACKUP_DIR" "fukuii-full-$DATE"
# rm -rf "$BACKUP_PATH"

# Restart Fukuii
echo "Restarting Fukuii..."
# ./bin/fukuii etc &
# Or restore your startup method

echo "Backup completed: $BACKUP_PATH"

Method 3: Live Database Backup (Online)¶

Backup while node is running using RocksDB checkpoint feature.

Note: This requires RocksDB checkpoint API support in Fukuii. Check if available.

#!/bin/bash
# backup-live.sh

DATADIR=~/.fukuii/etc
BACKUP_DIR=/backup/fukuii/live
DATE=$(date +%Y%m%d-%H%M%S)
BACKUP_PATH="$BACKUP_DIR/fukuii-checkpoint-$DATE"

# Create RocksDB checkpoint (if supported)
# This would require exposing checkpoint functionality via CLI or RPC
# Example (hypothetical):
# ./bin/fukuii cli create-checkpoint --output "$BACKUP_PATH"

# Alternative: Use filesystem snapshots (LVM, ZFS, Btrfs)
# LVM example:
# sudo lvcreate -L 10G -s -n fukuii-snap /dev/vg0/fukuii-lv
# sudo mount /dev/vg0/fukuii-snap /mnt/snapshot
# rsync -avh /mnt/snapshot/ "$BACKUP_PATH/"
# sudo umount /mnt/snapshot
# sudo lvremove -f /dev/vg0/fukuii-snap

echo "Live backup requires snapshot support - see disk-management.md"

Method 4: Incremental Backup¶

Backup only changes since last backup.

#!/bin/bash
# backup-incremental.sh

DATADIR=~/.fukuii/etc
BACKUP_DIR=/backup/fukuii/incremental
DATE=$(date +%Y%m%d-%H%M%S)
LINK_DEST="$BACKUP_DIR/latest"

mkdir -p "$BACKUP_DIR"

# Use rsync with hard links to save space
rsync -avh --delete \
  --link-dest="$LINK_DEST" \
  "$DATADIR/" \
  "$BACKUP_DIR/backup-$DATE/"

# Update latest link
rm -f "$LINK_DEST"
ln -s "$BACKUP_DIR/backup-$DATE" "$LINK_DEST"

echo "Incremental backup completed: backup-$DATE"

Method 5: Cloud Backup¶

Upload to cloud storage (S3, Google Cloud Storage, Azure Blob, etc.)

#!/bin/bash
# backup-to-s3.sh

DATADIR=~/.fukuii/etc
S3_BUCKET=s3://my-fukuii-backups
DATE=$(date +%Y%m%d-%H%M%S)

# Backup essentials to S3
aws s3 sync "$DATADIR/keystore/" "$S3_BUCKET/keystore-$DATE/" --exclude "*"
aws s3 cp "$DATADIR/node.key" "$S3_BUCKET/node.key-$DATE"
aws s3 cp "$DATADIR/app-state.json" "$S3_BUCKET/app-state-$DATE.json"

# Optionally backup database (expensive and slow)
# aws s3 sync "$DATADIR/rocksdb/" "$S3_BUCKET/rocksdb-$DATE/"

echo "Cloud backup completed"

Configure AWS CLI first:

aws configure

Encrypting Backups¶

For sensitive data (especially keys):

#!/bin/bash
# backup-encrypted.sh

DATADIR=~/.fukuii/etc
BACKUP_DIR=/backup/fukuii/encrypted
DATE=$(date +%Y%m%d-%H%M%S)

# Create archive
tar -czf - "$DATADIR/keystore" "$DATADIR/node.key" | \
  gpg --symmetric --cipher-algo AES256 \
  -o "$BACKUP_DIR/fukuii-keys-$DATE.tar.gz.gpg"

echo "Encrypted backup created"
echo "Decrypt with: gpg -d fukuii-keys-$DATE.tar.gz.gpg | tar -xzf -"

Restore Procedures¶

Restore Essential Files¶

Scenario: Fresh installation, need to restore node identity and accounts.

#!/bin/bash
# restore-essentials.sh

BACKUP_FILE=/backup/fukuii/essentials/fukuii-essentials-20250102-030000.tar.gz
DATADIR=~/.fukuii/etc

# Stop node if running
pkill -f fukuii

# Extract backup
mkdir -p "$DATADIR"
tar -xzf "$BACKUP_FILE" -C /tmp/

# Restore files
cp /tmp/fukuii-essentials-*/node.key "$DATADIR/"
cp -r /tmp/fukuii-essentials-*/keystore "$DATADIR/"
cp /tmp/fukuii-essentials-*/app-state.json "$DATADIR/" 2>/dev/null
cp /tmp/fukuii-essentials-*/knownNodes.json "$DATADIR/" 2>/dev/null

# Set permissions
chmod 600 "$DATADIR/node.key"
chmod 700 "$DATADIR/keystore"

# Cleanup
rm -rf /tmp/fukuii-essentials-*

echo "Essential files restored"
echo "Database will sync from network on next start"

Restore Full Database¶

Scenario: Hardware failure, need complete restoration.

#!/bin/bash
# restore-full.sh

BACKUP_PATH=/backup/fukuii/full/fukuii-full-20250101-030000
DATADIR=~/.fukuii/etc

# Stop node
pkill -f fukuii

# Remove existing data (be careful!)
read -p "This will delete $DATADIR. Continue? (yes/no) " confirm
if [ "$confirm" != "yes" ]; then
    echo "Aborted"
    exit 1
fi

rm -rf "$DATADIR"

# Restore from backup
mkdir -p "$(dirname $DATADIR)"
rsync -avh --progress "$BACKUP_PATH/" "$DATADIR/"

# Verify critical files
if [ ! -f "$DATADIR/node.key" ]; then
    echo "ERROR: node.key not found in backup!"
    exit 1
fi

echo "Full restoration completed"
echo "Start Fukuii normally: ./bin/fukuii etc"

Restore from Cloud¶

#!/bin/bash
# restore-from-s3.sh

S3_BUCKET=s3://my-fukuii-backups
DATADIR=~/.fukuii/etc
DATE=20250102-030000

mkdir -p "$DATADIR"

# Restore from S3
aws s3 sync "$S3_BUCKET/keystore-$DATE/" "$DATADIR/keystore/"
aws s3 cp "$S3_BUCKET/node.key-$DATE" "$DATADIR/node.key"
aws s3 cp "$S3_BUCKET/app-state-$DATE.json" "$DATADIR/app-state.json"

chmod 600 "$DATADIR/node.key"
chmod 700 "$DATADIR/keystore"

echo "Restored from cloud backup"

Restore from Encrypted Backup¶

#!/bin/bash
# restore-encrypted.sh

BACKUP_FILE=/backup/fukuii/encrypted/fukuii-keys-20250102-030000.tar.gz.gpg
DATADIR=~/.fukuii/etc

# Decrypt and extract
gpg -d "$BACKUP_FILE" | tar -xzf - -C "$DATADIR/"

chmod 600 "$DATADIR/node.key"
chmod 700 "$DATADIR/keystore"

echo "Decrypted and restored"

Selective Restore¶

Scenario: Only restore specific components.

# Restore only node.key
tar -xzf fukuii-essentials-DATE.tar.gz \
  --strip-components=1 \
  -C ~/.fukuii/etc/ \
  fukuii-essentials-DATE/node.key

# Restore only keystore
tar -xzf fukuii-essentials-DATE.tar.gz \
  --strip-components=1 \
  -C ~/.fukuii/etc/ \
  fukuii-essentials-DATE/keystore/

Portable Chain Snapshots (Export / Import)¶

A database backup only restores into the same client. A chain file moves blocks between nodes and between clients: fukuii writes and reads go-ethereum's chain format, so a file from geth export / geth's admin_exportChain imports into fukuii, and a fukuii export imports into geth (geth import).

Format. A chain file is RLP-encoded blocks back to back, with no header and no framing: each block is its consensus encoding (header, transactions, ommers, and withdrawals from Shanghai on), so its hash is unchanged. A file name ending in .gz is gzipped on export and gunzipped on import. A geth export file may carry a fifth item per Amsterdam block (the EIP-7928 access list); fukuii accepts it and recomputes the list from execution anyway. fukuii writes the four-item form, which every geth import path reads.

Files written by fukuii's admin_exportChain before the chain-format fix (CQ-020-01) put a 4-byte length before every block. The importer still reads them (it logs a warning); re-export to get a file other clients can read.

Export¶

Offline, with the node stopped. If a node is still running on the datadir, the command stops at once with Cannot open the database in <datadir>: it is locked. Is a fukuii node running on <datadir>?. The offline commands bind no port and open no connection: no P2P server, discovery, JSON-RPC, Engine API, sync or mining start, only the database and an idle actor system.

sudo systemctl stop fukuii
./bin/fukuii etc export /backups/etc-0-20000000.rlp.gz 0 20000000   # first and last are optional
sudo systemctl start fukuii

Online, from a running node (geth-compatible admin_exportChain(file, first?, last?); the admin API must be enabled):

curl -s -X POST -H 'Content-Type: application/json' localhost:8546 \
  --data '{"jsonrpc":"2.0","id":1,"method":"admin_exportChain","params":["/backups/range.rlp.gz",135090,147609]}'

Both write the canonical blocks first..last (default: genesis to head) and:

fail on the first missing block — export failed on #N: not found (...) — instead of writing a file with a gap. A block past the local head, or below a SNAP-sync pivot where the node holds no bodies, is missing. A SNAP-synced node can only export from the point where it holds full blocks;
fail if a stored body does not match its header (the stored body does not match its header), because no client could import such a block;
fail if the chain reorganises under the export (chain reorg during export);
never overwrite an existing file, and write through a temporary file that is renamed into place only when the whole range is written, so the target either does not exist or is complete.

Block numbers on the command line are decimal or 0x hex. Check the range you ask for — 0x24039 is 147513, not 147609.

Import¶

Offline, with the node stopped, onto the datadir that should receive the blocks:

sudo systemctl stop fukuii
./bin/fukuii etc import /backups/etc-0-20000000.rlp.gz     # several files may follow, imported in order
sudo systemctl start fukuii

Online: admin_importChain(file). Each block is imported under the chain insert lock that every other import path (sync, Engine API newPayload) also takes, as geth's admin_importChain holds its chain lock, so they cannot interleave with it. It is refused while the node is syncing (the node is syncing): the file must continue the local head, and sync is moving it. The lock does not cover head writes outside block execution, such as a consensus client's engine_forkchoiceUpdated: detach or stop the consensus client (and any miner) while running admin_importChain on a live node, or use the offline command. If the head moves anyway, the import stops at that block with "head moved during import" instead of overwriting the head. The call is synchronous, as in geth: a file that takes longer than the HTTP request timeout (pekko.http.server.request-timeout, 20 s by default) times out on the client side while the import carries on; its progress and result are in the node log. For anything large, use the offline command.

Every block goes through the same checks as a synced block — header and body validation, execution, then gas used, receipts root, state root, logs bloom, requests hash and access-list hash. Nothing in the file is trusted. The import:

skips a genesis block in the file after checking it is this network's genesis (a file for another network fails);
skips blocks already on the local canonical chain, so re-running an import, or importing a file that overlaps the local chain, is safe;
needs every other block to continue the local head: the first block of the file must be the child of the head (or already known), and the file must have no gaps. To restore onto a fresh datadir, export from genesis;
stops at the first invalid, undecodable or non-continuing block with an error naming it. Blocks before it stay imported. Each block, with its state, receipts and the head pointer, is written in one atomic commit, so a failed block — or a killed process — leaves nothing half-written, and re-running the same command resumes.

The startup option -Dfukuii.import-chain-file=<file> reads the same format but is the hive / test-fixture loader: it continues past invalid blocks (consensus fixtures deliberately contain them) instead of stopping. Use the import command or admin_importChain for operator restores.

Verify¶

After an import, compare the head with a trusted source:

curl -s -X POST -H 'Content-Type: application/json' localhost:8546 \
  --data '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockByNumber","params":["latest",false]}' | jq '.result | {number, hash, stateRoot}'

Disaster Recovery¶

Scenario 1: Corrupted Database¶

Symptoms: Node won't start, RocksDB errors

Recovery Steps:

Try automatic repair (see disk-management.md)

# Restart - RocksDB may auto-repair
./bin/fukuii etc

If repair fails, restore from backup
```
./restore-full.sh
```

If no backup, resync from genesis

# Backup keys first
cp ~/.fukuii/etc/node.key ~/node.key.backup
cp -r ~/.fukuii/etc/keystore ~/keystore.backup

# Remove database only
rm -rf ~/.fukuii/etc/rocksdb/

# Restore keys
cp ~/node.key.backup ~/.fukuii/etc/node.key
cp -r ~/keystore.backup ~/.fukuii/etc/keystore/

# Resync (will take days)
./bin/fukuii etc

Scenario 2: Lost Node Key¶

Symptoms: node.key file deleted or lost

Recovery:

If you have a backup:

tar -xzf fukuii-essentials-DATE.tar.gz fukuii-essentials-DATE/node.key
cp fukuii-essentials-DATE/node.key ~/.fukuii/etc/
chmod 600 ~/.fukuii/etc/node.key

If NO backup: - Node will generate a new key on next start - You will have a new node identity - Known peers will not recognize your node - Impact: Minimal - node will still work, just with new identity

Scenario 3: Lost Keystore¶

Symptoms: Keystore directory deleted or lost

Recovery:

If you have a backup:

tar -xzf fukuii-essentials-DATE.tar.gz fukuii-essentials-DATE/keystore/
cp -r fukuii-essentials-DATE/keystore ~/.fukuii/etc/
chmod 700 ~/.fukuii/etc/keystore

If NO backup: - CRITICAL: Private keys are permanently lost - Accounts are inaccessible - Funds cannot be recovered - Prevention: ALWAYS backup keystore after creating accounts

Scenario 4: Hardware Failure¶

Complete server/disk failure

Recovery Steps:

Provision new hardware
Install Fukuii (see first-start.md)
Restore from backup
```
./restore-full.sh
```

Verify restoration

./bin/fukuii etc
# Check logs, RPC, peer count

Resume operations

Time estimate: 1-3 hours (if database backup exists), 1-7 days (if resync needed)

Scenario 5: Accidental Data Deletion¶

Recovery:

Stop immediately to prevent more writes

Attempt file recovery (if just deleted)

# Linux - may recover recently deleted files
sudo extundelete /dev/sdX --restore-directory /home/user/.fukuii

Restore from backup

Implement safeguards:

# Make critical files immutable
sudo chattr +i ~/.fukuii/etc/node.key

Testing and Validation¶

Regular Backup Testing¶

Test restores regularly - A backup you can't restore is useless.

#!/bin/bash
# test-restore.sh

BACKUP_FILE=/backup/fukuii/essentials/fukuii-essentials-latest.tar.gz
TEST_DIR=/tmp/fukuii-restore-test

# Extract to test directory
mkdir -p "$TEST_DIR"
tar -xzf "$BACKUP_FILE" -C "$TEST_DIR"

# Verify critical files exist
if [ ! -f "$TEST_DIR"/fukuii-essentials-*/node.key ]; then
    echo "FAIL: node.key missing"
    exit 1
fi

if [ ! -d "$TEST_DIR"/fukuii-essentials-*/keystore ]; then
    echo "FAIL: keystore missing"
    exit 1
fi

echo "PASS: Backup is valid"
rm -rf "$TEST_DIR"

Schedule monthly:

0 4 1 * * /path/to/test-restore.sh && mail -s "Backup Test: PASS" [email protected]

Verification Checklist¶

After any restore:

Best Practices¶

For All Deployments¶

3-2-1 Rule: 3 copies, 2 different media, 1 offsite
Backup keys immediately after generation
Test restores regularly (monthly)
Automate backups (cron jobs)
Monitor backup success (alerting)
Document procedures (this runbook)
Encrypt sensitive backups (keys, keystore)

For Production Nodes¶

Multiple backup locations (local + cloud)
Frequent essentials backups (hourly)
Weekly database backups
Versioned backups (keep multiple generations)
Offsite replication (different datacenter)
Automated testing (restore to test environment)
Disaster recovery plan (documented, tested)
RTO/RPO targets (defined and measured)

For Personal Nodes¶

Daily essentials backup (minimum)
Manual database backup before upgrades
Cloud backup for keys (encrypted)
Document restore procedure

Security Considerations¶

Encrypt backups containing private keys
Restrict backup access (file permissions)
Secure backup storage (encrypted at rest)
Secure transfer (SSH, TLS)
Key management (store encryption keys separately)
Audit backup access (log who accessed backups)

Backup Automation Example¶

Complete automated backup solution:

#!/bin/bash
# /usr/local/bin/fukuii-backup-automation.sh

DATADIR=~/.fukuii/etc
BACKUP_BASE=/backup/fukuii
LOG_FILE=/var/log/fukuii-backup.log

log() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a "$LOG_FILE"
}

# Daily essentials backup
daily_essentials() {
    log "Starting daily essentials backup"
    /usr/local/bin/backup-essentials.sh >> "$LOG_FILE" 2>&1

    # Upload to cloud
    aws s3 sync "$BACKUP_BASE/essentials/" s3://my-backups/fukuii/essentials/

    log "Daily backup completed"
}

# Weekly full backup (Sunday)
weekly_full() {
    log "Starting weekly full backup"
    /usr/local/bin/backup-full-offline.sh >> "$LOG_FILE" 2>&1
    log "Weekly backup completed"
}

# Monthly test restore
monthly_test() {
    log "Starting monthly restore test"
    /usr/local/bin/test-restore.sh >> "$LOG_FILE" 2>&1

    if [ $? -eq 0 ]; then
        log "Restore test: PASSED"
    else
        log "Restore test: FAILED - ALERT"
        mail -s "ALERT: Fukuii Backup Test Failed" [email protected] < "$LOG_FILE"
    fi
}

# Run appropriate backup based on day
DAY=$(date +%u)  # 1-7 (Monday-Sunday)
if [ "$DAY" -eq 7 ]; then
    weekly_full
fi

daily_essentials

# First day of month
if [ "$(date +%d)" -eq "01" ]; then
    monthly_test
fi

Cron schedule:

# Daily at 3 AM
0 3 * * * /usr/local/bin/fukuii-backup-automation.sh

First Start - Initial setup and configuration
Disk Management - Storage and database management
Known Issues - Database corruption and recovery

Document Version: 1.0
Last Updated: 2025-11-02
Maintainer: Chippr Robotics LLC

Backup & Restore Runbook¶

Overview¶

Table of Contents¶

Backup Strategies¶

Strategy Comparison¶

Recommended Strategy¶

What to Backup¶

Essential Files (MUST backup)¶

Database (Optional but recommended)¶

Configuration Files¶

Logs (Usually not needed)¶

Backup Size Estimates¶

Backup Procedures¶

Method 1: Essential Files Only (Recommended for All)¶

Method 2: Full Database Backup (Offline)¶

Method 3: Live Database Backup (Online)¶

Method 4: Incremental Backup¶

Method 5: Cloud Backup¶

Encrypting Backups¶

Restore Procedures¶

Restore Essential Files¶

Restore Full Database¶

Restore from Cloud¶

Restore from Encrypted Backup¶

Selective Restore¶

Portable Chain Snapshots (Export / Import)¶

Export¶

Import¶

Verify¶

Disaster Recovery¶

Scenario 1: Corrupted Database¶

Scenario 2: Lost Node Key¶

Scenario 3: Lost Keystore¶

Scenario 4: Hardware Failure¶

Scenario 5: Accidental Data Deletion¶

Testing and Validation¶

Regular Backup Testing¶

Verification Checklist¶

Best Practices¶

For All Deployments¶

For Production Nodes¶

For Personal Nodes¶

Security Considerations¶

Backup Automation Example¶

Related Runbooks¶