KPI Baselines - Fukuii Test Suite¶

Status: ✅ Established
Date: November 16, 2025
Related ADRs: TEST-001, TEST-002

Overview¶

This document establishes baseline Key Performance Indicators (KPIs) for the Fukuii test suite and performance benchmarks. These baselines provide objective criteria for test suite health, execution efficiency, and system performance.

Test Execution Time Baselines¶

Tier 1: Essential Tests¶

Target: < 5 minutes
Warning Threshold: > 7 minutes
Failure Threshold: > 10 minutes

Components: - Core unit tests (bytes, crypto, rlp modules) - Fast unit tests (excluding slow and integration tests) - Critical consensus logic tests

Baseline Measurement (as of Nov 16, 2025):

Estimated execution time: 3-5 minutes
- bytes / test:        ~30 seconds
- crypto / test:       ~45 seconds  
- rlp / test:          ~30 seconds
- testOnly (filtered): ~2-3 minutes

SBT Command:

sbt testEssential

Tier 2: Standard Tests¶

Target: < 30 minutes
Warning Threshold: > 40 minutes
Failure Threshold: > 60 minutes

Components: - All unit tests (including slower tests) - Selected integration tests - RPC API validation tests - Basic ethereum/tests validation

Baseline Measurement (as of Nov 16, 2025):

Estimated execution time: 15-30 minutes
- All unit tests:           ~10-15 minutes
- Integration tests:        ~5-10 minutes
- RPC tests:                ~2-5 minutes
- Coverage report generation: ~2-3 minutes

SBT Command:

sbt testCoverage

Tier 3: Comprehensive Tests¶

Target: < 3 hours
Warning Threshold: > 4 hours
Failure Threshold: > 5 hours

Components: - Complete ethereum/tests BlockchainTests suite - Complete ethereum/tests StateTests suite - Performance benchmarks - Long-running stress tests

Baseline Measurement (as of Nov 16, 2025):

Estimated execution time: 45 minutes - 3 hours
- All standard tests:       ~30 minutes
- Ethereum/tests suite:     ~30-60 minutes
- Benchmark tests:          ~15-30 minutes
- Stress tests:             ~30-60 minutes

SBT Command:

sbt testComprehensive

Test Health KPI Baselines¶

Test Success Rate¶

Target: > 99%
Measurement: (Passing tests / Total tests) × 100

Current Baseline: - Essential tests: 100% (all tests passing) - Standard tests: ~98-99% (with known excluded tests) - Comprehensive tests: ~95-98% (ethereum/tests in Phase 3)

Test Flakiness Rate¶

Target: < 1%
Measurement: (Tests with inconsistent results / Total tests) × 100

Current Baseline: - Actor-based tests: < 2% (improved with cleanup fixes) - Database tests: < 1% - Network tests: < 3% (inherently variable) - Pure unit tests: 0%

Test Coverage¶

Target: > 80% line coverage, > 70% branch coverage
Measurement: scoverage reports

Current Baseline (Phase 2 Complete):

Line Coverage:   70-80% (target: > 80%)
Branch Coverage: 60-70% (target: > 70%)

Excluded from Coverage: - Protobuf generated code - BuildInfo generated code - Managed sources

Actor Cleanup Success Rate¶

Target: 100%
Measurement: (Actor systems shut down / Actor systems created) × 100

Current Baseline: - Post-TEST-002 Phase 1: 100% (cleanup fixes implemented) - Pre-Phase 1: ~80-90% (hanging tests issue)

Ethereum/Tests Compliance KPI Baselines¶

GeneralStateTests (Berlin Fork)¶

Target Pass Rate: > 95%
Current Status: ✅ Phase 2 Complete

Baseline Measurement: - SimpleTx tests: 100% passing (4/4 tests) - Extended StateTests: Pending Phase 3 rollout

Test Categories: - Value transfers - Contract creation - Contract calls - Storage operations - Gas calculations

BlockchainTests (Berlin Fork)¶

Target Pass Rate: > 90%
Current Status: ✅ Phase 2 Complete

Baseline Measurement: - SimpleTx_Berlin: 100% passing - SimpleTx_Istanbul: 100% passing - Extended BlockchainTests: Pending Phase 3 rollout

State Root Validation:

Initial state root: cafd881ab193703b83816c49ff6c2bf6ba6f464a1be560c42106128c8dbc35e7
Final state root:   cc353bc3876f143b9dd89c5191e475d3a6caba66834f16d8b287040daea9752c

TransactionTests¶

Target Pass Rate: > 95%
Current Status: ⏳ Pending Phase 3

Planned Coverage: - Transaction parsing - Signature validation - Gas limit validation - Value transfer validation

VMTests¶

Target Pass Rate: > 95%
Current Status: ⏳ Pending Phase 3

Planned Coverage: - All 140+ EVM opcodes - Gas cost validation - Stack operations - Memory operations - Storage operations