Statistical Performance Regression Harness

The native console harness measures HotPDF on both Win32 and Win64 and returns a failing process status when a workload exceeds its versioned threshold

Run the gate

Tests\Performance\Run-HotPDFPerformance.bat

python tools\validate_performance_report.py ^
  Tests\Performance\Reports\performance-Win32.json ^
  Tests\Performance\Reports\performance-Win64.json

The batch file builds release runners, performs three warmups followed by 30 measured samples per workload, applies Tests\Performance\thresholds.json, and writes one report per platform under Tests\Performance\Reports

Run the gate in isolation on an otherwise idle machine because scheduler contention and unrelated process I/O are deliberately visible to the measurements

Workloads

load opens the deterministic four-page fixture and materializes page geometry
save serializes the shared loaded document to a fresh file and flushes it to storage
render rasterizes every fixture page at 72 DPI
extract extracts and verifies text across every fixture page

The fixture has fixed PDF bytes and a pinned SHA-256 identity so an input change cannot silently reuse old thresholds

Report schema

hotpdf.performance.v1 records the product and harness versions, UTC timestamp, warm-cache state, environment, fixture identity, raw samples, summaries, applied thresholds, failures, and overall pass state

Each summary contains minimum, nearest-rank p50 and p95, maximum, arithmetic mean, N-1 sample variance, standard deviation, observed and process-peak RSS, maximum positive working-set growth, and total read, write, and other process I/O

The validator independently recomputes timing statistics, RSS, I/O totals, threshold decisions, and aggregate pass state from the raw samples

Runner options and exit codes

HotPDFPerformanceRunner.exe \
  --output=report.json \
  --thresholds=thresholds.json \
  --warmup=3 \
  --samples=30

--samples cannot be lower than 30 and --warmup cannot be lower than one

Exit code 0 means every workload passed
Exit code 1 means configuration, fixture, parsing, or execution failed
Exit code 2 means at least one performance threshold failed

Threshold maintenance

Keep separate Win32 and Win64 ceilings for p50, p95, observed RSS, working-set growth, read bytes, and write bytes

Only revise a threshold after reviewing raw samples, machine conditions, fixture identity, implementation intent, and repeated isolated runs on both platforms