Cross-cutting future requirements

Schema evolution

Version every YAML and persisted result schema.
Provide forward migrations for supported old versions.
Never silently reinterpret an old score or metric definition.
Preserve source and resolved configurations.

Compatibility matrix

Track tested combinations of:

vLLM versions.
Benchmark backend versions.
CUDA and driver versions.
Python versions.
GPU architectures.

Passthrough flags should remain allowed even when the combination is untested; the matrix informs warnings rather than becoming an argument allowlist.

Observability

Structured internal logs with run and trial identifiers.
Optional tracing of lifecycle phases.
Storage-size and artifact-retention reporting.
Diagnostic bundles that redact secrets.

Testing strategy

Unit tests for schemas, rendering, scoring, and fingerprinting.
Fake vLLM and benchmark processes for deterministic lifecycle tests.
Fault injection for hangs, partial output, OOM text, signals, and corrupt artifacts.
Golden tests for reports and exports.
Hardware integration tests on supported GPU configurations.
Long-running soak tests.
Schema-upgrade and interrupted-run recovery tests.

Documentation

Quick start with a small model.
Complete configuration reference.
Metric definitions and units.
Benchmark comparability guidance.
Troubleshooting by structured failure category.
Reproducibility and security model.
Adapter-development guide.