Skip to content

v0.1.0a16

Highlights

  • Added adaptive benchmark repeat policies for evidence-driven measurement budgets.
  • Improved repeat aggregation and ranking while retaining repeat-level provenance.
  • Hardened reports and comparisons against incomplete evidence.

Release artifacts

Platform Install
PyPI pip install "vllm-optimizer==0.1.0a16"
Linux/WSL runtime pip install "vllm-optimizer[runtime]==0.1.0a16"

The universal wheel, source distribution, and SBOM are attached to the release.

Adaptive benchmark repeats

The new adaptive repeat policy can request more measurements when the current evidence is too noisy or incomplete for a reliable decision. Configuration is validated before execution, and repeat budgets are represented explicitly in the run state and reports. Fixed repeat behavior remains available for users who need a deterministic measurement budget.

benchmark:
  repeats: 4
  min_repeats: 2
  adaptive_repeats:
    minimum_relative_score: 0.7

This runs every trial at least twice, then skips its remaining repeats when its initial positive score is below 70% of the best eligible trial. See the benchmark repeat guide for validation rules and fixed-repeat behavior.

Repeated experiments are ranked by arithmetic mean. Individual repeat results, coverage, and eligibility remain available so the aggregate does not hide failed or incomplete measurements.

Reporting and evidence

  • Comparisons now distinguish complete, workload-matched evidence from partial coverage and mark unavailable aggregates explicitly.
  • Added evidence-aware comparison and leaderboard support.
  • Refreshed the published RTX 3080 report examples and provenance to match the current results JSON and HTML report format.
  • Added readable workload summaries with full redacted configuration JSON in expandable details.
  • Made large TPE preflight validation lazy and snapshotted external vLLM YAML settings for stable reproduction exports.

Packaging and documentation

  • Made PyPI publishing safe to retry after a partial upload with skip-existing enabled.
  • Expanded benchmark-repeat, configuration, reporting, and results guidance.

Validation

The public and available private suites, package checks, lint, types, coverage, strict documentation build, audit, and release workflow are run for this release. Hardware smoke and real-backend validation remain dependent on the manual GPU workflow.

Full changelog: v0.1.0a15...v0.1.0a16.