Search and scoring

Request failures are an eligibility gate controlled by benchmark.max_failure_percentage. Among eligible trials, the configured optimization.maximize metric is the primary objective in search, persisted rankings, baseline comparison, terminal output, CSV, and HTML. Ties use failure percentage, failure count, and trial identifier in that order.

Choose grid for exhaustive small spaces, random for a bounded sample, or tpe for guided exploration. Random and TPE searches do not execute the same resolved configuration twice.

optimization:
  maximize: output_tokens_per_second
  sampler: tpe
  trials: 20
  seed: 42

maximize names the metric used to rank trials. Direction is inferred: the declared metric is always maximized. For each named benchmark, the vllm-opt CLI averages the eligible workload metric values, takes the median when it was repeated, then averages named benchmark scores into the trial score. A workload with an errored or incomplete percentage above benchmark.max_failure_percentage is excluded; the setting defaults to 0. A trial without an eligible workload is not ranked.

Grid evaluates every unique configuration and is best for small spaces. TPE uses completed trial results to choose promising configurations, so it is the better default for larger spaces where exhaustive Grid search is impractical.

The fixed server configuration runs first as the baseline. The report shows the best observed configuration and its difference from that baseline.

TPE state is persisted in the run's SQLite study. Runs themselves remain immutable: a retry creates a new linked run instead of rewriting history.