Baseline wins — keep the baseline
Repeat ranges favor the baseline in every matched workload; independent validation is still limited.
Baseline retained; no observed improvement is claimed.
| Measurement | Baseline | Best observed |
|---|---|---|
| Mean output throughput | 925.5 tok/s | 925.5 tok/s |
| Average benchmark duration | 6.618 s | 6.618 s |
Arithmetic means of recorded measurements; benchmark duration excludes server startup. The configured optimization score may use a different aggregation.
Best observed settings
baseline · Best observed eligible configuration.
No settings changed.
Full configuration and launch command
Full resolved vLLM YAML
Experiment settings, snapshotted external settings, and runtime host/port; unspecified vLLM internal defaults are not captured. Redacted values require replacement.
dtype: float16 enable-chunked-prefill: true enforce-eager: true gpu-memory-utilization: 0.4 host: 127.0.0.1 max-model-len: 512 max-num-batched-tokens: 512 max-num-seqs: 16 model: /var/tmp/vllm-optimizer-demo/models/opt-125m port: 8766 seed: 42 tensor-parallel-size: 1
Selected environment variables
CUDA_VISIBLE_DEVICES: '0' VLLM_LOGGING_LEVEL: INFO VLLM_USE_FLASHINFER_SAMPLER: '0' VLLM_USE_V2_MODEL_RUNNER: '0'
Launch command (POSIX shell)
env CUDA_VISIBLE_DEVICES=0 VLLM_LOGGING_LEVEL=INFO VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_USE_V2_MODEL_RUNNER=0 vllm serve /var/tmp/vllm-optimizer-demo/models/opt-125m --dtype float16 --enable-chunked-prefill --enforce-eager --gpu-memory-utilization 0.4 --host 127.0.0.1 --max-model-len 512 --max-num-batched-tokens 512 --max-num-seqs 16 --port 8766 --seed 42 --tensor-parallel-size 1
Performance breakdown
Baseline vs best observed
This comparison exposes latency trade-offs, but tuning a different objective is not the best way to search for the lowest latency. Use a latency objective for that purpose.
Each value is the arithmetic mean of available repeat measurements for this exact workload. P50/P95/P99 are means of backend-supplied percentiles, not pooled request percentiles. Positive throughput changes improve performance; negative latency/failure changes improve performance. Percentage changes are unavailable for mismatched workloads, unequal repeat counts, or missing measurements. Overall means retain the same workload weighting only when coverage matches.
Overall means
| Metric | Baseline | Recommended | Difference |
|---|---|---|---|
| Mean E2E latency | 275.9 ms | 275.9 ms | +0.00% (same) |
| Mean output throughput | 925.5 tok/s | 925.5 tok/s | +0.00% (same) |
| Mean benchmark duration | 6.618 s | 6.618 s | +0.00% (same) |
Mean duration by benchmark
| Benchmark | Baseline | Recommended | Difference |
|---|---|---|---|
| short-chat | 6.618 s | 6.618 s | +0.00% (same) |
short-chat / workload 0 — requests: 16 · concurrency: 8 · request rate: unlimited · burstiness: 1.0
Full workload configuration JSON
{
"backend": "vllm",
"burstiness": 1.0,
"max_concurrency": 8,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf"
}| Metric | Baseline | Recommended | Change | Repeats (base / rec) |
|---|---|---|---|---|
| Output throughput · Mean | 925.5 tok/s | 925.5 tok/s | +0.00% | 3 / 3 |
| Request throughput · Mean | 28.92 req/s | 28.92 req/s | +0.00% | 3 / 3 |
| TTFT · Mean | 35.81 ms | 35.81 ms | +0.00% | 3 / 3 |
| TTFT · P50 | 25.94 ms | 25.94 ms | +0.00% | 3 / 3 |
| TTFT · P95 | 52.3 ms | 52.3 ms | +0.00% | 3 / 3 |
| TTFT · P99 | 52.47 ms | 52.47 ms | +0.00% | 3 / 3 |
| Time per output token · Mean | 7.745 ms | 7.745 ms | +0.00% | 3 / 3 |
| Time per output token · P50 | 7.773 ms | 7.773 ms | +0.00% | 3 / 3 |
| Time per output token · P95 | 10.8 ms | 10.8 ms | +0.00% | 3 / 3 |
| Time per output token · P99 | 11.28 ms | 11.28 ms | +0.00% | 3 / 3 |
| Inter-token latency · Mean | 7.97 ms | 7.97 ms | +0.00% | 3 / 3 |
| Inter-token latency · P50 | 5.149 ms | 5.149 ms | +0.00% | 3 / 3 |
| Inter-token latency · P99 | 22.79 ms | 22.79 ms | +0.00% | 3 / 3 |
| End-to-end latency · Mean | 275.9 ms | 275.9 ms | +0.00% | 3 / 3 |
| End-to-end latency · P50 | 275.9 ms | 275.9 ms | +0.00% | 3 / 3 |
| End-to-end latency · P95 | 381.1 ms | 381.1 ms | +0.00% | 3 / 3 |
| End-to-end latency · P99 | 381.3 ms | 381.3 ms | +0.00% | 3 / 3 |
| Failed + incomplete requests | 0 requests | 0 requests | Unavailable | 3 / 3 |
Confidence and validation
Confidence in the result
Repeat ranges favor the baseline in every matched workload; independent validation is still limited.
Descriptive repeat evidence for the optimization metric, per workload. Range overlap is a conservative inconclusive flag, not a statistical significance test. These measurements are not independent proof of production performance. Drift compares the first and second halves when at least four repeats exist.
At least 3 measured repeats are required by the configured confidence policy. Sequential drift of more than 5% triggers finalist validation.
| Trial / phase | Workload | Individual repeats | n | Mean | Sample SD | Range | Drift |
|---|---|---|---|---|---|---|---|
| baseline / Initial search (accepted) | short-chat / workload 0 — requests: 16 · concurrency: 8 · request rate: unlimited · burstiness: 1.0 | 1: 919.4; 2: 931.7; 3: 925.5 | 3 | 925.5 | 6.127 | 919.4 – 931.7 | Unavailable (<4 repeats) |
| trial-0001 / Initial search (accepted) | short-chat / workload 0 — requests: 16 · concurrency: 8 · request rate: unlimited · burstiness: 1.0 | 1: 205.9; 2: 187.6; 3: 207.7 | 3 | 200.4 | 11.1 | 187.6 – 207.7 | Unavailable (<4 repeats) |
All trials
Every trial, including failures. Mean benchmark duration excludes server startup. Expand Full details for repeat timings, settings, errors, and all recorded metrics. Click a heading to sort.
| Details | ||||||
|---|---|---|---|---|---|---|
| 1 | baseline (baseline) | completed | 925.5 | +0.00% | 6.618 s | Full detailsFailed requests (all repeats): 0; Score SD (largest workload): 6.127 Duration vs baseline: +0.00% (same) Changed settings: {} short-chat / workload 0 — requests: 16 · concurrency: 8 · request rate: unlimited · burstiness: 1.0: score SD 6.127; n=3 short-chat, repeat 1: duration 6.621 s short-chat, repeat 2: duration 6.635 s short-chat, repeat 3: duration 6.599 s {
"host": "127.0.0.1",
"port": 8766,
"dtype": "float16",
"seed": 42,
"tensor-parallel-size": 1,
"gpu-memory-utilization": 0.4,
"max-model-len": 512,
"enforce-eager": true,
"enable-chunked-prefill": true,
"max-num-seqs": 16,
"max-num-batched-tokens": 512,
"env.CUDA_VISIBLE_DEVICES": "0",
"env.VLLM_USE_V2_MODEL_RUNNER": "0",
"env.VLLM_USE_FLASHINFER_SAMPLER": "0"
}All recorded trial data (JSON){
"schema_version": 1,
"trial_id": "baseline",
"status": "completed",
"benchmarks": [
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/results.json",
"repeat": 1,
"elapsed_seconds": 6.6211034309963,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001015",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 0.5568704510005773,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 28.731996770975037,
"request_goodput": null,
"output_throughput": 919.4238966712012,
"total_token_throughput": "<redacted>",
"ttfts": [
0.027390028000809252,
0.052197485994838644,
0.051786725998681504,
0.05180259900225792,
0.05179955500352662,
0.05132502799824579,
0.05124900900409557,
0.05118700999446446,
0.023685018997639418,
0.017565855996508617,
0.023381894003250636,
0.023378498997772112,
0.02286308899783762,
0.022546439002326224,
0.022504008004034404,
0.022462788998382166
],
"max_output_tokens_per_s": 498.0,
"max_concurrent_requests": 16,
"rtfx": 0.0,
"mean_ttft_ms": 35.445314624666935,
"median_ttft_ms": 25.537523499224335,
"std_ttft_ms": 14.376934877886665,
"p50_ttft_ms": 25.537523499224335,
"p95_ttft_ms": 51.9013207504031,
"p99_ttft_ms": 52.138252945951535,
"mean_tpot_ms": 7.816520802422136,
"median_tpot_ms": 7.843694241948411,
"std_tpot_ms": 3.0229491101809094,
"p50_tpot_ms": 7.843694241948411,
"p95_tpot_ms": 10.935117975790373,
"p99_tpot_ms": 11.418450975777946,
"mean_itl_ms": 8.043560060169847,
"median_itl_ms": 5.212225001741899,
"std_itl_ms": 5.741247223539477,
"p50_itl_ms": 5.212225001741899,
"p95_itl_ms": 20.642403695092042,
"p99_itl_ms": 21.952506191737484,
"mean_e2el_ms": 277.75745949975317,
"median_e2el_ms": 277.83887300029164,
"std_e2el_ms": 106.66682276035688,
"p50_e2el_ms": 277.83887300029164,
"p95_e2el_ms": 384.9628047464648,
"p99_e2el_ms": 385.07883214915637,
"requests_per_second": {
"average": 28.731996770975037
},
"output_tokens_per_second": {
"average": 919.4238966712012
},
"total_tokens_per_second": {
"average": 2758.271690013604
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 277.75745949975317,
"median": 277.83887300029164,
"p99": 385.07883214915637
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
},
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/results.json",
"repeat": 2,
"elapsed_seconds": 6.634794607001822,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001021",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 0.5495458990044426,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 29.11494750299402,
"request_goodput": null,
"output_throughput": 931.6783200958087,
"total_token_throughput": "<redacted>",
"ttfts": [
0.028065067999705207,
0.05394198099384084,
0.05361263800296001,
0.05350894799630623,
0.053490347003389616,
0.052931407997675706,
0.05289208600152051,
0.05281375200138427,
0.02378148699790472,
0.017953021000721492,
0.02343496599496575,
0.02285615599976154,
0.02283722500578733,
0.02279422900028294,
0.022567385996808298,
0.022093879997555632
],
"max_output_tokens_per_s": 498.0,
"max_concurrent_requests": 16,
"rtfx": 0.0,
"mean_ttft_ms": 36.22341112441063,
"median_ttft_ms": 25.923277498804964,
"std_ttft_ms": 15.183399028614506,
"p50_ttft_ms": 25.923277498804964,
"p95_ttft_ms": 53.69497375068022,
"p99_ttft_ms": 53.89257954520872,
"mean_tpot_ms": 7.6712394758003715,
"median_tpot_ms": 7.694385887119889,
"std_tpot_ms": 3.0112599328328096,
"p50_tpot_ms": 7.694385887119889,
"p95_tpot_ms": 10.779751007982169,
"p99_tpot_ms": 11.274841298337934,
"mean_itl_ms": 7.894058360980728,
"median_itl_ms": 4.912910000712145,
"std_itl_ms": 5.861273226742792,
"p50_itl_ms": 4.912910000712145,
"p95_itl_ms": 20.29182394981033,
"p99_itl_ms": 23.307157570307027,
"mean_e2el_ms": 274.0318348742221,
"median_e2el_ms": 274.1764454985969,
"std_e2el_ms": 107.08313900691736,
"p50_e2el_ms": 274.1764454985969,
"p95_e2el_ms": 381.49258849807666,
"p99_e2el_ms": 381.66176569720847,
"requests_per_second": {
"average": 29.11494750299402
},
"output_tokens_per_second": {
"average": 931.6783200958087
},
"total_tokens_per_second": {
"average": 2795.0349602874257
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 274.0318348742221,
"median": 274.1764454985969,
"p99": 381.66176569720847
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
},
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/results.json",
"repeat": 3,
"elapsed_seconds": 6.5985015600017505,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001028",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 0.5531883360017673,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 28.92324179436221,
"request_goodput": null,
"output_throughput": 925.5437374195907,
"total_token_throughput": "<redacted>",
"ttfts": [
0.028088799001125153,
0.05140045999723952,
0.05126534100418212,
0.051071800000499934,
0.05112475399801042,
0.05063521500414936,
0.050631443002203014,
0.05025333400408272,
0.02461104199755937,
0.018343661999097094,
0.024472422002872918,
0.02450507799949264,
0.02393332299834583,
0.023918619997857604,
0.023928807000629604,
0.023920738000015263
],
"max_output_tokens_per_s": 498.0,
"max_concurrent_requests": 16,
"rtfx": 0.0,
"mean_ttft_ms": 35.75655237546016,
"median_ttft_ms": 26.34992049934226,
"std_ttft_ms": 13.48343145010384,
"p50_ttft_ms": 26.34992049934226,
"p95_ttft_ms": 51.29912075244647,
"p99_ttft_ms": 51.38019214828091,
"mean_tpot_ms": 7.746803197615889,
"median_tpot_ms": 7.782221758163595,
"std_tpot_ms": 2.8557270133423724,
"p50_tpot_ms": 7.782221758163595,
"p95_tpot_ms": 10.693517096789888,
"p99_tpot_ms": 11.151018412983243,
"mean_itl_ms": 7.971818367236306,
"median_itl_ms": 5.32274699799018,
"std_itl_ms": 5.596056037418811,
"p50_itl_ms": 5.32274699799018,
"p95_itl_ms": 20.30544309855029,
"p99_itl_ms": 23.110008718213066,
"mean_e2el_ms": 275.9074515015527,
"median_e2el_ms": 275.80378499988,
"std_e2el_ms": 100.64567682264229,
"p50_e2el_ms": 275.80378499988,
"p95_e2el_ms": 376.9401125045988,
"p99_e2el_ms": 377.24082650420314,
"requests_per_second": {
"average": 28.92324179436221
},
"output_tokens_per_second": {
"average": 925.5437374195907
},
"total_tokens_per_second": {
"average": 2776.631212258772
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 275.9074515015527,
"median": 275.80378499988,
"p99": 377.24082650420314
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
}
],
"artifacts": {
"vllm_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/vllm.log",
"benchmark_short-chat_warmup_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/benchmark.log",
"benchmark_short-chat_warmup_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/results.json",
"benchmark_short-chat_warmup_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/drain.json",
"benchmark_short-chat_repeat_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/benchmark.log",
"benchmark_short-chat_repeat_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/results.json",
"benchmark_short-chat_repeat_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/drain.json",
"benchmark_short-chat_repeat_2_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/benchmark.log",
"benchmark_short-chat_repeat_2_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/results.json",
"benchmark_short-chat_repeat_2_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/drain.json",
"benchmark_short-chat_repeat_3_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/benchmark.log",
"benchmark_short-chat_repeat_3_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/results.json",
"benchmark_short-chat_repeat_3_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/drain.json",
"manifest": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/manifest.json"
},
"execution": {
"mode": "sequential"
},
"attempts": [
{
"index": 1,
"status": "completed",
"artifacts": {
"vllm_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/vllm.log",
"benchmark_short-chat_warmup_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/benchmark.log",
"benchmark_short-chat_warmup_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/results.json",
"benchmark_short-chat_warmup_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/warmups/001/short-chat/drain.json",
"benchmark_short-chat_repeat_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/benchmark.log",
"benchmark_short-chat_repeat_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/results.json",
"benchmark_short-chat_repeat_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/001/short-chat/drain.json",
"benchmark_short-chat_repeat_2_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/benchmark.log",
"benchmark_short-chat_repeat_2_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/results.json",
"benchmark_short-chat_repeat_2_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/002/short-chat/drain.json",
"benchmark_short-chat_repeat_3_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/benchmark.log",
"benchmark_short-chat_repeat_3_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/results.json",
"benchmark_short-chat_repeat_3_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/baseline/attempts/001/repeats/003/short-chat/drain.json"
},
"failure": null
}
]
} |
| 2 | trial-0001 | completed | 205.9 | -77.76% | 8.488 s | Full detailsFailed requests (all repeats): 0; Score SD (largest workload): 11.1 Duration vs baseline: +28.26% (worse) Changed settings: {"max-num-seqs": [16, 1]} short-chat / workload 0 — requests: 16 · concurrency: 8 · request rate: unlimited · burstiness: 1.0: score SD 11.1; n=3 short-chat, repeat 1: duration 8.373 s short-chat, repeat 2: duration 8.684 s short-chat, repeat 3: duration 8.409 s {
"host": "127.0.0.1",
"port": 8766,
"dtype": "float16",
"seed": 42,
"tensor-parallel-size": 1,
"gpu-memory-utilization": 0.4,
"max-model-len": 512,
"enforce-eager": true,
"enable-chunked-prefill": true,
"max-num-seqs": 1,
"max-num-batched-tokens": 512,
"env.CUDA_VISIBLE_DEVICES": "0",
"env.VLLM_USE_V2_MODEL_RUNNER": "0",
"env.VLLM_USE_FLASHINFER_SAMPLER": "0"
}All recorded trial data (JSON){
"schema_version": 1,
"trial_id": "trial-0001",
"status": "completed",
"benchmarks": [
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/results.json",
"repeat": 1,
"elapsed_seconds": 8.37268743399909,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001101",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 2.4869544280008995,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 6.433571849911764,
"request_goodput": null,
"output_throughput": 205.87429919717644,
"total_token_throughput": "<redacted>",
"ttfts": [
0.025231797000742517,
0.2852520300002652,
0.43206471799931023,
0.5842921779985772,
0.7439827830021386,
0.8861063029980869,
1.0314784820002387,
1.1794528050013469,
1.054604721000942,
1.0508805669960566,
1.0448919800037402,
1.0369755889987573,
1.0383470980013954,
1.041974866995588,
1.0434260529946187,
1.0364189360043383
],
"max_output_tokens_per_s": 211.0,
"max_concurrent_requests": 13,
"rtfx": 0.0,
"mean_ttft_ms": 844.7113066872589,
"median_ttft_ms": 1036.6972625015478,
"std_ttft_ms": 325.7372667708726,
"p50_ttft_ms": 1036.6972625015478,
"p95_ttft_ms": 1085.8167420010432,
"p99_ttft_ms": 1160.7255924012861,
"mean_tpot_ms": 4.64734799395721,
"median_tpot_ms": 4.4128143549202035,
"std_tpot_ms": 0.8932748656185375,
"p50_tpot_ms": 4.4128143549202035,
"p95_tpot_ms": 5.615733750023685,
"p99_tpot_ms": 7.575070233912312,
"mean_itl_ms": 4.792281725544825,
"median_itl_ms": 4.464744997676462,
"std_itl_ms": 1.791302287169032,
"p50_itl_ms": 4.464744997676462,
"p95_itl_ms": 5.91104799968889,
"p99_itl_ms": 15.239301000838166,
"mean_e2el_ms": 988.7790944999324,
"median_e2el_ms": 1170.17078400022,
"std_e2el_ms": 307.3709691693757,
"p50_e2el_ms": 1170.17078400022,
"p95_e2el_ms": 1220.3382165007497,
"p99_e2el_ms": 1299.0667041009144,
"requests_per_second": {
"average": 6.433571849911764
},
"output_tokens_per_second": {
"average": 205.87429919717644
},
"total_tokens_per_second": {
"average": 617.6228975915293
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 988.7790944999324,
"median": 1170.17078400022,
"p99": 1299.0667041009144
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
},
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/results.json",
"repeat": 2,
"elapsed_seconds": 8.683645363002142,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001109",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 2.7290979890021845,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 5.862742951875442,
"request_goodput": null,
"output_throughput": 187.60777446001416,
"total_token_throughput": "<redacted>",
"ttfts": [
0.029022063994489145,
0.368944341003953,
0.5336852059990633,
0.6980273760054843,
0.8487287270036177,
0.9976122059961199,
1.1448938110042945,
1.2886425090036937,
1.0797745699965162,
1.0644910979972337,
1.055728701998305,
1.0519022369990125,
1.1245925199982594,
1.167566356001771,
1.1626358499997878,
1.1670223960027215
],
"max_output_tokens_per_s": 201.0,
"max_concurrent_requests": 13,
"rtfx": 0.0,
"mean_ttft_ms": 923.9543730627702,
"median_ttft_ms": 1060.1098999977694,
"std_ttft_ms": 335.2804852002237,
"p50_ttft_ms": 1060.1098999977694,
"p95_ttft_ms": 1197.8353942522517,
"p99_ttft_ms": 1270.4810860534053,
"mean_tpot_ms": 5.1072168528281185,
"median_tpot_ms": 4.47684332257561,
"std_tpot_ms": 1.5371002345284,
"p50_tpot_ms": 4.47684332257561,
"p95_tpot_ms": 7.653173596824467,
"p99_tpot_ms": 9.989521145231969,
"mean_itl_ms": 5.2774638187808405,
"median_itl_ms": 4.631096999219153,
"std_itl_ms": 4.900567988974282,
"p50_itl_ms": 4.631096999219153,
"p95_itl_ms": 7.124159847808186,
"p99_itl_ms": 15.161198868227075,
"mean_e2el_ms": 1082.2780955004419,
"median_e2el_ms": 1215.7386344988481,
"std_e2el_ms": 305.81978883766527,
"p50_e2el_ms": 1215.7386344988481,
"p95_e2el_ms": 1333.8289702496695,
"p99_e2el_ms": 1405.9156804545637,
"requests_per_second": {
"average": 5.862742951875442
},
"output_tokens_per_second": {
"average": 187.60777446001416
},
"total_tokens_per_second": {
"average": 562.8233233800424
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 1082.2780955004419,
"median": 1215.7386344988481,
"p99": 1405.9156804545637
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
},
{
"name": "short-chat",
"backend": "vllm",
"backend_version": "0.28.0",
"raw_artifact": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/results.json",
"repeat": 3,
"elapsed_seconds": 8.408828316001745,
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0,
"workloads": [
{
"index": 0,
"configuration": {
"backend": "vllm",
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8
},
"metrics": {
"date": "20260914-001118",
"endpoint_type": "vllm",
"backend": "vllm",
"label": null,
"model_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"tokenizer_id": "/var/tmp/vllm-optimizer-demo/models/opt-125m",
"num_prompts": 16,
"request_rate": "inf",
"burstiness": 1.0,
"max_concurrency": 8,
"duration": 2.4654356940009166,
"completed": 16,
"failed": 0,
"total_input_tokens": 1024,
"total_output_tokens": 512,
"request_throughput": 6.489725138210825,
"request_goodput": null,
"output_throughput": 207.6712044227464,
"total_token_throughput": "<redacted>",
"ttfts": [
0.030523202003678307,
0.2954079150003963,
0.4415855539991753,
0.588207856999361,
0.7314118879949092,
0.873938390002877,
1.0248617409961298,
1.1753475999939837,
1.038549677003175,
1.036741397001606,
1.0322472370025935,
1.0308321389966295,
1.0319554329980747,
1.0245435350007028,
1.0203452959976858,
1.0163023899949621
],
"max_output_tokens_per_s": 214.0,
"max_concurrent_requests": 13,
"rtfx": 0.0,
"mean_ttft_ms": 837.0500781866212,
"median_ttft_ms": 1022.4444154991943,
"std_ttft_ms": 318.1408527960537,
"p50_ttft_ms": 1022.4444154991943,
"p95_ttft_ms": 1072.7491577508772,
"p99_ttft_ms": 1154.8279115453624,
"mean_tpot_ms": 4.590657264147481,
"median_tpot_ms": 4.335310887155731,
"std_tpot_ms": 0.9334812561006373,
"p50_tpot_ms": 4.335310887155731,
"p95_tpot_ms": 5.456045596738771,
"p99_tpot_ms": 7.646020087087243,
"mean_itl_ms": 4.743685393774891,
"median_itl_ms": 4.442456003744155,
"std_itl_ms": 1.7213657403808786,
"p50_itl_ms": 4.442456003744155,
"p95_itl_ms": 5.589907550165662,
"p99_itl_ms": 15.319291791602154,
"mean_e2el_ms": 979.3604533751932,
"median_e2el_ms": 1156.675633501436,
"std_e2el_ms": 299.95692985065455,
"p50_e2el_ms": 1156.675633501436,
"p95_e2el_ms": 1208.2840897528513,
"p99_e2el_ms": 1290.5742139504582,
"requests_per_second": {
"average": 6.489725138210825
},
"output_tokens_per_second": {
"average": 207.6712044227464
},
"total_tokens_per_second": {
"average": 623.0136132682392
},
"time_to_first_token_ms": "<redacted>",
"time_per_output_token_ms": "<redacted>",
"inter_token_latency_ms": "<redacted>",
"end_to_end_latency_ms": {
"average": 979.3604533751932,
"median": 1156.675633501436,
"p99": 1290.5742139504582
},
"request_totals": {
"successful": 16,
"errored": 0,
"incomplete": 0
},
"request_total": 16,
"requests": {
"input": [
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64,
64
],
"output": [
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32,
32
],
"error_msg": {}
}
},
"successful_requests": 16,
"failed_requests": 0,
"errored_requests": 0,
"incomplete_requests": 0,
"total_requests": 16,
"failure_percentage": 0.0
}
]
}
],
"artifacts": {
"vllm_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/vllm.log",
"benchmark_short-chat_warmup_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/benchmark.log",
"benchmark_short-chat_warmup_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/results.json",
"benchmark_short-chat_warmup_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/drain.json",
"benchmark_short-chat_repeat_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/benchmark.log",
"benchmark_short-chat_repeat_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/results.json",
"benchmark_short-chat_repeat_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/drain.json",
"benchmark_short-chat_repeat_2_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/benchmark.log",
"benchmark_short-chat_repeat_2_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/results.json",
"benchmark_short-chat_repeat_2_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/drain.json",
"benchmark_short-chat_repeat_3_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/benchmark.log",
"benchmark_short-chat_repeat_3_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/results.json",
"benchmark_short-chat_repeat_3_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/drain.json",
"manifest": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/manifest.json"
},
"execution": {
"mode": "sequential"
},
"attempts": [
{
"index": 1,
"status": "completed",
"artifacts": {
"vllm_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/vllm.log",
"benchmark_short-chat_warmup_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/benchmark.log",
"benchmark_short-chat_warmup_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/results.json",
"benchmark_short-chat_warmup_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/warmups/001/short-chat/drain.json",
"benchmark_short-chat_repeat_1_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/benchmark.log",
"benchmark_short-chat_repeat_1_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/results.json",
"benchmark_short-chat_repeat_1_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/001/short-chat/drain.json",
"benchmark_short-chat_repeat_2_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/benchmark.log",
"benchmark_short-chat_repeat_2_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/results.json",
"benchmark_short-chat_repeat_2_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/002/short-chat/drain.json",
"benchmark_short-chat_repeat_3_log": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/benchmark.log",
"benchmark_short-chat_repeat_3_json": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/results.json",
"benchmark_short-chat_repeat_3_drain": "/var/tmp/vllm-optimizer-demo/public/baseline-control/20260913-210946-683068/trials/trial-0001/attempts/001/repeats/003/short-chat/drain.json"
},
"failure": null
}
]
} |
Diagnostics and methodology
Accepted score comparison
Accepted scores in recorded trial order
This is trial order, not elapsed time. Superseded search executions are excluded.
Parameter associations
Not shown: fewer than 5 eligible tuned trials. A percentage here would look precise without enough evidence.
Observed score means by tested value, with sample counts; associations, not controlled ablations.
No parameter had multiple evaluated values.
Failures and interruptions
No failed or interrupted trials.
How metrics are calculated
- Requests/s
- Completed requests divided by the benchmark measurement time.
- Output tokens/s
- Generated output tokens divided by measurement time. Total tokens/s includes prompt and output tokens.
- TTFT
- Time from sending a request until its first generated token.
- TPOT
- Average time per generated output token after the first token.
- ITL
- Delay between consecutive streamed output tokens.
- E2E latency
- Time from request start until the complete response is received.
- Average / median / P99
- Arithmetic mean, 50th percentile, and 99th percentile. P99 means 99% of observations were at or below that value.
- Elapsed
- vLLM Optimizer wall-clock time from benchmark subprocess launch through JSON result parsing; vLLM server startup is measured separately.
The benchmark backend calculates request distributions and percentiles. vLLM Optimizer maps backend names and units into one schema without deriving missing percentiles. Report comparisons use repeat means. For scoring, eligible workloads are averaged within a benchmark execution, repeated executions use their median score, and named benchmark scores are averaged into the trial score.
Scoring evidence
| Trial | Metric | Mean benchmark duration | Duration vs baseline | Successful | Errored | Incomplete | Error rate | Excluded workloads |
|---|---|---|---|---|---|---|---|---|
| baseline (baseline) | 925.5437 | 6.618 s | +0.00% (same) | 48 | 0 | 0 | 0.00% | 0 |
| trial-0001 | 205.8743 | 8.488 s | +28.26% (worse) | 48 | 0 | 0 | 0.00% | 0 |
A workload is excluded when failed/incomplete requests exceed 0% of all requests. Eligible workload metrics use their mean per named run, the median across repeats, then the mean of named runs. This configured objective is separate from the workload comparisons above.