Skip to content

v0.1.0a5 — Native benchmarks and explicit parallel trials

August 31, 2026

This alpha adds vLLM's native serving benchmark and opt-in concurrent trials without changing the default GuideLLM or sequential workflows.

Highlights

  • Run either GuideLLM or vllm bench serve from the same experiment engine.
  • Execute independent trials concurrently on explicit, non-overlapping GPUs.
  • Keep scoring, error accounting, reports, reproduction, and lifecycle behavior consistent across benchmark engines.

vLLM Bench Serve

  • Added benchmark.engine: vllm while keeping GuideLLM as the default.
  • Made --backend vllm, model, server address, and result paths automatic.
  • Added forward-compatible runs[].args rendering for native benchmark flags.
  • Added canonical throughput aliases for backend-independent optimization.
  • Counted completed, failed, and incomplete requests through the existing error-aware ranking policy.
  • Preserved raw JSON, benchmark.log, exact commands, versions, repeats, retries, and timeout diagnostics.
  • Documented random, ShareGPT, Hugging Face, custom, and prefix-repetition datasets with links to current upstream options.
benchmark:
  engine: vllm
  runs:
    - name: throughput
      args:
        dataset-name: random
        random-input-len: 512
        random-output-len: 128
        num-prompts: 1000
        request-rate: inf
        max-concurrency: 32

Parallel trials

  • Added execution.mode: local_parallel with explicit worker names, GPU sets, and deterministic ports.
  • Rejected GPU sharing, overlapping device sets, manually assigned ports, and incompatible tensor-parallel configurations before launching processes.
  • Kept search suggestions, Optuna updates, ranking, and persistence under one coordinator while trial managers execute concurrently.
  • Isolated trial failures and cleaned up every active owned process on cancellation.
  • Stored execution mode, worker, devices, and port in trial artifacts.
  • Labeled reports with a shared-resource warning so parallel measurements are not mistaken for contention-free sequential results.

Validation

  • All 147 private tests and 23 subtests passed.
  • A real OPT-125M baseline and tuned trial completed with vLLM 0.28.0 and native Bench Serve on WSL2 with an RTX 3080; all requests succeeded.
  • GuideLLM regression tests, offline report regeneration, reproduction, strict documentation, Python compilation, packaging, installation, and metadata validation passed.
  • Python 3.11 and 3.12 GitHub package checks passed before release preparation.
  • Controlled two-process parallel integration passed. A real multi-GPU parallel vLLM run was not possible on the available single-GPU host.

Install

pip install vtune==0.1.0a5

Use pip install "vtune[runtime]==0.1.0a5" to install vLLM and GuideLLM with vTune. The universal pure-Python wheel works for inspection on Linux, Windows, and macOS; running vLLM experiments remains Linux-only.