Serve an Open Model at Speed with vLLM: Setup, Configuration and Benchmarking

Ollama is wonderful for personal use, but when you need to serve a model to an application — handling concurrent requests with low latency — you want vLLM. It’s the open-source inference server behind a large share of production LLM deployments, and its tricks (paged attention for the KV cache, continuous batching of requests) translate directly into more tokens per second per GPU. In this guide you’ll install vLLM, serve an open model behind an OpenAI-compatible API, tune the key configuration flags, and benchmark it properly.

This is for developers with access to an NVIDIA GPU who want to self-host model inference — for an internal tool, a product feature, or just to understand what “serving” actually entails.

Prerequisites

• OS: Linux. vLLM is CUDA-first; Windows users should use WSL2, and macOS isn’t supported for GPU serving.
• GPU: NVIDIA with 16 GB+ VRAM for 7–8B models (compute capability 7.0+). Check with nvidia-smi.
• Python: 3.10–3.12, and a CUDA 12.x toolkit matching your PyTorch.
• Disk: ~20 GB for the model weights (downloaded from Hugging Face on first run).

Step 1: Install vLLM

pip install vllm

That’s the whole install — wheels ship with prebuilt CUDA kernels. (Building from source is only needed for exotic setups.) Confirm the CLI:

vllm --help | head -20

The two commands you’ll use are vllm serve (start the server) and vllm bench serve (measure it).

Step 2: Serve your first model

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

The model downloads from Hugging Face on first launch — grab a coffee. Once you see Uvicorn running, the server speaks the OpenAI API dialect:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Explain vLLM in one sentence."}],
    "max_tokens": 100,
    "temperature": 0.7
  }'

Any OpenAI client library works against this endpoint — just change the base URL. Health checks live at /health, and /metrics exposes Prometheus stats.

Step 3: Understand the flags that matter

vLLM has dozens of options; five do most of the work. But first, the two ideas that make vLLM fast, so the flags make sense. PagedAttention manages the KV cache (the model’s short-term memory of the conversation) in fixed-size blocks instead of one contiguous chunk per request, which nearly eliminates wasted memory. Continuous batching means incoming requests join the GPU’s work queue immediately instead of waiting for the current batch to finish — so ten simultaneous users each get steady progress rather than taking turns. Every flag below is just tuning how aggressively vLLM exploits those two mechanisms:

• --tensor-parallel-size 2 — split the model across 2 GPUs. Required for models that don’t fit on one card; also the simplest way to raise throughput on multi-GPU machines.
• --gpu-memory-utilization 0.90 — fraction of VRAM vLLM may use for weights plus KV cache. The default 0.90 is sane; lower it if you share the GPU.
• --max-model-len 8192 — caps context length. The KV cache is allocated per this value, so cutting it from 32k to 8k frees enormous memory for more concurrent requests. Set the smallest value your use case tolerates.
• --enable-prefix-caching — reuses KV cache for repeated prompt prefixes (system prompts, few-shot examples). Nearly free throughput for chat applications.
• --quantization awq — serve a 4-bit AWQ-quantised model (e.g. a -AWQ repo from Hugging Face) at roughly half the memory. See our quantisation guide for how these are made.

A tuned single-GPU launch for a chat workload might look like:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --port 8000 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-seqs 128

Step 4: Benchmark it honestly

Don’t trust vibes — measure. vLLM ships a serving benchmark that fires synthetic traffic at your server and reports throughput and latency percentiles:

vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --model Qwen/Qwen2.5-7B-Instruct \
  --endpoint /v1/chat/completions \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 128 \
  --num-prompts 50 \
  --max-concurrency 8

You’ll get output like this:

============ Serving Benchmark Result ============
Successful requests:            50
Request throughput (req/s):     3.41
Output token throughput (tok/s): 412.55
---------------Time to First Token----------------
Mean TTFT (ms):                 812.34
P99 TTFT (ms):                  1450.21
---------------Time per Output Token--------------
Mean TPOT (ms):                 19.44

The numbers that matter: output token throughput (capacity), TTFT (responsiveness — dominated by prefill), and TPOT (streaming smoothness). Compare configurations by re-running the same benchmark — change one flag at a time. For realistic traffic shapes, swap --dataset-name random for a ShareGPT-format JSON file.

Step 5: Run it in Docker (optional but recommended)

For anything beyond experiments, pin the environment:

docker run -d --gpus all --name vllm \
  -p 8000:8000 --shm-size 8g \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.90 --max-model-len 8192

The --shm-size 8g matters — vLLM uses shared memory heavily, and Docker’s tiny default causes cryptic hangs at startup.

Calling it from Python

Curl is fine for smoke tests; applications use a client library. The openai package works unchanged — only the base URL differs:

pip install openai

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Summarize this in 20 words: ..."}],
    max_tokens=60,
    temperature=0.3,
)
print(resp.choices[0].message.content)

For streaming responses — the token-by-token effect users expect in chat UIs — pass stream=True and iterate the chunks. One operational note: vLLM logs every request, so put it behind a reverse proxy with authentication before exposing it beyond localhost. The server itself has no auth; that’s your infrastructure’s job.

Where things go wrong

CUDA out of memory at startup. Lower --max-model-len first (it’s usually the culprit, not the model weights), then --gpu-memory-utilization. A 7B model at 32k context needs far more KV cache than the same model at 8k.

Slow first request. vLLM compiles CUDA graphs and warms up on the first tokens. Benchmark after warmup, not before.

Multi-GPU NCCL errors. Usually a driver or networking issue between GPUs. Verify nvidia-smi sees all cards and that no other process holds them.

Garbled output with quantised models. The quantisation format must match the flag — AWQ weights need --quantization awq, GPTQ needs --quantization gptq. Mismatches produce confident nonsense.

Throughput plateaus no matter the flags. You’re likely memory-bandwidth bound, which is normal for decode-heavy workloads on a single GPU — the fix is a smaller or quantised model, not more tuning. And if latency spikes only under concurrent load, check --max-num-seqs: set too high, long prefills starve everyone else’s decodes.

Wrap-up

You now have a production-grade inference server: OpenAI-compatible, tunable, and measurable. The discipline to take away is benchmarking before and after every config change — vLLM’s defaults are good, but the right --max-model-len for your workload is worth more than any exotic flag. When you’re ready to compare models rather than servers, our eval harness guide shows how to score them on standard benchmarks.

Featured image: Wil Weterings via Wikimedia Commons (Public domain)