The right vLLM settings for your GPU.

Tell it what you want to run and what you are running it on. You get a vllm serve command, a Docker setup, and an honest answer about whether it will fit in your VRAM.

Runs entirely in your browser - nothing is uploaded, nothing is installed. Tuned for vLLM v0.28.

1 What are you running?

From Hugging Face A local folder

Model id

Paste any id from huggingface.co and we read its config.json for you. Try , , or .

Model not on Hugging Face, or gated?

Drop config.json here or click to pick a file

Or just tell us the size in billions of parameters

Overrides anything read from a config file. Useful for vision models.

It is private or gated Adds a Hugging Face token slot to the generated .env.

2 What are you running it on?

Your GPU

VRAM per GPU GiB

How many

Longest prompt tokens

Longest reply tokens

These add up to --max-model-len: 5,120 tokens.

Requests at the same time

How many people can be mid-request at once. Every one of them needs its own KV cache.

3 Fine-tuning everything below is optional

Precision

Quantization

Number format --dtype

KV cache format

Compilation -O

Cache shared prefixes Big win for chat and RAG, where every request repeats a long system prompt. Skip CUDA graphs Saves about half a gigabyte and starts faster, at the cost of throughput.

Memory

How much VRAM vLLM may take 0.90

vLLM's own default is 0.92. Go lower if the same GPU is also driving your monitor.

Offload to system RAM GiB

Last resort. Weights held in RAM cross the PCIe bus on every token.

Fix KV cache size GiB

Setting this lets vLLM skip memory profiling, so restarts are quicker.

Random seed optional

Force full attention Ignores the model's sliding window.

Throughput

Tokens per batch optional

Above 8192 favours throughput; around 2048 favours a steady token stream.

API server processes optional

Raise this if the GPU is idle but a CPU core is pinned at 100%.

Queue order

Long-prefill threshold optional

Chunked prefill

Breaks long prompts into pieces so replies in flight keep streaming. Rarely worth changing.

Multiple GPUs and machines

Pipeline parallel

Splits layers across machines. Leave at 1 on a single host.

Data parallel

Whole extra copies of the model, for more throughput.

Expert parallel For Mixture-of-Experts models: spreads experts over GPUs instead of slicing each one.

Worker backend

Naming and loading

Name clients use optional

Tokenizer override optional

Revision optional

Weight format

Download folder optional

Trust remote code Runs Python from the model repo at load time. Some models need it.

LoRA adapters Serve LoRA adapters One base model, many fine-tunes, swapped per request.

Adapters in a batch

Largest rank

Adapters one per line, name=path

Speculative decoding

A small model guesses several tokens ahead and the big one checks them in a single pass. Often 1.5x to 2x faster with identical output.

Method

Tokens to guess

Tools, reasoning and chat Enable function calling Lets clients send OpenAI-style tools and get tool calls back.

Tool-call parser

Reasoning parser

Splits a model's thinking from its answer in the API response.

Chat template path optional

The HTTP server Require an API key Generates one into .env. Do this before exposing the port.

Allowed browser origins optional

Only needed when a web page calls the server directly.

Log every request

Truncate logs at chars

Images, video and anything else

Media allowed per request optional

Docker image

Left empty, we pick the image that matches your GPU.

Ignore the model's sampling defaults Recommended. Otherwise a repo's generation_config.json can silently override temperature.

Anything else added verbatim

-- of VRAM

Tell us your model to begin

Weights: 0.0 GiB
KV cache: 0.0 GiB
Runtime: 0.0 GiB
Total: 0.0 GiB
Available: 0.0 GiB

The command

vllm serve ...

Back

The right vLLM settings for your GPU.

1 What are you running?

2 What are you running it on?

3 Fine-tuning everything below is optional

Saved on this browser

deployment