Get vLLM running on your GPU.

Tell us which model you want and what you'll run it on. We'll check that it fits, then give you the command, the Docker files and a short guide to start it.

It all runs in your browser, so nothing you type is uploaded. Made for vLLM v0.31.0.

Simple Full control

A few plain questions. Switch to full control to see every vLLM setting.

1 Which model?

From Hugging Face From a folder

Model

Paste its name from huggingface.co, like owner/model, and we'll look up its size. Or try , or .

Can't find it, or it's private?

Drop its config.json here, or click to choose it

Or type its size billions of parameters

This wins over anything read from a file. It's handy for vision models, where we only count the language part.

It's gated or private We'll add a spot for your Hugging Face token in .env.

2 What will it run on?

Your GPU

How many GPUs? in this machine

NVIDIA driver

Not sure? Run nvidia-smi and look at CUDA Version in the top right.

Memory per GPU GB

Machines usually 1

Use more than one when the model is too big for a single machine.

How long are your conversations?

Up to 5,120 tokens per conversation, prompt and reply together.

Longest prompt tokens

Longest reply tokens

How many people will use it at the same time?

Each person waiting on a reply needs their own share of GPU memory.

Requests at the same time exact number

What matters most?

Tunes vLLM for speed or for volume (--performance-mode). Your numbers above stay as they are.

3 Advanced settings optional, the defaults are fine

Precision

Quantization

Number format --dtype

KV cache format --kv-cache-dtype

Compilation -O

Reuse shared prompt starts On by default. A long system prompt or the same documents are only processed once. Untick to turn it off. Skip CUDA graphs Saves about half a gigabyte and starts faster, but serves fewer people (--enforce-eager).

Memory

How much of the GPU vLLM may use 0.90

vLLM's own default is 0.92. Go lower if the same GPU also drives your screen.

Spill weights to system RAM GB

A last resort. Every token has to fetch those weights over PCIe, so it's slow.

Fixed KV cache size GB

With this set, vLLM skips measuring memory at startup, so restarts are quicker.

Random seed optional

Force full attention Ignores the model's sliding window.

Speed and batching

Tokens per batch optional

Above 8192 serves more people. Around 2048 keeps each reply streaming smoothly.

API server processes optional

Raise this if the GPU sits idle while one CPU core is stuck at 100%.

Queue order

Long-prompt threshold optional

Chunked prefill

Splits long prompts into pieces so other replies keep streaming. You'll rarely need to change it.

Several GPUs or machines

Pipeline parallel

Splits the layers between groups of GPUs. It's set to your number of machines for you. On one machine, use it only for uneven splits or GPUs without NVLink.

Data parallel

Runs whole extra copies of the model, so more people can be served.

Expert parallel For Mixture-of-Experts models. Each GPU holds whole experts instead of a slice of every one.

Worker backend

Naming and loading

Name your apps will use optional

Tokenizer optional

Revision optional

Weight format

Download folder optional

Trust remote code Runs Python code from the model's repo when it loads. Some models need this. Only tick it for models you trust.

LoRA adapters Serve LoRA adapters One base model with many fine-tunes on top, picked per request.

Adapters in one batch

Largest rank

Adapters one per line, name=path

Speculative decoding

A quick guesser writes a few tokens ahead and the model checks them all at once. The output is the same, and it often comes 1.5 to 2 times faster.

Method

Tokens to guess

Tools, reasoning and chat Turn on function calling Your apps can send OpenAI-style tools and get tool calls back.

Tool-call parser

Reasoning parser

Keeps the model's thinking separate from its answer in the API response.

Chat template file optional

The web server Require an API key We'll put a key in .env. Turn this on before anyone else can reach the port.

Websites allowed to call it optional

Only needed when a web page talks to the server straight from the browser.

Log every request

Cut logged prompts at characters

Images, video and anything else

Images or videos per request optional

Written as JSON. If you type the older image=4,video=1 form, we'll convert it.

Docker image

Leave it empty and we'll pick the image that matches your GPU.

Ignore the model's sampling defaults Leave this off to use the temperature and top-p the model's makers recommend. Tick it to use vLLM's plain defaults instead.

Anything else added as typed

-- of memory

Pick a model to start

Model weights: 0.0 GiB
KV cache: 0.0 GiB
Working memory: 0.0 GiB
Total: 0.0 GiB
vLLM may use: 0.0 GiB

The command

vllm serve ...

Back

Get vLLM running on your GPU.

Your setups

1 Which model?

2 What will it run on?

3 Advanced settings optional, the defaults are fine

setup