Most open-source inference stacks assume the internet is always there. vLLM resolves model names against Hugging Face, Python packages expect an index, and some models fetch extra files the first time they load. On a site with no link, each of those assumptions turns into a crash or a long hang.
The fix is not one flag. It is a habit: anything the server touches at startup has to be on disk before the network goes away. Here is how to stage vLLM and llama.cpp so they start cold with no connection.
What does “fully offline” actually mean?
Fully offline inference means the server installs, starts, loads the model and answers requests with every network interface down. Not “works on a slow link”, and not “works once it has cached things”. If a first start needs the internet, the system is not offline yet.
That gives you a simple test, which we come back to at the end: start it with networking removed. If it works there, it works anywhere.
What has to be staged before you lose the network?
Four things, and teams usually remember only the first.
| What | Why it breaks offline | How to stage it |
|---|---|---|
| Model weights | vLLM resolves a repo name like org/model against the Hub | hf download to a plain folder, pinned to a commit |
| Python packages | pip install vllm needs an index | A wheelhouse, or a pinned container image |
| Runtime downloads | Some models fetch tokenizer or vocab files on first load | Fetch them ahead and set the variable that points to them |
| Telemetry | Usage reporting tries to phone home | Turn it off with environment variables |
How do you download weights for an air-gapped machine?
Use the hf CLI on a connected machine and download into a normal folder rather than the hidden cache:
hf download Qwen/Qwen3-8B --revision <commit-sha> --local-dir ./models/qwen3-8b
Two habits matter here. Pin --revision to a commit hash, so the model you tested is byte for byte the model you ship. And use --local-dir, so the folder is self-contained and easy to copy, rather than a tree of symlinks inside ~/.cache/huggingface.
For llama.cpp, download only the GGUF file you plan to run. One quantisation is usually several gigabytes, so pick it before you copy, not after.
How do you install vLLM without internet access?
You have two routes. Pick one and pin every version.
Route 1: a wheelhouse. On a connected machine with the same OS, Python version and CPU architecture as the target, download every package vLLM needs, then install from that folder only. pip documents exactly this pattern:
# connected side
python -m pip download --destination-directory ./wheels vllm==<version>
# offline side
python -m pip install --no-index --find-links=./wheels vllm==<version>
--no-index is the important part. It stops pip from even trying to reach PyPI, so a missing package fails straight away with a clear message.
Route 2: a container image. Pull a pinned tag, never latest, and export it:
docker pull vllm/vllm-openai:<version>
docker save vllm/vllm-openai:<version> -o vllm-openai.tar
# offline side
docker load -i vllm-openai.tar
The image carries the CUDA libraries and Python together, which removes a whole class of version mismatch. The wheelhouse is lighter when the host already has a known good CUDA stack.
Which environment variables make vLLM work offline?
Set these before the server starts:
export HF_HUB_OFFLINE=1 # no HTTP calls to the Hub, cached files only
export VLLM_NO_USAGE_STATS=1 # turn off vLLM usage reporting
export DO_NOT_TRACK=1 # also disables Hugging Face telemetry
vllm serve /models/qwen3-8b --served-model-name qwen3-8b
HF_HUB_OFFLINE=1 does two jobs. It stops lookups to the Hub, and it turns a silent timeout into a fast, explicit error when a file is missing. Passing a local path to vllm serve, instead of a repo name, means there is nothing to resolve in the first place. --served-model-name keeps the model name your clients see clean.
vLLM collects anonymous usage data by default, according to its own docs. On a cut-off site that traffic goes nowhere, but it is still an outbound attempt your security team will ask about, so switch it off.
llama.cpp is simpler. llama-server -m /models/model-Q4_K_M.gguf --host 0.0.0.0 --port 8080 reads a local file and makes no network calls for inference. Only the -hf download option needs a connection, so use it on the connected side.
Which errors break offline installs, and how do you fix them?
“We couldn’t connect to ‘https://huggingface.co’”. The server was given a repo name and tried to resolve it. Pass the local folder path instead, and set HF_HUB_OFFLINE=1.
A missing file error saying outgoing traffic has been disabled. Offline mode is on and working, but a file never made it into the folder or cache. Download again with the same revision and compare against your checksum manifest. Partial copies are the usual cause.
gpt-oss: “error downloading or loading vocab file”. The openai_harmony library downloads tiktoken encodings at startup. The vLLM recipe for gpt-oss gives the fix: fetch o200k_base.tiktoken and cl100k_base.tiktoken ahead of time, put them in a folder, and set TIKTOKEN_ENCODINGS_BASE to it.
pip: “No matching distribution found” with --no-index. The wheelhouse was built on a machine that does not match the target, often a different Python minor version or CPU architecture. Rebuild it on a matching machine.
Out of memory on first start. Not an offline problem, but it shows up at the same moment. Our guide to fixing vLLM out of memory errors and the GPU memory planning guide cover the budget.
How do you prove an inference server is really offline?
Take the network away and start it cold:
docker run --rm --gpus all --network none \
-e HF_HUB_OFFLINE=1 -e VLLM_NO_USAGE_STATS=1 \
-v /models:/models vllm/vllm-openai:<version> \
--model /models/qwen3-8b
--network none gives the container no network at all apart from loopback. Send it a test prompt from inside the container. If it starts and answers, the bundle is complete. If it hangs or errors, you found the missing piece in testing rather than at a remote site at 2am.
Before you ship, write a manifest with sha256sum over every file in the bundle and check it on the far side with sha256sum -c. That catches a truncated copy before it becomes a mystery crash.
What does this not solve?
Staging gets one machine running once. It does not answer what happens next: how you update the model at fifty sites with no link, how you roll back a bad version, or how you show afterwards which model produced which answer. Those are fleet problems, and our design guide for AI on intermittent links is a good place to start on them.
Running offline AI at more than one site
Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. It keeps models serving when the network is gone or untrusted, and keeps a durable record of what ran. See how our technology works, or talk to us about the sites you need to run.