How to run LLM inference in an air-gapped environment

Short answer: you need four things. Model weights present locally before inference starts, the inference process blocked from outbound network access, a self-hosted model catalog, and no telemetry in the platform itself. Miss any one of them and you have a system that mostly stays inside your perimeter, which is not the same thing.

This guide covers each, and then the part that is usually skipped: how to verify that nothing leaves.

What "air-gapped" actually requires

Most teams handle the obvious part. Inference runs on your hardware, prompts go to a local endpoint, responses come back locally. Done.

Then something reaches out anyway. The usual culprits, in rough order of how often they surprise people:

Model weight acquisition. Weights have to get onto the machine somehow. The question is which component fetches them and when. If the inference engine pulls a tokenizer or a shard at load time, you have an outbound connection inside your serving path, triggered by a cache miss nobody planned for. If a provisioning step fetches them once, before serving, you have a scoped operation you can audit and then switch off. Same download, different architecture, very different risk.

Model catalog lookups. The management layer asks a remote service what models exist, what their specifications are, and which driver strings to use. This is metadata rather than your data, which makes it easy to overlook and still a network call out of a network that is not supposed to make them.

Telemetry and analytics. Usage pings, crash reporting, license checks, update checks. Vendor-default behaviour, frequently undocumented, occasionally on by default.

Tool and retrieval integrations. Web search, page fetching, external API calls inside an agent loop. These need internet access by definition. The question is whether they are off by default or discovered later.

An air-gapped deployment is one where you have addressed all four deliberately, not one where inference happens to run locally.

Step 1: Decide where weight acquisition happens

Two patterns work. Pick one before you build.

The strict pattern. Download weights on a connected machine, verify checksums, transfer them into the environment, and place them where the engine expects to find them. Nothing inside the environment ever contacts a model hub. This is what a hard air gap means, and it is the right answer for classified, defence and some heavily regulated deployments.

The practical pattern. Allow one component, on a deliberate provisioning path, to fetch weights from Hugging Face. Everything else stays blocked. Most teams who say "air-gapped" want this. Staging every model and every quantisation by hand is a real operational cost, and pretending otherwise just means people never get started.

Either way, the serving engine must never fetch. For vLLM and anything else built on the Hugging Face libraries, that is HF_HUB_OFFLINE=1. Without it, a missing tokenizer file turns into an outbound request instead of a clean error. You want the clean error. An error tells you your staging was incomplete. A silent successful download tells you nothing and defeats the point.

In Xinity, vLLM runs with offline mode set by default. Weight acquisition sits outside the engine entirely: the management layer checks whether the model is already present locally, and queues a download only if it is not. The serving process has no fetch path at all, and the single component that can reach Hugging Face is a provisioning step you can inspect, schedule, restrict to a maintenance window, or block outright once your models are in place.

That is the distinction worth drawing, and it is not pull versus pre-staged. It is whether the fetch happens inside the serving process or outside it. A tool that resolves a model from a registry at run time puts an outbound dependency in your inference path, and you cannot remove it without removing the convenience. A separate provisioning step can be cut off without touching the engine.

Step 2: Block egress at the inference process, not just at the firewall

Perimeter firewall rules are necessary and not sufficient, because they are managed by a different team on a different change cycle than the inference platform, and a rule that gets relaxed for an unrelated reason silently re-opens the path.

Run the inference process on a container network with no outbound route. The process then cannot reach the internet regardless of what the perimeter allows, and the guarantee lives next to the workload rather than three teams away.

This is defence in depth in the ordinary sense: perimeter rules are the outer layer, the container network is the inner one, and you want the inner one because it fails closed.

Step 3: Self-host the model catalog

The management layer needs to know what models are available, their memory requirements, quantisation options and driver configuration. That catalog is often fetched from a vendor-hosted endpoint.

Host it yourself. In Xinity's case this is the Info Server, a stateless service reading a local YAML catalog. Point the platform at your own instance and catalog lookups stay internal. If no self-hosted Info Server is configured, the dashboard falls back to a Xinity-hosted metadata endpoint.

That fallback behaviour is the thing to check with any platform you evaluate, and the question to ask is specific: what does this system do when a component is not configured? Silent fallback to a vendor endpoint is a common and reasonable default for convenience, and in an air-gapped deployment it is the exact behaviour you need to switch off. Ask for the list of endpoints the platform can contact and under what conditions. A vendor who cannot produce that list has not thought about your deployment.

Step 4: Confirm there is no mandatory telemetry

There is a real difference between a platform with telemetry you can disable and a platform that has none in the core path.

The first requires you to trust a configuration flag, and to re-verify it after every upgrade. The second requires no trust because there is nothing to switch off.

Ask whether inference requests, responses, model weights and GPU telemetry stay local by default, and whether any external call is required for core operation. Then ask which outbound calls exist as opt-in features, because a system with zero optional integrations is a system without web search or retrieval tooling, and you may want those in a different deployment even if not this one. Open source helps here: you can read the code rather than the datasheet.

Step 5: Keep authentication internal

Air-gapping the inference path and then authenticating against a cloud identity provider is a common inconsistency.

Workable options inside the perimeter: OIDC against your internal identity provider, passkeys using WebAuthn or FIDO2 with hardware keys or platform authenticators, and TOTP two-factor with backup codes. All function without external calls. Verify that domain verification, if the platform uses it, is not doing a DNS lookup against a public resolver as part of login.

Step 6: Verify, then verify again after upgrades

This is the step that separates a claim from a control, and almost nobody documents it.

Capture traffic at the boundary. Run tcpdump or your equivalent on the inference host and the control-plane host during a full exercise: cold start, model load, inference, model swap, dashboard use, a deliberate failure. Look at every outbound connection attempt. Attempts that are blocked still tell you what the software wanted to do, and that is the information you want.

Test both layers separately. Delete a tokenizer file and confirm the engine errors rather than fetches. Then check what the layer above the engine does about it, because a management plane that repairs missing weights by downloading them is doing its job, and you need to know that is what happened rather than assuming the engine went quiet. Stop the local catalog service and confirm the dashboard errors rather than falling back. Request a model that is not staged. The happy path never reveals a fallback. Only the broken path does.

Check DNS. Query logs on your internal resolver will show lookups for external hostnames even when the connections themselves fail. This is often the fastest way to find a component nobody told you about.

Re-run after every upgrade. New versions add features, and features add endpoints. Whatever you verified against version 1.2 is not verified against 1.4. Make this part of the upgrade checklist rather than a one-time acceptance test.

Checklist

• Weight acquisition pattern chosen deliberately: manual staging, or a scoped provisioning fetch

• Checksums verified on every weight set entering the environment

• Offline mode enforced at the serving engine, and re-confirmed after upgrades

• No fetch path inside the inference process itself

• Inference process on a network with no outbound route

• Model catalog self-hosted and configured, so no fallback to a vendor endpoint occurs

• No telemetry required for core operation

• Optional outbound integrations explicitly off

• Authentication resolves internally

• Packet capture and DNS logs reviewed across cold start, inference, model swap and failure

• Verification repeated as part of the upgrade process

Xinity Runtime is open source under Apache 2.0. No outbound call is required to serve inference. Requests, responses, labels, model weights and GPU telemetry stay inside your infrastructure by default, and every optional outbound integration is opt-in. Weight acquisition is the one outbound path in a default deployment: it runs in the management layer rather than the inference engine, it brings public model weights in rather than sending your data out, and you can replace it with manual staging where your policy requires a hard air gap. You can read the code rather than take our word for it.

[Read the architecture whitepaper →]