Does a Sovereign AI Gateway Cost You Speed?

Every team evaluating on-premise AI asks the same question. A gateway that adds authentication, request admission, retries and observability sounds like overhead. Overhead sounds like latency. So does putting an enterprise layer in front of your model make it slower?

We measured it, and we designed the test to work against us.

Key takeaways

• On identical hardware and model weights, the Xinity gateway matched plain vLLM on per-stream token rate and beat it on peak throughput.

• At 512 concurrent users on 65,536-token prompts, Xinity completed 85.0% of requests. Plain vLLM completed 4.6%.

• That is up to 18x more requests finishing under extreme load, the point where an unmanaged server falls over.

• The only cost is about 56 ms of extra time to first token, which is the network path, not the model.

• The gateway ran across the public internet with WireGuard and TLS. Plain vLLM ran on loopback with no network at all. We still won.

The honest test

The temptation with a vendor benchmark is to tilt the field. We tilted it the other way.

Both runs used the same benchmark harness, the same hardware and the same model weights. The only variable was the network path.

• Xinity gateway: requests traveled over the public internet, through a WireGuard tunnel, then TLS termination. About 60 IP hops before a token was generated.

• Plain vLLM: requests hit the model on loopback. Localhost. Zero network in the way.

If a gateway adds a speed tax, this setup exposes it. We gave the baseline every advantage and handicapped ourselves on purpose.

The setup

• Hardware: 1x NVIDIA DGX Spark, 128 GB

• Model: Qwen 3.6-35B-A3B-FP8

• Scenarios: 120, sweeping 5 input sizes, 3 output sizes and 8 concurrency levels

• Prompt size: 256 to 65,536 tokens

• Output length: 64 to 2,048 tokens

• Concurrency: 1 to 512 in-flight requests

• Volume: 31,200 requests per run

The results

The line that matters is completion rate under load. Both systems serve single streams at the same speed. The difference appears when the box is saturated. Plain vLLM holds up to roughly 8 concurrent users, then falls off a cliff. Xinity stays near the top of the chart the whole way to 512.

Why is the gateway faster under load, not slower?

This is the counterintuitive part, so here is the mechanism in plain terms.

The first-token gap is only the network

The 56 ms difference in time to first token is the round trip across the public internet, WireGuard and TLS. It is the cost of the network path, not the gateway logic. Once generation starts, per-stream token rate is identical at 50.0 tok/s.

Bounded admission keeps the scheduler healthy

Peak aggregate throughput is higher through the gateway because controlled request admission keeps vLLM's scheduler inside its efficient operating range. An unmanaged server accepts everything at once, thrashes, and its effective throughput drops. Admission control is not overhead here. It is what lets the hardware do its job.

Transparent retries catch failures before the client sees them

Under sustained overload, bare vLLM starts emitting malformed streaming chunks. The client sees stream parse errors and the request fails. The Xinity gateway catches these internally and re-dispatches before the failure ever reaches the caller. That is the difference between a 4.6% completion rate and an 85.0% one.

The result is a system that stays usable in production instead of one that falls over the moment traffic spikes.

What this means for regulated industries

If you run AI in finance, healthcare, legal or the public sector, the interesting number is not tokens per second on a quiet box. It is how many requests complete when every user hits the system at 9 a.m. on a Monday.

A model that is fast in isolation but drops 95% of requests under load is not a production system. It is a demo. The gateway layer is what turns raw inference into something you can put in front of real users, on your own hardware, without sending data to anyone else's cloud.

That is what we mean by sovereign by architecture, not by contract. The control lives in the infrastructure, not in a data processing agreement.

The honest caveats

We would rather you trust the numbers than be surprised by them later.

• Time to first token is about 56 ms slower through the gateway. For a system on the public internet versus one on loopback, that is expected and small. If your workload is many tiny single-shot prompts with no concurrency, plain vLLM on the same box will feel marginally snappier to first token.

• "Up to 18x" describes the extreme. It is measured at 512 concurrent users on 65,536-token prompts, the hardest corner of the sweep. At low concurrency the two systems are close, because there is nothing to fall over yet. The gateway earns its keep as load rises.

• This is a network-path comparison. We isolated one variable on purpose. It is a fair way to answer "does the gateway cost speed," and the answer is no.

Frequently asked questions

Does adding an enterprise gateway to vLLM reduce inference speed? No. In our benchmark, per-stream token generation rate was identical at 50.0 tok/s and peak throughput was higher through the gateway. The only measurable cost was about 56 ms of extra time to first token, which is the network path.

What is the completion rate difference between Xinity and plain vLLM under load? At 512 concurrent users on 65,536-token prompts, the Xinity gateway completed 85.0% of requests (870 of 1024) while plain vLLM completed 4.6% (47 of 1024), roughly 18x more.

Why does an unmanaged vLLM server fail under high concurrency? Without request admission control, the scheduler accepts too many requests at once and leaves its efficient operating range. Under sustained overload it also emits malformed streaming chunks that surface as errors to the client. A gateway with bounded admission and transparent retries prevents both.

What hardware and model were used? A single NVIDIA DGX Spark with 128 GB, running Qwen 3.6-35B-A3B-FP8, across 120 scenarios and 31,200 requests per run.

Is this benchmark reproducible? Yes. Both runs used identical harness code, identical hardware and identical model weights. The only difference was the network path. Full per-scenario reports and aggregate summaries are published with the data.

Methodology and data

Both runs used the same benchmark harness, the same DGX Spark and the same Qwen 3.6-35B-A3B-FP8 weights. The Xinity path ran from a public endpoint over the internet, through WireGuard, to TLS termination on the box. The plain vLLM path ran on loopback. Each run covered 120 scenarios, 5 input sizes by 3 output sizes by 8 concurrency levels, at 31,200 requests total.

Full per-scenario reports and aggregate summaries are available on request. If you want to run it on your own hardware, we will share the harness.

Sovereign by architecture, not by contract. See how Xinity runs on your infrastructure at xinity.ai.