How Does On-Prem Infrastructure Scale?

It is the first question every CTO asks when on-premise AI comes up, and it is usually asked with a raised eyebrow. Cloud is supposed to be the thing that makes capacity someone else's problem. Owning hardware sounds like the opposite: rigid, slow, a return to the server room era.

The eyebrow is fair. It is also worth noticing that at the top of the GPU market, cloud does not really make capacity someone else's problem either. Current-generation accelerators come with quotas, waitlists, regional availability limits and reservation commitments. Cloud has not removed the capacity question from your roadmap. It has turned it into a procurement negotiation you conduct through a sales rep instead of a purchase order.

So the real comparison is not elastic versus rigid. It is which kind of growth you want to plan for. Owned AI infrastructure scales differently than cloud, not worse, and for inference workloads specifically the difference often works in your favor. Here is what scaling actually looks like when the hardware is yours.

Start with what scaling means for inference

Training runs are bursty. Inference is not. An internal AI assistant, a document processing pipeline, or a coding agent produces a load curve that is remarkably flat: high during working hours, predictable week over week, growing steadily as adoption spreads.

On-demand cloud pricing is built for the opposite pattern, and it is worth being precise about why it costs what it costs. The premium in an on-demand rate is not a charge for the hardware. It is the price of an option: the right to add or drop capacity at any moment, without commitment. That option has real value when your load is spiky. When your load curve is flat, you buy it every hour and never exercise it.

That is the trade we call the Utilization Inversion. Not a new discovery in cloud economics, but a name for the specific shape it takes in AI inference: the workloads driving most enterprise value are the ones that get the least out of what cloud pricing charges for. Always-on agents running at high sustained utilization are precisely the case where renting elasticity you never use is the most expensive way to buy compute.

How much cheaper depends entirely on your numbers, which is why we would rather you run them than take a headline percentage from a vendor. Our ROI calculator asks for your GPU class, your load profile and your amortization assumptions, and shows the crossover point for your case.

It also reframes the scaling question. You are not asking "how do I burst to 10x capacity for an hour?" You are asking "how do I grow capacity in step with steadily rising demand?" That is a much easier engineering problem, and it is the one owned infrastructure is good at.

Scaling up: prepare once, join with one command

In Xinity, the control plane, meaning the gateway, dashboard, database, and cache, is deployed once. Compute capacity is a separate, independent layer. A new GPU machine needs its driver stack and an inference engine in place first, vLLM and/or Ollama depending on which models you intend to serve, and that is usually done as part of provisioning the box. Once the machine is prepared, joining it to the pool is a single command:

xinity up daemon

xinity up daemon

xinity up daemon

The daemon connects to the shared PostgreSQL instance, registers itself, detects its accelerators, and begins receiving model deployments. Detection covers NVIDIA through nvidia-smi, AMD through sysfs with an rocm-smi fallback, and Intel through xpu-smi. Unified memory systems like DGX Spark are recognized when GPUs report zero VRAM, with 90 percent of system RAM treated as usable capacity, and CPU-only inference works when there is no GPU at all. The workflow is identical whether your control plane runs on bare metal, Docker, or NixOS.

Two things worth stating plainly here rather than leaving you to discover them.

The engine is Apache 2.0 and runs without the dashboard. Node selection is a dashboard function, so multi-node orchestration requires a paid dashboard tier. The free tier covers one organization and one inference node, which is enough to evaluate the platform or run single-node production.

And the single command is the registration step, not the whole story. Integrating a machine into the stack means installing our software alongside an inference engine, vLLM and/or Ollama, with a modest amount of configuration, normally before the daemon ever comes up. Ordering the machine, racking it, cabling it and verifying power and cooling is still work with a lead time. What it does mean is that capacity planning becomes procurement planning rather than an engineering project. The Compute page polls every GPU node every twelve seconds, and Prometheus picks up new nodes automatically through a service discovery endpoint, so the trend line that tells you when to order is already in front of you. When utilization approaches your ceiling, you order a machine, prepare it, and run one command. There is no re-architecture, no migration, and no application changes, because the gateway routes traffic across whatever nodes exist.

Scaling smart: placement is a policy, not a guess

A fleet of GPU machines is only as good as the logic that decides where models run. The deployment sync service supports four node-selection strategies, and choosing one is a one-line configuration:

Balanced, the default, places each model on the node with the most absolute free VRAM, spreading load naturally and giving you high availability as a side effect.

Bin-pack does the opposite: it consolidates workloads as tightly as possible, keeping some nodes idle and drainable. This is the strategy for teams that want to power down machines overnight or keep spare nodes ready for maintenance.

Proportional targets the lowest percentage utilization, which matters when your fleet is heterogeneous, mixing older and newer GPUs, and absolute free memory would mislead the scheduler.

First-fit takes the first node that fits the model's requirements. Deterministic and boring, which is exactly what some compliance environments want: the same input always produces the same placement.

None of these strategies is novel on its own. They are the standard vocabulary of schedulers. The point is that on owned infrastructure, placement is a policy you set rather than an opaque decision a cloud scheduler makes on your behalf.

Scaling under load: where the gateway earns its keep

More machines only help if traffic reaches them intelligently. The gateway offers three load-balancing strategies: least-connections, the default, which picks the node with the fewest in-flight requests tracked in Redis and falls back to random if Redis is unavailable; round-robin, using an atomic Redis counter per model; and uniform random.

On top of that, least-connections and random implement prefix-cache affinity. Conversation prefixes are hashed and mapped to nodes with a five-minute TTL, so repeat conversations return to the node that already holds the relevant KV cache. Under least-connections the affinity is deliberately conditional: it is honored only when the hinted node is within two connections of the least-loaded one, so cache locality never wins at the cost of piling requests onto a saturated node.

Under sustained high concurrency, this layer does something counterintuitive. We measured it on a single DGX Spark with 128 GB, serving Qwen 3.6-35B-A3B-FP8 across 120 scenarios that swept prompt sizes from 256 to 65,536 tokens and concurrency from 1 to 512, at 31,200 requests per run.

Per-stream token generation was identical through the gateway and direct: 50 tokens per second either way, so the gateway is not a bottleneck. Peak aggregate throughput was higher through the gateway, 352 tokens per second against 325, because bounded request admission keeps the engine in its efficient operating range. The interesting number is completion rate at the extreme: at 512 concurrent requests with 65k-token prompts, 85 percent of requests completed through the gateway against 4.6 percent direct, because bare vLLM starts emitting malformed streaming chunks under overload and the gateway's retry layer re-dispatches them before the client sees a failure.

Two caveats we would rather state than have you find. The gateway path in that test traversed the public internet, a WireGuard tunnel and TLS termination while the baseline ran on loopback, which costs us about 56 milliseconds of time-to-first-token and means the throughput result was achieved from the disadvantaged network path. And it is a single-node benchmark: it measures what the gateway does for the capacity you have, not what multi-node scaling adds. Identical harness, identical hardware, identical weights, network path as the only variable, per-scenario data committed to the repository.

Scaling is not just adding capacity. It is protecting the capacity you have from the chaos of real-world traffic.

Scaling without downtime

Growth also means change: new model versions, engine upgrades, node maintenance. Canary deployments split traffic between model versions either at a fixed percentage you advance manually, or interpolating linearly over a configured time window until all traffic has moved. Multi-replica deployments keep a model available if a node fails, offline nodes are detected and excluded from host selection immediately, and daemon-managed inference processes restart automatically up to three times before giving up. Fatal errors like GPU out-of-memory fail fast instead of retrying, which is the right behaviour when a retry cannot succeed.

For a 24/7 production posture, the recommended setup is straightforward: multiple gateway replicas behind a load balancer, a PostgreSQL cluster with replication and failover, high-availability Redis, and the same models deployed across several GPU machines. The compute layer scales horizontally and the control plane runs highly available, so no single box is a hard limit on the system.

One thing we will not claim: because Xinity runs on your infrastructure, uptime depends on your infrastructure choices and operational practices. We cannot offer a hard SLA on hardware we do not run. What the platform can do is make high availability straightforward for operators who want it.

The honest ceiling

Owned infrastructure has real constraints, and two are worth naming.

The first is elasticity. You cannot conjure a hundred GPUs for a one-week experiment. If your workload is genuinely spiky and experimental, cloud bursting has its place, and a hybrid setup is a legitimate answer.

The second is more specific, and it is the question that usually follows "how does it scale?" Adding machines adds replicas. It gives you concurrent throughput and redundancy for models that fit on the hardware you have. Within a single node, vLLM splits a model across GPUs with tensor parallelism. Splitting a model across machine boundaries is a different problem, and it is on our roadmap as distributed split inference rather than something you can deploy today. If your requirement is serving a model larger than any single node you can buy, that is worth a conversation before anything else.

But for the workloads that drive most enterprise AI value, the steady, growing, always-on ones, the ceiling you hit on owned hardware is a purchase order, not an architecture. And every step of growth you pay for once, you keep.

Curious what your growth curve would cost on your own hardware? Our ROI calculator gives you a first estimate, or book 30 minutes and we will model it with your numbers.

This post was drafted with AI assistance. All benchmark figures, engine behaviour and platform claims were verified by the Xinity team before publication.