Build Your Own AI Data Center, Part 1

The phrase "build your own data center" makes people picture Frankfurt. Raised floors, diesel generators, a nine-figure budget and a three-year construction plan.

That picture is why most companies never even consider running AI on their own hardware. It is also wrong.

For AI inference, a data center is not a building. It is a unit of control: hardware you own, in a room you control, running models you chose. That can be a single compact machine on a shelf. It can be one rack in the server room you already have. Very few companies ever need more.

Here is how to actually do it.

Step 1: Size the workload, not the ambition

Do not start with hardware. Start with one question: what will this system do on a normal Tuesday?

Concrete numbers to gather:

How many people will use it, and how many of them at the same time? A 200-person company rarely has more than 15 to 20 concurrent AI users. Which tasks? Document summarization, internal chat, RAG over your file shares, code assistance. Which model size covers those tasks? For most internal enterprise work, models in the 8B to 70B parameter range are enough. You do not need a frontier model to summarize contracts.

One of our customers, a media company, runs around 14,000 API calls a day for 120 users. That is a real production workload, and it does not require a hall of GPUs.

Step 2: Pick your hardware tier

There are three realistic entry points.

A single inference box. Machines like the ASUS Ascent GX10 are compact desktop-format systems built for exactly this. They run serious models, plug into a normal power socket, and sit in a locked office. This is the right starting point for pilots and for companies under a few hundred employees. Investment: five figures, low end.

A GPU workstation or single server. One or two data center GPUs in a tower or 2U chassis. More memory, more concurrent users, still fits in an existing server room. Investment: mid five figures.

A rack. Multiple GPU nodes, redundant power, proper networking. This is for organizations serving thousands of users or heavy batch workloads. Most companies reading this will never need it, and that is fine. You can grow into it. The software does not change.

The common mistake is buying tier three capacity for tier one workloads. Buy for the Tuesday workload, with roughly 30 percent headroom. Scaling later means adding a node, not starting over: the gateway load-balances requests across every node it sees, so a second machine joins the pool and starts taking traffic without any change to your applications. More on that in Step 5.

Step 3: Solve power, cooling and placement honestly

This is where "own data center" sounds intimidating and mostly is not.

A single inference box draws less power than a coffee machine. A GPU server draws 1 to 3 kW, which a standard office circuit and a normal air-conditioned room handle without drama. Only at rack scale do you need to talk to facilities about dedicated circuits and cooling.

What you do need from day one: a lockable room, a UPS so a power blip does not corrupt anything, and a clear answer to who has physical access. Physical access control is not bureaucracy. It is the foundation of every compliance claim you will make later.

Step 4: Keep the network boring

The strongest security feature of on-premise AI is the one you get for free: the traffic never leaves.

Put the inference system in its own VLAN. Expose one endpoint to your internal network. No inbound connections from the internet, and ideally no outbound ones either. Model weights come in once, through a controlled update process, not a live connection to someone's cloud.

If your firewall rules for the AI system fit on one screen, you did it right.

Step 5: The software layer is the actual product

Hardware is the easy part. What turns a GPU in a room into infrastructure your company can use is the software on top: model serving, user management, an OpenAI-compatible API so existing tools connect without rewrites, monitoring, and audit logs.

This is the layer we build. Xinity runs on your hardware, serves open models through an OpenAI-compatible endpoint, and gives you the operational layer around it. The core is open source under Apache 2.0, so you can verify every claim in this post by reading the code.

It is also the layer that makes on-prem scalable. Xinity's gateway load-balances every request across your inference nodes, picking the least loaded machine, retrying transient failures before the user ever sees one, and routing repeat conversations back to the node that already has the context cached. One box or ten, your applications talk to a single endpoint and the gateway does the distribution. Under sustained load this matters more than raw GPU count: bounded request admission keeps each node in its efficient operating range instead of letting traffic spikes push it off a cliff.

The OpenAI compatibility matters more than it sounds. It means the tools your teams already use, from office suites to internal apps, can point at your own metal by changing a base URL. Migration is a config change, not a project.

Step 6: Do the compliance work while it's cheap

On-premise does not make you compliant by itself. It makes compliance provable.

Under GDPR, the EU AI Act and, where they apply, NIS2 and DORA, the questions are always the same: where is the data, who can access it, what happened when. When the answer is "in this room, these people, here are the logs," audits get short. When the answer involves a foreign cloud provider's sub-processors, audits get long and legal opinions get expensive.

DORA deserves its own sentence for anyone in financial services. It treats dependence on critical ICT third parties as a risk you must register, assess, and be able to exit. An inference stack on your own hardware is one less third party in the register, one less concentration risk to explain, and an exit strategy that is trivially true because there is nothing to exit from.

Document three things from day one: the physical and network access rules, the model and update process, and the logging. That is most of your audit evidence, produced as a side effect of running the system properly. Architecture as evidence, not paperwork as promise.

Step 7: Do the math over 36 months, not one invoice

Cloud AI pricing is built on a quiet assumption: that you will stay small. Per-token pricing is cheap at low volume and brutal at production volume.

Owned hardware inverts this. The costs are front-loaded and then flat. The more you use it, the cheaper every request gets. One of our customers cut their per-request cost from €2.00 to €0.06 after moving inference in-house. That is a 97 percent reduction, and it did not come from a discount. It came from utilization.

The break-even point depends on your workload, but for any company with steady daily usage it typically lands within the first one to two years. Everything after that is margin you keep instead of rent you pay.

To be clear about the trade-off: if your usage is sporadic or experimental, cloud is the right tool. Buy hardware when the workload is real. Not before.

Start with one box, not a blueprint

You do not need a construction plan. You need one machine, one pilot use case, and 30 days of real usage data.

That is exactly how our pilot works: an ASUS Ascent GX10 with the full Xinity software in your building, free for 30 days, your data never leaving the premises. You run your actual workload and see whether the math holds up for your company.

Apply at xinity.ai/pilot.

Next in this series: Part 2, where Jonas takes the first node from bare metal to serving its first model, terminal session and all.