How to evaluate on-premise AI for banks

The short answer

A bank should evaluate on-premise AI the way its supervisor will: by asking who can see customer data, what gets logged, and how the bank stays in control if a vendor fails. The model matters less than the layer around it, meaning access control, a per-request audit trail and policies you can show an auditor.

This guide covers why cloud AI is a hard fit for banks, what supervisors and internal audit will ask, and ten questions to put to any vendor before you sign.

Why can't banks just use cloud AI?

They can for some tasks, but not for anything that touches customer data without a lot of extra work. Three rules make it hard.

Bank secrecy. In Austria, §38 of the Banking Act (BWG) protects everything a bank learns about its customers. A prompt with a client's name, account history or loan file is covered, and so is the answer the model sends back.

DORA. The Digital Operational Resilience Act, Regulation (EU) 2022/2554, has applied since 17 January 2025. It treats an external AI API as an ICT third-party service: it goes into the register of information, needs an exit strategy, and counts toward concentration risk if many banks rely on the same provider.

Outsourcing rules. Under the EBA guidelines on outsourcing, sending data to an external AI service can count as outsourcing, with its own contract, audit and notification duties.

None of this bans AI. It means every request that leaves the building creates paperwork, risk and a dependency. Running the model on your own servers removes most of that, because the data never leaves.

What will supervisors and auditors ask?

Expect the same questions internal audit asks about any core system, now pointed at AI:

• Who used it, and for what? Which employee sent which request, from which team, on which day.

• Which model answered? The model and version behind a given output, so a result can be explained later.

• Where did the data go? Whether prompts, documents or outputs were stored, and where.

• Who is allowed to do what? How access is granted, reviewed and removed when someone changes role.

• What happens if the vendor disappears? Whether the bank can keep running, switch models or leave without losing its setup.

The EU AI Act adds a sharper version for some uses. Credit scoring and creditworthiness checks of individuals are classed as high-risk, with obligations now due from 2 December 2027. Banks building toward those use cases need the logging and oversight in place well before then.

What should a bank evaluate in an on-premise AI platform?

Running a model on your own hardware is the easy part. What makes it fit for a bank is the control layer around it.

Ten questions to ask any on-premise AI vendor

1. Does any prompt, document or output ever leave our servers, including for telemetry, updates or support?

2. Can we see, per request, which user, team and model was involved?

3. Does access control connect to our existing identity provider through single sign-on?

4. Can we restrict which teams use which models?

5. Which models are validated to run on the platform, and who decides when to change them?

6. Can our existing applications connect without rewriting them?

7. Which security and compliance controls do you provide, and which stay with us? Is that written down?

8. Can the software run fully offline or air-gapped if our policy requires it?

9. If we stop working with you, what keeps running and what do we lose?

10. Can our internal audit or an external auditor review the code and the logs?

A vendor who answers all ten clearly, in writing, is one you can take to your risk committee.

Where Xinity fits

Xinity is the control layer around the model. It runs validated open-weight models on your own servers and adds role-based access, single sign-on and a per-request audit trail. The API is OpenAI-compatible, so existing applications connect without a rewrite.

The core software is open source and public on GitHub, so your security team can read the code before anything is installed. Our shared responsibility model sets out which controls Xinity provides and which stay with your teams.

See how this applies to your bank on our Sovereign AI for Banks page, or start with the 30-day pilot.

FAQ

Is on-premise AI compliant with bank secrecy? Running AI on your own servers keeps customer data inside the bank, which removes the main secrecy risk of external AI services. Compliance still depends on your access rules, logging and internal processes.

Does DORA apply to on-premise AI? DORA covers your ICT risk either way. On-premise AI reduces third-party dependency, but the software vendor can still be an ICT third-party provider, so it belongs in your assessment.

Are open-weight models good enough for banking? For most internal tasks, such as summarising documents, drafting and searching policies, current open-weight models are capable. The gap is usually governance, not model quality.

What does the EU AI Act mean for AI in banks? Credit scoring and creditworthiness checks of individuals are high-risk under the EU AI Act, with obligations due from 2 December 2027. Most internal assistant use cases fall outside that category.

Can we start small? Yes. Most banks start with one internal use case on existing or pilot hardware, then expand once audit and security have signed off.

AI declaration

This article was drafted with the help of AI and reviewed and edited by the Xinity team. It is general information, not legal advice.