Cloud, private cloud or on-premise: where your AI should run
A decision matrix by data sensitivity and criticality: total cost, sovereignty, open versus proprietary models, and exit strategy.
*Ninth article in our series on enterprise AI, legacy integration and information governance.*
Executive summary
There is no single answer. There is a matrix: the more sensitive the data and the more critical the operation, the more control the company needs over where processing happens — and the higher the bill. The wrong decision costs money one way (idle infrastructure) or credibility the other (sensitive data processed where it should not be). The good news is that the choice does not have to be uniform across the company.
The three options, without the marketing
Public model API. Least effort, best available quality, pay per use. Data leaves the perimeter; control is contractual, not technical.
Private cloud or dedicated region. Models hosted in an isolated environment under contract, with a defined region and no shared inference infrastructure. Intermediate cost, substantially more control.
On-premise with open models. Processing inside your own datacenter. Maximum control and predictable cost at high volume; requires staff, hardware and update discipline — and quality tracks the available open model, not the commercial state of the art.
| Criterion | Public API | Private cloud | On-premise |
|---|---|---|---|
| Time to first value | Days | Weeks | Months |
| Upfront cost | Low | Medium | High |
| Cost at high volume | High | Medium | Predictable |
| Control over data | Contractual | Contractual + isolation | Total |
| Model quality | State of the art | Near state of the art | Good, with a lag |
| Operating effort | Minimal | Moderate | High |
The deciding criterion: data classification
Combine two axes — data sensitivity and failure impact — and the answer emerges per use case, not per company:
- 1.Public or low-impact internal data: a public API is fine.
- 2.Confidential business data with moderate impact: private cloud or dedicated region.
- 3.Sensitive personal data, trade secrets or critical operations: a controlled environment, with a serious on-premise evaluation.
Hybrid architectures are the norm in mature companies: the general assistant uses a public API with masking; the engine analyzing product formulas runs internally.
Sovereignty and data residency
Regulated industries and public-sector contracts often require data to stay in a given jurisdiction. That constrains vendors and regions — and must be verified at the inference subprocessor level, not only at the interface vendor's, as detailed in Where what your employees type into AI ends up.
Total cost: the line items that vanish from spreadsheets
- Hardware and its depreciation on-premise, including idle capacity outside peak hours.
- People: running models internally requires a team that did not exist before.
- Reindexing and storage of the context archive.
- Switching cost: what would it take to migrate if today's decision proves wrong in 18 months?
Exit strategy from day one
Whatever the choice, keep three things outside the platform: the context archive, versioned prompts and templates, and the audit trail. With those assets preserved, changing where execution happens becomes an infrastructure project rather than a restart.
What to do on Monday
- Classify your top five use cases on both axes (sensitivity and impact).
- Ask each vendor for the effective inference region and the subprocessor list.
- Compare cost across three volume scenarios, not only at pilot volume.
- Write, in half a page, how a migration to another vendor would work — if you cannot write it, dependency is already too high.
Conclusion
Where AI runs is a risk decision, not a technology preference. Classify the data, decide per use case, and preserve the assets that keep the choice reversible.
Further reading
Engineering track:
- Serving LLMs on-premise: GPU, quantization, and throughput — the technical deep dive on this topic.