Reference architecture for enterprise AI: layers, contracts, and perimeter

We published on the executive blog that "your company's ChatGPT is not an AI strategy." Here is the technical counterpart: the architecture design that turns that statement into something buildable, testable, and portable across vendors.

The problem the architecture solves

Subscribing to a vendor's enterprise plan doesn't create an AI platform — it creates a dependency. Without explicit layers, every team couples its business logic directly to a specific provider's API, its prompt format, and its cost model. Switching vendors, adding a second model for a specific use case, or simply auditing what was decided and why becomes a months-long project.

The reference architecture below separates these responsibilities into layers with stable contracts between them.

The seven layers

Applications and agents
        │
   Orchestration
        │
  Model gateway ── Retrieval (RAG) ── Memory/store
        │
     Identity
        │
 Observability and audit trail

1. Model gateway

Single exit point for any call to an LLM, internal or external. Responsibilities:

• Normalize the interface (an internal contract, not a vendor's SDK).
• Route by use case, cost, and latency across multiple models.
• Enforce rate limits, timeouts, and retry policies.
• Log every call (prompt, response, tokens, cost, model) before it leaves the perimeter.

Without this layer, portability across providers is theoretical: it exists in the vendor contract, not in your code.

2. Orchestration

Coordinates sequences of calls — to a model, to tools, to other services — to complete a task. This is where agents, reasoning chains, and state machines live. Orchestration should know nothing about the model vendor's specifics; it only talks to the gateway.

3. Retrieval (RAG)

Searches internal sources for relevant context before assembling the final prompt. Involves indexing, chunking, hybrid search (vector + lexical), and reranking. Retrieval quality usually matters more to the final result than model choice.

4. Memory and store

Persistent state across interactions: conversation history, preferences, prior decisions. Should be treated as corporate data — with an owner, a retention policy, and access control — not as disposable cache belonging to the AI provider.

5. Identity

Every AI call carries identity: user, application, agent. This enables scoped authorization (who can access which source, which tool, which model) and is a prerequisite for any meaningful audit trail.

6. Observability

Latency, cost, error rate, and quality metrics by route, model, and use case. Without this, switching models is a gamble, not a decision.

7. Audit trail

Immutable record of who asked what, which context was used, which model responded, and what was decided from it. This is the artifact that backs explainability and compliance when someone asks "why did the system decide this."

Portability across providers

Real portability happens on two fronts:

Switching models should be a configuration change in the gateway, followed by an evaluation cycle — not an application rewrite.

What stays inside the company perimeter

A practical rule: anything representing intellectual property, customer data, or business decision logic should live on infrastructure the company controls — vector store, memory store, audit logs, prompt definitions. What can cross the perimeter, under contract and with outbound DLP, is the minimum needed for inference: the final prompt and the context strictly relevant to that call.

Common implementation mistakes

1. 1.Orchestration talking directly to the vendor's API. Removes any possibility of switching models without touching business code.
2. 2.Prompts embedded in application code. Prevents versioning, review, and independent testing (the topic of the next article in this series).
3. 3.RAG without per-source access control. A single index mixing public and confidential documents returns what it shouldn't, to whoever it shouldn't.
4. 4.Observability treated as generic logging. Without specific per-token cost and per-route quality metrics, architecture decisions become opinion.

What to do on Monday

• Map, for each existing AI integration, whether it goes through a central gateway or talks directly to the provider.
• List where your production prompts currently live: code, spreadsheet, vendor console. If they aren't in a versioned repository, that's the next priority.
• Check whether any record exists of "who asked what" for production AI calls. If not, the audit trail is the most urgent gap.
• Pick one low-risk use case to test switching models through the gateway, without touching application code, and measure the real effort.

Further reading

Executive track:

• Your company's ChatGPT is not an AI strategy — the business view of this same topic.