Technical readiness checklist for AI in production

AI systems pass demos with a ease that says nothing about production. The right question isn't "does it work?" — it's "what evidence shows it keeps working under real load, real data, and real adversaries?" This checklist is organized into eight blocks. Each item asks for concrete evidence — a log, a dashboard, a document, a test — not a verbal claim that "it's fine."

Use it as a launch gate. If a block has no evidence, the system doesn't go to production; it goes to a backlog with an owner and a deadline.

This article is the technical counterpart to "Executive checklist: 30 questions before scaling AI". The executive asks whether to scale; this document verifies whether scaling is safe without breaking things.

1. Data and index

An index without a version is a black box: nobody can explain why Tuesday's answer differs from Thursday's.

2. Identity and permissions

• Every model call carries the end user's identity, not just a service credential.
• Data scope returned respects the source's RBAC/ABAC — the model doesn't "leak" beyond what the user could otherwise see.
• API keys and service tokens rotate automatically, with evidence of the last rotation.
• An automated test attempts to access data outside the user's scope and fails as expected.

Evidence: authorization test report (not just authentication) run in the last release cycle.

3. Evaluation and regression

• Evaluation set (golden set) with real cases, not only synthetic ones, updated with past incidents.
• Quality metric defined before deployment (accuracy, hallucination rate, format adherence) with a minimum accepted threshold.
• Evaluation pipeline runs in CI on every prompt, model, or index change.
• Side-by-side comparison between the current version and the candidate before promotion.

Evidence: automated evaluation report with date, model version, and delta against baseline.

4. Observability and audit trail

• Every interaction is logged with: input, output, retrieved context, model version, cost, and latency.
• Audit trail allows reconstructing why the system produced a specific answer.
• Quality and cost dashboards are visible to whoever operates the system, not only to whoever built it.
• Alerts configured for quality drops, latency increases, and cost spikes.

Evidence: an example of a full interaction reconstruction from logs, from scratch.

5. Cost and limits

Evidence: cost report per business unit (per user, per ticket, per document processed).

6. Security

• Prompt injection: tests exist with known direct and indirect injection payloads (via retrieved documents), with a documented block rate.
• Secrets: no key, password, or token appears in a prompt, log, or model response — verified by an automated scanner.
• Egress: outbound traffic from the inference environment is restricted to approved destinations; tool calls (function calling) go through an allowlist.
• Tool sandboxing: any action the model can trigger (sending email, executing code, calling an external API) runs with minimal permission and mandatory logging.

Evidence: report from the most recent red-team or pentest exercise specific to AI, with findings and remediation status.

7. Incident response

• Specific runbook for AI incidents (toxic response, data leak, hallucination with business impact).
• Kill switch channel: ability to disable or roll back an AI feature without a full deployment.
• Defined role for who decides to pause the system, with an objective trigger criterion.
• Pre-approved communication for affected users in case of an incident.

Evidence: record of the most recent tabletop exercise for an AI incident, with date and participants.

8. Vendor exit plan

• Contract or architecture allows switching model providers without rewriting the entire application.
• Prompts, evaluations, and fine-tuning data are stored in a provider-independent format.
• Migration time and cost estimate documented and tested at least once (even partially).
• Abstraction layer (model router, anti-corruption layer) isolates the application from provider API changes.

Evidence: exit plan document, reviewed within the last 12 months, with a recorded portability test.

How to use this checklist

This isn't meant to become a six-month audit. For each block, assign a simple status:

• Green: evidence exists, is up to date, and was reviewed by someone outside the team that built it.
• Yellow: intent or a partial process exists, without complete evidence.
• Red: nothing exists.

A system with any red block in identity, security, or incident response shouldn't go to production with real data, no matter how well it performed in the demo.

What to do on Monday

1. 1.Run this checklist against the AI system closest to production today, and mark green/yellow/red for each of the eight blocks.
2. 2.For every red item in security, identity, or incidents, open a launch-blocking ticket — not a "later" note.
3. 3.Schedule the first AI incident response tabletop exercise within the next two weeks, even if the system has no real users yet.
4. 4.Write the vendor exit plan before signing any long-term contract — it gets more expensive afterward.

Further reading

Executive track:

• The executive checklist: 30 questions before scaling AI — the business view of this same topic.