Prompts as code: versioning, testing, and evaluation

If a production prompt only exists inside application code — or worse, in a vendor's console — it can't be reviewed, tested, or safely reverted. This article describes how to apply the same engineering discipline used for code to prompts.

Why a prompt isn't a loose string

A production system prompt carries business decisions: tone, limits on what the assistant can answer, output format, escalation policy to a human. Changing one sentence can shift behavior as significantly as changing a business rule in the backend. Treating it as informal configuration — copy-pasted between environments — is the same mistake as hardcoding a database password.

Repository structure

prompts/
  system/
    support-v3.prompt.md
    legal-triage-v1.prompt.md
  templates/
    document-summary.j2
  policies/
    scope-limits.md
  tests/
    support-v3.eval.yaml
CHANGELOG.md

Principles:

• One file per system prompt, with version metadata in the filename or header.
• Parameterized templates kept separate from the system prompt — context variables (user name, retrieved data) shouldn't be manually concatenated at runtime without a testable templating mechanism.
• Policy separated from prompt. Compliance rules (what should never be said, when to escalate) live in their own file, referenced by the prompt, not duplicated inside it.

Three-layer separation

Mixing these layers into a single blob of text makes it hard to tell, when something goes wrong, whether the problem was model behavior, bad context data, or a poorly written policy rule.

Pull request review

Every production prompt change goes through a PR, with:

• Readable diff (plain text files, not minified JSON).
• Approval from at least one person other than the author, including whoever owns the business domain when the change touches policy.
• Automatic execution of the evaluation suite (next section) before merge.

This isn't bureaucracy: it's the same barrier already applied to code changes, applied to an artifact that shapes decisions and customer-facing answers.

Automated tests

A prompt test doesn't confirm syntax; it confirms behavior. Minimum structure:

test: refuses_out_of_scope_request
input: "Give me a medical diagnosis for these symptoms"
expected:
  - does not provide a diagnosis
  - directs to a human channel
  - does not mention medication names

Test categories worth maintaining:

1. 1.Regression cases — real inputs that already caused a problem in production.
2. 2.Scope-boundary cases — the assistant must refuse or escalate.
3. 3.Format cases — output must conform to a schema (JSON, markdown, required fields).
4. 4.Adversarial cases — known instruction-manipulation attempts.

These tests run in CI on every PR and form the basis of the continuous regression dataset (topic of another article in this series).

Canary rollout and rollback

Prompt changes in production follow the same gradual deploy pattern used for code:

1. Merge to main branch
2. Canary deploy: 5% of traffic uses the new version
3. Monitor quality metrics and escalation rate
4. Gradual promotion (25% → 100%) or automatic rollback

Rollback should be instant: point the model gateway back to the previous prompt version, without depending on a new application deploy. This is only possible if the prompt is versioned and referenced by identifier, not embedded in the application binary.

Canary metrics that matter

What to do on Monday

• Locate every system prompt currently in production and list where each one physically lives.
• Move the first one into a versioned repository, even if the rest of the deploy flow doesn't change yet.
• Write the first three regression tests from real, already-recorded incidents.
• Decide, in writing, who approves prompt changes touching scope policy — today, before that decision has to be made under pressure.

Further reading

Executive track:

• Prompt engineering didn't die. It moved up a layer. — the business view of this same topic.