Catch prompt regressions in CI before your users do

Change a prompt to fix one case and you can silently break five others. Nothing throws, no test goes red, the app still returns confident-looking text — and the regression ships to your biggest customer. If you edit prompts on vibes, you will ship one of these eventually. The fix is to treat prompts like the code they are: version them, and run an eval suite against the real production path before every change.

An eval suite points at your actual prompt

The trap with prompt tests is grading a copy of the prompt that drifts from what production runs. So a suite here calls the real helper — the same function your route calls — and grades its output. The shape is deliberately small:

export interface EvalSuite {
  name: string;
  // The production code path under test — call your real prompt/helper here.
  run: (input: string) => Promise<string>;
  cases: EvalCase[];
}

Because the suite runs the production code path, a prompt edit is exercised the moment you run the suite — no separate fixture to keep in sync.

Two kinds of grader: cheap-and-exact, then fuzzy

Most of what you want to assert is format and policy: does the output contain the disclaimer, does it avoid a banned phrase, does it match a shape. Those are deterministic string checks — instant and free. Reach for the model only for things a string cannot capture ("politely declines without inventing a policy"), and write the rubric as a checkable statement, not a vibe:

export const llmJudge =
  (rubric: string, model: ModelId = "claude-opus-4-8"): Grader =>
  async (output) => {
    const { data } = await generateObject({
      model,
      schema: verdictSchema, // { pass: boolean, reasoning: string }
      system:
        "You are grading the output of an AI product feature against a rubric. " +
        "Judge strictly: pass only if the output clearly satisfies the rubric.",
      messages: [{ role: "user", content: rubric + "\n\n" + output }],
    });
    return { pass: data.pass, detail: data.reasoning };
  };

The judge returns a typed verdict via structured output, so a grader is just a function returning pass plus a reason — deterministic and LLM graders compose in the same list.

Make CI fail, not just print

An eval that logs a warning gets ignored. The runner returns a single boolean — true only if every case passed — and the suite script exits nonzero on false, so a red eval blocks the merge exactly like a failing unit test:

const ok = await runSuite(suite);
process.exit(ok ? 0 : 1);

Pair that with a versioned prompt registry: each prompt carries a version, and the eval output prints it, so when a suite goes red you can tell whether the cause was a prompt edit or a model upgrade — the same deterministic registry that makes prompt caching actually work.

Things that bite

• Evals cost real tokens. They make live API calls, and the LLM judge is itself a model call. Keep deterministic graders first and gate them so the cheap checks fail fast before you pay for a judge.
• A vague rubric grades vaguely. "Is it good" gives you a coin flip. "Declines the refund and offers to escalate, without inventing an exception" is checkable.
• Judge strictly. A lenient judge passes everything and tells you nothing; the system prompt here forces a strict pass bar on purpose.
• Version the prompt, or you are debugging blind. Without a version stamped on the run, a regression could be the prompt, the model, or the weather.

Shipwright — the Next.js and Claude starter kit this blog documents — ships this eval harness wired into CI, alongside the versioned prompt registry and per-request cost metering, so quality regressions surface in a pull request instead of a support ticket. You can try the prompts it grades in the live demo.