Building a Golden Dataset to Regression-Test LLM Features

A golden dataset is a versioned collection of realistic inputs, each paired with an expected answer or a set of properties the answer must have, that you rerun automatically whenever you change a prompt, model, retrieval step or tool. Run it before and after a change, compare the scores, and block the change if quality drops. This guide builds one in Python, from collecting cases to wiring it into CI.

Who this is for and what you will have

This article is for developers who have shipped, or are about to ship, an LLM feature such as a support assistant, extractor or classifier, and who currently check changes by trying a few prompts by hand. By the end you will have a JSONL dataset format, a runner with three kinds of graders, pass thresholds, a CI step, and a routine for growing the set from real failures.

OpenAI's evaluation guidance names the failure mode this prevents: "vibe-based evals", or putting evaluation off until after launch. Anthropic's guidance for building tests makes the same case: define success criteria first, then design evals that mirror your real task.

Step 1: Define success criteria

Before collecting data, write down what good means. Anthropic's guidance says criteria should be specific, measurable, achievable and relevant, and that most use cases need several at once. "Good performance" is not a criterion; "accurate sentiment classification" is, and so is "an F1 score of at least 0.85" or "answers cite a policy document ID".

Typical criteria for common features:

Pick the thresholds from your own needs and baseline measurements, not from a rule of thumb.

Step 2: Collect cases that mirror reality

Both vendors give the same advice: make the dataset reflect your real task distribution, and include edge cases. OpenAI lists sources for building a dataset: synthetic data, domain-specific data, human-curated examples, production data and historical logs. Anthropic lists edge cases to cover:

• Irrelevant or nonexistent input.
• Overly long input.
• Poor or harmful user input.
• Ambiguous cases where humans would also disagree.

A practical collection plan:

1. Take 20 to 50 real, anonymised inputs from logs or support tickets. Remove personal data before storing them.
2. Add edge cases from the list above, plus cases for each behaviour you promise, such as refusal when the answer is not in the documents.
3. Add known failures from bug reports. Each one is a regression test waiting to happen.
4. Use a capable model to suggest more variations if you need volume, but have a person review them. OpenAI's guidance suggests a model can help generate diverse scenarios while human experts label.

Choose a file format

JSONL, one JSON object per line, diffs cleanly in version control and is easy to append to.

{"id": "refund-001", "input": "How long do I have to return an item?", "expected": "30 days", "type": "contains", "tags": ["policy", "typical"]}
{"id": "refund-002", "input": "Can I return a gift card?", "expected": "no", "type": "contains", "tags": ["policy", "edge"]}
{"id": "scope-001", "input": "What is the capital of France?", "rubric": "Politely declines because the question is outside the support policy.", "type": "judge", "tags": ["refusal"]}
{"id": "format-001", "input": "Order CONTOSO-1042 shipped via Fabrikam Freight.", "expected": {"order_id": "CONTOSO-1042", "carrier": "Fabrikam Freight"}, "type": "json", "tags": ["extraction"]}

Each case has a stable id, the input, the way to grade it (type), and either an expected value or a rubric. The contains examples use Contoso-style placeholder policies; replace them with your own.

Step 3: Choose graders

Anthropic's guidance recommends automating grading when possible, and describes three families. OpenAI's guidance describes metric-based graders, human graders and LLM-as-judge. Combine them:

Anthropic's exact-match example is simply a comparison after stripping whitespace and lowercasing. OpenAI notes metric-based graders are useful for regression testing but can miss nuance, which is why you pair them with judged cases.

LLM-as-judge rules

From the vendors' guidance:

• Use a different model from the one that generated the output (Anthropic).
• Favour pairwise comparison or pass/fail scoring over open-ended scores, and ask for reasoning before the score (OpenAI).
• Watch for position bias and verbosity bias, and vary response order or control for length (OpenAI).
• Write a clear, detailed rubric (both).
• Validate the judge against human labels before relying on it (OpenAI).

Step 4: Write the runner

The runner below loads the JSONL file, calls your feature, grades each case and prints a summary. answer is a placeholder for your application's entry point; it takes a question and returns a string.

import json
import sys
from collections import defaultdict
from pathlib import Path
 
import anthropic
 
from myapp import answer  # your feature: str -> str
 
judge_client = anthropic.Anthropic()
JUDGE_MODEL = "claude-sonnet-5-5"  # use a different model from the one in `answer`
 
 
def grade_contains(output, case):
    return str(case["expected"]).lower() in output.lower()
 
 
def grade_json(output, case):
    try:
        data = json.loads(output)
    except json.JSONDecodeError:
        return False
    return all(data.get(k) == v for k, v in case["expected"].items())
 
 
def grade_judge(output, case):
    prompt = (
        "You are grading an AI assistant's answer.\n"
        f"Question: {case['input']}\n"
        f"Answer: {output}\n"
        f"Rubric: {case['rubric']}\n"
        "Think briefly, then finish with a final line that is exactly PASS or FAIL."
    )
    reply = judge_client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=300,
        messages=[{"role": "user", "content": prompt}],
    )
    text = reply.content[0].text.strip()
    return text.splitlines()[-1].strip() == "PASS"
 
 
GRADERS = {"contains": grade_contains, "json": grade_json, "judge": grade_judge}
 
 
def run(path: str, threshold: float = 0.9) -> int:
    cases = [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
    failures, by_tag = [], defaultdict(lambda: [0, 0])
 
    for case in cases:
        output = answer(case["input"])
        ok = GRADERS[case["type"]](output, case)
        for tag in case.get("tags", []):
            by_tag[tag][1] += 1
            by_tag[tag][0] += ok
        if not ok:
            failures.append((case["id"], output[:200]))
 
    passed = len(cases) - len(failures)
    rate = passed / len(cases)
    print(f"Passed {passed}/{len(cases)} ({rate:.0%})")
    for tag, (p, n) in sorted(by_tag.items()):
        print(f"  {tag}: {p}/{n}")
    for case_id, out in failures:
        print(f"FAIL {case_id}: {out!r}")
    return 0 if rate >= threshold else 1
 
 
if __name__ == "__main__":
    sys.exit(run(sys.argv[1]))

Run it with python run_evals.py golden.jsonl. The process exits non-zero when the pass rate falls below the threshold, which is what CI needs.

Handle non-determinism

OpenAI's guidance asks you to identify where variability enters: developer and user inputs, model outputs, tool selection and arguments, and agent handoffs. Practical consequences:

• Grade properties, such as "contains 30 days", instead of exact wording where you can.
• Use a pass-rate threshold across the set, not a requirement that every case passes every run.
• Re-run borderline cases a few times before declaring a regression.
• Record the model name, prompt version and dataset version with every run so results are comparable.

Step 5: Validate your graders

An LLM judge is itself a component that can be wrong. Before you trust it:

1. Pick 30 to 50 judged cases and have a person label each answer pass or fail.
2. Run the judge on the same answers.
3. Compare. Where they disagree, fix the rubric or the case, not the labels.
4. Re-check whenever you change the judge model or the rubric.

OpenAI's guidance says to check automated scores against human feedback, and not to ignore human feedback when calibrating automated metrics.

Step 6: Run it in CI

Gate changes that can alter behaviour: prompt files, model identifiers, retrieval settings and tool definitions. A GitHub Actions job, for example:

name: llm-evals
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'app/**'
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install -r requirements.txt
      - run: python run_evals.py golden.jsonl
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Keep a fast subset, tagged smoke, for every pull request, and run the full set nightly or before release, so cost and latency stay reasonable. If many eval calls run at once, expect rate limit errors; see Handling LLM API Rate Limits: Retry, Backoff and Queues in Python. For a batch of non-urgent judged cases, a provider's batch interface can also help.

A note on hosted evaluation tools

If you use a hosted evaluation product, check its lifecycle. OpenAI's evaluation best practices page states that the OpenAI Evals platform is being deprecated, with read-only access starting on 31 October 2026 and shutdown scheduled for 30 November 2026. A dataset stored in plain JSONL in your repository, with a runner you own, is unaffected by vendor changes.

Step 7: Grow the dataset

A golden dataset is never finished. OpenAI's guidance recommends logging everything during development so you can mine the logs for good test cases, running evals on every change, monitoring production and continuing to grow the set.

• Every production failure, bad answer or support complaint becomes a new case with an id, the failing input and the corrected expectation.
• Tag cases by behaviour so you can see where a change helps or hurts.
• Review the set quarterly. Remove cases that no longer reflect the product, and keep a changelog of dataset versions.
• Keep the dataset independent of any framework. A runner that calls answer(question) works with a hand-written SDK loop and with any framework discussed in LangChain vs Semantic Kernel vs the Plain SDK: When a Framework Pays Off.

The same dataset is also how you compare retrieval backends fairly; the test plan in pgvector vs Azure AI Search vs Qdrant: Choosing a Vector Store reuses it.

Common mistakes

• Testing only happy-path questions, so edge cases and refusals regress unnoticed.
• Using the same model as generator and judge.
• Tuning prompts until the dataset passes, then never adding fresh cases. The set stops predicting production quality.
• Storing real customer data in the dataset without removing personal information.
• Requiring a perfect score, so flaky cases train the team to ignore failures.

Troubleshooting

Checklist

• Success criteria are written down and measurable.
• Cases cover typical, edge, adversarial and refusal behaviour, and contain no personal data.
• Each case has a stable ID, a grader type and tags.
• Judges use a different model and have been checked against human labels.
• The runner exits non-zero below a threshold and runs in CI.
• Every production failure is added as a new case.

References

• Anthropic, Create strong empirical evaluations: https://platform.claude.com/docs/en/test-and-evaluate/develop-tests
• OpenAI, Evaluation best practices: https://developers.openai.com/api/docs/guides/evaluation-best-practices