Getting Started with LLM Evals: A Small-Team Setup with Code

To start evaluating an LLM feature, write 20 to 50 realistic test cases with expected answers in a JSONL file, run each one through your prompt, grade the outputs with simple code, and record the score. Re-run the same set after every prompt or model change and compare. This article builds that setup in Python with the Claude API, with no framework, so a small team can have a working baseline in an afternoon.

Who this is for and what you will have

This is for developers and product owners who have an LLM feature, such as a classifier, a support-reply drafter or an extractor, and have been checking it by eye. By the end you will have a test set file, a runner script that produces a score and a list of failures, a place to store results, and a routine for growing the set. If you already have a larger regression suite in CI, see Building a Golden Dataset to Regression-Test LLM Features; this guide covers the first step before that.

What an eval is

An eval is a repeatable test for model output. It has three parts: inputs (test cases), a way to produce outputs (your prompt and model call), and a grader that turns each output into a score. Anthropic's documentation on designing evals gives three principles worth adopting from the start:

1. Be task-specific. Mirror your real task distribution and include edge cases such as irrelevant or nonexistent input, overly long input, poor or harmful input, and ambiguous cases where even humans would find it hard to agree.
2. Automate when possible. Structure the task so it can be graded automatically, for example by multiple choice, string match, code, or an LLM grader.
3. Prioritise volume over quality. The documentation says more questions with slightly lower-signal automated grading are better than fewer questions with high-quality human-graded evals.

It also recommends defining success criteria that are specific, measurable, achievable and relevant. "Good performance" is not a criterion; "classifies the ticket into one of five categories correctly" is.

Step 1: Write the success criteria

Write one or two sentences per quality you care about. For a support-ticket classifier:

1. The category label is one of: billing, bug, how-to, account, other.
2. The label matches the label a support lead would choose.
3. The output contains only the label, with no extra text.

Each line should be checkable. Line 3 is a code check; line 2 needs expected answers in the test set.

Step 2: Build the test set

Use a JSONL file, one case per line, so you can append cases without merge conflicts. Save as evals/cases.jsonl:

{"id": "t001", "input": "I was charged twice for my March invoice.", "expected": "billing", "tags": ["core"]}
{"id": "t002", "input": "The export button does nothing when I click it.", "expected": "bug", "tags": ["core"]}
{"id": "t003", "input": "How do I add a second user to my plan?", "expected": "how-to", "tags": ["core"]}
{"id": "t004", "input": "Great, another outage. Just what I needed today.", "expected": "bug", "tags": ["edge", "sarcasm"]}
{"id": "t005", "input": "", "expected": "other", "tags": ["edge", "empty"]}

How to choose cases:

• Take them from real inputs. Use past tickets, emails or queries with personal details removed. Real inputs show you the variety synthetic ones miss.
• Cover the edge cases named above: sarcasm, mixed intent, typos, very long text, empty input, and input in another language if your users send it.
• Add the expected answer yourself. Have the person who knows the domain label them. If two people disagree on a case, that is useful: it tells you the criteria need tightening.
• Tag cases so you can see scores by group.
• Aim for 20 to 50 to start. The number matters less than keeping them realistic and adding to them.

Step 3: Write the runner

Install the SDK and set your key as an environment variable:

pip install anthropic
export ANTHROPIC_API_KEY="your-key-here"

Create evals/run_eval.py:

import json
import sys
import time
from pathlib import Path
 
import anthropic
 
MODEL = "claude-sonnet-5-5"
LABELS = {"billing", "bug", "how-to", "account", "other"}
PROMPT = (
    "Classify this support message into exactly one of: "
    "billing, bug, how-to, account, other. "
    "Output only the label.\n\nMessage: {input}"
)
 
client = anthropic.Anthropic()
 
 
def generate(text: str) -> str:
    message = client.messages.create(
        model=MODEL,
        max_tokens=20,
        messages=[{"role": "user", "content": PROMPT.format(input=text or "(empty)")}],
    )
    return next(b.text for b in message.content if b.type == "text")
 
 
def grade(output: str, case: dict) -> dict:
    cleaned = output.strip().lower()
    return {
        "format_ok": cleaned in LABELS,
        "correct": cleaned == case["expected"].lower(),
    }
 
 
def main(path: str) -> None:
    cases = [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()]
    results = []
    for case in cases:
        output = generate(case["input"])
        results.append({**case, "output": output, **grade(output, case)})
 
    accuracy = sum(r["correct"] for r in results) / len(results)
    format_rate = sum(r["format_ok"] for r in results) / len(results)
    print(f"cases={len(results)} accuracy={accuracy:.1%} format_ok={format_rate:.1%}")
    for r in results:
        if not r["correct"]:
            print(f"FAIL {r['id']}: expected={r['expected']!r} got={r['output']!r}")
 
    out_dir = Path("evals/results")
    out_dir.mkdir(parents=True, exist_ok=True)
    out_file = out_dir / f"run-{int(time.time())}.json"
    out_file.write_text(json.dumps({"model": MODEL, "prompt": PROMPT, "results": results}, indent=2))
 
 
if __name__ == "__main__":
    main(sys.argv[1] if len(sys.argv) > 1 else "evals/cases.jsonl")

Run it:

python evals/run_eval.py evals/cases.jsonl

The output is a one-line summary followed by each failing case. The results file stores the model, the prompt and every output, so you can compare runs later. Note that the model ID is a constant at the top; use the identifier from Anthropic's current model list when you run this.

Step 4: Choose graders

Match the grader to the question.

Anthropic's documentation lists exact match, cosine similarity and ROUGE-L as code-based methods, and Likert-scale and binary-classification prompts as LLM-based methods. Start with exact match or a rule wherever you can. Code graders are free, fast and the same every time.

Adding an LLM grader for tone

Use an LLM grader only for qualities that you cannot check by rule. The documentation advises using a different model from the one that produced the output, where possible. This grader scores a reply from 1 to 5:

def grade_tone(reply: str, tone: str = "polite and clear") -> int:
    prompt = (
        f"Rate this customer service response on a scale of 1-5 for being {tone}.\n"
        f"<response>{reply}</response>\n"
        f"1: Not at all {tone}\n5: Perfectly {tone}\n"
        "Output only the number."
    )
    message = client.messages.create(
        model=GRADER_MODEL,  # ideally different from MODEL
        max_tokens=5,
        messages=[{"role": "user", "content": prompt}],
    )
    text = next(b.text for b in message.content if b.type == "text").strip()
    return int(text)

Define GRADER_MODEL at the top of the file. The int() call will raise an error if the grader adds words; if that happens, tighten the instruction or catch the error and mark the case as ungraded rather than as zero.

Check the grader itself

Before trusting an LLM grader, read 10 to 20 of its scores next to the outputs. If you disagree with it often, change the rubric. A grader you have not checked is another unverified component.

Step 5: Read the failures, not only the score

A score of 84% tells you little. The failing cases tell you what to do:

• A wrong label on a clear case means the prompt needs work.
• A wrong label on an ambiguous case may mean the expected answer is wrong or the criteria are unclear. Fix the test set.
• A format failure such as extra words around the label means the instruction or the output constraint needs tightening, or the grader needs to normalise more.
• A cluster of failures with the same tag points at a category of input to handle explicitly.

Step 6: Use it when you change something

The eval earns its value when you change the prompt, the model or the settings:

1. Run the eval on the current version and keep that results file as the baseline.
2. Make one change.
3. Run again on the same cases.
4. Compare the score and look at the cases that changed from pass to fail and from fail to pass.
5. Keep the change only if the failures you care about improved and nothing important regressed.

Outputs can vary between runs, so differences of one or two cases may be noise. If the variation matters, run each case several times and look at the average.

Step 7: Grow the set

Every real failure is a new test case. When a user reports a wrong output, add it to cases.jsonl with the correct expected answer and a tag such as reported. Over a few weeks the set will reflect what actually goes wrong. Keep a short note of who labelled each batch and when.

Common mistakes

• Testing only easy cases. If every case passes on the first try, the set is not telling you anything.
• Writing the prompt to fit the test set. If you tune the prompt against the same 30 cases until they pass, keep a few cases aside that you do not look at while tuning.
• Grading with the same model that wrote the output and not checking the grader.
• Treating a number as proof. A score from a small set shows direction, not a guarantee of quality in production.
• Putting real personal data in test files. Remove names, numbers and addresses before committing inputs to a repository.

Troubleshooting

Checklist

• Success criteria written as checkable lines.
• 20 to 50 realistic cases in evals/cases.jsonl, including edge cases and tags.
• A runner that saves the model, prompt and every output.
• Code graders first; an LLM grader only where needed, using a different model, and spot-checked.
• Baseline stored before any change.
• Failures from real use added to the set.

References

• Anthropic, Designing evals (test and evaluate): https://platform.claude.com/docs/en/test-and-evaluate/develop-tests