# Getting Started with LLM Evals: A Small-Team Setup with Code > Build your first LLM eval in an afternoon: write 30 test cases in a JSONL file, grade them with simple code, add an LLM grader for tone, and track the score between prompt changes. Source: https://www.epifive.com/blog/ai-engineering-llm-evals-small-team-getting-started · Published 2026-10-28 · EpiFive ## Key takeaways - Start with 20 to 50 realistic test cases stored in a JSONL file, including edge cases such as sarcasm, empty input and ambiguous input. - Use code-based graders such as exact match or JSON validity first, and an LLM grader only for qualities that code cannot judge. - Grade with a different model from the one that generated the output where possible. - Re-run the same set after every prompt or model change and compare the score before accepting it. To start evaluating an LLM feature, write 20 to 50 realistic test cases with expected answers in a JSONL file, run each one through your prompt, grade the outputs with simple code, and record the score. Re-run the same set after every prompt or model change and compare. This article builds that setup in Python with the Claude API, with no framework, so a small team can have a working baseline in an afternoon. ## Who this is for and what you will have This is for developers and product owners who have an LLM feature, such as a classifier, a support-reply drafter or an extractor, and have been checking it by eye. By the end you will have a test set file, a runner script that produces a score and a list of failures, a place to store results, and a routine for growing the set. If you already have a larger regression suite in CI, see [Building a Golden Dataset to Regression-Test LLM Features](https://www.epifive.com/blog/llm-golden-dataset-regression-testing); this guide covers the first step before that. ## What an eval is An eval is a repeatable test for model output. It has three parts: inputs (test cases), a way to produce outputs (your prompt and model call), and a grader that turns each output into a score. Anthropic's documentation on designing evals gives three principles worth adopting from the start: 1. **Be task-specific.** Mirror your real task distribution and include edge cases such as irrelevant or nonexistent input, overly long input, poor or harmful input, and ambiguous cases where even humans would find it hard to agree. 2. **Automate when possible.** Structure the task so it can be graded automatically, for example by multiple choice, string match, code, or an LLM grader. 3. **Prioritise volume over quality.** The documentation says more questions with slightly lower-signal automated grading are better than fewer questions with high-quality human-graded evals. It also recommends defining success criteria that are specific, measurable, achievable and relevant. "Good performance" is not a criterion; "classifies the ticket into one of five categories correctly" is. ## Step 1: Write the success criteria Write one or two sentences per quality you care about. For a support-ticket classifier: ```text 1. The category label is one of: billing, bug, how-to, account, other. 2. The label matches the label a support lead would choose. 3. The output contains only the label, with no extra text. ``` Each line should be checkable. Line 3 is a code check; line 2 needs expected answers in the test set. ## Step 2: Build the test set Use a JSONL file, one case per line, so you can append cases without merge conflicts. Save as `evals/cases.jsonl`: ```json {"id": "t001", "input": "I was charged twice for my March invoice.", "expected": "billing", "tags": ["core"]} {"id": "t002", "input": "The export button does nothing when I click it.", "expected": "bug", "tags": ["core"]} {"id": "t003", "input": "How do I add a second user to my plan?", "expected": "how-to", "tags": ["core"]} {"id": "t004", "input": "Great, another outage. Just what I needed today.", "expected": "bug", "tags": ["edge", "sarcasm"]} {"id": "t005", "input": "", "expected": "other", "tags": ["edge", "empty"]} ``` How to choose cases: - **Take them from real inputs.** Use past tickets, emails or queries with personal details removed. Real inputs show you the variety synthetic ones miss. - **Cover the edge cases** named above: sarcasm, mixed intent, typos, very long text, empty input, and input in another language if your users send it. - **Add the expected answer yourself.** Have the person who knows the domain label them. If two people disagree on a case, that is useful: it tells you the criteria need tightening. - **Tag cases** so you can see scores by group. - **Aim for 20 to 50** to start. The number matters less than keeping them realistic and adding to them. ## Step 3: Write the runner Install the SDK and set your key as an environment variable: ```bash pip install anthropic export ANTHROPIC_API_KEY="your-key-here" ``` Create `evals/run_eval.py`: ```python import json import sys import time from pathlib import Path import anthropic MODEL = "claude-sonnet-5-5" LABELS = {"billing", "bug", "how-to", "account", "other"} PROMPT = ( "Classify this support message into exactly one of: " "billing, bug, how-to, account, other. " "Output only the label.\n\nMessage: {input}" ) client = anthropic.Anthropic() def generate(text: str) -> str: message = client.messages.create( model=MODEL, max_tokens=20, messages=[{"role": "user", "content": PROMPT.format(input=text or "(empty)")}], ) return next(b.text for b in message.content if b.type == "text") def grade(output: str, case: dict) -> dict: cleaned = output.strip().lower() return { "format_ok": cleaned in LABELS, "correct": cleaned == case["expected"].lower(), } def main(path: str) -> None: cases = [json.loads(line) for line in Path(path).read_text().splitlines() if line.strip()] results = [] for case in cases: output = generate(case["input"]) results.append({**case, "output": output, **grade(output, case)}) accuracy = sum(r["correct"] for r in results) / len(results) format_rate = sum(r["format_ok"] for r in results) / len(results) print(f"cases={len(results)} accuracy={accuracy:.1%} format_ok={format_rate:.1%}") for r in results: if not r["correct"]: print(f"FAIL {r['id']}: expected={r['expected']!r} got={r['output']!r}") out_dir = Path("evals/results") out_dir.mkdir(parents=True, exist_ok=True) out_file = out_dir / f"run-{int(time.time())}.json" out_file.write_text(json.dumps({"model": MODEL, "prompt": PROMPT, "results": results}, indent=2)) if __name__ == "__main__": main(sys.argv[1] if len(sys.argv) > 1 else "evals/cases.jsonl") ``` Run it: ```bash python evals/run_eval.py evals/cases.jsonl ``` The output is a one-line summary followed by each failing case. The results file stores the model, the prompt and every output, so you can compare runs later. Note that the model ID is a constant at the top; use the identifier from Anthropic's current model list when you run this. ## Step 4: Choose graders Match the grader to the question. | Question | Grader | Example | | --- | --- | --- | | Is the label right? | Exact match after normalising case and whitespace | The code above | | Is the output valid JSON with required keys? | Parse with `json.loads` and check keys | Extraction tasks | | Does the answer contain a required fact? | Substring or regular expression | "Mentions the refund window" | | Is the tone right? | LLM grader | Customer replies | Anthropic's documentation lists exact match, cosine similarity and ROUGE-L as code-based methods, and Likert-scale and binary-classification prompts as LLM-based methods. Start with exact match or a rule wherever you can. Code graders are free, fast and the same every time. ### Adding an LLM grader for tone Use an LLM grader only for qualities that you cannot check by rule. The documentation advises using a different model from the one that produced the output, where possible. This grader scores a reply from 1 to 5: ```python def grade_tone(reply: str, tone: str = "polite and clear") -> int: prompt = ( f"Rate this customer service response on a scale of 1-5 for being {tone}.\n" f"{reply}\n" f"1: Not at all {tone}\n5: Perfectly {tone}\n" "Output only the number." ) message = client.messages.create( model=GRADER_MODEL, # ideally different from MODEL max_tokens=5, messages=[{"role": "user", "content": prompt}], ) text = next(b.text for b in message.content if b.type == "text").strip() return int(text) ``` Define `GRADER_MODEL` at the top of the file. The `int()` call will raise an error if the grader adds words; if that happens, tighten the instruction or catch the error and mark the case as ungraded rather than as zero. ### Check the grader itself Before trusting an LLM grader, read 10 to 20 of its scores next to the outputs. If you disagree with it often, change the rubric. A grader you have not checked is another unverified component. ## Step 5: Read the failures, not only the score A score of 84% tells you little. The failing cases tell you what to do: - A **wrong label** on a clear case means the prompt needs work. - A **wrong label** on an ambiguous case may mean the expected answer is wrong or the criteria are unclear. Fix the test set. - A **format failure** such as extra words around the label means the instruction or the output constraint needs tightening, or the grader needs to normalise more. - A cluster of failures with the same tag points at a category of input to handle explicitly. ## Step 6: Use it when you change something The eval earns its value when you change the prompt, the model or the settings: 1. Run the eval on the current version and keep that results file as the baseline. 2. Make one change. 3. Run again on the same cases. 4. Compare the score and look at the cases that changed from pass to fail and from fail to pass. 5. Keep the change only if the failures you care about improved and nothing important regressed. Outputs can vary between runs, so differences of one or two cases may be noise. If the variation matters, run each case several times and look at the average. ## Step 7: Grow the set Every real failure is a new test case. When a user reports a wrong output, add it to `cases.jsonl` with the correct expected answer and a tag such as `reported`. Over a few weeks the set will reflect what actually goes wrong. Keep a short note of who labelled each batch and when. ## Common mistakes - **Testing only easy cases.** If every case passes on the first try, the set is not telling you anything. - **Writing the prompt to fit the test set.** If you tune the prompt against the same 30 cases until they pass, keep a few cases aside that you do not look at while tuning. - **Grading with the same model that wrote the output** and not checking the grader. - **Treating a number as proof.** A score from a small set shows direction, not a guarantee of quality in production. - **Putting real personal data in test files.** Remove names, numbers and addresses before committing inputs to a repository. ## Troubleshooting | Symptom | Likely cause and fix | | --- | --- | | `anthropic.AuthenticationError` | The `ANTHROPIC_API_KEY` variable is missing or wrong in this shell | | `anthropic.RateLimitError` | Too many requests too fast; add a short pause or retry with backoff. See [LLM API Rate Limits: Retry and Backoff in Python](https://www.epifive.com/blog/llm-api-rate-limits-retry-backoff-python) | | Every case fails on format | The output has extra text; tighten the instruction or inspect one raw output | | `ValueError` in `grade_tone` | The grader returned words, not a number; tighten its instruction | | Scores jump between runs | Run each case several times and compare averages | ## Checklist - Success criteria written as checkable lines. - 20 to 50 realistic cases in `evals/cases.jsonl`, including edge cases and tags. - A runner that saves the model, prompt and every output. - Code graders first; an LLM grader only where needed, using a different model, and spot-checked. - Baseline stored before any change. - Failures from real use added to the set. ## References - Anthropic, Designing evals (test and evaluate): https://platform.claude.com/docs/en/test-and-evaluate/develop-tests ## Frequently asked questions ### How many test cases do I need to start with LLM evals? Start with about 20 to 50 cases that mirror real inputs, including edge cases. Anthropic's documentation says more automatically graded questions beat fewer hand-graded ones, so add cases over time rather than perfecting a small set. ### Should I use an LLM to grade the outputs? Use code-based grading wherever the answer can be checked by rule, such as exact match or valid JSON. Use an LLM grader for qualities that code cannot judge, such as tone, and prefer a different model from the one that generated the output. ### Why does my eval score change between runs on the same prompt? Model outputs can vary between runs. Run each case more than once if the variation matters, compare scores before and after a change on the same cases, and treat very small differences as noise. ### Do I need an evaluation framework to start? No. A JSONL file, a short Python script and a results file are enough to start. Move to a framework when you need dashboards, many graders or team workflows.