Confident AI

Garbage in, garbage out applies directly to LLM evaluation. Your evaluation is only as good as the dataset you run it against. A poorly constructed dataset — too small, unrepresentative of production traffic, or focused on easy cases — gives you a false sense of security. You get high scores on your tests while your users experience poor quality in the real world.

Building a good evaluation dataset is not glamorous work, but it is the foundation everything else rests on. This guide covers the principles and practical steps for constructing datasets that accurately reflect your production use case.

The Representativeness Problem

The most common evaluation dataset failure mode is the representativeness gap: the test cases in your dataset do not match the actual distribution of inputs in production. This happens for several reasons:

Hand-crafted datasets skew toward easy cases. When developers construct test cases manually, they naturally think of the canonical inputs — the clear, well-formed queries that the system is obviously designed to handle. Production users send edge cases, ambiguous queries, typo-laden inputs, and requests that straddle the boundaries of what the system is supposed to do. If these are absent from your dataset, your evaluation scores are systematically optimistic.

Public benchmark datasets measure different distributions. Academic benchmarks measure model capabilities on standardized tasks. They tell you something about model quality in general, but not about model quality on your specific task with your specific users. A model that scores 90% on a general Q&A benchmark may score 70% on your application's actual queries.

Early-stage datasets freeze early assumptions. A dataset built before you have production data reflects your assumptions about what users will ask. As you get real production data, you will almost always find that your assumptions were partially wrong. Datasets need to be updated as you learn more about your actual usage patterns.

The Three-Source Framework

Effective evaluation datasets draw from three sources that each provide different types of coverage.

Source 1: Production logs. Once your application is in production, production logs are your most valuable evaluation data source. They contain exactly the inputs your users send, which makes them the most realistic test cases available. A sampling strategy for production data:

• Random sample from all production queries (captures the core distribution)
• Failure-weighted sample from queries that resulted in user complaints, low satisfaction ratings, or explicit negative feedback (captures the cases that matter most to fix)
• Diversity sample using clustering to select cases that cover a wide range of query types, even if some types are rare in production (prevents over-weighting common cases)

For each production query you include in your evaluation dataset, you need a reference answer or quality judgment. This is where annotation comes in — either human annotation of correct outputs, or LLM-assisted annotation where a strong judge model produces candidate reference answers that a human reviews and approves.

Source 2: Adversarial and edge cases. Production logs capture natural user behavior. Adversarial cases represent deliberate attempts to probe the system's limits. Both are necessary. A dataset composed entirely of representative production cases will not catch vulnerabilities that only appear under adversarial conditions. A dataset composed entirely of adversarial cases is disconnected from real usage.

Adversarial cases to include: inputs that test the model's handling of ambiguous or underspecified queries, inputs that probe policy boundaries, inputs at the edge of the system's domain scope, inputs with unusual formatting or encoding, and inputs that a malicious user might craft to extract sensitive information or bypass safety measures.

Source 3: Synthetic generation. Synthetic data generation — using an LLM to create diverse test cases — is useful for filling coverage gaps when production data is limited or when you need to test specific scenarios systematically. The caveat is quality control: LLM-generated test cases need human review, because models will generate test cases that are plausible but may not reflect real user behavior or may be ambiguous in ways that make them hard to annotate correctly.

Annotation Strategy

Most evaluation metrics require reference answers or quality judgments to compare against. Annotation — the process of producing these reference answers — is one of the most time-consuming parts of dataset construction, and it is frequently done poorly.

Who should annotate. Domain experts who understand the correct answer produce more reliable annotations than general-purpose annotators, but they are more expensive and harder to coordinate. For most production applications, a tiered approach works: domain experts annotate the most important and difficult cases, while general annotators with clear guidelines handle high-volume, more straightforward cases.

Annotation guidelines. The quality of annotations depends heavily on the clarity of your guidelines. Annotators need to know: what counts as a correct answer, how to handle cases with multiple acceptable answers, how to handle genuinely ambiguous cases, and the specific scoring rubric for quality dimensions beyond binary correctness. Invest time in annotation guidelines — ambiguous guidelines produce inconsistent annotations that weaken your evaluation.

Inter-annotator agreement. Have multiple annotators label a subset of cases independently, then measure agreement. Low inter-annotator agreement signals that your annotation guidelines are unclear or that the cases themselves are genuinely ambiguous. Both situations require resolution before the annotations are useful for evaluation.

LLM-assisted annotation. Using a strong judge model to produce initial annotation candidates that humans review and approve is significantly faster than annotation from scratch. The key is maintaining human oversight — the judge model produces candidates, human annotators accept, reject, or edit them. This keeps quality high while reducing annotation time by 60-80% compared to manual annotation.

Dataset Size and Coverage

How large does your evaluation dataset need to be? The answer depends on what level of change you need to detect reliably. With a dataset of 100 cases, you can detect changes of roughly 5% or more with reasonable statistical confidence. With 500 cases, you can detect changes of roughly 2-3%. With 2,000 cases, you can detect 1% changes.

For most production applications, a dataset of 200-500 cases is a practical starting point that catches meaningful regressions without requiring enormous annotation investment. Expand from there as you accumulate more production data and your quality requirements become more demanding.

Coverage matters as much as size. A dataset of 500 cases that all test the same query pattern is less useful than 200 cases that cover 20 distinct query types with 10 examples each. Structure your dataset to cover the range of scenarios your users encounter, not just the most common ones.

Dataset Maintenance

Evaluation datasets are not one-time artifacts. They require ongoing maintenance to remain relevant as your application and user base evolve.

Establish a regular dataset review cadence — at minimum quarterly, more frequently if your application is evolving rapidly. In each review:

• Add new cases from recent production logs that cover patterns not well-represented in the current dataset
• Retire cases that no longer reflect real usage patterns or that your system now handles correctly without exception
• Update reference answers for any cases where the expected behavior has changed due to product decisions
• Review and retire cases where annotator confidence was low or where inter-annotator agreement was poor

Maintain a separate "holdout" test set that is never used in development decisions. Your main evaluation dataset will inevitably be partially optimized against — developers will see failing cases and make targeted improvements. The holdout set should only be consulted for major release validation, to give you an unbiased quality signal.

← Back to Blog