Defining “Correct” in AI Systems

Defining “Correct” in AI Systems

Traditional testing has a clear idea of correctness: an expected value, defined in advance, that the actual result either matches or doesn't. AI systems break this model almost immediately, because a single prompt can have several different acceptable answers, all correct in different ways. Before you can write a single meaningful test case for an AI feature, the team has to agree on what 'correct' actually means for that feature.

This article covers:

1. Why traditional pass/fail falls short
2. Four ways to define correctness
3. Building an evaluation rubric
4. Handling subjectivity and ambiguity

Why Traditional Pass/Fail Falls Short

Ask an AI assistant to 'summarize this article in two sentences' and give the same request to three skilled writers — you will get three different, all reasonably correct, summaries. There is no single expected string to assert against. Trying to force AI output into a strict pass/fail test either produces false failures (the answer was good but didn't match the expected text exactly) or false passes (the test was loosened so much it stopped catching real problems). Neither outcome gives the team a trustworthy signal, which is why AI evaluation needs its own approach rather than reusing traditional assertion-based testing as-is.

Four Ways to Define Correctness

Most AI evaluation strategies combine several of these approaches, rather than relying on just one:

• Exact or fuzzy match — still useful for outputs that genuinely have one correct answer, such as extracting a specific number from a document.
• Semantic similarity — comparing the meaning of the output to a reference answer rather than the exact wording, often using another model to judge how close the two are.
• Human evaluation — a person reviews the output against a defined standard; slower and harder to scale, but often the most reliable for judgment-heavy tasks.
• Rubric-based scoring — breaking 'correct' into specific, checkable criteria (accurate, relevant, appropriately toned, complete) and scoring each one separately instead of a single pass/fail.

Building an Evaluation Rubric

A rubric turns a vague idea of quality into something a tester — or an automated evaluator — can apply consistently. A workable rubric usually defines, for each criterion, what a top score, a middle score, and a failing score actually look like, with a concrete example of each. For a customer support chatbot, a rubric might score every response on accuracy (is the information correct), completeness (does it fully answer the question), tone (is it appropriate for the brand), and safety (does it avoid anything harmful or policy-violating) — each scored independently, since a response can be perfectly accurate and still fail on tone. Involving the whole team — product, QA, and whoever owns the AI system — in defining the rubric matters, because 'correct' is ultimately a product decision, not just a testing one.

Handling Subjectivity and Ambiguity

Some correctness questions never fully resolve into an objective rule, and pretending otherwise creates rubrics nobody actually agrees with in practice. When a criterion is genuinely subjective — tone, helpfulness, creativity — it helps to have more than one reviewer score the same output independently and compare results; large disagreement between reviewers is itself useful information, since it usually means the rubric's definition needs to be sharper, not that the reviewers are wrong. Over time, disagreements should be resolved by refining the rubric with more concrete examples, not by simply picking whichever reviewer's opinion is more convenient.

Further Reading

Defining “Correct” in AI Systems — full case study
Ensuring Quality & Coverage in AI-Generated Questions