Evaluating AI output: one good answer is not enough

A strong AI answer shows the system can do the task once. It doesn’t show how often it gets it right, which inputs break it, or whether the tool is worth keeping for that work. Output evaluation treats generation as a sample: several representative inputs, repeated runs, and an acceptance rule set before you start. Keep-or-drop is a per-task call, not a demo verdict.

Why one good answer proves nothing

Language systems aren’t deterministic. The same request can yield answers of different quality. The Nielsen Norman Group puts it this way: a single output is an example, not an evaluation. Decades of classic software trained teams to treat “it passed once” as proof. With AI, that habit doesn’t hold.

The article uses a customer-service example. The same return question can come out correct on one run, skip an exception on another, and promise a refund the customer isn’t owed on a third. Which one represents the system? All three together, and none alone.

What evaluation has to measure

The same NN/g piece compares AI evaluation to a quantitative usability study. Instead of one participant and one task, you watch many people on representative tasks and report an average with a confidence interval. In AI, a “participant” is one run; a “task” is one test input.

Two questions need answers:

How the system does across the range of inputs real people will send.
How consistent it is when the same input comes back.

The first is test-input variability. The second is run-to-run variability. More inputs don’t replace more runs, and the reverse isn’t true either.

How many inputs and how many runs

The article illustrates with 10 representative questions and 5 runs each, 50 outputs in total. Those numbers aren’t a universal recommendation. They show the design: several inputs, repetition per input, and a summary of the distribution.

Before you run, define what counts as acceptable. In the support example, NN/g asks for a correct answer, with conditions and exceptions, aligned to policy, and with a next step when it matters. Failing any item makes the output unacceptable.

At minimum, report the share of acceptable outputs, a confidence interval, variation across questions, and consistency on the same question. Document model, version, instructions, tools, and date. If any piece changes, the evaluation is a new snapshot.

What a final score hides

Two systems can both hit 80% of 50 runs and still have very different problems. NN/g shows the contrast: one gets 8 of 10 questions right on every run and always fails the other 2; the other gets 4 of 5 right on each question.

The first is predictable: you can map the failing themes and hand them to a person. The second is unpredictable: any question may come out wrong, and safe use usually needs a human review before the customer sees it. The average alone hides that difference.

Decide by task: the PROVE framework

Pressure to adopt AI isn’t evidence that a tool helps. NN/g’s PROVE framework tests one tool against one task and ends with a provisional decision you can defend. The letters cover problem alignment, risk, output quality, total velocity, and experience of use.

Start with the recurring task, not the demo. Check that the data you’ll feed it is allowed. Compare the output to a real piece of your own work, not a friendly showcase case. Measure total time: setup, prompt, review, fixes, and formatting. Separate one-time friction from friction that comes back every week.

On the suggested scale, 3 means “about the same as what I already do.” One run on one input is a screen, not a verdict. PROVE doesn’t recommend a tool for the whole organization; it recommends whether you keep it, trial it for a window, or set it aside for that task.

Who checks: output literacy

Using AI often isn’t the same as using it well. NN/g’s study on AI literacy splits two skills: prompt fluency and output literacy. The first communicates intent, constraints, and context. The second judges the result: gaps, hallucinations, sources, and cross-checks when the stakes rise.

The two don’t grow together. Some people prompt fluently and still take polished answers at face value. Some stay skeptical, barely use the tools, and never build prompt skill. People who really check cross-check data, push back, and switch tools by task.

That matters for evaluation. Without someone with output literacy, the round becomes applause for the slickest answer. The analysis of AI chat adoption in UX shows use concentrated in writing, design, and code, and much less in tasks with human interaction or complex analysis. Trying a task with AI doesn’t prove the output passed.

Measured productivity, with caveats

Jakob Nielsen summarizes three studies where average productivity rose 66% with generative AI. Support agents handled 13.8% more inquiries per hour. Business professionals wrote 59% more documents per hour. Programmers shipped 126% more projects per week.

The article itself lists caveats. The data came from earlier versions, in the ChatGPT 3.5 line. Two of the three studies measured a single use, often the person’s first. Gains apply to AI-supported tasks; across the rest of the day, the effect thins out. Cite the 66% with the link and these caveats in the same paragraph. High throughput without output evaluation can still hide quiet errors.

How Vexor applies it

At Vexor, my product for WhatsApp AI assistants, the rule before installing at a client is the same. One good demo answer isn’t an evaluation. The model is chosen per task. Only what survives repeated real questions, in the kinds of conversations the business will get, stays.

Measured cost landed in the cents per conversation, with a monthly ceiling per client, and the math is in the post on AI cost per conversation. That number only makes sense after output quality clears the task’s bar. A cheap model that fails and an expensive one you judged from one shiny demo share the same method error.

Where the sources diverge

Statistics versus decision. The evaluation article covers sampling, confidence intervals, and failure distribution. PROVE covers whether the tool is worth it on your task, with a scale and a provisional call.
Who verifies. Output literacy focuses on day-to-day checking behavior. Formal evaluation focuses on experimental design; they complement each other, but they answer different questions.
Productivity versus quality. The 66% piece measures throughput and, in part, deliverable quality. It doesn’t replace sampling outputs against your product’s policy or flow.
Use versus success. The adoption analysis counts conversations attempted by occupation. It doesn’t report how often those attempts were right.

The common mistake

The classic mistake is judging by the best answer that showed up in the test. The demo picks the easy input, runs once, and sells the highlight. Serious evaluation picks the real range, repeats, looks at where it fails, and only then decides whether the tool stays on that task. If the runs cost too much to repeat, the clear conclusion is that the evidence still isn’t enough.