A seven is not a theory
Imagine an answer rated 7 out of 7 that produces exactly the same completion, conversion, error, and support demand as the old one. The score improved; the product did not. Worse, the answer may be clearer but less complete, more persuasive but less calibrated, or more pleasant while increasing the wrong action.
LLM judges intensify the temptation because they make evaluation cheap and scalable. They are useful diagnostic instruments when the construct is specific and validity is checked. They do not convert a vague rubric into a meaningful product outcome. “Overall quality” remains a bundle of unexamined values no matter how many samples are scored.
Measurement starts before the metric
Wallach and colleagues distinguish a background concept from a systematized concept, the measurement instrument, and the resulting instance-level measurements. In plain language: decide what usefulness means here, make that definition explicit, choose observations that represent it, and examine whether the observations justify the claim.
A useful renewal answer might help an account owner identify the right plan, understand the cost change, and act without avoidable support, while not increasing later regret. That definition leads to several measures. It also reveals conflicts. Faster decisions are not automatically better decisions; fewer calls are not a success if confused customers simply give up.
Build a chain from answer to business
Answer diagnostics sit at the beginning: factual correctness, protocol completeness, clarity, evidence use, and latency. Human intermediate outcomes follow: comprehension, calibrated confidence, cognitive effort, perceived relevance, and trust. Behavior comes next: selection, trial, escalation, completion, adoption. Business outcomes arrive later: revenue, cost, retention, risk, and customer lifetime value.
The chain is a causal hypothesis, not an accounting identity. A clearer answer may improve comprehension without changing conversion because price dominates. A persuasive answer may increase trial without retention because product value is absent. Measuring several links lets the team locate the bottleneck instead of declaring the entire intervention a success or failure.
Null results need a decision
A/B testing is valuable precisely because plausible design ideas can win, do nothing, or make the outcome worse. A blog full of wins teaches teams to retrofit stories around noise. A research program should state the minimum effect worth acting on and what will happen when the threshold is not met.
Predefine four paths: scale, iterate, stop, or investigate harm. A statistically detectable improvement can still be commercially irrelevant. A null average can conceal a valuable segment or a damaged one. A negative result can save an organization from rolling out expensive answer complexity. Evidence creates value by changing the decision, not by staying positive.
Start with the decision the evidence must change
Take one proposed answer improvement and write its measurement model before implementation. Name the customer behavior, causal mechanism, primary metric, risk guardrail, minimum meaningful effect, target population, and decision owner. Add diagnostics only when they help explain the primary result. Register the analysis rules internally before looking at data.
Then run the smallest credible controlled test. If a team cannot say what action follows each result, it is not ready to experiment. If it cannot measure the consequential behavior, call the study exploratory rather than pretending a proxy is the outcome. “Did the answer get better?” becomes a useful question only after “better for whom, for what decision, and observable how?”