The benchmark has already won. Your customer may not care.
A field study of 5,179 support agents found that an AI assistant increased issues resolved per hour by 14 percent on average and by 34 percent for novice and lower-skilled workers, while delivering little benefit to the strongest agents. The same model was therefore not one product: its value changed sharply with the person, knowledge gap, and workflow around it.
Two earlier experiments make the pattern harder to dismiss. In bounded professional writing, 453 participants finished 40 percent faster and produced work rated 18 percent better. In a study of 758 consultants, AI improved speed and quality inside its capability frontier but made participants 19 percentage points less likely to reach the right answer on a task outside it. Capability matters; task fit decides whether that capability becomes help or harm.
The customer-support result is a clue, not a universal promise
The large field study reported in Generative AI at Work is useful because it measured work rather than applause. Across 5,179 support agents, access to an AI assistant raised productivity by roughly 14 percent, with larger gains among less-experienced workers. The mechanism was not merely “a smarter model.” The system made patterns from stronger performers available inside a specific workflow, where an agent could use them at the moment of need.
This should change how companies read AI case studies. The transferable lesson is not the effect size. It is the chain: relevant organizational knowledge, presented inside a real task, to a person who can act on it, measured with operational outcomes. Break any link and the same model can become expensive autocomplete. Reproduce the chain, and even an imperfect model may create meaningful value.
Usefulness needs a business definition
Human-centered AI starts from augmenting people and addressing human needs. That definition becomes operational only when a product team asks: useful to whom, at which step, under which constraints, and visible in what behavior? “The answer is better” is not yet a commercial claim.
For a procurement assistant, useful might mean fewer overlooked contractual risks without slowing review. For a consumer shopping assistant, it might mean fewer abandoned sessions and fewer regretful returns. For a physician, it may mean protocol-compliant escalation with uncertainty made visible. One generic “answer quality” score silently collapses these different jobs and can make a better model look like a better business even when nothing important changes.
Find the answer-to-action leak
Map one consequential journey from question to action. After the response appears, where does value leak? Perhaps users reread a dense paragraph, cannot compare alternatives, miss a warning, distrust a correct recommendation, or need to copy the answer into another tool. Those are not secondary UX details. They are candidate causal mechanisms between model output and business outcome.
Humanities and design become practical here. Rhetoric helps distinguish evidence from assertion. Information design makes relationships visible. Psychology helps predict overconfidence, overload, and avoidance. Ethnography reveals the local workarounds absent from product analytics. None makes the model “smarter” in the leaderboard sense. Each can make its existing intelligence more usable by a real person.
The test I would run first
Choose one answer that is already factually adequate and redesign only its human layer. Keep the model, retrieval, and underlying facts fixed. Compare the current response with a version organized around the customer’s decision: a one-sentence recommendation, the evidence, uncertainty, alternatives, and the next safe action. Predefine one behavioral primary metric and one risk guardrail.
If comprehension improves but action does not, the leak sits later in the journey. If action rises together with error, the design is persuasive but unsafe. If neither moves, stop polishing and investigate another mechanism. This test has one job: find out whether this particular design helps this particular customer make this particular decision.