Prompts are configuration; operations create reliability
Karaca reports that its AI shopping assistant doubled conversion versus search, reached five times the conversion rate of unaided sessions, and reduced session cost by 97.5 percent before launch. The public case also describes extensive guardrails, A/B testing, data work, and leadership involvement. The business result belongs to that operating system, not to one clever prompt.
Answer Operations treats the answer as a product system. It owns what the system may claim, how evidence enters, which protocols govern action, what forms the answer may take, how customers can correct context, when humans intervene, and how changes are measured. The name matters less than the accountability it creates.
Four versioned layers
The evidence layer contains sources, provenance, freshness, and access. The protocol layer contains conditions, obligations, prohibited transitions, required disclosures, and escalation. The presentation layer contains language, hierarchy, interaction, accessibility, tone, and the representation selected for a task. The measurement layer defines constructs, instruments, thresholds, guardrails, and decisions.
Version these layers separately. A policy update should not require retuning brand voice. A new model should be tested against the same protocol suite. A redesigned comparison card should use the same evidence. Separation makes failures legible and lets teams improve the human layer without confusing a visual effect with a knowledge change.
The team is necessarily interdisciplinary
Product owns the customer and business decision. Domain experts author rules and boundary cases. Engineering implements retrieval, generation, validation, observability, and rollback. Design owns task representation and interaction. Humanities and social science clarify rhetoric, culture, constructs, and lived context. Risk, legal, security, and frontline operations own the consequences that a lab test can miss.
Think of this as a division of control, not a committee reviewing every sentence. Each discipline owns artifacts that enter the release process: source policies, protocol tests, component constraints, qualitative findings, measurement plans, and escalation playbooks. Cross-functional work becomes scalable when every insight has an address in the system.
Tie releases to decisions
Wallach and colleagues’ measurement framework helps teams avoid declaring vague improvement. Before a release, state what human behavior should change, why the answer intervention might change it, the primary outcome, minimum meaningful effect, risk guardrail, and decision for each result. Use LLM evaluations for diagnostics where validated, not as a substitute for customer behavior.
Karaca’s reported shopping-assistant case is instructive because the public account includes A/B testing, guardrails, conversion comparisons, and session-cost reduction. The numbers remain externally reported and context-specific. The operational lesson is transferable: value emerged through product experimentation and economics, not through a model demo alone.
A 90-day Answer Operations start
Days 1–30: choose one consequential answer class, map its journey and failure modes, collect customer and frontline evidence, and write the measurement contract. Days 31–60: separate evidence, protocol, and presentation; build a matched target-and-contrast evaluation set; prototype one answer intervention. Days 61–90: run offline safety tests, then a controlled customer or employee study with rollback and monitoring.
End with a decision, not a showcase. Scale if the intervention clears the threshold without unacceptable harm. Iterate if mechanism diagnostics are promising but the primary outcome is not. Stop if value is too small. Investigate if a subgroup is harmed. The mature LLM company will not be the one with the longest prompt. It will be the one that can explain, test, and improve how intelligence becomes an answer a human can safely use.