AI applications are non-deterministic. This is not a limitation but a consequence of how the model performs inference. The question that follows: can the system around the model manage the non-determinism and contain the cost of errors it can produce?
Like any architecture decision, AI application design is a trade-off — between the autonomy the system grants the model and the cost of errors in its domain. The next section revisits common agentic terms; the rest of the post works through the trade-off and the knobs that move a system along it.
Definitions
Agent. A system that uses an AI model (typically an LLM) to make decisions and take actions, not just produce output. An agent has goals, can invoke tools or trigger workflows, and operates over multiple steps.
Tools. The set of capabilities an agent can invoke — APIs it can call, code it can run, data it can read or write, and integrations it has access to. Tools are the discrete primitives ("send an email," "query the database").
Skills. Higher-level capabilities composed as LLM prompt instructions ("triage this support ticket"). Skills and Tools together define the agent's action space: what it can do in the world.
Context. Everything inside the model's context window at invocation — system prompts, user inputs, tool outputs, retrieved documents, and conversation history. Context is the information available to the model when it produces an output: the most direct lever on output quality, and the most common source of poor outputs ("the model was not given the thing it needed to know").
Autonomy. The degree to which the system chooses its own next step rather than following code-defined paths. High autonomy means the model decides what to do and which tools to call; low autonomy means code decides, and the model fills in narrow gaps. The knobs in this post adjust autonomy.
Cost of error. The damage when the system is wrong — monetary, safety, reputational, or regulatory. Some applications absorb errors cheaply (a bad brainstorm, a re-rolled image); others pay heavily (a wrong medical recommendation, an unauthorized refund, a bad merge to main). Cost of error is a property of the domain and the workflow around the model, not of the model itself.
The trade-off
With those terms set: autonomy and exposure to error are linked. More autonomy — broader action space, more open-ended generation, more model-chosen steps — creates more opportunities for the system to be wrong, and those opportunities cost more in domains where errors are expensive. Tighter guardrails, deterministic steps, and narrower scope reduce that exposure, but also limit what the model can do.
This is a frontier, not a fixed line. Engineering moves the frontier outward — good orchestration, grounding, structured outputs, and verification let a system grant more autonomy at the same error cost, or hold the same autonomy at a lower one. Building a useful AI application means picking a position on that frontier and engineering to it.
The two axes
Plot two axes:
- Autonomy need — how much the use case requires the model to choose its own next step: which tools to call, which paths to take, when it is done.
- Cost of error — what it costs the user, the business, or the world when the system gets it wrong.
Four quadrants fall out, and most AI applications sit firmly in one of them — though the same use case can move between quadrants as more of its workflow is automated or its outputs reach more consequential downstream systems.
Creative & exploratory — high autonomy, low cost of error
Models earned their early reputation here: brainstorming, image and video generation, code scaffolding, research assistants, ideation tools. The model gets room to wander because its output is intermediate — a human reviews, edits, picks, or rerolls before anything ships.
The cost of a wrong answer is small; the cost of a boring answer is usually the bigger problem.
Design implication: lean into the model. Higher temperature, longer context, fewer guardrails. Optimize for surface area and surprise.
Mission-critical agentic — high autonomy, high cost of error
The hardest quadrant. Agents that act on production systems: refund issuance, code merges, incident response, autonomous trading, customer-impacting workflows. The use case demands both axes — the value of the agent comes from autonomy, but a wrong action carries real monetary, safety, or regulatory exposure.
Most of the engineering effort goes into containing error cost without smothering the autonomy the use case actually requires.
Design implication: the model is a component, not the system. Surround it with the full kit — orchestration, grounding, structured outputs, evaluator models, business-logic gates on every consequential action, and human review at well-chosen checkpoints. Treat every tool as a potential blast radius.
Structured & assistive — low autonomy, low cost of error
Two related flavors sit here. Structured automation: form filling, document extraction, classification, ETL transforms, deterministic workflow steps — narrow correct answers, fuzzy inputs. Casual assistive: conversational interfaces for non-critical domains, gaming NPCs, low-stakes chat. The model is not making real decisions; errors are cheap.
For the structured-automation flavor, the question to ask first is: does this use case need an LLM at all? When inputs are messy enough that rules miss edge cases, a model helps. But this is where teams most often over-apply LLMs to problems a regex would solve more cheaply and reliably.
For the casual flavor, the interesting design problems are usually personality, latency, and cost. On-device LLMs are good candidates here.
Design implication: decide whether a model is needed at all; when it is, constrain it heavily — JSON schemas, low temperature, narrow prompts, validators. Treat the model as a fuzzy parser, not a source of judgment.
High-stakes advisory — low autonomy, high cost of error
Medical diagnosis support, legal research, financial advisory, security incident analysis, regulatory review. The model produces reasoning that a human acts on — it does not take the action itself. Errors are expensive because the human downstream often trusts the output more than the system has earned.
Autonomy is bounded by design: a clinician, a lawyer, an analyst is in the loop. But the value comes from reasoning the human cannot do as quickly or as broadly, and the dominant failure mode is confident-but-wrong outputs absorbed without scrutiny.
Design implication: ground every claim. Retrieval against authoritative sources, citations the human can verify, structured outputs that surface uncertainty and dissent. Design the workflow so the human reviews structured artifacts — ranked options, sourced claims, flagged uncertainties — not raw model prose. Cheap verifiers run on every output; expensive human review goes at the right checkpoint.
When to build an AI agent vs. a traditional app
The structured-automation flavor raises the more general question of when to reach for AI at all. The default for any new feature should be a traditional application: deterministic code, well-understood inputs and outputs, predictable performance, and debuggable behavior. AI earns its place when the problem genuinely demands judgment, generation, or reasoning over fuzzy inputs.
A few heuristics that have held up:
Reach for a traditional app when:
- The input space is structured and bounded. A simple form with five fields and known validation rules does not need an LLM.
- The rules are knowable and stable. If you can write down the logic, write down the logic. Code is cheaper to run, faster, easier to debug, and more reliable than a model call.
- Determinism is a hard requirement. Tax calculations, payment processing, access control, and anything regulatory.
- Performance matters. A 200ms p99 budget is hard to hit when a model call takes a second or more.
Reach for an AI agent (or AI components) when:
- The input is messy. Free-form text, voice, images, mixed-format documents, and ambiguous user intent. The kind of data processing that a regex can't reliably parse and a schema can't fully describe.
- The edge cases are unbounded. New ones keep appearing, and the rules engine grows without converging. At that point, the cost of encoding more rules starts to exceed the cost of a model.
- Synthesis or judgment is the value. Summarization, nuanced classification, drafting, comparison, and recommendation in a fuzzy domain — tasks where the answer is not lookup-able.
- The task is cheaper to describe than to implement. A model that reads three pages of policy and decides accordingly can beat the engineering cost of encoding all those rules — for a while.
- A reasonable human could do this task by reading inputs and producing outputs, and you would happily replicate that human's quality at higher throughput.
The honest answer is usually a hybrid. Most production AI systems are not pure agents — they are traditional applications with AI components at the specific decision points where the messy-input or open-ended-judgment problem lives. The login flow is code. The form validation is code. The "extract the customer's intent from this support email" is a model call. The "decide whether to issue a refund" is business logic (knob #2) wrapped around a model recommendation — not a model decision.
A useful test: reach for AI only when the rules genuinely do not compress — when the cheapest honest description of the problem is "look at this and tell me what it is," not when nobody has sat down yet to enumerate the cases.
Where this gets interesting
The framework is most useful when an application moves between quadrants over its lifetime, or when different parts of the same application live in different quadrants.
A coding assistant in Creative & exploratory (suggesting code for review) becomes a Mission-critical agentic system the moment it merges to main without a human in the loop. A casual chatbot in Structured & assistive crosses into Mission-critical agentic the moment it can issue a refund. The error cost shifts not because the model changed, but because the autonomy did.
When sketching an AI application, ask the question explicitly: which quadrant is this in, and which quadrant should it be in? The gap between those two answers is the engineering work.
The knobs
Once the target quadrant is set, the design problem is: how to get there. Because more autonomy brings more exposure to error cost, dialing in the trade-off comes down to deciding how much determinism to wrap around the model — and where. Models do not have a "be more deterministic" dial, but the system around them does. The knobs below run roughly from coarsest to finest.
1. Amount of orchestration
The single most powerful lever is not letting the model choose the next step when a deterministic workflow would do. The amount of orchestration wrapped around the model determines how much of its behavior code fixes and how much the model infers at runtime.
Replace open-ended agent loops with state machines, directed graphs, or fixed pipelines wherever the steps are knowable. A workflow of "extract entities → look them up → compose a response" is safer than one of "figure out what to do." The system gives up some flexibility and gains behavior that can be reasoned about.
This also covers multi-step decomposition: breaking a complex task into smaller, individually verifiable steps — generate a plan, critique it, execute step one, validate, execute step two. Each step has a smaller scope, a tighter prompt, and a clear pass/fail criterion. When something breaks, the failing step is identifiable.
The pattern: autonomy at the leaves, determinism in the spine. Invoke the model only for the parts that genuinely require it; everything around it is plain code.
2. Business logic
Around the model sits the domain logic of the application — the rules that encode what the business is willing to allow, regardless of what the model produces.
A refund agent's model might propose a $5,000 refund. The business logic enforces "refunds over $500 require manager approval, period." A code-modifying agent might propose merging to main; the business logic enforces "no merges without a passing CI." These rules belong in code, not prompts.
Prompts are ignorable, jailbreakable, or lost under unusual context. Code-enforced policy gates are the difference between an agent that suggests compliant actions and one that can only take compliant actions. Compliance, regulatory, and safety constraints belong here — encoded, not asked-about.
3. Agent context
What enters the model's context window is the most direct lever on what comes out. Three sub-knobs:
- Grounding via retrieval (RAG, search tools, knowledge bases). A model citing real documents hallucinates less than one reasoning from training data alone.
- Scoping. Strip the context to what the current step needs. Long, irrelevant context is a hallucination accelerant, not a precaution.
- Instruction hierarchy. System prompts, user prompts, and tool outputs — be explicit about precedence when they conflict. This is also where prompt-injection defense lives.
Most "the model got it wrong" failures, on inspection, are really "the model was given the wrong context."
4. Agent skills and tools
The set of skills and tools granted to an agent is its action space. The blast radius of a bad inference equals the most destructive tool the agent has access to.
Apply least privilege: read before write, sandboxed before global, dry-run before execute, and human-confirmed before irreversible. The question to ask of every tool: if the model called this with the worst plausible arguments, what would happen? When the honest answer is "a bad day," the tool needs a guardrail in front of it, not just a good prompt.
Tool design also affects inference quality. A well-named tool with a clear schema and a focused purpose helps the model use it correctly. Five overlapping tools — or one God-tool with twenty parameters — invite misuse. Skill design is part of the safety surface, not just a UX concern.
5. Output structuring
Free-form text is the highest-variance output a model can produce. Constraining the output collapses the space of possible responses and makes downstream code reliable. The mechanisms differ in what they guarantee:
- JSON schemas and tool/function calling — the model is asked to follow a schema; the framework validates and may retry on miss. Cheap and well-supported, but the model can still produce a structurally valid response that is semantically wrong.
- Constrained decoding — the model is prevented from emitting tokens outside the schema. Validity is enforced; semantic correctness is not.
- Validators and grammars after generation — regex, JSON parsers, business-rule checks. Easy to layer with the above as a second line of defense.
A schema validates shape, not correctness. A JSON-valid refund recommendation can still be the wrong refund. Combine schemas with verification (knob #7) wherever the cost of being wrong is more than cosmetic.
Wherever errors are expensive — or where outputs feed downstream code — structured outputs are usually non-negotiable. Treat the schema as the contract between the model and the rest of the system.
6. Model parameters
The classic dials: temperature, top-p, top-k, max tokens, and stop sequences. Lower temperature for deterministic tasks; higher for creative. Do not leave these at defaults without a reason — defaults are tuned for general chat, not for a specific application.
This is the finest knob, not the most powerful one. A perfectly tuned temperature does not save a system whose orchestration is wrong.
7. Verification and evaluation
Two layers:
- Inline verification — judge models, validators, and sanity checks on outputs before downstream code acts on them. A second model checking the first one's work catches a meaningful fraction of errors, with a caveat: judges share failure modes with the models they judge, especially when the same family or training data is involved. A judge is most useful when it brings a different lens — a validator, a tool that runs the output, or a model from a different family — and least useful when it is the same model with a different prompt.
- Offline evaluation — eval suites, regression tests, and golden datasets. The same discipline that applies to any system whose behavior matters: when something breaks, the suite proves it broke and signals when it is fixed.
Evals become load-bearing on model upgrades. Prompts, tool definitions, and orchestration tuned for one model often regress on the next — sometimes silently, because the new model is better on average but worse on the specific cases the application depends on. An eval suite that runs on every upgrade is what turns "pick the latest model" from a leap of faith into a measurable change.
A note on latency
Every knob that improves output quality has a latency cost. Verification chains add a round trip; multi-step decomposition adds several; grounding via retrieval adds the search; judge models double the per-output time. The right configuration depends on what the application can absorb. A real-time UX with a sub-second budget cannot afford the same verification stack as an overnight batch workflow.
Within a quadrant, the choice is usually not whether to add a defense but which one fits the latency budget: cheap and parallelizable (schemas, validators, fast judges) before slow and sequential (full re-generation, multi-stage review, human checkpoints).
Capabilities for managing non-determinism
The knobs above are design-time choices, set when the system is built. Even a well-tuned agentic system retains residual non-determinism — the same prompt produces different outputs, the same tool call hits different state, and the same workflow takes different paths. Running agents responsibly in production requires a set of operational capabilities that match what production already demands of any other system: telemetry, tests, verification, baselines, and budgets.
Agent audit
Capture every step the agent takes — every prompt, tool call, output, and decision branch — with enough fidelity to replay or reason about the run after the fact. The audit log is to an agent what tracing is to a microservice.
When something goes wrong in production, the first question is what did the agent do? Without an audit trail, the answer is a guess. The audit log is also the substrate the rest of this section depends on: evals replay it, verification checks it, and efficacy reports aggregate over it.
Agent evaluation
Repeatable test cases that exercise the agent end-to-end against known inputs and expected outcomes. Eval suites run in CI, on every prompt change, and on every model upgrade. They provide evidence that the system got better or worse — instead of the vibes of the engineer who pushed the change.
A non-trivial agentic system without an eval suite is, functionally, untested code shipping to production. Treat evals as a first-class artifact: version them, review them, and grow the suite every time a new failure mode appears.
Agent verification
The ability to confirm that the agent's output is correct — not just plausible. For an agent generating SQL, that means running the query against a sandbox and checking the result. For one writing code, the tests pass. For one drafting a contract, a structured review against required clauses.
Verification closes the loop between "the model produced something" and "the output is trustworthy enough to act on." Where the verifier is cheap and automated, run it on every output. Where verification requires a human, route to one — and design the workflow so the human reviews structured artifacts, not raw transcripts.
Agent efficacy
A comparison against a baseline: human work, the previous version of the agent, or a simpler alternative. Without efficacy measurement, the question that funds the project goes unanswered: is this agent better than the previous approach?
Efficacy also catches drift. An agent that was net-positive at launch can degrade because inputs shift, models update, or downstream systems change. The metric tracked at launch — accuracy, throughput, escalation rate, or customer outcome — is the same metric that signals when the agent has quietly stopped delivering net value.
Agent budget
Track LLM token usage, tool-call counts, latency, and dollar cost — and enforce limits. An agent that loops in a degenerate way can burn a quarter's API budget in an afternoon. A per-run cap, a per-tenant quota, and a circuit breaker on cost are basic hygiene, not premature optimization.
Budget tracking doubles as a leading indicator of quality issues: a sudden cost spike usually means the agent started retrying, looping, or thrashing. The cost graph often catches problems before the eval suite does.
A short checklist
For any AI application under design or review:
- Place the application on the grid. Which quadrant is it in today? Which quadrant does the use case demand?
- Identify the autonomy boundary. Where does the model choose, and where does code choose? Push the boundary toward code where possible.
- Encode business logic in code, not prompts. Compliance and policy rules belong in gates, not instructions.
- Audit the context. Does the agent get what it needs and nothing else? Is grounding in place where claims need to be verifiable?
- Map the skills and tools. What can this agent do, and what is the worst case when it picks the wrong one?
- Constrain the output. Where is structured output appropriate, and is it enforced?
- Tune the parameters deliberately. Temperature is not 0.7 by accident; document the choice.
- Wire the operational capabilities. Audit, evals, verification, efficacy baselines, and budget caps — each one before scale, not after the first incident.
Closing thought
The trade-off does not go away. No model release eliminates it, no prompt technique resolves it, and no agent framework hides it. What changes over time is the position of the frontier — what is achievable for a given level of error cost — and good engineering is what keeps a system on that frontier rather than well behind it.
Treat autonomy and cost of error as a single design dimension, not two separate concerns to balance after the fact. Pick a position on it, and use the knobs to get there.