Choosing AI Application Architecture: Six Patterns and Their Trade-offs

Most AI features are not hard to prototype. A developer can call a model API, write a prompt and demonstrate something impressive in a day. The hard part starts when the feature has to answer from your own data, serve many users at a predictable cost, survive provider outages, respect permissions and, eventually, take actions in other systems.

AI application architecture is the set of decisions that determines how models, data, application code, integrations and controls fit together. It extends ordinary software architecture rather than replacing it: you still need clear boundaries, observability and failure handling. What changes is that one component — the model — produces outputs that vary, can be wrong in plausible ways and can be manipulated through its inputs.

This guide compares six common patterns. Instead of describing each in isolation, it follows one product through all six so you can see exactly which requirement justifies each additional piece of architecture, and what that piece costs.

The running example

A business-to-business software company runs an order-management platform. Its support team handles questions from customers about orders, delivery, invoices and refund policies. The product team wants to add AI in stages:

1. Summarise long support tickets for agents.
2. Answer agents’ questions about policies and product documentation.
3. Let several internal teams use AI features without each managing provider keys.
4. Keep costs under control as usage grows.
5. Prepare refund recommendations that a supervisor approves.
6. Investigate unusual order problems that need several lookups.

Each stage maps to one pattern. Treat the example as a thought experiment; the costs and latencies described are relative, not measured benchmarks.

Pattern overview

Pattern 1: Direct model call

Stage in the example: summarise a long support ticket so an agent can read it in 20 seconds.

The application sends the ticket text and an instruction to a model and displays the result. All the information the model needs is in the request, the output is advisory, and a person reads it before acting.

When it is the right choice

• The task is narrow: summarise, rewrite, translate, classify or extract from text you already have.
• A wrong answer is cheap because a person checks it.
• You do not need knowledge outside the request.

Design details that matter

• Structured output. Ask for a defined format (for example JSON with fixed fields) and validate it. Treat invalid output as an error with a retry limit, not as text to display.
• Input as data, not instructions. Ticket text written by customers can contain instructions aimed at the model. OWASP lists prompt injection as the first risk in its Top 10 for LLM applications. Keep system instructions separate, never let input text change what the application does, and do not pass the model’s output into other systems without validation.
• Timeouts and a fallback. If the call fails, show the original ticket. The feature should degrade to “no summary”, never block the agent.

When to move on

Agents start asking the summariser questions it cannot answer, such as “what is our refund window for enterprise customers?”. The answer lives in internal documents, not in the ticket. That is the trigger for retrieval.

Pattern 2: Retrieval-augmented generation (RAG)

Stage in the example: answer agents’ questions using approved policy documents and product documentation.

Before calling the model, the system searches a knowledge store for passages relevant to the question and includes them in the prompt, with an instruction to answer only from those passages and cite them. We compare RAG with fine-tuning in a separate guide, RAG or fine-tuning?; for most knowledge questions, retrieval is the better starting point because documents can be updated without retraining.

What the architecture adds

• An ingestion pipeline that splits documents into passages, creates embeddings and stores them with metadata (source, owner, version, access group, last updated).
• A retrieval step, often combining keyword and vector search, sometimes followed by reranking.
• A prompt that includes the passages and asks for citations.
• A user interface that shows sources so agents can check them.

Where RAG fails, and how to design for it

Permissions deserve special attention. A RAG system must never give a user broader access than they already have in the source system. The safest pattern is to store access metadata with each passage and apply the user’s permissions as a filter during retrieval. Our guide to planning an internal AI knowledge assistant covers permission-aware retrieval and evaluation in more detail.

When to move on

Once the marketing, finance and customer-success teams want their own AI features, each team creating its own provider keys, logging and rate limits becomes a governance problem. That is the trigger for a gateway.

Pattern 3: AI gateway

Stage in the example: several internal applications call models; the company wants one place for keys, policies, logging and quotas.

An AI gateway is a service between applications and model providers. Applications call the gateway with an application identity; the gateway holds provider credentials, applies policies and records every request.

What a gateway is good for

• Credential management. Applications never see provider keys.
• Consistent logging. Every call is recorded with application, workflow, model, token counts, latency and outcome — the basis for the cost attribution described in our AI FinOps guide.
• Quotas and rate limits. Per application and per user, which also mitigates OWASP’s Unbounded Consumption risk (LLM10).
• Policy enforcement. For example, blocking certain data classes from being sent to external providers, or redacting fields before a request leaves your network.
• Provider abstraction. A common request format makes switching or adding providers less disruptive, although prompts and output quality still differ between models.

The cost of a gateway

A gateway is another production service. It adds a network hop, needs high availability and becomes a single point of failure if designed carelessly. Run it with the same discipline as an API gateway: health checks, horizontal scaling, timeouts and a documented behaviour when an upstream provider fails. Decide early whether to build a thin internal proxy, use an open-source gateway or use a managed service; the right choice depends on how much custom policy you need and who will operate it.

When to move on

Usage grows. Monthly spend becomes visible to finance, and analysis shows that many requests are simple and do not need the most capable model. Meanwhile a provider outage blocked every AI feature for an afternoon. That is the trigger for routing.

Pattern 4: Model routing

Stage in the example: send simple requests to a smaller model, complex ones to a larger model, and fail over when a provider is unavailable.

Routing selects a model per request. The decision can be rule-based (request type, input length, customer tier), classifier-based (a small model estimates difficulty) or fallback-only (use model B when model A errors or times out).

Design rules

• Route by measured quality, not assumed difficulty. Before routing a category of requests to a cheaper model, compare outputs on a representative evaluation set. A cheaper model that produces more rejected answers can increase total cost per successful outcome.
• Keep fallbacks compatible. The fallback model must accept the same prompt structure and produce the same output format, or your parsing will fail at the worst moment.
• Log the routing decision. Record which route was chosen and why, so you can analyse quality and cost per route.
• Review routes regularly. Model prices and capabilities change. A rule that made sense six months ago may now waste money or quality.

When routing is premature

If you use one model for one feature, routing adds complexity with no benefit. Start with a fallback route only, and add difficulty-based routing when cost data shows a clear opportunity.

When to move on

The support team now wants AI help with refunds: check the order, check the policy, calculate an amount and prepare a refund for approval. This is no longer “answer a question”. It is a business process that changes records. That is the trigger for a controlled workflow.

Pattern 5: Controlled workflow

Stage in the example: prepare a refund recommendation that a supervisor approves before money moves.

In a controlled workflow, the application defines the steps. AI is used inside specific steps where judgement or language understanding helps, and deterministic code handles everything that must be exact.

Example refund workflow

Why this pattern is often the right end state

Most business processes have known steps. Encoding them in a workflow gives you predictability, testability and clear audit trails. You can measure accuracy per step, add approval where risk is high and remove it where the data shows it is unnecessary. Our guide on when to use AI and when rules are enough explains how to split a process between deterministic logic and AI.

Design details

• Idempotent actions. Each step that changes a record should be safe to retry, using an idempotency key, so a timeout does not create duplicate refunds.
• State management. Store workflow state outside the model. If step 5 fails, resume from step 5, not from the beginning.
• Approval with context. Show the supervisor the extracted values, the policy reference and the calculation, not just the draft message. Approval screens without evidence lead to rubber-stamping.
• Exception queue. Requests that do not fit the workflow go to a person with the information gathered so far.

When to move on

Occasionally an order problem does not fit any defined process: a delivery partner, a warehouse system and an invoice disagree, and an agent spends an hour checking each system. The steps cannot be fixed in advance. That is a possible trigger for an agent — and the point where restraint matters most.

Pattern 6: Agent

Stage in the example: investigate an unusual order discrepancy by looking up several systems and summarising findings for an agent.

An agent is given a goal and a set of tools, and the model decides which tools to call, in what order, until it reaches an answer or a limit. This flexibility is the reason to use an agent and the source of its risks.

Constrain the agent before you give it tools

• Read before write. Start with read-only tools (look up order, shipment, invoice). Add any write tool only after the read-only version has been evaluated in production.
• Least privilege. Each tool gets the narrowest permission that works, scoped to the current user’s access. OWASP’s Excessive Agency risk (LLM06) describes what happens when systems receive more autonomy or permissions than they need.
• Step and cost budgets. Set a maximum number of tool calls and a maximum token budget per run, and stop with a partial summary when they are reached.
• Human confirmation for consequential actions. Any action that changes data, contacts a customer or spends money requires explicit approval, ideally by routing into the controlled workflow from Pattern 5.
• Full trace. Record every tool call, input and output with the run ID. Without traces, agent failures cannot be debugged or audited.

Signs you do not need an agent

• You can list the steps in advance. Use a controlled workflow.
• The task needs one lookup and one answer. Use RAG or a direct call with a tool.
• You cannot describe what a good result looks like. Clarify the requirement before choosing any architecture.

Relative cost, latency and operational load

Each step up the pattern ladder changes the cost and operations profile of the system. The table summarises the direction of change for the running example. It is qualitative: measure your own workload before committing to a design.

Two practical consequences follow. First, RAG and agents are the patterns most likely to surprise you on cost, because tokens per request grow with retrieved context and tool calls; plan budgets and alerts accordingly. Second, the gateway and workflow patterns add operational components rather than tokens, so their cost shows up as engineering and on-call time.

Context and memory: designing what the model sees

Every pattern above eventually runs into the same constraint: the model only knows what is in its context for that call. Context design is therefore an architectural concern, not just prompt writing.

Separate the kinds of context

• Instructions: the stable system prompt describing the task, format and rules. Version it and keep it as short as the task allows.
• Reference knowledge: retrieved passages or documents. Select them per request and include their identifiers so the answer can cite them.
• Conversation state: what the user and assistant have said. Long conversations grow quickly; summarise older turns or keep only the turns relevant to the current question.
• Business state: facts from systems of record, such as the order status or customer tier. Fetch them fresh from the source system with deterministic code rather than relying on the model to remember them from earlier turns.

Set a context budget

Decide how many tokens each kind of context may use for a given feature, for example a fixed allowance for instructions, a cap on retrieved passages and a rolling window for conversation history. A budget keeps cost predictable and forces useful trade-offs: three highly relevant passages often produce better answers than ten loosely related ones.

Keep memory outside the model

If the product needs memory across sessions — user preferences, previous cases, open tasks — store it in your own database with clear ownership, retention and access rules, and retrieve it like any other context. Do not rely on a provider’s conversation features as the system of record for business information.

Security controls in depth

Security for AI applications combines familiar controls with a few new ones. The OWASP Top 10 for LLM applications is a useful checklist; the controls below map to the risks that appear most often in the running example.

The most durable principle is simple: rules that must hold — who can see what, what amount can be refunded, which records can change — are enforced by deterministic code and access controls, never by instructions in a prompt.

Moving between patterns without a rewrite

Architecture should allow the system to grow from one pattern to the next. A few habits make that possible:

• Wrap model calls from the start. Even a direct-call feature should call the model through one internal function or service. That wrapper later becomes the place to add logging, routing or a gateway without touching every feature.
• Keep prompts and retrieval settings as configuration. Store them with versions, separate from application code, so changes can be tested and rolled back independently.
• Introduce new patterns behind a flag. Run the RAG version of a feature for a subset of users, or in shadow mode where answers are generated and evaluated but not shown, before switching everyone over.
• Promote, do not replace. A controlled workflow can reuse the RAG component for its explanation step; an agent can call the same order-lookup tools the workflow uses. Reuse keeps behaviour consistent and testing manageable.
• Retire what you no longer need. If routing saves less than it costs to maintain, remove it. Architecture is not a ratchet that only adds components.

Team and ownership

Patterns imply responsibilities. A RAG system needs someone who owns document quality and freshness; a gateway needs an on-call owner; a workflow needs a business owner who decides where approvals are required; an agent needs someone who reviews traces and adjusts tool permissions. Before choosing a more complex pattern, confirm that these owners exist. A simpler architecture with clear ownership will outperform a sophisticated one that nobody maintains.

Decision matrix

Use this matrix to choose a starting pattern. Answer each question for the feature you are designing and start with the simplest pattern that satisfies all answers.

Trust boundaries

Draw trust boundaries before choosing frameworks. For the example product, four boundaries matter:

1. User and content input → application. Customer emails, uploaded files and retrieved web content are untrusted. They may contain instructions aimed at the model.
2. Application → model provider. Data leaves your network. Decide which data classes may be sent, under which contractual terms and in which region.
3. Model output → application logic. Model output is untrusted until validated. Parse it against a schema, check values against source systems and never execute it directly. OWASP calls the failure here Improper Output Handling (LLM05).
4. Application → systems of record. Only deterministic code with explicit permissions writes to order, payment or customer systems.

For each boundary, write down what crosses it, who is allowed to send it and what validation happens. This short document becomes the basis for security review and for the controls described in frameworks such as the NIST AI Risk Management Framework.

Failure handling by component

A conventional health check tells you a server responds. It does not tell you the answers are correct. Design failure handling for both kinds of failure.

Evaluation built into the architecture

Every pattern needs an evaluation plan, and the architecture should make evaluation easy.

• A fixed evaluation set per feature. Real, anonymised examples with expected outcomes agreed by the business owner. Run it before every prompt, model, retrieval or routing change.
• Component-level metrics. For RAG: whether the right passage was retrieved, whether the answer is supported by it and whether the model correctly abstained when it should. For workflows: accuracy per step. For agents: task completion, steps per run and how often limits were hit.
• Production signals. Acceptance rates, edits, escalations and user feedback, tied to workflow run IDs.
• Versioning. Version prompts, retrieval settings and model identifiers together, so you can tell which change caused a regression and roll back.

Hosting: provider API or self-hosted model

The patterns above work with both provider-hosted and self-hosted models. The choice affects cost structure and operations more than architecture.

Document the decision: an ADR template

Architecture decision records, proposed by Michael Nygard in 2011, are short documents that capture a decision, its context and its consequences. They are especially useful for AI systems because models and prices change quickly and the reasons behind a choice are easily forgotten.

ADR: Use RAG for policy questions in the support assistant

• Status: Accepted.
• Context: Agents ask policy questions that require internal documents updated monthly. Documents have access restrictions by customer tier. We need answers with sources that agents can check.
• Decision: We will use retrieval-augmented generation over the approved policy library, with permission filtering at retrieval time, mandatory citations and a “not found” response when retrieval confidence is low. We will not fine-tune a model for this use case.
• Consequences: We must maintain an ingestion pipeline and version metadata. Answer quality depends on document quality, so policy owners must keep documents current. We will run a 150-question evaluation set before each change. Cost per question increases because prompts include passages.
• Review trigger: Revisit if the evaluation pass rate falls below the agreed threshold for two consecutive releases, or if a new requirement asks the assistant to take actions.

Keep ADRs next to the code. When the next team asks “why does this use a gateway?” or “why is there no agent here?”, the answer is already written down.

Operational readiness checklist

• Each AI feature has an owner, a written success condition and an evaluation set.
• Trust boundaries are documented, with validation at every boundary.
• Every external call has a timeout, a retry limit and a defined fallback.
• Model output is validated against a schema before use.
• Retrieval applies user permissions before ranking.
• Write actions go through deterministic code, idempotency keys and, where consequential, human approval.
• Requests are logged with run ID, model, token counts, route, latency and outcome.
• Prompts, retrieval settings and model versions are versioned together and can be rolled back.
• Quotas and per-run budgets exist for cost and abuse control.
• ADRs record the major decisions and their review triggers.

Frequently asked questions

Do we need an agent framework to build an AI application?

No. Many production systems use a direct call, RAG or a controlled workflow written in ordinary application code. Frameworks can help with agent tool calling and tracing, but choose one after deciding the pattern, not before.

Is RAG always required for business data?

Not always. If the data fits in the request — a single document, a ticket, a form — a direct call is simpler. RAG is the right pattern when answers depend on a larger or changing body of knowledge.

Should we build our own gateway?

A thin internal proxy is often enough for one or two applications. As the number of teams and policies grows, an open-source or managed gateway can save effort. Either way, treat it as critical infrastructure.

How do we keep architecture flexible as models change?

Keep clear boundaries: a gateway or wrapper around model calls, prompts stored and versioned separately from code, and an evaluation set that lets you test a new model in hours rather than weeks.