Stateful or Stateless AI Agents? A Product Architecture Decision

“Should our agent be stateful or stateless?” sounds like a simple architecture question. In practice it hides several separate decisions: what the agent needs to see during one request, what must survive while a task is in progress, what — if anything — it should remember about a user next week, and who is allowed to see each of those things.

This guide untangles those decisions. If you are still deciding whether you need an agent at all, start with our guide to what AI agents are.

The model is always stateless

A language model call takes input and returns output. It does not retain anything afterwards. When a chatbot appears to remember earlier messages, your application — or the provider’s conversation feature — is storing those messages and sending them back with each new request.

That has two consequences. First, “stateful agent” really means “agent whose surrounding system stores and reuses state”. Second, every piece of state you send back costs tokens and latency, and every piece you store is data you must secure, retain correctly and be able to delete.

Four kinds of agent state

Shared knowledge is usually provided through retrieval rather than memory, and it is not personal. The other three are where the stateful-versus-stateless decision really lies.

When stateless is enough

A stateless agent receives everything it needs in the request and keeps nothing afterwards. It suits:

• Single-shot tasks: classify a ticket, extract fields from a document, summarise an article, draft a reply from a supplied thread.
• High-volume APIs where any instance can serve any request and horizontal scaling must be simple.
• Sensitive data where retaining inputs creates risk — for example, processing documents that contain personal or health information.
• Deterministic pipelines where reproducibility and testing matter more than personalisation.

Benefits are significant: no session affinity, easier load balancing, simpler failure recovery (just retry), smaller attack surface and straightforward testing with fixed inputs.

The limits appear when tasks span multiple turns or steps. A stateless design either asks the client to send the whole history each time — which moves the burden and the risk to the client — or forces users to repeat themselves.

When you need state

State becomes necessary when:

• Conversations are long or multi-turn, and users expect the agent to follow the thread.
• Tasks pause for human approval, a customer reply, a payment or a slow external system, and must resume exactly where they stopped.
• Personalisation adds clear value, such as remembering a user’s preferred language, product configuration or recurring requests.
• Work is expensive to redo, so losing progress after a crash or deployment would waste time and money.

The costs are real too: storage, more complex scaling, the need to keep state consistent with the systems of record, larger prompts as context grows, and privacy obligations for anything remembered.

Design patterns that work

Stateless workers with durable workflow state

Keep the agent’s reasoning steps stateless, but record task progress in a durable workflow engine or database with checkpoints. Any worker can pick up the next step. This combines easy scaling with reliable resumption, and is the pattern behind frameworks that offer checkpointing and human-in-the-loop pauses.

Rolling summaries for long conversations

Keep the last few turns verbatim and replace older turns with a summary that preserves decisions, facts and open questions. Store the full transcript separately if you need it for audit, not in the prompt.

Explicit, scoped long-term memory

Rather than letting an agent store anything it likes, define what may be remembered — for example, “preferred delivery address label” or “product edition in use” — with a schema, a source and an expiry. Show users what is remembered and let them correct or delete it.

Reference, do not copy

Store references to records in your systems of record (an order ID, a ticket ID) rather than copies of their data. Fetch the current values when needed. This avoids an agent acting on a stale order status or price.

Worked example: a B2B procurement assistant

Consider a hypothetical assistant that helps buyers at business customers reorder supplies, check contract prices and track deliveries.

The design deliberately keeps prices out of memory. If the assistant “remembered” last month’s contract price, it could quote a figure that has since changed. It also makes the approval pause durable: if the service restarts while waiting, the workflow resumes from its checkpoint rather than asking the buyer to start again.

The cost of carrying state

State has a direct token cost. Suppose each turn adds about 400 tokens of user message and agent response, and the system prompt and tool definitions take 3,000 tokens. Replaying the full history, the twentieth turn sends roughly 3,000 + 19 × 400 = 10,600 tokens of input. Over a 20-turn conversation the input totals around 136,000 tokens. With the last six turns kept verbatim and older turns replaced by a 500-token summary, no turn exceeds about 3,000 + 500 + 6 × 400 = 5,900 tokens, and the conversation total falls to roughly 106,000 — about 22% less input, with a larger saving the longer conversations run. Prompt caching of the stable prefix can reduce cost further. Test summaries against your evaluation set, because a summary that drops a key fact costs more than the tokens it saves.

Choosing storage

Privacy, security and correctness risks of memory

• Cross-user leakage. Memory must be strictly scoped by user and tenant; a retrieval query that forgets the filter can expose another person’s data.
• Retention and deletion. Memory is personal data in many jurisdictions. Apply retention periods and make deletion requests remove it from every store, including caches and vector indexes.
• Poisoned memory. Content from documents or other users can plant false “facts” that the agent later trusts. Only write memory from trusted sources or after confirmation, and record where each memory came from.
• Stale facts. Preferences and circumstances change. Expire memories and prefer live system data for anything transactional.
• Context bloat. Sending all memory with every request raises cost and can distract the model. Retrieve only what is relevant to the current task.

A decision checklist

Measuring the trade-off

Before adding memory, measure what it buys. Compare task success rate, number of turns to completion, user satisfaction and cost per completed task with and without each form of state on an evaluation set. Memory that does not improve these metrics is cost and risk without benefit.

Frequently asked questions

Do provider conversation features make our agent stateful?

They store conversation history on the provider side, which is convenient but means your data lives there under their retention terms. Check those terms and whether you can delete conversations on request.

Is a vector database the same as agent memory?

A vector database is one storage option. Memory is the policy about what to store, for whom, for how long and how it is retrieved. Many memories are better stored as structured fields.

Are stateless agents more secure?

They have a smaller attack surface because they keep nothing between requests. Security still depends on how inputs, tools and permissions are handled.

How do we test stateful agents?

Test scenarios end to end, including pauses, resumption after failure, memory updates and deletion, using fixed seed data so results are reproducible.