Reducing LLM Token and Cloud Costs without Losing Answer Quality

Generative AI applications usually become expensive gradually. A system prompt grows as edge cases are added, retrieval returns a few more passages “just in case”, conversations get longer, an agent receives every tool the team has ever built, and a GPU instance sized for a demo keeps running at night. Each change is small. Multiplied by thousands of requests a day, together they decide whether a feature is commercially viable.

This guide covers both sides of the bill: tokens sent to and from models, and the cloud infrastructure around them. It explains where waste comes from, compares nine techniques, and — most importantly — shows how to prove that a saving did not damage the answers your users rely on. For the wider question of how to measure AI spend by feature and outcome, see our companion guide to AI FinOps and cost per successful workflow.

Start with a baseline, not a technique

Teams often start optimising with whatever technique they read about last. A better first step is a baseline that tells you where tokens and money actually go. Without it, you cannot tell whether a change saved anything, and you cannot tell whether quality moved.

Capture, for one representative week:

• Tokens per call, split by component: system instructions, retrieved context, conversation history, tool definitions, user input and output.
• Calls per task: how many model calls a single user task triggers, including retries and agent steps.
• Cost per successful task: total cost divided by tasks that met your quality bar.
• Latency: median and 95th percentile per call and per task.
• Infrastructure cost: GPU or CPU hours, vector database, storage and data transfer, allocated to the feature.
• Quality: pass rate on a fixed evaluation set and production signals such as acceptance or escalation.

The component split matters most. In many systems we review, the user’s actual question is a small fraction of the input; most tokens are instructions and context that were added once and never revisited.

Where tokens leak

Nine techniques and their trade-offs

The techniques below are ordered roughly from “remove waste” to “accept a trade-off”. Earlier ones are usually safe; later ones need careful measurement.

1. Trim and modularise instructions

Review the system prompt line by line. Remove rules that duplicate others, examples that no longer help and instructions for features that were removed. Then split what remains: core rules that apply to every request, and task-specific instructions loaded only when needed. Risk: removing an instruction that handled a rare case. Mitigate by keeping those cases in your evaluation set.

2. Retrieve instead of injecting

Instead of sending whole documents, retrieve the passages relevant to the question. Chunk by document structure, combine keyword and semantic search where it helps, and limit the number of passages. Our guide to RAG or fine-tuning explains when retrieval is the right approach. Risk: the right passage is not retrieved and the model answers from general knowledge. Mitigate with citations, a “not found” path and retrieval metrics.

3. Manage conversation history

Keep the last few turns verbatim, summarise older turns, and store durable facts — the customer’s order number, chosen plan, stated preferences — as structured data you inject when relevant. Risk: summaries drop details that later matter. Mitigate by testing multi-turn scenarios and keeping key facts outside the summary.

4. Load tools on demand

For agents, group tools by task and expose only the group relevant to the current step, or give the agent a lightweight way to discover tools when it needs them. Smaller tool lists also tend to reduce wrong tool choices. Risk: the agent cannot find a tool it needs. Mitigate with tests that cover cross-group tasks.

5. Constrain output

Ask for the exact format the application needs — fields, short answers, bullet points — and set a maximum output length per step. Structured output features offered by several providers help ensure responses match a schema, which also reduces retries caused by malformed output. Risk: truncated or overly terse answers. Mitigate by checking completeness in evaluation and by setting limits per step rather than globally.

6. Cache stable prompt prefixes

Many providers discount repeated input. Anthropic, for example, documents cache reads at 0.1× the base input price, with cache writes at 1.25× for a five-minute cache or 2× for a one-hour cache, and a minimum cacheable length for most models. Keep instructions and reference material identical and at the start of the prompt so the cache can hit. Risk: small edits to the prefix silently disable caching. Monitor cache hit rate.

7. Cache answers to repeated questions

For questions that recur — “what is your refund policy?” — you can cache final answers, keyed by normalised question and the version of the underlying content. Risk: stale or wrong answers served to many users, and personal data leaking between users. Cache only answers that do not depend on the user, and invalidate when source content changes.

8. Batch work that can wait

Summaries, classification and enrichment that do not need an immediate response can often run as batch jobs. OpenAI’s Batch API, for example, offers a 50% discount compared with synchronous requests in exchange for completion within 24 hours. Risk: latency that users notice if the job is not truly asynchronous.

9. Route by difficulty

Send simple requests to smaller, cheaper models and difficult ones to stronger models, or try a cheap model first and escalate when confidence is low. Research supports the idea: the FrugalGPT paper (Chen, Zaharia and Zou, 2023) reported matching GPT-4’s performance at up to 98% lower cost on the tasks it studied, and RouteLLM (Ong and colleagues, 2024) reported cost reductions of more than two times in certain cases while maintaining response quality. Risk: these results come from specific benchmarks; your traffic may differ. Misrouted hard requests produce worse answers, which can raise total cost if people must fix them.

Infrastructure: the other half of the bill

If you host models, run embeddings at scale or operate vector databases, infrastructure can rival token spend. Most of the advice is standard cloud cost practice, applied to AI workloads.

Two AI-specific points are worth adding. First, vector indexes grow quietly: re-embedding a whole corpus after every small document change multiplies cost, so embed incrementally and remove superseded content. Second, evaluation and experimentation environments often run on expensive hardware far longer than needed; schedule them, shut them down automatically and give each experiment a budget.

A worked example: one support assistant, five changes

The following example is illustrative. It shows how changes compound and why each one must be measured.

A support assistant answers agents’ questions. The baseline drafting call uses 9,000 input tokens: 3,000 of instructions, 4,500 of retrieved passages and 1,500 of history and question. Output averages 500 tokens. The team applies five changes in sequence and measures each against a 200-question evaluation set.

The result: input tokens fell by roughly 43% and a large part of the remaining prefix is cached, with quality unchanged. One change was rejected because it reduced quality, which would have been invisible if the team had only watched the token chart.

A benchmark plan that protects quality

Treat every optimisation as an experiment with a hypothesis, a measurement and a decision.

1. Fix the evaluation set. Use real, anonymised examples with expected outcomes agreed by the business owner, including rare and difficult cases. Keep it stable so results are comparable.
2. Measure the baseline. Record pass rate, tokens per task, cost per successful task and latency.
3. Change one thing. Combine changes only after each has been measured on its own.
4. Compare on all four dimensions. A change that cuts tokens but adds latency or reduces pass rate is a trade-off, not a saving.
5. Run in production with a holdback. Release to a share of traffic behind a flag, keep a control group on the old configuration and compare production signals for at least a full business cycle.
6. Keep a kill switch. Be able to revert a prompt, retrieval setting, cache or route in minutes.

Decision rules

• Accept a change if quality stays within an agreed tolerance and cost per successful task falls.
• Reject a change if quality falls beyond the tolerance, even if tokens fall dramatically.
• Escalate to the product owner when a change trades a small quality loss for a large saving; that is a business decision, not an engineering one.

Measuring token efficiency in production

After rollout, a few metrics tell you whether savings hold and where new waste appears. Review them weekly for new features and monthly once stable.

Use the metrics to set budgets per feature and alerts on unusual changes. A sharp rise in input tokens after a release usually points to a prompt or retrieval change; a rise in calls per task often points to retries or an agent taking more steps.

Security and privacy implications of each technique

Cost techniques interact with data handling. Review them with whoever owns security and privacy:

Who approves changes

Cost optimisations change behaviour, so they need the same discipline as feature changes. A lightweight process works well:

• Engineering proposes the change with expected savings and the evaluation results.
• The feature’s product owner approves any change that affects answer content, tone, completeness or latency users will notice.
• Security or privacy owners review changes involving new providers, caching of user data or new data flows.
• Operations or support are told before release, so they recognise any change in behaviour reported by users.

Record each change, its measured effect and its rollback steps. Over time this log becomes a valuable record of what worked for your workload.

Review model choices regularly

Model prices, capabilities and context limits change frequently. A configuration that was optimal six months ago may now be expensive or underperforming. Every quarter, re-run your evaluation set on the current model and one or two alternatives, including smaller models for your simplest tasks. Compare cost per successful task, not price per token, and include latency and operational considerations such as rate limits and regional availability.

Cache invalidation and data safety

Caching is where cost optimisation most often creates new risks:

• Version every cache key with the content version it depends on — policy document version, product catalogue version, prompt version.
• Never cache personalised answers across users. If an answer contains account data, it must not be served to anyone else.
• Set expiry times that reflect how often the underlying content changes, and purge on publish for content you control.
• Log cache hits with the version used, so you can trace which answer a user received.

What not to optimise

Some savings are false economies:

• Removing safety or validation steps to save a model call. A single bad output in a regulated or customer-facing context costs more than months of tokens.
• Downgrading the model for complex reasoning without measuring the impact on human review time.
• Over-compressing context for tasks where precision matters, such as contracts, prices or medical information.
• Optimising a feature nobody uses. Check usage before spending engineering time.

A practical checklist

• A baseline exists with tokens split by component, cost per successful task, latency and quality.
• System instructions are reviewed and modular.
• Retrieval returns a limited number of relevant passages with citations.
• Conversation history is summarised and key facts are stored separately.
• Agents load tools by task and have step budgets.
• Output formats and per-step length limits are defined.
• Stable prefixes are cached and cache hit rate is monitored.
• Non-urgent jobs run in batch where discounts apply.
• Routing decisions are based on measured quality per route.
• Infrastructure is right-sized, tagged, scheduled and scaled to demand.
• Every change is tested on a fixed evaluation set and released with a holdback and kill switch.

Frequently asked questions

Which technique usually saves the most?

It depends on the leak. In systems with long instructions or large retrieved context, trimming and better retrieval usually save the most with the least risk. In high-volume, repetitive workloads, prefix caching and batching can be significant. Measure your baseline first.

Does a shorter prompt always mean a worse answer?

No. Irrelevant context can distract a model, so removing it sometimes improves answers. The only way to know is to compare on your evaluation set.

Should we self-host an open model to cut costs?

Self-hosting replaces token prices with infrastructure, operations and engineering time. It can pay off at steady high volume or with strict data requirements, but compare total cost per successful task including the people needed to run it.

How often should we revisit optimisations?

Whenever prompts, models, prices or traffic change significantly, and at least quarterly. Provider pricing and model capabilities change often.