Home / Insights / Generative AI and LLMs
Generative AI and LLMs
From Prototype to Production: A Readiness Checklist for LLM Features
An LLM prototype can be built in days; a feature that behaves predictably for thousands of users takes deliberate engineering. This checklist covers the reliability work that turns a promising demo into software your team can operate and change safely.
TopicsGenerative AI and LLMsArchitecture and ModernizationQA and Software Testing
Key takeaways
- Treat the model call as an unreliable network dependency: set timeouts, bounded retries and a fallback path.
- Use schema-constrained structured outputs where available, and still validate every response before acting on it.
- Version prompts like code and run an evaluation suite whenever prompts, models or retrieval change.
- Log prompts, responses, latency, cost and user feedback from day one, with sensitive data redacted.
- Decide in advance what the feature does when the model is unavailable, slow or refuses.
On this page
- Input handling: validate, redact and route
- Context assembly: give the model only what it needs
- The model call: timeouts, retries, fallbacks and typed outputs
- Post-processing: never act on raw model output
- Change management: prompts, models and evaluation suites
- The readiness checklist
- Frequently asked questions
- Sources and methodology
The prototype worked. A product manager pasted a few real examples into a notebook, the model produced good classifications or summaries, and the team agreed to ship it. Then production arrived: the provider had a slow afternoon and requests hung for a minute, one response in a few hundred came back as malformed JSON and broke a downstream job, a minor prompt edit improved one case and silently degraded three others, and nobody could say what the feature was costing per user.
None of these problems is exotic. They are the ordinary concerns of any external dependency, plus a few that are specific to probabilistic output. At PoliteTech we treat LLM applications as normal software with an AI component, and this checklist is how we make that concrete. It is organised around four layers: input handling, context assembly, the model call and post-processing.
Input handling: validate, redact and route
- Validate inputs before they reach the model. Enforce length limits, reject empty or binary content, and normalise encodings. Very long inputs inflate cost and latency and can push important instructions out of the context window.
- Redact sensitive data you do not need. If the task is classifying a support ticket, the model rarely needs full card numbers, national IDs or home addresses. Mask them first.
- Route by task complexity. Simple steps such as language detection or short classification can often use a smaller, faster model; reserve larger models for harder reasoning. Routing rules should be explicit and tested.
- Separate user content from instructions. Mark untrusted user text clearly in the prompt and never let it be interpreted as system instructions. Our article on securing LLM applications against the OWASP Top 10 for LLMs covers prompt injection in more depth.
Context assembly: give the model only what it needs
Most quality problems in production LLM features trace back to context: too little, too much, stale or badly formatted. Context assembly deserves its own tested component rather than string concatenation spread across the codebase.
- Define, per task, exactly which records, documents and conversation turns are included, and in what order.
- Set a token budget for each part of the context and truncate predictably, keeping the most relevant material.
- Include identifiers with retrieved passages so outputs can cite sources and post-processing can verify them.
- Cache stable parts, such as long instructions and reference material, where the provider supports prompt caching.
For a broader treatment of this topic, see context engineering for AI agents.
The model call: timeouts, retries, fallbacks and typed outputs
Resilience settings
| Concern | Production practice |
|---|---|
| Slow responses | Per-request timeout tied to the user experience; stream output where users are waiting |
| Transient errors and rate limits | Bounded retries with exponential backoff and jitter; respect provider retry headers |
| Provider outage | Fallback to a secondary model or provider tested against the same evaluation suite, or a graceful non-AI path |
| Runaway cost | Max output tokens per call, per-user and per-tenant quotas, budget alerts |
| Non-determinism | Low-variance settings for extraction and classification; pin model versions where the provider allows |
Structured outputs
When the output feeds another system, ask for structured data rather than prose. Several providers now support schema-constrained generation. OpenAI’s documentation, for example, distinguishes Structured Outputs, which make responses follow a supplied JSON Schema, from the older JSON mode, which only guarantees valid JSON. It also notes that safety refusals are returned in a programmatically detectable way, which your code must check before parsing.
Even with schema enforcement, validate business rules after the call: an enum may be valid but wrong for this customer, or a date may be well formed but in the past. Schema guarantees shape, not truth.
Post-processing: never act on raw model output
- Validate against the schema and business rules. Reject or repair invalid outputs; do not silently pass them downstream.
- Check citations. If the model claims a fact came from a document, confirm the cited ID was actually in the context.
- Encode output for its destination. Treat model text as untrusted when rendering HTML, building SQL or calling APIs.
- Apply confidence routing. Low-confidence or high-impact outputs go to a human review queue rather than straight through.
- Log the decision. Record what was generated, what was changed by validation and what the user did next.
Change management: prompts, models and evaluation suites
LLM features change in ways ordinary code does not. A provider updates a model, a teammate edits a prompt, or retrieval starts returning different passages after new documents arrive. Each can shift behaviour without a single line of application code changing.
- Version prompts in source control alongside the code that uses them, with reviews like any other change.
- Maintain an evaluation set of representative inputs and expected outputs, including edge cases and known past failures.
- Run the suite in CI on every prompt, model or retrieval change, and compare against the previous baseline.
- Roll out gradually behind a feature flag or to a percentage of traffic, watching quality and cost metrics.
- Keep a rollback path to the previous prompt and model combination.
This is the core of LLMOps; our article on MLOps and LLMOps covers the operating model around it.
The readiness checklist
| Area | Ready when… |
|---|---|
| Task definition | Target user, task and success criteria are written down |
| Evaluation | An evaluation set exists, has a baseline score and runs automatically |
| Resilience | Timeouts, bounded retries and a tested fallback are in place |
| Output safety | Responses are schema-validated, rule-checked and encoded before use |
| Data handling | Sensitive fields are minimised or redacted; provider retention settings are confirmed |
| Cost control | Token limits, quotas and a cost dashboard exist |
| Observability | Prompts, responses, latency, cost and feedback are logged with redaction |
| Degraded mode | The product behaves sensibly if the model is slow, down or refuses |
| Ownership | Someone owns prompt changes, evaluation results and incident response |
Frequently asked questions
Do we need structured outputs if our feature only shows text to users?
Not necessarily, but structured output still helps when you need to separate an answer from its citations or metadata. For anything another system consumes, structured output plus validation is strongly recommended.
How do we stop a prompt change from breaking things?
Keep prompts in version control, maintain an evaluation set with a baseline, and run it automatically on every change. Roll out behind a flag and keep the previous version available for rollback.
Should we use a fallback model from a different provider?
It improves availability, but the fallback must be tested against the same evaluation set because prompts rarely transfer perfectly. For some features a non-AI fallback, such as a rules-based default, is simpler.
Can PoliteTech improve an LLM feature that is already live?
Yes. We usually start by evaluating current quality, cost and failure cases on real examples, then prioritise fixes such as structured outputs, better context assembly or an evaluation suite.
Sources and methodology
Guidance reflects PoliteTech delivery experience. The external references below informed specific points in this article.
- OpenAI: Structured Outputs (API documentation) — Difference between schema-constrained Structured Outputs and JSON mode, and programmatic refusal detection.
Related reading
Continue your research
Is your LLM prototype ready for real users?
We can review your prototype against this checklist and plan the engineering needed to run it reliably in production.
