Building APIs for AI Features: Streaming, Limits, Versioning and Safety

Most teams add their first AI feature by calling a model provider from the backend and returning the result. That works for a prototype. In production, the properties of language models start to shape the API around them: a single request can take a fraction of a second or two minutes, cost depends on how much text goes in and out, the same input can produce different outputs, and the model you depend on will be updated or retired.

This guide is about the API you build between your applications and the AI features behind them — the endpoints your web app, mobile app or partners call. It draws on patterns that model providers use in their own APIs, because they solved the same problems at much larger scale. For the overall architecture of AI applications, see our guide to choosing AI application architecture.

Why you need your own AI API layer

Calling a provider directly from a browser or mobile app exposes your API key, gives up control of cost and makes it impossible to enforce business rules. A dedicated layer — sometimes called an AI gateway — gives you one place to:

• hold secrets and rotate provider keys without app releases;
• authenticate and authorise users, and apply tenant or plan entitlements;
• assemble context — retrieve documents, add instructions, trim history — on the server, where it cannot be tampered with;
• budget tokens and cost per user, tenant and feature;
• route between models or providers and fail over when one is unavailable;
• validate outputs before returning them;
• log and measure usage, latency, cost and quality.

A clean pattern is to expose feature-level endpoints — summarise a ticket, draft a reply, extract invoice fields — rather than a generic “send this prompt” endpoint. Feature endpoints keep prompts on the server, make permissions and limits specific, and let you change models or prompts without touching clients.

Choose the interaction style by response time

Streaming done properly

Server-sent events (SSE) are the most common transport for streaming model output. They run over ordinary HTTP, work through most proxies and suit one-way token streams. Anthropic’s Messages API is a useful reference design: a stream begins with a message_start event, then each content block is sent as a start event, a series of delta events and a stop event, followed by message_delta events carrying cumulative usage and a final message_stop. Ping events keep the connection alive, and errors such as an overloaded condition can arrive as an error event in the middle of the stream.

Lessons to copy into your own API:

• Use typed events. Separate events for text deltas, tool or progress updates, usage and completion make clients simpler and let you add event types later. Tell clients to ignore unknown types.
• Send keep-alives. Proxies and load balancers often close idle connections; periodic ping events prevent that during long pauses.
• Report errors in-band. Once streaming has started, the HTTP status is already 200. Send a structured error event and let the client decide whether to retry or show partial output.
• Send usage at the end. Final token counts let clients and your billing system record the true cost.
• Support cancellation. When a user closes the view or presses stop, cancel the upstream model call so you stop paying for tokens nobody will read.
• Plan for interruption. Store partial output server-side with a response ID, so a reconnecting client can fetch what was generated rather than starting again.

Long-running work as jobs

Anything that can exceed roughly 30 seconds — parsing a 200-page contract, running an agent, processing a batch — should be a job, not a held-open request. Mobile networks drop, users background apps and gateways time out.

1. POST /v1/document-reviews creates the job and returns 202 Accepted with a job ID and a status URL.
2. GET /v1/document-reviews/{id} returns status (queued, running, needs_input, succeeded, failed, cancelled), progress and, when finished, the result or a link to it.
3. Webhooks notify servers of completion; mobile apps can use push notifications. Our guide to event-driven APIs covers signing and retries.
4. POST /v1/document-reviews/{id}/cancel stops work and spending.

Accept an Idempotency-Key header on job creation so a client retrying after a network failure does not start, and pay for, the same job twice.

Manage context and input size

Model cost and latency grow with input size, and every model has a context limit. Design for it explicitly:

• Validate size at the edge. Reject or truncate inputs above a documented limit with a clear error (413 or 422 with a machine-readable code), rather than failing deep inside the pipeline.
• Count tokens, not characters. Use the provider’s tokenizer or token-counting endpoint for budgets; character estimates vary by language.
• Retrieve rather than send everything. For large documents, chunk and retrieve relevant passages instead of passing whole files.
• Summarise long histories. Keep recent turns verbatim and replace older turns with a summary.
• Expose limits in documentation and responses so client teams can design around them.

Our guide to reducing LLM token costs covers these techniques and their quality trade-offs in more depth.

Rate limits and quotas that reflect cost

A request that sends 50 tokens and one that sends 50,000 cost very different amounts. Providers therefore limit not just requests per minute but input and output tokens per minute. Anthropic, for example, returns headers on every response showing the limit, remaining allowance and reset time for requests, input tokens and output tokens, and a retry-after header with a 429 response.

Apply the same thinking to your own API:

Queue requests when you are close to provider limits rather than failing immediately, and prioritise interactive traffic over background jobs.

Errors and retries

Version the feature, pin the model

Model providers release new versions and retire old ones on their own schedule. Your clients should never notice unless you decide they should.

• Version your API contract (for example /v1/) based on request and response shapes, not on which model runs behind it.
• Pin specific model versions in configuration, not floating aliases that change silently.
• Upgrade models through evaluation. Run the new model against your evaluation set for each feature, compare quality, latency and cost, then roll out gradually behind a flag.
• Return a model or prompt version identifier in responses and logs so you can trace a complaint to the exact configuration that produced it.
• Plan for deprecation dates. Track provider retirement notices and schedule evaluations well before deadlines.

Structured outputs

When the result feeds other software — extracted fields, classifications, actions — ask the model for JSON that conforms to a schema, using the provider’s structured-output or tool-calling features where available. Then validate the result on your server anyway. Reject or repair invalid output, enforce business rules (an extracted invoice total must equal the sum of lines) and never pass unvalidated model output to a database, shell or HTML page.

Security and privacy

The OWASP Top 10 for LLM Applications highlights the risks most relevant to an AI API layer:

• Prompt injection (LLM01). User input and retrieved documents can contain instructions. Keep system instructions on the server, separate untrusted content clearly, and limit what the model’s output can trigger.
• Sensitive information disclosure (LLM02). Enforce permissions before retrieval, redact personal data where possible and avoid sending secrets to models.
• Improper output handling (LLM05). Escape and validate outputs before rendering or executing them.
• System prompt leakage (LLM07). Assume prompts can be extracted; never put credentials or sensitive logic in them.
• Unbounded consumption (LLM10). Enforce token, cost and concurrency limits.

Add the usual API controls too: strong authentication, per-tenant isolation, encryption in transit, data-retention rules for prompts and outputs, and contractual checks on how providers use your data.

Authentication and multi-tenancy

AI features often process the most sensitive data in a product — documents, conversations, customer records — so the access model needs particular care:

• Authenticate the end user, not just the app. Pass the user’s identity to the AI layer so retrieval and tools act with that user’s permissions.
• Isolate tenants everywhere: in retrieval indexes, caches, conversation history, logs and fine-tuning data. A cache key that omits the tenant is a data leak waiting to happen.
• Separate service credentials per feature so a compromised or misbehaving feature cannot reach data used by another.
• Keep provider keys server-side, rotate them regularly and set spending limits at the provider as a backstop.

Testing an AI API

Conventional API tests still apply — contracts, authentication, error codes, load — but AI features need more:

Observability

Log, for every call: feature, user and tenant identifiers, model and prompt version, input and output tokens, latency to first token and to completion, cost, cache use, finish reason, validation result and any user feedback. Redact personal data before storage. Dashboards by feature and tenant answer the questions that will come up first: what is this costing, who is using it, is it getting slower and is it still good?

A checklist before launch

• Feature-level endpoints with server-side prompts and context assembly.
• Streaming with typed events, keep-alives, cancellation and in-band errors.
• Jobs with idempotency keys for long-running work.
• Documented input limits and token-based quotas with clear headers.
• Pinned model versions and an evaluation gate for changes.
• Schema validation of structured outputs.
• Prompt-injection, permission and data-retention controls.
• Logging of tokens, cost, latency and versions per call.

Frequently asked questions

Should mobile apps call the model provider directly to reduce latency?

No. It exposes credentials and removes your ability to enforce limits and rules. The extra hop through your API is small compared with model generation time, especially with streaming.

SSE or WebSockets for streaming AI responses?

SSE is usually enough: model output flows one way and SSE works over ordinary HTTP. Choose WebSockets when you need frequent two-way messages on the same connection, such as real-time voice or collaborative editing.

Can we cache AI responses?

Yes, for deterministic or repeated requests such as classifying identical inputs or answering common questions from fixed content. Include the model and prompt version in the cache key, set expiry times, and never share cached answers across users when content depends on their permissions.

How do we handle a provider outage?

Retry with backoff, then fail over to a second model or provider for features where you have evaluated one, or degrade gracefully with a clear message. Test the failover path regularly.