Event-Driven APIs: Webhooks, Streams and Async Patterns That Hold Up in Production

Most APIs start as request–response: a client asks, the server answers. That works well until a client needs to know when something changes. Polling every few seconds wastes resources and still delivers updates late; polling every few minutes keeps users waiting. Event-driven APIs reverse the flow: the provider notifies consumers when something happens.

Events are now expected in most platforms — payment providers notify merchants of settled payments, SaaS products notify customers’ systems of new records, and internal services react to each other without tight coupling. Getting them right is harder than it looks. This guide covers the design decisions and the production details that are easy to miss.

Events, commands and queries

Three kinds of messages flow through modern systems, and mixing them up causes design problems:

• Queries ask for current state: “what is the status of order 1042?” They suit request–response APIs.
• Commands ask a system to do something: “cancel order 1042.” They have a single intended handler and can be rejected.
• Events state that something has already happened: “order 1042 was cancelled.” They are facts in the past tense, can have many consumers and cannot be rejected.

Event-driven APIs deal with the third kind. A good test of an event name is whether it reads as a past-tense fact — order.cancelled, invoice.paid, user.email_changed.

Choosing a delivery mechanism

Design the event contract first

An event is a public interface. Once consumers depend on it, changing it is as disruptive as changing a REST endpoint. Design it deliberately.

Use a standard envelope

CloudEvents, a specification that graduated in the Cloud Native Computing Foundation on 25 January 2024, defines common metadata for events so that tools and consumers can handle them consistently. Its core attributes include:

• id — unique identifier for the event;
• source — the context that produced it;
• type — what happened, such as com.example.order.cancelled;
• specversion — the CloudEvents version;
• subject, time, datacontenttype and dataschema — optional attributes describing the subject, timestamp, payload format and schema.

Fat or thin events?

• Thin events carry identifiers only (“order 1042 changed”); consumers call the API for details. They are small and never stale, but create load and depend on the API being available.
• Fat events carry the relevant state. Consumers can act without calling back, but payloads are larger and you must think carefully about sensitive data.

A common middle path is to include the fields most consumers need and a link to the full resource.

Version and document

• Treat adding optional fields as non-breaking. Removing or renaming fields, or changing their meaning, is breaking.
• For breaking changes, publish a new event type or version and run both in parallel during a deprecation period.
• Document channels, messages and schemas in AsyncAPI, which plays the role for event-driven interfaces that OpenAPI plays for REST. Generate documentation and validate payloads from it in CI.

Webhooks done properly

Webhooks look simple — send a POST — but most integration incidents we investigate involve webhooks. The Standard Webhooks specification, an open effort to make webhook implementations consistent, codifies good practice that applies whether or not you adopt it formally.

Sign every request

Consumers must be able to verify that a webhook came from you and was not altered. Standard Webhooks uses three headers:

• webhook-id — a unique message identifier, kept the same across retries;
• webhook-timestamp — Unix time in seconds;
• webhook-signature — one or more signatures, separated by spaces, which allows secret rotation without downtime.

The signed content is the message ID, timestamp and raw body joined by full stops. Symmetric signatures use HMAC-SHA256. Consumers should verify against the exact raw bytes received — re-serialising parsed JSON can change them — compare signatures in constant time, and reject messages whose timestamp falls outside an acceptable window to block replay attacks.

Retry with backoff, and know when to stop

Consumers will be down sometimes. Retry failed deliveries with exponential backoff and random jitter, spread over hours or days. Treat only 2xx responses as success. Stop sending to an endpoint that returns 410 Gone, slow down on 429 or gateway errors and honour Retry-After. After prolonged failure, notify the customer through another channel and disable the endpoint rather than retrying forever.

Help consumers respond quickly

Advise consumers to verify the signature, store the event and return 2xx immediately, then process asynchronously. Slow handlers cause timeouts, which cause retries, which cause duplicates.

Give customers control

Provide a dashboard or API to register endpoints, choose event types, rotate secrets, view delivery attempts and replay events. This reduces support load dramatically.

Reliability: delivery guarantees, idempotency and ordering

Assume at-least-once delivery

Exactly-once delivery across networks is, for practical purposes, not achievable end to end. Systems deliver at least once and expect duplicates. Consumers must therefore be idempotent: processing the same event twice must have the same effect as processing it once. The usual technique is to record processed event IDs and skip repeats, within the same transaction as the side effect where possible.

Publish reliably with the outbox pattern

A common bug is to update the database and then publish an event in a separate step. If the process crashes between the two, the event is lost; if publishing happens first, consumers may see an event for a change that was rolled back. The transactional outbox fixes this: write the event to an outbox table in the same database transaction as the change, then have a separate relay publish from the outbox. Change-data-capture tools can serve as the relay.

Decide what ordering you need

Global ordering is expensive and rarely necessary. Usually you need ordering per entity — all events for order 1042 in sequence. Brokers such as Kafka provide ordering within a partition, so partition by entity ID. For webhooks, which can arrive out of order after retries, include a sequence number or version on each entity and have consumers ignore events older than the state they already hold.

Provide replay and reconciliation

Consumers will miss events through outages, bugs or deployments. Offer a way to recover: a replay API for a time range, an events list endpoint, or a regular reconciliation job comparing state via the REST API. Our guide to zero-downtime migration covers reconciliation techniques in more detail.

Worked example: events for an order platform

Consider a hypothetical B2B ordering platform with a web front end, a warehouse system, an accounting integration and customers who want order updates in their own ERP systems.

The order service writes each change and its event to the outbox in a single transaction. A relay publishes to the broker. A webhook dispatcher subscribes to the broker, looks up which customer endpoints want each event type, signs the payload and manages retries. Customers can see delivery attempts and replay any event from the last 30 days in the portal. Because every event carries an order version number, a customer’s ERP can safely ignore an older status that arrives after a newer one.

Governing an event catalogue

As the number of events grows, governance prevents chaos:

• Keep a catalogue listing every event type, its owner, schema, consumers and status (active, deprecated, retired).
• Name events consistently, for example domain.entity.past_tense_verb, and agree naming rules before the first public release.
• Require a design review for new public events, as you would for new public API endpoints.
• Announce deprecations with dates and keep deprecated events running for an agreed period, tracking which consumers still receive them.
• Avoid leaking internals. Public events should describe business facts, not database changes. Raw change-data-capture streams are useful internally but make poor public contracts.

Security checklist

• Sign webhooks and verify signatures; reject stale timestamps.
• Require HTTPS for all endpoints and refuse to follow redirects.
• Protect against server-side request forgery when customers register URLs: block internal and private address ranges.
• Keep sensitive personal or financial data out of payloads unless necessary; prefer identifiers with authenticated fetch.
• Authenticate and authorise WebSocket and SSE connections, and re-check authorisation for long-lived connections.
• Rate-limit and size-limit inbound events on the consumer side.
• Log delivery attempts without logging secrets or sensitive payload fields.

Observability

Track, per event type and per consumer endpoint: events published, delivery success rate, latency from occurrence to delivery, retry counts, endpoints disabled and consumer lag for streams. Propagate a correlation or trace ID from the originating request through the event to downstream processing, so one business transaction can be followed across services.

Testing event-driven integrations

• Contract tests validate published events against the AsyncAPI or JSON Schema definitions in CI.
• Consumer simulation sends duplicates, out-of-order events, invalid signatures and expired timestamps to confirm consumers handle them.
• Failure tests take consumers offline and confirm retries, backoff, disabling and replay behave as designed.
• Load tests check that bursts — such as a bulk import producing thousands of events — do not overwhelm consumers or delivery workers.

Frequently asked questions

Should we use webhooks or a message broker?

Use a broker or stream for internal services that you operate, and webhooks for external consumers who should not need access to your infrastructure. Many platforms run a broker internally and deliver webhooks from it.

Do we need Kafka?

Not necessarily. Kafka suits high-volume streams that need replay and partitioned ordering. For modest volumes, a managed queue or a database-backed outbox with a worker can be simpler and cheaper to run.

How long should we retry webhooks?

Long enough to cover a realistic consumer outage, often a day or more, with exponential backoff and jitter. Publish your retry schedule so consumers know what to expect.

Should events include personal data?

Only when consumers genuinely need it. Thin events with identifiers, fetched through an authenticated API, reduce exposure and make access easier to control and audit.

Is CloudEvents required?

No, but it gives you a well-understood envelope and SDKs in many languages, and saves debating basic metadata. Adopt it unless you have a clear reason not to.