Observability with OpenTelemetry: Traces, Metrics and Logs Without Vendor Lock-In

Modern applications are distributed: dozens of services, managed cloud components, queues, third-party APIs and front ends across devices. When something goes wrong, a dashboard showing CPU usage rarely explains why checkout is slow for some customers in one region. Observability is about being able to answer new questions about system behaviour from the data it emits.

Monitoring versus observability

The three main signals

• Traces follow a request across services, showing where time is spent and where errors occur.
• Metrics are numerical measurements over time — request rates, error rates, latency, resource use.
• Logs are timestamped records of events with details.

Correlation is what makes them powerful: a slow trace links to the logs of the failing service and to metrics showing a saturated database.

What OpenTelemetry provides

OpenTelemetry is an open-source observability framework, now a graduated project of the Cloud Native Computing Foundation. It offers:

• APIs and SDKs for many languages to instrument applications.
• Automatic instrumentation for common frameworks and libraries.
• Semantic conventions — standard names for attributes such as HTTP routes and database operations.
• The Collector, a service that receives, processes and exports telemetry to one or more backends.
• Context propagation so trace identifiers flow across service boundaries.

Because instrumentation is vendor-neutral, you can change or combine backends without re-instrumenting code.

Reference architecture

1. Applications instrumented with OpenTelemetry SDKs, using automatic instrumentation plus custom spans for business operations.
2. Collectors deployed as agents or gateways, adding metadata, filtering, sampling and batching.
3. Backends for traces, metrics and logs — open-source or commercial — chosen by capability and cost.
4. Dashboards and alerts built on service level indicators.
5. Front-end and mobile telemetry linked to back-end traces for end-to-end visibility.

Service level objectives

Rather than alerting on every metric, define service level indicators (SLIs) — such as the share of checkout requests completing successfully within two seconds — and objectives (SLOs) for them. Alert when the error budget is being consumed too quickly. This focuses on-call attention on user impact and reduces alert fatigue, an approach described in Google’s Site Reliability Engineering book.

Controlling cost

Telemetry volume can grow faster than traffic. Practical controls:

• Sampling: keep all errors and slow requests, sample routine successful traces.
• Attribute discipline: avoid high-cardinality attributes such as user IDs in metrics.
• Log levels: debug logs off in production unless investigating.
• Retention tiers: short retention for detailed data, longer for aggregates.
• Collector processing: drop or redact unnecessary data before export.

Worked example: diagnosing slow checkout

A hypothetical e-commerce platform receives complaints that checkout is sometimes slow. Metrics show average latency is fine. With OpenTelemetry tracing across the web front end, API gateway, order service, payment service and inventory service, the team filters traces for checkout requests above the 95th percentile. Most slow traces share one pattern: the inventory service makes a call to a warehouse API for each basket line, and the warehouse API is slow for one region. Logs linked to those spans show timeouts and retries.

The fix — batching inventory checks and caching availability for a short period — is verified by comparing trace durations before and after. Without traces linking services, the team would have been guessing across five services and two teams.

Custom instrumentation that pays off

• Business operations as spans: “place order”, “calculate price”, “reserve stock”.
• Meaningful attributes: order size bucket, payment method, region, tenant tier — avoiding personal data.
• Events for key decisions: cache hits, fallbacks used, retries.
• Links to deployments and feature flags so changes can be correlated with behaviour.

Security and privacy

Telemetry often contains personal or sensitive data: URLs with identifiers, request bodies, user information. Redact at the source or in the Collector, restrict access to telemetry backends and apply retention rules consistent with privacy obligations.

Rollout plan

1. Start with critical journeys: instrument the services involved in checkout, login or another key flow.
2. Deploy a Collector and connect one backend.
3. Define SLOs for those journeys and alert on burn rates.
4. Add context propagation across all services in the journey, including queues.
5. Expand service by service using templates in your internal platform.
6. Review costs and usefulness quarterly.

Frequently asked questions

Do we need to replace our monitoring tools to use OpenTelemetry?

No. Many commercial and open-source tools accept OpenTelemetry data. You can adopt it gradually alongside existing agents.

Is automatic instrumentation enough?

It gives a quick start, but custom spans and attributes for business operations make traces far more useful.

How much does observability cost?

It depends on volume and backend. Without sampling and retention policies, costs can rise quickly; design controls from the beginning.

Does observability help AI features?

Yes. Tracing model calls, retrieval steps and tool use — with token counts and latency — is essential for debugging and cost control.