Observability in Spring Boot with Micrometer and OpenTelemetry: what metrics to export before the client asks why it's slow
Almost every Spring Boot service we inherit already has an infrastructure dashboard: CPU, heap memory, active threads, database pool connections. That dashboard almost never answers the question that actually comes in first: "a user's checkout took eight seconds yesterday at 3pm, what happened." CPU was at a normal level at that time. Memory had no spikes. The dashboard says the system was healthy, yet one specific request was slow. That disconnect is what made us change approach: instead of starting from which infrastructure metrics are easy to expose, we start from the question we need to answer and work backward from there.
Why infrastructure metrics fall short
An infrastructure metric describes the state of the process or the machine: CPU usage, Tomcat pool threads, open connections in the HikariCP pool. These are necessary to know whether the system as a whole is under pressure, but they're aggregated across all traffic hitting the service. If ten percent of requests to a specific endpoint are slow for a localized reason -a locked row in a table, a call to an external provider that started degrading- that signal gets diluted among the ninety percent of requests that are fine, and the average CPU doesn't move enough to trigger any alert.
What answers "why was this specific request slow" isn't an aggregated metric, it's a distributed trace of that particular request, with the timing of each segment: how long the database query took, how long the call to the external provider took, how long the controller's own processing took. And what answers "how many requests of this kind are hitting this problem" isn't CPU, it's a business metric with the right labels: checkout endpoint duration, broken down by outcome, not the generic duration of all HTTP requests lumped together.
Micrometer and OpenTelemetry aren't competing, they complement each other
Micrometer is the instrumentation facade Spring Boot already ships with: it exposes a MeterRegistry and automatically records things like HTTP request duration, JPA queries and outgoing RestClient calls. OpenTelemetry is the export standard and the distributed tracing model. In Spring Boot 3 there's no need to choose one over the other: Micrometer Tracing uses OpenTelemetry as its underlying implementation, and the same micrometer-registry-otlp exports both metrics and traces to the same collector, in the same format.
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
management:
otlp:
metrics:
export:
url: http://otel-collector:4318/v1/metrics
step: 15s
tracing:
endpoint: http://otel-collector:4318/v1/traces
tracing:
sampling:
probability: 1.0
step: 15s defines how often metrics are aggregated and sent to the collector, not how often they're measured: Micrometer still records every event the moment it happens, and aggregates internally before exporting. A trace sampling probability of 1.0 makes sense while you're still figuring out what to export; on a high-volume service it's worth lowering, but that's a decision that comes after knowing what you need to capture, not before.
Business metrics: the Timer isn't free, you have to decide the labels
Spring Boot already exposes http.server.requests for every endpoint automatically, tagged with HTTP method, route and status code. That's enough to know an endpoint is generally slow, but it doesn't tell you why. When the reason for slowness depends on a business condition -for instance, whether checkout requires extra fraud verification or not- that condition needs its own label, not something folded into the endpoint's generic duration.
@Service
public class CheckoutService {
private final MeterRegistry registry;
public CheckoutService(MeterRegistry registry) {
this.registry = registry;
}
public CheckoutResult process(Order order) {
Timer.Sample sample = Timer.start(registry);
String fraudPath = order.requiresFraudCheck() ? "with_fraud_check" : "standard";
try {
CheckoutResult result = doProcess(order);
sample.stop(Timer.builder("checkout.process.duration")
.tag("fraud_path", fraudPath)
.tag("outcome", result.outcome().name())
.register(registry));
return result;
} catch (Exception ex) {
sample.stop(Timer.builder("checkout.process.duration")
.tag("fraud_path", fraudPath)
.tag("outcome", "error")
.register(registry));
throw ex;
}
}
}
With that metric, a dashboard can separate the duration percentile of checkouts that go through fraud verification from those that don't, instead of showing a single blended percentile that doesn't say which of the two paths the slow tail belongs to. That separation is the difference between a dashboard that confirms something is wrong in general and one that points to which part of the flow to check first.
What didn't work: labels with uncontrolled cardinality
The first version of this metric included the customer identifier as a label, with the idea of being able to filter by a specific customer if someone reported a particular issue:
Timer.builder("checkout.process.duration")
.tag("customer_id", order.customerId())
.tag("fraud_path", fraudPath)
.register(registry);
Every distinct value of a label in Micrometer creates a new time series in the metrics backend. With thousands of distinct customers, that single line multiplied the number of active series by an enormous factor, and the collector started dropping metrics and consuming increasing amounts of memory until we restarted it. We reverted that label within days. The customer identifier still exists, but as a distributed trace attribute, not a metric label: OpenTelemetry doesn't fan out series per trace attribute the same way, so the high-cardinality cost doesn't apply there, and to find a specific customer's case it's enough to search their traces, not to keep a separate metric per customer.
Span currentSpan = tracer.currentSpan();
if (currentSpan != null) {
currentSpan.tag("customer.id", order.customerId());
}
The rule we settled on is simple to state even if it's sometimes hard to follow: a metric label can only have a bounded, known set of possible values -a result code, a boolean, an external provider name from a fixed list. Any value that grows with the number of users, orders or business entities goes into a trace, never into a metric label.
Connecting metrics and traces with exemplars
The step that closes the loop is being able to see a spike in the p99 of checkout.process.duration and jump straight to a real trace that fell into that spike, without having to search for it by hand over a time range. That's what an exemplar does: a trace identifier attached to a point in the exported metric.
management:
metrics:
distribution:
percentiles-histogram:
checkout.process.duration: true
With the histogram enabled and the same collector receiving both metrics and traces, the backend can associate a histogram bucket with concrete traces that fell into that duration range, as long as the trace ID is available in context at the moment the metric is recorded -which happens automatically if the metric is recorded on the same thread handling the request, inside the active span.
The rule we apply now
Before adding a new metric we ask ourselves one question: if this metric spikes or drops abnormally, would anyone know what to look at next. A high CPU metric doesn't answer that on its own; a checkout duration metric split by fraud path does, because it points directly at where to look. Infrastructure metrics are still there, because they're needed to know whether the whole system is under pressure, but they stopped being the first place we look when someone asks about one particular slow request. The first place now is the trace of that request and the business metric that groups it with others that hit the same problem.