Horizontal Pod Autoscaler with Custom Metrics: Why CPU Isn't Always the Right Signal to Scale On

The default Horizontal Pod Autoscaler scales on CPU or memory, read through metrics-server. For a service that handles each request synchronously and whose work is mostly computation, that signal makes sense: more traffic means more CPU burned, and scaling on CPU scales in the right direction. The problem shows up when the service's real bottleneck has no direct relationship to how much CPU each pod is using.

We ran into this with a worker that consumes messages from a queue and calls a third-party service for each one. The worker spends most of its time waiting on that external service's response, not computing anything. When the external provider slows down, the queue starts piling up unprocessed messages, but each individual pod keeps reporting low CPU usage because it is, quite literally, waiting, not working. The HPA, watching only CPU, had no reason to scale. The result was a queue growing out of control while the autoscaler sat there looking at a flat CPU graph.

Why metrics-server isn't enough

metrics-server exposes only CPU and memory, aggregated from kubelet via cAdvisor. It has no way of knowing how many messages sit in a RabbitMQ queue, what a downstream endpoint's p99 latency is, or how many active connections a pod is holding. Those signals exist, but they live elsewhere: in Prometheus, if the application exposes a /metrics endpoint, or inside the queueing system itself.

The HPA, though, doesn't talk to Prometheus directly. It talks to three Kubernetes APIs: metrics.k8s.io for resource metrics (what metrics-server covers), custom.metrics.k8s.io for metrics tied to a Kubernetes object (pods, services), and external.metrics.k8s.io for metrics that aren't tied to any object in the cluster at all, like the size of an external queue. Prometheus Adapter is the piece that translates PromQL queries into responses those last two APIs can return, so the HPA can consume them just like any other resource metric.

The full picture

The flow looks like this: the application exposes Prometheus-formatted metrics (or an exporter does it on its behalf, like the RabbitMQ exporter), Prometheus scrapes them on its usual interval, Prometheus Adapter queries Prometheus with a PromQL rule every time the HPA asks for that metric, and the HPA decides how many replicas it needs based on that answer.

That means there are two new pieces to maintain: the Prometheus Adapter rule configuration, which defines which PromQL query corresponds to each metric name the HPA will request, and the HPA manifest itself, which no longer points at cpu but at an external or custom metric under that name.

Configuring Prometheus Adapter

The configuration lives in a ConfigMap that Prometheus Adapter reads on startup. To expose a RabbitMQ queue's length as an external metric:

apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules: []
    externalRules:
      - seriesQuery: 'rabbitmq_queue_messages_ready{queue="pedidos-pendientes"}'
        resources:
          overrides:
            namespace:
              resource: "namespace"
        name:
          matches: "rabbitmq_queue_messages_ready"
          as: "pedidos_pendientes_en_cola"
        metricsQuery: 'avg(rabbitmq_queue_messages_ready{queue="pedidos-pendientes"})'

The field that actually matters here is metricsQuery: the PromQL query Prometheus Adapter runs every time the HPA asks for this metric. We used avg instead of the raw value on purpose, and that wasn't the first version we tried.

What didn't work: raw queue depth with no smoothing

The first version of this rule used the instantaneous value of rabbitmq_queue_messages_ready with no aggregation at all. The HPA polled that metric every 15 seconds — the controller's default sync interval — and scaled up or down based on that single point-in-time reading. The result was a jumpy autoscaler: the queue rose and fell in short bursts as batches of messages arrived and got drained quickly, and the HPA reacted to every burst by adding replicas that, fifteen seconds later, were no longer needed.

That produced scale-flapping: pods spinning up and getting torn down in cycles under two minutes, each one's startup time wasted because by the time the new pod was ready to consume messages, the burst that triggered it had already been drained by the existing workers. Two changes fixed it. The first was wrapping the query in avg() to smooth the point-in-time reading, though with a single time series the real fix came from the second change: setting behavior.scaleUp.stabilizationWindowSeconds and behavior.scaleDown.stabilizationWindowSeconds on the HPA itself, which forces the controller to look at a window of recent decisions instead of reacting to the last isolated reading.

The HPA manifest

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-pedidos
  namespace: pedidos
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: worker-pedidos
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: pedidos_pendientes_en_cola
          selector:
            matchLabels:
              namespace: pedidos
        target:
          type: AverageValue
          averageValue: "30"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 4
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

The asymmetry between scaleUp and scaleDown is deliberate. Scaling up fast is cheap: one extra pod doing little work costs far less than a queue growing out of control. Scaling down fast, on the other hand, is exactly the pattern that produced the flapping described above, so we gave scale-down a five-minute window before it considers cutting replicas, giving the real queue time to settle at a stable level before the controller decides those workers are no longer needed.

The target.type: AverageValue set to 30 means the HPA tries to keep the average number of pending messages per replica at 30: if there are 90 messages queued across 3 active replicas, the HPA calculates that one more replica is needed to bring that average back down. It's the same calculation the HPA uses for CPU — the only difference is that the input metric comes from Prometheus Adapter instead of metrics-server.

When to use Pods versus External

Prometheus Adapter can also expose metrics under custom.metrics.k8s.io instead of external.metrics.k8s.io, matched to the Pods type in the HPA. The practical difference is what the metric is attached to: a Pods type makes sense when the value comes from each individual pod — active connections per pod, the size of an in-memory queue kept by each instance — and Kubernetes automatically averages those values across active replicas. An External type makes sense when the metric doesn't belong to any particular pod, like the length of a shared RabbitMQ queue that producers outside the cluster write to. Trying to force a shared queue's length into a Pods metric doesn't work well: there's no natural way to split that number across pods, because the number was never theirs individually.

What to watch once it's live

Two things only show up in production. The first is latency across the whole chain: Prometheus Adapter doesn't cache aggressively by default, so every HPA query triggers a real query against Prometheus, and if that query is expensive — long ranges, too many series — Prometheus Adapter's response time becomes part of the HPA's reconciliation loop. We kept the queries simple, over a single series, specifically to keep this from becoming a problem.

The second is what happens when Prometheus is down or the metric has no recent data: the HPA doesn't scale on a metric it can't read, and in practice it just stays at whatever replica count it already had. For a queue worker that's tolerable — worst case, it stops scaling further during the window when Prometheus isn't responding — but it's worth knowing before deciding how urgently to treat the Prometheus cluster as a critical piece of the autoscaling path, same as any other.