Circuit Breakers with Resilience4j in Spring Boot: Keeping a Failing Provider From Taking Down the Rest of the System

A typical Spring Boot microservice handles HTTP requests with a bounded thread pool: by default, the embedded Tomcat connector caps out at 200 threads, and every incoming request occupies one of them while it's being processed. As long as processing stays fast, that limit never comes up. The problem shows up when part of that processing involves a network call to an external service, and that service starts responding slowly instead of failing outright. A misconfigured -or simply missing- timeout on the HTTP client means every request that depends on that provider holds its thread hostage, waiting far longer than the system can afford. If that happens often enough, the thread pool runs dry, and so does the service's ability to handle any other endpoint, including ones that have nothing to do with the provider that's down.

This is exactly what we ran into on a checkout service integrated with a third-party fraud-verification provider. The provider had an availability incident and started taking 20 to 30 seconds to respond instead of failing fast with an error. The service's HTTP client had no explicit timeout configured, so every checkout request that went through fraud verification sat blocked waiting for that response. Within minutes, the Tomcat pool filled up with threads waiting on the provider, and the order-status endpoint -which doesn't call the fraud provider at all- started timing out too, simply because there were no free threads left to serve it.

Why a timeout alone isn't enough

The first, reasonable reaction was to add an explicit timeout to the HTTP client:

RestClient restClient = RestClient.builder()
    .requestFactory(ClientHttpRequestFactorySettings.DEFAULTS
        .withConnectTimeout(Duration.ofSeconds(2))
        .withReadTimeout(Duration.ofSeconds(3))
        .build())
    .build();

This bounds the damage of any single request: instead of waiting 30 seconds, each call to the fraud provider now fails after 3. But it doesn't fix the underlying problem, because if the provider is still down, the service keeps attempting the call on every new request, and keeps spending 3 seconds of a thread on every failed attempt. With enough traffic, 3 seconds per request is still enough to drain the pool -just slower than 30 would have been. The timeout keeps a single call from hanging forever, but it doesn't stop the system from repeatedly calling a provider it already knows isn't answering.

What's missing is a mechanism that, after seeing several calls fail in a row, stops attempting the real call for a while and responds immediately with a fallback, giving the provider room to recover and freeing up threads on our own side. That's what a circuit breaker does.

CircuitBreaker and TimeLimiter with Resilience4j

Resilience4j exposes the pattern as a combination of two independent mechanisms decorated together: TimeLimiter, which bounds how long an asynchronous operation is allowed to wait, and CircuitBreaker, which tracks recent failures and decides whether to allow a new attempt at all.

Configuration lives in application.yml:

resilience4j:
  circuitbreaker:
    instances:
      fraudCheck:
        sliding-window-type: COUNT_BASED
        sliding-window-size: 20
        minimum-number-of-calls: 10
        failure-rate-threshold: 50
        wait-duration-in-open-state: 15s
        permitted-number-of-calls-in-half-open-state: 5
  timelimiter:
    instances:
      fraudCheck:
        timeout-duration: 3s
  thread-pool-bulkhead:
    instances:
      fraudCheck:
        max-thread-pool-size: 10
        core-thread-pool-size: 5

sliding-window-size: 20 combined with minimum-number-of-calls: 10 means the circuit breaker evaluates the failure rate over the last 20 calls, but won't make any decision until it has seen at least 10. With failure-rate-threshold: 50, if half or more of those calls fail, the circuit trips open: for the next 15 seconds (wait-duration-in-open-state), no real call is attempted, and every invocation returns the fallback immediately, without spending a single thread waiting on the provider. After that window, the circuit moves to half-open and lets through 5 trial calls; if those succeed, the circuit closes and resumes normal operation, and if they fail again, it opens for another 15 seconds.

The part that's easy to miss is that TimeLimiter needs to operate on a CompletableFuture running on a thread pool separate from the one serving the incoming HTTP request. If the timeout is applied to a blocking call executed on the same request thread, that thread stays occupied until the blocking call actually returns -even though TimeLimiter has already decided, from the outside, that the operation took too long. The thread-pool-bulkhead in the configuration above exists precisely for this: it isolates calls to the fraud provider in their own pool of at most 10 threads, separate from the Tomcat pool serving every other endpoint.

The client code ends up looking like this:

@Service
public class FraudCheckClient {

    private final RestClient restClient;

    public FraudCheckClient(RestClient restClient) {
        this.restClient = restClient;
    }

    @CircuitBreaker(name = "fraudCheck", fallbackMethod = "fallback")
    @TimeLimiter(name = "fraudCheck")
    @Bulkhead(name = "fraudCheck", type = Bulkhead.Type.THREADPOOL)
    public CompletableFuture<FraudVerdict> check(Order order) {
        return CompletableFuture.supplyAsync(() ->
            restClient.post()
                .uri("/verify")
                .body(new FraudCheckRequest(order.customerId(), order.amount()))
                .retrieve()
                .body(FraudVerdict.class)
        );
    }

    private CompletableFuture<FraudVerdict> fallback(Order order, Throwable ex) {
        return CompletableFuture.completedFuture(FraudVerdict.manualReview());
    }
}

The fallback method takes the same signature as the original plus the exception that triggered it -a timeout, a connection error, or the circuit already being open- and here it returns a manual-review verdict instead of automatically approving or rejecting the order. That business decision matters: the fallback can't pretend the provider said everything's fine, because that would silently disable fraud checking without anyone noticing. Flagging the order for manual review is an explicit, visible degradation, not a silent approval.

What didn't work: circuit breaker without a thread bulkhead

The first version we shipped had CircuitBreaker and TimeLimiter in place, but no Bulkhead of type THREADPOOL. TimeLimiter was still running the CompletableFuture via supplyAsync with no explicit executor, which in practice means it runs on the JVM's common ForkJoinPool, shared with any other code in the application that also uses CompletableFuture without specifying an executor. When the fraud provider started failing, the circuit opened correctly and stopped attempting real calls, but while it sat half-open, the trial calls kept competing for threads in the common ForkJoinPool against other asynchronous tasks in the service that had nothing to do with fraud checking, and those tasks started lagging with no apparent connection to the external provider. Isolating calls to the provider in their own dedicated, bounded thread-pool-bulkhead fixed that cross-interference.

The rule we settled on

Not every timeout needs a circuit breaker on top of it. If an external call fails in isolation, occasionally, a simple retry with backoff is enough. A circuit breaker becomes necessary when a provider's failures tend to come in bursts -the whole provider is down or degraded, not just one request- and when continuing to try during that burst eats into a shared resource, like the thread pool that also serves traffic unrelated to that provider. In that scenario, backing off for a while and answering with an explicit fallback isn't a degradation of the service: it's what keeps a problem in a third-party provider from turning into a full outage of your own system.