Building Resilient HTTP Clients with Polly: Retry and Circuit Breaker Patterns

In today's interconnected world, many applications rely on external APIs and services. However, network calls can fail due to various reasons such as temporary network issues, server overloads, or maintenance downtime. To build robust and fault-tolerant applications, it's crucial to implement resilience patterns. In this blog post, we'll explore how to use Polly, a .NET resilience and transient-fault-handling library, to create a resilient HTTP client using retry and circuit breaker patterns.

Understanding Polly Retry Handler

The retry pattern is a simple yet effective way to handle transient failures. When a request fails, instead of immediately throwing an exception, the application waits for a short period and then retries the request. This can often resolve issues caused by temporary network glitches or brief service unavailability.

Polly's retry handler allows you to specify:

1. The number of retry attempts

2. The delay between retries (which can be fixed or can increase with each attempt)

3. The types of exceptions or return results that should trigger a retry

In our example, we're using an exponential backoff strategy, where the delay increases exponentially with each retry attempt.

Understanding Circuit Breaker Policy

The circuit breaker pattern prevents an application from repeatedly trying to execute an operation that's likely to fail. It's like an electrical circuit breaker: when a problem is detected, it "trips" and stops the flow to protect the system.

The circuit breaker has three states:

1. Closed: When everything is normal, the circuit is closed, and requests are allowed through.

2. Open: If the number of failures exceeds a threshold, the circuit opens, and requests are not allowed through for a certain period.

3. Half-Open: After the timeout period, the circuit switches to half-open, allowing a limited number of requests through to test if the problem still exists.

This pattern is particularly useful for preventing cascading failures in distributed systems.

Implementing Resilient HTTP Clients

Let's break down the code and see how we're implementing these patterns:

We need to install following nuget packages:

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly

public static void AddResilientApiClient(this IServiceCollection services, ApiClientConfiguration apiClientConfiguration)
{
    services.AddHttpClient("ResilientHttpClient")
        .AddPolicyHandler(GetRetryPolicy(apiClientConfiguration))
        .AddPolicyHandler(GetCircuitBreakerPolicy(apiClientConfiguration));
}

This method extends IServiceCollection to add a named HttpClient with both retry and circuit breaker policies.

Retry Policy

private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy(ApiClientConfiguration config)
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(
            config.RetryCount,
            retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt) * config.RetryAttemptInSeconds)
        );
}

This method creates a retry policy that:

• Handles transient HTTP errors (NetworkError, 5xx, and 408 status codes)

• Retries for a specified number of times

• Uses exponential backoff for the delay between retries

Circuit Breaker Policy

private static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy(ApiClientConfiguration config)
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .CircuitBreakerAsync(
            config.HandledEventsAllowedBeforeBreaking,
            TimeSpan.FromSeconds(config.DurationOfBreakInSeconds)
        );
}

This method creates a circuit breaker policy that:

• Handles the same transient HTTP errors as the retry policy

• Opens the circuit after a specified number of consecutive failures

• Keeps the circuit open for a specified duration before allowing a trial request

Usage

To use this resilient HTTP client in your application, you would typically:

1. Define your `ApiClientConfiguration` with appropriate values.

2. Call `AddResilientApiClient` in your service configuration.

3. Inject and use the named `HttpClient` in your services.

For example:

public class ApiClientConfiguration
{
    public int RetryCount { get; set; }
    public int RetryAttemptInSeconds { get; set; }
    public int HandledEventsAllowedBeforeBreaking { get; set; }
    public int DurationOfBreakInSeconds { get; set; }
}

services.AddResilientApiClient(new ApiClientConfiguration
{
    RetryCount = 3,
    RetryAttemptInSeconds = 2,
    HandledEventsAllowedBeforeBreaking = 5,
    DurationOfBreakInSeconds = 30
});

Then, in your service:

public class MyApiService

{
    private readonly HttpClient _httpClient;
    public MyApiService(IHttpClientFactory httpClientFactory)
    {
        _httpClient = httpClientFactory.CreateClient("ResilientHttpClient");
    }
    Use _httpClient for your API calls
}

Conclusion

By implementing retry and circuit breaker patterns with Polly, we can create more resilient applications that can better handle transient failures and prevent cascading failures in distributed systems. This approach not only improves the reliability of our applications but also enhances the user experience by reducing the impact of temporary issues.

Remember, while these patterns can significantly improve your application's resilience, they're not a silver bullet. It's important to combine them with other best practices such as logging, monitoring, and alerting to create truly robust systems.