Retries in Python

Networks are unreliable and applications typically experience transient errors when making outbound requests. At times it’s necessary to retry the request instead of throwing an exception and bailing. Ideally, retries are moved out of process to a job queue (Celerly).

For the times when a jobbing framework isn’t available, the Backoff module by litl comes in super handy. A simple decorator around your method or function, and boom, you have retries.

Here’s what it looks like in action:

@backoff.on_predicate(
    backoff.expo,
    lambda resp: resp.status_code in [429, 500],
    max_value=3,
    jitter=backoff.random_jitter
)
def get_url(url):
    return requests.get(url, timeout=(3.00, 7))

The Backoff readme has examples on how to use the module with requests, but it’s not limited to requests only. Speaking of requests, here’s what the code looks like using requests’ built-in retry functionality:

def get_url(url):
    retry_strategy = Retry(total=3, backoff_factor=2, status_forcelist=(429, 500))
    adapter = HTTPAdapter(max_retries=retry_strategy)
    http = requests.Session()
    http.mount("https://", adapter)

    return http.get(url, timeout=(3.00, 7))

👍