What Is a 520 Status Code and How Do You Avoid It?

TL;DR

A 520 status code is Cloudflare's catch-all error: the origin server returned an empty, malformed, or unexpected response that Cloudflare could not pass along. In web scraping, 520s spike when aggressive request rates overload the origin or when anti-bot rules crash the response midway, so the fix is slower pacing, realistic headers, and retries with backoff.

What 520 Actually Tells You

520 is not in the HTTP standard. It belongs to Cloudflare's private 52x family, and it means "the origin behind us returned something we cannot interpret": an empty reply, a crashed connection, malformed headers, or a response that exceeded internal limits. Two facts matter for scrapers:

• The request got through Cloudflare. A 520 is not a challenge page; the edge accepted you and the failure happened at the origin.
• The origin is struggling or misbehaving, and scraper traffic is often the reason.

Bursts of concurrent requests against a small origin are one of the most common causes of 520 waves. If your own crawl triggered them, no retry strategy fixes a problem you are actively creating.

How to Avoid 520s

Slow down first. Cut concurrency and add delays between requests. If 520s drop when you halve your rate, the diagnosis is complete: you were overloading the origin. Respect it, because a crashed origin serves nobody, including you.

Send realistic requests. Some origin-side security modules kill connections for suspicious headers, producing 520s instead of clean 403s. Browser-consistent headers remove that trigger.

Retry with patience. Genuine origin hiccups are transient. Retry a small number of times with growing delays:

for attempt in range(4):
    response = requests.get(url, headers=headers, timeout=(5, 30))
    if response.status_code != 520:
        break
    time.sleep(2 ** attempt * 5)  # 5s, 10s, 20s, 40s

Check whether it is only you. If the site 520s in a normal browser too, the origin is down and the correct move is to pause the job and come back later.

The Rest of the 52x Family

520 travels with siblings that pinpoint the failure more precisely: 521, 522, and 525 mean the origin refused the connection, timed out, or failed the SSL handshake respectively. Logging them separately from 520 shortens every future debugging session.

Key Takeaways

• 520 means Cloudflare accepted you but the origin returned garbage or nothing.
• Your own request rate is suspect number one; halve it and measure.
• Retry transient 520s with exponential delays, and stop if the site is down for browsers too.

How ScrapeGraphAI Handles This

ScrapeGraphAI paces traffic per target, retries transient 52x errors with backoff across multiple fetch providers, and marks persistent origin failures explicitly. Point the scrape endpoint at a Cloudflare-fronted site and the platform handles the edge cases that produce 520s.