Cloudflare 520 Errors Explained and Investigated

What is a Cloudflare 520 Error?

The Cloudflare 520 error code is similar to an HTTP server error code. The Cloudflare worker returns these errors when they’re unable to complete their task. There is no HTTP server error code 520, it’s a specific Cloudflare error code, here’s the definition.

520 Web Server Returned an Unknown Error. The origin server returned an empty , unknown, or unexplained response to Cloudflare

https://support.cloudflare.com/hc/en-us/articles/115003011431-Troubleshooting-Cloudflare-5XX-errors#520error

What causes a Cloudflare 520 Error?

There is many reasons why, the Cloudflare support article talks about the main points as to why a 520 error occurs.

• Origin web server application crashes
• Cloudflare IPs not allowed at your origin
• Headers exceeding 16 KB (typically due to too many cookies)
• An empty response from the origin web server that lacks an HTTP status code or response body
• Missing response headers or origin web server not returning proper HTTP error responses
• 520 errors are prevalent with certain PHP applications that crash the origin web server.

You can read more about Cloudflare’s 520 Error on their support site. But it’s not too helpful.

How can I troubleshoot a Cloudflare 520 Error?

Cloudflare’s Support Article Troubleshooting Steps

If 520 errors continue after contacting your hosting provider or site administrator, provide the following information to Cloudflare Support:

• Full URL(s) of the resource requested when the error occurred
• Cloudflare cf-ray from the 520 error message
• Output from http://www.example.com/cdn-cgi/trace (replace www.example.com with your hostname and domain where the 520 error occurred)
• Two HAR files:
  • one with Cloudflare enabled on your website, and
  • the other with Cloudflare temporarily disabled.

Cloudflare Support

If you’re on a free plan, you won’t have access to Cloudflare support. And you’ll have to post your request on the community forums with your RayID. If you have a pro plan on your domain, then you’ll be able to submit a ticket.

Cloudflare Error Logs

This is directly from a support rep.

“I understand that you wish us to investigate RayID 680cd166037c2f79, but we cannot since we only keep sampled logs for zones that are below Enterprise level. Have you checked these most common causes for that type of error yet?”

If you’re on the Pro or Business plan, you can provide the RayID to Cloudflare support and they will provide you with error logs or at least review them and give you further details.

Tracking 520 Errors

The biggest issue some people have is tracking the 520 errors as they’re not always reproducible, and sometimes a client of theirs saw the issue but didn’t take a screenshot or theirs no RayID to provide to support.

This is where you could use something like Logflare.app which can be added as a Cloudflare App to your site. The free plan of Logflare allows for 5 requests per second, so if you’re adding it to a busy site you will need to upgrade to the $15/month plan. Once you get set up, you can then filter our 520 errors to a separate endpoint and have that endpoint alert on Slack or email.

There is also the option of using Sentry.io but you would need to code up a Cloudflare worker to get the data into Sentry. You can also self-host Sentry as it’s open-sourced.

Potential Causes

Network Issues

If there is a network issue between Cloudflare and your origin (server hosting your site). Then this could be a potential cause for the 520 errors.

Provider Network Filtering and DDoS Protection

So now you need to confirm what’s sitting between Cloudflare and your server. Reach out to your provider, for instance, OVH has a DDoS firewall that can block Cloudflare. If your provider has any sort of firewall or filtering service that rate limits or blocks attacks. You have to reach out to them to whitelist Cloudflare’s IP’s or contact their support to investigate why Cloudflare is being blocked.

GridPane WAF

If you’re using 6G/7G/modsec. Simply disable all of it, at least to see if you can stop the 520’s. You can also look at the various logs to see if any errors are coming up and match up the dates and times of the 520 from Cloudflare to the logs. This is where Logflare.app works well. I also use Sentry.io which is a paid service, but you can self-host too and I have an instance for myself to track 520’s using a Cloudflare custom worker as well as 404’s.

Cloudflare Argo Tunnels

You could also just build an Argo tunnel directly to your server and potentially bypass some of these issues. Potentially… the tunnel still goes through your provider’s network. But it might help the situation or provide logs or debugging on 520s.

https://blog.cloudflare.com/argo-tunnels-that-live-forever/

It’s decently easy, I stood one up for access to Cypberpanels admin interface from localhost, and put up a login page using Cloudflare’s Zerotrust setup.

Allowing Cloudflare only Firewall Rules (Test)

Just a note. Technically you should be firewalling all traffic to your server’s 80/443 to only allow Cloudflare’s IP ranges. Since nothing else should be hitting these ports that don’t filter through Cloudflare. This might sound silly but in the case of OVH and their automated alerts of a DDoS attack. If you did put these firewall rules in place and got a notification of a DDoS attack, you know it’s going to be Cloudflare that was blocked and you could send them a request to look into it and whitelist Cloudflare.

Application Issue / PHP

• Checking php-fpm logs to see if anything bad happened. You could also look at Newrelic.

Linux OS

• Check iptables rules
• Check dmesg
• Check /var/log/syslog for any strange errors.

Web Server

It’s possible your site’s web server is causing an issue resulting in Cloudflare generating a 520 error. This could be from SSL, HTTP Headers, or filtering.

Cloudflare HTTP Header Limit 16KB

There is a limit on the HTTP Header that Cloudflare will accept and proxy, it’s 16kb. Which includes all HTTP headers, such as status codes and cookies.

https://developers.cloudflare.com/workers/platform/limits

Nginx Header default is 8KB

If you’re running Nginx, make sure you confirm that you have increased the default 8KB to 16KB. You would be getting a different error however from Nginx if this was the case. You can increase the default 8KB to 16KB and match Cloudflare.

https://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers

Too Many Cookies

Whenever you have a cookie store in your browser for a specific site, the cookies are sent automatically upon each visit to the site. If you have too many cookies stored in your browser, it can exceed the 16KB limit. However, the 16KB limit isn’t just counting cookies, it’s counting all headers sent.

The following article talks about too many cookies on a website and has some interesting information about cookies and HTTP headers.

https://hajekj.net/2017/10/30/the-dangers-of-too-many-cookies-on-a-website/

If you’re using Chrome and want to see what Cookies you’re sending to the site with the 520 error. You can do so within the Chrome Console via Application Tab->Storage->Cookies. Here’s an example

If you want to edit your cookies or set a cookie, you can do so with the Chrome extension Cookie Editor which provides a great way to see current cookies, delete or edit them or create new cookies.

Other Information

Cloudflare Support Interaction

During my communication with Cloudflare support, they sent the following log based on my pro plan site. As you can see there is quite a bit of information, granted this site wasn’t a site I was having issues to my knowledge.

You can see there is an arrow next to each error message, I’m assuming this would provide more details about the request.

Cloudflare Web Traffic Analytics (Pro Only)

It turns out that Cloudflare tracks 520 errors within their new Web Traffic Analytics feature.

However, it’s a Pro-only feature, which isn’t great for those under free plans.

Support Threads on FB and Communities

Issue #1 – 520 Errors Backend Error

This person got a response from Cloudflare

Thank you for contacting Cloudflare Support. Sorry to hear that you are experiencing some difficulties here.

A 520 error happens when your origin returns a response that is unexpected.

I went ahead and checked our logs for the past 24 hours and was able to observe the following 520 errors.


I ran some tests on my side from an example location on the most common URL generating this error but was unable to observe this error which means this is likely an intermittent problem.

< HTTP/2 200 
< date: Sun, 06 Feb 2022 07:39:06 GMT
When reviewing the responses from your origin via our logs I was however able to observe the following error.

upstream prematurely closed connection while reading upstream
This indicates that your origin web server was having issues during this period which caused us to generate those 520 errors. I recommend checking with your hosting provider/system administrator and forwarding this particular response for more information on what may have caused this.

This indicates that your origin web server was having issues during this period which caused us to generate those 520 errors. I recommend checking with your hosting provider/system administrator and forwarding this particular response for more information on what may have caused this.

This is good, your origin closed the connection. So most likely something between Cloudflare and your server closed the connection, and it was prematurely and not due to a timeout. This is good because it’s not an SSL issue now. WRONG. We don’t know what “prematurely” means, this is why I’ve been fighting with Cloudflare to categorize these errors. Was it an SSL issue with verifying the certificate, was it a bad header returned? Or was there a header sent from the client to the origin that caused an issue?

It’s hard because Cloudflare would have to have turned on debugging to see the entire connection stand up and down. It would be great if they just provided the HTTP headers between the hosts.

Issue #1 Update

The person having this issue continued with Cloudflare support and found that the issue was related to a large number of cookies being sent. Removing those cookies resolved the issues.

Upates

• 02-16-2022 – Fixed broken images, moved content around and provided more content on HTTP headers.
• 02-08-2022 – Added network troubleshooting and support case about backend unusable.