Cloudfare bug led to sensitive data being leaked and indexed by search engines

A simple missing piece of a C routine led to a buffer overflow and what's techincally known as 'a total disaster'

Written 4 years ago on Feb 24, 2017

Cloudfare, an internet service provider of hosting, DNS, etc were informed last week by a Google employee that their servers were returning web pages that seemed to contain sensitive information dumped into them.

It turns out the issue was a result of a bug in the code that modifies web pages as they move through the Cloudfare system. They use this to provide services to customers that redirect requests, inject code into pages etc - it's all very useful, until it goes wrong.

The bug was a simple missing instruction in the C program that runs this service. The bug was only exposed when a web page moving through the code was malformed - for example a web page with an unclosed script tag.

This led to the program getting confused about where it should be reading from and it 'jumped' out of the memory where the HTML was kept and into memory from other requests/programs that were also running. This means that a broken web page could display a piece of authorisation header information, api keys, content etc from a totally unconnected website, they just happen to both use Cloudfare's services.

The Impact

The impact of this bug should not be underestimated. Some huge brands and services use Cloudfare services and as a result data from those services has been injected into badly written web pages. That in itself isn't a huge concern until you think about two things.... search engine spiders and browser caches. Lots of this data has been cached by search engine crawlers, you can still find it now (I won't post a link). There's also the issue that a lot of it is also stored locally in people's browser caches. That's a huge potential security breach with all kinds of personal and transaction data being dumped out.

What to do

The best thing to do is check whether your are using any of the websites and services listed here: https://github.com/pirate/sites-using-cloudflare/blob/master/README.md - if you are then you should probably change your passwords, API keys etc as a precaution.

Below is a detailed timeline of the incident from Cloudfare themselves.

Detailed Timeline

We are very grateful to our colleagues at Google for contacting us about the problem and working closely with us through its resolution. All of which occurred without any reports that outside parties had identified the issue or exploited it.
All times are UTC.
2017-02-18 0011 Tweet from Tavis Ormandy asking for Cloudflare contact information 
2017-02-18 0032 Cloudflare receives details of bug from Google 
2017-02-18 0040 Cross functional team assembles in San Francisco 
2017-02-18 0119 Email Obfuscation disabled worldwide 
2017-02-18 0122 London team joins 
2017-02-18 0424 Automatic HTTPS Rewrites disabled worldwide 
2017-02-18 0722 Patch implementing kill switch for cf-html parser deployed worldwide
2017-02-20 2159 SAFE_CHAR fix deployed globally
2017-02-21 1803 Automatic HTTPS Rewrites, Server-Side Excludes and Email Obfuscation re-enabled worldwide

Report from Cloudfare here: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

Cloudfare seem to have been completely transparent about this and have sorted it very quickly. Hopefully lessons will be learned here by everyone and the result will be better systems and checks going forward......hopefully.


Article Category Tags

Click one to see related articles hand-written by Cowshed Works

Business

Want to discuss your next web project?

We're happy to host meetings at the Cowshed or via conference call, we're equally happy to come and meet you to discuss your project.

Just drop us a line and we'll get it booked in and get the ball rolling.

Cowshed Works

What our clients say...