A Big Chunk Of The Internet Went Down This Morning

While you were asleep this morning, a big chunk of the Internet briefly went down:

Among those affected are Amazon, Twitch, Reddit, The Verge, The Guardian, ZDnet, The New York Times, Freetrade, The Financial Times, Pinterest, Kickstarter, Ebay, The Telegraph, CNN, and Imgur. Google searches are also partially impacted, as is the Google Cloud Platform. While Twitter is up, its emoji platform is offline.

The issue has been traced back to content delivery network Fastly, which is down. The company runs an Edge cloud between companies’ data centers and the end user, reducing latency, protecting from DDoS attacks, and helping them handle traffic spikes.

Fastly is a content delivery network (CDN), an intermediary that brings data closer to Internet end users so their interactions don’t need to go all the way back to the company’s central servers. This is the market Akamai pioneered, and other companies in the space include CDNetwork and Cloudflare. “Edge computing” is a sort of catch-all term for intermediary cloud services that became one of those buzzwords that VC companies threw money at about two years ago.

With Amazon down, a lot of people jumped to the conclusion that AWS, Amazon’s 800 pound cloud service gorilla, was experience an outage, but it turned out to be Fastly, who evidently fixed the problem at 10:57 UTC (5:57 AM CDT):

“The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.” On Twitter, the company added: “We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration. Our global network is coming back online.”

The modern Internet is decentralized, widely distributed and pretty efficient, but its very decentralized nature means that there are more moving parts to break, and also more attack surfaces for hackers to exploit. Delivering rich content over the Internet (be it text, images, video or shopping) usually involves dozens, if not hundred of software pieces, protocols, companies, etc. for every web page served up. Any of them can go down. Network engineers design in as much redundancy as possible, but there’s only so much you can do. I worked for a company in 2020 whose computer testing lab went down because antifa rioters in Minneapolis physically destroyed a fiber optic cable.

All I can tell you is to keep multiple rotating backups of your most valuable data, because anything that can go wrong eventually will…

Tags: , , , , , ,

6 Responses to “A Big Chunk Of The Internet Went Down This Morning”

  1. ant7 says:

    “more attack surfaces for hackers to exploit”

    and the chinese have literally 100 hackers assigned to specialize in each one and to be ready to shut it down when given the word.

  2. Ygolonac says:

    “Experts baffled and concerned after Internet falls over! But first, let’s all watch this funny video of a cat wandering into an open server room…”

  3. Howard says:

    Bah, nation-state hacking? ’tis but an annoyance while we try to watch the public execution of “B” from “LGBT” for implying there are 2 genders …

  4. ant7 says:

    “implying there are 2 genders”

    “B” only means they have two genders, it doesn’t say which ones ….

  5. Kurt says:

    Oddly enough, it wasn’t hacking – it was a valid customer configuration change that exposed/triggered a bug.

    Weird, huh?

    Kurt

  6. ant7 says:

    “it wasn’t hacking – it was a valid customer configuration change”

    imagine what a hack would do. you can bet the chinese are imagining a whole lot.

Leave a Reply