Spectrum is out right now, so the LinkSwarm is going to be extra late.
I mean, it was already going to be extra late, but now I have an excuse…
As Jerry Pournelle used to say “This day eaten by locusts.”
So: Open thread!
Archive for the ‘Uncategorized’ Category
Internet Outage
Friday, October 8th, 2021The System is Down…The System Is Down…
Monday, October 4th, 2021Facebook, Instagram and WhatsApp are all down right now. Botched upgrade? Misconfigured router? Expired signing certificate? Who knows? I’m just going to assume its a problem with their latest SuppressAllPostsQuestioningTheHolyVaccineMarrative.yml file. But it’s a reminder of how deeply interconnected all online systems are these days, and how many different things can go wrong at different layers.
Expect a sudden burst of productivity from American companies.
And just in case you didn’t get the reference:
Edited to add: Additional detail:
Facebook—and apparently all the major services Facebook owns—are down today. We first noticed the problem at about 11:30 am Eastern time, when some Facebook links stopped working. Investigating a bit further showed major DNS failures at Facebook…
DNS—short for Domain Name System—is the service which translates human-readable hostnames (like arstechnica.com) to raw, numeric IP addresses (like 18.221.249.245). Without working DNS, your computer doesn’t know how to get to the servers that host the website you’re looking for.
The problem goes deeper than Facebook’s obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services—which are hosted on Amazon rather than being internal to Facebook’s own network—were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services’ load balancers were reachable, the application servers that should be feeding the load balancers were not.
A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. (BGP—short for Border Gateway Protocol—is the system by which one network figures out the best route to a different network.)
With no BGP routes into Facebook’s network, Facebook’s own DNS servers would be unreachable—as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR.
DNS—short for Domain Name System—is the service which translates human-readable hostnames (like arstechnica.com) to raw, numeric IP addresses (like 18.221.249.245). Without working DNS, your computer doesn’t know how to get to the servers that host the website you’re looking for.
The problem goes deeper than Facebook’s obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services—which are hosted on Amazon rather than being internal to Facebook’s own network—were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services’ load balancers were reachable, the application servers that should be feeding the load balancers were not.
A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. (BGP—short for Border Gateway Protocol—is the system by which one network figures out the best route to a different network.)
With no BGP routes into Facebook’s network, Facebook’s own DNS servers would be unreachable—as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR.
Speculation is that Facebook engineers have locked themselves out of their own network, meaning someone with physical access to the servers will have to fix things…
Edited to add 2: Krebs offers more details:
Facebook and its sister properties Instagram and WhatsApp are suffering from ongoing, global outages. We don’t yet know why this happened, but the how is clear: Earlier this morning, something inside Facebook caused the company to revoke key digital records that tell computers and other Internet-enabled devices how to find these destinations online.
Doug Madory is director of internet analysis at Kentik, a San Francisco-based network monitoring company. Madory said at approximately 11:39 a.m. ET today (15:39 UTC), someone at Facebook caused an update to be made to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which Internet service providers of the world share information about which providers are responsible for routing Internet traffic to which specific groups of Internet addresses.
In simpler terms, sometime this morning Facebook took away the map telling the world’s computers how to find its various online properties. As a result, when one types Facebook.com into a web browser, the browser has no idea where to find Facebook.com, and so returns an error page.
In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded.
“Not only are Facebook’s services and apps down for the public, its internal tools and communications platforms, including Workplace, are out as well,” New York Times tech reporter Ryan Mac tweeted. “No one can do any work. Several people I’ve talked to said this is the equivalent of a ‘snow day’ at the company.”
Developing…
Edited to add 3: Seeing reports that Gmail is down for some people. It’s not down for me. I just tested and it’s working fine.
Updated to add 4: Facebook appears to be back up, but is way wonky…
Technical Difficulties
Tuesday, September 7th, 2021My blogs were down this morning (both BattleSwarm and https://www.lawrenceperson.com/), and some features (like tag auto-population) still don’t seem to be working. No explanation from BlueHost for the outage except “There are too many process, I killed all the process but it keeps re-generating,” which is a symptom of the issue, not the issue itself.
Anyway, the upshot is no real blog post this morning, so instead enjoy a compilation of random Golden Retriever videos.
Why The Hell Does Twitter Need Access To Network Devices?
Tuesday, August 24th, 2021What fresh hell is this?
Why on earth would Twitter, of all things, need access to my network devices from my iPhone?
The answer, of course, is to sell my data to advertisers. But why would I want Twitter to have access to that data?
I wouldn’t. And there are precious few scenarios I can imagine where anyone would want to give Twitter that access.
Evidently this has been going on for a while, but it seems like a huge security hole to me, and another vector by which a malicious attacker could launch a cross-app privilege escalation attack.
Happy Independence Day!
Sunday, July 4th, 2021I hope you’re enjoying America’s 245th Birthday today. In celebration, enjoy some vaguely-related videos. Here are some fireworks fails, complete with lots of cussing:
Speaking of profanity, here’s “America Fuck Yeah” from Team America: World Police.
(This version is funnier, but not embedable.)
And here’s “America” from West Side Story:
(Did you know that Stephen Spielberg has a remake due out this year? I don’t think that’s a good idea…)
Also, open thread. Talk amongst yourselves, and feel free to share stories of America’s greatness.
LinkSwarm Held Hostage: Week Two
Friday, July 2nd, 2021I got my Mac back on Monday with the battery, bottom and trackpad replaced, and everything works fine. But between contractors redoing my bathroom and cleaning ladies coming in today to get ready for July 4, this week, has, if anything, been even busier than last week. So enjoy this low-calorie LinkSwarm substitute featuring only the crowd-pleasing LinkSwarm favorites: The Babylon Bee, dogs, guns and explosions.
WATCH: Sky9 was overhead when police attempted to detonate a stash of illegal fireworks that ended in an explosion in the 700 block of East 27th Street in South L.A. https://t.co/KpWiAEoyqv pic.twitter.com/UbIXLDoZjh
— CBS Los Angeles (@CBSLA) July 1, 2021
Oh my goodness. pic.twitter.com/h8tMCDqWxD
— Janice Dean (@JaniceDean) June 21, 2021
— Mike Baron (@BloodyRedBaron) June 21, 2021
There Is No LinkSwarm, Only Zuul
Friday, June 25th, 2021A few days ago, I found it strange that my MacBookPro lid wouldn’t close properly. Then it seemed to spontaneously reboot. Then yesterday I noticed this:
That, dear reader, is the bottom of my Mac bowed outward due to the swollen battery issue. A trip to the Apple store ensued. They’re going to replace the battery, and possibly the bottom of the case, due to the issue, free of charge, despite it being out of warranty. Plus the Mac was still working, so I was able to back everything up just in case they need to swap out the drive. And I have a backup MacBookPro, a lovely parting gift from my previous employer, so I’m not dead in the water. So all I lost was two hours.
But let me tell you: two hours is a lot when you’re constantly running the needle into the red. This week I’m finishing up cranking on a project at work that ships today, I have to get started on a new Lame Excuse Books catalog that’s already more than a month later than I usually send it out, I’m trying to get up to full speed on my summer exercise regime, etc. Frankly, I need a break. So I’m using the repair time to enjoy a blogging semi-hiatus for a week or so.
But a semi-hiatus for me is going to look like a lesser mortal’s blog at full tilt. I think I’ve managed to put up something here every day for probably over 700 days (maybe 1,000 or more), and I want to keep that streak alive, but expect something more like a single video or tweet each day rather than full-blown posts. Just don’t expect long, detailed posts for a bit.
I need to use the time to get some work done around the house, and clean up in advance of a July 4th gathering.
Next week: Maybe two LinkSwarms. We’ll see how it goes…
Bosch’s New Fab And The State of European Semiconductors
Wednesday, June 23rd, 2021This is interesting for what it says about the state of European semiconductor manufacturing:
German tech giant Bosch has opened the doors to a new semiconductor manufacturing facility in Dresden, which it hopes will plug the significant hole currently plaguing automotive supply chains.
The site – which was estimated to cost the company in excess of €1 billion (AU$1.57 billion) to build – will reportedly begin producing much-needed chips for vehicles by early September this year. It is so far unclear how the product will be distributed, and if German manufacturers – including Volkswagen, BMW, and Mercedes-Benz – will be given priority in the queue.
“The new [factory] is the single largest investment in the company’s history. This cannot be stressed too much. Its size and additional production capacity alone are impressive. The very latest methods of data-driven continuous improvement in production make the Dresden plant a smart factory,” said German Chancellor Angela Merkel.
“To put it another way: in this plant, natural and artificial intelligence have joined forces with the internet of things to form a productive symbiosis.”
Here Alte Jungfer Merkel is no doubt repeating some generic technomarketingspeak handed to her by an aide, as I suspect Merkel is as ignorant of current developments in artificial intelligence as I am of the songs she sang in the Freie Deutsche Jugend.
Here’s a Deutsche Weil English video segment on the fab, for which the headline says “Europe aims for independence from Asia”:
How cutting edge will the Bosch fab be?
The 300mm fab has been under construction since June 2018 and as recently as October 2019 it was stated that the Dresden wafer fab was expected to start operations in spring 2020…
The fab makes use of 5G communications and artificial intelligence [there’s that phrase again -LP] for extensive automation and will be used for the manufacture of power semiconductors and ASICs for automotive applications down to a minimum geometry of 65nm, according to a company spokesperson.
(record scratch)
300mm is industry standard, and Dresden makes sense, as Infineon, Global Foundries and X-FAB already have fabs there, so there’s a skilled labor pool to draw on. The strange thing here is spending $1.2 building a new 65nm fab, since that’s about seven process node generations behind the cutting edge. 65nm was cutting edge way back in 2005.
You don’t need cutting edge process technology to build automotive integrated circuits, which tend to use older process nodes, but 65nm doesn’t give you much headroom for the long haul. It also doesn’t really do much to “compete with Asia,” since it’s hopelessly behind not only leaders TSMC and Samsung, but also about half the fabs run by Chinese companies SMIC and Unigroup. It’s hard to see investing $1 billion in building a older technology fab as opposed to contracting out production to foundries. (Of course, foundry capacity is highly constrained right now, and maybe Bosch leadership was able to see that coming when they broke ground in 2018.)
Could it still turn out to be a decent investment? Possibly. In the short haul it will relieve the current capacity crunch and let Bosch gain share from competitors who can’t book foundry production. Plus semiconductor manufacturing seems like the sort of thing European governments like to subsidize. And indeed, the DW piece notes that “The German government also invested in the plant.” (It also wouldn’t surprise me to find out that various inscrutable, multi-acronym EU agencies unknown to voters in Barcelona and Gdansk are also kicking in money.) Fueled by taxpayer money, building a 65nm fab could still be a profitable proposition for Bosch.
But fabs close all the time, even 300mm fabs. Building a fab dedicated to such an old process node makes profitability a much more challenging proposition, especially given the next (inevitable) industry downturn. You only have so many years before everything is obsolete. Bosch has given themselves a very short runway to profitability for this investment.
Back in the 1990s, Europe was competitive with American and Japanese semiconductor companies, with fabs built in the UK, Ireland, France, Germany, the Netherlands and even Italy. Today, of those countries, only Ireland (Intel) and Germany have any 300mm fabs at all, despite having a highly educated workforce and modern technological infrastructure. (Russia evidently has one as well, owned by Crocus Nano Electronics (about which I know next to nothing), also running 65nm, that started production in 2015.) Europe’s semiconductor industry has been passed not only by Taiwan and South Korea, but also by China.
While Europe was integrating, its semiconductor manufacturing capacity was stagnating. Highly integrated, yes, but also highly regulated and highly unionized.
As I’ve said before, building cutting edge fabs is a very expensive game to play. Collectively, the world has decided that Europe is not a suitable location for one.
A Big Chunk Of The Internet Went Down This Morning
Tuesday, June 8th, 2021While you were asleep this morning, a big chunk of the Internet briefly went down:
Among those affected are Amazon, Twitch, Reddit, The Verge, The Guardian, ZDnet, The New York Times, Freetrade, The Financial Times, Pinterest, Kickstarter, Ebay, The Telegraph, CNN, and Imgur. Google searches are also partially impacted, as is the Google Cloud Platform. While Twitter is up, its emoji platform is offline.
The issue has been traced back to content delivery network Fastly, which is down. The company runs an Edge cloud between companies’ data centers and the end user, reducing latency, protecting from DDoS attacks, and helping them handle traffic spikes.
Fastly is a content delivery network (CDN), an intermediary that brings data closer to Internet end users so their interactions don’t need to go all the way back to the company’s central servers. This is the market Akamai pioneered, and other companies in the space include CDNetwork and Cloudflare. “Edge computing” is a sort of catch-all term for intermediary cloud services that became one of those buzzwords that VC companies threw money at about two years ago.
With Amazon down, a lot of people jumped to the conclusion that AWS, Amazon’s 800 pound cloud service gorilla, was experience an outage, but it turned out to be Fastly, who evidently fixed the problem at 10:57 UTC (5:57 AM CDT):
“The issue has been identified and a fix has been applied. Customers may experience increased origin load as global services return.” On Twitter, the company added: “We identified a service configuration that triggered disruptions across our POPs globally and have disabled that configuration. Our global network is coming back online.”
The modern Internet is decentralized, widely distributed and pretty efficient, but its very decentralized nature means that there are more moving parts to break, and also more attack surfaces for hackers to exploit. Delivering rich content over the Internet (be it text, images, video or shopping) usually involves dozens, if not hundred of software pieces, protocols, companies, etc. for every web page served up. Any of them can go down. Network engineers design in as much redundancy as possible, but there’s only so much you can do. I worked for a company in 2020 whose computer testing lab went down because antifa rioters in Minneapolis physically destroyed a fiber optic cable.
All I can tell you is to keep multiple rotating backups of your most valuable data, because anything that can go wrong eventually will…