You should backup your data. You should have multiple backups of your data. You should have multiple rotating or rolling backups of your data. You should have multiple rotating or rolling backups of your data saved to different types of backup media. Ideally, you should have multiple rolling or rotating backups of your data saved to different types of backup media including off-site backups.
Evidently, a PBS station didn’t follow any of these rules. Instead they handed their data to a third-party, said “Here, we’ll pay you to store it for us,” and now they’re screwed because the company went defunct.
Yeah, maybe tossing all of the physical media and using the cloud to store your memories wasn’t the best move?
Especially if you’ve got 70 years of archival footage that could disappear at the drop of a hat, as this St. Louis PBS station has learned.
They stored 50 terabytes of data with a cloud storage company, that company went kaput, and now they’re having to fight tooth and nail to get that data back.
Here are more details from tech-news site Tom’s Hardware:
Nine PBS is afraid that it will lose over 70 years of archival materials and programming after its contracted cloud storage vendor has apparently gone out of business. According to Current, the channel used Open Source Storage, or OSS, for storing over 50TB of data — but it suddenly lost access to its data earlier this year. Nine PBS has been working with OSS and its predecessor since 2019 and intended to renew its contract on March 6 of this year. The company never responded and abruptly cut off the station’s data access even though it still had 30 days to retrieve its data after the contract ended.
The station dug a little bit deeper when it discovered that the OSS website no longer existed and was listed as delinquent under the Colorado Secretary of State.
As it turns out, however, OSS used another data company, Iron Mountain, to host the archive.
Iron Mountain is still in business, but since its client was OSS, it is refusing to hand over the files.
The PBS station is now suing.
More from Gizmodo:
Nine PBS is now asking the court to make sure the company preserves the archive while the dispute gets sorted out. A judge granted temporary relief preventing Iron Mountain from deleting or overwriting the data.
As another twist in this confusing mess, OSS was sold after the station first tried to take legal action, then reverted back to the original owners when the new owner claimed he’d been defrauded.
Nine PBS sued OSS in April, but the station later paused the case after James Tramel, who identified himself as a managing partner of a group that had acquired OSS’s assets, contacted Nine PBS and confirmed that its data was being housed at Iron Mountain’s Denver facility.
About a month later, Tramel stopped responding to the station too. An automatic reply from his email account said he was no longer affiliated with OSS. Nine PBS says Tramel later told the station by phone that he ‘had been defrauded’ into purchasing the company and that it had reverted back to its previous owners.
Pretty sus, bro.
Even though the court has now ruled in the station’s favor, Iron Mountain still hasn’t given them access and OSS remains in limbo.
Iron Mountain still maintains that it doesn’t have to give PBS the files that PBS owns because PBS was not its client.
When the station’s attorney again contacted Iron Mountain, the company acknowledged that it had the data and initially signaled that it wanted to comply with Nine PBS’s request. But Iron Mountain later declined to turn it over, citing OSS’s ownership of the infrastructure housing the data.
This is a pretty bad look for Iron Mountain, a company big enough to be on the S&P 500. First level resistance for legal reasons (“You’re not our customer”) seems perfectly justified, but digging in your heels after a court ruling seems pig-headed. “Suck it, Elmo! We’re not giving your data back!”
Is there a Louis Rossmann rant on the topic?
Yes. Yes there is.
Remember, when you store your data in “the Cloud,” you’re simply storing your data on someone else’s computer. I have no philosophical objection to larger corporations using offsite cloud backup as part of a larger array of offsite backup solutions. And there are types of offsite data storage (iCloud and GitHub/Lab/Etc. come to mind) that are cheap or free and work perfectly well for their intended purposes. But cloud storage should never be your only data backup.
In the case of this PBS station, if you’re backing up 50TB of data, you probably want to go with both a local NAS array and a cartridge tape backup system, the latter of which is going to run you some $5,000—$20,000 for the machine and $20 and up for multi-terrabyte cartridges. The startup cost is higher, but the monthly fees are lower than cloud storage, and you have your data right there if you need it, plus you can have multiple backups in a fire-proof safe and an off-site storage facility for less than it costs in continuing fees for cloud storage.
There are other considerations (encryption or no encryption, regular tests of backup and restore software, etc.) that will vary from organization to organization, but that’s less important than having backups of your own data where you can lay hands on it, without having to trust third parties or pay lawyers.
I have my most important data saved to a thumb drive, I run Time Machine backups of my Mac to an SSD, plus I burn periodic DVDs of important data. (I used to occasionally save files as attachments in GMail, but with Google using your data to train AI, I can’t really recommend that any more.) Every now and then I’ll rotate a DVD out to my car as a half-assed implementation of “offsite backup.” I’m not even as religious about backup as I should be, but I still have a manifestly more robust and secure backup policy than a St. Louis PBS station…
Tags: data security, Iron Mountain, James Tramel, Lawsuit, Louis Rossmann, Not The Bee, Open Source Storage, PBS, St. Louis, technology, video
The very word “backup” is a rather fraught subject with me, and probably always will be. In the early ’80s you had nothing but ****** FLOPPY DISCS, at 360k, so you had boxes and boxes of the things, created in some arcane format which gave this ordinary user with a reasonably high IQ no confidence at all that every **** one of the disks would work, and which took all the blooming day to create, when you had a nasty boss from Hell, and actual important work to get done and later, to bigger discs, then outboard drives, which all eventually suddenly failed, ……….. and then we got to imaging entire files vs. file-by-file approaches.
Now I have my stuff on some cloud drive, being absorbed by the Borg or something, and an outboard solid state drive with a very well-regarded program, which will copy away for literally hours then ALWAYS “fail” five minutes before completion, blaming a disk error, which neither checkdisk or anything else can ever find… I do not think my experience is in any way unique.
My main backup methods is:
1. Mount blank DVD-R on desktop.
2. Drag files I want to backup to the DVD-R (which macOS understands means “copy” rather than “move”)
3. Click “Burn”
“All their data are belong to us. Someone set them up the zip bomb.”
As is so common in bankruptcy cases, Iron Mountain is holding out for more money. My guess is they locked OSS into a long-term back-loaded contract (first year cheap, then the vise tightens, yada yada) and then got stiffed. Oops, the parasite killed its host! I read the whole “PBS is not the customer” bit as lawyer code for, “We want to be compensated by the extortive terms OSS signed.”
The discovery process could be interesting if we find out what Iron Mountain was charging, and the courts rule on what constitutes a reasonable price.
I have two 1TB USB drives that handle backups. One does backups twice a day, while the other, once a month. We had to evacuate for a major fire in southern Oregon, but I had time to run the backups before packing those drives and a laptop and getting the hell out late that afternoon.
When we got back (the firebreaks held in our area), I plugged the USB drives back in the desktop and carried on. I’m a long-retired techie, and don’t much care for cloud storage.
(Bank statements and such go onto thumb drives, and thence to fire safes. We took one safe with us when it was considered likely we’d lose the house. Pain in the back, but it worked. Paper copies of such as time permits.)
I’m retired so the only things that I really care about are e-mails, web sites, and my photographs.
I’m running a 4-year-old iMac and routinely (about every 3 months) run a Time Machine back-up. I’ve got two 2-terabyte hard drives…one of which sits on the desk and the other stored in a (relatively) fire-proof safe. I also have another 2-terabyte hard drive on which I separately store my photographs. That’s tucked away in a corner of a gun safe.
Back when I used to do engineering work I had multiple back-ups, both on the company’s network and on my desktops (one of which was dedicated to running CFD software). Those files were huge and took a while to save, but it was ultimately worth it when one of the hard drives suffered an unrecoverable crash; I lost only a couple of hours of work on that project.
I’ve never understood why people opt for the “cloud” concept. Putting your information, both mission-critical and personal/private, into somebody else’s hands just strikes me as nuts.
They should probably hire you…
I keep all my data locally. All of it. I keep two full backups of all the data. One in an external hard drive on my desktop, and one on an external hard drive I keep with my laptop. Back up everything once a week. Keep the laptop external air-gapped except when I am backing it up or need to use it on the laptop. I also back up each book I write on a separate flash drive.
I have three separate 4Tb hard drives. Together they cost under $1000.00. That’s cheap.
1.) “This is the most incompetence that I’ve ever seen in my life. It’s inexcusable.”
2.) “They should probably hire you…”
PBS went the DEI route when they could have retained the services of Lawrence Persons. I have very little sympathy for their loss.
it always staggered me how shortsighted execs were about backups. I ran midrange shops, and had to beg and plead for the tape cartridge autoloaders we needed. these were the same execs who were intolerant of 5 minute partial downtimes for hot swap switch replacements.