[FoRK] Cloudy thoughts on backups

Sean Conner sean at conman.org
Tue Dec 15 13:57:50 PST 2009


It was thus said that the Great Ken Ganshirt @ Yahoo once stated:
> 
> With the recent events of service providers not being able to restore
> themselves after outages, it's unconscionable that any external service
> provider would not do such testing on a regular basis ... regardless of
> what you think of Joel's attitude...

  About a decade ago, I had the displeasure of working with a database [1]
and I asked my friend (who was more knowledgable in these things than I) how
I could get a "dump" of the database.  

  You'd think I asked him to sprout antennae or something.  "Why the @#$@#$
would you need something so @#$@#$ stupid as that?" 

  "I want to back up the database!"

  "What a #$$%# stupid thing to do," he said.  He then went on to explain
that "real" databases can't be backed up because 1) some of the "data" in a
data base isn't explicitely stored in the database, and 2) you're talking
terrabytes of data (to be fair, he was working in telephony at the time and
the "smallest" database he worked on was measured in the millions of rows;
same with the codebases he was used to---smallest was measured in millions
of lines written over 30 years in a proprietary langauge).  In his world
(enterprise-class software systems) "backups" just aren't possible because
it doesn't make all that much sense [6].

  I've also been on the back end of a outtage when the main server of the
company I worked for (different from The Company I work for now) was hacked
[2] during a particularly bad time of the year [3].  Even though we had a
full backup, it still took over 50 hours to rebuild, reinstall and restore
[4].  It's not something I'd like to repeat.

  And that was a small operation (just a few gigs of information).

  But I can see how restoring from backups can be difficult.  At our current
company, our bus number [5] is 1 (maybe 2) and I'm not counted in that
number (then again, out bus number for the network is also 1, and I'm
counted in that number).  IT is already considered a cost-center by near
every large company that doesn't specialize in IT (and even then, I'm
guessing the internal IT is *still* considered a cost-center) and recovery
testing costs extra (so sayeth the bean counters) so I'm guessing it doesn't
get tested all that often unless there's strong support from the C*O level.

  -spc (I think Joel was just saying, "test your restore procedure, not just
	your backup procedures ... ")

[1]	I hate databases.  Not quite as much as control panels, but still, I
	hate them (bad experiences and an insane instructor in college).

[2]	http://boston.conman.org/2004/09/19.1

	And yes, the hacker really did write me an email explaining what he
	did.

[3]	http://boston.conman.org/2004/09/

	The hurricanes *just* *wouldn't* *stop* *coming*!

[4]	Just in time for yet another hurricane.

[5]	http://c2.com/cgi/wiki?BusNumber

[6]	Then again, he showed me the closest were the previous version of
	the switch software was stored---printed out on about a million
	pages of paper, which I doubt was even of the acid-free archival
	type.


More information about the FoRK mailing list