1.5TB SAN for $35K -- this is how we do it...
Mon, 18 Mar 2002 17:07:27 -0800
Naively, I think that if they tried to run a high-throughput database on
this SAN, it would not perform as well as some of the SAN solutions being
sold for the big $$. The sound-file application has relatively low #s of
reads/writes, with high volumes of data being sent on each, a very
> -----Original Message-----
> From: email@example.com [mailto:firstname.lastname@example.org]On Behalf Of Rohit
> Sent: Monday, March 18, 2002 3:41 PM
> To: FoRK@xent.com
> Subject: 1.5TB SAN for $35K -- this is how we do it...
> This is how we do it, my homies -- a real geek knows what the real
> deal is, and knows how to stick it to the man!
> Jan. 1, 2002 Issue of CIO Magazine
> LEW GOLDSTEIN IS a sound supervisor editor for C5 Inc. in New York
> City. C5 does postproduction audio for major motion pictures-which
> means it creates or embellishes every sound you hear in a movie from
> a dog bark to every spoken word. They put the hurricane in Cape Fear.
> The woodchipper in Fargo too.
> Goldstein is also a closet IT guy. To store all those space-hogging
> audio clips, he built a 1.5 terabyte storage area network (SAN). He
> did this without a SAN vendor and for less than $35,000, a third of
> what vendors charge for equipment alone-never mind pesky consulting
> and integration fees.
> His SAN has never crashed. Once, he unplugged it on purpose in an
> attempt to cross it up. When he plugged it back in, sound editors
> returned to work as if nothing had happened.
> Goldstein didn't set out to build a SAN because SANs are trendy. He
> did it because the transition from tape to digital editing was
> wreaking all sorts of havoc in audio postproduction. Digital audio
> files are big, and Goldstein has more than 45,000 of them. Every
> sound from the natural world-and thousands not of this world-is
> stored on a server's hard drive at C5. Most of them are bigger than
> 1MB. Here's a tiny sample: In Get Shorty, a 20-second clip of a 767
> flying overhead was 8MB. Goldstein has gigabytes of "dins," which are
> long stretches of ambient city noise. Some dins run 15 minutes
> (120MB). Goldstein has a file called Aircraft Toilet Flush. He has a
> folder called simply Blowtorches.
> C5 not only edits the sounds; it creates them. Each new movie (he
> recently finished Men in Black 2) involves 15 days of recording with
> "foley artists," people who are recorded knocking on a door or
> walking on gravel and so forth. Hundreds of audio files emerge from
> that work.
> Work processes also contributed to C5's storage problem. Because
> editors at C5 couldn't share files, they made local copies of
> everything they worked on. They also made 6GB local copies of the
> movies in order to sync sound and picture. At any one time, C5 is
> working on four major motion pictures plus several documentaries and
> indie films, each having up to six editors. On top of that, directors
> will often change entire sections of movies during audio
> postproduction, which means everyone will stop what they're doing,
> upload their work, wait for the new video file, make a new local copy
> and then start editing again.
> Vendors offered to sell Goldstein a SAN, of course, but they wouldn't
> sell him what he wanted. If he wanted just an empty rack to put his
> own hard drives in, they'd tell him he had to buy the drives, too, at
> enormous markups. If he wanted fibre channel, they tried to sell him
> on SCSI-a technology his research taught him to avoid. One vendor
> offered a discount if he would beta-test its SAN. He thought that
> sounded like he would be working for them instead of the other way
> A SAN, Goldstein says, is just a big rack of hard drives everyone
> shares. With a hobbyist's background and some dedicated research, he
> was able to learn the technology on his own and avoid vendors'
> upselling, technology biases, and their price tags.
> Goldstein did have to call on a couple of vendors to complete his
> first SAN. He bought fibre channel switches and PCI-to-fibre channel
> cards. He found a humble little company that sold him empty racks at
> a good price.
> He picked up 10 9GB drives on eBay. He spent $51. Total.
> Don't scoff. They've never crashed.
> He put each $5.10 eBay drive in one of the empty racks and connected
> the rack to the switches that in turn connected to four end users. "I
> slapped it together, and it worked," Goldstein says. "I had probably
> spent less than $5,000 at this point."
> The thing hummed, and C5's editors started jumping on board. Four
> nodes became eight. Eight became 16. Soon, every sound editor was
> connected. Users could share files. No more local replications. The
> editing process became more efficient and more collaborative.
> Goldstein believes without his SAN, C5 couldn't have pulled off the
> audio postproduction on Ang Lee's Crouching Tiger, Hidden Dragon. Lee
> was dissatisfied with much of the movie's audio, and he leaned on C5.
> With the SAN, multiple members of the crack team could work
> simultaneously on a file. By the old method, each editor had to wait
> for the previous one to finish his work and upload it.
> The system keeps growing. Goldstein now buys 73GB drives for about
> $700 each-still a minor theft. He has more than 3 racks running 1.5
> terabytes of storage. He just added 500GB without a hitch. Each
> editor gets his own 20GB workspace, and each has access to the
> archive of 39,000 (and growing) audio files. In fact, Goldstein finds
> SAN-building so straightforward, he now sells them on the side.
> The sounds of Scorcese's New York, every wistful breeze in an Ang Lee
> film, every Coen brothers gunshot is sitting there on Lew Goldstein's
> first SAN at C5. That's probably a billion dollars in ticket
> receipts, and Goldstein has yet to spend $35,000.
> Senior Writer Scott Berinato loves a bargain. E-mail him at