RAIDing the Digital Studio
Posted Feb 19, 2004

There's nothing like a good day's shoot to fill up your workstation's storage. At the shooting stage, if the DV camera's tape is full, it's a simple task to pop out a cassette and drop in another. But it isn't that simple with workstation hard disk space. True to its name, fixed storage—when filled to capacity—can put you in quite a fix.

Of course, you're always free to add on. But consider what would happen if you just plugged in an endless succession of new hard disks to add space to your system. Since both FireWire and USB 2.0 support daisychaining drives together, it is conceivable that you could have an ever-growing set of disks laid out on the floor around your desk. However, you quickly realize that there are several problems with this setup.

First, every hard disk is its own entity to each application. So you have to go through disk after disk to locate your work files. Searching for data on even two disks becomes a challenge when time is of the essence. Second, if a file becomes too large for a single disk, you'll have to take time out to buy and add a larger disk to your system. Finally, even FireWire and USB channels lag in performance as the I/O burden becomes greater and greater and your workstation tries to talk to all these disks.

This means that rendering times soar, and you then have to face those oh-so-joyful delays in redrawing megapixel designs onscreen from virtual memory. All of which is the kiss of death for anyone doing post-production on budget or deadline, which means virtually anyone doing commercial video work.

This is where desktop RAID storage comes into play.

RAID is an acronym for Redundant Array of Independent (or Inexpensive) Disks. Time was when the "Inexpensive" was more of a joke than a genuine part of the acronym, because the RAID systems of the past were anything but inexpensive. In fact, RAID was primarily the province of the corporate server room where the system was patiently administered to by lab-coated gurus who knew how to appease the Storage Gods.

Now, thanks to the ever-decreasing price of hard disks and components, there are less expensive, pre-configured RAID systems for the desktop workstation that designers and editors can use and effectively ease the storage strain.

Have You Ever Seen the RAID?
RAID, in essence, is a way of making two or more hard disks appear as a single drive to the workstation. This "virtual" drive then can be far larger than any single drive making up the RAID set. Apple's Xserve RAID storage box, for example, can grow to almost three terabytes (or about 26 hours of uncompressed digital video footage at 30+ Mbps) with a full set of 14 drives installed. For OS X, however, all these drives appear as a single device, making management easier.

But RAID offers several more advantages beyond creating virtual disks that make it a compelling choice for the digital studio:
-Data protection
-Higher performance
-Fault tolerance
-Easier maintenance

Protecting data is the primary goal of RAID. Most RAID systems can readily recover from a single hard drive failure in the set. It only takes a single drive failure in a non-RAID protected workstation to demonstrate why RAID is important, given that the loss of even five or ten minutes of edited video might well be worth ten or twenty times that of any such storage system.

Equally important, RAID systems provide faster performance in reading and writing data to hard disks. That's because there is more than one drive available to answer the call when an application needs to record or retrieve data. This aspect of RAID is probably the most widely known advantage and thus a great reason to upgrade to RAID storage.

Another advantage comes along with protecting the data—fault tolerance. Since the RAID system can continue to work after a drive failure, there's no unscheduled downtime. This can again be a critical asset at crunch time when the video post team needs to meet a deadline.

Finally, replacing a failed drive in a RAID system is often only a matter of removing and replacing a snap-in module. This makes maintaining a RAID system much easier than storage drives on a conventional workstation. Compare that remove-and-replace process with that of recovering from a failed hard disk in a conventional workstation:
1. Power down the workstation
2. Open the case
3. Remove the hard drive cables
4. Detach the hard drive
5. Insert the replacement drive
6. Plug in the cables
7. Close the box
8. Reboot the workstation
9. Format the drive
10. Restore your data files from backup tape or optical disc

Any question which is easier or faster?

One last point—don't confuse RAID storage with a way to connect different devices to the CPU. There are RAID systems that support such connectors as SCSI, USB2.0, and FireWire. Internally, the box might have SCSI, ATA, or even SATA drives.

Level with Me
Naturally, since choosing to upgrade to desktop RAID seems such a simple proposition, those lab-coated gurus would have to make things difficult by having various types of RAID. In fact, there are more than a dozen varieties of RAID—known as Levels—and each includes its share of sub-types. The good news is, there are only four RAID Levels of real interest to the digital studio: RAID Levels 0, 1, 3, and 5. They differ in price, performance, and protection.

RAID 0 is the simplest Level of RAID. In effect, the storage of all the drives in a RAID 0 system is combined into one virtual drive. This provides you with the maximum amount of storage space. It also improves performance, since reading and writing data can occur on multiple disks in the RAID set simultaneously.

The big drawback to RAID 0 is that it offers no protection from a drive failure. If one drive goes out, all the data stored in the RAID 0 system is lost. (Well, not actually lost. But lost to most conventional tools. A disk recovery expert can still retrieve it, for a price.)

So yes, RAID 0 is in fact not a "Redundant" disk at all. As one observer put it, it's more like "AID" than RAID. A RAID 0 system can be as small as two drives or as many as the type of connector you have supports (16 with Ultra320 SCSI, 63 for FireWire, or 127 for USB 2.0).

A RAID 1 system keeps a duplicate copy of all data on separate disk drives. This provides the highest level of protection since there are two copies of everything. However, this is also its biggest drawback since you have to buy twice as much storage to hold your data. By definition, then, RAID 1 requires an even number of drives, with at least two to start with. On the plus side, you again get a performance boost by being able to read data from either copy—whichever is less busy.

RAID 3 is the level most suited to A/V production environments since it handles large data files (like video files) exceptionally well, and much better than the far more common RAID 5. It does this while still providing protection from a single drive failure by the use of a "parity" drive.

RAID 1 protects data by duplicating it. RAID 3 protects data with a less costly approach by using what is called "parity data" to reconstruct lost files. To do this, RAID 3 sets aside one drive from the set to keep the parity data. The RAID system creates this parity data as a type of code based on the unique contents of each stored file.

To understand this, let's say that your RAID 3 system has five drives in it. Four drives will hold the video files themselves while Drive 5 holds the parity code. If Drive 1, 2, 3, or 4 goes out, the RAID system can recreate the missing portion of the file stored on the dead drive based on the unique code for that file in the parity data on Drive 5. If Drive 5 happens to fail, then the RAID system still has access to your full data file; it just won't have RAID protection until the dead Drive 5 is replaced. (Once it is, the RAID system will recalculate the parity data based on the existing files on Drives 1 through 4.)

As its name suggests, RAID 3 requires at least three drives.

Probably the most familiar RAID level is RAID 5. RAID 5 is similar to RAID 3 in that it too uses parity data for protection. Unlike RAID 3, however, RAID 5 stores both the files and the parity data across all the disks in the set and not just on a single drive. This provides the best performance for typical office and database use, but poor performance for video files. That's because reading and writing to RAID 5 involves all the drives simultaneously. While this isn't an issue for the relatively small word processing files or database queries that office users work with, it is for the multi-megabyte or multi-gigabyte files native to digital video post-production.

Configuring RAID 5 also requires setting aside one third of the overall system capacity for parity data. This can prove expensive compared to the RAID 3 approach. RAID 5 also requires a minimum of three drives.

RAID 0+1 is a more recent addition to RAID levels. It overcomes limitations of both RAID Level 0 and 1 mentioned earlier—Level 0 provides no protection from a single-drive failure and Level 1 requires a duplicate drive for each and every drive in the set. RAID 0+1 instead duplicates or mirrors an entire RAID 0 virtual drive on a second RAID 0 virtual drive. This provides the redundancy required to overcome a drive failure while eliminating the requirement for an exact even number of drives in each set.

Instead, the overall capacity of each RAID 0+1 virtual drive must be the same so mirroring can take place but the exact number of drives can be different. (For example, if the first RAID 0+1 set is 120GB in size using three 40GB drives, the second or mirror set could be four 30GB drives or two 60GB drives, etc. With RAID 1, the mirror would have to be three 40GB drives.)

An advantage to RAID 0+1 over RAID 3 is that there's no need to continuously calculate parity data, thus increasing performance. The drawback is that you again have to give up 50% of your storage capacity for data protection. (Making it potentially twice as expensive.) RAID 0+1 requires at least four drives.

As we mentioned earlier, RAID today, at all its levels (in varying degrees), offers many advantages:
-Higher performance than internal standalone drives
-Greater capacity over single drives
-Reliable, fault-tolerant stability in the event a drive crashes
-Prosumer pricing compared with past iterations of RAID
-Easy, almost plug-and-play setup

The enhanced throughput performance will probably be the first and most compelling reason to upgrade. However, in the event of a hard disk crash, the value of redundant storage will eclipse everything else. Instead of a week of horror, you may just experience a coffee break of inconvenience, thanks to vendors like Medéa, Kano, or Apple.

RAID on Parade
Now that we've discussed how RAID works and how it vastly improves storage for the digital studio, we can turn to how to implement it and examine some RAID offerings from vendors specifically targeting collaborative video production and post-production environments and outfits looking to reap RAID's benefits without RAID-specific engineering expertise. Of course, there is always some learning curve involved; all these levels and variations would make it seem difficult to add a RAID system as a do-it-yourself project for an IT ingenue. Fortunately, vendors such as Medéa and Kano Technologies make the task simple and almost plug-and-play for around $1000-$3000.

Since our focus here is on upgrading to RAID for desktop workstations, we won't look at three other ways to RAID your system, but we'll at least mention them here:
1. Network-attached storage (NAS) units such as Snap Appliance's Snap Server include RAID. Although easily within our set budget, they require some network expertise to configure access (such as an IP address) and user rights management.
2. External RAID subsystems that require a RAID controller card within the workstation. While the units themselves are cheap enough, the task of installing and configuring a card can be a challenge. (Apple's delightful Xserve RAID unit is not a desktop system but requires an Xserve as a parent. For more on Xserve, see EMedia's September cover story, pp. 34-39)
3. Contemporary, mid- to top-of-the-line motherboards such as those from ASUSTek or ABIT now include RAID 0 and/or RAID 0 and 1 support. Again, while attaching new drives to the motherboard isn't difficult, the upgrade to an all-new motherboard is.

Medéa's Firefly
To call Medéa's FireFly stunning is an understatement. It is exactly what the contemporary Mac studio wants on the desktop: a stylish, almost whimsical set of drives that feel right at home with the iMac. If you need beautiful storage, this is the way to go.

Design isn't the only area where Medéa earns honors. Functionally, the FireFly works as RAID 0 (big disk) or RAID 1 (mirrored disks) using FireWire. Configured as RAID 1, our test unit consistently delivered an impressive 40MB/sec over FireWire 400. It supports dual-stream, real-time editing from NLEs like Adobe Premiere, Apple Final Cut Pro, and Avid Xpress DV.

All of which is fine for an external desktop unit, but where the FireFly excels is in meeting the needs of the portable storage crowd. A real boon for professional studios that want to protect all content—which we're guessing means "all of them"—Medéa has crafted the FireFly to allow one of the two drives to act as a portable unit. It can thus be used on location or at home for onsite storage, then returned to the cradle to get "re-mirrored" with its partner.

To have this flexibility, you have to reconfigure the FireFly from its shipping version as RAID 0 into two separate drives. Medéa then includes Peer Software's Save-N-Sync for Windows and Qdea's Sychronize! X Plus for Macs to perform the resync. Both are straightforward and easy enough to use. Thus the FireFly is a fantastic solution for combining portability and data protection in a way that is almost painless.

With traditional RAID 1, even though the two drives are mirrors of each other, they have to be locked together in the same box. They cannot be removed like the FireFly. (If you did, the workstation will say that the remaining RAID 1 drive is unreadable.)

This adds incredible value to the FireFly. It makes it a must-have for those who work between home, studio, and various client sites—especially given that most home systems lack any regular backup.

If there's a flaw to the brilliant FireFly, it's that it isn't as portable as it could be. Although the modules look incredible in the cradle, they are bulky once removed. They lack any kind of carrying handle, and their shape makes it hard to find a comfortable way to hold them. The small portable cradle that comes with the set seems destined to get lost in transit somewhere, forcing you then to lay the module on its side to use it.

Contrast this with nNovia's QuickCapture portable hard disk, for example. QuickCapture is a DV production tool for location use and looks the part. It comes already mounted in a 2x4x6-inch sturdy metal tool belt case. The FireFly's plastic exterior and large form factor suggest that it prefers indoor use even though it could plug into a DV camera and thus theoretically be used on location. (nNovia touts its QuickCapture as being expressly designed for just this purpose.)

Fortunately, the FireFly easily redeems itself with its "too cool for words" looks and its removable mirror capability. The unit starts at around $700 for 240GB (120GB when mirrored) and around $1300 for 500GB/250GB.

Note: Do not confuse Medéa's FireFly Personal RAID system with a similarly named unit from Dynamic Network Factory called the FireFly 2000 RAID system nor with ASACA's FireFly Digital Virtual Library.

Medéa's VideoRAID RTR 5/320
The VideoRAID RTR unit demonstrates why Medéa sets the standard for desktop RAID for digital content creation. This solid, all-business external tower just feels like a workhorse for daily production work. Tailored for RAID Level 3 (what Medéa calls "fail-safe"), the VideoRAID certainly won us over as a confidence builder while demonstrating superior performance when used in video editing.

Like the Firefly, the VideoRAID is pre-tested to work with NLEs like Premiere, or Xpress DV. We had no problem with our test Final Cut Pro system (as guaranteed by Medea), and we give high marks to Medéa for this software certification program. Too often hardware vendors simply assume that software will run on their stuff without going that extra step. Why should editors also be beta-testers?

Unlike other units discussed here with FireWire or USB connectors, the VideoRAID uses Ultra160 SCSI for its connection. (If you need the extra performance, Medéa offers the higher-end RT3 with Ultra320 SCSI support for maximum throughput in an HD NLE environment.) Many existing workstations have SCSI connections and can take advantage of the VideoRAID. For others, it might be nice if Medéa offered an option for FireWire connectivity.

At around $3000, the VideoRAID RTR 5/320 is at the top end of our budget. However, for that, you get everything in our short list of desktop RAID requirements—RAID 3 support, hot-swappable drives (meaning you can replace a bad drive with the system running), modular power and connectors, and two large fans to keep temperatures down. We even like the lovely blue LEDs on the front panel. The RTR 5/320 is the one to have if you are serious about protection and performance. (The 5/320 designation indicates that this model has five drives with a total capacity of 320GB; the Medéa line offers capacities all the way up to 10/2000.)

Firewire Direct's Vanguard III
Austin, Texas-based Firewire Direct offers one of the first FireWire 800 RAID systems on the market in the Vanguard III. Thus it provides some of the fastest throughput of any RAID unit (except for Medéa's Ultra320 system). Testing showed that the FireWire 800 sustained about 55-60MB/sec throughput, just a bit under Ultra320 SCSI's 75MB/sec.

The Vanguard is limited to RAID Levels 0 or 1 (big disk or mirror). While the system's hard drives can be hot-swapped, the power supply and connectors are fixed and not modular, requiring a return to the factory if there are problems with these components.

Design-wise, the Vanguard offers a polished aluminum case with a stylish blue LCD screen up front. (Vanguard is the only unit on our short list with this useful feature.) While the Vanguard is not likely to give the VideoRAID RTR a run for its money, the box is attractive enough for the average Mac user.

The Vanguard III 400GB unit costs around $950, going up to $1300 for 500GB. Conveniently, it also provides USB 2.0 in the same box with FireWire. Also, the Vanguard's hot-swap capability is a $500-or-so add-on option rather than a built-in feature. For the budget-minded, this might be one way to go to get into a fast RAID system with an upgrade path to hot-swap in the future. This is the one to have if you want high capacity and high throughput on a budget.

Kano's SureVault
Kano Technologies has a long history in the external storage arena, so it comes as no surprise that their newest SureVAULT RAID 5 covers the bases in desktop RAID—support for all three top connections (USB 2.0, FireWire 400, and 800), plug-and-play installation, hot swappable drives, and transparent support for Mac and PC. The only downside is a lack of options for RAID levels besides Level 5.

Our test unit was their mid-range SureVAULT RAID 5 240GB system with USB 2.0. Since the SureVault is designed to use only RAID 5, it would seem to be more suited to office automation tasks than to video production. However, in our tests, throughput kept up fine with our editing software. This may have been due in part to the unit's having only three drives in the set, thus limiting the amount of work to do with all the drives. It may also have been the small number of large files we were working with as well.

In a welcome move that's increasingly unpopular among storage solution vendors, Kano ships the SureVAULT with a small printed manual. Although somewhat amateurish in design and layout, the booklet covers all the major points with plenty of illustrations.

As tested, the 240GB SureVAULT RAID-5 system runs about $1500 with an effective capacity of 180GB. The newer FireWire 800 systems start at around $2200 for 360GB and up. The SureVAULT is a competent, straightforward package for an easy upgrade to RAID 5.

Who'll Stop the RAID?
The selection of desktop RAID systems increases almost daily. The primary questions remain: how much protection do I want (none with Level 0, Level 1 mirroring, Level 3, or Level 5)? What type of connection do I need (USB 2, FireWire 400 or 800, or SCSI)? What overall capacity? And finally, how important is it to proudly display my system?

Fortunately, it isn't a question of whether to RAID or not to RAID. The value of even a minute of content means that your studio will get RAIDed. Now it is a matter of being RAIDed in style.

Sidebar: How RAID Works
Think of RAID 0 as if you are dealing with paper and books. RAID 0 is like taking three stacks of paper and combining them into one big stack or book. If you print on the paper (like writing to the RAID disk) and you lose part of the big pile, then you are missing data. In the case of missing data with RAID 0, the disk is lost.

RAID 1 is like taking a journal and photocopying the contents each day to make a duplicate. If you lose the first book, you can start using the second as well as make a new photocopy duplicate to protect your contents.

RAID 3 is like having two or more books on a shelf with a separate Master Index book. Each "book" is a hard disk. The Master Index book is like RAID parity data, only with RAID the parity "index" lets you recreate any single book in the set that may go missing. If the RAID parity data (the Master Index in our example) disappears, then you can use the original disks to recreate it.

With RAID 5, each hard disk is like a volume in an encyclopedia. Put together on the shelf, the books can look like one big "virtual" book. Again, the encyclopedia's index is like RAID parity, letting your recreate any single volume in the encyclopedia that may go missing. If it disappears, then you can use the remaining disks to recreate it. Unlike an encyclopedia, the RAID 5 volume stores data across all the disks, like having paragraph 1 in Volume 1 of the encyclopedia, paragraph 2 in Volume 2, etc., rather than storing all of File 1 on Disk 1, all of File 2 on Disk 2, etc.

Companies Mentioned in this Article
Apple www.apple.com
FireWireDirect www.firewiredirect.com
FWB, Inc. www.fwb.com
Kano Technologies www.kanotechnologies.com
Medéa Corporation www.Medéa.com
Snap Appliance, Inc. www.snapappliance.com
SoftRAID LLC www.softraid.com