More Fibre Channel at Home 3/2/2002

I've added and changed my home setup a bit, and it seemed time to document some of the additions and changes.

First, I used a 4-bay chassis from Sandin Fuess (ebay username chirixi, he's also on the resources page.) This, combined with a single T-card from CSI (also on the resources page) and a used mini-tower, gave me a very nice 5-bay solution. I found 4 9G 10k RPM disks on ebay for $10 each to fill the four bays, and I had an extremely fast array.

Pictures of case and chassis

[anticipation-times-four.jpg]
anticipation-times-four.jpg
40357 bytes
529 x 428
[case-1.jpg]
case-1.jpg
46861 bytes
576 x 768
[case-2.jpg]
case-2.jpg
76228 bytes
571 x 646
[case-back-panel-2.jpg]
case-back-panel-2.jpg
46358 bytes
592 x 499
[case-back-panel.jpg]
case-back-panel.jpg
28838 bytes
533 x 369
[case-detail-1.jpg]
case-detail-1.jpg
114324 bytes
932 x 699
[chassis-detail-2.jpg]
chassis-detail-2.jpg
103831 bytes
845 x 675
[chassis-detail-3.jpg]
chassis-detail-3.jpg
139249 bytes
934 x 775
[chassis-detail.jpg]
chassis-detail.jpg
104699 bytes
708 x 770

Unsafe at Any Speed

So fast, in fact, that it crashes the Linux QLogic driver if used in any configuration with parallel writes. RAID 0 is an especially fast way to kill the box. However, reads work fine and are godlike. Here are some rough numbers from my P3/504, 512MB, Debian, 2.4.17 box:
RAID level     Read rate (MB/second), sustained transfers
5              45
0              75-86
linear         20-23
1+0            30-37

Ohmygod its fast

Yes, that's not a typo - eighty five megabytes per second, sustained! It'll read a 1G file into /dev/null in under twelve seconds!

Too bad about the crashes, though. I tested the beejesus out of the hardware with the aptly named "molest" disk test script, so I am 100% confident that the drives and chassis are not the problem. More on that in a bit.

Heeding the Clariion Call

Shortly after I got all of this figured out, I was able to check out from work a monster ten-bay Clariion chassis, as OEM'd by SGI. (They called it a "Deskside Fibre Vault"). This has ten bays with custom rails, power supply sufficient to the task, and is way way oversized. Not to mention noisy, but what can you expect when you put ten spindles in one enclosure?

Clariion pictures

[clar-front-1.jpg]
clar-front-1.jpg
55377 bytes
524 x 699
[clar-full.jpg]
clar-full.jpg
79907 bytes
648 x 864
[clar-mounting-detail.jpg]
clar-mounting-detail.jpg
99033 bytes
940 x 441
[clar-start-2.jpg]
clar-start-2.jpg
69766 bytes
576 x 432
[clar-start.jpg]
clar-start.jpg
63774 bytes
471 x 629
[success.jpg]
success.jpg
39664 bytes
576 x 432
I, of course, immediately needed to fill the case. If nothing else, the blinking lights are magnificent. I ended up buying four more of the 9G 10k RPM disks from Ebay, giving me a spiffy matched set of seven disks. I figured that this could easily saturate the 100+MB/sec loop. The case uses odd standoffs to mount drives into the rails; I was able to use a combination of washers and polymer discs.

So, I have 1 36G IBM system disk (7200RPM), two Seagate 18G 7200s for misc data (in a RAID0 stripe), and the seven 9G disks, currently in in JBOD ("linear") configuration. Due to the problems noted above, I cannot currently run the planned RAID0 7-drive setup. I need to email whoever is maintaining the QLogic driver, the person in code comments denied any responsibility. Ahh, the cons of Open Source. "If it's broke, you get to keep all of the pieces."

Good safety tip, Egon

When one gets this much storage, a nerds' thoughts turn naturally to backup. It's almost spring, after all.

This array is about 105G, depending on whether the arrays are parity'd or not, so you need a Big Backup Device. From my times at Fermilab, I went shopping for a used tape robot. I found a nice Exabyte 10h on Ebay for $250, this is a 7G drive (uncompressed) that holds ten tapes. I got it, it worked even after I took it apart and reassembled it. (Note to my Dad, if he ever reads this: See, I have grown up!)

[exb10h-1.jpg]
exb10h-1.jpg
66068 bytes
524 x 699
[exb10h-2]
exb10h-2
867705 bytes
465 x 622
[exb10h-3.jpg]
exb10h-3.jpg
76832 bytes
674 x 448
[exb10h-4.jpg]
exb10h-4.jpg
63188 bytes
547 x 401
[exb10h-5.jpg]
exb10h-5.jpg
47802 bytes
520 x 694

The Ferrari Testarossa of Serial Media

Of course, having put in the work, I managed to get my hands on a used Sony AIT-2 -- this is a nice toy. 50G uncompressed, 3.5 half-height, and 6MB/sec sustained transfers. Needless to say, I sold the tape robot and am happy with the Sony. To paraphrase Ferris Bueller, if you have the cash I highly recommend picking one up. It is so sweet.

Actually, I like the Exabyte Mammoth-2 quite a bit better, but none presented themselves. The LTO is even nicer in some ways if you're in that market. Feel free to mail me an eval unit!

Conclusions and further work

I've been experimenting with other Fibre channel cards - Interphase 5526 and Emulex LP-8000. Neither is as good as the Qlogic (non bootable, kernel driver is external). I can't get either into enough of a working state to see if the parallel-write crash is in the SCSI layer or in the qlogic driver itself. I've worked around it for now by using JBOD mode, which is slower but reliable. Of course, "slower" is relative since these are 10k RPM disks; it's certainly no slouch!

I'm thinking of moving the system disk to a 9G disk for speed and space, and doing some partitioning to speed the system up a bit. With both the Clariion and Feuss chassis, the system is reliable and fast. Both are recommended; I send my Feuss chassis to a friend since even I can live with only ten bays.

Credits

Thumbnails generated by HTMLThumbnail 1.1.4

Navigation Links

  • Introduction / HOWTO: Fibre Channel
  • FC at CDF page
  • FC at home page
  • FC at home, part two - chassis and tapes
  • Resources page
  • Back to home page