User Tools

Site Tools


technology:linux:creating_a_4-disk_raid_array

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

technology:linux:creating_a_4-disk_raid_array [2010/02/04 08:46]
Chris Freyer repairing table header
technology:linux:creating_a_4-disk_raid_array [2011/04/06 08:50] (current)
Chris Freyer
Line 1: Line 1:
 ====== Creating a 4-Disk RAID Array ====== ====== Creating a 4-Disk RAID Array ======
- 
- 
 ===== The Hardware ===== ===== The Hardware =====
 The hardware I'm using is pretty standard stuff.  Its not a gamer PC, but its relatively new technology.  And its very I/O-friendly: The hardware I'm using is pretty standard stuff.  Its not a gamer PC, but its relatively new technology.  And its very I/O-friendly:
-  * Gigabyte [[http://www.gigabyte.com.tw/Products/Motherboard/Products_Overview.aspx?ProductID=2324|GA 945P-S3]] motherboard - no RAID, no hotswap, 1 EIDE port, 4 SATA II ports+  * Gigabyte [[http://www.gigabyte.us/Products/Motherboard/Products_Overview.aspx?ProductID=2958|GA EP45T-UD3LR]] motherboard ([[http://www.newegg.com/Product/Product.aspx?Item=N82E16813128371|Newegg link]]
   * Pentium D dual-core 3.4 Ghz CPU   * Pentium D dual-core 3.4 Ghz CPU
-  * 2GB RAM+  * 4GB RAM
   * 1 80GB EIDE disk   * 1 80GB EIDE disk
   * 4 1TB SATA disks   * 4 1TB SATA disks
Line 13: Line 11:
  
 {{  :technology:linux:mobo_diagram.png?161x204|Hard disk connection diagram}}  For the sake of brevity, I'll get right to the I/O setup.  The drive configuration is shown to the right.  An 80GB EIDE drive is connected to the PATA connector on the motherboard.  The BIOS detects it as the first disk, which is perfect for this setup.  The four SATA II drives are snapped into a drive cage, and their data connections are plugged into the four SATA ports on the motherboard.  My board doesn't support hotswap, so I'll have to power off the system to replace a drive if one fails.   {{  :technology:linux:mobo_diagram.png?161x204|Hard disk connection diagram}}  For the sake of brevity, I'll get right to the I/O setup.  The drive configuration is shown to the right.  An 80GB EIDE drive is connected to the PATA connector on the motherboard.  The BIOS detects it as the first disk, which is perfect for this setup.  The four SATA II drives are snapped into a drive cage, and their data connections are plugged into the four SATA ports on the motherboard.  My board doesn't support hotswap, so I'll have to power off the system to replace a drive if one fails.  
 +
 ===== Hardware-based vs. Software based ===== ===== Hardware-based vs. Software based =====
  
Line 284: Line 283:
  
 </code> </code>
-==== Replacing bad partition/device ====+==== Re-adding a partition ====
 Suppose you forget to update your ''/etc/mdadm/mdadm.conf'' file after changing an array with ''mdadm'', and then you reboot (like I did)?  You'll end up with an array that looks like this... Suppose you forget to update your ''/etc/mdadm/mdadm.conf'' file after changing an array with ''mdadm'', and then you reboot (like I did)?  You'll end up with an array that looks like this...
 <code> <code>
Line 374: Line 373:
 </code> </code>
  
 +==== Replacing a bad device ====
 +Well, it eventually happens to everyone.  One of my RAID drives went bad.  I don't understand why--it wasn't doing anything demanding or different.  But luckily I got this email, thanks to a properly configured ''/etc/mdadm/mdadm.conf'' file:  
 +
 +<code>
 +This is an automatically generated mail message from mdadm
 +running on werewolf
 +
 +A FailSpare event had been detected on md device /dev/md0.
 +
 +It could be related to component device /dev/sdc1.
 +
 +Faithfully yours, etc.
 +
 +P.S. The /proc/mdstat file currently contains the following:
 +
 +Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
 +md0 : active raid5 sdc1[4](F) sdb1[0] sdd1[1] sde1[3]
 +     2930279808 blocks level 5, 128k chunk, algorithm 2 [4/3] [UU_U]
 +     [=>...................]  recovery =  7.6% (75150080/976759936) finish=1514.9min speed=9919K/sec
 +
 +unused devices: <none>
 +</code>
 +
 +The important thing is the ''(F)'' next to the ''sdc1'' partition.  It means the device has failed.   I power cycled the machine and the array came up in "degraded, recovering" status, but it failed after several hours of rebuilding.  After two or three attempts, I decided the drive was bad (or at least bad enough to warrant replacing).  Here are the stepssteps:
 +
 +  - Run ''mdadm --remove /dev/md0 /dev/sdc'' to remove the bad drive from the array
 +  - Replace the faulty drive with a new one
 +  - Use ''fdisk'' as described above to setup the drive like the others
 +  - Run ''mdadm --add /dev/md0 -/dev/sdc1'' to add the new drive to the array
 +
 +After that, ''cat /proc/mdstat'' reported the array was recovering.  It took nearly 6 hours to rebuild the data, but everything went back to normal.  No lost data.
  
 ===== Benchmarking for Performance ===== ===== Benchmarking for Performance =====
Line 452: Line 482:
 |# CPU seconds used     |   230.32     |  165.07      |   187.96      | |# CPU seconds used     |   230.32     |  165.07      |   187.96      |
  
-IOZONE provides some useful performance statistics for the disks.  The above stats were gathered while it was running (same tests for each filesystem).  Nothing significantly different here.+IOZONE provides some useful performance statistics for the disks.  The above stats were gathered while it was running (same tests for each filesystem).  EXT3 took longer to run the tests (3 minutes and 2.5 minutes longer), and took more CPU time (65 and 42 seconds more {39% and 26% extra} ).  JFS has a slight advantage over XFX, but EXT3 is in a distant 3rd place.
  
 ^  **5GB FILE CREATION**  ^^^^ ^  **5GB FILE CREATION**  ^^^^
Line 463: Line 493:
 |# CPU seconds used     |   23.88      |  13.72       |  14.00        | |# CPU seconds used     |   23.88      |  13.72       |  14.00        |
  
-Creation of multi-gigabyte files will be a routine event on this machine (since it will be recording TV shows daily).  Each filesystem took just over 1 minute to create the file.  As I expected, EXT3 had significantly higher CPU utilization than JFS and XFS (90% and 58%, respectively).  The number of CPU seconds used was higher too (74% and 70%, respectively).  These small numbers don't look significant until you think about running a media server and how much disk IO goes on.+Creation of multi-gigabyte files will be a routine event on this machine (since it will be recording TV shows daily).  Each filesystem took just over 1 minute to create the file.  As I expected, EXT3 had significantly higher CPU utilization than JFS and XFS (90% and 58% higher, respectively).  The number of CPU seconds used was higher too (74% and 70%, respectively).  These small numbers don't look significant until you think about running a media server and how much disk IO goes on.
  
 ^  **5GB FILE DELETION**  ^^^^ ^  **5GB FILE DELETION**  ^^^^
Line 474: Line 504:
 |# CPU seconds used     |    0.95      |   0.00       |  0.00         | |# CPU seconds used     |    0.95      |   0.00       |  0.00         |
  
-File deletion is a big deal when running a media server.  People want to delete a show and immediately be able to continue using their system.  But I've experienced long delays with EXT3 before -- sometimes 10-15 seconds when deleting a show.  The statistics here don't reflect that, but they do indicate a problem.  The elapsed time is 19X and 16x longer with EXT3 than with JFX and XFS.  CPU use and CPU seconds are simlar in nature.  +File deletion is a big deal when running a media server.  People want to delete a large file (i.e. a recorded program) and immediately be able to continue using their system.  But I've experienced long delays with EXT3 before -- sometimes 10-15 seconds when deleting a file.  The statistics here don't reflect that, but they do indicate a problem.  The elapsed time is 19X and 16x longer with EXT3 than with JFX and XFS.  CPU use and CPU seconds are simlar in nature.  
  
 Obviously, EXT3 is out of the running here, so I'll stop talking about it.  The real decision is between JFS and XFS.  Both have similar statistics, so I decided to search the internet for relevant info.  Here are some sources that swayed my opinion: Obviously, EXT3 is out of the running here, so I'll stop talking about it.  The real decision is between JFS and XFS.  Both have similar statistics, so I decided to search the internet for relevant info.  Here are some sources that swayed my opinion:
Line 481: Line 511:
   * [[http://www.mythtv.org/wiki/RAID|MythTV RAID guide]]   * [[http://www.mythtv.org/wiki/RAID|MythTV RAID guide]]
   * [[http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO.html|Linux Software RAID Guide]]   * [[http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO.html|Linux Software RAID Guide]]
 +
 ===== And the winner is... ===== ===== And the winner is... =====
 The winner is:  XFS.  I've been using it for several years on my MythTV box with no issues.  My recorded programs are stored on an LVM volume formatted with XFS.  The volume itself spans 4 drives from different manufacturers((Seagate, Maxtor, and Western Digital)) and with different capacities((200gb, 300gb, 500gb, and 250gb)) and interfaces((EIDE and SATA)).  My recording and playback performance are great, especially when you consider that my back-end machine serves 4 front-ends (one of which is on the back-end machine).  And the file system delete performance is perfect:  about 1 second to delete a recording (normally a 2-6gb file). The winner is:  XFS.  I've been using it for several years on my MythTV box with no issues.  My recorded programs are stored on an LVM volume formatted with XFS.  The volume itself spans 4 drives from different manufacturers((Seagate, Maxtor, and Western Digital)) and with different capacities((200gb, 300gb, 500gb, and 250gb)) and interfaces((EIDE and SATA)).  My recording and playback performance are great, especially when you consider that my back-end machine serves 4 front-ends (one of which is on the back-end machine).  And the file system delete performance is perfect:  about 1 second to delete a recording (normally a 2-6gb file).
/home/cfreyer/public_html/data/attic/technology/linux/creating_a_4-disk_raid_array.1265291188.txt.gz · Last modified: 2010/02/04 08:46 by Chris Freyer