How to setup and manage your disk software mirroring
From FedoraNEWS.ORG
Written by Anze Vidmar on 2005-10-13
| Table of contents |
About
This is quick HOW-TO setup your mirroring with two exactly the same hard disks. It assumes you're building a new system from scratch, and setup mirroring at install time. But let's get something straight first: Software mirroring does not mirror the disks - it mirrors partitions. Thanks to Ed Wilts from Redhat General Newsgroups for this remark. So in this guide I'll show you how to setup your partitions on one disk and mirror them to another. After that, we will learn how to rebuild our mirrored partitions after one disk dies.
Why do you need to mirror your disk?
Mirroring or RAID 1 if you like, is an easy and not painful (to setup) way of protecting the data, from beeing lost from disk crash. For example, If you keep your private stuff on one partition you can mirror this partition to another on the second drive, and if one disk fails, you will not lose your data, since you have the exact same copy of your files on the second disk. The same goes for the whole filesystem. If you setup mirroring correct, you might never have to reinstall your system again ;-)
About RAID 1 a.k.a. Mirroring
A little about RAID 1: Data are stored twice by writing them to both the data disk (or set of data disks) and a mirror disk (or set of disks) . If a disk fails, the controller uses either the data drive or the mirror drive for data recovery and continues operation. You need at least 2 disks for a RAID 1 array.
Advantages: RAID 1 offers excellent read speed and a write-speed that is comparable to that of a single disk. In case a disk fails, data do not have to be rebuild, they just have to be copied to the replacement disk.
Here is a nice picture that shows how RAID 1 is build:
You can read more about RAID levels here (http://www.prepressure.com/techno/raid.htm)
The scenario
Let's say we have two identical IDE disks, both have 4Gb of disk space. We will setup the partitions as follows:
- 100Mb partition for /boot - 3,5Gb partition for / - the rest of the free space for swap
We will create partition 1 and 2 as RAID1 Partitions. Since the swap partition is not mirrored, we don't need to make another Raid1 partition for swap. We will just leave as is. However, it's possible to have swap partition mirrored, but practicly you don't gain anything from this.
If you don't know how to setup Raid partitions at install time, take a look here (http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/sysadmin-guide/ch-software-raid.html).It's a manual for Redhat EL4, but it's the same thing for FC4, so don't get confused.
When we are done setting up the partitions, we will end up with the partitions set up as follows:
- 100Mb /dev/md0 for /boot - 3Gb /dev/md1 for / - the rest of the free space for swap
This configuration looks like this:
...and the RAID part
The first look & feel
After the partition setup part, feel free setup the rest of your system as you like. After the first reboot, If everything went well, your mirrored partitions should be working.
First, you might have a look at the boot messages, to do this issue
dmesg |grep md
...and will see something like this:
You can check the status of the mirroring with the following command
cat /proc/mdstat
You will be provided with a nice stats about your Raid1 partitions:
Now, let's look at the output, just to make sure we understand what is the output saying to us. This is very important for you to understand this, if you want to successfully manage and troubleshoot the mirrors!
So the first line
Personalities : [raid1]
Is telling us that we have setup a Raid1 system.
The next two lines are
md1 : active raid1 hdb2[1] hda2[0]
3582400 blocks [2/2] [UU]
The first line is showing that we have a Raid1 device named md1 which is made of two partitions hdb2 and hda2. The second lines shows how many block does the device have and the numbers below
[2/2]
Are telling us that both 2 out of 2 partitions are being used for mirroring.
The letters
[UU]
Are showing that the system is working and mirroring data on both partitions.
The same goes for all the other lines.
So this is the part were everything works as it should.
The GRUB part of setup
In order for your system to successfully boot from any of too hard disks, you need to tel GRUB manually that you want the MBR to be written on BOTH disks! Otherwise, your system won't boot at all from second disk if the first one fails.
You can do this by first starting GRUB setup with the command
grub
You will see a grub prompt as shown below:
grub>
Next, you need to insert the following commands inside the grub setup:
grub>root (hd0,0) grub>setup (hd0) grub>root (hd1,0) grub>setup (hd1) grub>quit
Note that the GRUB setup parameters applies to have fisrt disk as primary master and second as primary slave. Now your configuration of the partitions are saved on both MBR's on both disks.
Disaster planning
Or so to speak... Now, let's demonstrate that our second hard disk fails. Turn off your machine, and disconnect the second disks and let's see what will happen. In my case, I've disconnected the second hard disk (primary slave). As the system boots the first time after one disk fails, we actually don't see any changes, neither we don't feel anything different. So let's first look what boot messages (dmesg) has to say about this.
type into the console
dmesg |grep md
...and you should see something like this
Now, let's have a look into the /proc/mdstat and we should see something like:
The "numbers" shows that only 1 partition out of two is beeing used at the moment, and the "letters"
[U_]
indicates that second hard disk is missing. If you would disconnected the first disk, the letters would look like this:
[_U]
See the difference? It's very simple even if it's in ASCII ;-)
Disaster recovery / Rebuilding the mirror
Now it's time for us to insert the new blank disk and rebuild our mirror. The first step is to replace your damaged/dead/whatever hard disk. So plug it in and boot your machine. When the system boots up, get ready for some partitioning. Use fdisk, cfdisk or whatever you prefer and make new partitions for your mirror. You can check the current partition size either with
df -h
or by
fdisk /dev/hdX -l
You don't really need to make the partitions to be the exact same size as allready created ones on our working disk, the only thing that you shouldn't do is make the new partition smaller than the original. When you'll try to rebuild the mirror you will end up with an error like this:
mdadm: hot add failed for /dev/hdb1: No space left on device
So make sure the new partitions aren't smaller than the original once. Next, it's smart to reboot your machine, just to make sure the system is using the new partition table.
After the machine is back up, and you have your partitions ready to be mirrored, it's time to actuall rebuild the mirror.
The rebuilding mirror is done with the mdadm command, and adding one by one partition to the mirror. The following example shows, how to rebuild our two partitions / (root) and /boot from previous example.
So, let's start with rebuilding the boot partition /dev/md0 first. This is done by
[root@localhost ~]# mdadm --manage /dev/md0 --add /dev/hdb1 mdadm: hot added /dev/hdb1
You will be notified when partition is successfully added to the mirror. Rebuilding this partition is done in a second or so, for its being very small (in our case 100MB).
Now, we only need to add the second /dev/md1 (root) partition back to the mirror. This is done by
[root@localhost ~]# mdadm --manage /dev/md1 --add /dev/hdb2 mdadm: hot added /dev/hdb2
The mirror for root partition is now rebuilding ... This will take a few minutes, for the partition beeing "not so small" :-) You can see the rebuilding progress by issuing the command
watch /proc/mdstat
It should look something like this:
You can see the progress, remaining rebuilding time and the rebuilding speed.
When done, you'll be notified and can check the status again to make sure the mirrors were rebuild:
The only step left to be done is, to setup GRUB again, so that MBR is written to the new disk. You can use the same method to setup GRUB as mentioned in part #6 (http://fedoranews.org/mediawiki/index.php/How_to_setup_and_manage_your_disk_software_mirroring#The_GRUB_part_of_setup) of this article.
It's all done now. Reboot your machine. After the reboot, check boot messages for mirroring details. You should see now that everything is back to normal, as shown below:
That's it, at this point we have successfully rebuild the mirror and system is using both hard disks again.
Conclusion
Having mirroring at home on your desktop PC or on production servers it's a good and cheap-solution idea. You can sleep as calm as a baby again, knowing that when the time comes and your disk fails for real this time, you will not lose any of your data/personal stuff.
