linux磁盘管理软RAID的实现原理分析和方法分享

  [root@centos7 bash]$ fdisk /dev/sdb  welcome to fdisk (util-linux 2.23.2).    changes will remain in memory only, until you decide to write them.  be careful before using the write command.    device does not contain a recognized partition table  building a new dos disklabel with disk identifier 0x93d380cf.    command (m for help): n  partition type:   p primary (0 primary, 0 extended, 4 free)   e extended  select (default p): p  partition number (1-4, default 1):   first sector (2048-209715199, default 2048):   using default value 2048  last sector, +sectors or +size{k,m,g} (2048-209715199, default 209715199): +1g  partition 1 of type linux and of size 1 gib is set    command (m for help): t  selected partition 1  hex code (type l to list all codes): fd  changed type of partition 'linux' to 'linux raid autodetect'    command (m for help): p    disk /dev/sdb: 107.4 gb, 107374182400 bytes, 209715200 sectors  units = sectors of 1 * 512 = 512 bytes  sector size (logical/physical): 512 bytes / 512 bytes  i/o size (minimum/optimal): 512 bytes / 512 bytes  disk label type: dos  disk identifier: 0x93d380cf     device boot start  end blocks id system  /dev/sdb1  2048 2099199 1048576 fd linux raid autodetect    command (m for help): w  the partition table has been altered!    calling ioctl() to re-read partition table.  syncing disks.  [root@centos7 bash]$ fdisk /dev/sdc  welcome to fdisk (util-linux 2.23.2).    changes will remain in memory only, until you decide to write them.  be careful before using the write command.    device does not contain a recognized partition table  building a new dos disklabel with disk identifier 0xc56b90d8.    command (m for help): n  partition type:   p primary (0 primary, 0 extended, 4 free)   e extended  select (default p): p  partition number (1-4, default 1):   first sector (2048-41943039, default 2048):   using default value 2048  last sector, +sectors or +size{k,m,g} (2048-41943039, default 41943039): +1g  partition 1 of type linux and of size 1 gib is set    command (m for help): t  selected partition 1  hex code (type l to list all codes): fd  changed type of partition 'linux' to 'linux raid autodetect'    command (m for help): p    disk /dev/sdc: 21.5 gb, 21474836480 bytes, 41943040 sectors  units = sectors of 1 * 512 = 512 bytes  sector size (logical/physical): 512 bytes / 512 bytes  i/o size (minimum/optimal): 512 bytes / 512 bytes  disk label type: dos  disk identifier: 0xc56b90d8     device boot start  end blocks id system  /dev/sdc1  2048 2099199 1048576 fd linux raid autodetect    command (m for help): w  the partition table has been altered!    calling ioctl() to re-read partition table.  syncing disks.  [root@centos7 bash]$ fdisk /dev/sdd  welcome to fdisk (util-linux 2.23.2).    changes will remain in memory only, until you decide to write them.  be careful before using the write command.    device does not contain a recognized partition table  building a new dos disklabel with disk identifier 0x7e0900d8.    command (m for help): n  partition type:   p primary (0 primary, 0 extended, 4 free)   e extended  select (default p): p  partition number (1-4, default 1):   first sector (2048-41943039, default 2048):   using default value 2048  last sector, +sectors or +size{k,m,g} (2048-41943039, default 41943039): +1g  partition 1 of type linux and of size 1 gib is set    command (m for help): p    disk /dev/sdd: 21.5 gb, 21474836480 bytes, 41943040 sectors  units = sectors of 1 * 512 = 512 bytes  sector size (logical/physical): 512 bytes / 512 bytes  i/o size (minimum/optimal): 512 bytes / 512 bytes  disk label type: dos  disk identifier: 0x7e0900d8     device boot start  end blocks id system  /dev/sdd1  2048 2099199 1048576 83 linux    command (m for help): t  selected partition 1  hex code (type l to list all codes): fd  changed type of partition 'linux' to 'linux raid autodetect'    command (m for help): w  the partition table has been altered!    calling ioctl() to re-read partition table.  syncing disks.

  [root@centos7 bash]$ mdadm -c /dev/md5 -a yes -l 5 -n 3 /dev/sd{b1,c1,d1} -c 256 # -c指定创建， -a yes 自动创建设备 ， -l 设定level , -n 设定磁盘个数， -c chunk大小  continue creating array y  mdadm: defaulting to version 1.2 metadata  mdadm: array /dev/md5 started.  [root@centos7 bash]$ mdadm -ds        # 查看信息  array /dev/md5 metadata=1.2 name=centos7.magedu.com:5 uuid=2c8ae60d:a799fcb7:9008a046:ae6ea430  [root@centos7 bash]$ mdadm -ds >/etc/mdadm.conf      # 将软raid信息写入到配置文件中去  [root@centos7 bash]$ mkdir /mnt/md5       # 创建挂载点目录   [root@centos7 bash]$ mkfs.ext4 /dev/md5       # 创建文件系统  mke2fs 1.42.9 (28-dec-2013)  filesystem label=  os type: linux  block size=4096 (log=2)  fragment size=4096 (log=2)  stride=64 blocks, stripe width=128 blocks  131072 inodes, 523776 blocks  26188 blocks (5.00%) reserved for the super user  first data block=0  maximum filesystem blocks=536870912  16 block groups  32768 blocks per group, 32768 fragments per group  8192 inodes per group  superblock backups stored on blocks:   32768, 98304, 163840, 229376, 294912    allocating group tables: done      writing inode tables: done      creating journal (8192 blocks): done  writing superblocks and filesystem accounting information: done     [root@centos7 bash]$ mount /dev/md5 /mnt/md5      # 挂载设备   [root@centos7 bash]$ tail -n 1 /etc/mtab  /dev/md5 /mnt/md5 ext4 rw,seclabel,relatime,stripe=128,data=ordered 0 0  # 查看挂载信息  [root@centos7 bash]$ tail -n 1 /etc/mtab >>/etc/fstab     #添加到fstab文件中，确保开机启动，这里建议使用uuid

  [root@centos7 md5]$ mdadm -d /dev/md5       #查看详细raid5详细信息，可以发现有3个都是working状态的  /dev/md5:  version : 1.2  creation time : wed dec 6 19:28:22 2017  raid level : raid5  array size : 2095104 (2046.00 mib 2145.39 mb)  used dev size : 1047552 (1023.00 mib 1072.69 mb)  raid devices : 3  total devices : 3  persistence : superblock is persistent    update time : wed dec 6 19:39:06 2017  state : clean   active devices : 3  working devices : 3  failed devices : 0  spare devices : 0    layout : left-symmetric  chunk size : 256k    consistency policy : resync  name : centos7.magedu.com:5 (local to host centos7.magedu.com)  uuid : 2c8ae60d:a799fcb7:9008a046:ae6ea430  events : 18    number major minor raiddevice state   0 8 17 0 active sync /dev/sdb1   1 8 33 1 active sync /dev/sdc1   3 8 49 2 active sync /dev/sdd1  [root@centos7 md5]$ man mdadm  [root@centos7 md5]$ mdadm /dev/md5 -f /dev/sdc1       # -f 设定指定设备故障， 将/dev/sdc1 这个盘标记失败， 看是否数据能访问，我这里使用-f标记失败，工作中可以根据硬盘指示灯判断磁盘状态  mdadm: set /dev/sdc1 faulty in /dev/md5    [root@centos7 md5]$ mdadm -d /dev/md5        #在次查看信息，发现工作的是2个， 一个失败的设备   /dev/md5:  version : 1.2  creation time : wed dec 6 19:28:22 2017  raid level : raid5  array size : 2095104 (2046.00 mib 2145.39 mb)  used dev size : 1047552 (1023.00 mib 1072.69 mb)  raid devices : 3  total devices : 3  persistence : superblock is persistent    update time : wed dec 6 19:41:08 2017  state : clean, degraded        # 这里注意了。 我们的一个盘坏掉了。 raid5状态为降级使用了。  active devices : 2  working devices : 2  failed devices : 1  spare devices : 0    layout : left-symmetric  chunk size : 256k    consistency policy : resync  name : centos7.magedu.com:5 (local to host centos7.magedu.com)  uuid : 2c8ae60d:a799fcb7:9008a046:ae6ea430  events : 20    number major minor raiddevice state  0 8 17 0 active sync /dev/sdb1  - 0 0 1 removed  3 8 49 2 active sync /dev/sdd1  1 8 33 - faulty /dev/sdc1  [root@centos7 md5]$ cat a.txt      # 发现我们的数据还是能访问的。没有问题。

  [root@centos7 md5]$ mdadm /dev/md5 -a /dev/sde1   # 上面我们的sdc1数据损坏，我们需要更换新的磁盘来顶替他的位置。这里添加一个sde1的磁盘， fdisk操作这里省去了。  mdadm: added /dev/sde1  [root@centos7 md5]$ mdadm -ds      # 查看详细信息  array /dev/md5 metadata=1.2 name=centos7.magedu.com:5 uuid=2c8ae60d:a799fcb7:9008a046:ae6ea430  [root@centos7 md5]$ mdadm -d /dev/md5     # 查看详细信息  /dev/md5:  version : 1.2  creation time : wed dec 6 19:28:22 2017  raid level : raid5  array size : 2095104 (2046.00 mib 2145.39 mb)  used dev size : 1047552 (1023.00 mib 1072.69 mb)  raid devices : 3  total devices : 4  persistence : superblock is persistent    update time : wed dec 6 19:50:01 2017  state : clean       # 状态恢复正常了。没有问题  active devices : 3  working devices : 3  failed devices : 1  spare devices : 0    layout : left-symmetric  chunk size : 256k    consistency policy : resync  name : centos7.magedu.com:5 (local to host centos7.magedu.com)  uuid : 2c8ae60d:a799fcb7:9008a046:ae6ea430  events : 43    number major minor raiddevice state  0 8 17 0 active sync /dev/sdb1  4 8 65 1 active sync /dev/sde1  3 8 49 2 active sync /dev/sdd1    1 8 33 - faulty /dev/sdc1  # 这个盘是坏掉的，我们已经加入了新的磁盘， 这个盘可以干掉了  [root@centos7 md5]$ man mdadm  [root@centos7 md5]$ mdadm /dev/md5 --remove /dev/sdc1   # 这个盘我们从raid5中移除去。   mdadm: hot removed /dev/sdc1 from /dev/md5

  [root@centos7 mnt]$ mkadm -g -r /dev/md5 -n 4 -a /dev/sdxx   # 这里我就不测试了。使用/dev/sdxx代替一个设备。-g 是grown增长的意思，-r 是resizefs的意思，

  [root@centos7 mnt]$ umount /dev/md5     # 卸载设备  [root@centos7 mnt]$ mdadm -s /dev/md5     # 停止raid5   mdadm: stopped /dev/md5  [root@centos7 mnt]$ sed -i '$d' /etc/fstab    # 删除fstab中关于raid5挂载的行  [root@centos7 mnt]$ cat /etc/fstab     # 确保fstab没有大问题    #  # /etc/fstab  # created by anaconda on tue nov 7 16:07:01 2017  #  # accessible filesystems, by reference, are maintained under '/dev/disk'  # see man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info  #    uuid=59ccea87-3c4e-4bbc-9e2f-3fadb1dcf2e6 /   ext4 defaults 1 1  uuid=f4e867e8-bcde-43a2-adc7-c80b0948e85f /app   ext4 noatime,usrquota,grpquota 1 2  uuid=1d6cbe88-ffb4-4adf-bacf-76be1fa75708 /boot   ext4 defaults 1 2  #uuid=b2c064f5-1ee5-4b5c-9e75-ed41cb99c5aa swap   swap defaults 0 0  #uuid=a0516c4f-40e6-4919-905a-8b44db12ff7b swap  swap defaults,pri=0 0 0   #/dev/sdb2 /test ext4 rw,seclabel,relatime,data=ordered 0 0  #/dev/sdb1 /home xfs rw,seclabel,relatime,attr2,inode64,usrquota,grpquota 0 0  [root@centos7 mnt]$ rm -rf /etc/mdadm.conf     # 删除raid默认配置文件

  [root@centos7 mnt]$ mdadm --zero-superblock /dev/sd{b1,e1,d1,c1}  # 清空设置上的超级块信息

  [root@centos7 mnt]$ lsblk    # 就是使用fdisk 创建的设备， 具体这里就不写了。 最终使用lsblk显示，我们可以看到sdb1,sdb2,sdd1,sde1一共6个磁盘  name maj:min rm size ro type mountpoint  sda 8:0 0 200g 0 disk   ├─sda1 8:1 0 1g 0 part /boot  ├─sda2 8:2 0 128g 0 part   ├─sda3 8:3 0 48.8g 0 part /  ├─sda4 8:4 0 512b 0 part   └─sda5 8:5 0 19.5g 0 part /app  sdb 8:16 0 100g 0 disk   ├─sdb1 8:17 0 1g 0 part   └─sdb2 8:18 0 1g 0 part   sdc 8:32 0 20g 0 disk   ├─sdc1 8:33 0 1g 0 part   └─sdc2 8:34 0 1g 0 part   sdd 8:48 0 20g 0 disk   └─sdd1 8:49 0 1g 0 part   sde 8:64 0 20g 0 disk   └─sde1 8:65 0 1g 0 part   sdf 8:80 0 20g 0 disk   sr0 11:0 1 8.1g 0 rom /run/media/root/centos 7 x86_64

  [root@centos7 mnt]$ mdadm -c /dev/md11 -a yes -l 1 -n 2 /dev/sd{b1,c1}    # 创建第一个raid1  mdadm: /dev/sdb1 appears to be part of a raid array:   level=raid5 devices=3 ctime=wed dec 6 19:28:22 2017  mdadm: note: this array has metadata at the start and   may not be suitable as a boot device. if you plan to   store '/boot' on this device please ensure that   your boot-loader understands md/v1.x metadata, or use   --metadata=0.90  mdadm: /dev/sdc1 appears to be part of a raid array:   level=raid5 devices=3 ctime=wed dec 6 19:28:22 2017  continue creating array y  mdadm: defaulting to version 1.2 metadata  mdadm: array /dev/md11 started.  [root@centos7 mnt]$ mdadm -c /dev/md12 -a yes -l 1 -n 2 /dev/sd{b2,c2}    #创建第二个raid1  mdadm: note: this array has metadata at the start and   may not be suitable as a boot device. if you plan to   store '/boot' on this device please ensure that   your boot-loader understands md/v1.x metadata, or use   --metadata=0.90  continue creating array y  mdadm: defaulting to version 1.2 metadata  mdadm: array /dev/md12 started.  [root@centos7 mnt]$ mdadm -c /dev/md13 -a yes -l 1 -n 2 /dev/sd{d1,e1}     # 创建第三个raid1  mdadm: /dev/sdd1 appears to be part of a raid array:  level=raid5 devices=3 ctime=wed dec 6 19:28:22 2017  mdadm: note: this array has metadata at the start and  may not be suitable as a boot device. if you plan to  store '/boot' on this device please ensure that  your boot-loader understands md/v1.x metadata, or use  --metadata=0.90  mdadm: /dev/sde1 appears to be part of a raid array:  level=raid5 devices=3 ctime=wed dec 6 19:28:22 2017  continue creating array y  mdadm: defaulting to version 1.2 metadata  mdadm: array /dev/md13 started.  [root@centos7 mnt]$ mdadm -c /dev/md10 -a yes -l 0 -n 3 /dev/md{11,12,13}    # 将3个raid1 合并为一个raid0   mdadm: /dev/md11 appears to contain an ext2fs file system  size=2095104k mtime=wed dec 6 19:29:45 2017  mdadm: /dev/md13 appears to contain an ext2fs file system  size=2095104k mtime=wed dec 6 19:29:45 2017  continue creating array y  mdadm: defaulting to version 1.2 metadata  mdadm: array /dev/md10 started.  [root@centos7 mnt]$ mkfs.ext  mkfs.ext2 mkfs.ext3 mkfs.ext4    [root@centos7 mnt]$ mkfs.ext4 /dev/md10         # 创建文件系统              mke2fs 1.42.9 (28-dec-2013)  filesystem label=  os type: linux  block size=4096 (log=2)  fragment size=4096 (log=2)  stride=128 blocks, stripe width=384 blocks  196224 inodes, 784896 blocks  39244 blocks (5.00%) reserved for the super user  first data block=0  maximum filesystem blocks=805306368  24 block groups  32768 blocks per group, 32768 fragments per group  8176 inodes per group  superblock backups stored on blocks:   32768, 98304, 163840, 229376, 294912    allocating group tables: done      writing inode tables: done      creating journal (16384 blocks): done  writing superblocks and filesystem accounting information: done     [root@centos7 mnt]$ mdadm -ds          # 查看配置信息  array /dev/md11 metadata=1.2 name=centos7.magedu.com:11 uuid=0ce2cd6c:cd21fab6:3e65cfb5:64bd86f3  array /dev/md12 metadata=1.2 name=centos7.magedu.com:12 uuid=8af31dff:efab06ed:48e2613b:a599c774  array /dev/md13 metadata=1.2 name=centos7.magedu.com:13 uuid=a8c99d60:2d0c61e7:97a76809:9396c020  array /dev/md10 metadata=1.2 name=centos7.magedu.com:10 uuid=50b2fa58:4ce65d67:8c50c853:fa175a28  [root@centos7 mnt]$ mdadm -ds >> /etc/mdadm.conf        # 写配置文件到mdadm的配置文件中  [root@centos7 mnt]$ mkdir /mnt/md10         # 创建挂载目录  [root@centos7 mnt]$ mount /dev/md10 /mnt/md10        # 挂载文件系统  [root@centos7 mnt]$ tail -n 1 /etc/mtab         # 查看mtab文件中的最后一行， 也就是我们的md10挂载信息  /dev/md10 /mnt/md10 ext4 rw,seclabel,relatime,stripe=384,data=ordered 0 0  [root@centos7 mnt]$ tail -n 1 /etc/mtab >> /etc/fstab       #添加到开机启动

  [root@centos7 mnt]$ umount /dev/md10         # 取消挂载  [root@centos7 mnt]$ rm -rf /etc/mdadm.conf        # 删除mdadm的默认配置  [root@centos7 mnt]$ mdadm -s /dev/md10         # 停止raid0设置  mdadm: stopped /dev/md10  [root@centos7 mnt]$ mdadm -s /dev/md11         # 停止raid1设置  mdadm: stopped /dev/md11  [root@centos7 mnt]$ mdadm -s /dev/md12         # 停止radi1 设置  mdadm: stopped /dev/md12   [root@centos7 mnt]$ mdadm -s /dev/md13         # 停止raid 1 设置  mdadm: stopped /dev/md13  [root@centos7 mnt]$ sed -i '$d' /etc/fstab        # 删除fstab的挂载   [root@centos7 mnt]$ cat /etc/fstab         # 确保正确    #  # /etc/fstab  # created by anaconda on tue nov 7 16:07:01 2017  #  # accessible filesystems, by reference, are maintained under '/dev/disk'  # see man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info  #    uuid=59ccea87-3c4e-4bbc-9e2f-3fadb1dcf2e6 /   ext4 defaults 1 1  uuid=f4e867e8-bcde-43a2-adc7-c80b0948e85f /app   ext4 noatime,usrquota,grpquota 1 2  uuid=1d6cbe88-ffb4-4adf-bacf-76be1fa75708 /boot   ext4 defaults 1 2  #uuid=b2c064f5-1ee5-4b5c-9e75-ed41cb99c5aa swap   swap defaults 0 0  #uuid=a0516c4f-40e6-4919-905a-8b44db12ff7b swap  swap defaults,pri=0 0 0   #/dev/sdb2 /test ext4 rw,seclabel,relatime,data=ordered 0 0  #/dev/sdb1 /home xfs rw,seclabel,relatime,attr2,inode64,usrquota,grpquota 0 0  [root@centos7 mnt]$ mdadm -d           # 再次查看下mdadm信息，确保没有了  mdadm: no devices given.  [root@centos7 mnt]$ mdadm --zero-superblock /dev/sd{b1,b2,c1,c2,d1,e1}     # 请求md的元数据信息