๊ธฐ์กด posting http://hwengineer.blogspot.com/2017/03/ppc64le-power8-ubuntu-sw-raid.html ์—์„œ๋Š” Ubuntu ppc64le ํ™˜๊ฒฝ์—์„œ mdadm device (SW RAID1) ๊ตฌ์„ฑ์„ ํ†ตํ•ด 2์žฅ์˜ ๋‚ด์žฅ disk์— OS mirror๋ฅผ ๊ตฌ์„ฑํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ์†Œ๊ฐœ์‹œ์ผœ ๋“œ๋ ธ์Šต๋‹ˆ๋‹ค. ์•„์‹œ๋‹ค์‹œํ”ผ x86 ์•„ํ‚คํ…์ฒ˜์™€๋Š” ๋‹ฌ๋ฆฌ ppc64le์—๋Š” boot loader๋ฅผ ๋ณด๊ด€ํ•˜๋Š” PReP์ด๋ผ๋Š” ํŒŒํ‹ฐ์…˜์ด ๋””์Šคํฌ ๋งจ ์•ž ๋ถ€๋ถ„์— ์กด์žฌํ•ด์•ผ ํ•˜๋Š”๋ฐ, ์ด๊ฒƒ์€ mdadm์œผ๋กœ๋Š” mirror๊ฐ€ ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ์ด boot loader๋Š” ํ•œ๋ฒˆ ์„ค์น˜๋˜๋ฉด ๊ฑฐ๊ธฐ์— ์ถ”๊ฐ€๋กœ write๋  ์ผ์ด ์—†์œผ๋ฏ€๋กœ, dd ๋ช…๋ น์–ด๋ฅผ ํ†ตํ•ด source์—์„œ target์œผ๋กœ copy๋งŒ ๋–  ๋†“์œผ๋ฉด OS mirror๋ฅผ ์‚ฌ์šฉํ•˜๋Š”๋ฐ ์ง€์žฅ์ด ์—†์Šต๋‹ˆ๋‹ค. ๊ทธ๋ฅผ ์œ„ํ•ด์„œ ์‚ฌ์šฉํ•œ ์ž‘์€ trick์€, OS ์„ค์น˜์‹œ ๋จผ์ € ์–‘์ชฝ disk์— PReP์„ ํฌํ•จํ•œ ๋ชจ๋“  ํŒŒํ‹ฐ์…˜์„ ๋งŒ๋“ค์–ด๋†“๋˜, mdadm device๋Š” PReP์„ ๋บธ ๋‚˜๋จธ์ง€ device์— ๋Œ€ํ•ด์„œ๋งŒ ๊ฑธ์–ด๋‘๋Š” ๊ฒƒ์ด์—ˆ์Šต๋‹ˆ๋‹ค.

์ด์ œ POWER9๊ณผ V100์„ ์žฅ์ฐฉํ•œ AC922 ์„œ๋ฒ„๋Š” ํ˜„์žฌ๊นŒ์ง€๋Š” ์ฃผ๋กœ Redhat์„ ์ด์šฉํ•˜๋Š”๋ฐ, Redhat์—์„œ๋„ ์ด๋Ÿฐ OS mirror๊ฐ€ ๊ฐ€๋Šฅํ• ๊นŒ์š” ? ๋ถˆํ–‰ํžˆ๋„ Redhat installer๋Š” Ubuntu installer์™€๋Š” ์•ฝ๊ฐ„ ๋‹ฌ๋ผ์„œ, OS ์„ค์น˜์‹œ ๊ฐ๊ฐ์˜ disk์— ๋จผ์ € ํŒŒํ‹ฐ์…˜์„ ๋งŒ๋“  ๋’ค ์–ด๋Š ํŒŒํ‹ฐ์…˜์„ ๋ฌด์—‡์œผ๋กœ ์‚ฌ์šฉํ• ์ง€ ์ •ํ•˜๋Š” ๊ฒƒ์ด ์•„๋‹ˆ๋ผ, ๋จผ์ € ํŒŒํ‹ฐ์…˜ ์šฉ๋„๋ฅผ ์ง€์ •ํ•˜๋ฉด ์ž๋™์œผ๋กœ mirror device๊ฐ€ ์ƒ์„ฑ๋ฉ๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ SW RAID1์œผ๋กœ ๊ทธ๋ƒฅ ์„ค์น˜ํ•˜๋ฉด PReP๊ณผ /boot๋Š” 1๋ฒˆ disk์—๋งŒ ๋งŒ๋“ค์–ด์ง€๊ณ , 2๋ฒˆ ๋””์Šคํฌ ์•ž ๋ถ€๋ถ„์—๋Š” PReP ํŒŒํ‹ฐ์…˜์„ ๋งŒ๋“ค ๊ณต๊ฐ„์„ ๋”ฐ๋กœ ํ™•๋ณดํ•  ์ˆ˜ ์—†๊ฒŒ ๋ฉ๋‹ˆ๋‹ค.

์ด ๋ฌธ์ œ์— ๋Œ€ํ•œ workaround๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด ๋ฐฉ๋ฒ•์€ Kolon์˜ ์ •์œ ์˜ ๋Œ€๋ฆฌ๊ป˜์„œ ๊ณ ์•ˆํ•ด๋ƒˆ๊ณ , ํ…Œ์ŠคํŠธ์—์„œ 1๋ฒˆ disk๋ฅผ ๋ฝ‘์•„๋‚ธ ๋’ค ๋ฆฌ๋ถ€ํŒ…์„ ํ•ด๋ด๋„ ์ •์ƒ์ ์œผ๋กœ ๋ฆฌ๋ถ€ํŒ…์ด ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ์ˆ˜๊ณ ํ•ด์ฃผ์‹  ์ •์œ ์˜ ๋Œ€๋ฆฌ๊ป˜ ๊ฐ์‚ฌ๋“œ๋ฆฝ๋‹ˆ๋‹ค.

์ด workaround์˜ ์ ˆ์ฐจ๋ฅผ ์š”์•ฝํ•˜๋ฉด ์•„๋ž˜์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

1) ๋จผ์ € Redhat installer๊ฐ€ ์‹œํ‚ค๋Š” ๋Œ€๋กœ, PReP๊ณผ /boot๋ฅผ ์ œ์™ธํ•œ OS mirror๋ฅผ mdadm device๋ฅผ ๊ตฌ์„ฑํ•˜์—ฌ ์™„๋ฃŒํ•ฉ๋‹ˆ๋‹ค.
2) OS ์„ค์น˜ ์™„๋ฃŒ ๋’ค mdadm ๋ช…๋ น์œผ๋กœ 2๋ฒˆ ๋””์Šคํฌ์˜ mirror copy๋ฅผ ์ œ๊ฑฐํ•˜์—ฌ blank disk๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
3) ๊ทธ๋ ‡๊ฒŒ ๋น„์›Œ์ง„ 2๋ฒˆ disk์— 1๋ฒˆ disk์™€ ๋™์ผํ•œ ํฌ๊ธฐ๋กœ PReP๊ณผ /boot๋ฅผ ํฌํ•จํ•œ ํŒŒํ‹ฐ์…˜๋“ค์„ ์ƒˆ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
4) ๊ทธ ์ค‘ PReP๊ณผ /boot์—๋Š” dd ๋ช…๋ น์„ ํ†ตํ•ด 1๋ฒˆ disk์˜ ์›๋ณธ์„ copyํ•ฉ๋‹ˆ๋‹ค.
5) ๋‚˜๋จธ์ง€ OS์šฉ ํŒŒํ‹ฐ์…˜์—๋Š” mdadm ๋ช…๋ น์œผ๋กœ mirror copy๋ฅผ ์ƒˆ๋กœ add ํ•ฉ๋‹ˆ๋‹ค.

๊ฐ„๋‹จํ•˜์ง€์š” ? ๋‹ค๋งŒ ์ด๋ ‡๊ฒŒ ํ•  ๊ฒฝ์šฐ, ๋‚˜์ค‘์— ํ˜น์‹œ /boot์— ๋ญ”๊ฐ€ write๋ฅผ ํ•  ์ผ์ด ์žˆ๋‹ค๋ฉด ๊ทธ ๋‚ด์šฉ์„ 2๋ฒˆ์งธ disk์— ๋‹ค์‹œ dd ๋ช…๋ น์œผ๋กœ write๋ฅผ ํ•ด์ค˜์•ผ ํ•˜๊ธด ํ•ฉ๋‹ˆ๋‹ค. ๋ฌผ๋ก  /boot์—๋Š” write๊ฐ€ ์ผ์–ด๋‚  ์ผ์€ ๊ฑฐ์˜ ์—†์Šต๋‹ˆ๋‹ค. ์ด์ œ ์œ„์˜ ๋‚ด์šฉ์„ ์ข€ ๋” ์ž์„ธํžˆ ์ •๋ฆฌํ•ด๋†“๊ฒ ์Šต๋‹ˆ๋‹ค.

1. 2๊ฐœ์˜ ๋””์Šคํฌ์ค‘ “/”์™€ “swap” ๋ถ€๋ถ„์„ mdadm device (์ฆ‰ SW RAID1)์œผ๋กœ ๊ตฌ์„ฑ ํ•ด์„œ ์„ค์น˜ํ•ฉ๋‹ˆ๋‹ค. ์ด๋•Œ PReP๊ณผ /boot๋Š” ์ œ์™ธ๋ฉ๋‹ˆ๋‹ค.
– PReP์€ ๊ทธ๋ ‡๋‹ค์น˜๊ณ , ์™œ /boot๋„ mdadm device๋กœ ๊ตฌ์„ฑ์ด ์•ˆ๋˜๋Š”์ง€๋Š” ์˜์•„ํ•˜๊ธด ํ•ฉ๋‹ˆ๋‹ค.

2. OS ์„ค์น˜ ์™„๋ฃŒ ํ›„ OS ๋ช…๋ น์–ด mdadm๋ฅผ ํ†ตํ•ด MD device๋กœ ๊ตฌ์„ฑ๋œ /์™€ swap์—์„œ 2๋ฒˆ ๋””์Šคํฌ๋ฅผ ์ œ๊ฑฐ

# mdadm –manage /dev/md126 –remove /dev/sdb1 (swap ์˜์—ญ)
# mdadm –manage /dev/md127 –remove /dev/sdb2 (/ ์˜์—ญ)

3. ์ด์ œ ํ…… ๋น„๊ฒŒ ๋œ 2๋ฒˆ ๋””์Šคํฌ๋ฅผ 1๋ฒˆ ๋””์Šคํฌ์™€ ๋™์ผํ•˜๊ฒŒ fdisk ๋ช…๋ น์„ ์ด์šฉํ•ด์„œ ํŒŒํ‹ฐ์…”๋‹

# fdisk -l /dev/sda (1๋ฒˆ disk์˜ ํŒŒํ‹ฐ์…˜ ์ƒํ™ฉ ํŒŒ์•…)
# fdisk /dev/sdb (์œ„์˜ ์ •๋ณด์—์„œ ์–ป์€ 1๋ฒˆ disk ํŒŒํ‹ฐ์…˜๊ณผ ๋™์ผํ•˜๊ฒŒ 2๋ฒˆ disk์— ํŒŒํ‹ฐ์…˜์„ ๊ตฌ์„ฑ)
– ํŠนํžˆ ๋งจ ์•ž์— ๋งŒ๋“œ๋Š” /dev/sdb1์˜ disk type์€ 41 (PPC PReP Boot)์œผ๋กœ ๊ตฌ์„ฑํ•ด์ค˜์•ผ ํ•ฉ๋‹ˆ๋‹ค.
# fdisk /dev/sda

Command (m for help): t
Hex code (type L to list all codes): 41
Changed type of partition ‘Linux’ to ‘PPC PReP Boot’

๊ทธ๋ž˜์„œ ๋‹ค์Œ๊ณผ ๊ฐ™์ด ๋‘ disk๊ฐ€ ๋™์ผํ•œ ํŒŒํ‹ฐ์…˜ ์ •๋ณด๋ฅผ ๊ฐ€์ง€๋„๋ก ๋งŒ๋“ญ๋‹ˆ๋‹ค.

Device Boot Start End Blocks Id System
/dev/sda1 * 2048 18431 8192 41 PPC PReP Boot
/dev/sda2 18432 1067007 524288 83 Linux
/dev/sda3 1067008 9455615 4194304 82 Linux swap / Solaris
/dev/sda4 9455616 1875384319 932964352 5 Extended
/dev/sda5 9457664 1875384319 932963328 8e Linux LVM

Device Boot Start End Blocks Id System
/dev/sdb1 2048 18431 8192 41 PPC PReP Boot
/dev/sdb2 18432 1067007 524288 83 Linux
/dev/sdb3 1067008 9455615 4194304 82 Linux swap / Solaris
/dev/sdb4 9455616 1875384319 932964352 5 Extended
/dev/sdb5 9457664 1875384319 932963328 8e Linux LVM

4. ์ด์ œ mdadm ๋ช…๋ น์œผ๋กœ /์™€ swap ํŒŒํ‹ฐ์…˜์˜ mirror ๊ตฌ์„ฑ์„ ์‹ค์‹œํ•ฉ๋‹ˆ๋‹ค.

# mdadm –manage /dev/md126 –add /dev/sdb3 (swap ์˜์—ญ)
# mdadm –manage /dev/md127 –add /dev/sdb5 (/ ์˜์—ญ)

5. PReP ํŒŒํ‹ฐ์…˜์„ dd ๋ช…๋ น์œผ๋กœ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค.

# dd if=/dev/sda1 of=/dev/sdb1 bs=1024k

6. /boot ํŒŒํ‹ฐ์…˜์„ dd ๋ช…๋ น์œผ๋กœ ๋ณต์‚ฌํ•ฉ๋‹ˆ๋‹ค. ๋‹จ, ์ด ๊ฒฝ์šฐ๋Š” ๋จผ์ € ์‹ ๊ทœ /boot์— filesystem์„ ๋งŒ๋“ค๊ณ , ๋˜ ๊ธฐ์กด /boot๋ฅผ umountํ•˜๊ณ  ์ง„ํ–‰ํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

๋จผ์ € ๊ธฐ์กด /boot๊ฐ€ ์–ด๋–ค type์˜ filesystem์œผ๋กœ ๋งŒ๋“ค์–ด์กŒ๋Š”์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ์—ฌ๊ธฐ์„œ๋Š” xfs๋กœ ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.

# mount | grep boot
/dev/sda2 on /boot type xfs (rw,relatime,attr2,inode64,noquota)

๊ฑฐ๊ธฐ์— ๋Œ€์‘ํ•ด์„œ ๋™์ผ size๋กœ ๋งŒ๋“ค์–ด ๋‘์—ˆ๋˜ /dev/sdb2๋ฅผ xfs๋กœ format ํ•ฉ๋‹ˆ๋‹ค. (์ด format superblock์€ dd๋กœ copy๊ฐ€ ์•ˆ๋˜๋”๋ผ๊ณ ์š”)

# mkfs -t xfs -f /dev/sdb2

๊ทธ ๋‹ค์Œ์— ๋ฐ˜๋“œ์‹œ ๋จผ์ € ๊ธฐ์กด์˜ /boot๋ฅผ umount ํ•˜๊ณ  dd๋ฅผ ๋œน๋‹ˆ๋‹ค. (mount๋œ ์ƒํƒœ์—์„œ dd๋ฅผ ๋œจ๋ฉด superblock์ด copy ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.)

# umount /boot
# dd if=/dev/sda2 of=/dev/sdb2 bs=1024k

์ด์ œ ๋ณด๋ฉด xfs_admin ๋ช…๋ น์œผ๋กœ ํ™•์ธํ•ด๋ณด๋ฉด /dev/sdb2์—๋„ /dev/sda2์™€ ๋™์ผํ•œ UUID๊ฐ€ ๋“ค์–ด๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด๋ ‡๊ฒŒ ๋™์ผํ•œ UUID๋ฅผ ์‚ฌ์šฉํ•˜๋ฏ€๋กœ, /boot/grub2/grub.cfg๋ฅผ ๊ตณ์ด ์ˆ˜์ •ํ•  ํ•„์š”๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ์ด๋กœ ์ธํ•ด ์„œ๋ฒ„ ๋ถ€ํŒ… ์‹œ์— /boot๋กœ /dev/sda2๊ฐ€ ์•„๋‹ˆ๋ผ /dev/sdb2๊ฐ€ mount๋  ๊ฐ€๋Šฅ์„ฑ๋„ ์žˆ๋Š”๋ฐ, ์–ด์ฐจํ”ผ ๊ฐ™์€ copy์ด๊ณ  ๊ฑฐ๊ธฐ์— write๊ฐ€ ์ผ์–ด๋‚  ์ผ๋„ ๊ฑฐ์˜ ์—†์œผ๋ฏ€๋กœ ๋ฌธ์ œ๋  ๊ฒƒ์€ ์—†์Šต๋‹ˆ๋‹ค.

# xfs_admin -u /dev/sda2
UUID = 20a94402-8589-48db-b8cf-e82c16f415ee

# xfs_admin -u /dev/sdb2
UUID = 20a94402-8589-48db-b8cf-e82c16f415ee

๊ทธ๋ฆฌ๊ณ  mount ํ•ด์„œ ๋ณด๋ฉด ์›๋ž˜์˜ /boot์™€ ๋™์ผํ•œ file๋“ค์ด ๋™์ผํ•˜๊ฒŒ ๋ณด์ž…๋‹ˆ๋‹ค.

# mount /dev/sda2 /boot
# mount /dev/sdb2 /mnt

# ls -l /boot
total 183444
-rw-r–r–. 1 root root 138001 Oct 17 2017 config-4.11.0-44.el7a.ppc64le
drwxr-xr-x. 3 root root 17 May 30 14:58 efi
drwx——. 5 root root 118 May 30 17:11 grub2
-rw——-. 1 root root 64932747 May 30 15:07 initramfs-0-rescue-168b61eb0f2d43cb8783c83d162b35f9.img
-rw——-. 1 root root 44154461 May 30 17:11 initramfs-4.11.0-44.el7a.ppc64le.img
-rw——-. 1 root root 22527454 May 30 15:27 initramfs-4.11.0-44.el7a.ppc64lekdump.img
-rw-r–r–. 1 root root 10382937 May 30 15:06 initrd-plymouth.img
-rw-r–r–. 1 root root 250055 Oct 17 2017 symvers-4.11.0-44.el7a.ppc64le.gz
-rw——-. 1 root root 2754018 Oct 17 2017 System.map-4.11.0-44.el7a.ppc64le
-rwxr-xr-x. 1 root root 21346760 May 30 15:07 vmlinuz-0-rescue-168b61eb0f2d43cb8783c83d162b35f9
-rwxr-xr-x. 1 root root 21346760 Oct 17 2017 vmlinuz-4.11.0-44.el7a.ppc64le

# ls -l /mnt
total 183444
-rw-r–r–. 1 root root 138001 Oct 17 2017 config-4.11.0-44.el7a.ppc64le
drwxr-xr-x. 3 root root 17 May 30 14:58 efi
drwx——. 5 root root 118 May 30 17:11 grub2
-rw——-. 1 root root 64932747 May 30 15:07 initramfs-0-rescue-168b61eb0f2d43cb8783c83d162b35f9.img
-rw——-. 1 root root 44154461 May 30 17:11 initramfs-4.11.0-44.el7a.ppc64le.img
-rw——-. 1 root root 22527454 May 30 15:27 initramfs-4.11.0-44.el7a.ppc64lekdump.img
-rw-r–r–. 1 root root 10382937 May 30 15:06 initrd-plymouth.img
-rw-r–r–. 1 root root 250055 Oct 17 2017 symvers-4.11.0-44.el7a.ppc64le.gz
-rw——-. 1 root root 2754018 Oct 17 2017 System.map-4.11.0-44.el7a.ppc64le
-rwxr-xr-x. 1 root root 21346760 May 30 15:07 vmlinuz-0-rescue-168b61eb0f2d43cb8783c83d162b35f9
-rwxr-xr-x. 1 root root 21346760 Oct 17 2017 vmlinuz-4.11.0-44.el7a.ppc64le

ํ™•์ธ์ด ๋๋‚ฌ์œผ๋ฉด /mnt๋Š” umountํ•˜๊ณ  ์‚ฌ์šฉ์„ ์‹œ์ž‘ํ•˜์‹œ๋ฉด ๋ฉ๋‹ˆ๋‹ค.

# umount /mnt

์ด๋กœ์จ OS mirroring์ด ๋‹ค ๋๋‚œ ๊ฒƒ์ž…๋‹ˆ๋‹ค.

ํ† ๋ก  ์ฐธ๊ฐ€

์ด๋ฉ”์ผ์€ ๊ณต๊ฐœ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํ•„์ˆ˜ ์ž…๋ ฅ์ฐฝ์€ * ๋กœ ํ‘œ์‹œ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.