Prolog

I created this as a reference / CheatSheet for installing an OS on ZFS root. In other words; I needed a quick reference to create a bootable OS install on ZFS. This serves as a quick reference. As unless your daily job is to install Operating Systems. You're not likely to need to remember all this stuff. So this article serves for those times when you do need to remember. Much of this article draws from an article written by Miroslav Lachman on the FreeBSD wiki

NOTE: This applies to FreeBSD version 11, and forward (newer). On the so-called AMD64 (x86_64) platform.

1. Creating a bootable ZFS Filesystem
2. Installing FreeBSD to the ZFS filesystem
3. Finish install

1. Creating a bootable ZFS Filesystem

1. Boot FreeBSD install DVD or USB Memstick

2. Choose Live CD option in sysinstall

3. Create GPT Disks:

live# gpart create -s gpt ad0 live# gpart create -s gpt ad1

4. Create the boot, swap, and zfs partitions

Create 3 partitions on both drives ad0 and ad1. The first partition contains the gptzfsboot loader which is able to recognize and load the loader from a ZFS partition. The second partition is a 4 GB swap partition. The third partition is the partition containing the zpool (60GB).

live# gpart add -b 34 -s 512k -t freebsd-boot ad0 live# gpart add -s 4G -t freebsd-swap -l swap0 ad0 live# gpart add -s 60G -t freebsd-zfs -l disk0 ad0 live# live# gpart add -b 34 -s 512k -t freebsd-boot ad1 live# gpart add -s 4G -t freebsd-swap -l swap1 ad1 live# gpart add -s 60G -t freebsd-zfs -l disk1 ad1



While a ZFS Swap Volume can be used instead of the freebsd-swap partition, crash dumps can't be created on the ZFS Swap Volume.
Sizes and offsets are specified in sectors by default (1 sector is typically 512 bytes) but can have M or G suffixes for MB and GB.

5. Install the Protective MBR (pmbr) and gptzfsboot loader to both drives

live# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad0 live# gpart bootcode -b /mnt2/boot/pmbr -p /mnt2/boot/gptzfsboot -i 1 ad1

If either of the previous commands return Operation not permitted. Issue the following:

live# sysctl kern.geom.debugflags=0x10

this error can occur while the kernel is attempting to protect the critical areas of the disk.

6. Load ZFS kernel module

live# kldload /mnt2/boot/kernel/opensolaris.ko live# kldload /mnt2/boot/kernel/zfs.ko

7. Create ZFS Pool zroot

live# mkdir /boot/zfs live# zpool create zroot mirror /dev/gpt/disk0 /dev/gpt/disk1 live# zpool set bootfs=zroot zroot



NOTE: zroot is the name of the ZFS Pool However, you may choose any name you like. You should choose a name most appropriate for your intended goal/usage.

2. A_Installing_FreeBSD_to_the_ZFS_filesystem

1. Create ZFS filesystem hierarchy

The fletcher4 algorithm should be more robust than the fletcher2 algorithm.

live# zfs set checksum=fletcher4

live# zfs create -o compression=on -o exec=on -o setuid=off zroot/tmp live# chmod 1777 /zroot/tmp

live# zfs create zroot/usr live# zfs create zroot/usr/home live# cd /zroot ; ln -s /usr/home home



NOTE: If you use nullfs or nfs to mount /usr/ports to different locations/servers. You will also need to nullfs/nfs mount /usr/ports/distfiles and/or /usr/ports/packages.

live# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/usr/src

live# zfs create zroot/var live# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/crash live# zfs create -o exec=off -o setuid=off zroot/var/db live# zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/db/pkg live# zfs create -o exec=off -o setuid=off zroot/var/empty live# zfs create -o compression=lzjb -o exec=off -o setuid=off zroot/var/log live# zfs create -o compression=gzip -o exec=off -o setuid=off zroot/var/mail live# zfs create -o exec=off -o setuid=off zroot/var/run live# zfs create -o compression=lzjb -o exec=on -o setuid=off zroot/var/tmp live# chmod 1777 /zroot/var/tmp



NOTE: Compression may be set to on, off, lzjb, gzip, gzip-N (where N is an integer from 1 (fastest) to 9 (best compression ratio. gzip is equivalent to gzip-6).
Compression will cause some latency when accessing files on the ZFS filesystems. Use compression on ZFS filesystems which will not be accessed that often.

2. Install FreeBSD to zroot

live# cd /dist/8.0-* live# export DESTDIR=/zroot live# for dir in base catpages dict doc games info lib32 manpages ports; \ do (cd $dir ; ./install.sh) ; done live# cd src ; ./install.sh all live# cd ../kernels ; ./install.sh generic live# cd /zroot/boot ; cp -Rlp GENERIC/* /zroot/boot/kernel/

3. Make /var/empty readonly

live# zfs set readonly=on zroot/var/empty

4. chroot into /zroot

live# chroot /zroot

5. Create /etc/rc.conf

live# echo 'zfs_enable="YES"' > /etc/rc.conf live# echo 'hostname="beastie.mydomain.local"' >> /etc/rc.conf live# echo 'ifconfig_re0="DHCP"' >> /etc/rc.conf



NOTE: Replace re0 with the name of the your Network interface on the new system

6. Create /boot/loader.conf

live# echo 'zfs_load="YES"' > /boot/loader.conf live# echo 'vfs.root.mountfrom="zfs:zroot"' >> /boot/loader.conf

7. Change root's password

live# passwd

8. Add a user

live# adduser

9. edit /etc/mail/aliases aliasing the user you just added to root

live# ee /etc/mail/aliases

10. Set the local time zone

live# tzsetup

11. Exit from the /zroot

live# umount /dev live# exit

12. Install zpool.cache to the ZFS filesystem

live# cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

Finish install

1. Using swap

There are 2 ways to use the gpt/swap0 and gpt/swap1 partitions

1. Create /etc/fstab to use both swap partitions

   live# cat << EOF > /zroot/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/gpt/swap0 none swap sw 0 0 /dev/gpt/swap1 none swap sw 0 0 EOF

2. Use gmirror to mirror the swap partitions
   1. Load the gmirror kernel module

      live# kldload /mnt2/boot/kernel/geom_mirror.ko

   2. Create the gmirror swap

      live# gmirror label -b prefer swap gpt/swap0 gpt/swap1

      
      NOTE: The prefer balance algorithm can be replaced by round-robin. See the gmirror(8) man page about problem using the round-robin balance algorithm and kernel dumps.
   3. Create /etc/fstab to use the gmirror swap partition

      live# cat << EOF > /zroot/etc/fstab # Device Mountpoint FStype Options Dump Pass# /dev/mirror/swap none swap sw 0 0 EOF

   4. Add gmirror to /boot/loader.conf

      live# echo 'geom_mirror_load="YES"' >> /zroot/boot/loader.conf

3. export LD_LIBRARY_PATH

   live# export LD_LIBRARY_PATH=/mnt2/lib

4. Unmount all zfs filesystems

   live# zfs unmount -a

5. Change mount points for zroot pool

   live# zfs set mountpoint=legacy zroot live# zfs set mountpoint=/tmp zroot/tmp live# zfs set mountpoint=/usr zroot/usr live# zfs set mountpoint=/var zroot/var

6. Exit Live CD mode and sysinstall. Then remove the FreeBSD install DVD/Memstick
   The system will now reboot into the new ZFS root.

See also:
BSD forge
Damn Small BSD
Man Pages