Author: James Powell <james (at) hotmail (dot) com>

Date: 2014-08-14

License: MIT

Synopsis: Using ZFS with Linux From Scratch

===============================================================================

The MIT License

Copyright (c) 2014 James Powell

Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

===============================================================================

Description:

This document is provided to demonstrate how to effectively use the ZFS File
system using ZFS-On-Linux as the primary /(root) partition. ZFS is a highly
advanced file system with many features not normally found on GNU/Linux,
including the BtrFS file system which is still considered experimental and
unstable, even though it's been supported in the Linux kernel since 2.6.xx.
ZFS is licensed under the CDDL, which unfortunately in NOT compatible with the
GNU's GPL license the Linux kernel is under. This means that you can not get
a premade GNU/Linux distribution with built-in ZFS support.

Linux From Scratch, however, is an odd exemption from this. Because LFS is not
a prebuilt distribution, LFS falls outside the legal status of GPL and CDDL
problematics into a sort of grey area.

ZFS-On-Linux was developed by Lawrence Livermoore National Laboratories for
their own usage under the umbrella of the OpenZFS Project sponsored by the
Illumos Foundation for the Illumos Kernel and the OpenIndiana operating system.
The OpenZFS project is under a full disclosure license agreement and OpenZFS
is a fully open source project. The only implementation not sponsored or used
by OpenZFS is the Oracle-ZFS implementation which is closed source, and has
been developed separately from OpenZFS.

ZFS-On-Linux currently uses zpool version 5000 and fs version 5.

This software hint is considered experimental, so please use at your own risk
on a rightful system you are willing to sacrifice if needed.

===============================================================================

Prerequisites:

You must first run and assign partitioning using fdisk, cfdisk, gdisk
or any other partitioning tool.

Make sure you assign the following such as this example:

/dev/sda1: Primary Partition - 83 Partition type (Linux) - At least 512KB in
                                                           size.

This will serve at the primary /boot partition to where you will be installing
the Linux kernel and boot loader.

/dev/sda2: Primary Partition - BF Partition Type (Solaris) - Remaining disk
                                                             space.

This will serve as the ZFS partition space as well as the swap space.

You must now have obtained the Solaris Porting Layer (SPL) and ZFS-On-Linux
packages for your distribution, and have installed them.

NOTE: At this point, it does not matter whether ZFS is built-into or installed
      as a module, you just need to have it installed.

===============================================================================

Hint:

For the Remainder of the hint any partitions used will be examples only. Please
adjust partition values as needed.

First you must create a zpool to setup where the zfs-root will go. To perform
this action run the following command:

  zpool create -m none zfs-root /dev/sda2

This will create a blank zpool in the Solaris assigned disk space for ZFS to
work with. At this point, the partition is very useless for LFS, so we'll need
to make some adjustments to let LFS self manage things.

Let's now set the mountpoint.

First run this command:

  zfs get mountpoint zfs-root
  
The result should be at first it shows ZFS has no mountpoint. Because we want
the native mount tools to work with ZFS rather than the proprietary mount tools
we'll change the mountpoint:

  zfs set mountpoint=legacy zfs-root
  
Legacy allows ZFS to be mounted and unmounted using the traditional mount and
umount commands.

If you would like to see all the options of your zfs partition run:

  zfs get all zfs-root

Now let's mount our brand new ZFS partition.

  mount -v zfs-root -t zfs $LFS

Just like in Chapter 2: Mounting the New Partition, this will mount the ZFS
partition normally. To unmount it, simply run:

  umount -v $LFS
  
And the partition is unmounted just like a normal EXT*, JFS, or ReiserFS file
system.

Now as normal, create a boot partition:

  mkfs.ext4 /dev/sda1
  
ZFS isn't always boot friendly with various operating systems. Some like
FreeBSD can boot to ZFS, but not all can. Because various Linux bootloaders
like Grub don't always support every single file system, unless rebuilt for it,
which you can do with Grub, you'll need a boot partition.

ZFS doesn't allow swapfiles, but you can use a ZFS volume as swap. Let's create
a swap space now using a zvol

  zfs create -V 8G -b $(getconf PAGESIZE) \
	    -o primarycache=metadata \
	    -o sync=always \
	    -o com.sun:auto-snapshot=false zfs-root/swap
	   
In truth, you should not require more than 8GB swap space on any computer, for
any reason of usage. However, unlike a traditional file system, you can always
use the zfs tools to add or reduce swap space as needed.
           
Now we'll set the swap up like normal but using the zvol as the target:

  mkswap -f /dev/zvol/zfs-root/swap
  
Now let's turn on the swap partition:

  swapon /dev/zvol/zfs-root/swap
  
Now you should be able to build your system normally. When unmounting
everything should remount as normal, if not, that means trouble happened and
you should check your zpool status such as:

  zpool status zfs-root
  
When you set your fstab, you should now take care about how you setup the ZFS
parition and swap. ZFS uses these types of entries:

  zfs-root / zfs defaults,atime,dev,exec,suid,xattr,nomand,zfsutil 0 0
  /dev/zvol/zfs-root/swap none swap discard 0 0
  /dev/sda1 /boot ext4 defaults 0 2
  
This will setup the swap with discard feature. This will reduce fragmentation
levels in the swap area if it's not full. ZFS supports a vast deal of options
for the file system, as you can see, these are the recommended defaults.

Now when you first boot a system with ZFS, you'll notice that the mount tools
may complain about fsck.zfs missing. To be honest, there is no fsck.zfs utility
period. ZFS has it's own utility, but it's best used offline. To make sure you
don't get complaints on boot, let's make a stubfile:

  cat > /sbin/fsck.zfs << "EOF"
  #!/bin/sh
  exit 0
  "EOF"
  
That should keep the error read outs to a minimum when booting.

Now let's talk about the kernel. Becasue I, personally, recommend you use a
kernel with built-in modules, this next step will focus on just that.

Let's first get the SPL (Solaris Porting Layer) sources:

  http://archive.zfsonlinux.org/downloads/zfsonlinux/spl/spl-0.6.3.tar.gz
  
and the ZFS sources:

  http://archive.zfsonlinux.org/downloads/zfsonlinux/zfs/zfs-0.6.3.tar.gz
  
Save these into /sources where you've stored everything else.

Now unpack the SPL and ZFS sources, and we'll start with getting SPL ready:

First go to the kernel sources and run the following against an existing kernel
source that's been configured:

  make prepare scripts
  
This should prepare the kernel for a source merge. Now change to the spl
directory and get it prepped.

  ./configure \
      --prefix=/ \
      --libdir=/lib \
      --includedir=/usr/include \
      --datarootdir=/usr/share \
      --enable-linux-builtin=yes \
      --with-linux=/usr/src/linux-3.15.6 \
      --with-linux-obj=/usr/src/linux-3.15.6
  ./copy-builtin /usr/src/linux-3.15.6
  make
  make install
  
This will directly merge SPL into the kernel source, and build a few libraries.
Now let's merge in ZFS:

  ./configure \
      --prefix=/ \
      --libdir=/lib64 \
      --includedir=/usr/include \
      --datarootdir=/usr/share \
      --enable-linux-builtin=yes \
      --with-linux=/usr/src/linux-3.15.6 \
      --with-linux-obj=/usr/src/linux-3.15.6 \
      --with-spl=/root/src/spl-0.6.3
      --with-spl-obj=/root/src/spl-0.6.3
  ./copy-builtin /usr/src/linux-3.15.6
  make
  make install
  
This will merge ZFS into the kernel and built what amounts to the zfsprogs
package... if it did exist.

Now run make menuconfig. Under the root directory you'll see SPL support as a
new option in the menu. Now enable it as [*] for Built-in. Now open the
File-Systems menu and you'll see ZFS in there too. Again, enable it [*] for
built-in.

Now build and install your kernel normally.

Before you reboot, you'll need to create a hostid. To do this, you'll need to
generate it but there is not utility to do so, so use this instead:

  cd /sources
  mkdir -v hostid
  cd hostid

  cat > writehostid.c << "EOF"
  #include <stdio.h>
  #include <errno.h>
  #include <unistd.h>

  int main() {
      int res;
      res = sethostid(gethostid());
      if (res != 0) {
	  switch (errno) {
	      case EACCES:
	      fprintf(stderr, "Error! No permission to write the"
			  " file used to store the host ID.\n"
			  "Are you root?\n");
	      break;
	      case EPERM:
	      fprintf(stderr, "Error! The calling process's effective"
			      " user or group ID is not the same as"
			      " its corresponding real ID.\n");
	      break;
	      default:
	      fprintf(stderr, "Unknown error.\n");
	  }
	  return 1;
      }
      return 0;
  }
  "EOF"

Now compile it with:

  gcc -v writehostid.c -o writehostid
  
And run the utility:

  ./writehostid

This will generate a proper hostid file.

If all goes well, on the first system boot, you'll boot into ZFS just like any
other file system, but to get subsequent reboots to work without issue we'll
need to create a cache file for the zpool:

  zpool set cachefile=/etc/zfs/zpool.cache zfs-root

Now everything should work as intended. Congradulations on your new file
system, and enjoy.

If you'd like to learn more information on ZFS and it's capabilities which are
beyond the scope of this hint, visit: http://en.wikipedia.org/wiki/ZFS

===============================================================================

Acknowledgements:

I'd like to thank my partners in crime Keith Hedger and Arthur Radley for being
awesome teammates in our work in GNU/Linux.

I'd also like to thank SlackWiki and ArchWiki for their awesome coverage of ZFS
on Linux.

===============================================================================

Changelog:

2014-08-14

Version - 0.1 - Initial Draft and Public Release

===============================================================================