Creating fully encrypted ZFS pool

What I want to do?

I use my pool to securely store backups, archive my old documents and keep huge family’s photo library.

I have new disks. They were tortured with badblocks , so they’re ready to create ZFS pool.

I’ve read few documents about different approaches ^{1 2 3}. I wanted to be sure if anything changed during past years. One of articles recommends mirroring over RAIDZ. Resilvering is faster, at the same time putting IO less stress on whole pool. But pool as small as mine, relies on single drive which might die in between and data won’t be recoverable. Eventually, I decided to go for RAIDZ1 for now and in the future I rather move to RAIDZ2. For that, I have to buy one more disk - Black Friday is close, we will see.

One thing that is available now, but was not, when I created ZFS pool last time, is encryption. I want it. I won’t be running with my PC anywhere, but in case if disk will be damaged and I will have to return it. No worries, data is unavailable. So I want it 😄

Let’s do it

I use Ubuntu 21.10 but despite installation of ZFS utils, rest of commands is not Ubuntu specific and should work on any distro.

sudo apt install -y zfsutils-linux zfs-zed

Encryption key

First, I have to generate secure key/file, that will be used to encrypt/decrypt file system. I will keep it on my root filesystem, actually in /root directory, as it won’t be easily available for other users.

It’s possible to use password or raw file. On laptop, I will choose password, but on my desktop PC I prefer file. First, I was thinking to place it in /etc/zfs but it’s world readable. So I decided I will drop it in /root dir.

sudo dd if=/dev/random of=/root/.zfs-encrypt.key bs=1 count=32

I strongly advice to make backup of this key. Multiple backups actually :)

Create a fully encrypted pool

Now I can create pool. Most examples refer to disks as sda, sdb, etc. But I prefer to use their labels as drive model plus serial number. Just check /dev/disk/by-id/. Other way is by use of wwn-* ids, they’re also stable across restarts, cable changes, etc. I strongly recommend such ids over standard letters.

sudo zpool create \
  -o ashift=12 \
  -o feature@encryption=enabled \
  -O encryption=on \
  -O keylocation=file:///root/.zfs-encrypt.key \
  -O keyformat=raw \
  storage raidz1 \
    ata-WDC_WD140EDGZ-11B1PA0_9MGJK4YK \
    ata-WDC_WD140EDGZ-11B1PA0_Y6GVH40C \
    ata-WDC_WD140EDGZ-11B1PA0_Y6GWHD3C

What happen here?

• -o ashift=12 - treat disks sector size as 4KB, it was detected automatically but I wanted to be sure :)
• -o feature@encryption=enabled -O encryption=on -O keylocation=file:///root/.zfs-encrypt.key -O keyformat=raw - enable encryption on whole pool, use raw key and default encryption method: aes-256-gcm

Automatically decrypt on boot

So my pool is encrypted now, but I don’t want to decrypt it manually each time my PC starts. So I added such service:

sudo cat <<EOF > /etc/systemd/system/zfs-load-key.service
[Unit]
Description=Load encryption keys
DefaultDependencies=no
After=zfs-import.target
Before=zfs-mount.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs load-key -a
StandardInput=tty-force

[Install]
WantedBy=zfs-mount.service
EOF

It have to enabled it now:

sudo systemctl daemon-reload
sudo systemctl enable zfs-load-key

In ArchLinux wiki ⁴, you can find example for service, that can load individual key per pool, but I had trouble getting it to work. Service above loads all keys for all pools and it’s matching my case of just two pools ;)

Let’s configure pool

There are few options worth to setup just after pool creation:

# automatically expand pool when new disk is added
sudo zpool set autoexpand=on storage

# automatically replace failed disk with hot spare
sudo zpool set autoreplace=on storage

# enable LZ4 compression
sudo zfs set compression=lz4 storage

# disable access time - for better performance
sudo zfs set atime=off storage

For better explanation of options, check best practices on Aaron’s Toponce blog ⁵.

Create datasets (actual mount points)

Now I’m ready to create few datasets, that will be actually used to store files.

# photos are well compressed, so I don't need to compress them again
sudo zfs create storage/photos
sudo zfs set compression=off storage/photos

# on backups I often just copy files staring, so let use stronger compression ZSTF
sudo zfs create storage/backup
sudo zfs set compression=zstd storage/backup

# I like to keep my downloads on pool too
sudo zfs create -o mountpoint=/home/timor/Downloads storage/downloads

Setting TLER on boot

Having disks in RAID, it’s good to have TLER enabled to rely on RAID for error recovery instead of internal hard drive recover ⁶. My disks support it, but it have to be enabled

sudo smartctl -l scterc /dev/sdd
sudo smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.0-21-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

It’s there but disabled.

sudo smartctl -l scterc,70,70 /dev/sdd
sudo smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.13.0-21-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

SCT Error Recovery Control set to:
           Read:     70 (7.0 seconds)
          Write:     70 (7.0 seconds)

Make it persistent between restarts:

sudo cat <<EOF > /etc/rc.local
#!/bin/bash

# I have such disks, let's iterate over them
# /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_9MGJK4YK
# /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_Y6GVH40C
# /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_Y6GWHD3C

for i in 9MGJK4YK Y6GVH40C Y6GWHD3C; do
  echo smartctl -l scterc,70,70 /dev/disk/by-id/ata-WDC_WD140EDGZ-11B1PA0_\$i > /dev/null;
done
EOF

sudo chmod +x /etc/rc.local
sudo systemctl enable rc-local.service

That’s it. All done.

Enjoyed?