Sign in

Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ZFS storage

ZFS (OpenZFS) is the illumos filesystem and volume manager, and the substrate Solnix’s upgrade model is built on. If you come from NixOS you may have used ZFS as an optional root filesystem; on illumos it is the filesystem — the root pool, the swap, the dump device, and every boot environment live on it. This page covers the concepts (pools, datasets, snapshots, clones) and the zpool and zfs commands you need, then ties it back to boot environments.

On illumos generally ZFS is mature and load-bearing — illumos is a founding OpenZFS platform. On Solnix ZFS is inherited from the illumos gate; the commands below are the real illumos commands. Solnix ties each Nix generation to a ZFS boot environment (see that page).

The model: pools and datasets

Traditional systems layer a filesystem on a partition on a disk. ZFS collapses that: you build a pool (a zpool) out of one or more devices with a chosen redundancy, and then create datasets inside the pool that draw from its shared free space on demand. There are two kinds of dataset:

  • Filesystems — mountable trees of files (most datasets).
  • Volumes (zvols) — block devices carved from the pool, used for swap, the crash-dump device, and zone/VM disks.

The root pool is conventionally named rpool. Boot environments live under rpool/ROOT/.

zpool — manage pools

zpool list                       # pools, size, health, capacity
zpool status                     # per-device health, errors, scrub progress
zpool status -v                  # verbose, lists any damaged files
pfexec zpool scrub rpool         # verify every block against its checksum
pfexec zpool create tank mirror c1t0d0 c1t1d0   # a mirrored pool

zpool status is your first stop for disk health. A scrub reads every block and checks it against its checksum; in a redundant pool it repairs bad blocks from the good copy automatically (“self-healing”). Scrub on a schedule.

Redundancy is chosen at pool creation — mirror, raidz, raidz2, raidz3 — and ZFS integrates the volume-manager job (which disks, what redundancy) with the filesystem, so it always knows which copy is good.

zfs — manage datasets

zfs list                         # datasets, used/avail/refer, mountpoint
zfs list -t snapshot             # snapshots
pfexec zfs create rpool/data     # a new filesystem dataset
pfexec zfs set compression=on rpool/data      # a property
zfs get all rpool/data           # every property of a dataset

Datasets have properties (compression, quota, mountpoint, recordsize, …) that are inherited down the tree unless overridden — a clean way to manage a hierarchy of filesystems from one place.

Snapshots

A snapshot is a read-only, point-in-time image of a dataset. It is copy-on-write, so it costs nothing at creation and grows only as the live dataset diverges from it.

pfexec zfs snapshot rpool/data@before-upgrade   # take a snapshot
zfs list -t snapshot rpool/data                 # list them
pfexec zfs rollback rpool/data@before-upgrade   # revert the dataset to it
pfexec zfs destroy rpool/data@before-upgrade    # remove a snapshot

You can browse a snapshot’s contents read-only under the dataset’s hidden .zfs/snapshot/<name>/ directory without rolling back — handy for recovering one file.

Clones

A clone is a writable dataset created from a snapshot; it shares the snapshot’s blocks copy-on-write and only consumes space as it is written to. Clones are the mechanism behind boot environments:

pfexec zfs clone rpool/data@snap rpool/data-copy

send / receive — replication and backup

A snapshot can be serialized to a byte stream and applied elsewhere — the same pool, another disk, or another machine over the network:

pfexec zfs send rpool/data@snap | pfexec zfs receive tank/data
# incremental, only the delta between two snapshots:
pfexec zfs send -i @snap1 rpool/data@snap2 | ssh host pfexec zfs receive tank/data

This is the foundation of ZFS backup and disaster recovery — full and incremental, verifiable end to end.

Moving pools between Solnix and OpenZFS

Solnix’s ZFS is illumos’s, which has 27 pool features to OpenZFS’s 49. A pool made by OpenZFS 2.2 or later (current Linux, FreeBSD 14) with the default zpool create can’t be imported on Solnix, not even with -o readonly=on. OpenZFS 2.2 added com.delphix:head_errlog, which becomes active the moment it is enabled and is not read-only compatible, and illumos doesn’t have it. No later command on the OpenZFS side makes such a pool importable again.

To make a pool on OpenZFS that Solnix can import, restrict its features when you create it, e.g., zpool create -o compatibility=compat-2021 .... Every feature in OpenZFS’s compat-2018 through compat-2021 files exists in illumos. That is checked against the feature lists only; nobody has tried the import yet. Pools created on Solnix import on OpenZFS.

The longer-term plan, to close this gap with OpenZFS rather than work around it, is in the software catalog.

TRIM: autotrim is off, and avoid zpool trim on NVMe and virtio

Two open illumos bugs let a ZFS TRIM lose data:

  • illumos 18615: on nvme and vioblk disks, ZFS treats a TRIM as done before the disk has done it, and a later write to the same blocks can be wiped. It hits manual zpool trim as well as autotrim. sd disks (SATA, SAS) and file vdevs are not affected.
  • illumos 18384: autotrim can corrupt a pool.

Until both fixes are in Solnix’s illumos pin:

  • Every pool Solnix creates (the installer’s root pool, images, scratch pools) is made with -o autotrim=off, which is also illumos’s default. Check a pool with zpool get autotrim <pool>.
  • Don’t run zpool trim on NVMe or virtio disks (laptop NVMe, EC2 NVMe and EBS, QEMU/KVM virtio). On SATA and SAS disks it is safe. Solnix schedules no periodic trim.

The fix for 18615 is in upstream review and will ship with Solnix’s next illumos re-pin. This section changes when it does.

How this ties to boot environments

Everything on the ZFS boot environments page is these primitives applied to the root filesystem:

  • A boot environment is a dataset under rpool/ROOT/.
  • Creating a BE takes a snapshot of the current root and makes a clone of it — which is why it is instant and nearly free.
  • The clones share blocks, so twenty generations do not cost twenty full roots.
  • beadm is a convenience layer over exactly these zfs operations, aware of the boot loader.

So the same integrity guarantees (checksums, scrub, self-healing) and the same send/receive backup story that protect your data also protect your generations.

Further reading