ZFS storage
ZFS (OpenZFS) is the illumos filesystem and volume manager, and the substrate
Solnix’s upgrade model is built on. If you come from NixOS you may have used ZFS
as an optional root filesystem; on illumos it is the filesystem — the root
pool, the swap, the dump device, and every boot environment live on it. This
page covers the concepts (pools, datasets, snapshots, clones) and the zpool
and zfs commands you need, then ties it back to
boot environments.
On illumos generally ZFS is mature and load-bearing — illumos is a founding OpenZFS platform. On Solnix ZFS is inherited from the illumos gate; the commands below are the real illumos commands. Solnix ties each Nix generation to a ZFS boot environment (see that page).
The model: pools and datasets
Traditional systems layer a filesystem on a partition on a disk. ZFS collapses
that: you build a pool (a zpool) out of one or more devices with a chosen
redundancy, and then create datasets inside the pool that draw from its
shared free space on demand. There are two kinds of dataset:
- Filesystems — mountable trees of files (most datasets).
- Volumes (zvols) — block devices carved from the pool, used for swap, the crash-dump device, and zone/VM disks.
The root pool is conventionally named rpool. Boot environments live under
rpool/ROOT/.
zpool — manage pools
zpool list # pools, size, health, capacity
zpool status # per-device health, errors, scrub progress
zpool status -v # verbose, lists any damaged files
pfexec zpool scrub rpool # verify every block against its checksum
pfexec zpool create tank mirror c1t0d0 c1t1d0 # a mirrored pool
zpool status is your first stop for disk health. A scrub reads every block
and checks it against its checksum; in a redundant pool it repairs bad blocks
from the good copy automatically (“self-healing”). Scrub on a schedule.
Redundancy is chosen at pool creation — mirror, raidz, raidz2, raidz3 —
and ZFS integrates the volume-manager job (which disks, what redundancy) with
the filesystem, so it always knows which copy is good.
zfs — manage datasets
zfs list # datasets, used/avail/refer, mountpoint
zfs list -t snapshot # snapshots
pfexec zfs create rpool/data # a new filesystem dataset
pfexec zfs set compression=on rpool/data # a property
zfs get all rpool/data # every property of a dataset
Datasets have properties (compression, quota, mountpoint, recordsize, …) that are inherited down the tree unless overridden — a clean way to manage a hierarchy of filesystems from one place.
Snapshots
A snapshot is a read-only, point-in-time image of a dataset. It is copy-on-write, so it costs nothing at creation and grows only as the live dataset diverges from it.
pfexec zfs snapshot rpool/data@before-upgrade # take a snapshot
zfs list -t snapshot rpool/data # list them
pfexec zfs rollback rpool/data@before-upgrade # revert the dataset to it
pfexec zfs destroy rpool/data@before-upgrade # remove a snapshot
You can browse a snapshot’s contents read-only under the dataset’s hidden
.zfs/snapshot/<name>/ directory without rolling back — handy for recovering
one file.
Clones
A clone is a writable dataset created from a snapshot; it shares the snapshot’s blocks copy-on-write and only consumes space as it is written to. Clones are the mechanism behind boot environments:
pfexec zfs clone rpool/data@snap rpool/data-copy
send / receive — replication and backup
A snapshot can be serialized to a byte stream and applied elsewhere — the same pool, another disk, or another machine over the network:
pfexec zfs send rpool/data@snap | pfexec zfs receive tank/data
# incremental, only the delta between two snapshots:
pfexec zfs send -i @snap1 rpool/data@snap2 | ssh host pfexec zfs receive tank/data
This is the foundation of ZFS backup and disaster recovery — full and incremental, verifiable end to end.
Moving pools between Solnix and OpenZFS
Solnix’s ZFS is illumos’s, which has 27 pool features to OpenZFS’s 49. A pool made by
OpenZFS 2.2 or later (current Linux, FreeBSD 14) with the default zpool create can’t
be imported on Solnix, not even with -o readonly=on. OpenZFS 2.2 added
com.delphix:head_errlog, which becomes active the moment it is enabled and is not
read-only compatible, and illumos doesn’t have it. No later command on the OpenZFS side
makes such a pool importable again.
To make a pool on OpenZFS that Solnix can import, restrict its features when you create
it, e.g., zpool create -o compatibility=compat-2021 .... Every feature in OpenZFS’s
compat-2018 through compat-2021 files exists in illumos. That is checked against the
feature lists only; nobody has tried the import yet. Pools created on Solnix import on
OpenZFS.
The longer-term plan, to close this gap with OpenZFS rather than work around it, is in the software catalog.
TRIM: autotrim is off, and avoid zpool trim on NVMe and virtio
Two open illumos bugs let a ZFS TRIM lose data:
- illumos 18615: on
nvmeandvioblkdisks, ZFS treats a TRIM as done before the disk has done it, and a later write to the same blocks can be wiped. It hits manualzpool trimas well as autotrim.sddisks (SATA, SAS) and file vdevs are not affected. - illumos 18384: autotrim can corrupt a pool.
Until both fixes are in Solnix’s illumos pin:
- Every pool Solnix creates (the installer’s root pool, images, scratch pools) is made
with
-o autotrim=off, which is also illumos’s default. Check a pool withzpool get autotrim <pool>. - Don’t run
zpool trimon NVMe or virtio disks (laptop NVMe, EC2 NVMe and EBS, QEMU/KVM virtio). On SATA and SAS disks it is safe. Solnix schedules no periodic trim.
The fix for 18615 is in upstream review and will ship with Solnix’s next illumos re-pin. This section changes when it does.
How this ties to boot environments
Everything on the ZFS boot environments page is these primitives applied to the root filesystem:
- A boot environment is a dataset under
rpool/ROOT/. - Creating a BE takes a snapshot of the current root and makes a clone of it — which is why it is instant and nearly free.
- The clones share blocks, so twenty generations do not cost twenty full roots.
beadmis a convenience layer over exactly thesezfsoperations, aware of the boot loader.
So the same integrity guarantees (checksums, scrub, self-healing) and the same
send/receive backup story that protect your data also protect your
generations.
Further reading
zpool(8)·zfs(8)— command references.- ZFS boot environments — snapshots and clones as the upgrade mechanism.
- Further reading & resources — the canonical ZFS admin guide.