--- title: Diagnose a failed service with svcs -xv, fix it in the module area: smf impact: HIGH tags: [smf, maintenance, svcs, logs, svcadm-clear, troubleshooting] verified: 2026-11-04 --- # Diagnose a failed service with svcs -xv, fix it in the module For a service in `maintenance`, find the cause with `svcs -xv` and its log. If the service comes from an `smf.services.` declaration (yours, or a Solnix module's such as `nix-daemon` or `sshd`), fix it in that declaration, rebuild, then `svcadm clear`. A hand `svccfg setprop` is not in your configuration and re-importing an unchanged manifest does not undo it (svccfg import upgrades only properties unchanged since the last import, and reports a conflict on stderr only when the manifest changed that property too). A stock illumos service (activation/default.nix calls them "world-stock") or one imported by hand has no declaration: fix its cause (a config file, a dependency, hardware) and `svcadm clear` it as on illumos. **Antipattern** ```sh pfexec svcadm clear svc:/site/myapp:default # retry without reading why it failed pfexec svccfg -s svc:/site/myapp:default setprop config/port = astring: 9090 pfexec svcadm refresh svc:/site/myapp:default # works today, missing from the config ``` **The Solnix way** ```sh # maintenance: the method exited $SMF_EXIT_ERR_CONFIG or _FATAL, failed 3 times in a row, or restarted more than # once a second. A timed-out start is killed and counts as a failure (svc.startd(8); svcadm(8) NOTES says a hung # start ends in maintenance). A contract service also fails on a core dump or an outside kill (see ignoreError). svcs -xv # what is broken, why, its dependency path, its log svcs -l svc:/site/myapp:default # state, enabled, the logfile line, each dependency tail -50 /var/svc/log/site-myapp:default.log # the start method's stdout and stderr svcs -d svc:/site/myapp:default # what it depends on (is one of them not online?) svcs -D svc:/site/myapp:default # what depends on it (what else is down) svcs -p svc:/site/myapp:default # its processes, once it runs svccfg export svc:/site/myapp # the service as the repository holds it now, hand edits included # fix smf.services.myapp (start.exec, requires, user, start.timeoutSeconds, properties), then: pfexec solnix-rebuild switch --flake # # re-imports the generated manifest (no --flake needed if # /etc/nixos/flake.nix exists; without either, it re-activates the old one) pfexec svcadm refresh svc:/site/myapp:default # load the new configuration into the running snapshot pfexec svcadm clear svc:/site/myapp:default # mark it repaired; svc.startd tries again ``` The log is `/var/svc/log/.log`, e.g. `site-myapp:default.log` (`svcs -L ` prints it); svc.startd's own log is `/var/svc/log/svc.startd.log`. Where `/var` datasets exist, `/var/svc/log` is shared by all BEs and survives a rollback. A generated service with no `dependents` also depends on `svc:/system/manifest-import:default`. **The illumos way** (OpenIndiana, OmniOS, SmartOS) ```sh svcs -xv; svcs -d svc:/site/myapp:default; svcs -l svc:/site/myapp:default pfexec svccfg -s svc:/site/myapp:default setprop config/port = astring: 9090 # the repository is the config pfexec svcadm refresh svc:/site/myapp:default pfexec svcadm clear svc:/site/myapp:default ``` A `svccfg` change typically takes effect when the instance is next refreshed (`svcadm refresh`); some values need a restart (svcadm(8) refresh). To undo bad edits, `svccfg -s listsnap` lists the snapshots and `pfexec svccfg -s revert ` goes back to one; `pfexec svcadm refresh ` makes it active. Ask the user first, and on a declared service prefer a rebuild: the `initial` snapshot is the configuration first imported (smf(7)), and a Solnix method is a store-path wrapper (lib/smf/xml.nix), which a later GC may have removed. **The difference.** The diagnosis (`svcs -x`, `-l`, `-d`, `-D`, `-p`, the log) and `svcadm clear` are the same illumos commands; the repair is not. On Solnix the manifest comes from your configuration, so the fix belongs there, where it is built, versioned and reproduced on the next machine. svccfg(8) says re-import upgrades only properties unchanged since the last import, so a switch leaves a hand-set value in place, and a fresh install from the same configuration (which starts from the seed repository) does not have it. **Check.** `svcs -x` prints nothing: no enabled service is down or blocking another. **Sources** - [modules/config/nix.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/nix.nix) and [configurations/base.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/configurations/base.nix) (solnix 5a8e5ce), `smf.services.nix-daemon`, `smf.services.sshd` - [Handbook: SMF](https://solnix.io/docs/handbook/smf#logs) (site 307b67f), Logs; Commands you use every day; The repository - [Handbook: Debugging](https://solnix.io/docs/handbook/debugging#a-service-failed-to-start) (site 307b67f), A service failed to start - [svcs(1)](https://solnix.io/docs/man/svcs.1) (illumos, CDDL), OPTIONS -x, -v, -l, -L, -d, -D, -p; Example 6 - [svcadm(8)](https://solnix.io/docs/man/svcadm.8) (illumos, CDDL), SUBCOMMANDS clear, refresh; NOTES (start method timeout) - [svc.startd(8)](https://solnix.io/docs/man/svc.startd.8) (illumos, CDDL), Logs; SERVICE FAILURE - [svccfg(8)](https://solnix.io/docs/man/svccfg.8) (illumos, CDDL), DESCRIPTION; import; export; setprop; listsnap; revert - [smf_method(7)](https://solnix.io/docs/man/smf_method.7) (illumos, CDDL), Method Definition (exec); Exiting and Exit Status - [smf(7)](https://solnix.io/docs/man/smf.7) (illumos, CDDL), MAINTENANCE state; snapshots (`initial`: the configuration created by the administrator or at package installation) - [lib/smf/xml.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/lib/smf/xml.nix) (solnix 5a8e5ce), mkMethod (store-path wrapper), importDep on manifest-import - [modules/system/activation/default.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/system/activation/default.nix) (solnix 5a8e5ce), the `activate` script (svccfg import of each manifest); the world-stock services that are not in smf.services; solnix-rebuild (no --flake and no /etc/nixos/flake.nix: `default_top`, the newest registered generation) - [modules/system/activation/install-be.sh](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/system/activation/install-be.sh) (solnix 5a8e5ce), step 5: seed repository.db when the BE has none - [docs/design-smf-init.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/design-smf-init.md) (solnix 5a8e5ce), ยง3.3 (changed service: re-import, then `svcadm refresh`) - [modules/system/boot/init/smf.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/system/boot/init/smf.nix) (solnix 5a8e5ce), options start, requires, user, properties, ignoreError - [docs/design-var-state.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/design-var-state.md) (solnix 5a8e5ce), The layout (svc/log shared) - [OpenIndiana handbook: System Administration](https://docs.openindiana.org/handbook/systems-administration/#basic-strategy), Troubleshooting: Basic Strategy; Corrupt Service Configuration