--- title: Post-mortem a crashed program from /var/cores with mdb and pstack area: debug impact: HIGH tags: [debug, mdb, core, coreadm, pstack, ctf, gdb, crash-dump] verified: 2026-11-04 --- # Post-mortem a crashed program from /var/cores with mdb and pstack On Solnix a crashing program leaves a core file under `/var/cores`, not in its working directory, and you read it with `mdb` or `pstack`. Look in the wrong place and the core seems to be missing. Wait too long and the prune service deletes it. **Antipattern** ```sh # Linux/systemd habits that do nothing or mislead here cat /proc/sys/kernel/core_pattern # Linux core(5); illumos sets the pattern with coreadm coredumpctl gdb prog # systemd-coredump; illumos has no journal, cores are plain files ulimit -c 0 # does not stop global cores: they ignore RLIMIT_CORE (coreadm.nix) ls ./core # Solnix sends cores to /var/cores, not the working directory ``` **The Solnix way** ```sh # configuration.nix (defaults shown): solnix.coreadm.enable = true; solnix.coreadm.perProcess = true; # solnix.coreadm.developerUsers = [ "alice" ]; # creates /var/cores/ so alice can read her cores # environment.enableDebugInfo = true; # default false; adds each systemPackage's `debug` output where it has one # (mk-userland-pkg packages do; mk-src-pkg ones do not; see below) coreadm # global /var/cores/core.%f.%p.%t, init /var/cores/%u/core.%f.%p.%t ls -t /var/cores/"$(id -u)"/ # your own cores (root-owned copies sit in /var/cores) pstack /var/cores/1000/core.prog.4321.1730700000 # the quick answer: the stack at the crash mdb /nix/store/-prog-1.0/bin/prog /var/cores/1000/core.prog.4321.1730700000 > ::status # summary of the target: the signal, the faulting instruction > $c # C stack backtrace (::stack is the same; $C adds frame pointers) > ::regs # general registers of the representative thread ($r is the same) > myfunc::dis # disassemble a function named in the backtrace > $q pstack 4321; pldd 4321 # a live process: these stop it while they read it (ask first on a production process) gcore 4321 # snapshot a live process to core.4321, then use mdb as above mdb -p 4321 # attach (this stops it for the whole session: ask first); ::release sets it running again, then $q # kernel panic: pfexec dumpadm shows the dump setup. By default (solnix.dumpadm.sizeAtBoot), when no dedicated dump # device is set and /dump exists, that zvol is sized and made the dump device at boot. To set it yourself: # solnix.dumpadm.enable = true with solnix.dumpadm.device (and savecoreDir, applied only with enable). The boot # after a panic saves the dump (vmdump. when compressed, the default) in the savecore directory; # in that directory, pfexec savecore -vf vmdump. expands it, then mdb reads unix. + vmcore. # (the illumos user guide's example: savecore -vf vmdump.0, then mdb 0) ``` `mdb core` alone also works (mdb infers the executable; some symbols may be missing). The debug split keeps CTF for mdb and DTrace and moves DWARF to a `debug` output, meant for `pkgs.solnix.gdb` to find by build-id once `environment.enableDebugInfo` is on. Not proven on a real package yet: man-and-debug-delivery.md says the proof used a test binary and waits on the wave6 split-debug work, and docs/registry/recipes.tsv at 5a8e5ce has no row with a `debug` path. For panics, see [Handbook: Debugging](https://solnix.io/docs/handbook/debugging#the-system-panicked-or-hung-crash-dumps). **The illumos way** (OpenIndiana, OmniOS, SmartOS) ```sh coreadm # illumos user guide: global pattern empty, per-process on -> a file named core in the cwd pfexec coreadm -g /var/cores/core.%f.%p -e global -e process # coreadm(8) option names: global, process, log pstack core mdb core ``` Same tools and dcmds. The core-file policy is set by hand with `coreadm` and persists across reboots. **The difference.** On Solnix the policy comes from `solnix.coreadm.*`, which generates `/etc/coreadm.conf`: global and per-process cores go to `/var/cores`, setid cores are off, and each global core is logged. A user with no `/var/cores/` gets only the root-owned global copy. `solnix-core-prune` deletes cores older than 14 days, beyond the newest 3 per program, or past 4096 MiB in total (`solnix.coreadm.prune.*`), so copy a core to keep it. **Check.** `coreadm` prints `global core file pattern: /var/cores/core.%f.%p.%t` and `per-process core dumps: enabled`; then a crash lands in `/var/cores`, not the working directory. **Unverified** - Whether a hand-run `pfexec coreadm -g ...` survives the next generation switch; the sources do not say. - Whether every package carries CTF: the sources say the debug split keeps CTF, not that each build generates it. `elfdump -c ` lists a `.SUNW_ctf` section when it does. **Sources** - [docs/registry/recipes.tsv](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/registry/recipes.tsv) (solnix 5a8e5ce), the `debug` column - [id(8)](https://solnix.io/docs/man/id.8) (illumos, CDDL), /usr/bin/id -u - [coreadm(8)](https://solnix.io/docs/man/coreadm.8) (illumos, CDDL), DESCRIPTION (%d %f %p %t %u, no-argument report); OPTIONS -e (global, process, log), -g; FILES /var/cores - [mdb(1)](https://solnix.io/docs/man/mdb.1) (illumos, CDDL), SYNOPSIS `object [core]`; -p pid; ::release (an attached `-p` process is set running) - [Modular Debugger Guide: Built-in Commands](https://illumos.org/books/mdb/commands-1.html) (PDL), $C; $c and ::stack; ::regs and $r; ::status; ::dis; $q - [Modular Debugger Guide: Options](https://illumos.org/books/mdb/options-1.html) (PDL), -p pid; Operands (core without object: mdb infers the executable) - [Modular Debugger Guide: Notes](https://illumos.org/books/mdb/notes-1.html) (PDL), Limitations on Examining Process Core Files - [proc(1)](https://solnix.io/docs/man/proc.1) (illumos, CDDL), pstack, pldd; USAGE (pfiles, pldd and pstack stop the target; they also read core files) - [gcore(1)](https://solnix.io/docs/man/gcore.1) (illumos, CDDL), DESCRIPTION (core.process-id) - [ctf(5)](https://solnix.io/docs/man/ctf.5) (illumos, CDDL), DESCRIPTION (used by mdb and dtrace); `.SUNW_ctf` section name - [elfdump(1)](https://solnix.io/docs/man/elfdump.1) (illumos, CDDL), -c (section headers) - [dumpadm(8)](https://solnix.io/docs/man/dumpadm.8) (illumos, CDDL), DESCRIPTION (no-argument report; compressed vmdump.X vs unix.X + vmcore.X) - [savecore(8)](https://solnix.io/docs/man/savecore.8) (illumos, CDDL), DESCRIPTION; -f dumpfile - [Handbook: Debugging](https://solnix.io/docs/handbook/debugging#a-program-crashed-core-files) (site 307b67f), A program crashed (core files); The system panicked or hung (crash dumps) - [modules/config/coreadm.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/coreadm.nix) (solnix 5a8e5ce), options and generated coreadm.conf; solnix-core-prune - [modules/config/debug-info.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/debug-info.nix) (solnix 5a8e5ce), environment.enableDebugInfo - [modules/config/dumpadm.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/dumpadm.nix) (solnix 5a8e5ce), solnix.dumpadm options (savecoreDir default; compress: vmdump.N expanded with savecore -f) - [.kiro/steering/nix-conventions.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/.kiro/steering/nix-conventions.md) (solnix 5a8e5ce), Debug symbols (separate output, cached) - [docs/man-and-debug-delivery.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/man-and-debug-delivery.md) (solnix 5a8e5ce), debug-info.nix; gdb itself (pkgs.solnix.gdb 18.1 reads illumos cores) - [illumos user guide: Debugging broken systems](https://illumos.org/docs/user-guide/debug-systems/) (MPL-2.0), `pfexec savecore -vf vmdump.0`, then `mdb 0` - [illumos user guide: Debugging applications](https://illumos.org/docs/user-guide/debug-apps/) (MPL-2.0), Inspecting An Application Crash - [compare: coreadm(8)](https://solnix.io/docs/man/compare/coreadm.8), Linux core(5) /proc/sys/kernel/core_pattern