--- title: Answer runtime questions with read-only DTrace one-liners, and never allow destructive actions without asking area: dtrace impact: MEDIUM tags: [dtrace, observability, syscall, proc, io, aggregations, production] verified: 2026-11-04 --- # Answer runtime questions with read-only DTrace one-liners, and never allow destructive actions (-w, -x destructive, or the pragma) without asking DTrace is the tool for "what is this process or this machine doing right now". It is safe on a production system as long as you do not enable destructive actions. A probe costs nothing until you enable it, aggregations summarise in the kernel, and `dtrace` refuses destructive actions unless they are allowed with `-w`, `-x destructive`, or `#pragma D option destructive` in a script (the Dynamic Tracing Guide's Options and Actions chapters). Read a script before running it. Observability is DTrace, not eBPF, strace or perf (handbook: for Linux users); bpftrace's flags differ from dtrace's (compare: dtrace(8)). **Antipattern** ```sh strace -f -p 1234; bpftrace -e '...' # Linux habits: Solnix's strace is the illumos STREAMS # trace printer, a different tool; bpftrace is Linux's pfexec dtrace -n 'syscall:::entry { printf("%s\n", execname); }' # a line per syscall: drops on a busy box pfexec dtrace -w -n '...{ system("..."); }' # -w permits destructive actions ``` **The Solnix way** ```sh pfexec dtrace -l | wc -l # how many probes this kernel publishes pfexec dtrace -l -n 'syscall::*spawn*:entry' # does a probe exist? (Solnix's own check) pfexec dtrace -n 'syscall:::entry { @[execname] = count(); }' # syscalls by program; Ctrl-C prints pfexec dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }' # programs being started pfexec dtrace -n 'syscall::read:entry { self->t = timestamp; } syscall::read:return /self->t/ { @ = quantize(timestamp - self->t); self->t = 0; }' pfexec dtrace -n 'io:::start { @[args[1]->dev_statname, execname] = sum(args[0]->b_bcount); }' pfexec dtrace -n 'profile-997 { @[stack()] = count(); }' # where CPU time goes pfexec dtrace -n 'proc:::signal-send /args[2] == 12/ { printf("%s", execname); stack(); }' # who sends SIGSYS, and from where ``` Userland packages strip DWARF into a separate `debug` output and keep any CTF in the binary for mdb and DTrace (nix-conventions.md, "Debug symbols"). The build team traced a cmake SIGSYS this way: `syscall::*spawn*:entry` matched no probe, `proc:::signal-send` showed the path, and `fbt::nosys:entry` gave the syscall number. **The illumos way** (OpenIndiana, OmniOS, SmartOS) ```sh dtrace -n BEGIN -n END # the guide's second example dtrace -n 'syscall::write:entry { @counts[execname] = count(); }' dtrace -x bufsize=1m -n 'syscall:::entry { @[probefunc] = count(); }' # more buffer space if you see "drops on CPU"; if dtrace reports aggregation drops, raise -x aggsize ``` By default only the superuser can use DTrace. A user can be given `dtrace_proc`, `dtrace_user` or `dtrace_kernel` with a `defaultpriv=` entry in `/etc/user_attr`. **The difference.** Same as illumos on x86_64. DTrace comes from the gate (aarch64 illumos first shipped without mdb, kmdb and DTrace: design-riscv-port.md; check `which dtrace` there), and the handbook's one-liners are ordinary DTrace. On Solnix there is no `users.users` option for `defaultpriv`, so give DTrace to an administrator through an RBAC profile and `pfexec`. The `solnix` builder-zone brand prohibits `dtrace_kernel`. (The riscv64 port plans to bring up mdb, kmdb and DTrace after first boot: design-riscv-port.md.) **Check.** `pfexec dtrace -n 'syscall:::entry { @[execname] = count(); }'` prints `matched N probes`, and after Ctrl-C it prints a table of counts and no message about drops. **Sources** - [dtrace(8)](https://solnix.io/docs/man/dtrace.8) (illumos, CDDL), OPTIONS -l, -n, -P, -w, -x - [compare: dtrace(8)](https://solnix.io/docs/man/compare/dtrace.8), Linux counterpart bpftrace(8), -e and -f meanings - [compare: strace(8)](https://solnix.io/docs/man/compare/strace.8), the illumos strace at /usr/bin/strace - [Handbook: for Linux users](https://solnix.io/docs/handbook/for-linux-users) (site 307b67f), "Observability is DTrace, not eBPF/strace/perf" - [Dynamic Tracing Guide: Introduction](https://illumos.org/books/dtrace/chp-intro.html) (PDL 1.01), `dtrace -n BEGIN -n END`, `dtrace -l` - [Dynamic Tracing Guide: Aggregations](https://illumos.org/books/dtrace/chp-aggs.html) (PDL 1.01), count, quantize; aggregation drops and aggsize - [Dynamic Tracing Guide: proc Provider](https://illumos.org/books/dtrace/chp-proc.html), [syscall Provider](https://illumos.org/books/dtrace/chp-syscall.html), [io Provider](https://illumos.org/books/dtrace/chp-io.html), [profile Provider](https://illumos.org/books/dtrace/chp-profile.html) (PDL 1.01) - [Dynamic Tracing Guide: Buffers and Buffering](https://illumos.org/books/dtrace/chp-buf.html) (PDL 1.01), drops, bufsize - [Dynamic Tracing Guide: Actions and Subroutines](https://illumos.org/books/dtrace/chp-actsub.html) (PDL 1.01), Destructive Actions (`#pragma D option destructive`) - [Dynamic Tracing Guide: Options and Tunables](https://illumos.org/books/dtrace/chp-opt.html) (PDL 1.01), `destructive` (alias -w) - [Dynamic Tracing Guide: Security](https://illumos.org/books/dtrace/chp-sec.html) (PDL 1.01), Privileges, dtrace_proc, dtrace_user, dtrace_kernel - [Handbook: DTrace](https://solnix.io/docs/handbook/dtrace#safety--why-you-can-run-this-in-production) (site 307b67f), Providers, A few one-liners, Safety — why you can run this in production - [.kiro/steering/nix-conventions.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/.kiro/steering/nix-conventions.md) (solnix 5a8e5ce), Debug symbols (KEEP CTF for mdb/dtrace) - [docs/cmake-on-illumos.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/cmake-on-illumos.md) (solnix 5a8e5ce), the four dtrace measurements - [docs/design-riscv-port.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/docs/design-riscv-port.md) (solnix 5a8e5ce), §3.3 CTF for riscv64 - [modules/config/community/zone-brand/config.xml.in](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/community/zone-brand/config.xml.in) (solnix 5a8e5ce), prohibited `dtrace_kernel` - [modules/config/users.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/modules/config/users.nix) (solnix 5a8e5ce), user options (no defaultpriv)