--- name: solnix-illumos-dev description: > Change illumos-gate itself from a Solnix context: find code, history, reviewers and open issues with this server's illumos-gate index; build the gate with nightly and bldenv; run git pbchk, cstyle, hdrchk, wsdiff and check_rtime; debug the kernel with mdb, kmdb, CTF and DTrace (fbt, sdt, adding a DTRACE_PROBE); send a change upstream through an illumos issue, code review and a request to integrate (RTI). Use when changing the illumos kernel, a driver, libc or a base command, reading a kernel crash dump, or preparing a gate change for upstream. Says what Solnix takes from the gate (the proto area, through Nix); it keeps no IPS packages and has no pkg, which onu needs. Porting other software is solnix-porter. license: CC0-1.0 metadata: author: solnix version: "0.2.0" persona: illumos-dev --- # Solnix: illumos-gate developer Solnix takes the kernel and the load-bearing userland from illumos-gate ([handbook: how Solnix is built](https://solnix.io/docs/handbook/how-solnix-is-built#the-illumos-gate)). On x86_64 it pins one gate revision (`ac45d71fc2`); aarch64 uses a different proto (pkgs/illumos/aarch64/gate.nix). [pkgs/illumos/gate.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/pkgs/illumos/gate.nix) runs the gate's own `nightly` inside a derivation on an illumos host, with that host's `/opt/onbld` tools, and keeps the proto area as a store path; by default it takes a prebuilt proto pinned by hash and runs no build (protoUrl/protoHash). The x86_64 boot kernel is a separate non-DEBUG build with Solnix's patches applied ([kernel-nd.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/pkgs/illumos/kernel-nd.nix)). Solnix consumes the proto area, not IPS packages, and has no `pkg` ([product.md](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/.kiro/steering/product.md), "No IPS/`pkg` in the end product"), so `onu` has nothing to install. A change to the gate goes upstream through illumos's process, built and tested on an illumos distribution such as OpenIndiana or OmniOS. Read `steering/solnix-is-not-upstream-illumos.md` and `steering/upstream-ai-policies.md` (repo root) first. ## Tools this uses - This server: `illumos_search_code` (symbols and paths in illumos-gate, linked to the line on GitHub at the indexed commit), `illumos_commits` (history by message, path, date, author), `illumos_blame` (the commits that touched a file; file-level, not per line), `illumos_issue` / `illumos_issues` (illumos.org issue metadata and links; the issue text is not indexed), `search_manpages`, `search_handbook`. - On an illumos build host: `nightly`, `bldenv`, `dmake`, `git pbchk`, `cstyle`, `hdrchk`, `wsdiff`, `check_rtime`, `elfdump`, `ctfdump`, `ctfdiff`; on the system under test: `mdb`, `kmdb`, `savecore`, `dtrace`; `git`. ## Steps 1. **Locate the code.** `illumos_search_code {query: }`, or `{query: , kind: "path"}`. Done when you have the file and line link. 2. **Read why it is that way, and who knows it.** `illumos_blame {path}`, then `illumos_commits {path, query}` for the change that matters. illumos commit messages end in `Reviewed by:` and `Approved by:` lines ([Developer's Guide: Workflow](https://illumos.org/books/dev/workflow.html), Squashing Commits and Commit Messages), so past commits to a file name people who know that area, candidates to ask for review (the Integrating chapter values a reviewer who knows the subsystem). Done when you can name the commit, its issue number and its reviewers. 3. **Read the issue.** `illumos_issue {id}` for status and the link; follow the link for the discussion. Done when you have quoted what the issue says, from its page. 4. **Check for open work, and ask first.** `illumos_issues {query, status: "open"}` before starting a change. If none fits, the human files one at illumos.org (it needs a bug-tracker account). Before writing code, ask on the developer list or IRC; for a larger body of work, write an illumos Project Discussion (IPD) ([illumos docs: Contributing](https://illumos.org/docs/contributing/), Finding An Area To Contribute To). Done when you have an issue number, an existing one or the human's new one. 5. **Build.** On OpenIndiana or OmniOS: an env file copied from `usr/src/tools/env/illumos.sh`, a full `nightly`, then `bldenv` and `dmake install` in the directory you change (for a kernel module, its architecture build directory, such as `uts/intel/igb`: Developer's Guide, Workflow, Kernel Components). Rule: `gate-build-nightly-bldenv.md` (also what Solnix's gate.nix changes in the env file, and what it keeps). Done when a full nightly (not `-i`) leaves `log/latest/mail_msg` with no errors or warnings. 6. **Check.** `git pbchk` under `bldenv`; `cstyle -pP` for new C files; `wsdiff` and `check_rtime` for what the binaries became. Rule: `gate-checks-pbchk-cstyle.md`. Done when every `git pbchk` finding is fixed, except a copyright one, which is optional. 7. **Install and test.** On OpenIndiana or OmniOS, `pfexec onu -t -d /packages/i386/nightly` (as root) installs the build's IPS packages into a new BE ([onu(1onbld)](https://github.com/illumos/illumos-gate/blob/b56614870515295c253babb2b10fffdd3c60fdbe/usr/src/tools/scripts/onu.1onbld), DESCRIPTION: beadm(8) for the BE, pkg(1) to update it); then a reboot ([illumos docs: Building illumos](https://illumos.org/docs/developers/build/), Installing your build). The new BE and the reboot change what the machine runs: show the commands and wait for the user's yes. Solnix has no onu path: it has no `pkg` and keeps no packages from its gate build. For a single binary with standard library dependencies, a copy on a system close to vanilla illumos-gate can do; core kernel changes need a whole environment booted from your gate build; a downstream fork carrying other non-trivial changes is generally not enough ([illumos docs: Contributing](https://illumos.org/docs/contributing/), Testing). Solnix's x86_64 kernel carries its own patches (kernel-nd.nix), so test an upstream change on OpenIndiana or OmniOS. Done when the issue has testing notes: what you ran, on what, with what result. 8. **Send it upstream.** Gerrit review at code.illumos.org (or another web-based review), testing notes in the issue, then an RTI mail to the core team. Rule: `gate-upstream-rti.md`. The human sends; an agent drafts and stops (`steering/upstream-ai-policies.md`). Done when the human has the draft and the attachments. ## Rules (`../rules/`) | task | rule | |---|---| | build the gate; rebuild one directory; what Solnix builds from the gate | `gate-build-nightly-bldenv.md` | | the checks before an RTI: git pbchk, cstyle, hdrchk, wsdiff, check_rtime | `gate-checks-pbchk-cstyle.md` | | issue, Gerrit, testing notes, RTI, commit message, flag days, IPD | `gate-upstream-rti.md` | | a crash dump or a core file | `debug-mdb-postmortem.md` | | leaks or heap corruption in a userland daemon | `debug-libumem-leaks.md` | | trace the running system | `dtrace-for-admins.md` | ## Debugging the kernel - **A panic.** In the savecore directory, `pfexec savecore -vf vmdump.` expands the dump; `mdb ` reads `unix.` and `vmcore.` ([mdb(1)](https://solnix.io/docs/man/mdb.1) OPERANDS suffix; `-k` forces crash-dump mode, OPTIONS). Start with `::status` and `$C` ([Developer's Guide: Debugging](https://illumos.org/books/dev/debugging.html), Panics), then `::msgbuf` ([handbook: debugging](https://solnix.io/docs/handbook/debugging#the-system-panicked-or-hung-crash-dumps)). For a daemon or kernel change, the Integrating chapter suggests a DEBUG build and a leak check: `::findleaks` on a crash dump for the kernel, libumem for a daemon (Other Testing Areas; `debug-libumem-leaks.md`). Details and the Solnix dump setup: `debug-mdb-postmortem.md`. - **The live kernel.** `mdb -k` with no operands reads `/dev/ksyms` and `/dev/kmem` (read access to `/dev/kmem` is for group sys). `mdb -kw` writes to the running kernel, which can panic it (mdb(1) WARNINGS); `$` uses it. The gate build adds it with ctfconvert (DWARF to CTF, [ctfconvert(1onbld)](https://github.com/illumos/illumos-gate/blob/b56614870515295c253babb2b10fffdd3c60fdbe/usr/src/tools/scripts/ctfconvert.1onbld)) and ctfmerge; [gate.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/pkgs/illumos/gate.nix) requires both on the build host (hostPrereqs); a command gets it by including `Makefile.ctf` and calling `POST_PROCESS_O` and `POST_PROCESS` ([Developer's Guide: Component Anatomy](https://illumos.org/books/dev/anatomy.html), CTF Support). Check with `elfdump -c | grep .SUNW_ctf`; decode with [ctfdump(1)](https://solnix.io/docs/man/ctfdump.1); compare two objects with [ctfdiff(1)](https://solnix.io/docs/man/ctfdiff.1). Solnix copies kernel objects with `cp -a` and no strip, so `.SUNW_ctf` survives ([slices.nix](https://codeberg.org/solnix/solnix/src/commit/5a8e5ce75a33086d7be4410149847bdf7501b8fc/pkgs/illumos/slices.nix), kernel). - **DTrace providers.** `fbt` has an `entry` and a `return` probe for most kernel functions, but not for some leaf functions; the guide recommends it only for kernel development, or when other providers (syscall, sched, proc, io) are not enough ([Dynamic Tracing Guide: fbt Provider](https://illumos.org/books/dtrace/chp-fbt.html)). To add a static probe, use `DTRACE_PROBE` to `DTRACE_PROBE4` from ``; a `__` in the name becomes `-`; list yours with `dtrace -l -P sdt -m `; keep pointer dereferences and global loads out of the arguments ([Dynamic Tracing Guide: sdt Provider](https://illumos.org/books/dtrace/chp-sdt.html), Creating SDT Probes). The [sdt(4D)](https://solnix.io/docs/man/sdt.4d) and [fbt(4D)](https://solnix.io/docs/man/fbt.4d) drivers are not public interfaces; you reach them through DTrace. `dtrace -w` allows destructive actions: ask first (`dtrace-for-admins.md`). ## Unverified - Whether Solnix has sent its kernel-nd.nix patches upstream: the sources do not say (`gate-upstream-rti.md`). - Whether Solnix has an option for kernel boot flags such as `-k`: none was found in `modules/` at 5a8e5ce. - Whether the onbld tools run on a Solnix host: gate.nix takes them from the build host's `/opt/onbld`, and docs/registry/recipes.tsv at 5a8e5ce has no onbld row. ## Reference - [illumos Developer's Guide](https://illumos.org/books/dev/) (PDL): Workflow, Debugging, Integrating. - [illumos docs: Building illumos](https://illumos.org/docs/developers/build/), [Contributing](https://illumos.org/docs/contributing/), [Gerrit](https://illumos.org/docs/contributing/gerrit/), [Flag Days](https://illumos.org/docs/developers/flagdays/) (MPL-2.0). - [Modular Debugger Guide](https://illumos.org/books/mdb/), [Dynamic Tracing Guide](https://illumos.org/books/dtrace/) (PDL). - onbld man pages at illumos-gate b566148: nightly, onu, git-pbchk, cstyle, hdrchk, wsdiff, check_rtime (CDDL).