Tracing processes

We have implemented a new tracing tool for processes in nix, also suitable for Plan 9. Previous tools did not suffice because they either blocked the processes being traced or did not report all the process events we were interested on.

The idea is that the kernel adds events to a free-running log of events, which is read from a user program. As long as the user program is fast enough reading them, and the buffer is large enough, we can capture most of the events with a (hopefully) small interference to what the system would do if they were not traced.

The interesting point is that by translating the trace dump to a text format amenable for
an editor, it is easy to integrate a graphical tool that browses the trace with the editor, so you still are able to search, use awk to write tiny scripts interrogating the trace. etc.

Also, because the  trace records the kernel PC for changes from running to sleep, and the new sleep state, it is easy to see where the processes went to sleep, why, and for how much time. Awk is your friend here.

This is a video from an early version of the tracing tool.