The idea is that the kernel adds events to a free-running log of events, which is read from a user program. As long as the user program is fast enough reading them, and the buffer is large enough, we can capture most of the events with a (hopefully) small interference to what the system would do if they were not traced.
The interesting point is that by translating the trace dump to a text format amenable for
an editor, it is easy to integrate a graphical tool that browses the trace with the editor, so you still are able to search, use awk to write tiny scripts interrogating the trace. etc.
Also, because the trace records the kernel PC for changes from running to sleep, and the new sleep state, it is easy to see where the processes went to sleep, why, and for how much time. Awk is your friend here.
This is a video from an early version of the tracing tool.
