Go servers and clients using belts

In a previous post I described the new belt abstraction, something similar to a Go channel, but capable of propagating error information from producers to consumers and capable of notifying producers that the consumers are no longer interested on more data, so they could terminate gracefully.

In this post I describe an example server and client using belts.

First, the server.  This is the code from a a test of the net/srv package.
imsg := 0
smsg := ""
h := NewBeltConn(&imsg, &smsg)

// An echo server from inb to outb.
go func() {
inb := h.In
outb := h.Out
defer outb.Close()
for {
m, err := inb.Rcv()
if err == belt.ErrClosed {
break
}
if err != nil {
Printv("srv: err %v\n", err)
err = outb.Snd(err)
} else {
Printv("srv: msg %v\n", m)
err = outb.Snd(m)
}
if err != nil {
t.Fatalf("srv: send: %s", err)
}
}
}()

s := New("test", "8081", h)
if err := s.Serve(); err != nil {
t.Fatal(err)
}

Before looking at the different parts, just see how the echo server can simply keep on receiving from the input belt and can tell errors from the client apart from regular client data. In fact, we echo errors back to the client.

Furthermore, if at some point the server does not want to receive more data from the client, it would simply inb.Close()and that would cleanly stop the process reading from the underlying network connection and sending data to the input belt.

The function NewBeltConn is as follows:

func NewBeltConn(proto ...interface{}) BeltConn {

h := BeltConn{

In:  belt.New(),

Out: belt.New(),

}

h.In.SetProto(proto...)

h.Out.SetProto(proto...)

return h

}

It creates two belts, one for input and one for output. Furthermore, the concrete values given as arguments are used in SetProto to tell the belts which data types they should accept as valid conveyed values (besides errors, which are always valid).

The code 

s := New("test", "8081", h)
if err := s.Serve(); err != nil {
t.Fatal(err)
}

from the network server package creates a new server listening at the indicated port, named test. This server spawns two processes per client. One calls belt.PipeFrom to convey data from the network to the input server belt, and the other calls belt.PipeTo to convey messages sent to the output belt back to the network.

In the normal situation, when the client closes the connection, the PipeFrom process closes the input belt and the server loop notices, closing then the output belt, which leads to the server network connection (to the client) being closed.

In the abnormal situation that the server decides to stop, it may close its input belt (and not just its output belt) to cleanly stop the two auxiliary processes on the server side.

The nice thing is that the server code is exactly the same code that would be used if it were just echoing to the client within the same machine. Because errors are propagated along with data (a send on a channel won't fail, but a send on a network or a pipe might fail).

Going back to the client, it might be as follows. This one is also from the tests for the package.

// client: send an error and several int/string messages.

ch, err := DialBeltTCP("[::1]:8081", &imsg, &smsg)

if err != nil {

t.Fatal(err)

}

if err := ch.Out.Snd(errors.New("errmsg")); err != nil {

t.Fatalf("send: %s", err)

}

for i := 0; i < 10; i++ {

Printv("cli: send %v\n", i)

if i%2 == 0 {

if err := ch.Out.Snd("str"); err != nil {

t.Fatalf("send: %s", err)

}

continue

}

if err := ch.Out.Snd(i); err != nil {

t.Fatalf("send: %s", err)

}

}

ch.Out.Close()

The first line calls DialBeltTCP, which is a convenience function to dial a TCP address and link an input and output belt to the resulting connection. Like in the case of the server, it relies on a BeltConn and two processes using belt.PipeFrom and belt.PipeTo to relay between the belts and the connection.

The pointers to values following the dialled address specify the protocol for the belts, so they could check out which messages are sent as part of the protocol and could configure themselves to be able to marshal and un-marshal those. 

As expected, the client can send any of the types configured, plus error indications.

To receive replies, the client would simply call ch.In.Rcv, like shown here for sending.

The interesting bit is that,  considering the chain made out of "client sender", "pipe to network", "server pipe from network", server echo, "server pipe to network", "client pipe from network", "client receiver", if at any point one process decides to stop, it can close both its input and output and the entire chain is shutdown both to the left and to the right of the closing point, cleanly, without making the code more complex or adding extra channels to the mix.

More on Channels and belts

This is a second cut at channels done right, after a nice idea from Roger.
This time I prefer to link to a pdf version of the draft for the paper describing them,
so it is easier to read.

The source may be retrieved (as other lsub go public source) by cloning or go getting from git.lsub.org/go.git. (No web server there, only a git server).

Channels done right

This is from a early draft for some work we are doing on belts, a new abstraction for
a new system inheriting from nix and go.

[It seems the blogger editor mangled things a little bit when I pasted the text, which is
a shame. I'll post again when there's a TR and when the code is made public, so I simply
adjusted things a little bit by hand; sorry.]

Francisco J. Ballesteros
ABSTRACT
Channels in the style of CSP are a powerful abstraction. The Go language includes them, inheriting much from earlier languages from Plan 9 and Research UNIX. But, to use them as a system abstraction it is necessary to be able to send in-band errors along with data and to stop senders when receivers are no longer interested in the data being sent. This paper describes Belts, a new channel abstraction that includes such features.
Channels
The Go language includes channels as a builtin type. A channel is includes in its type the data type for ele- ments sent through it. For example,
var c chan int
declares a channel to send int values. They are created using make in either unbuffered or buffered variants. For example, this declares and creates a couple of channels:

             unbufc := make(chan int)
             bufc := make(chan int, 10)

Receiving from a channel blocks until an element can be received. Sending on an unbuffered channel blocks until another process receives from it. Sending on a buffered channel blocks only when the buffer is full. One operator is used to send or receive, depending on which side of the channel it is written. For example:

bufc <- 0
x := <-bufc
unbufc <- 0

// send 0, does not block
// receive 0, copied to x.
// send 0, blocks (no proc receiving)

Here the first two sentences proceed without blocking, because the message is buffered in the channel. The last blocks because nobody is receiving in this example.
There is a construct to select one among multiple send or receive operations. When no operation may proceed, the construct blocks. One some may proceed, one is executed at random and the construct termi- nates. For example:

             select {
             case c1 <- 3:

                     // send 3 as soon as we can send to c1
             case c2 <- 5:

                     // send 5 as soon as we can send to c2
             case x := c3:

                     // receive x from c3 as soon as we can.

}
This constructs admits a default label to execute when no send or receive operation may be used, which leads to non-blocking variants of send and receive. In this example,

             x := 0
             select {
             case x = <-c:

                     // receive x from c
             default:

                     // didn’t receive, and didn’t block

}
we pretend that we received a zero if we couldn’t receive anything.
Another feature of channels is that we may close a channel, to signal the receiver that no further data will be sent on it. In this example we send two values and close the channel, and later (or concurrently in another process) we receive from the channel until we note it is closed:

             // send something and close it.
             c <- 1
             c <- 2
             close(c)

             // receive from c until closed or a zero value is sent.
             for <-c != 0 {
             }

Once closed, a receive returns a zero value without blocking. But we can also check if the channel was closed or if a zero value was just sent:

             if x, ok := <-c; !ok {
                     // c was closed

}
In this case, ok is set to false if we couldn’t receive becase the channel was closed. Usually, to loop receiv- ing the range operator is used instead:

             for x := range c {
                     // use x as received from c

}
It is aware of close and behaves nicely when the sender is done.
There are more features documented in the Go Programming Language Specification, but what has
been said is enough to understand our motivation and the discussion that follows.
Going problems
There are problems with the behavior of channels as described in the previous section.
One problen is that, to use channels as the primary structure used to glue different processes in one application, we should be able to stop the sender when the receiver is no longer interested in data being sent.
It has been argued that a second channel can be used to convey a termination note from the receiver of data to the producer. However, this complicates the interfaces between the elements involved. If one process is a data producer and another is a consumer, we should be able to connect them in very much the same way we do with pipes:

producer | consumer

or in this case

             c := make(chan data)
             // start a producer
             go produce(c)
             // start a consumer
             go consume(c)

The code for the producer and the consumer should be as simple as follows:

             func produce(c chan data) {
                     for {

                             x := make a new item
                             if c <- x failed {

                                   break
                             }
                     }

             }
             func consume(c chan data) {

                     for x := range c {
                             // use x

                             if don’t want to use more {
                                     tell c we are done

                                     break
                              } 
                     }
             }
The argument against notifying the sender about the stop of the receiver is that a channel should convely data only from the sender to the receiver. But, to cleanly terminate the sender if the receiver decides to stop we have to complicate the code and do one of three things:

1. 1  Use a second channel to notify the sender that the receiver is done.
   
2. 2  Spawn a new process (or use the receiver process) to consume the rest of the stream of data without actually processing it.
   
3. 3  Let the sender block forever and forget about it.
   

The last two cases are a waste of resources, and thus should not be used in practice. In the first case, the code would be more complex:

where

c := make(chan data)
endc := make(chan bool)
// start a producer
go produce(c, endc)
// start a consumer
go consume(c, endc)

where 

func produce(c chan data, endc chan bool) {
        for {

             x := make a new item
             select {
             case c <- x:
             case <-endc:

                return 

             }
        }
}

and

func consume(c chan data, endc chan bool) {
        for {

               x, ok := <-c:
               if !ok {

                     break

                }
                // use x
                if don’t want more {

                        close(endc)
                        break

                 } 
        }
}

Considering this code, the connection between then sender and the receiver is now two-ways. As it would be if we could use the channel to indicate that we are done receiving, so the argument against a backward flow of information does not seem sound at this point. We are still sending information backward, but, the code is more complex and what would be a single abstraction for message passing is now two separate structures.
But there are more problems. Another important one is that if the sender fails to produce an item, the error indication is lost and can’t be send to the receiver. The receiver should have a way to know that the sender had a problem, perhaps to propagate the error to others interested in the result.
To achieve this, we must further complicate the scheme to use data structure that packs either data or an error indication, and then complicate the receiver code to unpack it. Or we must use a separate error channel to convey error messages, which is even more complex.
If should be easy to let the producer notify an error to the consumer and terminate, and then let the consumer notice at each reception if it was an error or a regular data message.
Belts: problems are gone
To fix this issues, a new abstraction, belt channels has been built. By now, we have not modified the lan- guage to include it, but written a package providing a data type for the new abstraction. A belt is defined as

     // An typed belt channel.
     type Chan struct {

             Donec chan bool        // notify the sender that the receiver is done.
             Datac chan interface{} // send data or errors.
             /* other unexported fields */

}
The Datac conveys data and the Donec conveys receiver-close indications to senders. We left these fields public to let clients use belts in select constructs using more than one channel or belt but in the future it is likely that all this will be hidden.
A belt can be used to send data or errors:
     func (b *Chan) Snd(d interface{}) error
Here, d would be the desired data type to be sent or an error indication. Unlike with channels, if the receiver is done, the send operation returns an error to the caller, and it decides what to do next. Perhaps stop.
The receive operation tells errors apart from data:

     func (b *Chan) Rcv() (interface{}, error)

If the sender is done, an error indication is returned. In the same way, if the sender posts an error through the belt the receiver will get an error indication instead of data. Otherwise, a piece of data sent is received. The following operations can be used to close a belt for sending or for receiving:

     func (b *Chan) CloseSnd()
     func (b *Chan) CloseRcv()

The constructor functions creates either a buffered or an unbuffered belt as it could be expected. 
     func New() *Chan

     func NewBuffered(nbuf int) *Chan

Another interesting feature enabled is that belts are polymorphic, unlike channels. Any type can be sent. The language and its type assertions and reflection can be use to keep type safety, but multiple different data items can be sent easily.
A protocol can be defined in a belt so that only certain message types (plus error indications) are accepted (and other messages are rejected with error when trying to be sent). To define a protocol, the next operation accepts an array (a slice actually) of example message values, each one being a pointer to a mes- sage instance.

     func (b *Chan) SetProto(msgptrs ...interface{})

In this case, the reflection interface in the language is used to accept or reject messages to be sent through the belt. This makes it easy to define protocols made by different message types without having to define facades for the set of messages.
There are wrappers that adapt belts to traditional reader and writer interfaces, so it would be easy to write to a belt, read from it, or write a belt to a writer (to copy the data streamed to an external writer).
Futher wrappers know how to pipe a belt to an external connection (a writer) and how to pipe a belt from an external connection (a reader). In this case, the connection carries messages encoded as Gobs that carry any of the types defined in the belt protocol or an error indication. This permits one belt to be con- veyed through a network connection or through a system file or pipe.
It would be hard to do this with standard channels, because of the problems mentioned. However, it is easy to maintain reasonable semantics that work fine in practice when writing clients and servers that use belts to and from the network. Each network connection requires two different belts if it is to be duplex.
Example
As an example, this is how our example producer and consumer processes might be written:

func produce(c *belt.Chan) {
     for {

        // make a new x
        if err := c.Snd(x); err != nil {

                // couldn’t send. done.
                break

        }
        // or to send an error...
        c.Send(err)

      }
      c.CloseSnd()

}

func consume(c *belt.Chan) {
        for { 

            x, err := c.Rcv()
            if err != nil {

                  // couldn’t receive. done.
                  break

            }
            // use x
            if don’t want more {

                  c.CloseRcv()

            } 
        }
}

The result is quite similar to that of a UNIX Pipe, although belts do not kill the sender process when the
receiver is gone (because they might have to terminate cleanly or might decide to do other things).

Early evaluation
The next is the output from the package benchmarks, which try to send 512 byte slices by different mecha- nisms.

BenchmarkChan   20000000
BenchmarkAlt     5000000
BenchmarkTSend   5000000
BenchmarkTWrite  5000000
BenchmarkTRead   1000000
BenchmarkPipe    1000000

 134 ns/op
 345 ns/op
 418 ns/op
 723 ns/op

1394 ns/op
2476 ns/op

The first one uses a native channel without being able to stop the sender. That is the fastest. The second uses a select construct to let the receiver stop the sender. The third one uses a belt, and is not too slow when compared to the second one. We have still to optimize the code, but it is fast enough to be used in practice as it stands, compared to using a select directly. The extra time taken is probably due to the use of reflec- tion.
The last three ones report the performance when using adaptors to write on the belt, or to read from the belt, and the performance of a standard Pipe as implemented by go as a reference for this case.
Future work
We will optimize and fine tune the interfaces for the belts in the near future, and will probably experiment by modifying the language to make belts first-class citizens, so they can be used like channels in select and other constructs. We will also experiment with belts used for network communication and conduct further evaluation for them. 

Mark IV

We have exported our mark iv kernel.
It can be found at the Nix page.
Beware it's a development version not yet in production. We have been using it but you should expect bugs and problems.
Enjoy either way.

A description of the selfish Nix allocator

This TR from the Lsub papers page describes the Nix allocator and provides a bit of initial evaluation for it. In short, half of the times allocation of resources could be done by the allocating process without interlocking with other processes or cores and without disturbing any other system component. Plus other benefits described in the TR.

Here is a picture of the allocator as a teaser:

Selfish allocators in action

This is the output from the alloc device. The last few lines show that for paths,
chans, and, what is more important, I/O blocks, most of the times a process
could do the allocation by itself. It took a free structure out of the small number
of structures that are kept in the process structure in case it later allocates
whatever it liberated before.
For example, 13891 block allocations were done without reaching the central block allocator,
and 14440 were done using the block allocator.

Ouch!, the writing was a mistake!. It's 13891 self-allocations out of a total of 14440 allocations. I'm making this note editing the post. Thus, note that almost all of the allocations are a self-service, not just half of the allocations.

This is utterly important for many-core machines. Half Most of the times it's a self-service.

I will post here a paper describing the allocator in a near future.

% cat /dev/alloc
3473408/67108864 cache bytes
72/100 cache segs 0/0 reclaims 1 procs
0/7 rpcs
98/102 segs
12/100 text segs 0/0 reclaims
1071104000 memory
15663104 kernel
0/0 1G pages 0 user 0 kernel 0 bundled 0 split
0/496 2M pages 0 user 0 kernel 0 bundled 6 split
423/968 16K pages 229 user 194 kernel 0 bundled 0 split
82/99 4K pages 0 user 82 kernel 3792 bundled 0 split
6/6 pgas
1447/1509 pgs
169/181 path 5535/5716 self allocs 5563/5575 self frees
103/123 chan 7269/7392 self allocs 7297/7317 self frees
13796608/14192640 malloc 1 segs
56/63 block 13891/14440 self allocs 13945/14438 self frees
0/8388608 ialloc bytes
61/82 mmu 4096 pages

Selfish processes

There is an important optimization not described in previous posts, and not
considered in the evaluation and the traces shown there. The idea is
to let processes keep a few of the resources they release in case
they are needed later.

In particular, we modified the process structure to keep up to 10 pages
(of the size used for user segments). When a process releases a page
and has less than 10 pages kept, it simply keeps the page without
releasing it. Later, if a new page is needed it would first try to use
one from the per-process pool. The pool is not released when a process dies.
Instead, the pool is kept in the process structure and will be used
again when a new process is allocated using it.

The trace output taken after applying this optimization shows that
most of the pages are reused, and that for small cached programs about 1/3
of the allocations are satisfied with the per-process pool. Thus,
the contention on the central page allocator is greatly reduced with this
change.

Per process resource pool should be used with care. For example, our
attempts to do the same with the kernel memory allocator indicated that
it is not a good idea in this case. Memory allocations have very different
sizes and some structures are very long lived while others are very short
lived. Thus, what happen was that memory was wasted in per process pools
and, at the same time, not many memory allocations could benefit from this
technique.

In general, per-process allocation pools are a good idea when the structures
are frequently used and have the same size. For example, this could be
applied also to Chan and Path structures as used on Nix.

A memory trace for command execution in Nix

This is the output for executing

% cat /dev/alloc

in the steady state. That is, after executing the same command a few times so that all the previous things that could be cached are indeed cached. The file used, by the way, is reporting the statistics of several allocators in Nix, including the page allocator. So there is more information in the trace described here than just the set of faults and pages allocated or deallocated.

The events include everything since we typed the command until the shell prints out its prompt.
Thus, there is no microbenchmark trick going on.

The first faults are due to fork, used by rc to spawn the child process. Then, we have more faults for the child while it proceeds to execute the cat command. Pid 21 is the original shell process; pid 27 is the child that will execute pwd.

Let's see the trace before and then make a few remarks near the end of this post.

A fault caused because fork makes the data segment memory shared, but copied on reference. Here the parent references a data page:

fault pid 21 0x400000 r
fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x3fe78000 r2 n1
newpage 0x400000 -> 0x3fe24000 src 0xfffffffff018711c
fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x400000 pg 0x3fe24000 ref 1


The child starts executing and references a new text address:

fault pid 27 0x20f9e0 r
fixfault pid 27 s /bin/rc sref 9 Text 0x200000 addr 0x20c000 pg 0x1160000 r1 n1
fixfaulted pid 27 s /bin/rc Text 0x200000 addr 0x20c000 pg 0x1160000 ref 1


Another fault in the parent due to the copy on reference. This is because we flushed the MMU state during fork, but the page is already there, there is nothing to do to handle this type of fault, other than updating the MMU state to refer to the page, which is already there.

fault pid 21 0x404b30 w
fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x3fe74000 r2 n1
newpage 0x404000 -> 0x3fe64000 src 0xfffffffff018711c
fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x404000 pg 0x3fe64000 ref 1


The child continues executing and references a new text address:

fault pid 27 0x20a9d9 r
fixfault pid 27 s /bin/rc sref 9 Text 0x200000 addr 0x208000 pg 0x1188000 r1 n1
fixfaulted pid 27 s /bin/rc Text 0x200000 addr 0x208000 pg 0x1188000 ref 1

The child continues executing and references a data address:

fault pid 27 0x400060 w
fixfault pid 27 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x3fe78000 r1 n1
fixfaulted pid 27 s /bin/rc Data 0x400000 addr 0x400000 pg 0x3fe78000 ref 1

Another fault in the parent due to the copy on reference, like before.

fault pid 21 0x409d5c r
fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x3fe70000 r2 n1
newpage 0x408000 -> 0x3fe68000 src 0xfffffffff018711c
fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x408000 pg 0x3fe68000 ref 1

The child continues executing and references a new text address:

fault pid 27 0x202aec r
fixfixfault pid 27 s /bin/rc sref 9 Text 0x200000 addr 0x200000 pg 0x10cc000 r1 n1
fixfaulted pid 27 s /bin/rc Text 0x200000 addr 0x200000 pg 0x10cc000 ref 1

Another fault in the parent due to the copy on reference, like before.

fault pid 21 0x40c178 r
fault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x3fe4c000 r2 n1
newpage 0x40c000 -> 0x3fe6c000 src 0xfffffffff018711c
fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x3fe6c000 ref 1

The child continues executing and references a data address:

fault pid 27 0x40a738 r
fixfault pid 27 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x3fe70000 r1 n1
fixfaulted pid 27 s /bin/rc Data 0x400000 addr 0x408000 pg 0x3fe70000 ref 1

The child continues executing and references a new text address:

fault pid 27 0x212b47 r
fixfault pid 27 s /bin/rc sref 9 Text 0x200000 addr 0x210000 pg 0x118c000 r1 n1
fixfaulted pid 27 s /bin/rc Text 0x200000 addr 0x210000 pg 0x118c000 ref 1

The child continues executing and references a data address:

fault pid 27 0x404b30 w
fixfault pid 27 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x3fe74000 r1 n1
fixfaulted pid 27 s /bin/rc Data 0x400000 addr 0x404000 pg 0x3fe74000 ref 1

The child continues executing and references a data address:

fault pid 27 0x40c17c r
fixfault pid 27 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x3fe4c000 r1 n1
fixfaulted pid 27 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x3fe4c000 ref 1


We are now doing the exec in the child process...



pgfree pg 0x3fe78000
pgfree pg 0x3fe74000
pgfree pg 0x3fe70000
pgfree pg 0x3fe4c000

The child references a data address, which is not paged in from the file. The diagnostic indicates that the page is paged in from the cached image of the executable file, but no I/O was required. Simply, a new page is allocated and initial values for data are copied.

fault pid 27 0x400018 w
fixfault pid 27 s /bin/cat sref 1 Data 0x400000 addr 0x400000 pg 0x0 r0 n-1
pagein pid 27 s 0x400000 addr 0x400000 soff 0x0
newpage 0x400000 -> 0x3fe4c000 src 0xfffffffff018711c
fixfaulted pid 27 s /bin/cat Data 0x400000 addr 0x400000 pg 0x3fe4c000 ref 1

0/67108864 cache bytes
0/10 cache segs 0/0 reclaims 0 procs
0/3 rpcs
22/23 segs
8/10 text segs 0/0 reclaims
1071140864 memory
15663104 kernel
0/0 1G pages 0 user 0 kernel 0 bundled 0 split
0/502 2M pages 0 user 0 kernel 0 bundled 0 split
134/236 16K pages 119 user 15 kernel 0 bundled 0 split
61/61 4K pages 0 user 61 kernel 884 bundled 0 split
6/6 pgas
1447/1509 pgs
66/67 path
69/70 chan
11780256/14229504 malloc 1 segs
0/8388608 ialloc bytes
61/61 mmu 4096 pages

And we are done.

pgfree pg 0x3fe4c000

There are a few things to note here. One important thing is that there are no faults in the stack segments. That is because we copy stack memory and pre-install the entries in the MMU. Stacks in nix are small and it pays to copy them.

Another thing to note is that the text is not paged in. It is served from the cache. Thus, only updating the MMU is needed.

Note also how page allocation and deallocation is quite reduced, compared to what it could be. The reason is that stacks retain their memory, and that the page size is 16K, and so, not many page allocations are required.

Also, if you compare this with the trace from a standard Plan 9, you will notice other effects, described in a previous post in the Nix memory management TR.

Early evaluation of memory management in Nix mark IV

In a previous post I published a link to a TR describing the
recent work on memory management for Nix. Such TR has
been updated to include some early evaluation, which I reproduce
here.

To measure the  impact  of  the  different  changes  in  the
behavior  of the system, we took the final system and run it
with diagnostic output enabled for page allocation and  page
faults,  and measured the different events of interest. Then
we did the same disabling one or more of  the  improvements.
The  results  are not fully precise because debugging output
may miss some events sometimes. Further evaluation will  use
counters instead, and compare with a stock Plan 9 system.

     We have to say that the impact of the changes  is  more
dramatic  than shown by the results, because early modifica-
tions for the mount driver and other parts of the system, on
their  own, do a good job reducing the impact of system load
(e.g., by better caching).

     The variations of the system executed are  given  these
names in the results:

all

     The standard Nix mark IV. All new features are in.

nopf

     Prefaulting code for text and stack  segments  is  dis-
     abled.  Such  code  installs  into  the MMU entries for
     those pages already present in the segments (because of
     the  cache  or other optimizations). The alternative is
     installing entries on demand.

flush

     Prefaulting code is disabled and the MMU is flushed  on
     forks, as it is customary on Plan 9. The alternative to
     MMU flushes is flushing just the entries for  the  data
     segment  (others do not have to be because of optimiza-
     tions explained before).

nodeep

     Deep stack forks are disabled. Deep stack  forks  imply
     copying  the  actual  stack  memory  during  forks. The
     alternative is the standard copy on reference to fork a
     stack segment.

nostk

     Stack segments are not cached (their memory is not kept
     and  recycled) and deep stack copies are not performed.
     The alternative is the standard construction  by  zero-
     fill  on demand (due to page faults) and full dealloca-
     tion and allocation of stacks when processes  exit  and
     are created.

none

     There are no prefaulting code, the MMU  is  flushed  on
     forks, deep stack copying is disabled, and stack memory
     is not cached. Quite similar to the state of the system
     before  any  modification was made, but for using 16KiB
     pages for user segments.


none4k

     This is the old system.  Like the  previous  variation,
     but using standard 4KiB pages for user segments.

The system booted normally from a remote file server to exe-
cute  a shell instead of the full standard start script, and
then we executed  pwd twice. The  first  table  reports  the
results  for the second run of  pwd, counting since we typed
the command to the time when the shell  printed  its  prompt
after   pwd completed. The first way to read the table is to
compare any row with the first or the last one, to  see  the
impact of a particular configuration.

     The second table shows the same counters  but  for  the
entire execution of the system, from a hardware reset to the
prompt after executing  pwd twice.

_________________________________________________________________________
 bench    page allocs   page frees   mmu faults   page faults   page-ins
_________________________________________________________________________
  all          6             5           14           11           1
_________________________________________________________________________
  nopf         6             5           16           12           1
_________________________________________________________________________
 flush         6             5           22           18           1
_________________________________________________________________________
 nodeep        6             6           17           12           1
_________________________________________________________________________
 nostk         6             7           16           15           1
_________________________________________________________________________
  none         8             8           24           17           1
_________________________________________________________________________
 none4k       15            15           65           68           1
_________________________________________________________________________

Page  allocations,  page  deallocations,  page  faults,  MMU
faults,  and  pages paged in for variations of the system on
the second execution of a simple command.

_________________________________________________________________________
 bench    page allocs   page frees   mmu faults   page faults   page-ins
_________________________________________________________________________
  all         229           41          219           210         107
_________________________________________________________________________
  nopf        231           41          246           232         109
_________________________________________________________________________
 flush        224           38          311           283         109
_________________________________________________________________________
 nodeep       227           41          244           237         109
_________________________________________________________________________
 nostk        232           56          245           233         109
_________________________________________________________________________
  none        236           60          321           296         109
_________________________________________________________________________
 none4k       501          107          847           843         313
_________________________________________________________________________

Page  allocations,  page  deallocations,  page  faults,  MMU
faults,  and pages paged in for variations of the system for
an entire boot and two executions of a simple command.

Several things can be seen:

∙    going from the old to the new system means  going  from
     68  down  to 11 page faults, just for running  pwd from
     the shell.  For the entire boot process it means  going
     from 843 down to 210.

∙    Using a more reasonable page size, without other  opti-
     mizations,  reduces a lot the number of page faults (as
     could be expected). We saw that almost all the 4K pages
     paged  in are actually used for 16K pages, thus it pays
     to change the page size. Also, the new page size has  a
     significant  impact  in  the size of reads performed by
     the mount driver, because it enables  concurrent  reads
     for larger sizes.

∙    Deep stack copying reduces a little the page faults  in
     the system, but it might not be worth if the time taken
     to zero out the stack pages kept in the cache is wasted
     when  they  are  not  used. In our system that does not
     seem to be case.

∙    More efforts should be  taken  to  avoid  flushing  MMU
     state.  As  it  could be expected, not flushing the MMU
     when it is not necessary reduces quite a bit the number
     of page faults.

As a reference, here we list the trace output for the

old  and  the new system for executing  pwd the second time.

It is illustrative to compare them.

This is for the new system:

% pwd

newpg 0x000000003fe44000 pgsz 0x4000 for 0x4000

fault pid 23 0x20f9e0 r

fault pid 21 0x400000 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20c000 pg 0x1174000 r1 n1

fixfixfault faulted pid 23 s /bin/rc Text 0x200000 addr 0x20c000 pg 0x1174000 ref 1

fault pid 23 0x20a9d9 r

pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x3fe6c000 r2 n1

newpg 0x000000003fe80000 pgsz 0x4000 for 0x4000

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x400000 pg 0x3fe80000 ref 1

fault pid 21 0x404b30 w

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x208000 pfixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x3fe70000 r2 n1

newpg 0x000000003fe84000 pgsz 0x4000 for 0x4000

g 0x11a4000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x404000 pg 0x3fe84000 ref 1

fault pid 21 0x409afc r

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x208000 pg 0x11a4000 ref 1

fault pid 23 0x400060 w

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x3fe78000 r2 n1

newpg 0x000000003fe88000 pgsz 0x4000 for 0x4000

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x408000 pg 0x3fe88000 ref 1

fault pid 21 0x40c178 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x3fe6c000 r1 n1

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x400000 pg 0x3fe6c000 ref 1

fault pid 23 0x202aec r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x3fe74000 r2 n1

newpg 0x000000003feaddr 0x200000 pg 0x10e4000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x200000 pg 0x10e4000 ref 1

fault pid 23 0x40a698 r

8c000 pgsz 0x4000 for 0x4000

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x3fe8c000 ref 1

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x3fe78000 r1 n1

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x408000 pg 0x3fe78000 ref 1

fault pid 23 0x212b47 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x210000 pg 0x11a8000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x210000 pg 0x11a8000 ref 1

fault pid 23 0x404b30 w

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x3fe70000 r1 n1

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x404000 pg 0x3fe70000 ref 1

fault pid 23 0x40c17c r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x3fe74000 r1 n1

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x3fe74000 ref 1

fault pid 23 0x7ffffeffbfe0 w

fixfault pid 23 s '' sref 1 Stack 0x7ffffdffc000 addr 0x7ffffeff8000 pg 0x3fe60000 r1 n1

fixfaulted pid 23 s '' Stack 0x7ffffdffc000 addr 0x7ffffeff8000 pg 0x3fe60000 ref 1

pgfree pg 0x3fe6c000

pgfree pg 0x3fe70000

pgfree pg 0x3fe78000

pgfree pg 0x3fe74000

fault pid 23 0x400018 w

fixfault pid 23 s /bin/pwd sref 1 Data 0x400000 addr 0x400000 pg 0x0 r0 n-1

pagein pid 23 s 0x400000 addr 0x400000 soff 0x0

newpg 0x000000003fe74000 pgsz 0x4000 for 0x4000

fixfaulted pid 23 s /bin/pwd Data 0x400000 addr 0x400000 pg 0x3fe74000 ref 1

/usr/nemo

pgfree pg 0x3fe74000

And this is for the old system:

fault pid 21 0x2000c4 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x200000 pg 0x116c000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x200000 pg 0x116c000 ref 1

fault pid 21 0x20170a r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x201000 pg 0x116f000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x201000 pg 0x116f000 ref 1

fault pid 21 0x205dfc r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x205000 pg 0x1132000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x205000 pg 0x1132000 ref 1

fault pid 21 0x40b834 w

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40b000 pg 0x11b3000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40b000 pg 0x11b3000 ref 1

fault pid 21 0x407c78 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x407000 pg 0x11ac000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x407000 pg 0x11ac000 ref 1

fault pid 23 0x20f9e0 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20f000 pg 0x1138000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20f000 pg 0x1138000 ref 1

fault pid 23 0x7fffffffefe8 w

fixfault pid 23 s /bin/rc sref 1 Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11b0000 r2 n1

newpg 0x00000000011d2000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s '' Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11d2000 ref 1

fault pid 23 0x20a9d9 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20a000 pg 0x1146000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20a000 pg 0x1146000 ref 1

fault pid 23 0x20be91 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20b000 pg 0x1135000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20b000 pg 0x1135000 ref 1

fault pid 23 0x400060 w

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x11aa000 r2 n1

newpg 0x00000000011d0000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x400000 pg 0x11d0000 ref 1

fault pid 23 0x202aec r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x202000 pg 0x1130000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x202000 pg 0x1130000 ref 1

fault pid 23 0x40a698 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x40a000 pg 0x119e000 r2 n1

newpg 0x00000000011d6000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x40a000 pg 0x11d6000 ref 1

fault pid 23 0x20977d r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x209000 pg 0x1139000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x209000 pg 0x1139000 ref 1

fault pid 23 0x20dbe3 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20d000 pg 0x113b000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20d000 pg 0x113b000 ref 1

fault pid 23 0x212b47 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x212000 pg 0x113c000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x212000 pg 0x113c000 ref 1

fault pid 23 0x401338 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x401000 pg 0x11af000 r2 n1

newpg 0x00000000011ce000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x401000 pg 0x11ce000 ref 1

fault pid 23 0x210670 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x210000 pg 0x113d000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x210000 pg 0x113d000 ref 1

fault pid 23 0x404b30 w

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x11ae000 r2 n1

newpg 0x00000000011d3000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x404000 pg 0x11d3000 ref 1

fault pid 23 0x21107c r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x211000 pg 0x1143000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x211000 pg 0x1143000 ref 1

fault pid 23 0x40c17c r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x11b2000 r2 n1

newpg 0x00000000011cf000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x11cf000 ref 1

fault pid 23 0x4097fc r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x409000 pg 0x1196000 r2 n1

newpg 0x00000000011ca000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x409000 pg 0x11ca000 ref 1

fault pid 23 0x20e02d r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20e000 pg 0x1147000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20e000 pg 0x1147000 ref 1

fault pid 23 0x20cbb0 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x20c000 pg 0x1129000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x20c000 pg 0x1129000 ref 1

fault pid 23 0x40204a r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x402000 pg 0x11a7000 r2 n1

newpg 0x00000000011c6000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x402000 pg 0x11c6000 ref 1

fault pid 23 0x2086dc r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x208000 pg 0x1165000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x208000 pg 0x1165000 ref 1

fault pid 23 0x213000 r

fixfault pid 23 s /bin/rc sref 9 Text 0x200000 addr 0x213000 pg 0x1142000 r1 n1

fixfaulted pid 23 s /bin/rc Text 0x200000 addr 0x213000 pg 0x1142000 ref 1

fault pid 23 0x4055b8 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x405000 pg 0x11a9000 r2 n1

newpg 0x00000000011d8000 pgsz 0x4000 for 0x1000

splitbundle pg 0x00000000011d8000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x405000 pg 0x11d8000 ref 1

fault pid 23 0x4081a8 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x11a8000 r2 n1

newpg 0x00000000011db000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x408000 pg 0x11db000 ref 1

fault pid 23 0x407c78 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x407000 pg 0x11ac000 r2 n1

newpg 0x00000000011da000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x407000 pg 0x11da000 ref 1

fault pid 23 0x406dd8 r

fixfault pid 23 s /bin/rc sref 1 Data 0x400000 addr 0x406000 pg 0x11ad000 r2 n1

newpg 0x00000000011d9000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/rc Data 0x400000 addr 0x406000 pg 0x11d9000 ref 1

fault pid 21 0x7fffffffefe8 w

fixfault pid 23 0x7ffffeffefe0 w

fixfault pid 23 s '' sref 1 Stack 0x7ffffdfff000 addr 0x7ffffeffe000 pg 0x0 r0 n-1

newpg 0x00000000011dc000 pgsz 0x4000 for 0x1000

splitbundle pg 0x00000000011dc000

fixfaulted pid 23 s '' Stack 0x7ffffdfff000 addr 0x7ffffeffe000 pg 0x11dc000 ref 1

pgfree pg 0x11d2000

pgfree pg 0x11d0000

pgfree pg 0x11ce000

pgfree pg 0x11c6000

pgfree pg 0x11d3000

pgfree pg 0x11d8000

pgfree pg 0x11d9000

pgfree pg 0x11da000

pgfree pg 0x11db000

pgfree pg 0x11ca000

pgfree pg 0x11d6000

pgfree pg 0x11cf000

faultfault pid 23 0x7fffffffef98 w

fixfault pid 23 s /bin/pwd sref 1 Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11dc000 r1 n1

fixfaulted pid 23 s '' Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11dc000 ref 1

fault pid 23 0x20008a r

fixfault pid 23 s /bin/pwd sref 3 Text 0x200000 addr 0x200000 pg 0x11cd000 r1 n1

fixfaulted pid 23 s /bin/pwd Text 0x200000 addr 0x200000 pg 0x11cd000 ref 1

fault pid 23 0x400018 w

fixfault pid 23 s /bin/pwd sref 1 Data 0x400000 addr 0x400000 pg 0x0 r0 n-1

pagein pid 23 s 0x400000 addr 0x400000 soff 0x0

newpg 0x00000000011cf000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/pwd Data 0x400000 addr 0x400000 pg 0x11cf000 ref 1

fault pid 23 0x201b6d r

fixfault pid 23 s /bin/pwd sref 3 Text 0x200000 addr 0x201000 pg 0x11d1000 r1 n1

fixfaulted pid 23 s /bin/pwd Text 0x200000 addr 0x201000 pg 0x11d1000 ref 1

 pid 21 s /bfault pid 23 0x2020ef r

fixfault pid 23 s /bin/pwd sref 3 Text 0x200000 addr 0x202000 pg 0x11d4000 r1 n1

fixfaulted pid 23 s /bin/pwd Text 0x200000 addr 0x202000 pg 0x11d4000 ref 1

fault pid 23 0x2040b2 r

fixfault pid 23 s /bin/pwd sref 3 Text 0x200000 addr 0x204000 pg 0x11d7000 r1 n1

fixfaulted pid 23 s /bin/pwd Text 0x200000 addr 0x204000 pg 0x11d7000 ref 1

/usr/nemo

fault pid 23 0x401098 r

fixfault pid 23 s /bin/pwd sref 1 Data 0x400000 addr 0x401000 pg 0x0 r0 n-1

pagein: zfod 0x401000

newpg 0x00000000011d6000 pgsz 0x1000 for 0x1000

fixfaulted pid 23 s /bin/pwd Data 0x400000 addr 0x401000 pg 0x11d6000 ref 1

pgfree pg 0x11dc000

pgfree pg 0x11cf000

pgfree pg 0x11d6000

in/rc sref 1 Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11b0000 r1 n1

fixfaulted pid 21 s '' Stack 0x7ffffefff000 addr 0x7fffffffe000 pg 0x11b0000 ref 1

fault pid 21 0x20f9e0 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20f000 pg 0x1138000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20f000 pg 0x1138000 ref 1

fault pid 21 0x20a9d9 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20a000 pg 0x1146000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20a000 pg 0x1146000 ref 1

fault pid 21 0x20bdca r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20b000 pg 0x1135000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20b000 pg 0x1135000 ref 1

fault pid 21 0x400000 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x400000 pg 0x11aa000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x400000 pg 0x11aa000 ref 1

fault pid 21 0x20ddb3 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20d000 pg 0x113b000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20d000 pg 0x113b000 ref 1

fault pid 21 0x212ea2 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x212000 pg 0x113c000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x212000 pg 0x113c000 ref 1

fault pid 21 0x401338 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x401000 pg 0x11af000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x401000 pg 0x11af000 ref 1

fault pid 21 0x210670 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x210000 pg 0x113d000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x210000 pg 0x113d000 ref 1

fault pid 21 0x404b30 w

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x404000 pg 0x11ae000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x404000 pg 0x11ae000 ref 1

fault pid 21 0x409afc r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x409000 pg 0x1196000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x409000 pg 0x1196000 ref 1

fault pid 21 0x211ba3 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x211000 pg 0x1143000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x211000 pg 0x1143000 ref 1

fault pid 21 0x20e02d r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20e000 pg 0x1147000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20e000 pg 0x1147000 ref 1

fault pid 21 0x208574 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x208000 pg 0x1165000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x208000 pg 0x1165000 ref 1

fault pid 21 0x202c39 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x202000 pg 0x1130000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x202000 pg 0x1130000 ref 1

fault pid 21 0x40a698 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40a000 pg 0x119e000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40a000 pg 0x119e000 ref 1

fault pid 21 0x2097b7 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x209000 pg 0x1139000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x209000 pg 0x1139000 ref 1

fault pid 21 0x213000 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x213000 pg 0x1142000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x213000 pg 0x1142000 ref 1

fault pid 21 0x40c178 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40c000 pg 0x11b2000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40c000 pg 0x11b2000 ref 1

fault pid 21 0x20ce7e r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x20c000 pg 0x1129000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x20c000 pg 0x1129000 ref 1

fault pid 21 0x204bba r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x204000 pg 0x1133000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x204000 pg 0x1133000 ref 1

fault pid 21 0x402611 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x402000 pg 0x11a7000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x402000 pg 0x11a7000 ref 1

fault pid 21 0x405e00 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x405000 pg 0x11a9000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x405000 pg 0x11a9000 ref 1

fault pid 21 0x408d48 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x408000 pg 0x11a8000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x408000 pg 0x11a8000 ref 1

fault pid 21 0x20310e r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x203000 pg 0x1136000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x203000 pg 0x1136000 ref 1

fault pid 21 0x40b834 w

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x40b000 pg 0x11b3000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x40b000 pg 0x11b3000 ref 1

fault pid 21 0x206ab9 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x206000 pg 0x113a000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x206000 pg 0x113a000 ref 1

fault pid 21 0x406820 r

fixfault pid 21 s /bin/rc sref 1 Data 0x400000 addr 0x406000 pg 0x11ad000 r1 n1

fixfaulted pid 21 s /bin/rc Data 0x400000 addr 0x406000 pg 0x11ad000 ref 1

fault pid 21 0x7fffffffce78 w

fixfault pid 21 s /bin/rc sref 1 Stack 0x7ffffefff000 addr 0x7fffffffc000 pg 0x11bd000 r1 n1

fixfaulted pid 21 s '' Stack 0x7ffffefff000 addr 0x7fffffffc000 pg 0x11bd000 ref 1

fault pid 21 0x2071d8 r

fixfault pid 21 s /bin/rc sref 7 Text 0x200000 addr 0x207000 pg 0x116b000 r1 n1

fixfaulted pid 21 s /bin/rc Text 0x200000 addr 0x207000 pg 0x116b000 ref 1