Nested control requests in Clive ZX file trees

In a previous post I wrote about ZX file system stacks in Clive. Now I look into their control interfaces, which exhibit a nice feature shown here.

When a ZX file tree operates by relying on another ZX tree deeper in the stack, it adapts its control interface to behave as a facade for it.

For example, the /Ctl file of a ZX file tree can be user to see the status of the debug flags, to see who is using the file tree (in the cases of trees served to the network) and to see usage statistics for the system.

This is the result of reading /Ctl on a stack made out of a CFS that uses a MFS as a cache, and a RFS as the remote tree. The RFS was connected to a remote server exporting a CFS that uses a MFS cache to serve a LFS. What CFS, MFS, RFS, and LFS means is not really relevant. They are just ZX file servers that rely on underlying file servers to do its job (but, CFS is a caching file sever, MFS is a RAM FS, RFS is a remote file server, and LFS is a local file server).

I have removed some of the lines when they do not help to further illustrate the case.

> cat /zx/Ctl

cfs:

fdebug off

vdebug off

noperm off

stat 8220 calls 252 errs 16440 msgs 0 bytes

bgn: min    4.911µs avg  625.628µs max 1.333559776s

end: min    5.045µs avg  625.722µs max 1.333559891s

get 3147 calls 0 errs 16501 msgs 38660497 bytes

bgn: min   54.389µs avg 2.657265ms max 856.808309ms

end: min   63.349µs avg 5.141675ms max 856.841591ms

...

cmfs:

debug off

noperm off

stat 78449 calls 2721 errs 156898 msgs 0 bytes

...

lsub:

user rfs:193.147.15.27:65348 nemo as nemo on 2015-03-26

user rfs:193.147.15.27:65349 nemo as nemo on 2015-03-26

ldebug off

rdebug off

stat 2656 calls 0 errs 5312 msgs 0 bytes

...

mfs:lsub:

debug off

noperm off

stat 47588 calls 2487 errs 95176 msgs 0 bytes

...

lsub:

debug off

rdonly off

noperm off

stat 2854 calls 0 errs 5708 msgs 0 bytes

...

Now, note the lines starting with cfs:, cmfs:, and mfs:lsub:. Only the first section comes from the top-level CFS. Remaining lines come from such server reading the /Ctl file of the underlying ZX file tree and appending it to the data served to the user; and so on as we go down on the stack.

Thus, the user may inspect the ZX stack as deep as it wants.

In the same way, we can set debug on for CFS by executing

> echo debug on > /zx/Ctl

The ZX file server understands writes to the control file as textual commands

to perform control requests.

But we can also set debug on the third tree in the stack:

> echo pass pass debug on > /zx/Ctl

When a control request starts with pass, this word is removed from the request

and the server writes what remains of the control requests to the next tree in the stack.

This makes the control interface for all the trees in the stack available to the final user.

Go and unix traditions are a good match

Recently I had start writing a paper on Clive's ZX (more on this soon) and had to add citations in it. I'm writing the paper using wr(1), the Clive text formatter used to write manual pages, papers, and everything.

Because we had the entire set of bibliography entries from Plan 9 and Plan B in troff's refer format, I though wr could be teach to use them.

Being short in time, I added a new Go package to Clive that eats all the entries in the lsub refer bib directory, that have a format like

Article in conference proceedings
%A M. Bishop
%A L. Snyder
%T The Transfer of Information and Authority
in a Protection System
%J Proceedings of the 7th SOSP
%P 45-54
%D 1979

The wr/refs package exploits Go maps, strings, and slices to record a map of words to bib entries, which are also maps from keys (author, title, etc.) to strings. As easy as this:

// A reference maps from the key (eg. 'A') to values (eg. authors)

type Ref {

Keys map[rune][]string

}

// A bib maps from words found in references to references

type Bib {

refs map[string] map[*Ref]bool

all []*Ref

}

Note how easy is in Go to create sets of arbitrary types by using a map of that type to booleans.

Writing the code for these types feels very much like writing lisp.

Citing now in wr is as easy as writing [bib: plan 9 networks] in the source text.

Wr simply takes all  the words in the citation and looks them up in the Go map to locate entries matching all the words. And that's all that has to be done, the result looks like [1] in the text, and the reference is added at the end of the output as shown here.

REFERENCES

1. The organization of networks in Plan 9. D. Presotto, P. Winterbottom. USENIX Association. Proceedings of the Winter 1993 USENIX Conference. 1993.

We can now easily cite in manual pages, web pages, papers in PDF, ...

The UNIX tradition of using simple text files with line oriented format and using Go code to process them is a very powerful combination. I'm still amazed.

Limiting the number of concurrent processes

In some cases, a task can be easily split into multiple concurrent child tasks, but it is not a good idea to spawn each of of the child tasks as a concurrent process.

For example, some commands in Clive go through a tree to synchronise it, or to poll each node for changes, etc. At each node in the tree, it is clear that the children can be processed concurrently. This reduces the total latency for the entire process. However, doing it this way would place a high load on the server at the other end of the line, and also on the line itself, and it might not be a good idea.

In Go it is fairly trivial to place a limit on the maximum number of concurrent tasks kept but still exploit concurrency up to that limit. The technique shown here is not new and, in fact, has been in use for decades, even in C for Plan 9, and probably in many other languages and by many other authors. Nevertheless, I think this tiny bit of code might help those that never saw it before.

The idea is to send functions through a channel and keep a fixed set of workers at the other end of the channel. All of them receive a function, call it, and loop again. The number of workers places the limit on the number of concurrent tasks running at the same time.

This creates the pool to run tasks:

func NewPool(n int) *Pool {
p := &Pool{calls: make(chan call)}
for i := 0; i < n; i++ {
go p.worker(i)
}
return p
}

A worker is simply

func (p* Pool) worker(id int) {

for c := range p.calls {

c.fn()

c.donec <- true

}

}

And we can run functions using this method:

func (p *Pool) Go(donec chan bool, fn func()) chan bool {

if donec == nil {

donec = make(chan bool, 1)

}

p.calls <- call{fn: fn, donec: donec}

return donec

}

Now the thing is fairly easy to use. Go closures make the interface of the previous method powerful enough to do anything and accept/return any input/output values.

For example:

// Create a pool of 5 workers

p := NewPool(5) 

// Run 20 functions in the pool

dc := make(chan bool, 20)

for i := 0; i < 20; i++ {

p.Go(dc, func() {

dprintf("running this...\n")

})

}

// Wait until all of them are done

for i := 0; i < 20; i++ {

<-dc

}

The testing code in this package plots a nice diagram of the execution, as a side effect of testing.

This is one such plot, time on the X axis and each task on the Y axis. 

trace: abcdeAfBCDEghijFkGHIJlmnoKpLMNOqrstPQRST

+----+                                  

 +-----+                                

  +-----+                               

   +-----+                              

    +-----+                             

      +--------+                        

           +-----+                      

            +-----+                     

             +-----+                    

              +-----+                   

                +--------+              

                     +-----+            

                      +-----+           

                       +-----+          

                        +-----+         

                          +--------+    

                               +----+   

                                +----+  

                                 +----+ 

                                  +----+

The tests run fake jobs like this one:

func fakefn(r rune, t time.Duration) {

n := atomic.AddInt32(&ncalls, 1)

calls[n-1] = r

dprintf("fake call %c\n", r)

time.Sleep(t)

r = unicode.ToUpper(r)

n = atomic.AddInt32(&ncalls, 1)

calls[n-1] = r

}

They record a trace of the sequence of calls in a string like

abcdeAfBCDEghijFkGHIJlmnoKpLMNOqrstPQRST

where lower-case indicate the start of a task and upper-case its end.

Thus, checking that the scheduling is as expected for the test is easy as is producing a plot for fun.

func plot(trz string, ncalls int) {

c := 'a'

dprintf("trace: %s\n", trz)

ch := map[bool]string{true: "-", false: " "}

for i := 0; i < ncalls; i++ {

st := c+rune(i)

end := unicode.ToUpper(st)

sted := false

for _, ev := range trz {

switch ev {

case st, end:

dprintf("+")

sted = !sted

default:

dprintf(ch[sted])

}

}

dprintf("\n")

}

}

File System Stacks in Clive

In clive, The different file systems and file system tools can be combined and used together to provide features like file access, backups, history dumps, and performance measurement. For example, this is a non-trivial stack out of the multiple ones we use:

This stack traces calls made to a caching file system that relies on a local file system to cache a remote file system (wait, there is more), the remote file system serves a caching file file system that relies on a local memory-only file system to cache a local file system (at the remote machine). Confusing, isn't it? But that is the nice thing, that you can build complex stacks out of simple file system trees.

This is an example of code:

             zx1, err := lfs.New("a tree", "/foo", lfs.RW)
             zx2, err := mfs.New("a cache")
             tr1 := trfs.New(zx1)
             tr2 := trfs.New(zx2)

             cfs, err := cfs.New("a cached tree", tr1, tr2, cfs.RW)
             err := <- cfs.Mkdir("/a/b", zx.Dir{"mode:": "0775"})

This code builds a different stack: Creates a ZX tree for an underlying UNIX tree at /foo, and then a ram-only file tree, a tree that traces calls to the first, one that traces calls to the second, and a caching file tree that relies on the two tracing trees. Now we can trace calls to a cached file tree and see the calls to each one of the trees (the cache and the cached ones).



There is yet another draft of how stacks work in Clive in the clive file system stacks TR found in the Lsub web site.