Conventions can do more with less: Clive options continued.

In a previous post I discussed a new package in Clive, opt, which deals with command arguments.
The package turned to be very easy to use and has unified how clive commands are used beyond understanding a common option syntax.

When defining new options, the user gives a pointer to a Go value, a name, and a help string:

func (f *Flags) NewFlag(name, help string, vp interface{})

The help string follows the convention that

• for options with no arguments, it simply describes the option.
• for options with arguments, it starts with  "argument name:" followed by the description of the option.

This allows the usage method to exploit this convention to build a usage string more descriptive than the ones in other Go programs. For example, compare the old

usage: flds -options [file...]

with the new

usage: flds [-1D] [-F sep] [-o sep] {-r range} [file...]

The method writing the usage description scans the defined flags for those that have a single rune as a name, and no argument, and combines them like in "-1D" above. Then it goes over flags that have longer names, or have arguments, and adds the usage for each one in the same line. The name of the argument is taken from the start of the help string (if found, or a generic name for the type of argument [in Go] is used). When arguments may be used more than once, curly brackets are printed instead of square ones. The method knows this because the value pointer given to NewFlag is a pointer to a slice of values.

Thus, there is no need to keep the usage summary synchronised with the source, it is always in sync.

But this convention has done even more work for Clive. Manual pages in clive (for all sections but the go packages section) are written in wr(1). Recently we have

• added a feature to the opt package so that "-?" can never be a valid flag, which makes all programs inform of their usage when called with this flag.
• replaced all the "synopsis" subsections in all command man pages with a new "usage" one.

The new manual pages start with something like:

_ LNS(1): print lines

* USAGE

[sh

lns -? >[2=1] | lns -r 1,-2

]

...

Here, a shell escape runs the command to make it inform about its usage, and all the output lines but for the last one (which is an exit status in clive) are inserted into the manual.

After doing so, we could remove most option description lists from manual pages, and we no longer have to document the usage of a command anymore. Plus, we are sure that the usage matches the binary installed in the system. For example, the manual for lns looks like:

LNS(1): PRINT LINES

USAGE

    usage: lns [-Dn] {-r range} [file...]

    -D: debug

    -n: print line numbers

    -r range: print this range

    range is addr,addr or addr

    addr is linenb, or +linenb, or -linenb

    

DESCRIPTION

...

Once again, we could do more by actually doing less.

Simple things are more powerful. The clive cmd/opt package.

The golang flag package is very powerful. It makes it easy to process your command arguments.
We have been using it for a long time to process Clive command arguments.

But, to say it in a few words, it is not able to operate on different argument vectors in a reasonable way (i.e., using the same interface used for processing the actual command line), it cannot handle combined -abc style flags instead of the utterly verbose -a -b -c, it cannot handle flags that repeat, and it is far more complex than needed. The clive opt package does all this by actually doing less.

For example, the flag package operates on os.Args to retrieve the arguments.  Using another array requires using another interface. Instead, opt takes always a []string and works on it. This makes it work on any given set of arguments, which makes it unnecessary to provide any functionality to process "sub-arguments" or "different command lines". Instead, the interface is used against the desired set of arguments.

Looking at the interface of flag provides more examples of what this post is about. There are multiple StringVar(), String(), IntVar(), Int(), ... functions to define new flags. There are many such functions and they define flags globally. Instead, opt provides a single method

    func (f *Flags) NewFlag(name, help string, vp interface{})

to define new flags. It can be used as in

        debug := false
        opts.NewFlag("d", "debug enable", &debug)
        odir := "/tmp"
        opts.NewFlag("o", "output dir, defaults to /tmp", &odir)

which has several advantages:

1. The flag is defined on a set of flags, not globally. This permits using different sets of flags yet the code is the same that it would be to define global flags, as discussed above.
2. There is a single way of defining a new flag. This is a benefit, not a drawback.
3. There is a single method to call for any flag, not many to remember.
4. It does not re-do things the language is doing. For example, a default value is simply the value of the variable unless the options change it. Go already has assignment and initialisation, thus, it is not necessary to supply that service in the flag definition. 
5. It is not possible to introduce errors due to initialisation of option variables not matching the default value given to the flag definition call.

The implementation has also important differences. The flag package is an elaborate set of internal types defined to provide methods to handle each one of the flag types. Adding a new flag requires defining types and methods and must be done with care. A consequence is that Flag has 850 lines and is harder to understand. Opt has 356 lines. In both cases including all the documentation.

Instead of the approach used in flag to define and implement the arguments, opt relies on a type switch on the pointers given when defining each one of the flags. All the package does during parsing is to set the pointer to the value found in the argument, using the type of the pointer to determine how to parse the value by default. This is the an excerpt of the code:

switch vp := d.valp.(type) {

case *bool:

*vp = true

if len(argv[0]) == 0 {

argv = argv[1:]

} else { // put back the "-" for the next flag

argv[0] = "-" + argv[0]

}

case *string:

nargv, arg, err := optArg(argv)

if err != nil {

return nil, fmt.Errorf("option '%s': %s", d.name, err)

}

argv, *vp = nargv, arg

case *[]string:

nargv, arg, err := optArg(argv)

if err != nil {

return nil, fmt.Errorf("option '%s': %s", d.name, err)

}

argv, *vp = nargv, append(*vp, arg)

And that suffices. There is another type switch when a flag is defined to make sure that the pointer type can be handled later by the package during parsing. Because of this, the code walking the arguments trying to parse each option can be fully shared and is not highly coupled with the per-option parsing code.

What about more complex arguments? Easy: the program may define string arguments and then parse them as desired. Or, if the new argument type becomes very popular, it can be added to opt by

1. Adding an empty case to the type switch of NewFlag (to check the pointer type for validity)
2. Adding a new case to the type switch of Parse (the excerpt shown above) to actually do the parsing.

All this is not to say that opt is the greatest argument processing package. In fact, it is just born and it is likely that there are still bugs and things to improve. All this is to say that more can be done by doing less in the interface and in the implementation of the package considered.

Go is a very nice language. I'd like its interfaces to be as clean, tiny and powerful as possible. If we want complex interfaces and implementations, we know where to find Java and C++.

Clive for users: finders, trees, and name spaces

0. The first example shows that listing all sources files under /zx/sys/src can be done with this command 
1. 
   

                ; lz /zx/sys/src/,
                /zx/sys/src
                /zx/sys/src/clive
                /zx/sys/src/clo
   

   Here, describing a file (or a set of files) is done by using a combination of a name and a predicate. In this case, all files starting at /zx/sys/src and matching an empty predicate (the empty string) are listed. 
   
   Or we can list only directories using
                             ; lz /zx/sys/src/,type=d
   
   
   Or we can remove all regular files in a hierarchy by using
   

                ; rm /zx/sys/src/,type=-
   

   Commands operating on files find the involved directory entries and then operate on them. A directory entry is a set of name/value pairs, and may have any number of attributes with names and values chosen at will. Of course, there are some conventions about which attributes are expected to be there (like size, mode, etc). 
   
   Predicates used to find files are general expressions that may use directory attributes as values, which makes it easy for a command to issue an expression to find the entries of interest in a single or a few RPCs. Directory entries are self-describing entities (eg., they report also the address of the server and the name of the resource in the server). 
   
   This makes it easy for a program to issue requests for a directory entry it found. In short, file trees in clive are split into two important entities: Finders used to find directory entries, and  file trees that accept operations for directory entries
2. 
   
3. Each process groups one or more finders into a name space, built from a textual representation (it might inherit the name space from the parent). For example, we can use 
4. 
   
5. • ; NS=’/ /
   0. • ;;  /zx tcp!zxserver!zx
        
      • ;;  /dump tcp!zxserver!zx!dump
        ;; ’
        ; lz /zx/usr/nemo,type=d
        
        
7. to define a new name space and then issue commands that work in it. In this example, we defined as / the root of the host OS file tree, and then mounted at /zx our main tree and at /dump its dump file system. To say it in a different way, the name space is a finder that may groups other finders (among other things). The name space is more powerful, and can mount at a given name a set of directory entries (be they for files or not), but the example suffices for now. 
8. 
   
9. 
   

bytes.Buffer or builtin concatenation to build strings in Go?

During profiling of clive Go code I have found an interesting bit.
This is the same old discussion about deciding on using a string buffer or using raw string concatenation to build strings.

A Dir is actually a map[string]string, and, the network format is a string with form

    attr1=val1 attr2=val2 ....

The function Dir.String() builds the string from the map. The question is:

Do we use bytes.Buffer and Fprintf to append data to the string, asking at the end to Buffer.String the resulting string? Or do we use a native Go string and concatenate using "+="?

Using bytes.Buffer and Fprintf in Dir.String() yields:
BenchmarkDirString              500000      5163 ns/op

Using strings and += 
BenchmarkDirString              500000      5077 ns/op

Surprisingly (perhaps), it's both easier to write and faster to use the strings and forget
about using the bytes.Buffer. It's likely this will place more pressure in the GC, because it builds and discards intermediate strings, but, it's faster and easier.

Early clive distribution

We just placed in the Clive's web site a draft for a paper
describing the system, a link to the manual and indications to download our
(development) source tree.

This is a research system, still under construction. Be warned. Nevertheless, we
are already using it, and our main file tree and its dump are under the control of
Clive.

As an appetiser, this is the command to list some files
> l -l /zx/nautilus/src,name~*.go
--rw-r--r-- /zx/nautilus/src/bufs/bufs.go 683 15 May 14 12:43 CEST
--rw-r--r-- /zx/nautilus/src/cmd/auth/auth.go 1312 15 May 14 12:48 CEST
--rw-r--r-- /zx/nautilus/src/cmd/hist/hist.go 5875 19 May 14 14:17 CEST
--rw-r--r-- /zx/nautilus/src/cmd/ns/ns.go 4254 15 May 14 13:06 CEST
--rw-r--r-- /zx/nautilus/src/cmd/nsh/nsh.go 14719 21 May 14 15:24 CEST

and this is how the more interesting rm /zx/nautilus/src,name~*.go is implemented

// In our example, path is /zx/nautilus/src, and pred is name~*.go

dirc := rns.Find(path, pred, "/", 0)

errors := []chan error{}

for dir := range dirc {

// get a handle for the directory entry server

wt, err := zx.RWDirTree(dir)

if err != nil {

dbg.Warn("%s: tree: %s", dir["path"], err)

continue

}

errc := wt.Remove(dir["path"])

errors = append(errors, ec)

}

for _, errc := range errors {

if err := <-errc; err != nil {

dbg.Warn("%s: %s", dir["path"], err)

}

}

Enjoy. The clivezx group at googlegroups is a public discussion group where we will make further announces regarding clive, and host any public discussion about it. You are invited to join.

Using files in clive

This is just a few snaps from a clive shell, a more detailed description or TR will follow...

List the ns (two trees mounted at /zx/abin, and /zx/xbin, nothing else mounted)

> ns
/ 0
/zx/abin 1
"" path:"/zx/abin" mode:"020000000755" type:"d" size:"3" name:"/" mtime:"1396356469000000000" addr:"tcp!Atlantis.local!8002!abin" proto:"zx finder" gc:"y"
/zx/xbin 1
"" path:"/zx/xbin" mode:"020000000755" type:"d" size:"1" name:"/" mtime:"1396356453000000000" addr:"tcp!Atlantis.local!8002!xbin" proto:"zx finder" gc:"y"

List just files at /zx

> l -l /zx

d-rwxrwx--- /zx/abin 3 01 Apr 14 14:47 CEST

d-rwxrwx--- /zx/xbin 1 01 Apr 14 14:47 CEST

List all files under / with mode 750

> l -l /,mode=0750

- 0-rwxr-x--- /zx/xbin/bin/Go 202 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/Watch 2351948 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/a 212 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/cgo 2452606 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/drawterm 837032 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/drawterm.old 843880 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/dt 60 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/ebnflint 1149721 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/gacc 32 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/gnot 52 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/hgpatch 1328046 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/mango-doc 2957328 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/nix 60 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/quietgcc 1326 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/t+ 23 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/t- 22 01 Apr 14 14:47 CEST

Then chmod all of them to mode 755

> l -a /,mode=0750 | chmod 0755

How did chmod find out the files?

Here's how

> zl -a /zx

tcp!Atlantis.local!8002!abin /

tcp!Atlantis.local!8002!xbin /

The interesting thing is that the names are separated from the files (i.e., a file server serves a finder interface to find out about directory entries, and then a file tree to serve operations on those entries).

Commands operate on finders to discover entries, and then use them to reach the servers and perform operations on them. Of course, connections to servers are cached so that we don't create a new one per directory entry, which would be silly.

All these requests are streamed through channels, as are the replies, so that you can do things like this:

dc := sh.files(true, args...)

n := 0

for d := range dc {

n++

// we could process +-rwx here using d["mode"] to

// build the new mode and then use just that one.

nd := zx.Dir{"mode": mode}

ec := rfs.Dir(d).Wstat(nd)

<-ec

err := cerror(ec)

if err != nil {

fmt.Printf("%s: %s\n", d["path"], err)

}

}

Here, the code is waiting for one reply at a time, but it could easily stream 50 at a time like another command does:

calls := make(chan chan error, 50)

for i := len(ents)-1; i >=0; i-- {

ec := rfs.Dir(ents[i]).Remove()

select {

case calls <- ec:

case one := <-calls:

<-one

err := cerror(one)

if err != nil {

fmt.Printf("%s\n", err)

}

}

}

Network servers in Clive

The TR draft kept at TR describes, including some code excerpts how network services are built leveraging the tools described in the previous posts (and the TRs they mention).

In short, it is very easy to build services in a CSP like world while, at the same time, those services may be supplied across the network. And that considers also a set of pipes, or fifos, as a network!. It is not just TCP.

This is a teaser:

func (s *Srv) put(m *Msg, c <-chan []byte, rc chan<- []byte) {
wt, ok := s.t.(zx.RWTree)
if !ok {
close(rc, "not a rw tree")
return
}
err := wt.Put(m.Rid, m.D, c)
close(rc, err)
}

And this is another:

cc, err := Dial("fifo!*!ftest", nil)

if err != nil {

t.Fatal(err)

}

for i := 0; i < 10; i++ {

cc.Out <- []byte(fmt.Sprintf("<%d>", i))

msg := string(<-cc.In)

printf("got %s back\n", msg)

}

close(cc.In)

close(cc.Out)