expr + test = xp

The clive command xp(1) replaces the venerable UNIX commands expr and test. It is interesting to note why. In few words, both commands are doing the same, and thus they are a single command in Clive.

The purpose of a command line calculator is to evaluate expressions. Now, the purpose of the test command in UNIX is also to evaluate expressions. Only that test knows how to evaluate expressions on file attributes.

Considering that in clive directory entries are generic maps from attribute names to attribute values, it seems that a single command can do both. The result is that we can do calculations on, for example, file sizes and file modification times in very much the same way we can calculate on floating point numbers. Or we can perform bitwise operations on file permissions. As a result, the grammar known to the calculator is also available for free to the "test" command, because it is the same command.

For example, it is clear that we can do things like
> xp 2 + 2
4
> xp 1k
1024
> xp 1m
1048576

But, we can use the same command to print metadata for files. For example, to print out
the file type:
> xp type .
d
> xp type xp.go
-

Now we can write general purpose expressions as we see they fit; e.g. is the size larger than 1Kbyte?

> xp size xp.go '>' 1k

false

Or, how many full Mbytes are used by a binary?

> xp size '"'`which xp`'"' / 1m

5

Another example. Let's check the mode for a file

> xp mode xp.go

0644

Now we can write an expression to get the write permissions,

> xp mode xp.go '&' 0222

128

or to see if any of them is set

> xp mode xp.go '&' 0222 '!=' 0

true

which can be done easier, by the way:

> xp w xp.go

true

Compare go source files in two file trees in clive

I'm still amazed at how pieces fit together in Clive. This is like a recursive diff in unix, but in Clive, and compares just source files in go as found within two trees:

    ; comp ',~\.go$' '/tmp/x,~\.go$'
    #diff comp.go:208: and /tmp/x/comp.go:207:
    -   x.diff()
    #diff: comp.go:209: and /tmp/x/comp.go:209:
    +  x.diff()
    ...

There are several things working together here

• Commands accept names made of two elements: a path and a predicate (separated by a comma). In the example the predicates match the name against what would be "*.go" in UNIX.
• Each name is issued to a finder to find a stream of directory entries
• Directory entries are self-describing, and thus a stream of directory entries is enough to reach the different file servers involved in the name space used for the search.
• File and directory contents may be retrieved along with the find request; they are streamed through channels, like everybody else.
• Directories always report and find entries in the same order, i.e. they are always sorted

As a result, the compare command only has to use a library function to make each name become a stream and then compare the two streams.

Conventions can do more with less: Clive options continued.

In a previous post I discussed a new package in Clive, opt, which deals with command arguments.
The package turned to be very easy to use and has unified how clive commands are used beyond understanding a common option syntax.

When defining new options, the user gives a pointer to a Go value, a name, and a help string:

func (f *Flags) NewFlag(name, help string, vp interface{})

The help string follows the convention that

• for options with no arguments, it simply describes the option.
• for options with arguments, it starts with  "argument name:" followed by the description of the option.

This allows the usage method to exploit this convention to build a usage string more descriptive than the ones in other Go programs. For example, compare the old

usage: flds -options [file...]

with the new

usage: flds [-1D] [-F sep] [-o sep] {-r range} [file...]

The method writing the usage description scans the defined flags for those that have a single rune as a name, and no argument, and combines them like in "-1D" above. Then it goes over flags that have longer names, or have arguments, and adds the usage for each one in the same line. The name of the argument is taken from the start of the help string (if found, or a generic name for the type of argument [in Go] is used). When arguments may be used more than once, curly brackets are printed instead of square ones. The method knows this because the value pointer given to NewFlag is a pointer to a slice of values.

Thus, there is no need to keep the usage summary synchronised with the source, it is always in sync.

But this convention has done even more work for Clive. Manual pages in clive (for all sections but the go packages section) are written in wr(1). Recently we have

• added a feature to the opt package so that "-?" can never be a valid flag, which makes all programs inform of their usage when called with this flag.
• replaced all the "synopsis" subsections in all command man pages with a new "usage" one.

The new manual pages start with something like:

_ LNS(1): print lines

* USAGE

[sh

lns -? >[2=1] | lns -r 1,-2

]

...

Here, a shell escape runs the command to make it inform about its usage, and all the output lines but for the last one (which is an exit status in clive) are inserted into the manual.

After doing so, we could remove most option description lists from manual pages, and we no longer have to document the usage of a command anymore. Plus, we are sure that the usage matches the binary installed in the system. For example, the manual for lns looks like:

LNS(1): PRINT LINES

USAGE

    usage: lns [-Dn] {-r range} [file...]

    -D: debug

    -n: print line numbers

    -r range: print this range

    range is addr,addr or addr

    addr is linenb, or +linenb, or -linenb

    

DESCRIPTION

...

Once again, we could do more by actually doing less.

Simple things are more powerful. The clive cmd/opt package.

The golang flag package is very powerful. It makes it easy to process your command arguments.
We have been using it for a long time to process Clive command arguments.

But, to say it in a few words, it is not able to operate on different argument vectors in a reasonable way (i.e., using the same interface used for processing the actual command line), it cannot handle combined -abc style flags instead of the utterly verbose -a -b -c, it cannot handle flags that repeat, and it is far more complex than needed. The clive opt package does all this by actually doing less.

For example, the flag package operates on os.Args to retrieve the arguments.  Using another array requires using another interface. Instead, opt takes always a []string and works on it. This makes it work on any given set of arguments, which makes it unnecessary to provide any functionality to process "sub-arguments" or "different command lines". Instead, the interface is used against the desired set of arguments.

Looking at the interface of flag provides more examples of what this post is about. There are multiple StringVar(), String(), IntVar(), Int(), ... functions to define new flags. There are many such functions and they define flags globally. Instead, opt provides a single method

    func (f *Flags) NewFlag(name, help string, vp interface{})

to define new flags. It can be used as in

        debug := false
        opts.NewFlag("d", "debug enable", &debug)
        odir := "/tmp"
        opts.NewFlag("o", "output dir, defaults to /tmp", &odir)

which has several advantages:

1. The flag is defined on a set of flags, not globally. This permits using different sets of flags yet the code is the same that it would be to define global flags, as discussed above.
2. There is a single way of defining a new flag. This is a benefit, not a drawback.
3. There is a single method to call for any flag, not many to remember.
4. It does not re-do things the language is doing. For example, a default value is simply the value of the variable unless the options change it. Go already has assignment and initialisation, thus, it is not necessary to supply that service in the flag definition. 
5. It is not possible to introduce errors due to initialisation of option variables not matching the default value given to the flag definition call.

The implementation has also important differences. The flag package is an elaborate set of internal types defined to provide methods to handle each one of the flag types. Adding a new flag requires defining types and methods and must be done with care. A consequence is that Flag has 850 lines and is harder to understand. Opt has 356 lines. In both cases including all the documentation.

Instead of the approach used in flag to define and implement the arguments, opt relies on a type switch on the pointers given when defining each one of the flags. All the package does during parsing is to set the pointer to the value found in the argument, using the type of the pointer to determine how to parse the value by default. This is the an excerpt of the code:

switch vp := d.valp.(type) {

case *bool:

*vp = true

if len(argv[0]) == 0 {

argv = argv[1:]

} else { // put back the "-" for the next flag

argv[0] = "-" + argv[0]

}

case *string:

nargv, arg, err := optArg(argv)

if err != nil {

return nil, fmt.Errorf("option '%s': %s", d.name, err)

}

argv, *vp = nargv, arg

case *[]string:

nargv, arg, err := optArg(argv)

if err != nil {

return nil, fmt.Errorf("option '%s': %s", d.name, err)

}

argv, *vp = nargv, append(*vp, arg)

And that suffices. There is another type switch when a flag is defined to make sure that the pointer type can be handled later by the package during parsing. Because of this, the code walking the arguments trying to parse each option can be fully shared and is not highly coupled with the per-option parsing code.

What about more complex arguments? Easy: the program may define string arguments and then parse them as desired. Or, if the new argument type becomes very popular, it can be added to opt by

1. Adding an empty case to the type switch of NewFlag (to check the pointer type for validity)
2. Adding a new case to the type switch of Parse (the excerpt shown above) to actually do the parsing.

All this is not to say that opt is the greatest argument processing package. In fact, it is just born and it is likely that there are still bugs and things to improve. All this is to say that more can be done by doing less in the interface and in the implementation of the package considered.

Go is a very nice language. I'd like its interfaces to be as clean, tiny and powerful as possible. If we want complex interfaces and implementations, we know where to find Java and C++.

Clive for users: finders, trees, and name spaces

0. The first example shows that listing all sources files under /zx/sys/src can be done with this command 
1. 
   

                ; lz /zx/sys/src/,
                /zx/sys/src
                /zx/sys/src/clive
                /zx/sys/src/clo
   

   Here, describing a file (or a set of files) is done by using a combination of a name and a predicate. In this case, all files starting at /zx/sys/src and matching an empty predicate (the empty string) are listed. 
   
   Or we can list only directories using
                             ; lz /zx/sys/src/,type=d
   
   
   Or we can remove all regular files in a hierarchy by using
   

                ; rm /zx/sys/src/,type=-
   

   Commands operating on files find the involved directory entries and then operate on them. A directory entry is a set of name/value pairs, and may have any number of attributes with names and values chosen at will. Of course, there are some conventions about which attributes are expected to be there (like size, mode, etc). 
   
   Predicates used to find files are general expressions that may use directory attributes as values, which makes it easy for a command to issue an expression to find the entries of interest in a single or a few RPCs. Directory entries are self-describing entities (eg., they report also the address of the server and the name of the resource in the server). 
   
   This makes it easy for a program to issue requests for a directory entry it found. In short, file trees in clive are split into two important entities: Finders used to find directory entries, and  file trees that accept operations for directory entries
2. 
   
3. Each process groups one or more finders into a name space, built from a textual representation (it might inherit the name space from the parent). For example, we can use 
4. 
   
5. • ; NS=’/ /
   0. • ;;  /zx tcp!zxserver!zx
        
      • ;;  /dump tcp!zxserver!zx!dump
        ;; ’
        ; lz /zx/usr/nemo,type=d
        
        
7. to define a new name space and then issue commands that work in it. In this example, we defined as / the root of the host OS file tree, and then mounted at /zx our main tree and at /dump its dump file system. To say it in a different way, the name space is a finder that may groups other finders (among other things). The name space is more powerful, and can mount at a given name a set of directory entries (be they for files or not), but the example suffices for now. 
8. 
   
9. 
   

bytes.Buffer or builtin concatenation to build strings in Go?

During profiling of clive Go code I have found an interesting bit.
This is the same old discussion about deciding on using a string buffer or using raw string concatenation to build strings.

A Dir is actually a map[string]string, and, the network format is a string with form

    attr1=val1 attr2=val2 ....

The function Dir.String() builds the string from the map. The question is:

Do we use bytes.Buffer and Fprintf to append data to the string, asking at the end to Buffer.String the resulting string? Or do we use a native Go string and concatenate using "+="?

Using bytes.Buffer and Fprintf in Dir.String() yields:
BenchmarkDirString              500000      5163 ns/op

Using strings and += 
BenchmarkDirString              500000      5077 ns/op

Surprisingly (perhaps), it's both easier to write and faster to use the strings and forget
about using the bytes.Buffer. It's likely this will place more pressure in the GC, because it builds and discards intermediate strings, but, it's faster and easier.

Early clive distribution

We just placed in the Clive's web site a draft for a paper
describing the system, a link to the manual and indications to download our
(development) source tree.

This is a research system, still under construction. Be warned. Nevertheless, we
are already using it, and our main file tree and its dump are under the control of
Clive.

As an appetiser, this is the command to list some files
> l -l /zx/nautilus/src,name~*.go
--rw-r--r-- /zx/nautilus/src/bufs/bufs.go 683 15 May 14 12:43 CEST
--rw-r--r-- /zx/nautilus/src/cmd/auth/auth.go 1312 15 May 14 12:48 CEST
--rw-r--r-- /zx/nautilus/src/cmd/hist/hist.go 5875 19 May 14 14:17 CEST
--rw-r--r-- /zx/nautilus/src/cmd/ns/ns.go 4254 15 May 14 13:06 CEST
--rw-r--r-- /zx/nautilus/src/cmd/nsh/nsh.go 14719 21 May 14 15:24 CEST

and this is how the more interesting rm /zx/nautilus/src,name~*.go is implemented

// In our example, path is /zx/nautilus/src, and pred is name~*.go

dirc := rns.Find(path, pred, "/", 0)

errors := []chan error{}

for dir := range dirc {

// get a handle for the directory entry server

wt, err := zx.RWDirTree(dir)

if err != nil {

dbg.Warn("%s: tree: %s", dir["path"], err)

continue

}

errc := wt.Remove(dir["path"])

errors = append(errors, ec)

}

for _, errc := range errors {

if err := <-errc; err != nil {

dbg.Warn("%s: %s", dir["path"], err)

}

}

Enjoy. The clivezx group at googlegroups is a public discussion group where we will make further announces regarding clive, and host any public discussion about it. You are invited to join.

Using files in clive

This is just a few snaps from a clive shell, a more detailed description or TR will follow...

List the ns (two trees mounted at /zx/abin, and /zx/xbin, nothing else mounted)

> ns
/ 0
/zx/abin 1
"" path:"/zx/abin" mode:"020000000755" type:"d" size:"3" name:"/" mtime:"1396356469000000000" addr:"tcp!Atlantis.local!8002!abin" proto:"zx finder" gc:"y"
/zx/xbin 1
"" path:"/zx/xbin" mode:"020000000755" type:"d" size:"1" name:"/" mtime:"1396356453000000000" addr:"tcp!Atlantis.local!8002!xbin" proto:"zx finder" gc:"y"

List just files at /zx

> l -l /zx

d-rwxrwx--- /zx/abin 3 01 Apr 14 14:47 CEST

d-rwxrwx--- /zx/xbin 1 01 Apr 14 14:47 CEST

List all files under / with mode 750

> l -l /,mode=0750

- 0-rwxr-x--- /zx/xbin/bin/Go 202 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/Watch 2351948 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/a 212 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/cgo 2452606 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/drawterm 837032 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/drawterm.old 843880 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/dt 60 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/ebnflint 1149721 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/gacc 32 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/gnot 52 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/hgpatch 1328046 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/mango-doc 2957328 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/nix 60 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/quietgcc 1326 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/t+ 23 01 Apr 14 14:47 CEST

- 0-rwxr-x--- /zx/xbin/bin/t- 22 01 Apr 14 14:47 CEST

Then chmod all of them to mode 755

> l -a /,mode=0750 | chmod 0755

How did chmod find out the files?

Here's how

> zl -a /zx

tcp!Atlantis.local!8002!abin /

tcp!Atlantis.local!8002!xbin /

The interesting thing is that the names are separated from the files (i.e., a file server serves a finder interface to find out about directory entries, and then a file tree to serve operations on those entries).

Commands operate on finders to discover entries, and then use them to reach the servers and perform operations on them. Of course, connections to servers are cached so that we don't create a new one per directory entry, which would be silly.

All these requests are streamed through channels, as are the replies, so that you can do things like this:

dc := sh.files(true, args...)

n := 0

for d := range dc {

n++

// we could process +-rwx here using d["mode"] to

// build the new mode and then use just that one.

nd := zx.Dir{"mode": mode}

ec := rfs.Dir(d).Wstat(nd)

<-ec

err := cerror(ec)

if err != nil {

fmt.Printf("%s: %s\n", d["path"], err)

}

}

Here, the code is waiting for one reply at a time, but it could easily stream 50 at a time like another command does:

calls := make(chan chan error, 50)

for i := len(ents)-1; i >=0; i-- {

ec := rfs.Dir(ents[i]).Remove()

select {

case calls <- ec:

case one := <-calls:

<-one

err := cerror(one)

if err != nil {

fmt.Printf("%s\n", err)

}

}

}

Network servers in Clive

The TR draft kept at TR describes, including some code excerpts how network services are built leveraging the tools described in the previous posts (and the TRs they mention).

In short, it is very easy to build services in a CSP like world while, at the same time, those services may be supplied across the network. And that considers also a set of pipes, or fifos, as a network!. It is not just TCP.

This is a teaser:

func (s *Srv) put(m *Msg, c <-chan []byte, rc chan<- []byte) {
wt, ok := s.t.(zx.RWTree)
if !ok {
close(rc, "not a rw tree")
return
}
err := wt.Put(m.Rid, m.D, c)
close(rc, err)
}

And this is another:

cc, err := Dial("fifo!*!ftest", nil)

if err != nil {

t.Fatal(err)

}

for i := 0; i < 10; i++ {

cc.Out <- []byte(fmt.Sprintf("<%d>", i))

msg := string(<-cc.In)

printf("got %s back\n", msg)

}

close(cc.In)

close(cc.Out)

Files and name spaces meet channels in Clive

This is the third post on a series of posts describing what we did as part of the ongoing effort to build a new OS, which we just named as "Clive". The previous two posts described how channels were used and bridged to the outside pipes and connections, and how they where multiplexed to make it easy to write protocols in a CSP style.

This post describes what we did regarding file trees and name spaces. This won't be a surprise considering that we come from the Plan 9, Inferno, Plan B, NIX, and Octopus heritage.

But first things last. Yes, last. A name space is separated from the conventional file tree abstraction. That is, name spaces exist on their own and map prefixes to directory entries.  A name space is searched using this interface

type Finder interface {
Find(path, pred string, depth0 int) <-chan zx.Dir
}

The only operation lets the caller find a stream of matching directory entries for the request.

Directory entries are within the subtree rooted at the given path, and must match the supplied predicate.

A predicate here is a very powerful thing, inspired by the unix command named before our operation.

For example,  the predicate

~ name "*.[ch]" & depth < 3

can be used to find files with names matching C source file names but no deeper than three

levels counting from the path given.

It is also important to notice that the results are delivered through a channel. This means that the operation may be issued, reach a name space at the other end of the network, and the caller may just retrieve the stream of replies when desired (or pass the stream to another process).

Things like removing all object files within a subtree can now be done in a couple of calls. One to find the stream of entries, and another to convert that stream into a stream of remove requests.

And here is where the first thing (deferred until now) comes. The interface for a file tree to be used across the network relies on channels (as promises) to retrieve the result of each operation requested. Furthermore, those channels are buffered in many cases (eg., on all error indications) and might be even ignored if the caller is going to checkout later the status of the entire request in some other way.

Thus, we can actually stream the series of removes in this example. Note that the call to remove simply issues the call and returns a channel that can be used to receive the reply later on.

oc := ns.Find("/a/b", "~ name *.[ch] & depth < 3", 0)

for d := range oc {

fs.Remove(d["path"]) // and ignore the error channel

}

This is just an example. A realistic case would not ignore the error channel returned by

the call to remove. It would probably defer the check until later, or pass the channel to another process checking out that removes were made. 

Nevertheless, the example gives a glance of the power of file system interfaces used in Clive.

Reading and Writing is also made relying on input and output channels.

If you are interested in more details, you can read zx, which describes the file system and name space interfaces, and perhaps also nchan, which describes CSP like tools used in Clive and important to understand how these interfaces are used in practice.

Channels, pipes, connections, and multiplexors

This post is the second in a series to describe how we adapted the CSP style of programming used in Go to the use of external pipes, files, network connections, and related artefacts.
This was the first post. In what follows we refer as "pipe" to any external file-descriptor-like artefact used to reach the external world (for input or output).

Connections
The nchan package defines a connection as

 type Conn struct {
  Tag string // debug
  In  <-chan []byte
  Out chan<- []byte
 }

This joins two channels to make a full-duplex connection. A process talking to an external entity relies on this structure to bridge the system pipe used to a pair of channels. There are utilies that leverage the func- tions described in the previous section and build a channel interface to external pipes, for example:

 func NewConn(rw io.ReadWriteCloser, nbuf int, win, wout chan bool) Conn

The function creates processes to feed and drain the connection channels from and to the external pipe. Fur- thermore, if rw supports closing only for reading or writing, a close on the input or output channels would close the respective halves of the pipe. Because of the message protocol explained in the previous section, errors are also propagated across the external pipe and the process using the connection can very much ignore that the source/sink of data is external.
It is easy to build pipes where the Out channel sends elements through the In channel:

        func NewPipe(nbuf int) Conn

And, using this, we can create in-memory connections that do not leave the process space:

 func NewConnPipe(nbuf int) (Conn, Conn)

This has been very useful during testing, because this connection can be created with no buffering and it is easier to spot dead-locks that involve both ends of the connection. Once the program is ready, we can replace the connection based pipe with an actual system provided pipe.

Multiplexors
Upon the channel based connections shown in the previous sections, the nchan package provides multiplex- ors.

 type Mux struct {
  In chan Conn
  ...
 }
 func NewMux(c Conn, iscaller bool) *Mux
 func (m *Mux) Close(err error)
 func (m *Mux) Out() chan<- []byte
 func (m *Mux) Rpc() (outc chan<- []byte, repc <-chan []byte)

A program speaking a protocol usually creates a new Conn connection by dialing or accepting connections and then creates a Mux by calling NewMux to multiplex the connection among multiple requests.

The nice thing of the multiplexed connection is that requests may carry a series of messages (and not just one message per request) and may or not have replies. Replies may also be a full series of messages. Both ends of a multiplexed connetion (the process using the mux and its peer at the other end of the pipe) may issue requests. Thus, this is not a client-server interaction model, although it may be used as such.
To issue new outgoing requests through the multiplexor, the process calls Out (to issue requests with no expected reply):

 oc := mux.Out()
 oc <- []byte("no reply")
 oc <- []byte("expected")
 close(oc)

Or the process may call Rpc (to issue requests with an expected reply).

 rc, rr := mux.Rpc()
 rc <- []byte("another")
 rc <- []byte("request")
 close(rc)
 for m := range rr {
  Printf("got %v as part of the reply\en", m)
 }
 Printf("and the final error status is %v\en", cerror(rr))

In the first case, the multiplexor returns a Conn to the caller with just the Out channel. Of course, this can be done multiple times to issue several concurrent outgoing requests:

In the figure, the two connections of the left were built by two calls to mux.Out(), which returns a Conn with an Out chan to issue requests. The process using the Out channel may issue as many messages as desired and then close the channel.

If the request depicted below requires a reply, mux.Rpc() is be called finstead of mux.Out() and the resulting picture is as shown.

The important part is that messages (and replies) sent as part of a request (or reply) may be streamed with- out affecting other requests and replies, other than by the usage of the underlying connection. That is, an idle stream does not block other streams.
The interface for the receiving part of the multiplexor is a single In channel that conveys one Conn per incoming request. The request has only the In channel if no reply is expected, and has both the In and Out channels set if a reply is expected.

To receive requests from the other end of the pipe, the code might look like this:

for call := range mux.In {
// call is a Conn
for m := range call.In {
Printf("got %v as part of the request\en", m)
}
if call.Out != nil {
call.Out <- []byte("a reply")
call.Out <- []byte("was expected, but...")
close(call.Out, "Oops!, failed")
}
}

For example, if a process received two requests, one with no reply expected and another with a reply expected, the picture would be:

Here, the two connections on the left represent requests that were received through the In channel depicted on top of the multiplexor.

The important thing to note is that processes may now issue streams of requests, or replies, through channels and they are fed to external pipes (or from them) as required. The interfaces shown have greatly simplified programming for (networked) system serviced being written for the new system.

Channels and pipes: close and errors

This post describes changes made to Go channels and tools built upon those to provide system and network-wide services for a new OS under construction. Further posts will follow and describe other related set of tools, but the post is already long enough. Yes, belts are gone, we use our modified channels directly now.

A channel is an artifact that can be used to send (typed) data through it. The Go language operations on channels include sending, receiving, and selection among a set of send and receive operations. Go permits also to close a channel after the last item has been sent.
On most systems, processes and applications talk through pipes, network connections, FIFOS, and related artifacts. In short, they are just file descriptors once open, permit the application to write data (for sending) and/or read data (for receiving). Some of these are duplex, but they can be considered to be a pair of devices (one for each direction). In what follows we will refer to all these artifacts as pipes (e.g., a net- work connection may be considered as a pair of simplex pipes).
There is a mismatch between channels and pipes and this paper shows what we did to try to bridge the gap between both abstractions for a new system. The aim is to let applications leverage the CSP style of programming while, at the same time, let them work across the network.
We assume that the reader is familiar with channels in the Go language, and describes only our modi- fications and additions.

Close and errors

When using pipes, each end of the pipe may close and the pipe implementation takes care of propagating the error to the other end. That is not the case with standard Go channels. Furthermore, upon errors, it is desirable for one end of the pipe to learn about the error that did happen at the other end.
We have modified the standard Go implementation to:

1. 1  Accept an optional error argument to close.
   
2. 2  Make the send operation return false when used on a closed channel (instead of panicing; the receive operation already behaves nicely the case of a closed channel).
   
3. 3  Provide a primitive, cerror, that returns the error given when the channel was closed.
   
4. 4  Make a close of an already closed channel a no-operation (instead of a panic).
   

With this modified tool in hand, it is feasible to write the following code:

var inc, outc chan[]byte
...
for data := range inc {
ndata := modify(data)
if ok := outc <-ndata; !ok {
close(inc, cerror(outc))
break
}
}
close(outc, cerror(inc))

Here, a process consumes data from inc and produces new data through outc for another one. The image to have in mind is

where the code shown corresponds to the middle process. Perhaps the first process terminates normally (or abnormally), in which case it would close inc. In this case, our code closes outc as expected. But this time, the error given by the first process is known to the second process, and it can even forward such error to the third one.
A more interesting case is when the third process decides to cease consuming data from outc and calls close. Now, our middle process will notice that ok becomes false when it tries to send more data, and can break its loop cleanly, closing also the input channel to singal to the first process that there is no point in producing further data. In this second example, the last call to close is a no-operation because the output channel was already closed, and we don’t need to add unnecessary code to prevent the call.
The important point is that termination of the data stream is easy to handle for the program without resorting to exceptions (or panics), and we know which one is the error, so we can take whatever measures are convenient in that case.

Channels and pipes
There are three big differences between channels and pipes (we are using pipe to refer to any ‘‘file descrip- tor’’ used to convey data, as stated before). One is that pipes may have errors when sending or receiving, but channels do not. Another one is that pipes carry ony streams of bytes and not separate messages. Yet another is that channels convey a data type but pipes convey just bytes.
The first difference is mostly dealt with the changes made to channels as described in the previous section. That is, channels may have errors while sending and or receiving, considered the changes made. Therefore, the code using a channel must consider errors in very much the same way it would do if using a pipe.
To address the third difference we are going to consider channels of byte arrays by now.
The second difference can be dealt with by ensuring that applications using channels to speak through a pipe preserve message boundaries within the pipe. With this in mind, a new nchan package pro- vides new channel tools to bridge the gap between the channel and the pipe domains.
The following function writes each message received from c into w as it arrives. If w preserves mes- sage boundaries, that is enough. The second function is its counterpart.

func WriteBytesTo(w io.Writer, c <-chan []byte) (int64, error)
func ReadBytesFrom(r io.Reader, c chan<- []byte) (int64, error)

 However, is most cases, the transport does not preserve message boundaries. Thus, the next function writes all messages received from c into w, but precedes each such write with a header indicating the message length. The second function can rely on this to read one message at a time and forward it to the given chan- nel.

  func WriteMsgsTo(w io.Writer, c <-chan []byte) (int64, error)
func ReadMsgsFrom(r io.Reader, c chan<- []byte) (int64, error)

One interesting feature of WriteMsgsTo and ReadMsgsFrom is that when the channel is closed, its error sta- tus is checked out and forwarded through the pipe. The other end notices that the message is an error indi- cation and closes the channel with said error.
Thus, code like the excerpt shown for our middle process in the stream of processes would work correctly even if the input channel comes from a pipe and not from a another process within the same program.

The the post in the series will be about channel connections and channel multiplexors.

Tiny go debug tools

A recent post from Rob Pike in his blog reminded me that I didn't post about some tiny tools I use to debug programs by enabling traces and conditional prints. I think these were borrowed or inspired by those written by him or by others using Go time ago.

These are packaged in a tiny git.lsub.org/go.git/dbg.go package.

First, this is a well known idiom to trace function calls, only that I modified it to report also the file and line number in a convenient way.

It is used as in this excerpt

func ReadMsgsFrom(r io.Reader, c chan<- []byte) (int64, error) {
defer dbg.Trace(dbg.Call("ReadMsgsFrom"))

...

And produces messages like these ones:

bgn ReadMsgsFrom nchan.go:116

end ReadMsgsFrom nchan.go:116

The functions are like follows:

// For use as in defer dbg.Trace(Call("funcname"))

func Trace(s string) {

fmt.Printf("end %s\n", s)

}

// For use as in defer dbg.Trace(Call("funcname"))

func Call(s string) string {

if _, file, lno, ok := runtime.Caller(1); ok {

rel, _ := filepath.Rel(cwd, file)

s = fmt.Sprintf("%s %s:%d", s, rel, lno)

}

fmt.Printf("bgn %s\n", s)

return s

}

The second tool is a couple of functions that enable prints only if a flag is set or if another function says so. They are used like in

var Printf = dbg.FuncPrintf(os.Stdout, testing.Verbose)

func TestChanSend(t *testing.T) {

...

Printf("receiver %v\n", msg)

}

The variable name should reflect that it is a conditional print, but in this case I took the code from a testing file, which prints only if verbose is set during the test. The idea is that you can declare as many print functions (variables) you want to print conditionally when certain flags are enabled.

This is the code from the debug package

type PrintFunc func(fmts string, arg ...interface{}) (int, error)

/*

Return a function that calls fmt.Printf only if fn returns true.

To be used like in

var Printf = verb.PrintfFunc(testing.Verbose)

...

Printf(...)

 */

func FuncPrintf(w io.Writer, fn func()bool) PrintFunc {

return func(fmts string, arg ...interface{}) (int, error) {

if fn() {

return fmt.Fprintf(w, fmts, arg...)

}

return 0, nil

}

}

The function returns a function that prints, only that it prints only if the function given as an argument says so. Thus, when you declare your print function variable you can set the condition to trigger the print and the writer the print should use. The rest of the code is relieved from the burden of testing the condition or typing more to use a particular output stream.

There is a similar function (returning a conditional printing function) that takes a pointer to a boolean flag instead of a function, for those cases when checking a flag suffice.

New Go channels at lsub

After some experience with belts (see a previous post), we have changed the go language to provide the functionality we were missing from their channel abstraction. We refer to the modified environment as Lsub Go.

NB: I'm editing this post to reflect the new interface used after some discussion and reading the comments. 

The modified compiler and runtime can be retrieved with

git clone git://git.lsub.org/golang.git

This required changes to both the compiler and the run-time.
We also tried to keep the core language as undisturbed as feasible yet be able to track the changes made.

• The builtin close can be called now also on a chan<- data type (a channel to send things through). This won't cause a panic as it did anymore.
• A variant, close(chan, interface{})  has been added to both close a channel (for sending or receiving) and report the cause for the closing. close(c) is equivalent to cclose(c, nil)
• A new builtin cerror(chan) error has been added to report the cause for a closed channel (or  nil or is not closed or was closed without indicating the reason)
• The primitive send operation now returns a bool value indicating if the send could be made or not (the channel might be closed). Thus, ok := c <- 0 now indicates if we could send or the channel was closed and we couldn't send anymore. 

Not directly related to this change, but because we already changed the language and because we missed this construct a lot...

• A new doselect construct has been added, which is equivalent to a for loop with a single select construct as the body. It is a looping select.

The doselect construct is very convenient. Many times a process would just loop serving a set of channels, and

1. There is no actual reason for having to indent twice (the loop and the select)
2. It would be desirable to be able to break the service loop directly or to continue with the next request.

For example:

doselect {

case x, ok := <-c1:

if !ok {

...

break

}

...

case x, ok := <-c2:

...

if ... {

    continue

}

default:

...

}

Here, the break does break the entire loop and there is no need to add a label to the loop implied by the construct.

Also, continue would continue with the next request selected in the loop.

But this is just a convenience, and not really a change that would make a difference.

Now, the important change is to be able to

for x := range inc {

dosomethingto(x)

if ok := outc <- x; !ok {

close(inc, "not interested in your stream")

break

}

}

In this example, we loop receiving elements from an input channel,  inc, and after some processing, send the output to another process through outc. Now, if somehow that process is no longer interested in the stream we are piping to it, there is no point in

• forcing it to drain the entire stream, or
• forcing our interface to have another different channel to report the error, or
• ignoring the sender and leaving it forever blocked trying to send

Instead, the receiver of outc might just close(outc) to indicate to any sender that nobody will ever be interested in more elements sent through there. 

After that, our attempt to send would return false (recorded in ok in this example), and we can check that value to stop sending.

Furthermore, because we are sending data received from an input channel, inc,  we can also tell the sender feeding us data that we are done, and also use close to report why.

Thus, an entire pipeline of processes can cleanly shut down and stop when it is necessary to do so. The same could have been done recovering from panics, but this is not really a panic, it is not really that different from detecting EOF when reading and should be processed by normal error checking code and not by panic/recover tricks. Moreover, such tricks would be inconvenient if the function is doing more than just sending through the channel, because a recover does not continue with the execution of the panicking function.

To continue with the motivating example, this code could report to the sender the possible cause that did lead to the closing of outc, by doing something like:

close(inc, cerror(outc))

Unlike before, this time the actual cause is reported through the pipeline of processes involved and anyone might now issue a reasonable error diagnostic more useful than "the channel was closed".

The modified compiler and runtime can be retrieved with

git clone git://git.lsub.org/golang.git

The repository there contains a full go distribution as retrieved from the Go mercurial, modified a few times to make these changes. The compiler packages from the standard library (used by go fmt among other things) are also updated to reflect the new changes.

To reduce the changes required, we did not modify the select cases to let you send with the

ok := outc <- x construct. Thus, within select you can only try to send and it would just fail to send on a closed channel, but would not panic nor would it break your construct. You should probably check in the code that the send could be done if you have to do so. We might modify further the Lsub version of go to let select handle this new case, but haven't done it yet.