Showing posts with label tools. Show all posts
Showing posts with label tools. Show all posts

Saturday, June 13, 2015

Using build constraints to skip entire hierarchies in go

This is a nice change to let go apply build constraints to prune a hierarchy of packages. After the change, if there is a file skip.go in a package, then the build constraint for the file refers to the entire package and not just to the file. Also, if the package is not selected by the constraint, any sub-directory is also left out.

This is an example file, src/net/internal/socktest/skip.go, which I use to avoid compiling this package for Clive:

// +build !clive 

package socktest


Once this file is in place, running "go install std", or whatever happens to mention that package, will print a line

skip: socktest

and skip that package (and any directories within it).

In src/go/build/build.go, you add a new flag NotToBuild to the Package:

--- a/src/go/build/build.go
+++ b/src/go/build/build.go
@@ -360,6 +360,8 @@ type Package struct {
  AllTags       []string // tags that can influence file selection in this directory
  ConflictDir   string   // this directory shadows Dir in $GOPATH
 
+ NotToBuild bool // the skip.go file indicates not to build this in this context
+
  // Source files
  GoFiles        []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles)
  CgoFiles       []string // .go source files that import "C"

and then set NotToBuild in the package if skip.go is there and does not match the context.
In that case, all files are added to the ignored list and also we return an error in case the caller
might be tempted to do anything with the package that we want to discard.

We must look for the skip file before processing any other file because we might have a skip in place because other files do not even compile, for example.

--- a/src/go/build/build.go
+++ b/src/go/build/build.go
@@ -610,6 +612,20 @@ Found:
  return p, err
  }
 
+ for i, d := range dirs {
+ name := d.Name()
+ if name != "skip.go" {
+ continue
+ }
+ dirs[i] = dirs[len(dirs)-1]
+ dirs = dirs[:len(dirs)-1]
+ match, _, _, _ := ctxt.matchFile(p.Dir, name, true, make(map[string]bool))
+ if !match {
+ fmt.Fprintf(os.Stderr, "skip: %s\n", p.Dir)
+ p.NotToBuild = true
+ }
+ break
+ }
  var Sfiles []string // files with ".S" (capital S)
  var firstFile, firstCommentFile string
  imported := make(map[string][]token.Position)
@@ -670,6 +686,10 @@ Found:
  continue
  }
 
+ if p.NotToBuild {
+ p.IgnoredGoFiles = append(p.IgnoredGoFiles, name)
+ continue
+ }
  pf, err := parser.ParseFile(fset, filename, data, parser.ImportsOnly|parser.ParseComments)
  if err != nil {
  return p, err
@@ -777,7 +797,9 @@ Found:
  if len(p.GoFiles)+len(p.CgoFiles)+len(p.TestGoFiles)+len(p.XTestGoFiles) == 0 {
  return p, &NoGoError{p.Dir}
  }
-
+ if p.NotToBuild {
+ return p, &NoGoError{p.Dir}
+ }
  for tag := range allTags {
  p.AllTags = append(p.AllTags, tag)
  }

Then, the flag is copied into the Package structure used by the go command.

--- a/src/cmd/go/pkg.go
+++ b/src/cmd/go/pkg.go
@@ -95,6 +95,8 @@ type Package struct {
  coverMode    string               // preprocess Go source files with the coverage tool in this mode
  coverVars    map[string]*CoverVar // variables created by coverage analysis
  omitDWARF    bool                 // tell linker not to write DWARF information
+
+ NotToBuild bool `json:",omitempty"` // package is a skip
 }
 
 // CoverVar holds the name of the generated coverage variables targeting the named file.
@@ -137,6 +139,7 @@ func (p *Package) copyBuild(pp *build.Package) {
  p.TestImports = pp.TestImports
  p.XTestGoFiles = pp.XTestGoFiles
  p.XTestImports = pp.XTestImports
+ p.NotToBuild = pp.NotToBuild 
 }

And finally we make that command do the discard. The list of packages matched is adjusted
to remove those NotToBuild and children of NotToBuild.

--- a/src/cmd/go/main.go
+++ b/src/cmd/go/main.go
@@ -516,7 +516,7 @@ func matchPackages(pattern string) []string {
  have["runtime/cgo"] = true // ignore during walk
  }
  var pkgs []string
-
+ var skip []string
  for _, src := range buildContext.SrcDirs() {
  if (pattern == "std" || pattern == "cmd") && src != gorootSrc {
  continue
@@ -554,12 +554,23 @@ func matchPackages(pattern string) []string {
  if !match(name) {
  return nil
  }
- _, err = buildContext.ImportDir(path, 0)
+ var p *build.Package
+ p, err = buildContext.ImportDir(path, 0)
  if err != nil {
+ if p != nil && p.NotToBuild {
+ skip = append(skip, path+"/")
+ return nil
+ }
  if _, noGo := err.(*build.NoGoError); noGo {
  return nil
  }
  }
+ for _, s := range skip {
+ if strings.HasPrefix(path, s) {
+ // fmt.Fprintf(os.Stderr, "skip child %s\n", path)
+ return nil
+ }
+ }
  pkgs = append(pkgs, name)
  return nil
  })
@@ -597,6 +608,7 @@ func matchPackagesInFS(pattern string) []string {
  match := matchPattern(pattern)
 
  var pkgs []string
+ var skip []string
  filepath.Walk(dir, func(path string, fi os.FileInfo, err error) error {
  if err != nil || !fi.IsDir() {
  return nil
@@ -624,12 +636,23 @@ func matchPackagesInFS(pattern string) []string {
  if !match(name) {
  return nil
  }
- if _, err = build.ImportDir(path, 0); err != nil {
+ var p *build.Package
+ if p, err = build.ImportDir(path, 0); err != nil {
+ if p != nil && p.NotToBuild {
+ skip = append(skip, path+"/")
+ return nil
+ }
  if _, noGo := err.(*build.NoGoError); !noGo {
  log.Print(err)
  }
  return nil
  }
+ for _, s := range skip {
+ if strings.HasPrefix(path, s) {
+ // fmt.Fprintf(os.Stderr, "skip child %s\n", path)
+ return nil
+ }
+ }
  pkgs = append(pkgs, name)
  return nil
  })

Tuesday, March 24, 2015

Go and unix traditions are a good match

Recently I had start writing a paper on Clive's ZX (more on this soon) and had to add citations in it. I'm writing the paper using wr(1), the Clive text formatter used to write manual pages, papers, and everything.

Because we had the entire set of bibliography entries from Plan 9 and Plan B in troff's refer format, I though wr could be teach to use them.

Being short in time, I added a new Go package to Clive that eats all the entries in the lsub refer bib directory, that have a format like

Article in conference proceedings
%A M. Bishop
%A L. Snyder
%T The Transfer of Information and Authority
in a Protection System
%J Proceedings of the 7th SOSP
%P 45-54
%D 1979


The wr/refs package exploits Go maps, strings, and slices to record a map of words to bib entries, which are also maps from keys (author, title, etc.) to strings. As easy as this:

// A reference maps from the key (eg. 'A') to values (eg. authors)
type Ref {
Keys map[rune][]string
}

// A bib maps from words found in references to references
type Bib {
refs map[string] map[*Ref]bool
all []*Ref
}

Note how easy is in Go to create sets of arbitrary types by using a map of that type to booleans.
Writing the code for these types feels very much like writing lisp.

Citing now in wr is as easy as writing [bib: plan 9 networks] in the source text.
Wr simply takes all  the words in the citation and looks them up in the Go map to locate entries matching all the words. And that's all that has to be done, the result looks like [1] in the text, and the reference is added at the end of the output as shown here.

REFERENCES

1. The organization of networks in Plan 9. D. Presotto, P. Winterbottom. USENIX Association. Proceedings of the Winter 1993 USENIX Conference. 1993.

We can now easily cite in manual pages, web pages, papers in PDF, ...

The UNIX tradition of using simple text files with line oriented format and using Go code to process them is a very powerful combination. I'm still amazed.


Tuesday, March 10, 2015

File System Stacks in Clive

In clive, The different file systems and file system tools can be combined and used together to provide features like file access, backups, history dumps, and performance measurement. For example, this is a non-trivial stack out of the multiple ones we use:


This stack traces calls made to a caching file system that relies on a local file system to cache a remote file system (wait, there is more), the remote file system serves a caching file file system that relies on a local memory-only file system to cache a local file system (at the remote machine). Confusing, isn't it? But that is the nice thing, that you can build complex stacks out of simple file system trees.

This is an example of code:


             zx1, err := lfs.New("a tree", "/foo", lfs.RW)
             zx2, err := mfs.New("a cache")
             tr1 := trfs.New(zx1)
             tr2 := trfs.New(zx2)
             cfs, err := cfs.New("a cached tree", tr1, tr2, cfs.RW)
             err := <- cfs.Mkdir("/a/b", zx.Dir{"mode:": "0775"})
This code builds a different stack: Creates a ZX tree for an underlying UNIX tree at /foo, and then a ram-only file tree, a tree that traces calls to the first, one that traces calls to the second, and a caching file tree that relies on the two tracing trees. Now we can trace calls to a cached file tree and see the calls to each one of the trees (the cache and the cached ones).
There is yet another draft of how stacks work in Clive in the clive file system stacks TR found in the Lsub web site.

Tuesday, December 9, 2014

expr + test = xp

The clive command xp(1) replaces the venerable UNIX commands expr and test. It is interesting to note why. In few words, both commands are doing the same, and thus they are a single command in Clive.

The purpose of a command line calculator is to evaluate expressions. Now, the purpose of the test command in UNIX is also to evaluate expressions. Only that test knows how to evaluate expressions on file attributes.

Considering that in clive directory entries are generic maps from attribute names to attribute values, it seems that a single command can do both. The result is that we can do calculations on, for example, file sizes and file modification times in very much the same way we can calculate on floating point numbers. Or we can perform bitwise operations on file permissions. As a result, the grammar known to the calculator is also available for free to the "test" command, because it is the same command.

For example, it is clear that we can do things like
> xp 2 + 2
4
> xp 1k
1024
> xp 1m
1048576

But, we can use the same command to print metadata for files. For example, to print out
the file type:
> xp type .
d
> xp type xp.go
-

Now we can write general purpose expressions as we see they fit; e.g. is the size larger than 1Kbyte?

> xp size xp.go '>' 1k
false


Or, how many full Mbytes are used by a binary?

> xp size '"'`which xp`'"' / 1m
5

Another example. Let's check the mode for a file
> xp mode xp.go
0644

Now we can write an expression to get the write permissions,
> xp mode xp.go '&' 0222
128

or to see if any of them is set
> xp mode xp.go '&' 0222 '!=' 0
true

which can be done easier, by the way:
> xp w xp.go
true

Saturday, January 25, 2014

Tiny go debug tools

A recent post from Rob Pike in his blog reminded me that I didn't post about some tiny tools I use to debug programs by enabling traces and conditional prints. I think these were borrowed or inspired by those written by him or by others using Go time ago.

These are packaged in a tiny git.lsub.org/go.git/dbg.go package.

First, this is a well known idiom to trace function calls, only that I modified it to report also the file and line number in a convenient way.

It is used as in this excerpt

func ReadMsgsFrom(r io.Reader, c chan<- []byte) (int64, error) {
defer dbg.Trace(dbg.Call("ReadMsgsFrom"))
...

And produces messages like these ones:

bgn ReadMsgsFrom nchan.go:116
end ReadMsgsFrom nchan.go:116

The functions are like follows:

// For use as in defer dbg.Trace(Call("funcname"))
func Trace(s string) {
fmt.Printf("end %s\n", s)
}

// For use as in defer dbg.Trace(Call("funcname"))
func Call(s string) string {
if _, file, lno, ok := runtime.Caller(1); ok {
rel, _ := filepath.Rel(cwd, file)
s = fmt.Sprintf("%s %s:%d", s, rel, lno)
}
fmt.Printf("bgn %s\n", s)
return s
}

The second tool is a couple of functions that enable prints only if a flag is set or if another function says so. They are used like in

var Printf = dbg.FuncPrintf(os.Stdout, testing.Verbose)

func TestChanSend(t *testing.T) {
...
Printf("receiver %v\n", msg)
}

The variable name should reflect that it is a conditional print, but in this case I took the code from a testing file, which prints only if verbose is set during the test. The idea is that you can declare as many print functions (variables) you want to print conditionally when certain flags are enabled.

This is the code from the debug package

type PrintFunc func(fmts string, arg ...interface{}) (int, error)


/*
Return a function that calls fmt.Printf only if fn returns true.
To be used like in
var Printf = verb.PrintfFunc(testing.Verbose)
...
Printf(...)
 */
func FuncPrintf(w io.Writer, fn func()bool) PrintFunc {
return func(fmts string, arg ...interface{}) (int, error) {
if fn() {
return fmt.Fprintf(w, fmts, arg...)
}
return 0, nil
}
}

The function returns a function that prints, only that it prints only if the function given as an argument says so. Thus, when you declare your print function variable you can set the condition to trigger the print and the writer the print should use. The rest of the code is relieved from the burden of testing the condition or typing more to use a particular output stream.

There is a similar function (returning a conditional printing function) that takes a pointer to a boolean flag instead of a function, for those cases when checking a flag suffice.


Thursday, January 26, 2012

Wasting the time.

NIX knows how to record the worst/total/number of waiting times for each lock used in the kernel. Including queueing locks and other types of lock. This lets us retrieve from /dev a list of the ones where programs waste their time (be that for useful work or not). These are the winners as of today, as reported by the kernel


what pc #times maxtime tottime where meantime
lock 0xfffffffff017d490 50 12710074 215909070 /sys/src/nix/port/page.c:560 4318181
lock 0xfffffffff017d486 4 6326408 16216101 /sys/src/nix/port/page.c:560 4054025
lock 0xfffffffff017ce18 112 6763350 293233176 /sys/src/nix/port/page.c:309 2618153
lock 0xfffffffff017cbc3 78 12051128 126828266 /sys/src/nix/port/page.c:222 1626003
lock 0xfffffffff01620c0 162 1366053 14176025 /sys/src/nix/port/devmnt.c:1052 87506
qlock 0xfffffffff01773cc 581 1009266 40825474 /sys/src/nix/port/qio.c:1187 70267
qlock 0xfffffffff012cf2e 155 1238110 10673584 /sys/src/nix/k10/ether82563.c:893 68861
qlock 0xfffffffff01402af 376 574149 24205418 /sys/src/nix/ip/tcp.c:547 64376
qlock 0xfffffffff0143bee 350 492911 14229109 /sys/src/nix/ip/tcp.c:2118 40654
lock 0xfffffffff0119308 593 1121568 18533782 /sys/src/nix/k10/qmalloc.c:275 31254
lock 0xfffffffff0119a17 11183 1365890 294252474 /sys/src/nix/k10/qmalloc.c:500 26312
lock 0xfffffffff016226b 80 459279 1898897 /sys/src/nix/port/devmnt.c:1101 23736

The list is not interesting unless you are optimizing the kernel, but it shows how powerful this simple instrument can be. It's easy to spot some parts which are responsible for making the programs wait, perhaps more than really needed.