tag:blogger.com,1999:blog-75460291553358788112024-02-06T19:43:43.400-08:00System SoftwareQuick notes regarding system software issues, references to related work, ideas for future work, and any interesting result along the way.
See <a href="http://lsub.org/">the lsub web site.</a>Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.comBlogger59125tag:blogger.com,1999:blog-7546029155335878811.post-81933140974600149492016-02-23T09:30:00.000-08:002016-02-23T09:30:09.870-08:00Clive channels and plumbingIn Clive, we can run<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><b>eco http://lsub.org >[out:ink]</b></span></blockquote>
to display a web page. Here, <span style="font-family: Courier New, Courier, monospace;"><b><a href="http://lsub.org/sys/man/1/eco.html">eco</a></b> </span>is Clive's <span style="font-family: Courier New, Courier, monospace;"><b>echo</b></span>, and the redirection at the end is the standard <i><a href="http://lsub.org/sys/man/1/ql.html">ql</a>(1)</i> syntax to make the standard output channel be the channel named ink. <i>Ql</i> is the Clive's shell.<br />
<br />
In the same way,<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><b>eco look:ix.go:4 >[out:ink]</b></span></blockquote>
opens <span style="font-family: Courier New, Courier, monospace;"><b>ix.go</b></span> in the <i>ix</i> shell and selects line 4; also<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><b>eco exec:cmd >[out:ink]</b></span></blockquote>
opens a new window in <i>ix</i> with the cmd output.<br />
<br />
I/O in Clive relies on named channels. Instead of 0, 1, and 2 file descriptors in UNIX, Clive has <span style="font-family: Courier New, Courier, monospace;"><b>in</b></span>, <span style="font-family: Courier New, Courier, monospace;"><b>out</b></span>, and <span style="font-family: Courier New, Courier, monospace;"><b>err</b></span> channels. A command may use <span style="font-family: Courier New, Courier, monospace;"><b>cmd.In("in")</b></span> to get the input channel <span style="font-family: Courier New, Courier, monospace;"><b>in</b></span>, or <span style="font-family: Courier New, Courier, monospace;"><b>cmd.Out("err")</b></span> to get the output channel <span style="font-family: Courier New, Courier, monospace;"><b>err. </b></span>The command gets a nil channel if there's no such channel.<br />
<br />
This is used by <a href="http://lsub.org/sys/man/1/ix.html">ix(1)</a> to provide an extra output channel named <span style="font-family: Courier New, Courier, monospace;"><b>ink</b></span>. This channel is used by commands to output graphical user interfaces through it. If a command creates a web control, writes it to a buffer, and sends the buffer through this channel, the user interface will show the control and get in touch with the command.<br />
<br />
The convention is that through the <span style="font-family: Courier New, Courier, monospace;"><b>ink</b></span> channel we get URLs, or HTML, or strings starting with "<span style="font-family: Courier New, Courier, monospace;"><b>look:</b></span>" or "<span style="font-family: Courier New, Courier, monospace;"><b>exec:</b></span>".<br />
<div>
<br /></div>
<div>
A <i>look</i> package used by <i>ix</i> inspects the look strings read from the <i>ink</i> channel and executes the commands configured by the user. Thus, the <i>ink</i> channel is enough to provide the plumbing facilities provided in Plan 9 by its plumber command.</div>
<div>
<br /></div>
<div>
As another example, with this configuration file, a click with the button-3 at ix(1) opens a window with the manual.</div>
<div>
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><b># $home/lib/look: rules for look(2), used by ix(1)<br />^([a-zA-Z.]+)\(([0-9]+)\)$<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>doc \2 \1|rf<br />^http://.*$<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>open \0<br />^https://.*$<br /><span class="Apple-tab-span" style="white-space: pre;"> </span>open \0</b></span></blockquote>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAi1mm3W6WPnSJPNLZKdsbhyphenhyphenoPbGvZSZdmyU2ByUpbwLBWsQNw6MH2-KWZYoEQb47SB0IIg46ESPJDlqx0TL3w1B656SdusYAZ0TNCpA5F_q10USQQeL8xyWo_AqGnaolhhdfNxAnA_B5x/s1600/plumb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhAi1mm3W6WPnSJPNLZKdsbhyphenhyphenoPbGvZSZdmyU2ByUpbwLBWsQNw6MH2-KWZYoEQb47SB0IIg46ESPJDlqx0TL3w1B656SdusYAZ0TNCpA5F_q10USQQeL8xyWo_AqGnaolhhdfNxAnA_B5x/s1600/plumb.png" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-70624882647357109012016-02-09T09:32:00.000-08:002016-02-09T09:32:24.602-08:00Clive's Ink screenshotWe'll write about Ink in the future. It's the Clive UIMS, written in Go (as all other Clive apps) and using a viewer in HTML5/javascript so the browser is the terminal device.<br />
<br />
IX is the Clive shell (a descendant of Acme, Omero, and O/live) and, using Ink,<br />
it can do nice things, eg., click 3 on ",~*.go" and it finds and loads all Go files under ".".<br />
The "command language" is actually a set of external commands, thanks to using named channels as the standard I/O mechanism. A new "ink" standard output channel is along "in", "out", and "err", and let's commands output their user interfaces.<br />
All Ink controls can be replicated in one or more web pages, which is also nice.<br />
<br />
This is a screenshot of our development version of IX with a few commands running.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3vRr2pFF8nF8aRfpGAG31o2wlXuoYAUzis3E0Y66pBTELLvQxVQf6lvI9ElXICCAKEVkpOrpZ5CyGbYX8b3FPfKqxavMDumaUnWu3T-XSCTyKosJ8-vJ0O3gwBjlrZNXxtzwM1AdWGzYI/s1600/ink.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3vRr2pFF8nF8aRfpGAG31o2wlXuoYAUzis3E0Y66pBTELLvQxVQf6lvI9ElXICCAKEVkpOrpZ5CyGbYX8b3FPfKqxavMDumaUnWu3T-XSCTyKosJ8-vJ0O3gwBjlrZNXxtzwM1AdWGzYI/s400/ink.tiff" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<span id="goog_420920852"></span><span id="goog_420920853"></span><br />Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-328608913366367352015-11-13T08:26:00.000-08:002015-11-13T08:26:02.247-08:00Preventing dynamic memory problems in CWhen writing C programs that use structures kept in dynamic memory, there are a few techniques that are useful to prevent problems. The ones I mention here are not new, and well written software (e.g, much of Plan 9 from Bell Labs) use them a lot. This post is mostly for new programmers or not-so-old programmers in the hope that it might help them.<br />
<br />
For example, consider a server that allocates/releases some structures to serve clients:<br />
<span style="font-family: Courier New, Courier, monospace;"><b>typedef struct MBuf MBuf;</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>struct MBuf {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>uint64_t pc;</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>MBuf *next;</b></span><b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>// next in free or used list</b><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>MBuf *anext;<span class="Apple-tab-span" style="white-space: pre;"> </span>// next in global allocation list</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>... the actual data goes here ...</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>};</b></span><br />
<div>
<br /></div>
<div>
It does not matter what this structure is used for. But the code will do many (de)allocations for these. Instead of using malloc/free directly, we keep all allocated structures in a single list linked through the <span style="font-family: Courier New, Courier, monospace;">anext</span> field, and all unused structures in a free list, linked through the <span style="font-family: Courier New, Courier, monospace;">next</span> field.</div>
<div>
<br /></div>
<div>
For allocation, if there are structures in the free list, we are done. Otherwise we allocate a new one and link it in the global list.</div>
<div>
<br /></div>
<div>
Now, in the allocation, we take the program counter of the caller and store it in the <span style="font-family: Courier New, Courier, monospace;">pc</span> field. In deallocation, we set the <span style="font-family: Courier New, Courier, monospace;">pc</span> to zero and set most of the structure (but for the pc and the link fields) to a silly value. That is, we poison the data when it is free.</div>
<div>
<br /></div>
<div>
Doing so, makes it quite easy to detect references to free structures. Also, it is easy to write a leak function that, at the end of the program or at any other point, scans the entire global allocation list and prints those program counters that are not zero, so we know the places in the program where we are allocating the structures without actually releasing them.</div>
<div>
<br /></div>
<div>
It's an easy thing to do, it makes the program faster because it avoids extra allocations, and it makes the program safer because it aids in testing and debugging. Hope you enjoy it.</div>
<div>
<br /></div>
<div>
<div>
The same thing can be done to structures pre-allocated in arrays, thus, the leaks detected are those that are actually leaks (because the leak tool provided by the OS or the language does not know if something that we allocated is actually a leak or not; it might be a correct pre-allocation that was never used).</div>
</div>
<div>
<br /></div>
<div>
For example, this might be the code:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>MBuf*</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>bufalloc(void)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>{</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>MBuf *buf;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>buf = freebuf;</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if (buf != NULL) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>freebuf = buf->next;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>} else {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>buf = xmalloc(sizeof(MBuf));</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>buf->anext = allbufs;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>allbufs = buf;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>CALLERPC(buf->pc); // set buf->pc with the caller pc</b></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return buf;</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>void</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>buffree(MBuf *buf)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>{</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>MBuf *n;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if (buf != NULL) {</b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>n = buf->anext;</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if (buf->pc == 0) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>xfatal("buffree: double free");</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>memset(buf, 3, sizeof(*buf));// poison</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>buf->anext = n;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>buf->pc = 0;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>buf->next = freebuf;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>freebuf = buf;</b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>void</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>bufleaks(void)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>{</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>MBuf *buf;</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for (buf = allbufs; buf != NULL; buf = buf->anext) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if (buf->pc != 0) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>xwarn("bufalloc: leak pc %llx", buf->pc);</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>xwarn("\tlldb %s", argv0);</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>xwarn("\tdi -m -s %llx", buf->pc);</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-83583637067782829872015-06-13T09:02:00.001-07:002015-06-13T09:02:25.695-07:00Using build constraints to skip entire hierarchies in goThis is a nice change to let <i>go</i> apply build constraints to prune a hierarchy of packages. After the change, if there is a file <span style="font-family: Courier New, Courier, monospace;"><b>skip.go</b></span> in a package, then the build constraint for the file refers to the entire package and not just to the file. Also, if the package is not selected by the constraint, any sub-directory is also left out.<br />
<br />
This is an example file, <span style="font-family: Courier New, Courier, monospace;"><b>src/net/internal/socktest/skip.go</b></span>, which I use to avoid compiling this package for Clive:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>// +build !clive </b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span>
<span style="font-family: Courier New, Courier, monospace;"><b>package socktest</b></span><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
Once this file is in place, running "go install std", or whatever happens to mention that package, will print a line</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>skip: socktest</b></span></div>
</div>
<div>
<br /></div>
<div>
and skip that package (and any directories within it).</div>
<div>
<br /></div>
<div>
In src/go/build/build.go, you add a new flag NotToBuild to the Package:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--- a/src/go/build/build.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+++ b/src/go/build/build.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -360,6 +360,8 @@ type Package struct {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>AllTags []string // tags that can influence file selection in this directory</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>ConflictDir string // this directory shadows Dir in $GOPATH</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>NotToBuild bool<span class="Apple-tab-span" style="white-space: pre;"> </span>// the skip.go file indicates not to build this in this context</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>// Source files</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>GoFiles []string // .go source files (excluding CgoFiles, TestGoFiles, XTestGoFiles)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>CgoFiles []string // .go source files that import "C"</b></span></div>
</div>
<div>
<br /></div>
<div>
and then set NotToBuild in the package if skip.go is there and does not match the context.</div>
<div>
In that case, all files are added to the ignored list and also we return an error in case the caller</div>
<div>
might be tempted to do anything with the package that we want to discard.</div>
<div>
<br /></div>
<div>
We must look for the skip file before processing any other file because we might have a skip in place because other files do not even compile, for example.</div>
<div>
<span style="font-size: x-small;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--- a/src/go/build/build.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+++ b/src/go/build/build.go</b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span style="font-size: x-small;">@@ -610,6 +612,20 @@ Found:</span></b></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return p, err</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>for i, d := range dirs {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>name := d.Name()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if name != "skip.go" {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>continue</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>dirs[i] = dirs[len(dirs)-1]</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>dirs = dirs[:len(dirs)-1]</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>match, _, _, _ := ctxt.matchFile(p.Dir, name, true, make(map[string]bool))</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if !match {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Fprintf(os.Stderr, "skip: %s\n", p.Dir)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>p.NotToBuild = true</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>break</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>var Sfiles []string // files with ".S" (capital S)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>var firstFile, firstCommentFile string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>imported := make(map[string][]token.Position)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -670,6 +686,10 @@ Found:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>continue</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if p.NotToBuild {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>p.IgnoredGoFiles = append(p.IgnoredGoFiles, name)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>continue</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>pf, err := parser.ParseFile(fset, filename, data, parser.ImportsOnly|parser.ParseComments)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return p, err</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -777,7 +797,9 @@ Found:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if len(p.GoFiles)+len(p.CgoFiles)+len(p.TestGoFiles)+len(p.XTestGoFiles) == 0 {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return p, &NoGoError{p.Dir}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>-</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if p.NotToBuild {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>return p, &NoGoError{p.Dir}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>for tag := range allTags {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>p.AllTags = append(p.AllTags, tag)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
</div>
<div>
<br /></div>
<div>
Then, the flag is copied into the Package structure used by the go command.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--- a/src/cmd/go/pkg.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+++ b/src/cmd/go/pkg.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -95,6 +95,8 @@ type Package struct {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>coverMode string // preprocess Go source files with the coverage tool in this mode</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>coverVars map[string]*CoverVar // variables created by coverage analysis</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>omitDWARF bool // tell linker not to write DWARF information</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>NotToBuild bool `json:",omitempty"` // package is a skip</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> // CoverVar holds the name of the generated coverage variables targeting the named file.</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -137,6 +139,7 @@ func (p *Package) copyBuild(pp *build.Package) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>p.TestImports = pp.TestImports</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>p.XTestGoFiles = pp.XTestGoFiles</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>p.XTestImports = pp.XTestImports</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>p.NotToBuild = pp.NotToBuild </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> }</b></span></div>
</div>
<div>
<br /></div>
<div>
And finally we make that command do the discard. The list of packages matched is adjusted</div>
<div>
to remove those NotToBuild and children of NotToBuild.</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--- a/src/cmd/go/main.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+++ b/src/cmd/go/main.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -516,7 +516,7 @@ func matchPackages(pattern string) []string {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>have["runtime/cgo"] = true // ignore during walk</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>var pkgs []string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>-</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>var skip []string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>for _, src := range buildContext.SrcDirs() {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if (pattern == "std" || pattern == "cmd") && src != gorootSrc {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>continue</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -554,12 +554,23 @@ func matchPackages(pattern string) []string {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if !match(name) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>-<span class="Apple-tab-span" style="white-space: pre;"> </span>_, err = buildContext.ImportDir(path, 0)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>var p *build.Package</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>p, err = buildContext.ImportDir(path, 0)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if p != nil && p.NotToBuild {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>skip = append(skip, path+"/")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if _, noGo := err.(*build.NoGoError); noGo {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>for _, s := range skip {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if strings.HasPrefix(path, s) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>// fmt.Fprintf(os.Stderr, "skip child %s\n", path)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>pkgs = append(pkgs, name)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>})</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -597,6 +608,7 @@ func matchPackagesInFS(pattern string) []string {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>match := matchPattern(pattern)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>var pkgs []string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>var skip []string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>filepath.Walk(dir, func(path string, fi os.FileInfo, err error) error {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil || !fi.IsDir() {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>@@ -624,12 +636,23 @@ func matchPackagesInFS(pattern string) []string {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if !match(name) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>-<span class="Apple-tab-span" style="white-space: pre;"> </span>if _, err = build.ImportDir(path, 0); err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>var p *build.Package</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if p, err = build.ImportDir(path, 0); err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if p != nil && p.NotToBuild {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>skip = append(skip, path+"/")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>if _, noGo := err.(*build.NoGoError); !noGo {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>log.Print(err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>for _, s := range skip {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>if strings.HasPrefix(path, s) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>// fmt.Fprintf(os.Stderr, "skip child %s\n", path)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>+<span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>pkgs = append(pkgs, name)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>return nil</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>})</b></span></div>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-72569970440000544802015-06-11T04:35:00.001-07:002015-06-11T04:35:54.010-07:00Lsub go changesFor the Clive OS being developed at Lsub, we have modified the Go
compiler in several important aspects. This post is a copy of a TR
documenting the changes we made.
<p>
<a name="sec2"></a><h2>INTRODUCTION</h2>
Clive is written using the Go programming language
[<a href="#bib1">1</a>]. Clive system services are organized
by connecting them through a pipe-like abstraction. Like it has been
done in UNIX for decades. The aim is to let applications leverage the
CSP programming style while, at the same time, make them work across
the network.
<p>
The problem with standard Go (or CSP-like) channels is that:
<ol>
<li>They do not behave well upon errors, regarding termination of
pipelines.
<li>They do not convey error messages when errors happen.
<p>
</ol>
Therefore, we modified the channel abstraction as provided by Go to
make it a better replacement for traditional pipes. When using
channels in Clive's Go, each end of the pipe may close it and the
channel implementation takes care of propagating the error indication
to the other end. Furthermore, an error string can be supplied when
closing a channel and the other end may inquire about the cause of the
error. This becomes utterly important when channels cross the network
because errors do happen.
<p>
For example, consider the pipeline
<p>
<hr>
<center>
<a name="fig1"></a>
<img src="http://lsub.org/export/golsub_pic1.pdf"></img></center>
<b>Figure 1:</b> <em>Example pipeline of processes in clive </em>
<hr><p>
<p>
In Clive, <code>proc2</code> can execute this code to
receive data from an input channel, modify it, and send the result to
an output channel:
<p><ul style="list-style:none;">
<code><pre>
var inc, outc chan[]byte
...
for data := range inc {
ndata := modify(data)
if ok := outc <-ndata; !ok {
close(inc, cerror(outc))
break
}
}
close(outc, cerror(inc))
</pre></code>
</ul><p>
Should the first process, <code>proc1</code>, terminate
normally (or abnormally), it calls <code>close</code> on
the <code>inc</code> channel shown in the code excerpt. At
this point, the code shown for <code>proc2</code> executes
<code>close(outc, cerror(inc))</code>, which does two
things:
<ol>
<li>retrieves the cause for the close of the input channel, by
calling <code>cerror(inc)</code>.
<li>closes the output channel providing exactly that error
indication, by calling <code>close</code> with a second
argument that provides the error.
</ol>
Therefore, the error at a point of the pipe can be nicely propagated
forward. In its interesting to to reconsider the implications of this
for examples like that shown for removing files, and for similar
system tools.
<p>
The most interesting case is when the third process,
<code>proc3</code>, decides to cease consuming data. For
example, because of an error or because it did find what it wanted. In
this case, it calls <code>close</code> on the
<code>outc</code> channel shown in the code.
<p>
The middle process is not able to send more data from that point in
time. Instead of panicing, as the standard Go implementation would do,
the send operation now returns <code>false</code>, thus
<code>ok</code> becomes <code>false</code>
when <code>proc2</code> tries to send more data. The loop
can be broken cleanly, closing also the input channel to signal to the
first process that there is no point in producing further data.
<p>
Furthermore, all involved processes can retrieve the actual error
indicating the source of the problems (which is not just
<em>"channel was closed"</em> and can be of more help).
<p>
As an aside, the last call to <code>close</code> becomes
now a no-operation, because the output channel was already closed, and
we don't need to add unnecessary code to prevent the call because in
Clive this does not panic, unlike in standard Go.
<p>
The important point is that termination of the data stream is easy to
handle for the program without resorting to exceptions (or panics),
and we know which one is the error, so we can take whatever measures
are convenient in each case.
<p>
There is a second change required by Clive: application contexts. We
had to modify the runtime to include the concept of an application id
that is inherited when new processes (goroutines) are created. Also,
we had to access the current process (goroutine) id.
<p>
These were the two required changes. But, once we had to maintain our
own Go compiler, we introduced other changes as well, as a
convenience.
<p>
The following sections describe the changes made, as a reference for
further ports. In all the changes we tried to be conservative and
preserve as much as possible the existing structure, to make it easy
to upgrade to future versions of the compiler.
<p>
Also, just in case we made a mistake regarding assumptions made by the
compiler, adding more checks was preferred. The changes look worse but
are safer.
<p>
<a name="sec3"></a><h2>CLOSE</h2>
The <code>close</code> operation accepts now an optional
second argument with the error status, and does not panic if the
channel is already closed or is <code>nil</code>. Sending
or receiving from a closed channel does not block and does not do
anything. A new function <code>cerror</code> returns such
error status, if any, for a given channel.
<p>
These calls are now equivalent:
<p><ul style="list-style:none;">
<code><pre>
close(c)
close(c, nil)
close(c, "")
</pre></code>
<p>
</ul><p>
<a name="sec3x1"></a><h3>CHANGES IN THE RUNTIME PACKAGE</h3>
The type <code>hchan</code> is changed to include an error
string embedded in the structure, to preserve the invariant that there
are no pointers to collect. This will change in the future, and we
will keep an <code>error</code> instead garbage collected
as everybode else.
<li><code>runtime/chan.go:/^type.hchan</code>
<p><ul style="list-style:none;">
<code><pre>
type hchan struct {
qcount uint // total data in the queue
dataqsiz uint // size of the circular queue
buf unsafe.Pointer // points to an array of dataqsiz elements
elemsize uint16
errlen uint16
closed uint32
elemtype *_type // element type
sendx uint // send index
recvx uint // receive index
recvq waitq // list of recv waiters
sendq waitq // list of send waiters
err [maxerr]byte
lock mutex
}
</pre></code>
</ul><p>
The new fields are <code>errlen</code> and
<code>err</code>.
<p>
The standard <code>closechan</code> is now a call to
<code>closechan2</code> with <code>nil</code>
as the second argument.
<li><code>runtime/chan.go:/^func.closechan</code>
<p><ul style="list-style:none;">
<code><pre>
func closechan(c *hchan) {
closechan2(c, nil)
}
</pre></code>
<p>
</ul><p>
A new <code>chanerrstr</code> function returns the error
string for the types accepted as a second argument to
<code>close</code>:
<li><code>runtime/chan.go:/^func.chanerrstr</code>
<p><ul style="list-style:none;">
<code><pre>
func chanerrstr(e interface{}) string {
if e == nil {
return ""
}
switch v := e.(type) {
case nil:
return ""
case stringer:
return v.String()
case error:
return v.Error()
case string:
return v
default:
panic("close errors must be a string or an error")
}
}
</pre></code>
<p>
</ul><p>
The old <code>closechan</code> is now
<code>closechan2</code>:
<li><code>runtime/chan.go:/^func.closechan2</code>
<p><ul style="list-style:none;">
<code><pre>
func closechan2(c *hchan, e interface{}) {
if c == nil {
return
}
estr := chanerrstr(e)
lock(&c.lock)
if c.closed != 0 {
unlock(&c.lock)
return
}
...
c.errlen = uint16(0)
if estr != "" {
n := (*stringStruct)(unsafe.Pointer(&estr)).len
if n > maxerr {
n = maxerr
}
c.errlen = uint16(n)
c.err[c.errlen] = 0
p := (*stringStruct)(unsafe.Pointer(&estr)).str
memmove(unsafe.Pointer(&c.err[0]), p, uintptr(c.errlen))
}
...
}
</pre></code>
<p>
</ul><p>
The <code>chansend</code> function is changed not to panic
when sending on a closed channel. It will be changed again later to
return a boolean indicating if the send could proceed or not. For now,
it returns <code>true</code> indicating the send is
complete (and discarded).
<li><code>runtime/chan.go:/^func.chansend</code>
<p><ul style="list-style:none;">
<code><pre>
func chansend(...) bool {
...
if c.closed != 0 {
unlock(&c.lock)
return true
}
// and the same in a few other places that did panic.
...
}
</pre></code>
<p>
</ul><p>
In <code>selectgoImpl</code> we have to change the case
for <code>sclose</code> so it does not panic. Instead,
selects proceeds without actually doing anything.
<p>
<li><code>runtime/select.go:/^sclose</code>
<p><ul style="list-style:none;">
<code><pre>
func selectgoImpl(...) (uintptr, uint16) {
...
sclose:
selunlock(sel)
goto retc
...
}
</pre></code>
<p>
</ul><p>
A new type and a couple of functions permits the user to call
<code>cerror()</code> and retrieve the error for a channel
(or nil), and to learn if the channel is closed and drained.
<li><code>runtime/chan.go:/^type.chanError</code>
<p><ul style="list-style:none;">
<code><pre>
type chanError string
func (e chanError) Error() string {
return string(e)
}
</pre></code>
<p>
</ul><p>
<li><code>runtime/chan.go:/^func.cerror</code>
<p><ul style="list-style:none;">
<code><pre>
func cerror(c *hchan) error {
if c == nil {
return nil
}
lock(&c.lock)
if c.closed == 0 || c.errlen == 0 || c.err[0] == 0 {
unlock(&c.lock)
return nil
}
msg := gostringn(&c.err[0], int(c.errlen))
unlock(&c.lock)
return chanError(msg)
}
</pre></code>
<p>
</ul><p>
<li><code>runtime/chan.go:/^func.cclosed</code>
<p><ul style="list-style:none;">
<code><pre>
func cclosed(c *hchan) bool {
if c == nil {
return true
}
lock(&c.lock)
closed := c.closed != 0 && (c.dataqsiz == 0 ||
c.qcount <= 0)
unlock(&c.lock)
return closed
}
</pre></code>
<p>
</ul><p>
<a name="sec3x2"></a><h3>CHANGES IN THE COMPILER</h3>
The compiler must add <code>cerror</code> and
<code>cclosed</code> as new builtins, and must decide
which one of <code>closechan</code> and
<code>closechan2</code> should be called.
<p>
We define new constants for nodes that are calls to
<code>cerror</code> or <code>cclosed</code>.
<li><code>cmd/compile/internal/gc/syntax.go:/OCCLOSED</code>
<p><ul style="list-style:none;">
<code><pre>
// Node ops.
const (
OXXX = iota
...
OCCLOSED // cclosed
OCERROR // cerror
OCLOSE // close
...
)
</pre></code>
<p>
</ul><p>
We give names for the new constants when printed:
<li><code>cmd/compile/internal/gc/fmt.go:0+/goopnames/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
var goopnames = []string{
...
OCCLOSED: "cclosed",
OCERROR: "cerror",
OCLOSE: "close",
...
}
</pre></code>
<p>
</ul><p>
Precedence must be given to <code>cclosed</code> and
<code>cerror</code>:
<li><code>cmd/compile/internal/gc/fmt.go:0+/opprec/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
var opprec = []int{
...
OCCLOSED: 8,
OCERROR: 8,
OCLOSE: 8,
...
}
</pre></code>
<p>
</ul><p>
Also, <code>exprfmt</code> has to check out if
<code>close</code> has one or two arguments and must add
cases for <code>cclosed</code> and
<code>cerror</code>.
<li><code>cmd/compile/internal/gc/fmt.go:/^func.exprfmt/+/OCLOSE/</code>
<p><ul style="list-style:none;">
<code><pre>
func exprfmt(n *Node, prec int) string {
...
case OCLOSE:
// nemo: close with 2nd arg
if n.Left != nil && n.Right != nil {
return fmt.Sprintf("%v(%v, %v)",
Oconv(int(n.Op), obj.FmtSharp),
n.Left, n.Right)
}
fallthrough
case OREAL,
OIMAG,
...
OCERROR, OCCLOSED,
...
}
</pre></code>
<p>
</ul><p>
The predefined <code>syms</code> at
<code>lex.go</code> must add
<code>cerror</code> and <code>cclosed</code>.
<li><code>cmd/compile/internal/gc/lex.go:/cclosed</code>
<p><ul style="list-style:none;">
<code><pre>
var syms = []struct {...} {
...
{"cclosed", LNAME, Txxx, OCCLOSED},
{"cerror", LNAME, Txxx, OCERROR},
{"close", LNAME, Txxx, OCLOSE},
...
}
</pre></code>
<p>
</ul><p>
The <code>opnames</code> array is auto-generated and we
don't have to add entries, but these are them.
<li><code>cmd/compile/internal/gc/opnames.go:/CCLOSED</code>
<p><ul style="list-style:none;">
<code><pre>
var opnames = []string{
...
OCCLOSED: "CCLOSED",
OCERROR: "CERROR",
OCLOSE: "CLOSE",
...
}
</pre></code>
<p>
</ul><p>
In <code>order.go</code> we must add
<code>cclosed</code> and <code>cerror</code>
to <code>orderstmt</code>.
<li><code>cmd/compile/internal/gc/order.go:/CCLOSED</code>
<p><ul style="list-style:none;">
<code><pre>
case OAS2,
OCLOSE,
OCCLOSED,
OCERROR,
...
</pre></code>
<p>
</ul><p>
In <code>racewalk.go</code> must do the same for
<code>racewalknode</code>.
<li><code>cmd/compile/internal/gc/racewalk.go:/CCLOSED</code>
<p><ul style="list-style:none;">
<code><pre>
// should not appear in AST by now
case OSEND,
ORECV,
OCCLOSED,
OCERROR,
OCLOSE,
</pre></code>
<p>
</ul><p>
In <code>typecheck1</code>,
<code>OCLOSE</code> must accept an optional second
argument and don't fail for send-only channels:
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.typecheck1/+/OCLOSE/</code>
<p><ul style="list-style:none;">
<code><pre>
case OCLOSE:
// nemo: accept opt. second arg and don't fail on close for
// send only channels.
args := n.List
if args == nil {
Yyerror("missing argument for close()")
n.Type = nil
return
}
if args.Next != nil && args.Next.Next != nil {
Yyerror("too many arguments for close()")
n.Type = nil
return
}
</pre></code>
<code><pre>
// nemo: this probably isn'tneeded. n should be ok already.
n.Left = args.N
if args.Next != nil {
n.Right = args.Next.N
} else {
n.Right = nil
}
n.List = nil
</pre></code>
<code><pre>
typecheck(&n.Left, Erv)
defaultlit(&n.Left, nil)
l := n.Left
t := l.Type
if t == nil {
n.Type = nil
return
}
if t.Etype != TCHAN {
Yyerror("invalid operation: %v (non-chan type %v)", n, t)
n.Type = nil
return
}
</pre></code>
<code><pre>
if n.Right != nil {
typecheck(&n.Right, Erv)
defaultlit(&n.Right, nil)
t = n.Right.Type
if t == nil {
n.Type = nil
return
}
// TODO: check that the type is string or an error type.
}
ok |= Etop
break OpSwitch
</pre></code>
<p>
</ul><p>
Also in <code>typecheck1</code>,
<code>cclosed</code> and <code>cerror</code>
must be processed.
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.typecheck1/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
case OCCLOSED, OCERROR:
// nemo: new builtins
ok |= Erv
args := n.List
if args == nil {
Yyerror("missing argument for %v", n)
n.Type = nil
return
}
if args.Next != nil {
Yyerror("too many arguments for %v", n)
n.Type = nil
return
}
</pre></code>
<code><pre>
n.Left = args.N
n.List = nil
typecheck(&n.Left, Erv)
defaultlit(&n.Left, nil)
l := n.Left
t := l.Type
if t == nil {
n.Type = nil
return
}
if t.Etype != TCHAN {
Yyerror("invalid operation: %v (non-chan type %v)", n, t)
n.Type = nil
return
}
if n.Op == OCCLOSED {
n.Type = Types[TBOOL]
} else {
n.Type = errortype
}
break OpSwitch
</pre></code>
<p>
</ul><p>
In <code>checkdefergo</code> we must prevent discarding
the result of <code>cclosed</code> and
<code>cerror</code>.
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.checkdefergo/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
case OAPPEND,
OCAP,
OCCLOSED,
OCERROR,
</pre></code>
<p>
</ul><p>
In <code>walkstmt</code> we must check walk the two new
builtins.
<li><code>cmd/compile/internal/gc/walk.go:/^func.walkstmt/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
case OAS,
OCCLOSED,
OCERROR,
</pre></code>
<p>
</ul><p>
In <code>walkexpr</code>, we must check if we have one or
two arguments for <code>close</code> and then call one of
<code>closechan</code> and
<code>closechan2</code>.
<li><code>cmd/compile/internal/gc/walk.go:/^func.walkexpr/+/OCLOSE/</code>
<p><ul style="list-style:none;">
<code><pre>
case OCLOSE:
if n.Right == nil {
fn := syslook("closechan", 1)
substArgTypes(fn, n.Left.Type)
n = mkcall1(fn, nil, init, n.Left)
} else {
fn := syslook("closechan2", 1)
substArgTypes(fn, n.Left.Type)
n = mkcall1(fn, nil, init, n.Left, n.Right)
}
goto ret
</pre></code>
<p>
</ul><p>
In <code>walkexpr</code>, we must add calls for the two
new builtins:
<li><code>cmd/compile/internal/gc/walk.go:/^func.walkexpr/+/OCCLOSED/</code>
<p><ul style="list-style:none;">
<code><pre>
case OCCLOSED:
fn := syslook("cclosed", 1)
substArgTypes(fn, n.Left.Type)
n = mkcall1(fn, Types[TBOOL], init, n.Left)
goto ret
case OCERROR:
fn := syslook("cerror", 1)
substArgTypes(fn, n.Left.Type)
n = mkcall1(fn, errortype, init, n.Left)
goto ret
</pre></code>
<p>
</ul><p>
The file <code>builtin.go</code> is generated, but anway
these are the new runtime functions called:
<li><code>cmd/compile/internal/gc/builtin.go:/closechan</code>
<p><ul style="list-style:none;">
<code><pre>
"func @\"\".closechan (@\"\".hchan·1 any)\n" +
"func @\"\".closechan2 (@\"\".hchan·1 any, @\"\".err·2 interface {})\n" +
"func @\"\".cerror (@\"\".hchan·2 any) (? error)\n" +
"func @\"\".cclosed (@\"\".hchan·2 any) (? bool)\n" +
</pre></code>
<p>
</ul><p>
A new file <code>lsub_test.go</code> tests for the changes
in close.
<p>
<a name="sec4"></a><h2>SEND</h2>
The send operation on a closed chan was changed to proceed, doing
nothing in that case. It must be changed to report if the send could
be done or not, as in:
<p><ul style="list-style:none;">
<code><pre>
if ok := c <- v; !ok {
...
}
</pre></code>
<p>
</ul><p>
<a name="sec4x1"></a><h3>CHANGES IN THE RUNTIME PACKAGE</h3>
A new function <code>chansend2</code>, replaces
<code>chansend1</code> as the entry point for sends. It
returns a <code>bool</code> reporting if the send was done
or not (i.e., if the channel was open or closed).
<li><code>runtime/chan.go:/^func.chansend2</code>
<p><ul style="list-style:none;">
<code><pre>
func chansend2(t *chantype, c *hchan, elem unsafe.Pointer) bool {
if t == nil {
return false // prevent this from inlining
}
_, did := chansend(t, c, elem, true,
getcallerpc(unsafe.Pointer(&t)))
return did
}
</pre></code>
<p>
</ul><p>
The old <code>chansend</code> is changed to return two
booleans instead of one: could we send without blocking?, and did the
send happen? (i.e., was the channel not closed).
<p>
When it did return <code>false</code>, it now:
<li><code>runtime/chan.go:/^func.chansend\(</code>
<p><ul style="list-style:none;">
<code><pre>
func chansend(...) (bool, bool) {
...
if !block {
return false, false
}
...
}
</pre></code>
</ul><p>
When it did return <code>true</code> because it could
send, it now does
<p><ul style="list-style:none;">
<code><pre>
return true, true
</pre></code>
<p>
</ul><p>
Also, when the channel is found closed:
<p><ul style="list-style:none;">
<code><pre>
if c.closed != 0 {
unlock(&c.lock)
return true, false
}
</pre></code>
</ul><p>
Note that this did panic before we changed anything.
<p>
This part of the code is also changed:
<p><ul style="list-style:none;">
<code><pre>
gp.waiting = nil
done := true
if gp.param == nil {
if c.closed == 0 {
throw("chansend: spurious wakeup")
}
// nemo: don't panic("send on closed channel")
done = false
}
gp.param = nil
if mysg.releasetime > 0 {
blockevent(int64(mysg.releasetime)-t0, 2)
}
releaseSudog(mysg)
return true, done
</pre></code>
<p>
</ul><p>
Because of this change, <code>selectnbsend</code> has to
be changed to use one of the two returned values.
<li><code>runtime/chan.go:/^func.selectnbsend</code>
<p><ul style="list-style:none;">
<code><pre>
func selectnbsend(...) (selected bool) {
can, _ := chansend(...)
return can
}
</pre></code>
<p>
</ul><p>
The same happens to <code>reflect_chansend</code>.
<li><code>runtime/chan.go:/^func.reflect_chansend</code>
<p><ul style="list-style:none;">
<code><pre>
func reflect_chansend(...) (selected bool) {
can, _ := chansend(...)
return can
}
</pre></code>
<p>
</ul><p>
<a name="sec4x2"></a><h3>CHANGES IN THE COMPILER</h3>
The syntax must now accept using <code>c<-x</code> as a
value. In the grammar we must note that
<li><code>cmd/compile/internal/gc/go.y:0+/^expr/+/LCOMM</code>
<p><ul style="list-style:none;">
<code><pre>
expr LCOMM expr
{
$$ = Nod(OSEND, $1, $3);
}
</pre></code>
</ul><p>
is now a valid expression once again. This does not change the code,
but there was a comment indicating that this was here just to report
syntax errors.
<p>
The file <code>builtin.go</code> is generated, but anway
this function is added:
<li><code>cmd/compile/internal/gc/builtin.go:/closechan</code>
<p><ul style="list-style:none;">
<code><pre>
"func @\"\".chansend2 (@\"\".chanType·2 *byte, @\"\".hchan·3 chan<- any, @\"\".elem·4 *any) (@\"\".res·1 bool)\n" +
</pre></code>
<p>
</ul><p>
The function <code>hascallchan</code> is used to see if
something has a call to a channel, and must now consider
<code>OSEND</code> as part of expressions:
<li><code>cmd/compile/internal/gc/const.go:/^func.hascallchan/+/OSEND</code>
<p><ul style="list-style:none;">
<code><pre>
func hascallchan(n *Node) bool {
...
switch n.Op {
case OAPPEND,
...
OSEND:
return true
}
...
}
</pre></code>
<p>
</ul><p>
It is ok to use send in assignments in a
<code>select</code>. We introduce a new
<code>OSELSEND</code> node type that will later be used
like <code>OSELRECV</code> nodes. First we define the new
node type.
<li><code>cmd/compile/internal/gc/syntax.go:/OSELSEND</code>
<p><ul style="list-style:none;">
<code><pre>
// Node ops.
const (
OXXX = iota
OSELRECV // case x = <-c:
OSELRECV2 // case x, ok = <-c:
OSELSEND // case ok = c <- x:
...
)
</pre></code>
<p>
</ul><p>
This is generated, but anyway...
<li><code>cmd/compile/internal/gc/opnames.go:/opnames/</code>
<p><ul style="list-style:none;">
<code><pre>
var opnames = []string{
...
OSELRECV: "SELRECV",
OSELRECV2: "SELRECV2",
OSELSEND: "SELSEND",
...
}
</pre></code>
<p>
</ul><p>
In <code>order</code>, a send can now happen within a
expression.
<li><code>cmd/compile/internal/gc/order.go:/^func.orderexpr/</code>
<p><ul style="list-style:none;">
<code><pre>
func orderexpr(np **Node, order *Order, lhs *Node) {
...
case OSEND:
t := marktemp(order);
orderexpr(&n.Left, order, nil)
orderexpr(&n.Right, order, nil)
orderaddrtemp(&n.Right, order)
cleantemp(t, order)
}
</pre></code>
<p>
</ul><p>
In <code>select</code>, we must prepare to accept
assignments using sends.
<li><code>cmd/compile/internal/gc/select.go:/^func.typecheckselect/+/OAS/</code>
<p><ul style="list-style:none;">
<code><pre>
func typecheckselect(sel *Node) {
...
case OAS:
switch n.Right.Op {
case ORECV:
n.Op = OSELRECV
case OSEND:
// n.Op = OSELSEND
Yyerror("BUG: TODO")
default:
Yyerror("must have chan op on rhs")
}
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/select.go:/^func.walkselect/+/OSEND/</code>
<p><ul style="list-style:none;">
<code><pre>
func walkselect(sel *Node) {
...
// optimization: one-case select: single op.
...
case OSEND:
ch = n.Left
case OSELSEND:
Fatal("walkselect OSELSEND not implemented")
...
// convert case value arguments to addresses.
case OSELSEND:
Fatal("walkselect OSELSEND not implemented")
...
// optimization: two-case select but one is default
case OSELSEND:
Fatal("walkselect OSELSEND not implemented")
...
// register cases
case OSELSEND:
Fatal("walkselect OSELSEND not implemented")
}
</pre></code>
<p>
</ul><p>
In <code>typecheck</code>,
<code>callrecv</code> must be updated so it does not
indicate if a node is just a call or receive, but also a send.
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.callrecv</code>
<p><ul style="list-style:none;">
<code><pre>
func callrecv(n *Node) bool {
...
case OCALL,
OSEND,
...
}
</pre></code>
<p>
</ul><p>
The main change is making <code>typecheck1</code> accept
<code>OSEND</code> as <code>Erv</code>.
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.typecheck1/+/OSEND/</code>
<p><ul style="list-style:none;">
<code><pre>
func typecheck1(np **Node, top int) {
...
case OSEND:
ok |= Etop|Erv
...
// TODO: more aggressive
// n.Etype = 0
n.Type = Types[TBOOL]
break OpSwitch
}
</pre></code>
<p>
</ul><p>
Also, in <code>walk</code>, calling
<code>chansend2</code> so it can return its value.
<li><code>cmd/compile/internal/gc/walk.go:/^func.walkexpr/+/OSEND/</code>
<p><ul style="list-style:none;">
<code><pre>
func walkexpr(...) {
...
case OSEND:
n1 := n.Right
n1 = assignconv(n1, n.Left.Type.Type, "chan send")
walkexpr(&n1, init)
n1 = Nod(OADDR, n1, nil)
n = mkcall1(chanfn("chansend2", 2, n.Left.Type),
Types[TBOOL], init,
typename(n.Left.Type), n.Left, n1)
n.Type = Types[TBOOL]
goto ret
}
</pre></code>
<p>
</ul><p>
<a name="sec5"></a><h2>SEND IN SELECTS</h2>
This change permits using
<p><ul style="list-style:none;">
<code><pre>
select {
case ok := c <- v:
...
}
</pre></code>
<p>
</ul><p>
<a name="sec5x1"></a><h3>CHANGES IN THE RUNTIME</h3>
Two new functions accept a pointer to the returned value in sends, one
blocks and one doesn't.
<p>
<li><code>runtime/chan.go:/^func.chanselsend/</code>
<p><ul style="list-style:none;">
<code><pre>
func chanselsend(t *chantype, c *hchan, elem unsafe.Pointer, okp *bool) bool {
if t == nil {
return false // prevent this from inlining
}
ok, did := chansend(t, c, elem, true, getcallerpc(unsafe.Pointer(&t)))
if okp != nil {
*okp = did
}
return ok
}
</pre></code>
<p>
<code><pre>
func channbselsend(t *chantype, c *hchan, elem unsafe.Pointer, okp *bool) bool {
if t == nil {
return false // prevent this from inlining
}
ok, did := chansend(t, c, elem, false, getcallerpc(unsafe.Pointer(&t)))
if okp != nil {
*okp = did
}
return ok
}
</pre></code>
<p>
</ul><p>
<a name="sec5x2"></a><h3>CHANGES IN THE COMPILER</h3>
In <code>typecheckselect</code>, we will convert cases
like<code>ok=c<-v</code> to
<code>OSELSEND</code> nodes, like done for receives.
<li><code>cmd/compile/internal/gc/select.go:/^func.typecheckselect/+/OAS/</code>
<p><ul style="list-style:none;">
<code><pre>
case OAS:
...
switch n.Right.Op {
case ORECV:
n.Op = OSELRECV
case OSEND:
n.Op = OSELSEND
default:
Yyerror("select assignment must have receive on rhs")
}
</pre></code>
<p>
</ul><p>
In <code>orderstmt</code>, we must add a case for
<code>OSELSEND</code> within
<code>OSELECT</code>.
<li><code>cmd/compile/internal/gc/order.go:/^func.orderstmt/+/OSELECT/+/OSELSEND/</code>
<p><ul style="list-style:none;">
<code><pre>
case OSELSEND:
if r.Colas {
t = r.Ninit
if t != nil && t.N.Op == ODCL && t.N.Left == r.Left {
t = t.Next
}
if t != nil && t.N.Op == ODCL && t.N.Left == r.Ntest {
t = t.Next
}
if t == nil {
r.Ninit = nil
}
}
if r.Ninit != nil {
Yyerror("ninit on select send")
dumplist("ninit", r.Ninit)
}
</pre></code>
<code><pre>
// case ok = c <- x
// r->left is ok, r->right is SEND, r->right->left is c, r->right->right is x
// r->left == N means 'case c<-x'.
// c is always evaluated; ok is only evaluated when assigned.
orderexpr(&r.Right.Left, order, nil)
if r.Right.Left.Op != ONAME {
r.Right.Left = ordercopyexpr(r.Right.Left, r.Right.Left.Type, order, 0)
}
</pre></code>
<code><pre>
if r.Left != nil && isblank(r.Left) {
r.Left = nil
}
if r.Left != nil {
tmp1 = r.Left
if r.Colas {
tmp2 = Nod(ODCL, tmp1, nil)
typecheck(&tmp2, Etop)
l.N.Ninit = list(l.N.Ninit, tmp2)
}
r.Left = ordertemp(tmp1.Type, order, false)
tmp2 = Nod(OAS, tmp1, r.Left)
typecheck(&tmp2, Etop)
l.N.Ninit = list(l.N.Ninit, tmp2)
}
orderblock(&l.N.Ninit)
</pre></code>
<p>
</ul><p>
We keep the old <code>OSEND</code> case within selects to
leave the previous setup undisturbed, in case we introduce any bugs.
<p>
In <code>walkselect</code>, we must handle the new case.
First in the one-case select.
<li><code>cmd/compile/internal/gc/select.go:/^func.walkselect/+/OSELSEND/</code>
<p><ul style="list-style:none;">
<code><pre>
// optimization: one-case select: single op.
...
case OSELSEND:
ch = n.Right.Left
if n.Op == OSELSEND || n.Ntest == nil {
if n.Left == nil {
n = n.Right
} else {
n.Op = OAS
}
break
}
Fatal("walkselect OSELSEND with OAS2")
</pre></code>
<p>
</ul><p>
Then while converting case arguments to addresses.
<p><ul style="list-style:none;">
<code><pre>
// convert case value arguments to addresses.
...
case OSELSEND:
n.Left = Nod(OADDR, n.Left, nil)
typecheck(&n.Left, Erv)
n.Right.Right = Nod(OADDR, n.Right.Right, nil)
typecheck(&n.Right.Right, Erv)
</pre></code>
<p>
</ul><p>
Next, in the two-case select with default optimization.
<p><ul style="list-style:none;">
<code><pre>
// optimization: two-case select but one is default
...
case OSELSEND:
r = Nod(OIF, nil, nil)
r.Ninit = cas.Ninit
ch := n.Right.Left
r.Ntest = mkcall1(chanfn("channbselsend", 2, ch.Type),
Types[TBOOL], &r.Ninit, typename(ch.Type),
ch, n.Right.Right, n.Left)
</pre></code>
<p>
</ul><p>
Finally, in the plain select cases.
<p><ul style="list-style:none;">
<code><pre>
// register cases
...
case OSELSEND:
r.Ntest = mkcall1(chanfn("chanselsend", 2, n.Right.Left.Type),
Types[TBOOL], &r.Ninit, var_,
n.Right.Left, n.Right.Right, n.Left)
</pre></code>
<p>
</ul><p>
The file <code>builtin.go</code> is generated, but anway
this is added:
<li><code>cmd/compile/internal/gc/builtin.go:/channbselsend</code>
<p><ul style="list-style:none;">
<code><pre>
"func @\"\".channbselsend (@\"\".chanType·2 *byte, @\"\".hchan·3 chan<- any, @\"\".elem·4 *any, @\"\".okp·5 *bool) (@\"\".res·1 bool)\n" +
"func @\"\".chanselsend (@\"\".chanType·2 *byte, @\"\".hchan·3 chan<- any, @\"\".elem·4 *any, @\"\".okp·5 *bool) (@\"\".res·1 bool)\n" +
</pre></code>
<p>
</ul><p>
<a name="sec6"></a><h2>APP IDS</h2>
This change provides each process (goroutine) with a new application
id, inherited when new processes are created.
<p>
First, a new <code>gappid</code> is added to
<code>g</code>.
<li><code>runtime/runtime2.go:/^.readyg /</code>
<p><ul style="list-style:none;">
<code><pre>
type g struct {
...
readyg *g
gappid int64
...
}
</pre></code>
<p>
</ul><p>
It is initialized to the <code>goid</code> for top-level
processes.
<li><code>runtime/proc1.go:/^func.newextram</code>
<p><ul style="list-style:none;">
<code><pre>
func newextram() {
...
gp.goid = int64(xadd64(&sched.goidgen, 1))
gp.gappid = gp.goid
...
}
</pre></code>
<p>
</ul><p>
<li><code>runtime/proc.go:/^func.main</code>
<p><ul style="list-style:none;">
<code><pre>
func main() {
g := getg()
g.gappid = g.goid
...
}
</pre></code>
<p>
</ul><p>
And it is inherited. We pass the application id as an argumetn because
<code>systemstack</code> is likely to run on
<code>g0</code> and not on the caller process context.
<p>
<li><code>runtime/proc1.go:/^/func.newproc\(</code>
<p><ul style="list-style:none;">
<code><pre>
func newproc(...) {
argp := add(unsafe.Pointer(&fn), ptrSize)
pc := getcallerpc(unsafe.Pointer(&siz))
appid := int64(0)
if _g_ := getg(); _g_ != nil {
appid = _g_.gappid
}
systemstack(func() {
newproc1(fn, (*uint8)(argp), siz, 0, appid, pc)
})
}
</pre></code>
<p>
<code><pre>
func newproc1(..., appid int64,...) {
...
newg.goid = int64(_p_.goidcache)
newg.gappid = appid
...
</pre></code>
<p>
</ul><p>
The interface for the user is like follows.
<li><code>runtime/proc.go:/^/func.AppId</code>
<p><ul style="list-style:none;">
<code><pre>
// Return the application id for the current process (goroutine).
func AppId() int64 {
g := getg()
return g.gappid
}
// Return the process id (goroutine id)
func GoId() int64 {
g := getg()
return g.goid
}
// Make the current process the leader of a new application, with its own id
// set to that of the process id.
func NewApp() {
g := getg()
g.gappid = g.goid
}
</pre></code>
<p>
</ul><p>
<a name="sec7"></a><h2>LOOPING SELECT CONSTRUCT</h2>
This change was not strictly required, but, because we had to change
the compiler as shown before, it was made for the programmer's
convenience.
<p>
The change introduces a new <code>doselect</code>
construct that is a looping select (similar to CSP's
<em>do</em> control structure). Within the construct, a
<code>break</code> breaks the entire loop and a
<code>continue</code> continues looping. This is an
example:
<p><ul style="list-style:none;">
<code><pre>
doselect {
case <-a:
...
case <-b:
if foo {
break
}
case <-c: {
if bar {
continue
}
...
}
</pre></code>
<p>
</ul><p>
The meaning is:
<p><ul style="list-style:none;">
<code><pre>
Loop:
for {
select {
case <-a:
...
case <-b:
if foo {
break Loop
}
case <-c: {
if bar {
continue Loop
}
...
}
}
}
</pre></code>
<p>
</ul><p>
First, we add a new token for <code>doselect</code>.
<li><code>cmd/compile/internal/gc/go.y:/LDOSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
...
%token <sym> LTYPE LVAR
%token <sym> LDOSELECT
...
</pre></code>
<p>
</ul><p>
Then we add it to the lexer.
<li><code>cmd/compile/internal/gc/lex.go:/func._yylex/+/LDOSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
...
case LFOR, LIF, LSWITCH, LSELECT, LDOSELECT:
loophack = 1 // see comment about loophack above
...
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/lex.go:/^var.syms/+/LDOSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
var syms = ... {
...
{"default", LDEFAULT, Txxx, OXXX},
{"doselect", LDOSELECT, Txxx, OXXX},
{"else", LELSE, Txxx, OXXX},
...
}
</pre></code>
</ul><p>
<li><code>cmd/compile/internal/gc/lex.go:/^var.lexn/+/LDOSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
var lexn = ... {
...
{LDEFER, "DEFER"},
{LDOSELECT, "DOSELECT"},
{LELSE, "ELSE"},
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/lex.go:/^var.yytfix/+/LDOSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
var yytfix = ... {
...
{LDEFER, "DEFER"},
{LDOSELECT, "DOSELECT"},
{LELSE, "ELSE"},
...
}
</pre></code>
<p>
</ul><p>
The grammar is changed to include the construct. A
<code>doselect</code> is built as a
<code>for</code> with a <code>select</code> in
it, but the node for <code>select</code> uses
<code>ODOSELECT</code> instead of
<code>OSELECT</code>, to let us handle breaks.
<p>
<li><code>cmd/compile/internal/gc/go.y:/select_stmtd/</code>
<p><ul style="list-style:none;">
<code><pre>
%type <node> doselect_stmt doselect_hdr
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/go.y:/^non_dcl_stmt/</code>
<p><ul style="list-style:none;">
<code><pre>
non_dcl_stmt:
...
| select_stmt
| doselect_stmt
...
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/go.y:/^doselect_stmt/</code>
<p><ul style="list-style:none;">
<code><pre>
doselect_stmt:
LDOSELECT
{
// for
markdcl();
}
doselect_hdr
{
// select
typesw = Nod(OXXX, typesw, nil);
}
LBODY caseblock_list '}'
{
// select
nd := Nod(ODOSELECT, nil, nil);
nd.Lineno = typesw.Lineno;
nd.List = $6;
typesw = typesw.Left;
// for
$$ = $3;
$$.Nbody = list1(nd)
popdcl();
}
</pre></code>
<p>
</ul><p>
The header works like in a <code>for</code> construct, so
we can do things like limit the number of loops, etc.
<li><code>cmd/compile/internal/gc/go.y:/^doselect_hdr/</code>
<p><ul style="list-style:none;">
<code><pre>
doselect_hdr:
osimple_stmt ';' osimple_stmt ';' osimple_stmt
{
// init ; test ; incr
if $5 != nil && $5.Colas {
Yyerror("cannot declare in the doselect-increment");
}
$$ = Nod(OFOR, nil, nil);
if $1 != nil {
$$.Ninit = list1($1);
}
$$.Ntest = $3;
$$.Nincr = $5;
}
| osimple_stmt
{
// normal test
$$ = Nod(OFOR, nil, nil);
$$.Ntest = $1;
}
</pre></code>
<p>
</ul><p>
A new node <code>ODOSELECT</code> is added mainly to
handle <code>break</code> and
<code>continue</code> as expected in the new construct.
<p>
<li><code>cmd/compile/internal/gc/syntax.go:/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
// Node ops.
const (
OXXX = iota
...
OSELECT // select
ODOSELECT // doselect
...
)
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^var.goopnames/</code>
<p><ul style="list-style:none;">
<code><pre>
var goopnames = []string{
...
OSELECT: "select",
ODOSELECT: "doselect",
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^func.stmtfmt/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func stmtfmt(n *Node) string {
...
case OSELECT, ODOSELECT, OSWITCH:
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^var.opprec/</code>
<p><ul style="list-style:none;">
<code><pre>
var opprec = []int{
...
OSELECT: -1,
ODOSELECT: -1,
...
}
</pre></code>
<p>
</ul><p>
This one is generated, but anyway...
<li><code>cmd/compile/internal/gc/opnames.go</code>
<p><ul style="list-style:none;">
<code><pre>
...
OSELECT: "SELECT",
ODOSELECT: "DOSELECT",
...
</pre></code>
<p>
</ul><p>
Now we have to honor the new node. In general, a
<code>ODOSELECT</code> is to be handled as a
<code>OSELECT</code> node, because it is already within a
<code>OFOR</code> node.
<p>
<li><code>cmd/compile/internal/gc/inl.go:/^func.ishairy/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func ishairy(n *Node, budget *int) bool {
...
case OCLOSURE,
OCALLPART,
ORANGE,
OFOR,
OSELECT,
ODOSELECT,
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/order.go:/^func.orderstmt\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func orderstmt(n *Node, order *Order) {
...
case OSELECT, ODOSELECT:
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/racewalk.go:/^func.racewalknode\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func racewalknode(np **Node, init **NodeList, wr int, skip int) {
...
// just do generic traversal
case OFOR,
...
OSELECT,
ODOSELECT,
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.typecheck1\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func typecheck1(np **Node, top int) {
...
case OSELECT, ODOSELECT:
ok |= Etop
typecheckselect(n)
break OpSwitch
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.markbreak\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func markbreak(n *Node, implicit *Node) {
...
case OFOR,
OSWITCH,
OTYPESW,
OSELECT,
ODOSELECT,
ORANGE:
implicit = n
fallthrough
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.markbreaklist\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func markbreaklist(...) {
...
case OFOR,
OSWITCH,
OTYPESW,
OSELECT,
ODOSELECT,
ORANGE:
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.isterminating\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func isterminating(...) {
...
case OSWITCH, OTYPESW, OSELECT, ODOSELECT:
if n.Hasbreak {
return false
}
...
if n.Op != OSELECT && n.Op != ODOSELECT && def == 0 {
return false
}
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/walk.go:/^func.walkstmt\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func walkstmt(np **Node) {
...
case OSELECT, ODOSELECT:
walkselect(n)
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/gen.go:/^func.gen\(/+/OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
func gen(n *Node) {
...
if n.Defn != nil {
switch n.Defn.Op {
// so stmtlabel can find the label
case OFOR, OSWITCH, OSELECT, ODOSELECT:
n.Defn.Sym = lab.Sym
}
}
...
}
</pre></code>
<p>
</ul><p>
And this is the main change for a <code>ODOSELECT</code>.
It works like a <code>select</code> but does not redefine
the user break PC, so that breaks and continues always refer to the
enclosing, implicit, <code>for</code> loop.
<p>
The idea is that implicit <code>breaks</code> inserted by
the compiler will not be <code>OBREAK</code>, but
<code>OCBREAK</code>. The new
<code>OCBREAK</code> is a compiler-inserted break and
<code>gen.go</code> can skip those breaks when jumping on
<code>break</code> and <code>continue</code>
within <code>doselect</code> structures.
<p>
<li><code>cmd/compile/internal/gc/syntax.go:/OBREAK</code>
<p><ul style="list-style:none;">
<code><pre>
// Node ops.
const (
OXXX = iota
...
OBREAK // break
OCBREAK // break generated by the compiler
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/opnames.go:/OCBREAK</code>
<p><ul style="list-style:none;">
<code><pre>
...
OBREAK: "BREAK",
OCBREAK: "CBREAK",
...
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^var.goopnames/</code>
<p><ul style="list-style:none;">
<code><pre>
var goopnames = []string{
...
OBREAK: "break",
OCBREAK: "break",
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^func.stmtfmt/</code>
<p><ul style="list-style:none;">
<code><pre>
func stmtfmt(n *Node) string {
...
case OBREAK, OCBREAK,
OCONTINUE,
OGOTO,
OFALL,
OXFALL:
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/fmt.go:/^var.opprec/</code>
<p><ul style="list-style:none;">
<code><pre>
var opprec = []int{
...
OBREAK: -1,
OCBREAK: -1,
...
}
</pre></code>
<p>
</ul><p>
In <code>select</code> we insert
<code>OCBREAK</code> nodes instead of
<code>OBREAK</code>, which are now left for the user
breaks.
<li><code>cmd/compile/internal/gc/select.go:/^func.racewalknode/</code>
<p><ul style="list-style:none;">
<code><pre>
func walkselect(sel *Node) {
...
r.Nbody = concat(r.Nbody, cas.Nbody)
r.Nbody = list(r.Nbody, Nod(OCBREAK, nil, nil))
init = list(init, r)
...
}
</pre></code>
<p>
</ul><p>
The same must be done in <code>swt</code> for switches.
<li><code>cmd/compile/internal/gc/swt.go:/^func.casebody/</code>
<p><ul style="list-style:none;">
<code><pre>
func casebody(sw *Node, typeswvar *Node) {
...
var cas *NodeList // cases
var stat *NodeList // statements
var def *Node // defaults
br := Nod(OCBREAK, nil, nil)
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/swt.go:/^func.*exprswitch.*walk/</code>
<p><ul style="list-style:none;">
<code><pre>
func (s *exprSwitch) walk(sw *Node) {
...
if len(cc) > 0 && cc[0].typ == caseKindDefault {
def = cc[0].node.Right
cc = cc[1:]
} else {
def = Nod(OCBREAK, nil, nil)
}
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/swt.go:/^func.*typeSwitch.*walk/</code>
<p><ul style="list-style:none;">
<code><pre>
func (s *typeSwitch) walk(sw *Node) {
...
if len(cc) > 0 && cc[0].typ == caseKindDefault {
def = cc[0].node.Right
cc = cc[1:]
} else {
def = Nod(OCBREAK, nil, nil)
}
...
}
</pre></code>
<p>
</ul><p>
And almost all processing is shared with the user
<code>OBREAK</code> node.
<p>
<li><code>cmd/compile/internal/gc/order.go:/^func.orderstmt/</code>
<p><ul style="list-style:none;">
<code><pre>
func orderstmt(n *Node, order *Order) {
...
case OBREAK, OCBREAK,
OCONTINUE,
ODCL,
ODCLCONST,
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/racewalk.go:/^func.racewalknode/</code>
<p><ul style="list-style:none;">
<code><pre>
func racewalknode(...) {
...
case OFOR,
OBREAK,
OCBREAK,
OCONTINUE,
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.typecheck1/</code>
<p><ul style="list-style:none;">
<code><pre>
func typecheck1(np **Node, top int) {
...
case OBREAK,
OCBREAK,
OCONTINUE,
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/typecheck.go:/^func.markbreak/</code>
<p><ul style="list-style:none;">
<code><pre>
func markbreak(n *Node, implicit *Node) {
...
switch n.Op {
case OBREAK, OCBREAK:
...
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/walk.go:/^func.markbreak/</code>
<p><ul style="list-style:none;">
<code><pre>
func func walkstmt(np **Node) {
...
case OBREAK,
OCBREAK,
ODCL,
...
}
</pre></code>
<p>
</ul><p>
Here is where things start to change. A new
<code>ubreakpc</code> records the PC for user (not
compiler) breaks.
<li><code>cmd/compile/internal/gc/go.go:/^var.breakpc/</code>
<p><ul style="list-style:none;">
<code><pre>
var breakpc, ubreakpc *obj.Prog
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/pgen.go:/^func.compile/+/breakpc/</code>
<p><ul style="list-style:none;">
<code><pre>
func compile(fn *Node) {
...
continpc = nil
breakpc = nil
ubreakpc = nil
...
}
</pre></code>
<p>
</ul><p>
The code in <code>gen</code> is changed now so that
<code>ubreakpc</code> is recorded for user breaks but not
for compiler-inserted breaks.
<p>
The processing for <code>OBREAK</code> and
<code>OCBREAK</code> differs in the
<code>breakpc</code> used (which is be
<code>ubreakpc</code> for user breaks).
<p>
Processing for <code>ODOSELECT</code> is like that for
<code>OSELECT</code> but does not redefine the user break,
so that breaks and continues refer to the enclosing for loop inserted
by the compiler.
<p>
<li><code>cmd/compile/internal/gc/gen.go:/^func.gen/+/^.case.OBREAK/</code>
<p><ul style="list-style:none;">
<code><pre>
case OBREAK, OCBREAK:
...
if breakpc == nil || ubreakpc == nil {
Yyerror("break is not in a loop")
break
}
if n.Op == OBREAK {
gjmp(ubreakpc)
} else {
gjmp(breakpc)
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/gen.go:/^func.gen/+/^.case.OFOR/</code>
<p><ul style="list-style:none;">
<code><pre>
case OFOR:
sbreak, subreak := breakpc, ubreakpc
p1 := gjmp(nil) // goto test
breakpc = gjmp(nil) // break: goto done
ubreakpc = breakpc
...
Patch(breakpc, Pc) // done:
Patch(ubreakpc, Pc) // done:
continpc = scontin
breakpc, ubreakpc = sbreak, subreak
if lab != nil {
lab.Breakpc = nil
lab.Continpc = nil
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/gen.go:/^func.gen/+/^.case.OSWITCH/</code>
<p><ul style="list-style:none;">
<code><pre>
case OSWITCH:
sbreak, subreak := breakpc, ubreakpc
p1 := gjmp(nil) // goto test
breakpc = gjmp(nil) // break: goto done
ubreakpc = breakpc
// define break label
lab := stmtlabel(n)
if lab != nil {
lab.Breakpc = breakpc
}
Patch(p1, Pc) // test:
Genlist(n.Nbody) // switch(test) body
Patch(breakpc, Pc) // done:
Patch(ubreakpc, Pc) // done:
breakpc, ubreakpc = sbreak, subreak
if lab != nil {
lab.Breakpc = nil
}
</pre></code>
<p>
</ul><p>
<li><code>cmd/compile/internal/gc/gen.go:/^func.gen/+/^.case.OSELECT/</code>
<p><ul style="list-style:none;">
<code><pre>
case OSELECT, ODOSELECT:
sbreak, subreak := breakpc, ubreakpc
p1 := gjmp(nil) // goto test
breakpc = gjmp(nil) // break: goto done
if n.Op == OSELECT {
ubreakpc = breakpc
}
// define break label
lab := stmtlabel(n)
if lab != nil {
lab.Breakpc = breakpc
}
Patch(p1, Pc) // test:
Genlist(n.Nbody) // select() body
Patch(breakpc, Pc) // done:
breakpc = sbreak
if n.Op == OSELECT {
Patch(ubreakpc, Pc) // done:
ubreakpc = subreak
}
if lab != nil {
lab.Breakpc = nil
}
</pre></code>
<p>
</ul><p>
<a name="sec8"></a><h2>IMPLICIT STRUCTURE AND INTERFACE
DECLARATIONS</h2>
This is yet another convenience change, added because we already had
to change the compiler.
<p>
In most cases types are <code>struct</code> types. It can
be easy for the compiler in certain cases to assume that a type
declaration where the <code>struct</code> keyword is
missing is a <code>struct</code> type declaration. We
assume that a structure is declared if we see something like
<p><ul style="list-style:none;">
<code><pre>
type Point {
x, y int
}
</pre></code>
</ul><p>
while a type is declared (i.e., in the
<code>typedcl</code> node of the grammar).
<p>
In the same way, because <code>interface{}</code> is a
very popular type for channels in Clive, the
<code>interface</code> keyword can be removed when
declaring the type for a channel. These two are equivalent:
<p><ul style="list-style:none;">
<code><pre>
chan {}
chan interface{}
</pre></code>
<p>
</ul><p>
The changes in the grammar are as shown here.
<li><code>cmd/compile/internal/gc/go.y</code>
<p><ul style="list-style:none;">
<code><pre>
%type <node> implstructtype implinterfacetype
...
</pre></code>
<code><pre>
typedcl:
typedclname ntype
{
$$ = typedcl1($1, $2, true);
}
|
typedclname implstructtype
{
$$ = typedcl1($1, $2, true);
}
...
</pre></code>
<code><pre>
implstructtype:
lbrace structdcl_list osemi '}'
{
$$ = Nod(OTSTRUCT, nil, nil);
$$.List = $2;
fixlbrace($1);
}
| lbrace '}'
{
$$ = Nod(OTSTRUCT, nil, nil);
fixlbrace($1);
}
...
</pre></code>
<code><pre>
implinterfacetype:
lbrace '}'
{
$$ = Nod(OTINTER, nil, nil);
fixlbrace($1);
}
...
</pre></code>
<code><pre>
othertype:
...
| LCHAN non_recvchantype
{
$$ = Nod(OTCHAN, $2, nil);
$$.Etype = Cboth;
}
| LCHAN LCOMM ntype
{
$$ = Nod(OTCHAN, $3, nil);
$$.Etype = Csend;
}
| LCHAN implinterfacetype
{
$$ = Nod(OTCHAN, $2, nil);
$$.Etype = Cboth;
}
| LCHAN LCOMM implinterfacetype
{
$$ = Nod(OTCHAN, $3, nil);
$$.Etype = Csend;
}
...
</pre></code>
<code><pre>
recvchantype:
LCOMM LCHAN ntype
{
$$ = Nod(OTCHAN, $3, nil);
$$.Etype = Crecv;
}
|
LCOMM LCHAN implinterfacetype
{
$$ = Nod(OTCHAN, $3, nil);
$$.Etype = Crecv;
}
</pre></code>
<p>
</ul><p>
<a name="sec9"></a><h2>GO PACKAGE AND GO TOOLS</h2>
Previous changes should suffice, given that the compiler is now
written in Go. However, there is a <code>go</code> package
that contains yet another parser for the language, and it has to be
changed as well. Most Go tools (commands) use it, and we must update
it.
<p>
<a name="sec9x1"></a><h3>CHANNEL SENDS</h3>
We must add <code><-</code> in the predecende table. To
preserve the levels, hardwired into <code>gofmt</code>, we
set for the send operation the lowest one.
<li><code>/usr/local/go/src/go/token/token.go:/^.LowestPrec</code>
<p><ul style="list-style:none;">
<code><pre>
const (
LowestPrec = 0 // non-operators
UnaryPrec = 6
HighestPrec = 7
)
</pre></code>
<code><pre>
func (op Token) Precedence() int {
switch op {
case ARROW, LOR:
return 1
case LAND:
return 2
case EQL, NEQ, LSS, LEQ, GTR, GEQ:
return 3
case ADD, SUB, OR, XOR:
return 4
case MUL, QUO, REM, SHL, SHR, AND, AND_NOT:
return 5
}
return LowestPrec
}
</pre></code>
<p>
</ul><p>
<a name="sec9x2"></a><h3>LOOPING SELECTS</h3>
The main change id adding <code>DOSELECT</code> as a new
token.
<li><code>/usr/local/go/src/go/token/token.go</code>
<p><ul style="list-style:none;">
<code><pre>
// The list of tokens.
const (
...
DEFAULT
DEFER
DOSELECT
ELSE
FALLTHROUGH
FOR
...
)
</pre></code>
<code><pre>
var tokens = [...]string{
...
DEFAULT: "default",
DEFER: "defer",
DOSELECT: "doselect",
ELSE: "else",
FALLTHROUGH: "fallthrough",
FOR: "for",
...
}
</pre></code>
<p>
</ul><p>
The AST must include a <code>DoSelectStmt</code>.
<li><code>/usr/local/go/src/go/ast/ast.go:/^.DoSelectStmt</code>
<p><ul style="list-style:none;">
<code><pre>
// A DoSelectStmt node represents a doselect statement.
DoSelectStmt struct {
DoSelect token.Pos // position of "doselect" keyword
Init Stmt // initialization statement; or nil
Cond Expr // condition; or nil
Post Stmt // post iteration statement; or nil
Body *BlockStmt // CommClauses only
}
</pre></code>
<p>
</ul><p>
And its methods...
<li><code>/usr/local/go/src/go/ast/ast.go</code>
<p><ul style="list-style:none;">
<code><pre>
...
func (s *SelectStmt) Pos() token.Pos { return s.Select }
func (s *DoSelectStmt) Pos() token.Pos { return s.DoSelect }
...
func (s *SelectStmt) End() token.Pos { return s.Body.End() }
func (s *DoSelectStmt) End() token.Pos { return s.Body.End() }
...
func (*SelectStmt) stmtNode() {}
func (*DoSelectStmt) stmtNode() {}
</pre></code>
<p>
</ul><p>
Plus a <code>walk</code> for it.
<li><code>/usr/local/go/src/go/ast/walk.go</code>
<p><ul style="list-style:none;">
<code><pre>
func Walk(v Visitor, node Node) {
...
case *DoSelectStmt:
if n.Init != nil {
Walk(v, n.Init)
}
if n.Cond != nil {
Walk(v, n.Cond)
}
if n.Post != nil {
Walk(v, n.Post)
}
Walk(v, n.Body)
case *ForStmt:
...
}
</pre></code>
<p>
</ul><p>
Then the parser. There is a new statement to synchronize on errors.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.syncStmt\(</code>
<p><ul style="list-style:none;">
<code><pre>
func syncStmt(p *parser) {
for {
switch p.tok {
case token.BREAK, ...
token.DOSELECT, ...
token.VAR:
...
case token.EOF:
return
}
p.next()
}
}
</pre></code>
<p>
</ul><p>
And there is a new statement.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.parseStmt\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseStmt() (s ast.Stmt) {
...
case token.SELECT:
s = p.parseSelectStmt()
case token.DOSELECT:
s = p.parseDoSelectStmt()
...
}
</pre></code>
<p>
</ul><p>
The parsing is taken from the parsing of a
<code>for</code> header and a
<code>select</code> body.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.parseStmt\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseDoSelectStmt() *ast.DoSelectStmt {
if p.trace {
defer un(trace(p, "DoSelectStmt"))
}
pos := p.expect(token.DOSELECT)
p.openScope()
defer p.closeScope()
var s1, s2, s3 ast.Stmt
if p.tok != token.LBRACE {
prevLev := p.exprLev
p.exprLev = -1
if p.tok != token.SEMICOLON {
isRange := false
if p.tok == token.RANGE {
isRange = true
} else {
s2, isRange = p.parseSimpleStmt(basic)
}
if isRange {
p.error(pos, "unexpected range")
// but ignore it for now
}
}
if p.tok == token.SEMICOLON {
p.next()
s1 = s2
s2 = nil
if p.tok != token.SEMICOLON {
s2, _ = p.parseSimpleStmt(basic)
}
p.expectSemi()
if p.tok != token.LBRACE {
s3, _ = p.parseSimpleStmt(basic)
}
}
p.exprLev = prevLev
}
lbrace := p.expect(token.LBRACE)
var list []ast.Stmt
for p.tok == token.CASE || p.tok == token.DEFAULT {
list = append(list, p.parseCommClause())
}
rbrace := p.expect(token.RBRACE)
p.expectSemi()
body := &ast.BlockStmt{Lbrace: lbrace, List: list, Rbrace: rbrace}
return &ast.DoSelectStmt {
DoSelect: pos,
Init: s1,
Cond: p.makeExpr(s2, "boolean expression"),
Post: s3,
Body: body,
}
}
</pre></code>
<p>
</ul><p>
Now we can print it.
<li><code>/usr/local/go/src/go/printer/nodes.go:/^func.*printer.*stmt\(/</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *printer) stmt(stmt ast.Stmt, nextIsRBrace bool) {
...
case *ast.DoSelectStmt:
p.print(token.DOSELECT, blank)
p.controlClause(true, s.Init, s.Cond, s.Post)
body := s.Body
if len(body.List) == 0 && !p.commentBefore(p.posFor(body.Rbrace)) {
// print empty select statement w/o comments on one line
p.print(body.Lbrace, token.LBRACE, body.Rbrace, token.RBRACE)
} else {
p.block(body, 0)
}
...
}
</pre></code>
<p>
</ul><p>
<a name="sec9x3"></a><h3>IMPLICIT KEYWORDS</h3>
We are going to flag <code>StructType</code> for implicit
<code>struct</code> and <code>interface</code>
declarations.
<li><code>/usr/local/go/src/go/ast/ast.go:/^.StructType</code>
<p><ul style="list-style:none;">
<code><pre>
// A StructType node represents a struct type.
StructType struct {
Struct token.Pos // position of "struct" keyword
Fields *FieldList // list of field declarations
Incomplete bool
Implicit bool
}
</pre></code>
<p>
</ul><p>
<li><code>/usr/local/go/src/go/ast/ast.go:/^.InterfaceType</code>
<p><ul style="list-style:none;">
<code><pre>
// An InterfaceType node represents an interface type.
InterfaceType struct {
Interface token.Pos // position of "interface" keyword
Methods *FieldList // list of methods
Incomplete bool
Implicit bool
}
</pre></code>
<p>
</ul><p>
Globals in the parser records if we can accept implicit keywords.
<li><code>/usr/local/go/src/go/parser/parser.go:/^type.parser</code>
<p><ul style="list-style:none;">
<code><pre>
type parser struct {
...
implStructOk, implInterOk bool
}
</pre></code>
<p>
</ul><p>
In a global type declaration, we accept
<code>struct</code> to be implicit. This is not exactly
what the Go compiler does, but it is close enough.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.*parser.*parseDecl\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseDecl(sync func(*parser)) ast.Decl {
if p.trace {
defer un(trace(p, "Declaration"))
}
p.implStructOk = false
defer func() {p.implStructOk = false}()
var f parseSpecFunction
switch p.tok {
...
case token.TYPE:
p.implStructOk = true
f = p.parseTypeSpec
...
}
return p.parseGenDecl(p.tok, f)
}
</pre></code>
<p>
</ul><p>
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.*parser.*parseGenDecl\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseGenDecl(...) *ast.GenDecl {
...
old := p.implStructOk
for ... {
p.implStructOk = old
list = append(...)
}
...
}
</pre></code>
<p>
</ul><p>
Later, <code>parseStructType</code> can honor the flag.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.*parser.*parseStructType\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseStructType() *ast.StructType {
if p.trace {
defer un(trace(p, "StructType"))
}
var pos, lbrace token.Pos
implicit := p.implStructOk
if implicit && p.tok == token.LBRACE {
pos = p.expect(token.LBRACE)
lbrace = pos
} else {
pos = p.expect(token.STRUCT)
lbrace = p.expect(token.LBRACE)
}
old := p.implStructOk
p.implStructOk = false
defer func() {p.implStructOk = old}()
scope := ast.NewScope(nil) // struct scope
...
return &ast.StructType{
Struct: pos,
Fields: &ast.FieldList{
Opening: lbrace,
List: list,
Closing: rbrace,
},
Implicit: implicit,
}
}
</pre></code>
</ul><p>
The flag is saved, cleared, and restored to prevent implicit
<code>struct</code> declarations anywhere but at the
top-level.
<p>
To accept implicit <code>interface</code> declarations, we
set the flag while declaring a channel type.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.*parser.*parseChanType\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseChanType() *ast.ChanType {
...
p.implInterOk = true
value := p.parseType()
p.implInterOk = false
...
}
</pre></code>
<p>
</ul><p>
And <code>parseInterfaceType</code> takes care of the
flag.
<li><code>/usr/local/go/src/go/parser/parser.go:/^func.*parser.*parseInterfaceType\(</code>
<p><ul style="list-style:none;">
<code><pre>
func (p *parser) parseInterfaceType() *ast.InterfaceType {
if p.trace {
defer un(trace(p, "InterfaceType"))
}
var pos, lbrace token.Pos
implicit := p.implInterOk
if implicit && p.tok == token.LBRACE {
pos = p.expect(token.LBRACE)
lbrace = pos
} else {
pos = p.expect(token.INTERFACE)
lbrace = p.expect(token.LBRACE)
}
p.implInterOk = false
scope := ast.NewScope(nil) // interface scope
var list []*ast.Field
for p.tok == token.IDENT {
list = append(list, p.parseMethodSpec(scope))
}
if implicit && len(list) > 0 {
p.error(pos, "ok only for empty interfaces")
}
rbrace := p.expect(token.RBRACE)
return &ast.InterfaceType{
Interface: pos,
Methods: &ast.FieldList{
Opening: lbrace,
List: list,
Closing: rbrace,
},
Implicit: implicit,
}
}
</pre></code>
<p>
</ul><p>
This time we clear the flag right after using it, because the implicit
interface declaration works only right after the
<code>chan</code> keyword (but for send/receive only
indications).
<p>
In the printer, we define
<li><code>/usr/local/go/src/go/printer/printer.go:/^type.Config</code>
<p><ul style="list-style:none;">
<code><pre>
type Config struct {
Mode Mode // default: 0
Tabwidth int // default: 8
Indent int // default: 0 (all code is indented at least by this much)
DontPrintImplicits bool
}
</pre></code>
</ul><p>
The flag <code>DontPrintImplicits</code> may be set by the
code using this package to instruct nodes not to print the implicit
keywords. By default, they are printed.
<p>
The <code>gofmt</code> command is given a flag to set it.
<li><code>/usr/local/go/src/cmd/gofmt/gofmt.go</code>
<p><ul style="list-style:none;">
<code><pre>
var noImpls = flag.Bool("S", false,
"omit struct keyword in top-level type declarations")
</pre></code>
<p>
</ul><p>
And to process file...
<li><code>/usr/local/go/src/cmd/gofmt/gofmt.go:/^func.processFile</code>
<p><ul style="list-style:none;">
<code><pre>
func processFile(...) error {
cfg := printer.Config{..., DontPrintImplicits: noImpls}
res, err := format.Format(..., cfg)
}
</pre></code>
<p>
</ul><p>
<p>Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-73046817947558045772015-05-29T07:06:00.000-07:002015-05-29T07:06:40.183-07:00Random notes on the Go 1.4 runtime.These are some notes on the Go 1.4 Run-Time written while working on the new Clive kernel.
I just copied the text from the TR, which will be linked at the Lsub page and updated in the future.
They should be easy to follow using a Go 1.4 tree because the package and function names are
indicated. The line numbers are useful mostly for our local use in Acme at Lsub, using the
text output from wr(1).
<p>
<b>Francisco J. Ballesteros</b><br>
<b>TR LSUB-15-2 draft</b><br>
<hr>
<p>
<a name="sec"></a><h2>Abstract</h2>
This TR contains notes about the Go run time as of Go 1.4. They are
intended to aid in the construction of a kernel for Clive.
<p>
<a name="sec1"></a><h2>1. Initialization</h2>
<li><code>rt0_darwin_amd64.s:13 main()</code> is the
entry point for the binary. A direct jump to...
<p>
<li><code>asm_amd64.s:9: runtime·rt0_go</code> which is
the actual crt0. This initializes the stack calls
<code>_cgo_init</code> if needed, updates stack guards,
setups TLS, sets <code>m</code> to
<code>m0</code> and <code>g</code> to
<code>g0</code> and continues calling...
<p>
<ul>
<li><code>runtime·args</code> to save the arguments from
the OS
<li><code>os_darwin.c:54: runtime·osinit</code> to
initialize threading (they don't use their libc)
<li><code>proc.c:122: runtime·schedinit(void)</code> to
create a G that calls <code>runtime·main</code>.
<ul>
<li><code>traceback.go:48: func tracebackinit()</code>
<li><code>symtab.go:46: func symtabinit</code>
<li><code>stack.c:42: runtime·stackinit(void)</code>
calls this function for the stacks in the pool:
<ul>
<li><code>mheap.c:663: runtime·MSpanList_Init</code>
</ul>
<li><code>malloc.c:109: runtime·mallocinit</code>
<ul>
<li><code>mheap.c:57: runtime·MHeap_Init</code>
initializes the heap using these:
<ul>
<li><code>mfixalloc.c:16: runtime·FixAlloc_Init</code>,
for several allocs.
<li><code>mheap.c:663: runtime·MSpanList_Init</code>,
for free and busy lists.
<li><code>mcentral.c:21: runtime·MCentral_Init</code>,
for mcentral lists.
</ul>
<li><code>mcache.c:19: runtime·allocmcache</code> sets
<code>g->m->mcache</code>, a per-P malloc cache for small
objects.
</ul>
<li><code>proc.c:190: mcommoninit</code> calls
os-specific <code>mpreinit</code> and links m into the
sched list (<code>allm</code>).
<li><code>goargs</code> and
<code>goenvs</code> save the UNIX arguments.
<li><code>mgc0.c:1345: runtime·gcinit</code> sets GC
masks for data and bss.
<li><code>proc.c:2655: procresize</code> is called to
create <em>Ps</em> for
<code>GOMAXPROCS</code>.
</ul>
<li><code>proc.c:2154: runtime·newproc</code> is called
to run <code>runtime·main</code> using
<ul>
<li><code>asm_amd64.s:218: runtime·onM</code> to call
<code>newproc_m)</code> to run
<code>runtime·main</code>. This switches to g0, if not on
m stack (i.e., no switch this time).
<ul>
<li><code>proc.c:2128: newproc_m</code> is called by g0
and copies the args and then calls:
<ul>
<li><code>proc.c:2182: runtime·newproc1</code>. This
creates a G to run a function (<code>runtime·main</code>).
Using:
<ul>
<li><code>proc.c:2295: gfget</code> to get a G from the
free list, or
<li><code>proc.c:2102: runtime·malg</code> to allocate a
G, which ends up doing a <code>new(g)</code> in go.
</ul>
Then it copies arguments to the new G stack, sets the sched label to
call the function, using
<ul>
<li><code>stack.c:806: runtime·gostartcallfn</code>,
similar to set-label.
</ul>
Sets the state to Grunnable and calls
<ul>
<li><code>proc.c:3284: runqput</code> to put the new G
on a local runq or the global one.
<li><code>proc.c:1276: wakep</code>, which is a nop or a
call to sched the M for P or make a new M for P.
<ul>
<li><code>proc.c:1200: startm</code>
</ul>
</ul>
</ul>
</ul>
</ul>
<li><code>proc.c:818: runtime·mstart</code> starts the
M. It calls:
<ul>
<li><code>asminit</code> to do per-thread initialization
(nop for our arch).
<li><code>os_darwin.c:144: minit</code> initializes
signal handling for our arch.
<li><code>proc.c:1537: schedule</code> runs the
scheduller and calls this when gets something to run:
<ul>
<li><code>proc.c:1358: execute</code> is mostly a call
to a goto-label:
<ul>
<li><code>asm_amd64.s:143: gogo</code>
<p>
</ul>
</ul>
</ul>
</ul>
At the end, the new G calls
<li><code>proc.go:16: runtime·main()</code>, which calls
<ul>
<li><code>runtime_init</code>
<li><code>main_init</code>
<li><code>main_main</code>
<li><code>exit(0)</code>
<p>
</ul>
These are the initialization code and main program for the binary. The
<code>runtime·init</code> function in
<code>proc.go</code> spawns the goroutine to call
<code>gogc(1)</code> in a loop.
<p>
<a name="sec2"></a><h2>2. Labels</h2>
A <code>Gobuf</code> is similar to a
<em>Label</em>:
<p><ul style="list-style:none;">
<code><pre>
struct Gobuf
{
uintptr sp; // the actual label.
uintptr pc; // the actual label.
G* g; // the actual label.
void* ctxt; // aux context pointer.
uintreg ret; // return value
uintptr lr; // link register used in ARM.
};
</pre></code>
</ul><p>
The main operations are:
<ul>
<li><code>runtime·gosave</code>, i.e., set label. Clears
<code>ctxt</code> and <code>ret</code>.
<li><code>runtime·gogo</code>, i.e., goto label. Sets
<code>ctxt</code> in <code>DX</code>, used by
<code>runtime·morestack</code> and returns
<code>ret</code> after the actual jump to the label.
<p>
</ul>
Here, <code>pc</code>, <code>sp</code>, and
<code>g</code> are the actual context saved and restored.
That is the label.
<p>
The field <code>ctxt</code> is used to point to a frame in
the stack and seems to be the user context in other places. Seems to
be an auxiliary pointer but it is not clear (yet) how it is used. The
field <code>lr</code> seems to be the link register for
the ARM and is not used by our architecture.
<p>
<a name="sec3"></a><h2>3. Sleep/Wakeup</h2>
These are giant locks:
<p><ul style="list-style:none;">
<code><pre>
void runtime·stoptheworld(void);
void runtime·starttheworld(void);
extern uint32 runtime·worldsema;
</pre></code>
</ul><p>
used for example to dump all the stacks.
<p>
These are the usual locks with a user-level fast path:
<p><ul style="list-style:none;">
<code><pre>
void runtime·lock(Mutex*);
void runtime·unlock(Mutex*);
</pre></code>
<p>
</ul><p>
These are sleep/wakeup like structures, and timed out variants for
them with the usual conventions of at most one calling sleep and at
most one calling wakeup for it. Plus, there is a clear operation to
reset the thing for a further use:
<p><ul style="list-style:none;">
<code><pre>
void runtime·noteclear(Note*);
void runtime·notesleep(Note*);
void runtime·notewakeup(Note*);
bool runtime·notetsleep(Note*, int64); // false - timeout
bool runtime·notetsleepg(Note*, int64); // false - timeout
</pre></code>
<p>
</ul><p>
And these are futexes or semaphores (eg., on Darwin) to implement the
ones above:
<p><ul style="list-style:none;">
<code><pre>
uintptr runtime·semacreate(void);
int32 runtime·semasleep(int64);
void runtime·semawakeup(M*);
</pre></code>
<p>
</ul><p>
In many cases lock-free data structures are used; or at least parts of
them are handled lock-free by using atomic and CAS-like operations,
eg. to change statuses of processes and to link them to a list.
<p>
<a name="sec4"></a><h2>4. Scheduling</h2>
Most of the interesting code is in <code>proc.c</code>.
There are three central structures:
<dl>
<dt><em>G</em></dt><dd>
A goroutine.
<code>g</code> refers to the current and
<code>g0</code> is the idle process. This is a user-level
thread. <code>g</code> is a register on the ARM and a slot
in TLS everywhere else.
</dd>
<dt><em>M</em></dt><dd>
A machine, actually a UNIX
process. <code>m</code> refers to the current one. This is
a kernel-level thread that runs <em>Gs</em> or is idle or
is in a syscall.
</dd>
<dt><em>P</em></dt><dd>
A processor, actually a per
<code>GOMAXPROCS</code> sched.
<p>
</dd>
</dl>
The idea is that <em>M</em> takes a <em>P</em>
when it must run <em>Gs</em>. <em>Ps</em> were
introduced to do job stealing.
<p>
The interesting bits of <em>G</em> are:
<p><ul style="list-style:none;">
<code><pre>
struct G
{
Stack stack; // [stack.lo, stack.hi).
uintptr stackguard0; // stack.lo+StackGuard or StackPreempt
uintptr stackguard1; // stack.lo+StackGuard on g0 or ~0 on others
Panic* panic; // innermost panic - offset known to liblink
Defer* defer; // innermost defer
Gobuf sched;
uintptr syscallsp; // if status==Gsyscall, syscallsp = sched.sp to use during gc
uintptr syscallpc; // if status==Gsyscall, syscallpc = sched.pc to use during gc
void* param; // passed parameter on wakeup
int64 goid;
G* schedlink;
bool preempt; // preemption signal, dup of stackguard0 = StackPreempt
M* m; // for debuggers, but offset not hard-coded
M* lockedm;
SudoG* waiting; // sudog structures this G is waiting on
...
};
</pre></code>
<p>
</ul><p>
The interesting bits of <em>M</em> are:
<p><ul style="list-style:none;">
<code><pre>
struct M
{
G* g0; // goroutine with scheduling stack
G* gsignal; // signal-handling G
uintptr tls[4]; // thread-local storage (for x86 extern register)
void (*mstartfn)(void);
G* curg; // current running goroutine
G* caughtsig; // goroutine running during fatal signal
P* p; // attached P for executing Go code (nil if not executing Go code)
int32 mallocing,throwing,gcing;
int32 locks;
bool spinning; // M is out of work and is actively looking for work
bool blocked; // M is blocked on a Note
Note park;
M* alllink; // on allm
M* schedlink;
MCache* mcache;
G* lockedg;
uint32 locked; // tracking for LockOSThread
M* nextwaitm; // next M waiting for lock
uintptr waitsema; // semaphore for parking on locks
uint32 waitsemacount;
uint32 waitsemalock;
bool (*waitunlockf)(G*, void*);
void* waitlock;
uintptr scalararg[4]; // scalar argument/return for mcall
void* ptrarg[4]; // pointer argument/return for mcall
...
};
</pre></code>
</ul><p>
A value of <code>m->locks</code> greater than zero
prevents preemption and also prevents garbage colloection.
<p>
The interesting bits of <em>P</em> are:
<p><ul style="list-style:none;">
<code><pre>
struct P
{
Mutex lock;
uint32 status; // Pidle, Prunning, Psyscall, Pgcstop, Pdead
P* link;
uint32 schedtick; // incremented on every scheduler call
uint32 syscalltick; // incremented on every system call
M* m; // back-link to associated M (nil if idle)
MCache* mcache;
Defer* deferpool[5]; // pool of available Defers
// Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen.
uint64 goidcache;
uint64 goidcacheend;
// Queue of runnable goroutines.
uint32 runqhead;
uint32 runqtail;
G* runq[256];
// Available G's (status == Gdead)
G* gfree;
int32 gfreecnt;
};
</pre></code>
</ul><p>
This is a local scheduler plus cached structures per UNIX process used
to run Go code. And then there is a global scheduler structure that
also caches some structures.
<p>
<p><ul style="list-style:none;">
<code><pre>
struct SchedT
{
Mutex lock;
uint64 goidgen;
M* midle; // idle m's waiting for work
P* pidle; // idle P's
G* runqhead; // Global runnable queue.
Mutex gflock; // global cache go dead Gs
G* gfree;
uint32 gcwaiting; // gc is waiting to run
int32 stopwait;
Note stopnote;
...
};
</pre></code>
</ul><p>
There is a single <code>runtime·sched</code>, and
<code>runtime·sched.lock</code> protects it all.
<p>
<a name="sec4x1"></a><h3>4.1. Process creation</h3>
Code like <code>go fn()</code> is translated as a call to
<code>proc.c:2154: runtime·newproc</code> as shown in the
first section for initialization. This records the function and a
pointer to the <code>...</code> arguments yn
<code>g->m->ptrarg[]</code> and then calls
<code>newproc_m</code> using
<code>runtime·onM</code>.
<ul>
<li>There's a switch to <code>M</code>'s
<code>g0</code>
<li><code>newproc_m</code> runs there.
<li>And there's a switch back when done.
<p>
</ul>
Then <code>newproc_m</code> takes the function and
arguments from <code>m->ptrarg</code> and calls
<code>runtime·newproc1</code>:
<ul>
<li>This adds to <code>m->locks</code> to avoid
preemption and takes <em>P</em> from
<code>m->p</code>.
<li>A <em>G</em> is taken from the cache at
<em>P</em> or allocated (with
<code>StackMin</code> stack bytes).
<p>
</ul>
At this point <em>G</em> is
<code>Gdead</code>. Then arguments are copied into
<em>G</em>'s stack and it's prepared:
<ul>
<li>It's label <code>pc</code>,
<code>sp</code>, and <code>g</code> are set to
the function pointer and the new stack and <code>g</code>,
and then adjusted to pretend that such function did call
<code>gosave</code>, so we can
<code>gogo</code> (gotolabel) it. In this case, the label
<code>ctxt</code> is set to the
<code>FuncVal</code> value going. The return value is set
to call <code>goexit+PCQuantum</code> which is a call to
<code>goexit1</code>.
<li>Its state is changed to <code>Grunnable</code> (cas
op.)
<li>A new <code>id</code> is taken from the id cache at
<code>P</code> (perhaps refilling the cache).
<p>
</ul>
Now it's put in the scheduler queue by calling <code>runqput(p,
newg)</code>. This uses atomic load/store to put
<code>newg</code> at
<code>p->runq[tail]</code>, using
<code>p->runqtail</code> as a increasing counter for the
tail that is copied to <code>tail</code> and truncated as
an index for <code>runq</code>. This happens if
<code>p->runq</code> is not full.
<p>
If <code>p->runq</code> is full,
<code>runqputslow</code> is called to take half of the
local queue at <em>P</em> and place them at the global
<code>runtime·sched.runqhead/tail</code> queue while
holding the global scheduler lock. if the CAS operation to operate on
the local queue fails, the whole <code>runqput</code> is
retried.
<p>
<a name="sec4x2"></a><h3>4.2. Process termination</h3>
Performed by a call to
<ul>
<li><code>proc.c:1682: runtime·goexit1</code>, that does
an <code>mcall</code> to run
<code>goexit0</code> at <code>g0</code>.
<li><code>goexit0</code> (or any other mcall) never
returns and calls <code>gogo</code> to sched to somewhere
else, usually to <code>g->sched</code> to let it run
later. The <em>G</em> state is set to
<code>Gdead</code>. Then these are called:
<ul>
<li><code>proc.c:1598: dropg</code>, to release
<code>m->curg->m</code> and
<code>m->curg</code> (<code>g</code> is now
<code>g0</code>).
<li><code>proc.c:2259: gfput</code>, to put the dying
<em>G</em> at <code>p->gfree</code>, linked
through <code>g->schedlink</code>. If there are too many
free <em>G</em>s cached, they are moved to the global list
at <code>runtime·sched.gfree</code>.
<li><code>schedule</code>, runs one scheduler loop and
jumps to a new <em>G</em>.
<p>
</ul>
</ul>
Note that returning from the main function of a goroutine returns to
<ul>
<li><code>asm_amd64.s:2237: runtime·goexit</code>, which
calls <code>runtime·goexit1</code>.
<p>
</ul>
<a name="sec4x3"></a><h3>4.3. Context switches</h3>
Calls to <code>schedule</code> can be found at:
<ul>
<li><code>proc.c:661: mquiesce</code>, used to run code
at <code>g0</code> and then schedule back a
<em>G</em>.
<li><code>proc.c:868: mstart</code>, to jump to a new
<code>g0</code> for a new <em>M</em>.
<li><code>proc.c:1655: runtime·park_m</code>, used to
make <em>G</em> wait for something.
<li><code>proc.c:1675: runtime·gosched_m</code>, an
<code>mcall</code> from <code>Gosched</code>
at <code>proc.go</code>.
<li><code>proc.c:1718: goexit0</code>, to run
<code>g0</code> or other <em>G</em>s when
exiting.
<li><code>proc.c:2022: exitsyscall0</code>, called after
a system call at <code>g0</code>.
<li><code>malloc.go:482: gogc</code> calls
<code>Gosched</code> when the GC is done.
<li><code>mgc0.go:87: bgsweep</code> calls
<code>Gosched</code>.
<p>
</ul>
Plus, if <code>g->stackguard0</code> is
<code>StackPreempt</code>, then
<code>asm_amd64.s:824:runtime·stackcheck</code> calls
<code>morestack</code> and preemts if the goroutine seems
to be running user code (not runtime code), by calling
<code>gosched_m</code> like
<code>Gosched</code> does. I have not checked out if the
compiler inserts calls in loops that do not perform function calls
(which check the stack), and those might never be preempted; which
does not matter much for us now becase they would be bugs in the code
and not the normal case. The stack checks are inserted silently by the
linker unless <code>NOSPLIT</code> is indicated, and thus
normal function calls are a source of possible preemptions.
<p>
As a side-efect, a <code>NOSPLIT</code> function call is
not preemptible by this mechanism.
<p>
The code in <code>schedule</code> runs the scheduler:
<ul>
<li>There can be no <code>m->locks</code>, or it's a
bug.
<li>If <code>m->lockedg</code> then
<ul>
<li><code>proc.c:1287: stoplockedm</code> is called, to
stop the current <em>G</em> at <em>M</em>
until <em>G</em> can run again. This is done by lending
<code>m->p</code> to another <em>M</em>, if we
have a <em>P</em> using
<code>handoffp(releasep())</code>, and then calling
<code>runtime·notesleep(&g->m->park)</code> to sleep.
Later <code>acquirep(m->nextp)</code> is used to get a new
<em>P</em> before returning and...
<li><code>proc.c:1358: execute</code> is called for
<code>m->lockedg</code>.
<p>
</ul>
<li>If <code>runtime·sched.gcwaiting</code> then
<code>gcstopm</code> is called, which calls
<code>releasep</code> and <code>stopm</code>;
This happens when <code>stoptheworld</code> is called and,
once we are done <code>schedule</code> restarts again.
<p>
</ul>
At this point <code>schedule</code> picks a ready
<em>G</em>. Once in a while from the global pool, and
usually from <code>m->p</code>.
<p>
<ul>
<li><code>proc.c:3333: runqget</code> picks one
<em>G</em> from the queue using
<code>p->runqhead</code> and CAS.
<ul>
<li>When there is no <em>G</em> ready to run,
<code>proc.c:1381: findrunnable</code> tries to steal
calling <code>globrunqget</code> and,
<code>runtime·netpoll</code> (!!).
<li>Then if there's no work, <code>m->spinning</code> is
set and it tries to steal from others.
<li>When everything fails, it calls <code>stopm</code>
and blocks.
<p>
</ul>
</ul>
Once it gets a <em>G</em> to run:
<ul>
<li><code>proc.c:1358: execute</code> is called, which
is mostly a call to a goto-label:
<ul>
<li><code>asm_amd64.s:143: gogo</code>
<p>
</ul>
</ul>
<a name="sec5"></a><h2>5. System calls</h2>
All system calls call
<li><code>asm_darwin_amd64.s:19 syscall.Syscall</code>
or <code>Syscall6</code> that
<ul>
<li>calls <code>proc.c:1761
runtime·reentersyscall</code> (from
<code>runtime·entersyscall</code>)
<p><ul style="list-style:none;">
to prepare for a system call
</ul><p>
<ul>
<li>This increments <code>m->locks</code> to avoid
preemption, and tricks <code>g->stackguard0</code> to make
sure the no split-stack happens.
<li>Sets <em>G</em> status to
<code>Gsyscall</code>
<li>Saves the PC and SP in <code>g->sched</code>.
<li>Releases <code>p->m</code> and
<code>m->mcache</code>
<li>Sets <em>P</em> status to
<code>Psyscall</code>
<li>Calls <code>entersyscall_gcwait</code> when
<code>runtime·sched.gcwaiting</code>
<li>Decrements <code>m->locks</code> before proceding
further.
</ul>
<li>moves parameters into registers and executes
<li>executes <code>SYSCALL</code>, and, upon return from
the system call, calls
<li>calls <code>proc.c:1893 runtime·exitsyscall</code>
calls directly
<ul>
<li><code>exitsyscallfast</code> to either re-acquire
the last <em>P</em> or or get another idle
<em>P</em>.
<li>or runs <code>exitsyscall0</code> as an
<code>mcall</code> to run the scheduler and then return to
continue when the scheduler returns and <em>G</em> can run
again.
<ul>
<li>This calls <code>pidleget</code> and then
<code>acquirep</code> and <code>execute</code>
or <code>stopm</code> and
<code>schedule</code>, depending.
</ul>
</ul>
<li>and returns to the caller.
<p>
</ul>
There are other wrappers for system calls but in the end they call the
above entry points.
<p>
Two other functions, <code>syscall·RawSyscall</code> and
<code>·RawSyscall6</code> are wrappers that issue the
system call without doing anything: no calls to
<code>runtime·entersyscall</code> and no calls to
<code>runtime·exitsyscall</code>. I don't know why they
are not called from the entry points above.
<p>
<a name="sec6"></a><h2>6. Signals</h2>
There is a global <code>runtime·sigtab[sig]</code> table
with entries of this data type:
<p><ul style="list-style:none;">
<code><pre>
struct SigTab
{
int32 flags;
int8 *name;
};
enum
{
SigNotify = 1<<0, // let signal.Notify have signal, even if from kernel
SigKill = 1<<1, // if signal.Notify doesn't take it, exit quietly
SigThrow = 1<<2, // if signal.Notify doesn't take it, exit loudly
SigPanic = 1<<3, // if the signal is from the kernel, panic
SigDefault = 1<<4, // if the signal isn't explicitly requested, don't monitor it
SigHandling = 1<<5, // our signal handler is registered
SigIgnored = 1<<6, // the signal was ignored before we registered for it
SigGoExit = 1<<7, // cause all runtime procs to exit (only used on Plan 9).
};
</pre></code>
<p>
</ul><p>
Signals are handled in their own signal stack. When a new
<em>M</em> is initialized,
<ul>
<li><code>runtime·minit</code> is called, eg., for
Darwin, it calls
<ul>
<li><code>sys_darwin_amd64.s:265:
runtime·sigaltstack</code> to set the signal stack context, and
<li><code>sys_darwin_386.s:218:
runtime·sigprocmask</code> to set the signal mask to none.
<p>
</ul>
</ul>
Later on, when <code>dropm</code> is called to release an
<em>M</em>,
<ul>
<li><code>runtime·unminit</code> is called and it calls
<code>runtime·signalstack</code> to unset the signal
stack.
<p>
</ul>
The <code>signal</code> package
<code>init</code> function:
<ul>
<li>calls <code>runtime·signal_enable</code>, which when
called for the first time sets <code>sig.inuse</code> and
clears the <code>sig.note</code> note, to prepare for
handling signals.
<p>
<li>loops calling
<code>process(syscall.Signal(signal_recv()))</code>. Here,
<ul>
<li><code>sig.s:21 signal_recv</code> is just a jump to
<li><code>sigqueue.go:91 runtime·signal_recv</code>,
which returns a signal number when it's posted by the OS.
<p>
</ul>
</ul>
When <code>signal.Notify</code> is called to install a
handler:
<ul>
<li><code>signal.go:49 Notify</code> records in a
<code>handlers</code> table for the signal given or for
all the signals, and calls
<ul>
<li><code>signal_unix.go:47 signal·enableSignal</code>,
which is just a call to
<li><code>sigqueue.go:128 runtime·signal_enable</code>.
This sets the signal in <code>sig.wanted</code> and calls
<code>sigenable_go</code> which is just a call, using
<code>onM</code> to this:
<ul>
<li><code>signal.c:8 runtime·sigenable_m</code> enables
a signal calling with the signal number kept at
<code>m->scalararg[0]</code>. It is just a call to:
<li><code>signal_unix.c:44 runtime·sigenable</code>
enables a signal, making a call to the next with
<code>runtime·sighandler</code> as the handler function.
<ul>
<li><code>os_darwin.c:519: runtime·setsig</code> calls
<code>runtime·sigaction</code> specifying
<code>runtime·sigtramp</code> as the handler and supplying
the handler function (runtime·sighandler) in the
<em>sigaction</em> structure.
<p>
</ul>
</ul>
</ul>
</ul>
Later, when a signal arrives:
<ul>
<li><code>sys_darwin_386.s:240 runtime·sigtramp</code>
<ul>
<li>saves <code>g</code> and sets
<code>m->gsignal</code> as the current
<em>G</em>, and then calls the actual handler:
<ul>
<li><code>signal_amd64x.c:44 runtime·sighandler</code>
looks into the global <code>runtime·sigtab</code> and
might panic if not handled or deliver the signal by calling the next.
<ul>
<li><code>sigqueue.go:48 runtime·sigsend</code> is
called with a signal number to send the signal to receivers. The
signal is queued in a global if there is no receiver ready but the
signal is wanted and is not already in the queue.
<p>
</ul>
</ul>
</ul>
</ul>
From here, the loop started in the
<code>singal·init</code> function would send the signal to
the users channel with a non-blocking send. That is, if there is not
enough buffering in the channel signals are lost; but that's ok.
<p>
<a name="sec7"></a><h2>7. Memory allocation</h2>
Memory is allocated by calls to <code>new</code>. The
implementation for this builtin is:
<ul>
<li><code>malloc.go:348 newobject</code>, which calls to
<code>mallocgc</code> using
<code>flagNoScan</code> as a flag if the type is marked as
having no pointers.
<ul>
<li><code>malloc.go:46 mallocgc</code> is the main entry
point for memory allocation.
<p>
</ul>
</ul>
Slices are created by calling <code>malloc.go:357
newarray</code>, which does the same. Raw memory is allocated by
calls to <code>rawmem</code> which also calls
<code>mallocgc</code> with flags
<code>flagNoScan</code> and
<code>flagNoZero</code>; i.e., that's almost a direct
<code>malloc</code> call, but for the GC.
<p>
There are several allocation classes. Sizes are recorded in
<code>runtime·class_to_size[]</code>. Alignments are:
<ul>
<li>8, for sizes under 16 bytes and sizes that are not a power of 2.
<li>16, for sizes under 128 bytes,
<li>size/8, for sizes under 2KiB,
<li>256, for sizes starting at 2KiB.
</ul>
Regarding sizes:
<ul>
<li>Sizes under <code>MaxSmallSize</code> (32KiB), use
the larger size that keeps the same number of objects in each page.
They are allocated from a per-<em>P</em> cache.
<li>Sizes starting at 32KiB are rounded to
<code>PageSize</code>. They are allocated from the global
heap.
<p>
</ul>
The allocator used depends on the allocation:
<ul>
<li>Objects with no pointers and less than 16 bytes
(<code>maxTinySize</code>) are allocated within a single
allocation and share it. This is done in the <em>P</em>
cache. The allocation is released when all such objects are collected.
This is done in the <em>P</em> cache.
<li>Objects up to 1024-8 bytes go into a size class rounded to
multiples of 8 bytes.
<li>Objects up to 32KiB are rounded to multiples of 128 bytes. This
is done in the <em>P</em> cache.
<li>Objects larger than 32KiB are allocated on the heap by calling
<code>largeAlloc_m</code> <code>onM</code>.
<p>
</ul>
The heap operates using pages, but smaller allocators up in the
hierarchy uses bytes. This is how it works:
<ul>
<li>The tiny allocations use the <code>tiny</code>
allocation from the <code>MCache</code> structure at
<code>p->mcache</code>. There is one per
<em>P</em> and it requires no locks:
<code><pre>
struct MCache
{
byte* tiny;
uintptr tinysize;
// The rest is not accessed on every malloc.
MSpan* alloc[NumSizeClasses]; // spans to allocate from
StackFreeList stackcache[NumStackOrders];
SudoG* sudogcache;
...
};
</pre></code>
<p>
When <code>tiny</code> is exhausted, a new one is taken
from <code>Mcache.alloc</code>, and, if that is exhausted,
a new one is allocated by a call to
<code>runtime·MCache_Refill</code>, using
<code>onM</code>. This asks
<code>runtime·MCentral_CacheSpan</code> to allocate a new
span and places it into <code>Mache.alloc</code>.
<p>
<li>The allocations with pointers or starting at 16 bytes, try first
to use <code>p->cache.alloc</code> to get an allocation of
the right size. If this fails, <code>MCache_Refill</code>
is called as before, <code>onM</code>.
<p>
<li>Large allocations starting at 32KiB call
<code>largeAlloc_m</code> <code>onM</code>.
<ul>
<li><code>malloc.c:372: runtime·largeAlloc_m</code>
rounds the size to a multiple of <code>PageSize</code> and
calls:
<ul>
<li><code>mheap.c:231: runtime·MHeap_Alloc</code> calls
<code>mheap_alloc</code> directly if it's a call made by
<code>g0</code>, or <code>mcall</code>s it.
<ul>
<li><code>mheap.c:171 mheap_alloc</code> allocates n
memory pages.
</ul>
<li><code>mgc0.c:1815: runtime·markspan</code> is called
to tell the GC.
<p>
</ul>
</ul>
</ul>
Unless the allocation is <code>flagNoScan</code>, the GC
bitmap is updated by looking at the type given to
<code>mallocgc</code> to record which words are pointers.
Also, the unused part at the end of the actual allocation is marked in
the GC bitmap as dead.
<p>
Looking at the <code>MHeap</code> now, for allocations
under 1Mbyte (for 4KiB pages), there is a list of allocation with
exactly that page size. For larger allocations there is a final large
list used.
<p>
<hr><p>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-72664662824276557982015-05-22T10:39:00.000-07:002015-05-22T10:39:01.840-07:00When "cd" has no argumentsIn a <a href="http://syssoftware.blogspot.com.es/2015/05/commands-and-channels-in-clive-sam-is.html">previous post</a> I wrote about commands and channels in Clive. Since then, things have evolved a bit I think it's worth a post.<br />
<br />
The <span style="font-family: Courier New, Courier, monospace;"><b>cd</b></span> command, to change the current directory, no longer has a file name argument. Instead, it receives the directory entry to bet set as the new dot from the standard input, like everybody else does to operate on files!. That is:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b> /tmp | cd</b></span><br />
<br />
changes dot to /tmp.<br />
<br />
As it can be seen, the first component of the pipe is not a command, but a file name. The Clive shell, <i>Ql</i>, is departing even more from the venerable UNIX <i>sh</i>. Ql commands correspond to streams as explained in the <a href="http://syssoftware.blogspot.com.es/2015/05/commands-and-channels-in-clive-sam-is.html">previous post</a>. Since I wrote that post, Ql evolved as well.<br />
<br />
Pipe-lines in Ql can be as simple as a set of file names (that is, pairs of a UNIX-like file name and predicate to find files of interest). One example, list go source files:<br />
<span style="font-family: Courier New, Courier, monospace;"><b> % ,~*.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> Q/Q.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> bltin.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> lex.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> nd.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ql/ql.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ql.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ql_test.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ...</b></span><br />
<br />
(empty name, which defaults to "<span style="font-family: Courier New, Courier, monospace;"><b>.</b></span>", and <span style="font-family: Courier New, Courier, monospace;"><b>~*.go</b></span> as predicate to locate Go files).<br />
<br />
Further commands may be added to pipe-lines to process the directory entries found, like in the first example, or in this one:<br />
<span style="font-family: Courier New, Courier, monospace;"><b> % ,~*.go | pf -l</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> --rw-r--r-- 2.1k /zx/sys/src/clive/app/ql/Q/Q.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> --rw-r--r-- 6.3k /zx/sys/src/clive/app/ql/bltin.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> ...</b></span><br />
<div>
<br /></div>
<div>
For those cases when we really want to execute a command that takes no files as input we can just pipe nothing, so Ql does not assumes that the first series of names select files:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> % |date</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> Fri May 22 19:31:06 CEST 2015</b></span></div>
</div>
<div>
<br /></div>
<div>
Doing a recursive grep is now delightful:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> % ,~*go |> gr '"Q"'</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> Q/Q.go:27: <span class="Apple-tab-span" style="white-space: pre;"> </span>os.Args[0] = "Q"</b></span></div>
</div>
<div>
<br /></div>
<div>
This used "<span style="font-family: Courier New, Courier, monospace;"><b>|></b></span>" instead of "|" to ask Ql to retrieve file data and not just directory entries.</div>
<div>
<br /></div>
<div>
More on Ql soon.</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-4488743937342930722015-05-05T13:09:00.002-07:002015-05-05T13:09:26.237-07:00Commands and channels in Clive: Sam is gone.Recently I removed the port of the venerable Sam command language in Clive.<br />
This command language can do things like iterate for all portions of text matching addresses indicated by regular expressions and then, for example, for each portion matching another expression, apply a command or loop in some other way.<br />
<br />
As part of the Clive application framework (still WIP), I ported most of the commands to the new standard setup: commands have a series of I/O channels that forward <span style="font-family: Courier New, Courier, monospace;"><b>interface{}</b></span> data. When going outside of the Clive world, only <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span> is actually written into off-limits channels.<br />
<br />
By convention, commands expect data to be just <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span> messages, forwarded through std. IO channels (similar to file descriptors in UNIX, but using -modified- Go channels).<br />
<br />
Now, the interesting thing is the result for these conventions:<br />
<br />
<ul>
<li>All commands forward all messages they don't understand. </li>
<li>Commands finding files send not just file contents, but send a <span style="font-family: Courier New, Courier, monospace;"><b>zx.Dir</b></span> for each file found before actually sending the file data to standard output (more on this later in this post).</li>
<li>Only <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span> messages are actually considered data, all other types can be used to let friend commands talk without disrupting others in the command pipeline.</li>
</ul>
<div>
To illustrate the consequences, we can run this pipeline in Clive:</div>
<div>
<br /></div>
<br />
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>gf /tmp/echo,1 | gx '(^func[^\n]*\n)' '(^}\n)' | gg Run | </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>gp '~m.*&1' | trex -u | pf -ws,</b></span></div>
<div>
<br /></div>
<div>
This is just an example, I'll explain the bits and pieces; but the interesting thing is how things can work together so in Clive we can use separate commands to do what in UNIX we would have to do in a single process, or would be a lot harder to split into separate commands.</div>
<div>
<br /></div>
<div>
The first command, <b style="font-family: 'Courier New', Courier, monospace;">gf /tmp/echo,1 </b> issues a <span style="font-family: Courier New, Courier, monospace;"><b>Find</b></span> request to the ZX file system and sends to its std. output channel a stream of data for matching files. In this case, we asked for files under <span style="font-family: Courier New, Courier, monospace;"><b>/tmp/echo</b></span> with depth not greater than 1. Here, filtering of interesting files happens at the file server, which sends to the client machine a stream of directory entries for matching files plus data for each one (after its directory entry).</div>
<div>
<br /></div>
<div>
Now, <span style="font-family: Courier New, Courier, monospace;"><b>gx</b></span> is mostly a grep command. In this case, it simply sends to its output all text enclosed in text matched by the two given regexps. It can be used in other ways; the one in the example would report as a <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span> in the output each function or method defined in the go source.</div>
<div>
The funny thing is that <span style="font-family: Courier New, Courier, monospace;"><b>gx</b></span> (as called) also writes unmatched input to the output, but uses a string data type instead. Furthermore, it posts addresses (text lines and rune ranges) to its output, should the user want to print that.</div>
<div>
<br /></div>
<div>
The next command, <span style="font-family: Courier New, Courier, monospace;"><b>gg</b></span>, receives thus the entire text for the files found, but only the portions matched by the previous <span style="font-family: Courier New, Courier, monospace;"><b>gx</b></span> are <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span>, and thus all other data is forwarded but otherwise ignored. This command sends to the output those messages containing the given expression. At this point, only functions containing <span style="font-family: Courier New, Courier, monospace;"><b>Run</b></span> are in the output as <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span>.</div>
<div>
<br /></div>
<div>
To make the example more fun, <span style="font-family: Courier New, Courier, monospace;"><b>gp</b></span> greps directory entries in the stream and does not forward any file to the output unless its directory entry matches the predicate given. We could have given it to <span style="font-family: Courier New, Courier, monospace;"><b>gf</b></span> instead, but this is more fun. At this point, only functions containing Run in files with names starting with m and with a depth no greater than 1 are in the output. </div>
<div>
<br /></div>
<div>
To have more fun, <span style="font-family: Courier New, Courier, monospace;"><b>trex</b></span> translates the input text to uppercase. Note that all unmatched data is still forwarded (but not as <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span>), because we didn't suppress it from the stream in any of the previous commands.</div>
<div>
<br /></div>
<div>
Finally, <span style="font-family: Courier New, Courier, monospace;"><b>pf -ws</b></span> is a funny command that, one file at a time, reads all the file data into memory and, once the data is completed, writes the data back to the file (using its Dir entry as found in the stream to locate the file to write). We asked it to write the files and to include also data sent for them in string messages (and not just the "official" <span style="font-family: Courier New, Courier, monospace;"><b>[]byte</b></span> messages). Thus, this is like a "save the edited files" command in Sam.</div>
<div>
<br /></div>
<div>
All the files processed by previous commands in the stream are written to disk at this point. Because the command buffers in memory the files going through the stream, we don't have the "<span style="font-family: Courier New, Courier, monospace;"><b>cat f > f</b></span>" effect of truncating the files by mistake.</div>
<div>
<br /></div>
<div>
The pipe-line may be silly, but it shows how bits and pieces fit together in Clive. I was not able to do these things in any other system I used before. Clive is fun. Go is fun.</div>
<div>
<br /></div>
<div>
The output of the pipe is also fun (using <span style="font-family: Courier New, Courier, monospace;"><b>pf -x</b></span> instead):</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>/tmp/echo/mecho.go:#174,#716:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>FUNC RUN() {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> X := &XCMD{CTX: APP.APPCTX()}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> X.FLAGS = OPT.NEW("{ARG}")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> X.NEWFLAG("D", "DEBUG", &X.DEBUG)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> X.NEWFLAG("G", "DON'T ADD A FINAL NEWLINE", &X.NFLAG)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> ARGS, ERR := X.PARSE(X.ARGS)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> IF ERR != NIL {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> X.USAGE()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> APP.EXITS("USAGE")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> VAR B BYTES.BUFFER</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> FOR I, ARG := RANGE ARGS {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> B.WRITESTRING(ARG)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> IF I < LEN(ARGS)-1 {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> B.WRITESTRING(" ")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> IF !X.NFLAG {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> B.WRITESTRING("\N")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> OUT := APP.OUT()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> OK := OUT <- B.BYTES()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> IF !OK {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> APP.WARN("STDOUT: %S", CERROR(OUT))</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> APP.EXITS(CERROR(OUT))</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> }</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> APP.EXITS(NIL)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
I admit that Clive is even more user-friendly than UNIX ;)</div>
<div>
But I also admit I like that...</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-49417008948323334682015-03-26T05:03:00.001-07:002015-03-26T05:12:31.045-07:00Nested control requests in Clive ZX file treesIn a <a href="http://syssoftware.blogspot.com.es/2015/03/file-system-stacks-in-clive.html">previous post</a> I wrote about ZX file system stacks in Clive. Now I look into their control interfaces, which exhibit a nice feature shown here.<br />
<br />
When a ZX file tree operates by relying on another ZX tree deeper in the stack, it adapts its control interface to behave as a facade for it.<br />
<div>
<br /></div>
<div>
<div>
For example, the <span style="font-family: Courier New, Courier, monospace;"><b>/Ctl</b></span> file of a ZX file tree can be user to see the status of the debug flags, to see who is using the file tree (in the cases of trees served to the network) and to see usage statistics for the system.</div>
</div>
<div>
<br /></div>
<div>
<div>
This is the result of reading <span style="font-family: Courier New, Courier, monospace;"><b>/Ctl</b></span> on a stack made out of a CFS that uses a MFS as a cache, and a RFS as the remote tree. The RFS was connected to a remote server exporting a CFS that uses a MFS cache to serve a LFS. What CFS, MFS, RFS, and LFS means is not really relevant. They are just ZX file servers that rely on underlying file servers to do its job (but, CFS is a caching file sever, MFS is a RAM FS, RFS is a remote file server, and LFS is a local file server).</div>
<div>
<br /></div>
<div>
I have removed some of the lines when they do not help to further illustrate the case.</div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>> cat /zx/Ctl</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>cfs:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fdebug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>vdebug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>noperm off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>stat<span class="Apple-tab-span" style="white-space: pre;"> </span>8220 calls 252 errs 16440 msgs 0 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>bgn: min 4.911µs<span class="Apple-tab-span" style="white-space: pre;"> </span>avg 625.628µs<span class="Apple-tab-span" style="white-space: pre;"> </span>max 1.333559776s</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>end: min 5.045µs<span class="Apple-tab-span" style="white-space: pre;"> </span>avg 625.722µs<span class="Apple-tab-span" style="white-space: pre;"> </span>max 1.333559891s</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>get<span class="Apple-tab-span" style="white-space: pre;"> </span>3147 calls 0 errs 16501 msgs 38660497 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>bgn: min 54.389µs<span class="Apple-tab-span" style="white-space: pre;"> </span>avg 2.657265ms<span class="Apple-tab-span" style="white-space: pre;"> </span>max 856.808309ms</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>end: min 63.349µs<span class="Apple-tab-span" style="white-space: pre;"> </span>avg 5.141675ms<span class="Apple-tab-span" style="white-space: pre;"> </span>max 856.841591ms</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>cmfs:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>debug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>noperm off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>stat<span class="Apple-tab-span" style="white-space: pre;"> </span>78449 calls 2721 errs 156898 msgs 0 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: Courier New, Courier, monospace;"><b> </b></span></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>lsub:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>user<span class="Apple-tab-span" style="white-space: pre;"> </span>rfs:193.147.15.27:65348 nemo as nemo on 2015-03-26</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>user<span class="Apple-tab-span" style="white-space: pre;"> </span>rfs:193.147.15.27:65349 nemo as nemo on 2015-03-26</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>ldebug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>rdebug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>stat<span class="Apple-tab-span" style="white-space: pre;"> </span>2656 calls 0 errs 5312 msgs 0 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>mfs:lsub:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>debug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>noperm off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>stat<span class="Apple-tab-span" style="white-space: pre;"> </span>47588 calls 2487 errs 95176 msgs 0 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: Courier New, Courier, monospace;"><b> </b></span></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>lsub:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>debug off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>rdonly off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>noperm off</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>stat<span class="Apple-tab-span" style="white-space: pre;"> </span>2854 calls 0 errs 5708 msgs 0 bytes</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span></div>
</div>
<div>
<br /></div>
<div>
<div>
Now, note the lines starting with <span style="font-family: Courier New, Courier, monospace;"><b>cfs:</b></span>, <span style="font-family: Courier New, Courier, monospace;"><b>cmfs:</b></span>, and <span style="font-family: Courier New, Courier, monospace;"><b>mfs:lsub:</b></span>. Only the first section comes from the top-level CFS. Remaining lines come from such server reading the <span style="font-family: Courier New, Courier, monospace;"><b>/Ctl</b></span> file of the underlying ZX file tree and appending it to the data served to the user; and so on as we go down on the stack.</div>
<div>
Thus, the user may inspect the ZX stack as deep as it wants.</div>
</div>
<div>
<br /></div>
<div>
<div>
In the same way, we can set debug on for CFS by executing</div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>> echo debug on > /zx/Ctl</b></span></div>
</div>
<div>
<br /></div>
<div>
<div>
The ZX file server understands writes to the control file as textual commands</div>
<div>
to perform control requests.</div>
</div>
<div>
<br /></div>
<div>
<div>
But we can also set debug on the third tree in the stack:</div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>> echo pass pass debug on > /zx/Ctl</b></span></div>
</div>
<div>
<br /></div>
<div>
<div>
When a control request starts with <span style="font-family: Courier New, Courier, monospace;"><b>pass</b></span>, this word is removed from the request</div>
<div>
and the server writes what remains of the control requests to the next tree in the stack.</div>
<div>
This makes the control interface for all the trees in the stack available to the final user.</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-74573874428249311492015-03-24T03:53:00.000-07:002015-03-24T03:53:42.837-07:00Go and unix traditions are a good matchRecently I had start writing a paper on Clive's ZX (more on this soon) and had to add citations in it. I'm writing the paper using <a href="http://lsub.org/sys/man/1/wr.html">wr(1)</a>, the <a href="http://lsub.org/ls/clive.html">Clive</a> text formatter used to write manual pages, papers, and everything.<br />
<br />
Because we had the entire set of bibliography entries from Plan 9 and Plan B in troff's refer format, I though <i>wr</i> could be teach to use them.<br />
<br />
Being short in time, I added a new <a href="http://golang.org/">Go</a> package to <a href="http://lsub.org/ls/clive.html">Clive</a> that eats all the entries in the lsub refer bib directory, that have a format like<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>Article in conference proceedings</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%A<span class="Apple-tab-span" style="white-space: pre;"> </span>M. Bishop</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%A<span class="Apple-tab-span" style="white-space: pre;"> </span>L. Snyder</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%T<span class="Apple-tab-span" style="white-space: pre;"> </span>The Transfer of Information and Authority</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>in<span class="Apple-tab-span" style="white-space: pre;"> </span>a Protection System</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%J<span class="Apple-tab-span" style="white-space: pre;"> </span>Proceedings of the 7th SOSP</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%P<span class="Apple-tab-span" style="white-space: pre;"> </span>45-54</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>%D<span class="Apple-tab-span" style="white-space: pre;"> </span>1979</b></span><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span><br />
<div>
<br /></div>
<div>
The <i>wr/refs</i> package exploits Go maps, strings, and slices to record a map of words to bib entries, which are also maps from keys (author, title, etc.) to strings. As easy as this:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>// A reference maps from the key (eg. 'A') to values (eg. authors)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>type Ref {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>Keys map[rune][]string</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>// A bib maps from words found in references to references</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>type Bib {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>refs map[string] map[*Ref]bool</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>all []*Ref</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<br /></div>
</div>
<div>
Note how easy is in Go to create sets of arbitrary types by using a map of that type to booleans.</div>
<div>
Writing the code for these types feels very much like writing lisp.</div>
<div>
<br /></div>
<div>
Citing now in <span style="font-family: inherit;"><i>wr</i></span> is as easy as writing <span style="font-family: Courier New, Courier, monospace;"><b>[bib: plan 9 networks]</b></span> in the source text.</div>
<div>
<i>Wr</i> simply takes all the words in the citation and looks them up in the Go map to locate entries matching all the words. And that's all that has to be done, the result looks like <span style="font-family: Courier New, Courier, monospace;"><b>[1]</b></span> in the text, and the reference is added at the end of the output as shown here.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>REFERENCES</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>1. The organization of networks in Plan 9. D. Presotto, P. Winterbottom. USENIX </b></span><b style="font-family: 'Courier New', Courier, monospace;">Association. Proceedings of the Winter 1993 USENIX Conference. 1993.</b></div>
</div>
<div>
<br /></div>
<div>
<div>
We can now easily cite in manual pages, web pages, papers in PDF, ...</div>
</div>
<div>
<br /></div>
<div>
<div>
The UNIX tradition of using simple text files with line oriented format and using Go code to process them is a very powerful combination. I'm still amazed.</div>
<div>
<br /></div>
<div>
</div>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-30928687329033511172015-03-19T11:28:00.000-07:002015-03-19T11:28:09.258-07:00Limiting the number of concurrent processesIn some cases, a task can be easily split into multiple concurrent child tasks, but it is not a good idea to spawn each of of the child tasks as a concurrent process.<br />
<br />
For example, some commands in <a href="http://lsub.org/ls/clive.html">Clive</a> go through a tree to synchronise it, or to poll each node for changes, etc. At each node in the tree, it is clear that the children can be processed concurrently. This reduces the total latency for the entire process. However, doing it this way would place a high load on the server at the other end of the line, and also on the line itself, and it might not be a good idea.<br />
<br />
In <a href="http://golang.org/">Go</a> it is fairly trivial to place a limit on the maximum number of concurrent tasks kept but still exploit concurrency up to that limit. The technique shown here is not new and, in fact, has been in use for decades, even in C for Plan 9, and probably in many other languages and by many other authors. Nevertheless, I think this tiny bit of code might help those that never saw it before.<br />
<br />
The idea is to send functions through a channel and keep a fixed set of workers at the other end of the channel. All of them receive a function, call it, and loop again. The number of workers places the limit on the number of concurrent tasks running at the same time.<br />
<br />
This creates the pool to run tasks:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>func NewPool(n int) *Pool {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>p := &Pool{calls: make(chan call)}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := 0; i < n; i++ {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>go p.worker(i)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>return p</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span><br />
<div>
<br /></div>
<div>
A worker is simply</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>func (p* Pool) worker(id int) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for c := range p.calls {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>c.fn()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>c.donec <- true</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
And we can run functions using this method:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>func (p *Pool) Go(donec chan bool, fn func()) chan bool {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if donec == nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>donec = make(chan bool, 1)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>p.calls <- call{fn: fn, donec: donec}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>return donec</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
<div>
<br /></div>
</div>
<div>
Now the thing is fairly easy to use. Go <i>closures</i> make the interface of the previous method powerful enough to do anything and accept/return any input/output values.</div>
<div>
<br /></div>
<div>
For example:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// Create a pool of 5 workers</b></span></div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span></b><span style="font-family: Courier New, Courier, monospace;"><b>p := NewPool(5) </b></span><div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// Run 20 functions in the pool</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dc := make(chan bool, 20)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := 0; i < 20; i++ {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>p.Go(dc, func() {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf("running this...\n")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>})</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// Wait until all of them are done</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := 0; i < 20; i++ {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span><-dc</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
</div>
<div>
<br /></div>
<div>
The testing code in this package plots a nice diagram of the execution, as a side effect of testing.</div>
<div>
This is one such plot, time on the X axis and each task on the Y axis. </div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>trace: abcdeAfBCDEghijFkGHIJlmnoKpLMNOqrstPQRST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>+----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +--------+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +--------+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +-----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +--------+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +----+ </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> +----+</b></span></div>
</div>
<div>
<br /></div>
<div>
The tests run fake jobs like this one:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>func fakefn(r rune, t time.Duration) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>n := atomic.AddInt32(&ncalls, 1)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>calls[n-1] = r</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf("fake call %c\n", r)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>time.Sleep(t)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>r = unicode.ToUpper(r)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>n = atomic.AddInt32(&ncalls, 1)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>calls[n-1] = r</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
They record a trace of the sequence of calls in a string like</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>abcdeAfBCDEghijFkGHIJlmnoKpLMNOqrstPQRST</b></span></div>
<div>
<br /></div>
<div>
where lower-case indicate the start of a task and upper-case its end.</div>
<div>
Thus, checking that the scheduling is as expected for the test is easy as is producing a plot for fun.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>func plot(trz string, ncalls int) {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>c := 'a'</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf("trace: %s\n", trz)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>ch := map[bool]string{true: "-", false: " "}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := 0; i < ncalls; i++ {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>st := c+rune(i)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>end := unicode.ToUpper(st)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>sted := false</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for _, ev := range trz {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>switch ev {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case st, end:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf("+")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>sted = !sted</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>default:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf(ch[sted])</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dprintf("\n")</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-9821194778721399472015-03-10T11:34:00.000-07:002015-03-10T11:36:55.381-07:00File System Stacks in CliveIn clive, The different file systems and file system tools can be combined and used together to provide features like file
access, backups, history dumps, and performance measurement. For example, this is a non-trivial stack out of the multiple ones we use:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT1svlQ5i4Er0c27DwX0qS_VXdPHQWIUsKEikatiMa8QJhNkDMFyDLdfa6Fehpobl5lOBo2jORaV83xJx8XxlYIpV_-iuUjG2pEqJAr3PV06Bdrg-VGCuWm9RH2LydCV9mF8MWiSCT0TRI/s1600/zxstack.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhT1svlQ5i4Er0c27DwX0qS_VXdPHQWIUsKEikatiMa8QJhNkDMFyDLdfa6Fehpobl5lOBo2jORaV83xJx8XxlYIpV_-iuUjG2pEqJAr3PV06Bdrg-VGCuWm9RH2LydCV9mF8MWiSCT0TRI/s1600/zxstack.tiff" height="164" width="640" /></a></div>
<br />
This stack traces calls made to a caching file system that relies on a local file system to cache a remote file system (wait, there is more), the remote file system serves a caching file file system that relies on a local memory-only file system to cache a local file system (at the remote machine). Confusing, isn't it? But that is the nice thing, that you can build complex stacks out of simple file system trees.<br />
<br />
This is an example of code:<br />
<br />
<br />
<div class="page" title="Page 5">
<div class="layoutArea">
<div class="column">
<pre><span style="font-family: Courier; font-size: 8pt;"> </span><span style="font-family: 'Courier'; font-size: 8.000000pt;">zx1, err := lfs.New("a tree", "/foo", lfs.RW)
zx2, err := mfs.New("a cache")
tr1 := trfs.New(zx1)
tr2 := trfs.New(zx2)
</span></pre>
<pre><span style="font-family: 'Courier'; font-size: 8.000000pt;"> cfs, err := cfs.New("a cached tree", tr1, tr2, cfs.RW)
err := <- cfs.Mkdir("/a/b", zx.Dir{"mode:": "0775"})
</span></pre>
<pre><div style="font-family: Times; white-space: normal;">
</div>
<div style="font-family: Times; white-space: normal;">
This code builds a different stack: Creates a ZX tree for an underlying UNIX tree at /foo, and then a ram-only file tree, a tree that traces calls to the first, one that traces calls to the second, and a caching file tree that relies on the two tracing trees. Now we can trace calls to a cached file tree and see the calls to each one of the trees (the cache and the cached ones).</div>
<div style="font-family: Times; white-space: normal;">
</div>
<div style="font-family: Times; white-space: normal;">
There is yet another draft of how stacks work in Clive in the <a href="http://lsub.org/export/clivefsys.pdf">clive file system stacks TR</a> found in the Lsub web site.</div>
<div style="font-family: Times; white-space: normal;">
</div>
<div>
</div>
</pre>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-11008094284617494892014-12-09T10:11:00.000-08:002014-12-09T10:11:20.994-08:00expr + test = xpThe clive command <a href="http://lsub.org/sys/man/1/xp.html">xp(1)</a> replaces the venerable UNIX commands expr and test. It is interesting to note why. In few words, both commands are doing the same, and thus they are a single command in Clive.<br />
<br />
The purpose of a command line calculator is to evaluate expressions. Now, the purpose of the test command in UNIX is also to evaluate expressions. Only that test knows how to evaluate expressions on file attributes.<br />
<br />
Considering that in clive directory entries are generic maps from attribute names to attribute values, it seems that a single command can do both. The result is that we can do calculations on, for example, file sizes and file modification times in very much the same way we can calculate on floating point numbers. Or we can perform bitwise operations on file permissions. As a result, the grammar known to the calculator is also available for free to the "test" command, because it is the same command.<br />
<br />
For example, it is clear that we can do things like<br />
<span style="font-family: Courier New, Courier, monospace;"><b>> xp 2 + 2</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>4</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>> xp 1k</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>1024</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>> xp 1m</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>1048576</b></span><br />
<div>
<br /></div>
But, we can use the same command to print metadata for files. For example, to print out<br />
the file type:<br />
<span style="font-family: Courier New, Courier, monospace;"><b>> xp type .</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>d</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>> xp type xp.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>-</b></span><br />
<div>
<br /></div>
<div>
Now we can write general purpose expressions as we see they fit; e.g. is the size larger than 1Kbyte?</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp size xp.go '>' 1k</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>false</b></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Or, how many full Mbytes are used by a binary?</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp size '"'`which xp`'"' / 1m</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>5</b></span></div>
</div>
<div>
<br /></div>
<div>
Another example. Let's check the mode for a file</div>
<div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp mode xp.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>0644</b></span></div>
</div>
</div>
<div>
<div>
<br /></div>
<div>
Now we can write an expression to get the write permissions,</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp mode xp.go '&' 0222</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>128</b></span></div>
</div>
<div>
<br /></div>
<div>
or to see if any of them is set</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp mode xp.go '&' 0222 '!=' 0</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>true</b></span></div>
</div>
<div>
<br /></div>
<div>
which can be done easier, by the way:</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> xp w xp.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>true</b></span></div>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-59891084313242000352014-11-04T04:02:00.000-08:002014-11-04T04:02:19.896-08:00Compare go source files in two file trees in cliveI'm still amazed at how pieces fit together in Clive. This is like a recursive diff in unix, but in Clive, and compares just source files in go as found within two trees:<br />
<br />
<pre style="font-size: 13px;"> ; comp ',~\.go$' '/tmp/x,~\.go$'
#diff comp.go:208: and /tmp/x/comp.go:207:
- x.diff()
#diff: comp.go:209: and /tmp/x/comp.go:209:
+ x.diff()
...
</pre>
<div>
<br /></div>
<div>
There are several things working together here</div>
<div>
<ul>
<li>Commands accept names made of two elements: a path and a predicate (separated by a comma). In the example the predicates match the name against what would be "*.go" in UNIX.</li>
<li>Each name is issued to a finder to find a stream of directory entries</li>
<li>Directory entries are self-describing, and thus a stream of directory entries is enough to reach the different file servers involved in the name space used for the search.</li>
<li>File and directory contents may be retrieved along with the find request; they are streamed through channels, like everybody else.</li>
<li>Directories always report and find entries in the same order, i.e. they are always sorted</li>
</ul>
<div>
As a result, the compare command only has to use a library function to make each name become a stream and then compare the two streams.</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-1949714205946833792014-10-21T08:26:00.000-07:002014-10-21T08:26:01.941-07:00Conventions can do more with less: Clive options continued.In a <a href="http://syssoftware.blogspot.com.es/2014/10/simple-things-are-more-powerful-clive.html">previous post</a> I discussed a new package in <a href="http://lsub.org/ls/clive.html">Clive</a>, <a href="http://lsub.org/sys/man/2/cmd.opt.html">opt</a>, which deals with command arguments.<br />
The package turned to be very easy to use and has unified how clive commands are used beyond understanding a common option syntax.<br />
<br />
When defining new options, the user gives a pointer to a Go value, a name, and a help string:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>func (f *Flags) NewFlag(name, help string, vp interface{})</b></span><br />
<div>
<br /></div>
The help string follows the convention that<br />
<br />
<ul>
<li>for options with no arguments, it simply describes the option.</li>
<li>for options with arguments, it starts with "<i>argument name:</i>" followed by the description of the option.</li>
</ul>
<div>
This allows the <span style="font-family: Courier New, Courier, monospace;"><b>usage</b></span> method to exploit this convention to build a usage string more descriptive than the ones in other Go programs. For example, compare the old</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;"><b>usage: flds -options [file...]</b></span></div>
<div>
<br /></div>
<div>
with the new</div>
<div>
<br /></div>
<div>
<span class="Apple-tab-span" style="white-space: pre;"> </span><span style="font-family: Courier New, Courier, monospace;"><b>usage: flds [-1D] [-F sep] [-o sep] {-r range} [file...]</b></span></div>
<div>
<br /></div>
<div>
The method writing the usage description scans the defined flags for those that have a single rune as a name, and no argument, and combines them like in "<span style="font-family: Courier New, Courier, monospace;"><b>-1D</b></span>" above. Then it goes over flags that have longer names, or have arguments, and adds the usage for each one in the same line. The name of the argument is taken from the start of the help string (if found, or a generic name for the type of argument [in Go] is used). When arguments may be used more than once, curly brackets are printed instead of square ones. The method knows this because the value pointer given to <span style="font-family: Courier New, Courier, monospace;"><b>NewFlag</b></span> is a pointer to a slice of values.</div>
<div>
<br /></div>
<div>
Thus, there is no need to keep the usage summary synchronised with the source, it is always in sync.</div>
<div>
<br /></div>
<div>
But this convention has done even more work for Clive. Manual pages in clive (for all sections but the go packages section) are written in <a href="http://lsub.org/sys/man/1/wr.html">wr(1)</a>. Recently we have</div>
<div>
<ul>
<li>added a feature to the opt package so that "<span style="font-family: Courier New, Courier, monospace;"><b>-?</b></span>" can never be a valid flag, which makes all programs inform of their usage when called with this flag.</li>
<li>replaced all the "synopsis" subsections in all command man pages with a new "usage" one.</li>
</ul>
<div>
The new manual pages start with something like:</div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>_ LNS(1): print lines</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>* USAGE</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>[sh</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>lns -? >[2=1] | lns -r 1,-2</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>]</b></span></div>
<div>
...</div>
</div>
<div>
<br /></div>
<div>
Here, a shell escape runs the command to make it inform about its usage, and all the output lines but for the last one (which is an exit status in clive) are inserted into the manual.</div>
<div>
<br /></div>
<div>
After doing so, we could remove most option description lists from manual pages, and we no longer have to document the usage of a command anymore. Plus, we are sure that the usage matches the binary installed in the system. For example, the manual for <span style="font-family: Courier New, Courier, monospace;"><b>lns</b></span> looks like:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>LNS(1): PRINT LINES</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>USAGE</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> usage: lns [-Dn] {-r range} [file...]</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>-D: debug</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>-n: print line numbers</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>-r range: print this range</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>range is addr,addr or addr</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>addr is linenb, or +linenb, or -linenb</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b> </b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><br /></b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>DESCRIPTION</b></span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>...</b></span></div>
<div>
<br /></div>
<div>
Once again, we could do more by actually doing less.</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-67652116193358917272014-10-14T08:51:00.000-07:002014-10-14T08:51:32.938-07:00Simple things are more powerful. The clive cmd/opt package.The golang <a href="http://golang.org/pkg/flag/"><span style="font-family: Courier New, Courier, monospace;"><b>flag</b></span></a> package is very powerful. It makes it easy to process your command arguments.<br />
We have been using it for a long time to process Clive command arguments.<br />
<br />
But, to say it in a few words, it is not able to operate on different argument vectors in a reasonable way (i.e., using the same interface used for processing the actual command line), it cannot handle combined <span style="font-family: Courier New, Courier, monospace;"><b>-abc</b></span> style flags instead of the utterly verbose <span style="font-family: Courier New, Courier, monospace;"><b>-a -b -c</b></span>, it cannot handle flags that repeat, and it is far more complex than needed. The clive <a href="http://lsub.org/sys/man/2/cmd.opt.html"><span style="font-family: Courier New, Courier, monospace;"><b>opt</b></span></a> package does all this by actually doing less.<br />
<br />
For example, the <span style="font-family: Courier New, Courier, monospace;"><b>flag</b></span> package operates on <span style="font-family: Courier New, Courier, monospace;"><b>os.Args</b></span> to retrieve the arguments. Using another array requires using another interface. Instead, <span style="font-family: Courier New, Courier, monospace;"><b>opt</b></span> takes always a <span style="font-family: Courier New, Courier, monospace;"><b>[]string</b></span> and works on it. This makes it work on any given set of arguments, which makes it unnecessary to provide any functionality to process "sub-arguments" or "different command lines". Instead, the interface is used against the desired set of arguments.<br />
<br />
Looking at the interface of <span style="font-family: Courier New, Courier, monospace;"><b>flag</b></span> provides more examples of what this post is about. There are multiple <span style="font-family: Courier New, Courier, monospace;"><b>StringVar()</b></span>, <span style="font-family: Courier New, Courier, monospace;"><b>String()</b></span>, <span style="font-family: Courier New, Courier, monospace;"><b>IntVar()</b></span>, <span style="font-family: Courier New, Courier, monospace;"><b>Int()</b></span>, ... functions to define new flags. There are many such functions and they define flags globally. Instead, <span style="font-family: Courier New, Courier, monospace;"><b>opt</b></span> provides a single method<br />
<pre style="background-color: white; font-size: 13px;"></pre>
<pre style="background-color: white;"><span style="font-family: Courier New, Courier, monospace;"> <b>func (f *Flags) NewFlag(name, help string, vp interface{})</b></span></pre>
<br />
to define new flags. It can be used as in<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b> debug := false</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> opts.NewFlag("d", "debug enable", &debug)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> odir := "/tmp"</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b> opts.NewFlag("o", "output dir, defaults to /tmp", &odir)</b></span><br />
<div>
<br /></div>
which has several advantages:<br />
<br />
<ol>
<li>The flag is defined on a set of flags, not globally. This permits using different sets of flags yet the code is the same that it would be to define global flags, as discussed above.</li>
<li>There is a single way of defining a new flag. This is a benefit, not a drawback.</li>
<li>There is a single method to call for any flag, not many to remember.</li>
<li>It does not re-do things the language is doing. For example, a default value is simply the value of the variable unless the options change it. Go already has assignment and initialisation, thus, it is not necessary to supply that service in the flag definition. </li>
<li>It is not possible to introduce errors due to initialisation of option variables not matching the default value given to the flag definition call.</li>
</ol>
<div>
The implementation has also important differences. The <span style="font-family: Courier New, Courier, monospace;"><b>flag</b></span> package is an elaborate set of internal types defined to provide methods to handle each one of the flag types. Adding a new flag requires defining types and methods and must be done with care. A consequence is that <span style="font-family: Courier New, Courier, monospace;"><b>Flag</b></span> has 850 lines and is harder to understand. <span style="font-family: Courier New, Courier, monospace;"><b>Opt</b></span> has 356 lines. In both cases including all the documentation.</div>
<div>
<br /></div>
<div>
Instead of the approach used in <span style="font-family: Courier New, Courier, monospace;"><b>flag</b></span> to define and implement the arguments, <span style="font-family: Courier New, Courier, monospace;"><b>opt</b></span> relies on a type switch on the pointers given when defining each one of the flags. All the package does during parsing is to set the pointer to the value found in the argument, using the type of the pointer to determine how to parse the value by default. This is the an excerpt of the code:</div>
<br />
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>switch vp := d.valp.(type) {</b></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case *bool:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>*vp = true</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if len(argv[0]) == 0 {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>argv = argv[1:]</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>} else { // put back the "-" for the next flag</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>argv[0] = "-" + argv[0]</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case *string:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>nargv, arg, err := optArg(argv)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>return nil, fmt.Errorf("option '%s': %s", d.name, err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>argv, *vp = nargv, arg</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case *[]string:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>nargv, arg, err := optArg(argv)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>return nil, fmt.Errorf("option '%s': %s", d.name, err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>argv, *vp = nargv, append(*vp, arg)</b></span></div>
</div>
<div>
<br /></div>
<div>
And that suffices. There is another type switch when a flag is defined to make sure that the pointer type can be handled later by the package during parsing. Because of this, the code walking the arguments trying to parse each option can be fully shared and is not highly coupled with the per-option parsing code.</div>
<br />
What about more complex arguments? Easy: the program may define string arguments and then parse them as desired. Or, if the new argument type becomes very popular, it can be added to opt by<br />
<br />
<ol>
<li>Adding an empty case to the type switch of NewFlag (to check the pointer type for validity)</li>
<li>Adding a new case to the type switch of Parse (the excerpt shown above) to actually do the parsing.</li>
</ol>
<br />
All this is not to say that <span style="font-family: Courier New, Courier, monospace;"><b>opt</b></span> is the greatest argument processing package. In fact, it is just born and it is likely that there are still bugs and things to improve. All this is to say that more can be done by doing less in the interface and in the implementation of the package considered.<br />
<br />
Go is a very nice language. I'd like its interfaces to be as clean, tiny and powerful as possible. If we want complex interfaces and implementations, we know where to find Java and C++.<br />
<br />Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-74275793169183143082014-10-10T09:59:00.000-07:002014-10-10T09:59:38.685-07:00Clive for users: finders, trees, and name spaces<br />
<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
<ol start="0" style="list-style-type: none;">
<li>
<span style="font-size: 10pt;">The first example shows that listing all
sources files under </span><span style="font-family: 'Courier'; font-size: 10.000000pt;">/zx/sys/src </span><span style="font-size: 10pt;">can be done with this command </span></li>
<li><span style="font-size: x-small;"><br /></span>
<pre><span style="font-family: 'Courier'; font-size: 8.000000pt;"> ; lz /zx/sys/src/,
/zx/sys/src
/zx/sys/src/clive
/zx/sys/src/clo
</span></pre>
<pre><span style="font-family: 'Courier'; font-size: 8.000000pt;">
</span></pre>
<span style="font-size: 10pt;">Here, describing a file (or a set of files) is done by using a combination of a name and a predicate. In this
case, all files starting at </span><span style="font-family: 'Courier'; font-size: 10.000000pt;">/zx/sys/src </span><span style="font-size: 10pt;">and matching an empty predicate (the empty string) are listed. </span><br />
<span style="font-size: 10pt;"><br /></span>
<span style="font-size: 10pt;">Or
we can list only directories using</span><br />
<span style="font-size: 10pt;"> </span><span style="font-family: Courier; font-size: 8pt;"> ; lz /zx/sys/src/,type=d</span><br />
<span style="font-family: Courier; font-size: 8pt;"><br /></span>
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<span style="font-size: 10pt;">Or we can remove all regular files in a hierarchy by using
</span><br />
<pre><span style="font-family: 'Courier'; font-size: 8.000000pt;"> ; rm /zx/sys/src/,type=-
</span></pre>
<pre><span style="font-family: 'Courier'; font-size: 8.000000pt;">
</span></pre>
<span style="font-size: 10pt;">Commands operating on files find the involved directory entries and then operate on them. A directory
entry is a set of name/value pairs, and may have any number of attributes with names and values chosen at
will. Of course, there are some conventions about which attributes are expected to be there (like size, mode,
etc). </span><br />
<span style="font-size: 10pt;"><br /></span>
<span style="font-size: 10pt;">Predicates used to find files are general expressions that may use directory attributes as values, which
makes it easy for a command to issue an expression to find the entries of interest in a single or a few RPCs.
Directory entries are self-describing entities (eg., they report also the address of the server and the name of
the resource in the server). </span><br />
<span style="font-size: 10pt;"><br /></span>
<span style="font-size: 10pt;">This makes it easy for a program to issue requests for a directory entry it found.
In short, file trees in clive are split into two important entities: </span><span style="font-size: 10pt;">Finders used to find directory entries, and f</span><span style="font-size: 10pt;">ile trees that accept operations for directory entries</span></div>
</div>
</div>
</li>
<li><span style="font-size: 10pt;"><br /></span></li>
<li><span style="font-size: 10pt;">Each process groups one or more finders into a name space, built from a textual representation (it might
inherit the name space from the parent). For example, we can use </span><div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<ul style="list-style-type: none;">
</ul>
</div>
</div>
</div>
</li>
<li><span style="font-size: 10pt;"><br /></span></li>
<li><span style="font-size: 10pt;">
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<ol start="0" style="list-style-type: none;">
<ul>
<li><span style="font-family: Courier; font-size: 8pt;">; NS=’/ /</span></li>
</ul>
<li>
<ul style="list-style-type: none;">
<li>
<span style="font-family: 'Courier'; font-size: 8.000000pt;">;; /zx tcp!zxserver!zx
</span><br />
</li>
<li>
<span style="font-family: 'Courier'; font-size: 8.000000pt;">;; /dump tcp!zxserver!zx!dump</span><br />
<span style="font-family: 'Courier'; font-size: 8.000000pt;">;; ’</span><br />
<span style="font-family: Courier; font-size: 8pt;">; lz /zx/usr/nemo,type=d</span><br />
<br />
</li>
</ul>
</li>
</ol>
</div>
</div>
</div>
</span></li>
<li><span style="font-size: 10pt;">
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
</div>
</div>
</div>
</span></li>
<li><span style="font-size: 10pt;">to define a new name space and then issue commands that work in it. In this example, we defined as </span><span style="font-family: Courier; font-size: 10pt;">/ </span><span style="font-size: 10pt;">the
root of the host OS file tree, and then mounted at </span><span style="font-family: Courier; font-size: 10pt;">/zx </span><span style="font-size: 10pt;">our main tree and at </span><span style="font-family: Courier; font-size: 10pt;">/dump </span><span style="font-size: 10pt;">its dump file system.
To say it in a different way, the name space is a finder that may groups other finders (among other things).
The name space is more powerful, and can mount at a given name a set of directory entries (be they for files
or not), but the example suffices for now. </span></li>
<li><span style="font-size: 10pt;"><br /></span></li>
<li><span style="font-size: 10pt;"><br /></span></li>
<ol start="0" style="list-style-type: none;">
</ol>
</ol>
</div>
</div>
</div>
<ul style="list-style-type: none;">
</ul>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-90762534346882319372014-05-28T10:04:00.001-07:002014-05-28T10:04:22.298-07:00bytes.Buffer or builtin concatenation to build strings in Go?During profiling of clive Go code I have found an interesting bit.<br />
This is the same old discussion about deciding on using a string buffer or using raw string concatenation to build strings.<br />
<br />
A <span style="font-family: Courier New, Courier, monospace;">Dir</span> is actually a map[string]string, and, the network format is a string with form<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b> attr1=val1 attr2=val2 ....</b></span><br />
<br />
The function <span style="font-family: Courier New, Courier, monospace;">Dir.String()</span> builds the string from the map. The question is:<br />
<br />
Do we use <span style="font-family: Courier New, Courier, monospace;">bytes.Buffer</span> and <span style="font-family: Courier New, Courier, monospace;">Fprintf</span> to append data to the string, asking at the end to <span style="font-family: Courier New, Courier, monospace;">Buffer.String</span> the resulting string? Or do we use a native Go <span style="font-family: Courier New, Courier, monospace;">string</span> and concatenate using "<span style="font-family: Courier New, Courier, monospace;">+=</span>"?<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Using <span style="font-family: Courier New, Courier, monospace;">bytes.Buffer</span> and <span style="font-family: Courier New, Courier, monospace;">Fprintf</span> in <span style="font-family: Courier New, Courier, monospace;">Dir.String()</span> yields:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>BenchmarkDirString <span class="Apple-tab-span" style="white-space: pre;"> </span> 500000<span class="Apple-tab-span" style="white-space: pre;"> </span> 5163 ns/op<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Using <span style="font-family: Courier New, Courier, monospace;">strings</span> and <span style="font-family: Courier New, Courier, monospace;">+= </span><br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>BenchmarkDirString <span class="Apple-tab-span" style="white-space: pre;"> </span> 500000<span class="Apple-tab-span" style="white-space: pre;"> </span> 5077 ns/op<br />
<div>
<br /></div>
<br />
Surprisingly (perhaps), it's both easier to write and faster to use the strings and forget<br />
about using the <span style="font-family: Courier New, Courier, monospace;">bytes.Buffer</span>. It's likely this will place more pressure in the GC, because it builds and discards intermediate strings, but, it's faster and easier.<br />
<br />Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-85673810745731425332014-05-23T04:43:00.001-07:002014-05-23T04:43:31.681-07:00Early clive distributionWe just placed in the <a href="http://lsub.org/ls/clive.html">Clive's web site</a> a draft for a paper<br />
describing the system, a link to the manual and indications to download our<br />
(development) source tree.<br />
<br />
This is a research system, still under construction. Be warned. Nevertheless, we<br />
are already using it, and our main file tree and its dump are under the control of<br />
Clive.<br />
<br />
As an appetiser, this is the command to list some files<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>> l -l /zx/nautilus/src,name~*.go</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--rw-r--r-- /zx/nautilus/src/bufs/bufs.go<span class="Apple-tab-span" style="white-space: pre;"> </span>683<span class="Apple-tab-span" style="white-space: pre;"> </span>15 May 14 12:43 CEST</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--rw-r--r-- /zx/nautilus/src/cmd/auth/auth.go<span class="Apple-tab-span" style="white-space: pre;"> </span>1312<span class="Apple-tab-span" style="white-space: pre;"> </span>15 May 14 12:48 CEST</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--rw-r--r-- /zx/nautilus/src/cmd/hist/hist.go<span class="Apple-tab-span" style="white-space: pre;"> </span>5875<span class="Apple-tab-span" style="white-space: pre;"> </span>19 May 14 14:17 CEST</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--rw-r--r-- /zx/nautilus/src/cmd/ns/ns.go<span class="Apple-tab-span" style="white-space: pre;"> </span>4254<span class="Apple-tab-span" style="white-space: pre;"> </span>15 May 14 13:06 CEST</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b>--rw-r--r-- /zx/nautilus/src/cmd/nsh/nsh.go<span class="Apple-tab-span" style="white-space: pre;"> </span>14719<span class="Apple-tab-span" style="white-space: pre;"> </span>21 May 14 15:24 CEST</b></span><br />
<div>
<br /></div>
<div>
and this is how the more interesting <span style="font-family: Courier New, Courier, monospace;">rm </span><span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">/zx/nautilus/src,name~*.go </span><span style="font-family: Times, Times New Roman, serif;">is implemented</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// In our example, path is /zx/nautilus/src, and pred is name~*.go</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dirc := rns.Find(path, pred, "/", 0)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>errors := []chan error{}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for dir := range dirc {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// get a handle for the directory entry server</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>wt, err := zx.RWDirTree(dir)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dbg.Warn("%s: tree: %s", dir["path"], err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>continue</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>errc := wt.Remove(dir["path"])</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>errors = append(errors, ec)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for _, errc := range errors {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err := <-errc; err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dbg.Warn("%s: %s", dir["path"], err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
</div>
<div>
<br /></div>
<div>
Enjoy. The clivezx group at googlegroups is a public discussion group where we will make further announces regarding clive, and host any public discussion about it. You are invited to join.</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-45462066643762944402014-04-02T12:43:00.000-07:002014-04-02T12:43:09.204-07:00Using files in cliveThis is just a few snaps from a clive shell, a more detailed description or TR will follow...<br />
<br />
List the ns (two trees mounted at /zx/abin, and /zx/xbin, nothing else mounted)<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>> ns</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>/ 0</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>/zx/abin 1</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>"" path:"/zx/abin" mode:"020000000755" type:"d" size:"3" name:"/" mtime:"1396356469000000000" addr:"tcp!Atlantis.local!8002!abin" proto:"zx finder" gc:"y"</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>/zx/xbin 1</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>"" path:"/zx/xbin" mode:"020000000755" type:"d" size:"1" name:"/" mtime:"1396356453000000000" addr:"tcp!Atlantis.local!8002!xbin" proto:"zx finder" gc:"y"</b></span><br />
<div>
<br /></div>
<div>
List just files at /zx</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> l -l /zx</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>d-rwxrwx--- /zx/abin<span class="Apple-tab-span" style="white-space: pre;"> </span>3<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>d-rwxrwx--- /zx/xbin<span class="Apple-tab-span" style="white-space: pre;"> </span>1<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
</div>
<div>
<br /></div>
<div>
List all files under / with mode 750</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> l -l /,mode=0750</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/Go<span class="Apple-tab-span" style="white-space: pre;"> </span>202<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/Watch<span class="Apple-tab-span" style="white-space: pre;"> </span>2351948<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/a<span class="Apple-tab-span" style="white-space: pre;"> </span>212<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/cgo<span class="Apple-tab-span" style="white-space: pre;"> </span>2452606<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/drawterm<span class="Apple-tab-span" style="white-space: pre;"> </span>837032<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/drawterm.old<span class="Apple-tab-span" style="white-space: pre;"> </span>843880<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/dt<span class="Apple-tab-span" style="white-space: pre;"> </span>60<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/ebnflint<span class="Apple-tab-span" style="white-space: pre;"> </span>1149721<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/gacc<span class="Apple-tab-span" style="white-space: pre;"> </span>32<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/gnot<span class="Apple-tab-span" style="white-space: pre;"> </span>52<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/hgpatch<span class="Apple-tab-span" style="white-space: pre;"> </span>1328046<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/mango-doc<span class="Apple-tab-span" style="white-space: pre;"> </span>2957328<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/nix<span class="Apple-tab-span" style="white-space: pre;"> </span>60<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/quietgcc<span class="Apple-tab-span" style="white-space: pre;"> </span>1326<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/t+<span class="Apple-tab-span" style="white-space: pre;"> </span>23<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>- 0-rwxr-x--- /zx/xbin/bin/t-<span class="Apple-tab-span" style="white-space: pre;"> </span>22<span class="Apple-tab-span" style="white-space: pre;"> </span>01 Apr 14 14:47 CEST</b></span></div>
</div>
<div>
<br /></div>
<div>
Then chmod all of them to mode 755</div>
<div>
<b style="font-family: 'Courier New', Courier, monospace;">> l -a /,mode=0750 | chmod 0755</b></div>
<div>
<br /></div>
<div>
How did chmod find out the files?</div>
<div>
Here's how</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>> zl -a /zx</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>tcp!Atlantis.local!8002!abin<span class="Apple-tab-span" style="white-space: pre;"> </span>/</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>tcp!Atlantis.local!8002!xbin<span class="Apple-tab-span" style="white-space: pre;"> </span>/</b></span></div>
</div>
<div>
<br /></div>
<div>
<span style="font-family: inherit;">The interesting thing is that the names are separated from the files (i.e., a file server serves a finder interface to find out about directory entries, and then a file tree to serve operations on those entries).</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Commands operate on finders to discover entries, and then use them to reach the servers and perform operations on them. Of course, connections to servers are cached so that we don't create a new one per directory entry, which would be silly.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">All these requests are streamed through channels, as are the replies, so that you can do things like this:</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>dc := sh.files(true, args...)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>n := 0</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for d := range dc {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>n++</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// we could process +-rwx here using d["mode"] to</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// build the new mode and then use just that one.</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>nd := zx.Dir{"mode": mode}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>ec := rfs.Dir(d).Wstat(nd)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span><-ec</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>err := cerror(ec)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Printf("%s: %s\n", d["path"], err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
</div>
<div>
<br /></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
Here, the code is waiting for one reply at a time, but it could easily stream 50 at a time like another command does:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>calls := make(chan chan error, 50)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := len(ents)-1; i >=0; i-- {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>ec := rfs.Dir(ents[i]).Remove()</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>select {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case calls <- ec:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>case one := <-calls:</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span><-one</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>err := cerror(one)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Printf("%s\n", err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-55673193442221425242014-03-11T11:01:00.000-07:002014-03-11T11:01:05.698-07:00Network servers in CliveThe TR draft kept at <a href="http://lsub.org/export/srv.pdf">TR</a> describes, including some code excerpts how network services are built leveraging the tools described in the previous posts (and the TRs they mention).<br />
<br />
In short, it is very easy to build services in a CSP like world while, at the same time, those services may be supplied across the network. And that considers also a set of pipes, or fifos, as a network!. It is not just TCP.<br />
<br />
This is a teaser:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>func (s *Srv) put(m *Msg, c <-chan []byte, rc chan<- []byte) {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>wt, ok := s.t.(zx.RWTree)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if !ok {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(rc, "not a rw tree")</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>return</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>err := wt.Put(m.Rid, m.D, c)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(rc, err)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span><br />
<div>
<br /></div>
<div>
And this is another:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>cc, err := Dial("fifo!*!ftest", nil)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if err != nil {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>t.Fatal(err)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for i := 0; i < 10; i++ {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>cc.Out <- []byte(fmt.Sprintf("<%d>", i))</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>msg := string(<-cc.In)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>printf("got %s back\n", msg)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(cc.In)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(cc.Out)</b></span></div>
</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-24173716382624389902014-03-11T08:57:00.001-07:002014-03-11T08:57:45.415-07:00Files and name spaces meet channels in CliveThis is the third post on a series of posts describing what we did as part of the ongoing effort to build a new OS, which we just named as "Clive". The previous two posts described how channels were used and bridged to the outside pipes and connections, and how they where multiplexed to make it easy to write protocols in a CSP style.<br />
<br />
This post describes what we did regarding file trees and name spaces. This won't be a surprise considering that we come from the Plan 9, Inferno, Plan B, NIX, and Octopus heritage.<br />
<br />
But first things last. Yes, last. A name space is separated from the conventional file tree abstraction. That is, name spaces exist on their own and map prefixes to directory entries. A name space is searched using this interface<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b>type Finder interface {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>Find(path, pred string, depth0 int) <-chan zx.Dir</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span><br />
<div>
<br /></div>
<div>
The only operation lets the caller find a stream of matching directory entries for the request.</div>
<div>
Directory entries are within the subtree rooted at the given path, and must match the supplied predicate.</div>
<div>
<br /></div>
<div>
A predicate here is a very powerful thing, inspired by the unix command named before our operation.</div>
<div>
For example, the predicate</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>~ name "*.[ch]" & depth < 3</b></span></div>
</div>
<div>
can be used to find files with names matching C source file names but no deeper than three</div>
<div>
levels counting from the path given.</div>
<div>
<br /></div>
<div>
It is also important to notice that the results are delivered through a channel. This means that the operation may be issued, reach a name space at the other end of the network, and the caller may just retrieve the stream of replies when desired (or pass the stream to another process).</div>
<div>
<br /></div>
<div>
Things like removing all object files within a subtree can now be done in a couple of calls. One to find the stream of entries, and another to convert that stream into a stream of remove requests.</div>
<div>
<br /></div>
<div>
And here is where the first thing (deferred until now) comes. The interface for a file tree to be used across the network relies on channels (as promises) to retrieve the result of each operation requested. Furthermore, those channels are buffered in many cases (eg., on all error indications) and might be even ignored if the caller is going to checkout later the status of the entire request in some other way.</div>
<div>
<br /></div>
<div>
Thus, we can actually stream the series of removes in this example. Note that the call to remove simply issues the call and returns a channel that can be used to receive the reply later on.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>oc := ns.Find("/a/b", "~ name *.[ch] & depth < 3", 0)</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>for d := range oc {</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>fs.Remove(d["path"]) // and ignore the error channel</b></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span></div>
</div>
<div>
<br /></div>
<div>
This is just an example. A realistic case would not ignore the error channel returned by</div>
<div>
the call to remove. It would probably defer the check until later, or pass the channel to another process checking out that removes were made. </div>
<div>
<br /></div>
<div>
Nevertheless, the example gives a glance of the power of file system interfaces used in Clive.</div>
<div>
Reading and Writing is also made relying on input and output channels.</div>
<div>
<br /></div>
<div>
If you are interested in more details, you can read <a href="http://lsub.org/export/zx.pdf">zx</a>, which describes the file system and name space interfaces, and perhaps also <a href="http://lsub.org/export/nchan.pdf">nchan</a>, which describes CSP like tools used in Clive and important to understand how these interfaces are used in practice.</div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-86155574555726288772014-03-11T02:50:00.000-07:002014-03-11T02:50:18.452-07:00Channels, pipes, connections, and multiplexorsThis post is the second in a series to describe how we adapted the CSP style of programming used in Go to the use of external pipes, files, network connections, and related artefacts.<br />
This was the <a href="http://syssoftware.blogspot.com.es/2014/03/channels-and-pipes-close-and-errors.html">first post</a>. In what follows we refer as "pipe" to any external file-descriptor-like artefact used to reach the external world (for input or output).<br />
<br />
<br />
<div class="page" title="Page 3">
<div class="layoutArea">
<div class="column">
<span style="font-weight: 700;">Connections
</span><br />
The <span style="font-style: italic;">nchan </span>package defines a connection as
<br />
<br />
<pre><span style="font-family: Courier New, Courier, monospace;"><b> type Conn struct {
Tag string // debug
In <-chan []byte
Out chan<- []byte
}
</b></span></pre>
<div>
<br /></div>
This joins two channels to make a full-duplex connection. A process talking to an external entity relies on
this structure to bridge the system pipe used to a pair of channels. There are utilies that leverage the func-
tions described in the previous section and build a channel interface to external pipes, for example:
<br />
<pre><span style="font-family: Courier;">
</span></pre>
<pre><span style="font-family: Courier New, Courier, monospace;"><b> func NewConn(rw io.ReadWriteCloser, nbuf int, win, wout chan bool) Conn
</b></span></pre>
<div>
<br /></div>
The function creates processes to feed and drain the connection channels from and to the external pipe. Fur-
thermore, if <span style="font-style: italic;">rw </span>supports closing only for reading or writing, a close on the input or output channels would
close the respective halves of the pipe. Because of the message protocol explained in the previous section,
errors are also propagated across the external pipe and the process using the connection can very much
ignore that the source/sink of data is external.
<br />
It is easy to build pipes where the <span style="font-style: italic;">Out </span>channel sends elements through the <span style="font-style: italic;">In </span>channel:<br />
<span style="font-family: Courier;"><br /></span>
<b style="font-family: 'Courier New', Courier, monospace;"></b><span style="font-family: Courier New, Courier, monospace;"><b> f</b></span><b style="font-family: 'Courier New', Courier, monospace;"></b><b style="font-family: 'Courier New', Courier, monospace;">unc NewPipe(nbuf int) Conn</b><br />
<div>
<br /></div>
And, using this, we can create in-memory connections that do not leave the process space:
<br />
<pre><span style="font-family: Courier;">
</span></pre>
<pre><span style="font-family: Courier New, Courier, monospace;"><b> func NewConnPipe(nbuf int) (Conn, Conn)
</b></span></pre>
<div>
<br /></div>
This has been very useful during testing, because this connection can be created with no buffering and it is
easier to spot dead-locks that involve both ends of the connection. Once the program is ready, we can
replace the connection based pipe with an actual system provided pipe.<br />
<br />
<div class="page" title="Page 3">
<div class="layoutArea">
<div class="column">
<span style="font-weight: 700;">Multiplexors
</span><br />
Upon the channel based connections shown in the previous sections, the <span style="font-style: italic;">nchan </span>package provides multiplex-
ors.
<br />
<br />
<pre><span style="font-family: Courier New, Courier, monospace;"><b> type Mux struct {
In chan Conn
...
}
func NewMux(c Conn, iscaller bool) *Mux
func (m *Mux) Close(err error)
func (m *Mux) Out() chan<- []byte
func (m *Mux) Rpc() (outc chan<- []byte, repc <-chan []byte)
</b></span></pre>
<div>
<br /></div>
</div>
</div>
</div>
A program speaking a protocol usually creates a new <span style="font-style: italic;">Conn </span>connection by dialing or accepting connections
and then creates a <span style="font-style: italic;">Mux </span>by calling <span style="font-style: italic;">NewMux </span>to multiplex the connection among multiple requests.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyY0H47dFltyFRL8eZhUCwhI1LNJ6RwazbEPl3YjP119JQRjH3do92RlviGX4JnJMOfKzd9YZIYT2FaHNF0paBSogRVsSTMeQcpIkDd05EOyIO1BhBZWoHZaRC-NZ97WmSVm2uGRzz-uLC/s1600/pic1.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyY0H47dFltyFRL8eZhUCwhI1LNJ6RwazbEPl3YjP119JQRjH3do92RlviGX4JnJMOfKzd9YZIYT2FaHNF0paBSogRVsSTMeQcpIkDd05EOyIO1BhBZWoHZaRC-NZ97WmSVm2uGRzz-uLC/s1600/pic1.tiff" /></a></div>
<br />
<br />
<div class="page" title="Page 4">
<div class="layoutArea">
<div class="column">
The nice thing of the multiplexed connection is that requests may carry a series of messages (and not just
one message per request) and may or not have replies. Replies may also be a full series of messages. Both
ends of a multiplexed connetion (the process using the <span style="font-style: italic;">mux </span>and its peer at the other end of the pipe) may
issue requests. Thus, this is not a client-server interaction model, although it may be used as such.
<br />
To issue new outgoing requests through the multiplexor, the process calls <span style="font-style: italic;">Out </span>(to issue requests with
no expected reply):
<br />
<pre><span style="font-family: Courier;">
</span></pre>
<pre><span style="font-family: Courier New, Courier, monospace;"><b> oc := mux.Out()
oc <- []byte("no reply")
oc <- []byte("expected")
close(oc)
</b></span></pre>
<div>
<br /></div>
Or the process may call <span style="font-style: italic;">Rpc </span>(to issue requests with an expected reply).
<br />
<pre><span style="font-family: Courier;">
</span></pre>
<pre><span style="font-family: Courier New, Courier, monospace;"><b> rc, rr := mux.Rpc()
rc <- []byte("another")
rc <- []byte("request")
close(rc)
for m := range rr {
Printf("got %v as part of the reply\en", m)
}
Printf("and the final error status is %v\en", cerror(rr))
</b></span></pre>
<div>
<br /></div>
</div>
</div>
</div>
In the first case, the multiplexor returns a <span style="font-style: italic;">Conn </span>to the caller with just the <span style="font-style: italic;">Out </span>channel. Of course, this can
be done multiple times to issue several concurrent outgoing requests:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHU3Mz5jEcjXQrPz_uxM0svZfVpWp__CQHRd-w_cDBb3IfIz4imYfT1w2AP5YJ6eFDVg3zCakzyIoeZtZU2AWbnhtABHELSMnpcF3AHI-egvJjliDNzowMZwAbC7hleOWy4W5F7PddydUI/s1600/out1.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHU3Mz5jEcjXQrPz_uxM0svZfVpWp__CQHRd-w_cDBb3IfIz4imYfT1w2AP5YJ6eFDVg3zCakzyIoeZtZU2AWbnhtABHELSMnpcF3AHI-egvJjliDNzowMZwAbC7hleOWy4W5F7PddydUI/s1600/out1.tiff" /></a></div>
<br />
<br />
<div class="page" title="Page 4">
<div class="layoutArea">
<div class="column">
In the figure, the two connections of the left were built by two calls to <span style="font-style: italic;">mux.Out()</span>, which returns a <span style="font-style: italic;">Conn
</span>with an <span style="font-style: italic;">Out </span>chan to issue requests. The process using the <span style="font-style: italic;">Out </span>channel may issue as many messages as
desired and then close the channel.
<br />
</div>
</div>
</div>
If the request depicted below requires a reply, <span style="font-style: italic;">mux.Rpc() </span>is be called finstead of <span style="font-style: italic;">mux.Out() </span>and the
resulting picture is as shown.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8eZXfNPV_If7f6KJiT41dt5CGSt3PeQoWIkWTaMJclGGcetfJTJz3NjBX-boa3jJBriYvT-XcWNQMusL8uCcUs8tAjdcgv2tSq6GAaPQb7R2IMy8hni7CbKAo7RWvG3P2XQcRA1vSuLjP/s1600/out2.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8eZXfNPV_If7f6KJiT41dt5CGSt3PeQoWIkWTaMJclGGcetfJTJz3NjBX-boa3jJBriYvT-XcWNQMusL8uCcUs8tAjdcgv2tSq6GAaPQb7R2IMy8hni7CbKAo7RWvG3P2XQcRA1vSuLjP/s1600/out2.tiff" /></a></div>
<br />
The important part is that messages (and replies) sent as part of a request (or reply) may be streamed with-
out affecting other requests and replies, other than by the usage of the underlying connection. That is, an idle stream does not block other streams.<br />
The interface for the receiving part of the multiplexor is a single <span style="font-style: italic;">In </span>channel that conveys one <span style="font-style: italic;">Conn
</span>per incoming request. The request has only the <span style="font-style: italic;">In </span>channel if no reply is expected, and has both the <span style="font-style: italic;">In </span>and
<span style="font-style: italic;">Out </span>channels set if a reply is expected.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrLTVl4r9sCfDNaIeFHFpem5Qa99S22sfTncqnFCCnoXvfYXNRdKE6cZ6CtAkjbnMuJFLDBvP_9VCcynO1jXMrqo3UBsDjcpnNTQiY5XvqJepwX2vuygsOWkL3E4GPoC-N8CFVAf_8Yo5D/s1600/out3.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrLTVl4r9sCfDNaIeFHFpem5Qa99S22sfTncqnFCCnoXvfYXNRdKE6cZ6CtAkjbnMuJFLDBvP_9VCcynO1jXMrqo3UBsDjcpnNTQiY5XvqJepwX2vuygsOWkL3E4GPoC-N8CFVAf_8Yo5D/s1600/out3.tiff" /></a></div>
<br />
To receive requests from the other end of the pipe, the code might look like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for call := range mux.In {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>// call is a Conn</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>for m := range call.In {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>Printf("got %v as part of the request\en", m)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if call.Out != nil {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>call.Out <- []byte("a reply")</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>call.Out <- []byte("was expected, but...")</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(call.Out, "Oops!, failed")</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<div>
<br /></div>
<br />
<div class="page" title="Page 5">
<div class="layoutArea">
<div class="column">
For example, if a process received two requests, one with no reply expected and another with a reply
expected, the picture would be:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9NPM4N6dOBcJU6UC5TAobrKOTY4s_5ppatEhLP7Eg7plko72Lxd-mMX-ryyNlBkMp0B6Ctpl8CkxPQh3V3Iu3H0MPCPlHTJaVJQD9KqUwVI-v9umL5Qzylp30F2eN_Fh5bgc-Xo-Aktce/s1600/out4.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9NPM4N6dOBcJU6UC5TAobrKOTY4s_5ppatEhLP7Eg7plko72Lxd-mMX-ryyNlBkMp0B6Ctpl8CkxPQh3V3Iu3H0MPCPlHTJaVJQD9KqUwVI-v9umL5Qzylp30F2eN_Fh5bgc-Xo-Aktce/s1600/out4.tiff" /></a></div>
<br />
<br />
<div class="page" title="Page 5">
<div class="layoutArea">
<div class="column">
Here, the two connections on the left represent requests that were received through the <span style="font-style: italic;">In </span>channel depicted
on top of the multiplexor.
<br />
<br />
</div>
</div>
</div>
The important thing to note is that processes may now issue streams of requests, or replies, through
channels and they are fed to external pipes (or from them) as required. The interfaces shown have greatly
simplified programming for (networked) system serviced being written for the new system.<br />
</div>
</div>
</div>
<br />
</div>
</div>
</div>
<br />Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-14880166942343185042014-03-10T12:33:00.001-07:002014-03-10T12:33:09.335-07:00 Channels and pipes: close and errorsThis post describes changes made to
Go channels and tools built upon those to provide system and network-wide services for a
new OS under construction. Further posts will follow and describe other related set of tools, but the post is already long enough. Yes, belts are gone, we use our modified channels directly now.<br />
<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
<br />
A channel is an artifact that can be used to send (typed) data through it. The Go language operations on
channels include sending, receiving, and selection among a set of send and receive operations. Go permits
also to close a channel after the last item has been sent.
<br />
On most systems, processes and applications talk through pipes, network connections, FIFOS, and
related artifacts. In short, they are just file descriptors once open, permit the application to write data (for
sending) and/or read data (for receiving). Some of these are duplex, but they can be considered to be a pair
of devices (one for each direction). In what follows we will refer to all these artifacts as pipes (e.g., a net-
work connection may be considered as a pair of simplex pipes).
<br />
There is a mismatch between channels and pipes and this paper shows what we did to try to bridge
the gap between both abstractions for a new system. The aim is to let applications leverage the CSP style of
programming while, at the same time, let them work across the network.
<br />
We assume that the reader is familiar with channels in the Go language, and describes only our modi-
fications and additions.<br />
<br />
<br />
<div class="page" title="Page 1">
<div class="layoutArea">
<div class="column">
<span style="font-weight: 700;">Close and errors
</span><br />
<span style="font-weight: 700;"><br /></span>
When using pipes, each end of the pipe may close and the pipe implementation takes care of propagating
the error to the other end. That is not the case with standard Go channels. Furthermore, upon errors, it is
desirable for one end of the pipe to learn about the error that did happen at the other end.
<br />
We have modified the standard Go implementation to:
<br />
<ol style="list-style-type: none;">
<li>
1 Accept an optional <span style="font-style: italic;">error </span>argument to <span style="font-style: italic;">close</span>.
<br />
</li>
<li>
2 Make the send operation return <span style="font-style: italic;">false </span>when used on a closed channel (instead of panicing; the receive
operation already behaves nicely the case of a closed channel).
<br />
</li>
<li>
3 Provide a primitive, <span style="font-style: italic;">cerror</span>, that returns the error given when the channel was closed.
<br />
</li>
<li>
4 Make a close of an already closed channel a no-operation (instead of a panic).
<br />
</li>
</ol>
With this modified tool in hand, it is feasible to write the following code:<br />
<span style="font-size: 10pt;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"><b>var inc, outc chan[]byte</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>...</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>for data := range inc {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>ndata := modify(data)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>if ok := outc <-ndata; !ok {</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>close(inc, cerror(outc))</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>break</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>}</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b>close(outc, cerror(inc))</b></span><br />
<br />
<br />
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
Here, a process consumes data from <span style="font-style: italic;">inc </span>and produces new data through <span style="font-style: italic;">outc </span>for another one. The image to
have in mind is<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibvWKR_zs4qoVKvR8Kg-oz9KQKBTehINiSsPiRSuIHplch5D6LauvmAJ8vYO7Yrk6bETLeONNVkzgJvZGUvvAozAOjs8MrMxh736Iq2c28uOmL7HMK4nkaOiwSFl-0QP528e0zJ9DfTCbs/s1600/procs.tiff" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibvWKR_zs4qoVKvR8Kg-oz9KQKBTehINiSsPiRSuIHplch5D6LauvmAJ8vYO7Yrk6bETLeONNVkzgJvZGUvvAozAOjs8MrMxh736Iq2c28uOmL7HMK4nkaOiwSFl-0QP528e0zJ9DfTCbs/s1600/procs.tiff" height="107" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
where the code shown corresponds to the middle process. Perhaps the first process terminates normally (or
abnormally), in which case it would close <span style="font-style: italic;">inc</span>. In this case, our code closes <span style="font-style: italic;">outc </span>as expected. But this time,
the error given by the first process is known to the second process, and it can even forward such error to the
third one.
<br />
A more interesting case is when the third process decides to cease consuming data from <span style="font-style: italic;">outc </span>and
calls <span style="font-style: italic;">close</span>. Now, our middle process will notice that <span style="font-style: italic;">ok </span>becomes <span style="font-style: italic;">false </span>when it tries to send more data, and
can break its loop cleanly, closing also the input channel to singal to the first process that there is no point
in producing further data. In this second example, the last call to <span style="font-style: italic;">close </span>is a no-operation because the output
channel was already closed, and we don’t need to add unnecessary code to prevent the call.
<br />
The important point is that termination of the data stream is easy to handle for the program without
resorting to exceptions (or panics), and we know which one is the error, so we can take whatever measures
are convenient in that case.<br />
</div>
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<span style="font-weight: 700;">Channels and pipes
</span><br />
There are three big differences between channels and pipes (we are using <span style="font-style: italic;">pipe </span>to refer to any ‘‘file descrip-
tor’’ used to convey data, as stated before). One is that pipes may have errors when sending or receiving,
but channels do not. Another one is that pipes carry ony streams of bytes and not separate messages. Yet
another is that channels convey a data type but pipes convey just bytes.
<br />
The first difference is mostly dealt with the changes made to channels as described in the previous
section. That is, channels may have errors while sending and or receiving, considered the changes made.
Therefore, the code using a channel must consider errors in very much the same way it would do if using a
pipe.
<br />
To address the third difference we are going to consider channels of <span style="font-style: italic;">byte arrays </span>by now.
<br />
The second difference can be dealt with by ensuring that applications using channels to speak
through a pipe preserve message boundaries within the pipe. With this in mind, a new <span style="font-style: italic;">nchan </span>package pro-
vides new channel tools to bridge the gap between the channel and the pipe domains.
<br />
The following function writes each message received from <span style="font-style: italic;">c </span>into <span style="font-style: italic;">w </span>as it arrives. If <span style="font-style: italic;">w </span>preserves mes-
sage boundaries, that is enough. The second function is its counterpart.<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>func WriteBytesTo(w io.Writer, c <-chan []byte) (int64, error)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>func ReadBytesFrom(r io.Reader, c chan<- []byte) (int64, error)</b></span><br />
<div>
<br /></div>
However, is most cases, the transport does not preserve message boundaries. Thus, the next function writes
all messages received from <span style="font-style: italic;">c </span>into <span style="font-style: italic;">w</span>, but precedes each such write with a header indicating the message
length. The second function can rely on this to read one message at a time and forward it to the given chan-
nel.<br />
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<span style="font-family: Courier New, Courier, monospace;"><b> <span class="Apple-tab-span" style="white-space: pre;"> </span>func WriteMsgsTo(w io.Writer, c <-chan []byte) (int64, error)</b></span><br />
<span style="font-family: Courier New, Courier, monospace;"><b><span class="Apple-tab-span" style="white-space: pre;"> </span>func ReadMsgsFrom(r io.Reader, c chan<- []byte) (int64, error)</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><b><br /></b></span>
<br />
<div class="page" title="Page 3">
<div class="layoutArea">
<div class="column">
One interesting feature of <span style="font-style: italic;">WriteMsgsTo </span>and <span style="font-style: italic;">ReadMsgsFrom </span>is that when the channel is closed, its error sta-
tus is checked out and forwarded through the pipe. The other end notices that the message is an error indi-
cation and closes the channel with said error.
<br />
Thus, code like the excerpt shown for our middle process in the stream of processes would work
correctly even if the input channel comes from a pipe and not from a another process within the same program.<br />
<br />
The the post in the series will be about channel connections and channel multiplexors.<br />
<br />
</div>
</div>
</div>
<div>
<br /></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0tag:blogger.com,1999:blog-7546029155335878811.post-43767891785120798622014-01-25T16:23:00.001-08:002014-01-26T01:56:59.699-08:00Tiny go debug toolsA recent post from Rob Pike in his blog reminded me that I didn't post about some tiny tools I use to debug programs by enabling traces and conditional prints. I think these were borrowed or inspired by those written by him or by others using Go time ago.<br />
<br />
These are packaged in a tiny <span style="font-family: Courier New, Courier, monospace;">git.lsub.org/go.git/dbg.go</span> package.<br />
<br />
First, this is a well known idiom to trace function calls, only that I modified it to report also the file and line number in a convenient way.<br />
<br />
It is used as in this excerpt<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">func ReadMsgsFrom(r io.Reader, c chan<- []byte) (int64, error) {</span><br />
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>defer dbg.Trace(dbg.Call("ReadMsgsFrom"))</span><br />
<div>
<span class="Apple-tab-span" style="font-family: 'Courier New', Courier, monospace; white-space: pre;"> </span>...</div>
<div>
<br /></div>
<div>
And produces messages like these ones:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">bgn ReadMsgsFrom nchan.go:116</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">end ReadMsgsFrom nchan.go:116</span></div>
</div>
<div>
<br /></div>
<div>
The functions are like follows:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">// For use as in defer dbg.Trace(Call("funcname"))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func Trace(s string) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Printf("end %s\n", s)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">// For use as in defer dbg.Trace(Call("funcname"))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func Call(s string) string {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if _, file, lno, ok := runtime.Caller(1); ok {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>rel, _ := filepath.Rel(cwd, file)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>s = fmt.Sprintf("%s %s:%d", s, rel, lno)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Printf("bgn %s\n", s)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return s</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
The second tool is a couple of functions that enable prints only if a flag is set or if another function says so. They are used like in</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">var Printf = dbg.FuncPrintf(os.Stdout, testing.Verbose)</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func TestChanSend(t *testing.T) {</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>...</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Printf("receiver %v\n", msg)</span></div>
</div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
<div>
<br /></div>
<div>
The variable name should reflect that it is a conditional print, but in this case I took the code from a testing file, which prints only if verbose is set during the test. The idea is that you can declare as many print functions (variables) you want to print conditionally when certain flags are enabled.</div>
<div>
<br /></div>
<div>
This is the code from the debug package</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">type PrintFunc func(fmts string, arg ...interface{}) (int, error)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">/*</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Return a function that calls fmt.Printf only if fn returns true.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>To be used like in</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>var Printf = verb.PrintfFunc(testing.Verbose)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>...</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>Printf(...)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> */</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">func FuncPrintf(w io.Writer, fn func()bool) PrintFunc {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return func(fmts string, arg ...interface{}) (int, error) {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>if fn() {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return fmt.Fprintf(w, fmts, arg...)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>return 0, nil</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>}</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
The function returns a function that prints, only that it prints only if the function given as an argument says so. Thus, when you declare your print function variable you can set the condition to trigger the print and the writer the print should use. The rest of the code is relieved from the burden of testing the condition or typing more to use a particular output stream.</div>
<div>
<br /></div>
<div>
There is a similar function (returning a conditional printing function) that takes a pointer to a boolean flag instead of a function, for those cases when checking a flag suffice.</div>
<div>
<br /></div>
<div>
<br /></div>
Anonymoushttp://www.blogger.com/profile/09676516538823450651noreply@blogger.com0