What we need is a duck-typed replacement for sys.stdout that behaves like a writeable file, but also provides the ability to capture to thread-local storage. One of the ways to use threading.local is to subclass it. An instance of this subclass will have per-thread attributes, even if the instance itself is common to multiple threads.

Since StringIO intentionally doesn't implement isatty(), we need to make sure that gets passed through to the underlying file (we do this by catching the exception in getattr). And since we like seeing HTTP transactions when we're debugging, we include a writethrough mode that provides simultaneous capture and print.

import cStringIO, threading

class LocalCapturingWriter(threading.local):
    def __init__(self, fp, writethrough=False):
        self.__dict__['_fp'] = fp
        self.__dict__['_stringio'] = None
        self.__dict__['_writethrough'] = writethrough

    def start_capture(self):
        self.__dict__['_stringio'] = cStringIO.StringIO()

    def stop_capture(self):
        v = self._stringio.getvalue()
        self._stringio.close()
        self.__dict__['_stringio'] = None
        return v

    def write(self, s):
        if self._stringio:
            result = self._stringio.write(s)
        if not self._stringio or self._writethrough:
            result = self._fp.write(s)
        return result

    def __getattr__(self, name):
        if self._stringio is not None:
            try:
                return getattr(self._stringio, name)
            except:
                pass
        return getattr(self._fp, name)

    def __setattr__(self, name, value):
        if self._stringio:
            setattr(self._stringio, name, value)
        if not self._stringio or self._writethrough:
            setattr(self._fp, name, value)

And here's how to use it. First, the global settings:

import httplib2, sys

httplib2.debuglevel = 1

sys.stdout = LocalCapturingWriter(sys.stdout)

Then the code that runs in a thread to capture the debugging output. This will work as expected even in multiple threads simultaneously.

sys.stdout.start_capture()
try:
    response, content = \
        httplib2.Http().request("http://n01se.net")
finally:
    debug_trace = sys.stdout.stop_capture()

# Note that httplib2 doesn't include the content in its
# debug output.
debug_trace += "content: %r\n" % content

We're now using this in our Django app, with a custom Exception class (to hold the captured trace) and middleware that knows to look for it. The end result is that every time an exception occurs due to a problem talking to a backend server, the exception email includes the httplib2 trace. Yeah!

P.S. I wrote this entry less than a week after my previous entry, but then went on vacation and never managed to get it posted. Sorry for the delay...

One of the pieces of data I'd like to add to the email is the conversation between our server and the internal servers. On a development server, this is easy: Set httplib2.debuglevel=1 and watch the HTTP conversations scroll past on stdout.

On a staging or production server, one quickly discovers a crippling mistake made by the httplib authors: the library uses Python's "print" for debugging!

If the application were single-threaded, we could capture the trace by temporarily redirecting sys.stdout to an instance of StringIO (maybe using a context manager). Sure, it's more load on the server to capture the debug on every transaction, but I'll gladly pay that price for the hours we'll save when something goes wrong and we have the ability to debug it.

But it doesn't matter, because we haven't this option. Our app is multi-threaded and sys.stdout is global. We would have to serialize our HTTP transactions to prevent traces from being mixed together. Or fork to isolate sys.stdout. These aren't realistic approaches.

This sort of unfortunate shortcoming is to be expected in add-on libraries. After all, part of the reason they're not included with Python is that they don't necessarily meet the quality requirements of the core distribution. But I'm taken off-guard to find such an obvious shortcoming in the Python standard library. One of the things I'd hope to assume by using the standard library is a trust in the quality of the implementation, but a discovery like this forces me to question that assumption.

I'm pretty new to Python, so maybe I'm missing something. Is httplib a particularly poor example of the Python standard library? The existence of httplib2 seems to imply that (and also seems to imply that it's hard to get problems fixed in the core distribution). Maybe I need to find an add-on networking library that ignores httplib entirely...?

The problem is that the page's css has the following rules:

h1, h2, h3, h4 {
  color: #333;
}

pre {
  background-color: #fcfcfc;
}

Both these rules assume the usual black-on-white color scheme, but I'm using a gtk theme with a dark background and light foreground, which firefox respects. (Chromium doesn't have the issue because it enforces black-on-white defaults regardless of the gtk theme. I'm not sure how I feel about that; it fixes the problem but at the expense of ignoring my theme.)

This isn't to say anything bad about febootstrap, of course. I'm thrilled that somebody has finally written for Fedora what the excellent debootstrap has been providing for Debian and Ubuntu users for years! But I've come across this mistake enough times that Rich's site drew the unlucky number.

Movin' right along...

The basic process of moving a repo including all branches and tags is as follows:

git clone --mirror git@gitosis:my-repo.git
git --bare --git-dir my-repo.git push --mirror git@githubfi:my-user/my-repo.git

The above assumes that the git@githubfi:my-user/my-repo.git repo was created as an empty repo some point before the last command was run.

With the default config settings, git clone will create remote tracking branches for all branches found in the remote repo but will only create a local branch for the repository's currently active branch. In order to fully mirror the repository and all references (branches and tags) locally, you need to use the --mirror option which will also create the repo as a bare repository.

Rawhide!

A normal working copy has, at the top level, your checked out files and a .git/ sub-directory. A bare repository omits the working copy and has the contents of the .git/ directory at the top level. Bare repositories are intended to be used remotely (such as by a repository management system) and, by default, are named with a .git suffix to distinguish them as seen in the repo URLs above. (e.g. my-repo.git)

Since a bare repo has no working copy to sit in, we need to tell git where and how to find it when we call git commands against the repository, hence the --bare --git-dir my-repo.git options. The git push --mirror assures that all refs (branches/tags) will be pushed to the new location instead the default behaviors of only pushing refs (branches/tags) specified on the command line or pushing the branches that match by name between the local and remote repositories if none are specified on the command line.

More cowbell...

There's one further thing to mention that may be helpful. While transitioning from gitosis to GitHub:FI I took the opportunity to make our pre-receive hook a bit more stringent in its checking of emails, requiring that the committer email be from our corporate domain on non-upstream branches. In order to do this I had to use git filter-branch to do a little cleanup. Because we had a bare repo, the invocation looked a bit different. I'm showing a simplified version below:

git clone --mirror git@gitosis:my-repo.git
git --bare --git-dir my-repo.git filter-branch --commit-filter \
    '[ "$GIT_COMMITTER_EMAIL" = "joe@home.org"] && export GIT_COMMITER_EMAIL="joe@work.com"; \
        git commit-tree "$@"' \
    -- master devel staging v2.{30..45}
git --bare --git-dir my-repo.git push --mirror git@githubfi:my-user/my-repo.git

Note that history rewriting such as done by git filter-branch has implications for any repos that have been cloned out in the wild. Be sure you understand history rewriting before you use this command or there will be pain and sadness among the other developers. In our case, the above changes were done in a manner that was coordinated among the developers involved and was quite pain free. Note also that I did not rewrite the "upstream" branch which did not need to be purged of non-work addresses so I omitted it from the list of refs to be filtered. Further note that tags, since they are refs, also need to be rewritten so I passed our version tags to git filter-branch as well.

But if you find yourself in the unfortunate position of needing to daemonize an existing bash program, and you'd rather put off rewriting it in a more suitable language, keep reading! I found myself in that position recently and kept some notes.

Daemonizing a process consists of two primary tasks: forking to the background to return control to the shell, and preventing undesirable interaction between the process and the host. Rich Stevens enumerated the steps in his classic Advanced Programming in the UNIX Environment. Here's my summary of his formula with implementation notes for bash.

1. Call fork (to guarantee the child is not a process group leader, necessary for setsid) and have the parent exit (to allow control to return to the shell).

   Forking in bash is a simple matter of putting a command in the background using "&". To put a sequence of commands in the background, use a subshell: "( commands ) &". Note that bash doesn't provide any method for the child process to continue the same execution path as the parent, so the entirety of the child must be contained in the subshell. The easiest way to do this is implement the child as a bash function: "childfunc &".

2. Call setsid to create a new session so the child has no controlling terminal. This simultaneously prevents the child from gaining access to the controlling terminal (using /dev/tty) and protects the child from signals originating from the controlling terminal (HUP and INT, for example).

   Bash provides no method to call the setsid syscall for the current process. We have two less-than-ideal alternatives:

   1. The util-linux-ng package provides an external setsid command but this daemonizes an external command rather than the currently running script. It also makes collecting the PID of the child tricky because the setsid command will fork internally.

      Having said all that, if your application allows you to use the setsid command, it's a good choice because bash can't otherwise fully protect against the child process opening /dev/tty. It's still a good idea to redirect std* to prevent stray output to the terminal.

   2. Lacking the setsid syscall, there are steps we can take to partially protect the child process from the effects of the controlling terminal:
      1. Redirect std* to files or /dev/null
      2. Guard against HUP and INT by signal handler in child
      3. Guard against HUP by disown -h in parent

      Unfortunately without setsid there is no way to guard completely against a subchild opening /dev/tty until the terminal emulator exits, then /dev/tty will become unavailable.

3. Change working directory to / to prevent the daemon from holding a mounted filesystem open.

   Bash is good at this.

4. Set umask to 0 to clear file mode creation mask.

   I have to admit that I can't understand the point of this, in bash or any other language. It seems to me that the child will either set its umask explicitly before creating files, or it will set individual file permissions explicitly, or it will fall back on the caller's umask. In the last case, I want my inherited umask, not the wide-open zero.

   If anybody wants to explain a good reason for step 4, I'm all ears... Until then, it's commented out in my implementation below.

5. Close unneeded file descriptors.

   This step is fun in bash using eval and brace expansion...

With those notes in-hand, here's my implementation. There are two
functions here, "daemonize" for an external command using setsid,
"daemonize-job" for a function in the running script.

# redirect tty fds to /dev/null
redirect-std() {
    [[ -t 0 ]] && exec </dev/null
    [[ -t 1 ]] && exec >/dev/null
    [[ -t 2 ]] && exec 2>/dev/null
}

# close all non-std* fds
close-fds() {
    eval exec {3..255}\>\&-
}

# full daemonization of external command with setsid
daemonize() {
    (                   # 1. fork
        redirect-std    # 2.1. redirect stdin/stdout/stderr before setsid
        cd /            # 3. ensure cwd isn't a mounted fs
        # umask 0       # 4. umask (leave this to caller)
        close-fds       # 5. close unneeded fds
        exec setsid "$@"
    ) &
}

# daemonize without setsid, keeps the child in the jobs table
daemonize-job() {
    (                   # 1. fork
        redirect-std    # 2.2.1. redirect stdin/stdout/stderr
        trap '' 1 2     # 2.2.2. guard against HUP and INT (in child)
        cd /            # 3. ensure cwd isn't a mounted fs
        # umask 0       # 4. umask (leave this to caller)
        close-fds       # 5. close unneeded fds
        if [[ $(type -t "$1") != file ]]; then
            "$@"
        else
            exec "$@"
        fi
    ) &
    disown -h $!       # 2.2.3. guard against HUP (in parent)
}

The Clojure macro binding is frequently handy for mocking out functionality during testing, but this sometimes does not behave as desired in multi-threaded context. Fortunately, there's a solution...

For example, at work we have unit tests on functions that may attempt to talk to network services, which we'd rather they not do:

(defn send-request [request server]
  '(...do real RPC stuff here...))

(defn average-timestamp [time-servers]
  (/ (apply + (map #(send-request :get-timestamp %)
                   time-servers))
     (count time-servers)))

In order to test average-timestamp, we need a "mock" function that will stand in for send-request. This must be a function that takes a request and a server and returns a timestamp, just like the real send-request. For this example, it can take the timestamp itself as the "server" and simply return that timestamp. For bonus points we can make sure the request parameter is what we expect:

(defn mock-send-request [request server]
  (assert (= request :get-timestamp))
  server)  ; assume "server" is actually the timestamp

(mock-send-request :get-timestamp 5)
;=> 5

Using binding we can temporarily replace send-request with mock-send-request:

(binding [send-request mock-send-request]
  (send-request :get-timestamp 5))
;=> 5

(binding [send-request mock-send-request]
  (average-timestamp [5 15]))
;=> 10

So there's the background: a pretty normal way to mock out stuff in Clojure. What this doesn't address is when some clever co-worker (hi, Nathan!) realizes that average-timestamp would work better if it talked to multiple time-servers in parallel, and that this could be easily accomplished by replacing the use of map with pmap:

(defn average-timestamp [time-servers]
  (/ (apply + (pmap #(send-request :get-timestamp %) time-servers))
     (count time-servers)))

This is an easy single-letter change that indeed works quite well with the real send-request. But when we try to use binding to mock it out, we run into problems:

(binding [send-request mock-send-request]
  (average-timestamp [5 15]))
; java.lang.ClassCastException:
;   clojure.lang.PersistentList cannot be cast to java.lang.Number

It's not obvious from the error message, but what's happening is our original un-mocked send-request is getting called. This is because binding only has an effect on the current thread, but pmap causes send-request to be called in other threads.

The solution is simple enough in Clojure 1.0 -- you simply mock out pmap as well, and since the API is identical to map, it's quite easy to do. But as you can see, this does us no good at all in recent versions of Clojure:

(binding [send-request mock-send-request
          pmap map]
  (average-timestamp [5 15]))
; java.lang.ClassCastException:
;   clojure.lang.PersistentList cannot be cast to java.lang.Number

The reason is that starting with Clojure 1.1, most clojure.core Vars are linked directly into code that uses them. This means that our definition of average-timestamp above links directly to the actual definition of clojure.core/pmap, not just the Var that points to it, thus attempts to rebind the Var with binding are futile.

But do not despair, there is a solution even for this. All you need to do is tell Clojure not to directly link pmap when it compiles average-timestamp. This is done by adjusting pmap's metadata before average-timestamp is compiled, like so:

; Set pmap to be dynamically linked
(alter-meta! #'pmap assoc :dynamic true)

; Must now re-define the function that uses pmap
(defn average-timestamp [time-servers]
  (/ (apply + (pmap #(send-request :get-timestamp %)
                    time-servers))
     (count time-servers)))

; Finally, our mocking out of pmap works
(binding [send-request mock-send-request
          pmap map]
  (average-timestamp [5 15]))
;=> 10

The best place to put the alter-meta! depends on your particular use case. You might want to figure out how to do the alter-meta! only when in testing and not in production. Remember that the metadata on pmap is global -- you're changing the only metadata that clojure.core/pmap has, so the namespace you're in when you do it makes no difference at all.

In our case at work however, the performance impact of leaving pmap dynamic all the time was not enough to worry about. Dynamic linking is still pretty fast and is fine even for our production uses of pmap in this particular code base. So we simply put the alter-meta! at the top of the file that used pmap and got on with our unit testing.

The Joy of Clojure

Thinking the Clojure Way

by Michael Fogus and Chris Houser

I bought a Sony "Dream Machine" Clock Radio about a year ago and have liked it except for one problem: even when the brightness is set to "low", it's still too bright for my taste. Each night I have to rotate the clock away from me on the nightstand so I can fall asleep.

This morning I applied Insta-Cling Windshield Strip Professional Tint Film to the front and I think it's not bad for $4.88 and a few minutes with an x-acto knife!

Here's the before/after ("before" courtesy of Amazon since I didn't think of taking a photo):

This weekend I switched from irssi to weechat. The main reason I switched away from irssi was to gain support for protocols other than IRC. On the way to weechat, I tried Pidgin but I was immediately frustrated by two things.

First, I really love running my IM client in screen. I move regularly between three machines and I want persistent connectivity despite changing work context. Screen has provided that beautifully over the years and I don't know of any truly comparable technology for graphical programs.

Second, Pidgin seems to be barely customizable. For example, how would you like your timestamps? You have three choices:

• (HH:MM:SS)
• occasional iChat-style interruptions
• no timestamps

The Pidgin FAQ calls out plugins to customize timestamps but they're woefully underpowered. What I want is to supply a strftime-style format string: %H:%M. It seems preposterous to me that Pidgin wouldn't provide this capability, so maybe I'm just missing it?

In any case, irssi and weechat both make this and other customizations easy. It took about one minute to find and change the setting in weechat:

/set weechat.look.buffer_time_format "%H:%M"

Once I got weechat running, I found to my chagrin that I'd imagined weechat's support for other protocols. The front page mentions Jabber but it's still a work in progress; there's no stable support for anything other than IRC.

I was saved by bitlbee, a standalone server that gateways IM protocols (XMPP/Jabber, MSN Messenger, Yahoo! Messenger, AIM and ICQ) to IRC. Installing and configuring bitlbee took all of five minutes since it's in Ubuntu and there's a super-easy quickstart guide.

Of course this means that I could have stayed on irssi! But weechat seems to be a worthy replacement. It has a built in nicklist which has always been missing from irssi. It handles filtering better than irssi; whereas irssi's /ignore discards information, weechat's /filter simply hides it and you can toggle filtered text with alt+=. That means I can filter more aggressively because I don't have to worry about irretrievably missing something important.

We are, after all, searching for signal. (And good luck finding it in this post.)

One thing I liked from my brief experience with pidgin was libnotify alerts. You know, that little box that pops up so you don't need to keep your IM client on the screen at all times:

There's a weechat notify script that provides this functionality but it had some bugs. No problem, fixed them, then tried to submit my update to the site. The upload form suggests using pastebin, so I broke out my own nopaste tool and...

It failed. Where I expected to get a link, I got nothing! It seems Pastebin.com recently switched to a custom httpd:

$ curl -I http://pastebin.com/
HTTP/1.1 200 OK
Content-Type: text/html; charset=iso-8859-1
Date: Mon, 22 Feb 2010 20:37:36 GMT
Server: TorrentFly.org Custom Httpd

Curl sees "HTTP/1.1" and tries to be polite by using the 100-continue Expect request-header field. Unfortunately the TorrentFly.org Custom Httpd doesn't grok the request and halts the transaction with 417 (Expectation Failed).

Fine, it's an easy fix to force HTTP/1.0 and work around the broken server:

--- a/nopaste
+++ b/nopaste
@@ -52,7 +52,7 @@ main() {

 docurl() {
     declare u
-    u=$(curl --silent --show-error -o /dev/null -w "%{redirect_url}\n" \
+    u=$(curl -0 --silent --show-error -o /dev/null -w "%{redirect_url}\n" \
         --form-string "poster=$opt_nick" --form-string "format=$opt_language" \
         --form-string "expiry=$expire" \
         ${opt_parent:+--form-string "parent_pid=$opt_parent"} \

This reminded me that nopaste should be in externally-available source control so I created a github project for it. This is the first time I've used github to publish a project of my own and I can only say: Wow, that was easy.

I thought it would also be nice to publish proper releases of nopaste. In the past I've simply provided the individual file with a version number appended. Since I'm using git, I should be able to do something nifty with tags, right?

Enter tagver, a script to generate a "meaningful" version number based on tags, revisions and dirt. I've re-invented this wheel a few times, for example adding mercurial support to setlocalversion, but this is the first time I've made it a project in its own right.

Using tagver I released tarballs and an rpm for nopaste version 2.3 containing this trivial fix. You can find them both at http://agriffis.n01se.net/nopaste. I'm not really satisfied with the Makefile that calls tagver, but it'll do for now, at least until I look into integrating tagver into an autotools-driven distribution. Or cmake or scons or cons or whatever...

A normal Clojure REPL (or prompt) in a terminal window is by default a bit touchy about infinite seqs, deeply nested structures, and long-running operations. Any of these can cause your REPL to wander off into the weeds, busy spinning, perhaps printing pages of useless data, with the only apparent remedy being to press CTRL-C, which generally kills the entire JVM, Clojure and all, and dumps you back at your operating system prompt.

It's unfortunate that this is the default, but Clojure's youth shows particularly when it come to tools and settings like this. Happily it's sufficiently mature to provide several solutions that are not difficult to apply.

(set! *print-length* 103)

(iterate inc 0)
;=> (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
    24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
    45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65
    66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86
    87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 ...)

(set! *print-level* 15)

(let [x (atom 0)] (reset! x {:deeper x}))
;=> {:deeper #<Atom@4cb533b8: {:deeper #<Atom@4cb533b8:
    {:deeper #<Atom@4cb533b8: {:deeper #<Atom@4cb533b8:
    {:deeper #<Atom@4cb533b8: {:deeper #<Atom@4cb533b8:
    {:deeper #<Atom@4cb533b8: {:deeper #}>}>}>}>}>}>}>}

Run-away exercise bikes

But these settings only help take control of run-away printing. Sometimes the REPL gets hung up doing something other than printing, sitting there spinning away like a stationary bike, and neither of these settings will help you. My final advice is to use repl-utils to register your REPL thread as killable by CTRL-C:

(require 'clojure.contrib.repl-utils)
(clojure.contrib.repl-utils/add-break-thread!)

Now when you press CTRL-C, instead of shutting down the whole JVM, an exception will be thrown within the REPL thread, which is usually exactly what you need to halt run-away computation. Note that repl-utils does this using a deprecated Java API that is described as unsafe. And yet it remains useful for just such circumstances. For example:

(deref (promise))

Look at that -- instant deadlock! Now press CTRL-C, and you get:

java.lang.Exception: SIGINT (NO_SOURCE_FILE:0)

...and you're back to REPL prompt. But now you've violated Java, so I'd recommend saving what you need to save and then restart your JVM anyway. If you don't, weird things can happen. For example if you run the above deref-promise example again immediately, it may fail with a slightly different exception, even before you press CTRL-C. Similar strangeness can occur if you press CTRL-C at the REPL when no operation is running, though in that case it usually just kills your next expression, whatever it is.

Hopefully by setting your *print-length*, *print-level*, and break-thread, you can feel more comfortable at the REPL with less fear of getting stuck.

The Joy of Clojure

Thinking the Clojure Way

by Michael Fogus and Chris Houser

There are a few books about Clojure in the works now. But despite the variety in authorship and publisher, they all seem to be primarily in the vein of a language tutorial, starting from a basis of "common knowledge" among most programmers and taking the reader to a level of basic familiarity with Clojure.

Michael Fogus and I think there is more to Clojure that has yet to be addressed by any of the books currently being written. So we've started writing "The Joy of Clojure". Our vision for this book is to impart not just what Clojure is, but why it's the way it is; not just how to use it, but how to choose well from among the wide variety of options Clojure provides. And along the way we hope you'll experience the wonder, amazement, and, yes joy that Clojure can bring to programming tasks that often feel like nothing more than chores in other environments.

I also wanted to take this opportunity to brag a little on Fogus. He's been a fantastic partner throughout this process -- I wouldn't even be involved if it weren't for him. He has put into our book many references to papers and books that I've never heard of, but that illuminate much of Clojure's design -- clearly he has read a good deal more than I have. Once we're done writing, I intend to use the Bibliography as a reading list -- I'm sure it will be enlightening. He is dedicated to producing the best book we can, and he's the only reason we're meeting any of our deadlines. I'm looking forward to the next few months as we finish this book together.

The Joy of Clojure

Thinking the Clojure Way

by Michael Fogus and Chris Houser