@Jason: pycurl looks interesting, thanks. I may use it for a later project. I regularly use command-line curl for testing and debugging.

@Adam: Thanks for the pointer! But take a look at the commit message, “Restore suport for set_debuglevel” (sic). The original debuglevel and print statements go back to 1994 at least…

@Tom: I’ll definitely look at lazr.restfulclient, thanks.

@Ben: Our server portion is running under mod_wsgi with a mix of processes and threads. In any case, I don’t think I’d want to disallow threads in my application to solve this particular problem, though there may be lots of other good reasons to consider processes in favor over threads. Regarding Ruby, thanks for the suggestion. I’ll leave it at that.

@Stu: The point of the blog post was to ask for help and to express dismay at the current implementation. I will also pursue filing a bug, but for the sake of a fixing a future release, which is a different goal.

@Andrew: THANK YOU! I knew that I could replace sys.stdout to redirect print, but I didn’t consider that duck typing might actually work in this context. I looked at Alex’s implementation and also a 1994 email from Guido (see http://n01se.net/paste/sM72 ) and I now have a working implementation:

class LocalCapturingWriter(threading.local):
    def __init__(self, fp):
        self._fp = fp
        self._stringio = None

    def start_capture(self):
        self._stringio = cStringIO.StringIO()

    def stop_capture(self):
        v = self._stringio.getvalue()
        self._stringio.close()
        self._stringio = None
        return v

    def write(self, str):
        if self._stringio:
            return self._stringio.write(str)
        else:
            return self._fp.write(str)

sys.stdout = LocalCapturingWriter(sys.stdout)

I’m not sure yet if this minimalistic approach will be sufficient or if I’ll need to implement __setattr__ and maybe some other special methods. My testing suggests it’s enough. Immediately before calling Http.request() I call sys.stdout.start_capture(), then I call sys.stdout.stop_capture() afterward to fetch the trace and restore sys.stdout for this thread.

When I’ve had a chance to test this more thoroughly, especially in the context of our app rather than just a test program, I’ll update the post to include this info. Thanks again!

(Or even submit some patches too)

That solution might be more complete than you need, since it handles the “softspace” special attribute in file handles. That will only come up in httplib.py when the line ‘print “header:”, hdr,’ (because of the trailing “,”), but in this case I think that’s a newline so it’s not that important and you can fake a solution in other ways.

You can also get the thread id and prefix every output line with it, in order to help reassemble the debug statements.

http://docs.python.org/library/multiprocessing.html

These packages are quite polished, they were written for implementing Launchpad’s restful services but are designed as a general purpose just-add-water package for exposing Python objects as RESTful services and creating a pythonic client library.