notes on a python webserver

Last week I created a python webserver as a patch for make talos-remote.  This ended up being frought with performance issues, so I have started looking into it.  I based it off of the profileserver.py that we have in mozilla-central, and while it worked I was finding my tp4 tests were timing out.

I come to find out we are using a synchronous webserver, so this is easy to fix with a ThreadingMixIn, just like the chromium perf.py script:

class MyThreadedWebServer(ThreadingMixIn, BaseHTTPServer.HTTPServer):
    pass

Now the test was finishing, but very very slowly (20+ minutes vs ❤ minutes).  After doing a CTRL+C on the webserver, I saw a lot of requests hanging on log_message and gethostbyaddr() calls.  So I ended up overloading the log_message call and things worked.

class MozRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    # I found on my local network that calls to this were timing out
    def address_string(self):
        return "a.b.c.d"

    # This produces a LOT of noise
    def log_message(self, format, *args):
        pass

Now tp4m runs as fast as using apache on my host machine.

Advertisements

4 Comments

Filed under testdev

4 responses to “notes on a python webserver

  1. It would probably be better to overload address_string() instead since that’s where the issue comes from. It is pretty stupid however that this module always performs DNS lookups without a possibility to switch them off.

    • elvis314

      For my purposes log_message is too noisy, but it would be better to have the option. I will see how this works out when I get a patch ready to land, maybe just doing the address_string() only will be sufficient, or I could do a test at startup that ensures fqdn lookup will be available.

  2. Just curious, did you try just using the default http server provided by python? If so, how did that do for performance?

    • elvis314

      I ended up using BaseHTTPServer for the server and SimpleHTTPResponseHandler for the responses. I found that using BaseHTTPResonseHandler required me to basically rewrite SimpleHTTPResponseHandler. Nevertheless, the performance of either was horrible until I removed the calls that were timing out to gethostbyaddr() from log_message()

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s