By Zed A. Shaw

I spent the last few days figuring out how to go about doing something that could be "superpoll", and trying to hedge my bets on the idea. Ultimately I wanted just a poll-like abstraction that either I could have do the dual poll/epoll thing, or just say screw it and compile a version for whatever one you've got that works best.

Even though my goal from earlier this week was to try out the whole poll and epoll combined, I'm not an idiot. I'm not going to build something without extensive testing, assurances, and being able to back out of it if it doesn't work. It'd be stupid to bet the whole project on some idea that might work, and frankly, anyone who tries to tell you that Mongrel2 will have poor performance because of a couple of blog posts with me wondering outloud is full of it.

What I needed to do first is get the current poll usage out of the libtask coroutine library and into its own little abstraction it could hide behind. I had a few false starts, and then managed to boil it down to enough of an API to just abstract away what the tasks need to do poll, but in a way that I could then extend to use epoll, kqueue, etc.

In no way was I trying to add features. The first problem I could have is if I do a bad API then it can kill the performance on its own, even without epoll or any crazy schemes. In order to remove confounding I wanted to keep all the current polling mechanisms the same, and only test the new abstraction. If the performance when down after the refactoring, then I'd have to fix it. If it was the same or got better (unlikely) then I could safely move on with trying different internals.

The new abstraction works well enough, although it's missing a few pieces it'll need before I can use it in a generic way to support many different polling mechanisms. It's got a few bugs, but that's expected. To give you an idea of the performance, here's what a very simple test gets:

Total: connections 10000 requests 10000 replies 10000 test-duration 0.918 s
Connection rate: 10894.8 conn/s (0.1 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 0.1 avg 0.1 max 5.5 median 0.5 stddev 0.1
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000
Request rate: 10894.8 req/s (0.1 ms/req)
Request size [B]: 79.0
Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 0.0 transfer 0.0
Reply size [B]: header 192.0 content 23.0 footer 0.0 (total 215.0)
Reply status: 1xx=0 2xx=10000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 0.34 system 0.58 (user 37.0% system 63.2% total 100.2%)
Net I/O: 3128.0 KB/s (25.6*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

This is just requesting a basic file from /test/sample.json. Nothing big or fancy, just to make sure it can even do that. This performance matches the original code's performance, except sometimes it does this:

Total: connections 10000 requests 9278 replies 9278 test-duration 12.723 s
Connection rate: 786.0 conn/s (1.3 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 0.1 avg 1.4 max 8848.0 median 0.5 stddev 97.0
Connection time [ms]: connect 1.3
Connection length [replies/conn]: 1.000
Request rate: 729.2 req/s (1.4 ms/req)
Request size [B]: 79.0
Reply rate [replies/s]: min 729.1 avg 729.1 max 729.1 stddev 0.0 (1 samples)
Reply time [ms]: response 0.0 transfer 0.0
Reply size [B]: header 192.0 content 23.0 footer 0.0 (total 215.0)
Reply status: 1xx=0 2xx=9278 3xx=0 4xx=0 5xx=0
CPU time [s]: user 2.63 system 10.04 (user 20.7% system 78.9% total 99.6%)
Net I/O: 209.4 KB/s (1.7*10^6 bps)
Errors: total 722 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 722

Yeah, don't know what the hell that is but I'll have to track it down and fix it. Most likely it's just contention on the machine between httperf and Mongrel2, but I'll try to rule it out. It at least completes the requests and works so the API itself doesn't seem to slow things down.

This code is still rough so I'm not all that worried about it. It was more important to get it working and get the refactoring done before I did much more with it. Now that it's working I'll be able to tighten it up and get some testing going.

Instrumentation And Testing

After ripping out the use of poll in libtask the code for using it became much cleaner and nicer. That's a good sign you're doing it right. When you get more out of less code after you refactor. If you're refactoring and you get more code and it all does the same thing, well not so good an idea usually.

What I did next is a bit of exploration into what kind of ATR (active/total ratio) I'd get on various overloaded servers. To do that I just tossed in some simple prints that spewed out the ATR and various stats at different intervals and then tried to kill it in various ways.

I eventually had to run it under valgrind so that Mongrel2 slowed down enough that it could be overloaded. Otherwise I would need quite a few more CPUs than I had to make Mongrel2 choke. What I was shooting for is a simple test where I pound Mongrel2 until it stumbled and then open sockets started to pile up so poll got overladed.

Once I had that I ran it a few times, got more and more data, then did some basic graphs. I'm not particularly looking for anything, just exploring and look at what I might get from the superpoll API I've created. Things like potential thresholds, odd results, situations I could optimize, and just anything that's interesting.

NOTE: In no way should you assume that I am proposing this is a "real world" experiment. It is exploration. Just open musing about what could be, not a doctoral research paper published in Nature to get my Ph.D. Not a paper to a medical journal on a new cancer treatment. This is a blog with me talking about stuff I'm interested in. Adjust your expectations accordingly.

First graph to check out is the active and total levels showing the number of sockets that are getting active on each run through poll, and the total:

Not a lot going on here, just you can see how I hit the server, and it slowly climbed in the number of sockets it had to handle until it had a sudden burst toward the end when it was overloaded.

Alright, so how's that impact the ATR levels during the same time:

The red line is where 0.5 (50%) is located, and you can see that what happened is Mongrel2 was doing good, with the ATR being mostly about 50%, then around the 10k request it stumbled and it starts to behave erratically. The idea with using a poll/epoll combination is that in this implementation that's just using poll Mongrel2 has a hard time recovering from this. But, if it used epoll and poll combined then it'd be able to crank on the sockets it can deal with and leave the dead weight to epoll. Thus this graph would be "smoother".

Another thing to check out is the numerics of the ATR data:

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.0010  0.1760  0.3330  0.4826  1.0000  1.0000 

You can see that the majority of the ATR is near the 100% range, and then tails off down to very small ATR levels.

Looking at this I can see that I'd want to do more analysis on what the criteria for moving between them should be, and maybe it's something as simple as another part of the Mongrel2 code, like the connection state machine, tells the superpoll where to put it.

I also wanted to see if there's any correlations between the total or active levels and the ATR. Obviously since the ATR is using those in its own calculation they'll have some impact, but taking a look at a graph or two can help get an idea of what's going on between them:

Here's we see the ATR correlated with the active levels, and really you can see that the active levels don't have much promise in predicting ATR. The green line in this graph is the mean, and the two red lines are standard deviations from the mean. I am curious about the sudden bursts on the top right of the graph, but I'll have to look at that more later.

Looking at the ATR when you use total to predict it you can see it's a little better:

This kind of shows you what you'd expect, that as the number of open sockets starts piling onto Mongrel2 the ATR gets worse and worse. I also notice that the mean (green line) seems to be where the server starts to have trouble, and maybe the lower red line (mean - std.dev) is where epoll should kick in to save the day.

Further Exploration And Questions

Maybe it's possible to use the mean-std.dev of ATR levels to determine when to shunt sockets off to epoll out of poll?

I'm curious about simulating other loads and connection types, so I may write a little Python script that does a bunch of JSSocket connections and hangs out as a way to control active and idle socket levels.

I'm curious how all these super neckbeards with their awesome "real world" test things like this without using localhost? Do they simply surround themselves with EC2 nodes and never leave the house? The mind boggles.

One thing I'm happy about is the new API doesn't hurt performance, but I'm not so sure if it'll support more than just poll. I'll have to massage the API over the next week so expect it to change.

I'm hoping that if the epoll+poll stuff doesn't work out that it'll be easy to then just compile one or the other and be done with it. The superpoll API will probably work well as an abstraction for either, so it's a good bet.

Next up, fine tuning and trying the epoll, then more serious testing.