By Zed A. Shaw

I'm going to make the case that performance testing localhost is the best usability test for your server software you could make. This is contrary to pretty much anyone who's written a server nobody uses, but hopefully you can see why I find this important. I also give an example: MySQL vs. PostgreSQL popularity.

You Care About UX?

When I do my initial performance testing I always test the speed of localhost operations before I do anything else with my servers. In Mongrel2 and Mongrel(1) I went so far as to get in there and make sure the thing runs for all the different localhost test procedures people might run. Obviously I don't only test localhost, and have a large array of tests I run on a little crap cluster of old laptops and VPS slices I've got. In fact, right now mongrel2.org is running mongrel2 and I post links off of it to twitter to get it smashed.

The problem is, when I post metrics from localhost people scoff at them. The assertion is that I must only test by having a 20 node EC2 cluster with a wide array of tests. Well, apart from the confounding and difficult and cost of this design, it misses a very important purpose of the "localhost perf test": programmer usability.

The truth is, my servers get used by programmers off localhost way more than they get used in large clusters of machines. By making sure that my servers are snappy and responsive under localhost operations, I'm making sure that programmers and people who build things with my software are having a good experience. If the server seems snappy when you're coding, then you're happier and like the server better.

The first thing localhost perf naysayers will claim though is that means "real world" performance suffers. Well, true, but only if you don't also test real world operations. I do that as well. Like I said, I run a wide range of tests as I near releases, including the following:

1. localhost testing
2. single machine to machine over single cable
3. wireless "crappy" connection performance
4. cluster proxy performance
5. end to end performance of various web apps
6. fuzz testing and security audits
7. unit tests that have performance outputs
8. valgrind callgrind analysis for micro-optimizations

You don't have to take my word for it though, just check out the source in examples/kegogi to see the tool we're building (mostly being hacked on by agartrell) for doing this very thing. It's a port to C of my older rfuzz project. I used RFuzz to test the original mongrel, and part of that testing was all of the above tests. As I said earlier we're running mongrel2.org off it right now, and I'm setting up even more testing procedures.

However, all of that testing isn't nearly as important as getting localhost snappy, and doing that is all about usability. As I said, programmers use localhost way more frequently and for longer periods of the day than any other deployment scenario.

If localhost doesn't "feel good" then they won't use my stuff.

Just Ask PostgreSQL

PostgreSQL is an awesome database server, and my preferred one. It is stable, handles high load really well, has awesome features, and has improved so much from it's early "config from hell" days. It's sad it isn't more popular than MySQL because it really is the better RDBMS out there. But, there's a solid reason MySQL is more popular:

PostgreSQL "seems" slower than MySQL in localhost performance.

When a programmer spends all day with MySQL, they feel good because it actually is quicker for their small localhost operations. If you have to spend 8-16 hours a day reloading web pages and running queries, a faster RDBMS is a big deal. PostgreSQL just feels like a dog on localhost. It doesn't matter if PostgreSQL is actually faster at higher loads and works better under pressure because for the 90% of the work a coder does with a database, MySQL is faster and more enjoyable.

I agree, this is a little stupid, but if you want to make a good product you have to understand your users. And my users are partially programmers who have to reload their applications over and over and over off localhost and who hate configuration files. If I want them to be happy, I have to make sure localhost with Mongrel2 is snappy, easy to use, and works well.

If you look at many other servers and products, perceived performance on a local machine almost always is the deciding factor of user adoption. Especially if what you're trying to do is take on an established competitor. The best way to do that is say you're like them plus two or three things, and one of those things can easily be performance...because people just really can't judge performance visually.

Balancing Programmers With Operations

A lot of the work in making Mongrel2 usable is that I have to balance what operations needs with what programmers need. Programmers want a server that's fast as hell on localhost so they don't have to wait as much. Operations want a server that is tunable and will scale up to large numbers of requests without crashing. It's actually very hard to balance these two in a server design.

For localhost performance you have to make individual requests blast out as fast as possible, no matter what other requests are doing. You basically want a performance curve that is erratic but more frequently higher so the user (programmer) thinks it's fast. It's all about perception of their day jobs and your server making their day jobs easier.

For "real world" performance though you want a server that scales out and balances the load so that it serves things fast, but does it consistently as load increases. This performance profile is all about solid stable ramp up and conservative resources without losing speed. In addition operations folks need tunable parameters to handle their specific load and a way to easily cluster the servers.

With Mongrel2 I've done most of my research of superpoll, the configuration system, and ZeroMQ as my strategy for satisfying both parties. I focused on programmers first because they're the early adopters, and then I'll tune it up for operations and spend more time making it stable.

Right now we're working on things operations folks need. Stability, scalability of I/O, better epoll tuning, more parameters like buffers, and using it in many "real world" deployments. Hopefully by the 1.0 release it'll be balanced between programmers and operations needs rather well.

Performance Testing Tools Suck Though

One very dark spot on this business of getting programmers to use a web server is that the tools they user (and frankly everyone uses) just plain suck. If you were to go check out httperf, apache-bench, or siege you'd find that they behave nothing like a browser. Hell, apache-bench defaults to HTTP 1.0. Yes, 1.0.

Because none of these tools act like a real browser, the numbers they're getting are a little off or just plain wrong. Combine that with really bad stats from apache-bench and seige (httperf's are awesome) and you've got a huge disadvantage making programmers happy with performance.

This also doesn't include problems of rendering performance, TCP stack performance on different machines, lame file descriptor limits, and many other problems with localhost testing.

With Mongrel2 I've had to run all these tools and try to make them work only because other programmers run these tools and expect them to work. It doesn't really matter if they all suck, it's what people use. Until a better tool comes along that does it right we just gotta suck it up to keep the users happy.

We will be trying to turn kegogi into a better testing tool, and hopefully make it act like a real browser and collect statistics correctly. Right now it's focused more on unit testing and fuzz testing, but we'll start working to make it do performance measurements better too.