By Zed A. Shaw

Remember back when I said Mongrel2 would use sqlite3 for it's configuration system? Remember how everyone thought that meant they'd have to code SQL in order to configure Mongrel2? Remember how I kept saying no, I'd have some kind of configuration system that'd be better and more powerful than config files?

It's done, and it is fucking awesome. I've been using it for the last few days while I worked on the next demo (a ICY mp3 streaming handler) and the new configuration system works amazingly well. The design is basedon a Model-View-Controller design where:

• Mongrel2 == Controller
• Sqlite DB == Model
• View == m2sh config system

What's this m2sh you ask? Why, pure awesome is what. The m2sh is a simple Python command line program that acts as the View UI to a Mongrel2 config file. With m2sh you craft configurations using little Python scripts that are a lot like config files in other webservers. You can then use m2sh to start, stop, reconfigure, look at the config, create new ones, pretty much all the stuff you end up doing with janky git repos and shell scripts or etckeeper.

It works like this:

m2sh init -db myconf.sqlite
m2sh load -db myconf.sql -config myconf.py
m2sh start -db myconf.sql -host localhost -sudo
m2sh stop -db myconf.sql -host localhost

The config file looks like this:

from mongrel2.config import *

main = Server(
    uuid="f400bf85-4538-4f7a-8908-67e313d515c2",
    access_log="/logs/access.log",
    error_log="/logs/error.log",
    chroot="./",
    default_host="localhost",
    pid_file="/run/mongrel2.pid",
    port=6767,
    hosts = [
        Host(name="localhost", routes={
            '/tests/': Dir(base='tests/', index_file='index.html',
                             default_ctype='text/plain')
        })
    ]
)

commit([main])

And it is real Python, so you can hook into any code you need to figure out what this config has to do. NOTE: For those admins who hate executable configs we'll craft a non-executable style soon.

The end result is a fully working Mongrel2 database you can then deploy to your servers, but best of all, it's got a full transaction log system built in so you can look at any config and see who's been doing stuff to it:

# m2sh log -db config.sqlite 
[2010-07-18T04:14:53, mongrel2@zedshaw, init_command] /usr/bin/m2sh init
-db config.sqlite
[2010-07-18T04:15:06, mongrel2@zedshaw, load_command] /usr/bin/m2sh load
-db config.sqlite -config examples/python/tests/mongrel2_org.py
[2010-07-18T04:22:06, mongrel2@zedshaw, load_command] /usr/bin/m2sh load
-db config.sqlite -config examples/python/tests/mongrel2_org.py
[2010-07-18T04:23:32, mongrel2@zedshaw, load_command] /usr/bin/m2sh load
-db config.sqlite -config examples/python/tests/mongrel2_org.py
[2010-07-18T04:26:16, mongrel2@zedshaw, upgrade] Latest code for
Mongrel2.
[2010-07-18T18:05:59, mongrel2@zedshaw, load_command] /usr/bin/m2sh load
-db config.sqlite -config examples/python/tests/mongrel2_org.py
[2010-07-18T20:09:01, mongrel2@zedshaw, init_command] /usr/bin/m2sh
config -db config.sqlite -config examples/python/tests/mongrel2_org.py
[2010-07-18T20:09:02, mongrel2@zedshaw, load_command] /usr/bin/m2sh
config -db config.sqlite -config examples/python/tests/mongrel2_org.py
You have mail in /var/mail/zedshaw

That's what I've been doing the mongrel2.org site. You also have a commit command so you can add extra annotations:

# m2sh commit -db config.sqlite -what mongrel2.org -why "Upgrades."

This of course doesn't replace using git for the source that builds configs, but it gives you a nice log of what happened on your actually deployed servers, and solves the very real problem that it's difficult to craft configurations for heterogeneous hosts out of a source control repository.

With m2sh you can store the source to your configs in something like git or mercurial, and then use that source to generate the actual deployment config special for each host, but still know who did what on each deployment.

This separation of config files as View from config storage as Model means that you can solve the problem of storing your configurations in a repository, but generate different ones for each system.

In addition to this, there's ways to dump configurations, see what servers are registered, see if it's running, and more management tools and help will be on the way. You can read more about it in the GettingStarted documentation for early adopters.

Python Is Just My Preference

The other powerful thing about m2sh is that it's written entirely in Python and uses the Storm ORM to handle the database. It does only a couple of raw SQL operations when initializing and loading configs for speed, but otherwise it's using the ORM to work with the configuration.

This means, you could probably use any language you want to make your own tool. Don't like Python? Fine, use what works for your deployment. Mongrel2 is language agnostic even down to the configuration system. Of course hopefully people are smart about this, but if you need to write your own you can.

The important part is that this is all just a View of the Model-View-Controller design, so you can craft what works best for you. Create automation, use a different language, setup monitoring systems, automate deployments, merge into your setup.

Nothing about using SQLite restricts you from using configuration files or frankly any dream configuration you want. It's just a data storage mechanism similar to, but more powerful than, what Postfix and other MTAs have been doing for years.

So, for all of you who wanted config files, you've got them, and then some.

Even More Configuration Working

Mongrel2 also now respects all the configuration options you can give it, including chroot directories, pid files specs, handler specs, and everything else that the database has. If I missed something then we'll be adding it later.

This means that, apart from some bugs and not very good logging, it's got most of what you need to actually set it up and run it.

I plan on cutting over my major sites to it next week and eating my own dogfood entirely.

Daemonizing And Unixy Stuff

All of the usual unixy stuff is working and seems all the bugs are worked out so far. Mongrel2 will daemonize, chroot, and then drop privileges to whoever owns the chroot.

It also properly serves files from different directories and negotiates their realpath so that it doesn't leak files, even if not in a chroot. That means you can fire it up in a test situation and work with it still to make sure your config is good.

Full 304 Support

A big chore from last week was to also get the 304 support tight, and it's working great so far. You can hit the chat demo which is entirely self-hosted to see how they work. Basically, Mongrel2 is now a real web server.

Complete Async Streaming mp3stream Demo

There's also a prototype ICY (shoutcast) streamer that uses the Mongrel2 async abilities to stream mp3 files to multiple connected clients. The code is very rough, but works alright if you use MPlayer with other players really, really, hating the simplistic ICY implementation I've done.

You can check out the Python code for the sample in the repository or in the source at examples/mp3stream. It's rough though, so be gentle.

The main thing to understand about this code is that single requests are coming in off HTTP, and then one handler is responding with sequenced streams to all connected players at once. I cap it at 20 just to keep things stable, but the fact that you can do this is pretty damn hot. It also eats up almost no CPU to do this. If I wanted though, I could start 10 handlers and they'd all deal just fine.

A Big Announcement Tomorrow

I'm going to hack on this stuff today and get it all cleaned up, but tomorrow I'll have a pretty big announcement. Wait for it, it's gonna be good.