By Zed A. Shaw

I finally refactored the guts of Mongrel2 so that it is now configurable via a sqlite3 database. It's currently running stable at the chat demo and has all the necessary pieces to now complete the actual file serving and async routing parts. So far the choice to use sqlite3 for the "config file" is working out good for the code, but it remains to be seen if it'll work by itself for actual deployments. I thought I'd write up some of my thoughts on this idea, and also answer some common questions.

The first word of warning is that obviously this is going to change. It's an interesting idea, but it has potential to go horribly wrong if it's not constantly usability tested. If you have a knee-jerk reaction to this that's based on your dislike of SQL, just hold off on that. I wouldn't force SQL on anyone who didn't want it, so expect great tooling for this to come out.

Why Sqlite?

I want Mongrel2 to be very language agnostic, and that means you should be able to configure and manage it with your favorite ops friendly language. I'd rather have people work against a known and migrateable SQL database than force everyone out there to try to write hackjob tools that munge an ever changing config file format. Hopefully this will make Mongrel2 an operations wet dream where servers can be managed easily, have their configurations migrated (yes migrated), validated for consistency, and even live reconfiguration without restarts.

However, there's some problems with this design that I'll have to deal with before it's totally ready for general usage. Here's the ones I've identified so far.

People Hate SQL

It seems you say "SQL" and suddenly a massive ball of bile fills people's throats. They work around having to know SQL with every possible maneuver available to them. Whether that's wrapping their SQL in Ruby and Python clothes, or just turning the whole database into a giant Hastable, most programmers will avoid SQL at every cost.

It's sad really because learning SQL is kind of like learning Lisp, Prolog, or C. You learn something from it, a new way of thinking. You start to learn to think in terms of sets and operations on sets. You learn about good data structure designs and how to map those to people's ideas.

Anyway, I will say right now that there will be three interfaces to the Mongrel2 configuration system:

1. The Neckbeard interface, which is raw SQL for the super hackers.
2. A Mundane interface, which will be a shell that can manage the config.
3. A Designer interface, which will be a snazzy web application.

To make things easy on myself, the Neckbeard interface is already done and you can play with it now. Here's a sample configuration that's running the mongrel2 chat demo:

INSERT INTO server (uuid, access_log, error_log, chroot, default_host, port) 
    VALUES (
        'AC1F8236-5919-4696-9D40-0F38DE9E5861',
        '/mongrel2/logs/access.log',
        '/mongrel2/logs/error.log',
        '/var/www',
        'mongrel2.org', 6767);

INSERT INTO host (server_id, name, matching)
    VALUES (
        last_insert_rowid(),
        'mongrel2.org',
        '(.*).mongrel2.org'
    );

INSERT INTO handler (send_spec, send_ident, recv_spec, recv_ident)
    VALUES (
        'tcp://127.0.0.1:9999',
        '54c6755b-9628-40a4-9a2d-cc82a816345e',
        'tcp://127.0.0.1:9998',
        ''
    );

INSERT INTO route (path, host_id, target_id, target_type)
    VALUES (
        "@chat", 
        (select id from host where name="mongrel2.org"),
        last_insert_rowid(),
        "handler"
    );

INSERT INTO proxy (addr, port) VALUES ( '127.0.0.1', 80);

INSERT INTO route (path, host_id, target_id, target_type)
    VALUES (
        "/chat/",
        (select id from host where name="mongrel2.org"),
        last_insert_rowid(),
        "proxy"
    );

Yes, there is no way in hell I'll let this be the only way to configure Mongrel2, but for now, this works great.

It Will Be Slow

Some folks have worried it'll be too slow, because they actually aren't understanding the design I'm going for here.

First off, in my last post about TSTs in Mongrel2 I showed you how Mongrel2 will have some damn fast and sexy routing. This routing will blow the doors off most everything out there and be very easy to use at the same time. Now, if I spent a whole week developing that do you really think I'd load the routes out of SQLite every single request?

I am known for making fast and tight code. There is no way in hell I'd have SQLite be some kid of bottleneck on performance. Never.

Maybe there would be a debug mode where the whole configuration is loaded for each request, but that's a long time coming. So, this thought should be banished from your mind now. Done.

It's Not Diff/Git Friendly

This is very true. You can put the config.sqlite into git, but that's just basically a binary storage. When people ask for diff and git what they are usually talking about is two things:

1. I need to know what the last jerk changed on this broken box.
2. I need to share/version control it so I can deploy automatically.

Storing a raw config.sqlite file into git doesn't get you either of those features.

I think these two solutions are actually solving a different problem, one of automated and controlled migrations of a configuration. I'm going to propose that instead of just checking a config file into git and hoping it works on different machines and doesn't get clobbered, that instead there's a configuration migration system. Similar to the database migrations found in many web application frameworks.

With this idea, your configuration changes from a single config file to a starter SQL file (probably from an initial dump), and then a sequence of versioned migrations. These migrations would do your typical forward/backward migration operations, and make the changes needed in your configuration as you need to roll out new configurations. You'd then have these files checked in the same way you have your web application's migrations checked in.

Your deployment process then becomes:

1. Work out the new migration needed using the m2sh.
2. Save it to a .sql file with a version number.
3. Commit it to your VCS.
4. Pull these changes onto your target machine.
5. Use m2sh to run the migrations to sync up to the new one.

With the proposed m2sh you'll be able to make new migrations, run them, roll them back, and you'll know that they worked cleanly since the database will have proper standard migration version specifiers in it.

It Will Be Hard To Change Quickly

This proposed problem is actually related to "I HATE SQL", but let's look at the problem from the point of view of the three proposed UIs:

1. Neckbeards will just sqlite3 config.sqlite and then run SQL.
2. Mundanes will use the m2sh to do the same thing but without SQL.
3. Designers will use the web interface to click around.

It's my experience that creating a new configuration is usually only done once, but that changing the configuration is done over and over again during the life of the web server. This means that making it easy to change is way more important than making it easy to get started (although both are important).

In this way, using sqlite3 for the config storage wins hands down. Making a change to a SQL database is much easier and safer than doing it to a config file. With a config file you have to hunt around for the right stanza, type in the right exact change, restart the server, and pray it works. With SQL you can actually query the database, and run updates (then pray that works) which are much easier to get right and do.

Basically, with a config file vim and the / command are your UPDATE and SELECT, and they suck by comparison. For example, yes you can change one configuration pretty quick with vim, but could you change 1000? SQL will beat the pants off vim in an operations context.

Which leaves me to the last concern that I know of:

It Will Suck For Developers

I kind of think of Mongrel2 as my penance for Mongrel's ops-unfriendly deployment methods. Mongrel was crippled by a need to support Windows, requirements to be friendly to programmers (mongrel_start is all you need), and early bugs in Ruby that made forking not practical. If I were to redo Mongrel today it would basically be Unicorn. I wouldn't have to deal with Window's shitty process management (read: none). I wouldn't need to worry about early Rails programmers getting distracted by Web 2.0 eye candy. I wouldn't have massive bugs in Ruby's forking and garbage collection.

Mongrel2 though is written in C and will never run on Windows. Ever. Not even if Windows 8 is OSX and they buy me an Atelkof Super Bass VI. Mongrel2 is all unix, all the time, and heavily Linux focused with OSX only there because people will probably try it on OSX first, with NetBSD and FreeBSD probably later as I find a need (I use NetBSD).

The reason I'm so focused on Linux for Mongrel2 is that's what operations people use, and I want Mongrel2 to be an operations fantasy. Automation, management, scripting, fast changes without restarts, not caring about the languages being used, ability to carve up the application and scale. That's the goals I'm going for.

However, every web application starts with a Mundane..a programmer. Someone who just wants to build the damn web application and not care about migrations, SQL, server deployment, or any other operations guys don't care about. They want their mongrel2 start PONIES and have the world magically start working with next to no configuration.

In this contention between the two groups of users, Mongrel2 will try to be balanced, but side with operations users more than developers. That doesn't mean it'll suck for developers, but it does mean that if there's a bad habit I identify from programmers that's hurting operations, I'm going to make it unpleasant or impossible to do. It also means that programmers may have to learn some operations stuff to use Mongrel2, but operations folks will have to do the same.

Hopefully, this will get everyone to get along.

Next Steps

The next thing I have to get working is serving files and fully processing the HTTP and JSSocket protocols so the server will be mostly complete. After that I'll be rolling out the usual documentation so people can fire up a Mongrel2 and get their application behind it.

Comments?

If you have other thoughts about the SQLite configuration system feel free to let me know on twitter or email me at zedshaw-AT-zedshaw.com.