By Zed A. Shaw

Yesterday I talked about The BBS demo I did using Mongrel2's new JSON/XML/HTTP parser and Lua. In that post I just laid out how I used coroutines from Lua to organize the workflow of the BBS. In this post I'm going to talk about how Mongrel2 lets you use three different protocols on one port, and then show you a simple HTML+Javascript version of the Python client.

JSON/XML/HTTP on 80

Mongrel2 uses the Ragel state machine compiler to process the socket protocol your browser sends the Mongrel2 server. With Ragel Mongrel2 (and Mongrel1) has this very accurate and fast parser that handles various bizarre edge cases and turns out to be relatively small compared to a similar hand written parser.

When I sat down to create Mongrel2 though I wanted to add support for other "browser friendly" protocols and so I reverse engineered whatever Flash and JSSockets does and baked it into the parser. I did it with these lines:

SocketJSONStart = ("@" rel_path);
SocketJSONData = "{" any* "}" :>> "\0";
SocketXMLData = ("<" [a-z0-9A-Z\-.]+) >mark %request_path
    ("/" | space | ">") any* ">" :>> "\0";
SocketJSON = SocketJSONStart >mark %request_path " "
    SocketJSONData >mark @json;
SocketXML = SocketXMLData @xml;
SocketRequest = (SocketXML | SocketJSON);

This is after some evolution, and I don't expect you to understand it, but what I did was add specialized JSON and XML messaging to Mongrel2 with 8 lines to the parser code. It took a bit of wrangling to work out the simplest yet still correct version of this, but that is the meat of the entire protocol.

What does it do? Why it lets you send any of the following three to a Mongrel2 server from a single socket connection:

1. Regular HTTP requests.
2. A JSON message that starts with @path {} and ends with \0 (NUL).
3. An XML message that starts with any <tagname root tag and ends with >\0.

The parser, by virtue of being a parser, accurately identifies which is which based on what you send, and then peels out that one message and cooks it up for Mongrel2.

Mongrel2 then uses the @path, <tagname, or HTTP /uri to route your request to any handler you setup. All of the messages are orthogonal so you could serve up a file for all three, send to a 0mq handler for all three, or if you're really stupid/crazy proxy it to a backend server.

With this setup you can serve handler backends, files, and proxies to a very wide range of clients. HTTP clients, flash sockets, plain TCP/IP sockets, specialized XML clients, and the backend handlers really don't need to care. To them they just get messages with headers/body and different methods.

Let me repeat that, all three of these different protocols result in the same message format and response format for your handlers. No real special code needed to reparse it other than dealing with the XML or JSON.

The Python Client

If you look at the original Python client. you can see that I've got a very simple little TCP/IP client that connects to Mongrel2.org and just starts using the JSON mode to talk to our BBS. The BBS then just establishes a few simple message types like "prompt", "screen", "exit". It's all in JSON so very easy to work with and handle, and no specialized libraries.

Of course, doing this right would involve reconnects, checking the socket is live, timeouts, and various other things to make the client robust.

Being able to write a simple little client like this let me work out the backend protocol I'd be using without having to hunker down with a full on GUI and figure it out from there. I could test the different messages, what standards to adopt, and get raw debug access to the server very easily.

With that client, we had about 220 logins created since yesterday, and people left about 110 messages to the BBS.

The Browser Client

The goal was to get a browser accessible BBS "emulator" going to people could play with it. I personally am tired of chat demos, so a BBS seemed like a lot more fun and nostalgic for me. I sat down today to make a fun "black and bright green" BBS client for browsers.

You can check it out by going to http://mongrel2.org/bbs/ and it might be working. I'm still learning the Lua part of the project so it crashes sometimes. If it's up, play around with it and compare it to the Python client. The prompt is a bit different but otherwise works mostly the same.

How I did it was I just grabbed the code from examples/chat/ in the Mongrel2 source and then modified it to work with the BBS protocol I established in the Python client. Since I'd already worked out most of the kinks it was a fairly simple couple hours of quick coding.

First thing is I needed a BBS connection:

var BBS = {
    socket: null,
    init: function (fsm) {
        BBS.socket =  new jsSocket({
            hostname: 'mongrel2.org',
            port:     80,
            path:     '@bbs',
            onOpen:   BBS.onOpen,
            onData:   BBS.onData,
            onClose:  BBS.onClose
        });
        BBS.fsm = fsm;
    },
    onOpen: function () {
        BBS.fsm.handle('CONNECT', null);
    },
    onClose: function () {
        BBS.fsm.handle('CLOSE', null);
    },
    onData: function (data) {
        data = eval('(' + data + ')');
        BBS.fsm.handle(data.type.toUpperCase(), data);
    },
    send: function (message) { 
        BBS.socket.send({'type': 'msg', 'msg': message});
    },
}

This lets us connect to the Mongrel2 server @bbs and interact with it. It also takes any messages from the server and routes them through the fsm.

In my client I'm using a little bit of code that gives me a simple Finite State Machine for javascript. Nothing fancy, just enough state machine magic to keep the UI sane and control arbitrary events. Since JavaScript isn't so great at the coroutine thing, FSM is the next good choice for normalizing random events. Here's that code:

var FSM = function(states) 
{
    this.states = states;
    this.state = this.states.start;
    this.state_name = 'start';
    this.intervalId = null;
}
FSM.prototype.handle = function(event, data) 
{
    var res = null;
    var target = this.state[event];
    if(target) {
        if(this.onevent) this.onevent(this, event);
        try {
            res = target(this, data);
        } catch(e) {
            if(this.onexception) {
                this.onexception(this, e);
            } else {
                throw e;
            }
        }
    } else if(this.onunknownevent) {
        this.onunknownevent(this, event);
    }
    return res
}
FSM.prototype.trans = function(target) 
{
    var new_state = this.states[target]
    if(new_state) {
        if(this.ontrans) this.ontrans(this, target)
        this.state_name = target
        this.state = new_state
    } else if(this.onunknownstate) {
        this.onunknownstate(fsm, target)
    }
}
FSM.prototype.start = function(event) {
    this.states.start(this, event);
}

There's a couple helper functions for doing chained event handling but those are just sweeteners on top of this. All this is doing then is translating strings that are 'events' into callbacks on some object we hand it.

Finally, with those two pieces I have my little app.js that runs the whole thing:

state = new FSM({
    start: function(fsm, event) {
        display("Connecting to the Mongrel2 BBS....");
        fsm.trans('connecting');
        BBS.init(fsm);
    },
    connecting: {
        CONNECT: function(fsm, event) {
            display("Connected to Mongrel2 BBS.");
            BBS.send("connect");
            fsm.trans('connected');
        },
    },
    connected: {
        PROMPT: function(fsm, event) {
            prompt(event.msg);
            fsm.last_pchar = escapeHTML(event.pchar);
        },
        EXIT: function(fsm, event) {
            clear_prompt();
            display(event.msg);
        },
        SCREEN: function(fsm, event) {
            display(event.msg);
        },
        CLOSE: function(fsm, event) {
            display('<em>Disconnected, will reconnect...</em>');
            fsm.trans('connecting');
        },
        SEND: function(fsm, event) {
            var input = document.getElementById("prompt");
            BBS.send(input.value);
            display(fsm.last_pchar + input.value);
            clear_prompt();
        }
    }
});

Again I'm cutting out some of the helper functions like display() and clear_prompt() so you can see what's going on. Those functions don't use anything fancy like jquery and just do very simple printing or controlling the prompt.

The main point of this design though is that it reliably handles asynchronous events from the user and from the network. If the user types something while we're in the 'connecting' state then they're ignored. If we get a close they're notified. Adding state control to the events cleans up a lot of the callback-spaghetti code you get in a lot of javascript.

Take CLOSE for example. Let's say the user just typed something and we get the CLOSE callback from BBS (see above where it sends it). UI now has to display that they were disconnected and ignore whatever they may enter. Rather than try to juggle two competing events, this just accepts that event and transitions right into 'connecting'. If the user then hits enter, the 'connecting' state ignores it.

Those three little bits of code are all doing pretty much the same thing the Python client was doing. It's the same protocol between the two, which means I've actually created a kind of desktop+browser integration.

A Warning About FSMs

The downside to using an FSM though is they tend to not scale to large numbers of events or states. In this little case the GUI is really very simple. It gets connected and then prints crap out, either crap from the server or crap from the user, but that's all it does.

A more complicated GUI that had a large number of states, various screens, and lots of transitions would be damn hard to work with using just an FSM. I'd use an FSM to keep things straight using big events, and then leave the smaller little things to specific classes that know how to keep their little section of the world clean. I probably wouldn't cram everything into this simplistic FSM.

So they're a reliable way to clean up your event code, but they get a bit hairy when you have to code them by hand at a large scale.

The Mongrel2 Configuration

Gluing all this together is a simple mongrel2.conf file:

m2bbs = Handler(send_spec='tcp://127.0.0.1:9995',
                    send_ident='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
                    recv_spec='tcp://127.0.0.1:9994', recv_ident='')
main = Server(
    uuid="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    access_log="/logs/access.log"
    error_log="/logs/error.log"
    chroot="./"
    pid_file="/run/mongrel2.pid"
    default_host="localhost"
    name="main"
    port=6767
    hosts=[
        Host(name="localhost", routes={
            '@bbs': m2bbs
            '/bbs/': Dir(base='html/', 
                        index_file='index.html', 
                        default_ctype='text/plain')
        })
    ]
)
settings = {"zeromq.threads": 1}
servers = [main]

This is where you can see how I have the @bbs route setup to take JSON messages from a client off the socket, and send them to the m2bbs handler I've created at the top. It really doesn't care if the client is a browser using jssocket or the Python client as long as it follows the protocol. The backend handler then doesn't have to do jack all to deal with them either since it's just receiving the same 0MQ messages it always gets.

Hopefully you can see how this could be a powerful protocol for creating browser, desktop, and mobile clients that all speak the same messaging protocols and interact with each other.

The BBS Code

Some folks have expressed an interest in hacking on the BBS code, so I'm going to toss it up on github and bitbucket tomorrow with some instructions on getting it running. If people add features then I'll incorporate them and host the changes on the mongrel2.org/bbs/.

I think the next cool fun thing would be door games. I personally wouldn't have a lot of time to write door games, but I'd be interested in letting people write them and then host them. If there's enough interest in hacking on it then I may just get into hosting wacky BBSes and door games for people to test out Mongrel2 and because it's damn fun.

Once I announce the code release tomorrow I'll also setup a mailing list people can join if they want to help. Whatever anyone wants to do with it I'm cool and will try to host it. Of course you could also host your own, and maybe we can talk about that too some more.

Until then, enjoy the one I've got setup for you to play with (assuming it hasn't crashed yet).