My people, we have Mongrel2 1.0 ready for your glorious consumption. Started on June 1st in my spare times, and with the help of a few friends, now a working functioning web application server that:

• 10 programming languages, and with .NET being one of them, a lot more.
• Solid ZeroMQ messaging that has now been proven to work as one hell of a app server base protocol.
• A lot of the web server features you need like virtual hosts, long polling, directory serving, flexible routing, proxying (no 0MQ required).
• A solid and incredibly powerful configuration system that I'm hoping opens a whole new way of doing hosting and deployments.
• A complete manual covering every feature we've got so far.
• A very nice new control port feature for status and control without POSIX signals.
• All language agnostic right down the configuration system.
• Completely self-hosting and eating our own dog food.

I love this project. Even if it doesn't go anywhere and nobody uses it I am so happy I got to work on another cool idea nobody's really done before.

Nothing's Perfect

Well me and my faithful Mongrel2 cohorts have been knocking down tickets in Mongrel2 with our goal of Aug 31st to release Mongrel2 1.0. Since I announced the 1.0 goal we've completed everything but 5 tickets and implemented all the features we needed. There's also been some big leaps in the number of languages we support.

Getting 1.0beta4

First, you can grab the latest 1.0beta4 from the main page and you can follow the GettingStarted docs for a fast course in getting it up and running. After that we've been maintaining the Manual as we make features so everything is documented. We've adopted a policy that a feature isn't done until it's documented in the manual.

Quality

I'm going to make the case that performance testing localhost is the best usability test for your server software you could make. This is contrary to pretty much anyone who's written a server nobody uses, but hopefully you can see why I find this important. I also give an example: MySQL vs. PostgreSQL popularity.

You Care About UX?

When I do my initial performance testing I always test the speed of localhost operations before I do anything else with my servers. In Mongrel2 and Mongrel(1) I went so far as to get in there and make sure the thing runs for all the different localhost test procedures people might run. Obviously I don't only test localhost, and have a large array of tests I run on a little crap cluster of old laptops and VPS slices I've got. In fact, right now mongrel2.org is running mongrel2 and I post links off of it to twitter to get it smashed.

The problem is, when I post metrics from localhost people scoff at them. The assertion is that I must only test by having a 20 node EC2 cluster with a wide array of tests. Well, apart from the confounding and difficult and cost of this design, it misses a very important purpose of the "localhost perf test": programmer usability.

Two weeks. That's right, I want to try to get a 1.0 version of Mongrel2 out the door by the 31st of Aug. That would mean 3 months of development to produce a 1.0 release. It obviously wouldn't be feature complete if you compared it to older servers, but the goal for a quick 1.0 release will be to take the features we've got and make them rock solid. Solid enough that, if you need these features, then you can start building on them. That's what I want for a 1.0 by Aug 31st.

The thinking with this is that I've always waited forever to do a 1.0 release, and that to me and everyone else 1.0 means:

It's got two kitchen sinks, a winnebego, and can feed a mountain lion as well as server up web sites. Yep, it's got everything. Ship it!

For most of my other projects, and I think for many open source projects, 1.0 has meant that it has every feature they ever wanted. They slaved on it for 5 years or 10 years and finally they've got it all. It's 1.0 baby.

I've managed to work out a decent "polling" abstraction so that I can compenstate for later designs and try out some different methods for handling events. Today I committed the code that makes poll or epoll work inside superpoll without much configuration. It works and I have good unit tests, but I have to tighten it up tomorrow and make it usable by other people. Also need to make the epoll compile out if your platform doesn't support it.

In this blog post I'll show you a few graphs and some data that shows some improvements I've made to the system, and then what I plan to do with it before Mongrel2 1.0.

Current SuperPoll Status

Right now it works, but I don't have the epoll actually turned on. There's a small bug in that Mongrel2 won't shutdown if epoll is used so I need to fix that. I have a decent unit test that runs the whole system using both poll and epoll and then tracks performance, and then I'm using that to tune it a bit. Finally all the code for superpoll is about 370 lines long.

I received this email today:

Thank you for the initial python lessons via email; they were enough to get me going and really change my families living conditions and quality of life. You have made a real and tangible difference in my world I just wanted to let you know that before I started asking stupid questions.

That made me smile quite a lot.

Go check out Learn Python The Hard Way, my free book I'm writing to help people learn programming.

I'm currently working on the last few lessons in Learn Python The Hard Way and I want to include a lesson on general health problems programmers run into during their careers. I find many programmers seem to ignore their body's physical state when they're coding, most likely due to the intense concentration required. I'm hoping other people could benefit by simply understanding a few health related problems programming has almost caused me or caused many other people I know, and how I avoided them.

I probably won't put this whole blog post into LPTHW since it's a bit much, but I will make a shorter version of it. Please feel free to let me know if you hate it or like it or if you have some additional resources I could reference.

My Background And Qualifications

In the past I was a top qualified soldier in the US Army, and I have studied many martial arts. These days I'm not as into working out and studying martial arts as I used to be, instead focusing on yoga, meditation, and simpler activities. When I was younger I was incredibly fit, and still am because of habits and practices I ingrained in myself from an early age.

I spent the last few days figuring out how to go about doing something that could be "superpoll", and trying to hedge my bets on the idea. Ultimately I wanted just a poll-like abstraction that either I could have do the dual poll/epoll thing, or just say screw it and compile a version for whatever one you've got that works best.

Even though my goal from earlier this week was to try out the whole poll and epoll combined, I'm not an idiot. I'm not going to build something without extensive testing, assurances, and being able to back out of it if it doesn't work. It'd be stupid to bet the whole project on some idea that might work, and frankly, anyone who tries to tell you that Mongrel2 will have poor performance because of a couple of blog posts with me wondering outloud is full of it.

What I needed to do first is get the current poll usage out of the libtask coroutine library and into its own little abstraction it could hide behind. I had a few false starts, and then managed to boil it down to enough of an API to just abstract away what the tasks need to do poll, but in a way that I could then extend to use epoll, kqueue, etc.

In no way was I trying to add features. The first problem I could have is if I do a bad API then it can kill the performance on its own, even without epoll or any crazy schemes. In order to remove confounding I wanted to keep all the current polling mechanisms the same, and only test the new abstraction. If the performance when down after the refactoring, then I'd have to fix it. If it was the same or got better (unlikely) then I could safely move on with trying different internals.

When I wrote about epoll vs. poll I sort of knew people would have a hard time with it, but I didn't anticipate the huge response (mostly from a few folks). I mean, I understand that the majority of the history of programming has been more marketing, FUD, and meme spreading than actual analysis. I just figured epoll vs. poll is so banal a topic, and it's only about my Mongrel2 project so really what is up with all this insanity over something so little?

A Little Note About Trolls

I think the more important fact is that what I've learned since my Ruby on Rails days is you cannot ignore the trolls. Trolls have become more clever and are very motivated to shape public opinion. If you ignore them then huge swathes of people will simply blindly believe anything the trolls say no matter how wrong or weird it is. I guess it's a guy thing where they'll just believe whoever's tallest, and "tall" on a forum is who's most obnoxious and writes the most.

Before I would just ignore them, thinking that the general public would finally come to its senses. Later I realized the public never does. A simple comment like, "Never hire Zed Shaw because he will destroy your company" can have a major impact on your life if you're not careful.

I got to the end of my post about epoll vs. poll with regards to Mongre2 and was too damn tired to tell you how to repeat my tests. A cornerstone of science is that other people need to be able to repeat your experiments and report their own results. Another key piece of science is that there's always a probability that you are wrong.

I also consider much of my posting about Mongrel2 a side goal of educating people about building servers and the research involved. Whether I'm right or wrong I'm at least getting people to think and question what they've believed all these years, and teaching them something about my work. Hopefully that inspires many other people to do new things and take risks that might make them turn out wrong.

Today I hope people take my fully disclosed test gear and try to prove me wrong. Yes, I want you to prove me wrong so that I don't infect anyone with some weird ass idea that's actually going to hold people back for decades. I am going to make sure right now that everyone understands I am in no way advocating that I am right. I am merely showing an interesting observation which leads me to an idea I might try.

This is important, because I am offering up raw stats and data, my full R environment, the full C code to the tests, my shell script that runs it all, and letting you have at it. You could have a chance to be the guy that put me in my place, and I want you do to it. I want you to prove me wrong because then I won't feel like I've been living the "epoll lie" for 8+ years.

The next thing I wanted to solve for Mongrel2 is implementing an epoll backend for handling I/O. Rather than just implement it, I wanted to reconfirm that epoll was superior in the ways claimed based on recent research. I decided to rerun benchmarks everyone had used, but to change how they were being used based on some papers I read and original research I had done a while ago. Of course I totally can't find any of those papers because they're gone, but I decided to go try science and see what I found.

What I found is actually leading me to try an experiment in Mongrel2 I'm gonna call "superpoll" for lack of a better name. But first, let me get into why the touted benefits of epoll and most of the information out there is mostly bullshit because of some falacies people seem to have about epoll (mostly perpetuated by epoll's proponents).

Fallacy 1: epoll Is O(1)

The general belief among programmers is that epoll is O(1) and poll is O(N). Yes, they actually believe that epoll will process 1 event at the same speed it processes 10k or 100k or 1M events. That's what O(1) means to me, so I'm not sure where they're getting that epoll does anything in constant time.

It has been about 2 months since I started Mongrel2 and tonight I finished a rough first draft of the manual. This manual covers all of the features we have right now and goes to great lengths to make sure that you can install, deploy, manage, and hack on Mongrel2. I made it fun to read, have lots of examples, and be as complete as possible without overloading you. When you're done, you will know as much about Mongrel2 as anyone could possibly know right now without being me.

It's also full of little jabs, jokes, and tiny rants about important topics so you'll definitely find something to keep you interested.

You can read the first draft of the manual in HTML and PDF formats and probably the PDF is the better of the two if you're going to get serious. There will be some layout issues with the PDF and definitely some problems with grammar and spelling.

What the book covers is:

I believe that Mongrel2 has the chance to break the religious hold that programming languages have on organizations, businesses, and the average working programmer. Mongrel2's design and programming language agnostic philosophy could mean greater freedom, more agility, and better cooperation between everyone working at a high tech company today. Even though Mongrel2 is only a web server, its goals and direction could be a major inspiration for other projects and people to design their systems to be programming language agnostic as well.

I believe in this programming language agnostic movement so strongly that I'm going to bet the farm on the goal by giving Mongrel2 the most flexible license I possibly can so that it can reach the largest audience possible.

Mongrel2 is now BSD licensed with commit c4b9041f42 and I want everyone to use it and not have to worry about licensing issues or problems with contributors.

The license is the basic 3 clause BSD license:

Remember back when I said Mongrel2 would use sqlite3 for it's configuration system? Remember how everyone thought that meant they'd have to code SQL in order to configure Mongrel2? Remember how I kept saying no, I'd have some kind of configuration system that'd be better and more powerful than config files?

It's done, and it is fucking awesome. I've been using it for the last few days while I worked on the next demo (a ICY mp3 streaming handler) and the new configuration system works amazingly well. The design is basedon a Model-View-Controller design where:

• Mongrel2 == Controller
• Sqlite DB == Model
• View == m2sh config system

What's this m2sh you ask? Why, pure awesome is what. The m2sh is a simple Python command line program that acts as the View UI to a Mongrel2 config file. With m2sh you craft configurations using little Python scripts that are a lot like config files in other webservers. You can then use m2sh to start, stop, reconfigure, look at the config, create new ones, pretty much all the stuff you end up doing with janky git repos and shell scripts or etckeeper.

I'm proud to announce a first cut at an actual Python library for Mongrel2 that is turning out to be quite sexy. It's most likely too rudimentary right now for serious work, but it is simple enough to get your head around and use, and is full featured. You can actually implement full HTTP or JSON handlers with it.

After fixing up a bunch of bugs today and getting the HTTP handler to properly deal with request bodies, I sat down and implemented a first cut at a Python library for writing handlers. It turns out that since ZeroMQ handles a bunch of the protocol crap for me, all I have to do is parse the format and I'm done. I finalized the format last night, so tonight is just implementing some simple code.

You can install it by grabbing the Mongrel2 source and then doing:

$ cd examples/python && sudo python setup.py install

This is a short little post about the latest drop of Mongrel2 which is getting closer to an ALPHA release that regular folks can try out. This release features the beginning of it being a real server, and a nearly complete protocol format for 0MQ handlers no matter what kind of message is being sent. It also includes the rough code for two sample handlers that implement the chat demo and a basic HTTP handler.

If you want to try out the latest gear, read the Getting Started document that explains how to get up and running and do some demos and such. It runs through all the recent features, but is still definitely targeted at a programmer type who wants to try it out. Let's go over some of the big features.

GNU Make 4Life

I tried to like Premake4 but it just fails miserably on too many platforms. The fact that you need Premake4 to build Premake4 makes it nearly useless for many platforms, most notably OSX and a few Linux variants.

Today I'm dropping a fresh Mongrel2 that features a completely redesigned connection management algorithm that uses a bad ass Finite State Machine to keep everything straight. This state machine will make it possible for Mongrel2 to keep-alive connections no matter what kind of backend is requested and allow for developers to inject their own filters on the events that manage connections. This release also features a first (hackish) working HTTP->0MQ protocol based on SCGI and a simple demo.

I'm very excited about the connection state machine because it means that in addition to the original highly accurate Mongrel HTTP parser, I've now got a connection state system that's just as accurate. It will let Mongrel2 support bizarre proxy configurations, keep-alive state, any kind of backends, HTTP long poll operations, JSSocket, and WebSockets.

As I worked on it this last week I started stumbling into extra features I got for free with this design. HTTP long poll and keep-alive from HTTP->0MQ are probably the two sexiest. Thanks to this design, long polling isn't a special feature, it's just how things work. It also means that HTTP->0MQ has the ability to do N:M processing similar to the current chat demo but using plain HTTP.

What I'm most excited about is how, since the state machine is controlled by simple integer events through a fast Ragel FSM, that means people can write filters based on the events and the callbacks. In the same way people grabbed the Mongrel HTTP Parser and used it to build web servers, this new connection state machine will let you extend Mongrel2's internal connection state processing to meet whatever hairy problems you meet.

I no longer work at Dropbox since Jun 15th. There wasn't anything bad about my leaving. I wasn't fired. There isn't any drama. It just wasn't a good fit (to use that cliche term they love out here in The Valley) and I felt it was time to find something more challenging.

Don't read into that though. Dropbox has a lot of potential and if you want to work there, then go check out this job that they're promoting.

What's Next?

Rather than gossip about what's essentially a minor event in my life, you should go do something cool. I plan on dropping a major feature of Mongrel2 this weekend, assuming I can get the code right. I'm also going to play some guitar and try to get through my Bass Method book and learn "It Had To Be You". Maybe I'll see Toy Story 3, I hear it's good.

I managed to get Mongrel2 to serve files, although there's some bugs with how it works. This means that Mongrel2 is pretty close to being a full fledged server, minus all the usual edge case features like handling stupid J2ME clients that send chunked-encoding when they aren't supposed to (it's for servers morons). It also has bugs in how the proxying is done (which I'll explain in another post) and various other little things.

My next whack of work was to tighten up the HTTP processing so that it can handle all the bizarre ways that browsers use it, but before I did that I had some work to do.

You see, while Mongrel2 on Monday could served lots of requests without crashing, it wasn't really too tidy. I'd been slacking off in my usual C coding purity and just using all the stupid C string functions that invite buffer overflows. And while I'd been running mongrel2 under Valgrind during its entire development, I was playing fast and loose with the memory management.

It was time to get serious. Starting with this commit I finally set out to eliminate almost all use of the evil string functions using the bstring library. The bstring library is a heavily tested and very well designed library for working with strings. It gives you all the usual stuff you'd get in a modern language, and in a package that keeps things secure. I like this library because it doesn't try to become The One True String Replacement. Instead, you use it for your internal data structures, and then easily interface it to other libraries that already expect regular C strings. It's not totally foolproof, but it is a big help.

I finally refactored the guts of Mongrel2 so that it is now configurable via a sqlite3 database. It's currently running stable at the chat demo and has all the necessary pieces to now complete the actual file serving and async routing parts. So far the choice to use sqlite3 for the "config file" is working out good for the code, but it remains to be seen if it'll work by itself for actual deployments. I thought I'd write up some of my thoughts on this idea, and also answer some common questions.

The first word of warning is that obviously this is going to change. It's an interesting idea, but it has potential to go horribly wrong if it's not constantly usability tested. If you have a knee-jerk reaction to this that's based on your dislike of SQL, just hold off on that. I wouldn't force SQL on anyone who didn't want it, so expect great tooling for this to come out.

Why Sqlite?

I want Mongrel2 to be very language agnostic, and that means you should be able to configure and manage it with your favorite ops friendly language. I'd rather have people work against a known and migrateable SQL database than force everyone out there to try to write hackjob tools that munge an ever changing config file format. Hopefully this will make Mongrel2 an operations wet dream where servers can be managed easily, have their configurations migrated (yes migrated), validated for consistency, and even live reconfiguration without restarts.

Back when I did the first Mongrel I realized that the perfect data structure for routing requests by URL was the ternary search tree. This let Mongrel route large numbers of URLs very quickly by their prefix without having to parse the path at all. The downside though was you couldn't do anything more than prefix matching, which was fine in Mongrel. In Mongrel2 I wanted to go with the TST again, but I also wanted to try out a new idea, where I added the Lua pattern matching code so you get both fast prefix matches followed by patterns similar to a regular expression.

As of commit 079c177756 I have that working, and it took me just one day. Best of all, using the two data structures, the entire piece of code that meshes them together is:

int RouteMap_collect_match(void *value, const char *key, size_t len)
{
    assert(value && "NULL value from TST.");

Route *route = (Route *)value;


            Read more...
        

Hi,

I read your blog on "There are no Famous Programmers" (from reddit.com), and I have my own thoughts regarding programmers and programming. I have been programming for 30 years starting with Fortran then C and C++. I have a Masters in Mechanical engineering (not CS, but have taken night classes).

I've stumbled over your blog several times, most recently from reddit on your post about no famous programmers. The comment that amused me most was you getting another sysadmin job offer. Honestly you are right in that we love to code, but end up giving away much of what we do for free without anyone remembering. I mean, come on, it's not like Steve Jobs actually wrote OS X, and everyone knows about Bill Gates little OS purchase resell dance... Point being, fame does come to those in the software community, but it is to those who are leading something, and make a bunch of dough.

Over and over companies go through a hiring process to ramp up a project. Every time, programmers and engineers need to reprove themselves to some ahem douche. But the fault is not theirs, it is ours. We allow ourselves to be controlled and manipulated. We refuse to learn how to negotiate. We scoff at collective bargaining. In the end, we may get pissed off, threaten to go somewhere else, and finally get a raise as the realization of what we actually do sinks into some manager's head somewhere, and they panic and cave on the raise we demand. But really, we gave away our ingenuity, drive, and dedication for years at an undervalued rate.

Where am I going with this? The idea is for the programmer's "internet workshop". In the age of the internet, we can collaborate on projects like the Gnu/Linux OS. But we need to be on the same team as programmers. And we need to get paid. The "internet workshop" would be a place where companies come to get software built. Yeah, a consulting company... kind of... But it would have it's own internal mechanism for programmers from around the internet to work on projects, get paid, and establish reputation, not with clients, but internal to the "workshop community", so that they don't have to interview every time. When a hard problem comes up for a project, a talented programmer can be assigned (or choose to jump on it). When there is some "scut work", a junior programmer can get some of it out of the way, and get an opportunity for feedback from senior programmers. And they can get paid a decent rate...

It would take a lot for something like that to happen. But until it does, we will continue to compete amongst ourselves as programmers, drive our compensation down through competition, and devalue our abilities. Doctors realized this long ago, and created the AMA. Engineers (civil, mechanical) created the PE designation. Even actors created the SAG (but big stars still make millions). It's time for us to do the same as programmers.

I read your essay about computer science and some ideas you had bothered me so I wrote an article in which I try to set some things straight. Since I study computer science writing essays is not really my strength so it might lack some structure here and there.

Let me start off by saying that I don't think universities are a better place for learning culture than any other location with bright and interesting people. If you have time and interest you can learn as much as you want about any aspect of the collective human knowledge no matter what your daily occupancy is. If you really want to learn as much of the breadth and depth of the great human achievements as you can then work very hard for a few years and after that go travel around the world! Take an e-book reader with you with all the popular scientists and philosophers, poets and artists. If you think going to a university will just automatically teach you these things you are wrong. If you are lazy and you don't like to read or learn about culture, all you will learn at a university is what it is like to sit on your ass all day playing video games, and hang in a bar all night getting drunk as fuck. It will end eventually with either you dropping out or with you aging a bit, manning up and doing what it takes to actually get that degree.

So why would you go to a university and study computer science if it has no added cultural value? The answer is simple. Because you're interested in Computer Science. You've learned how to program at high school. You can make websites for shops or businesses. You could make them back-ends or optimize their data entry applications. You can use some abstract languages like PHP or Ruby, or even Java or C#. Or perhaps you're the kind of guy that works in C or C++, and you contribute patches to drivers and software you use on your linux operating system. These things are nice, and being able to do them will land you a nice job, but that is not enough for you. You might want to know what it takes to make a user interface that allows any person to use a device irrespective of complexity. You might want to know what it takes to design an operating system from the ground up. You might be interested in how to rigorously proof that a multithreaded application that monitors and controls a nuclear reactor is correct beyond a shadow of a doubt.

Do they teach you how to be a better programmer at a university? Yes and no, and probably not more than you would learn at a decent employer. A proper university will introduce you to some high level language, like Java and to test driven development using something like JUnit. The reason many universities chose Java for their introductory programming courses is not because they want to prepare you for 'the industry' or whatever organization might be giving them money. It's because java is a very strict and explicit language and that makes it an easy tool for explaining the concepts they are trying to teach. This is why software engineering courses might require you to use Java. Operating systems courses might require you to use c/c++ and functional programming courses might be in Haskell or whatever the professor thinks is appropriate. There is perhaps some truth in Zed Shaw's statement that you might be lucky if they teach you more than one language. In a computer organization course I followed the teacher explained how computer memory worked and then gave an assignment in C without a full introduction to the complicated art of C programming. He just assumed that you would be able to program in C after you had learned Java and heard about how memory access works. This is because languages are just tools. It is more about what and how you implement than in which language you do it, or using what process. (Although there is a branch of computer science that focuses on the development process, if you're interested in that!).

I spent this last week or so working on an idea. It's an idea I've had for a while. Ever since I wrote Mongrel and got stuck doing Ruby I've wanted a web server that loved all languages. A web server that broke the strict request/response pattern and let you scale out your application so that one browser request could be mapped to as many backends as you needed. A server that let you do both HTTP, Flash Sockets, and WebSockets on the same port with no configuration.

A web server simply written in C that loved all languages equally. When I wrote the original Mongrel I wanted to make a server that loved all Ruby web frameworks equally. Rather than make it Rails specific I wanted everyone to have a chance, and the end result was many nice frameworks and Merb eventually taking over Rails.

I wanted to do this same thing for all programming languages.

This week, I finally started that web server and I named it Mongrel2. It took me about 1 week to get a first functioning version up once inspiration hit me of how exactly I could do it. This version is functional enough to be self-hosting and support a mostly working web chat server. The code for this is currently about 366 lines of C and 57 lines of hack Python plus some supporting libraries.

I like to use Fossil for my projects these days. It's a great little SCM that's easy to use, easy to deploy, and with one single binary works on every platform you can imagine. What's kind of interesting though is proponents of git, bazaar, and mercurial like to insult my choice for various reasons. Usually their reasons for not liking fossil are either very shallow, or the exact same reasons people who liked SVN said they didn't want to use git/bazaar/mercurial.

For example, people who use git like to say fossil is weird. Fossil's actually pretty normal compared to git. The commands make sense, there's help, and it's simple so each thing does one thing. Git users however will avoid fossil because it's not like git, and don't want to learn anything new, yet when git came out those same people kept saying learning git was like learning C. It's good for you!

Well, what if learning fossil is good for you? Hell, I think you should know all the SCMs you can and be able to switch between them. You might not like it, and I complain all the time I have to commit a merge in mercurial, but I know how to use them anyway.

Another complaint people have about fossil is that it looks like crap. I'm sorry, but if you're comparing fossil to github or bitbucket then that's an unfair comparison. Github and bitbucket are hosted services that get paid money and have been designed to support many projects at one domain used by many people. Fossil is fairly new, and while it supports most of the features or more that github and bitbucket do, it's designed by programmers to be used by other programmers. It's not that great of a design, but the UI is pretty easy to use and gets the job done.

This is a quick one, because hopefully it will require very little explanation. This is wrong.

The author has converted a simple for loop into a usage of boost::bind, but has also attempted to format the for loop to make it look like the same amount of code. Let's compare if I simply rework the for loop to look normal:

int Dice::total() const {
  int total = 0;
  const_iterator i;


for(i = dice.begin(); i != dice.end(); ++i) {
        total += (*i)->faceValue();
  }

            Read more...
        

I frequently meet a friend for lunch and we talk. Usually I'll blab on and on about music, or some weirdo project I have going on. He'll tell me about jobs he's had or trips he might take now that he's sold a company and can chill out for a while. After one such meeting he said, "It's so refreshing to meet up with a geek who doesn't talk about VCs and term sheets the whole time."

VCs and term sheets? Really? Well shit, tomorrow I get a x0xb0x and tonight I was hacking on a cool new web server now that I'm done with MulletDB. And these guys just think about VCs and term sheets? That's kind of sad really.

Let me tell you about this cool new web server. I figured out how to merge the ZeroMQ event polling system with the libtask coroutine library so that you can use libtask to handle tons of TCP/UDP and ZeroMQ sockets in a single thread. I then took this very cool hack, and started building a web server using my Mongrel HTTP parser, but I modified the parser so that the same server on the same port can handle HTTP or Flash XMLSockets transparently. The next step is to get this server to route HTTP and XMLSocket JSON messages to arbitrary ZeroMQ backends. I was inspired by this so much that I registered utu.im and may try to bring it back. Not sure how or when though.

Sounds cool right? Totally doesn't matter one bit. I could hack on projects like this and nobody would care at all because I'm a famous programmer, and there is no such thing as famous programmers. I don't exist. I'm an enigma.

Another exercise I just wrote for Learn Python The Hard Way where they do some code reading. In Exercise 37 they reviewed all of the Python symbols in order to get their vocabulary straight. In this one they apply that exercise to try understanding code.

You should now go out and find some Python code to read. You should be finding and reading any Python code you can and trying to steal ideas that you find. You actually should have enough knowledge to be able to read, but maybe not understand what the code does. What I'm going to teach you in this lesson is how to apply things you've learned to understand other people's code.

First, take the code you want to understand and print it out. Yes, print it out, because your eyes and brain are more used to reading paper than computer screens. Make sure you only print a few pages at a time.

Second, you should go through your printout and take notes of the following:

Just finished this exercise for Learn Python The Hard Way and thought I'd put it up to see what people think. You can grab the PDF from the site if you want to see what Exercise 35 is like. Basically they make an Adventure game and in this one they make a bigger one on their own.

I'm going to give you some rules now that you know if-statements, for-loops, and while-loops that will keep you out of trouble. I'm also going to give you some tips on debugging so that you can figure out problems with your program. Finally, you're going to design a similar little game as in the last exercise but with a slight twist.

Rules For If-Statements

1. Every if-statement must have an else.
2. If this else should never be run, because it doesn't make sense, then you must use a die function in the else that prints out an error message and exits, just like we did in the last exercise. This will find many errors.
3. Never nest if-statements more than 2 deep and always try to do them 1 deep. This means if you put an if in an if then you should be looking to move that second if into another function.
4. Treat if-statements like paragraphs, where each if,elif,else grouping is like a set of sentences. Put blank lines before and after them.
5. Your boolean tests should be simple. If they are complex, move their calculations to variables earlier in your function and use a good name for the variable.

Here's how I'm teaching indexing into lists in LPTHW and I'm wondering what people think. The basic idea is if they know the difference between an ordinal and cardinal number then they can more easily figure out how to translate them in their head. Let me know if you have other ideas.

Exercise 34: Accessing Elements Of Lists

Lists are pretty useful, but unless you can get at the things in them they aren't all that good. You can already go through the elements of a list in order, but what if you want say, the 5th element? You need to know how to access the elements of a list. Here's how you would access the first element of a list:

animals = ['bear', 'tiger', 'penguin', 'zeebra'] bear = animals[0]

I frequently have young programmers ask me if they should bother with going to university. I'll call it university since that seems to be the more universal term for, "Studying something serious for four years after grade 12." They ask me because they frequently run into other programmers on the internet telling them university is pointless.

They're told, "If you can already code you won't learn anything in university you couldn't learn in a full time job."

So the young programmer, who's parents probably demand that they go to school and study so as not to be a loser, wonders why bother? All they want to do is code. They could skip those annoying four years of learning and just get down to doing it like a true professional. Right now. That's what everyone else says.

Sadly, I have to agree. If you go to a university and you can already code you probably won't learn much about programming. Chances are you'll end up going to some crappy Java school that's just now realizing standardizing on one language controlled by a corporation that had no clue how to actually turn a profit was a bad idea. For the next 4 to 10 years I think most university CS departments are going to be horribly behind, not just for the usual reasons of being out of touch with the current practices, but because they bet the wad on the wrong horse entirely.

I've named my latest project MulletDB which I think really sums up exactly what it is. I named it NoNoSQL originally, but then yesterday I did a little joke about The Valley Mullet and realized that the mullet represents my little database project perfectly. It combines SQL (the business) with NoSQL (the party) to create what's easily the Mullet of databases.

You can check it out in it's initial setup, which is just a Fossil repo I put up for collaborating. It's still a work in progress, but it works pretty well and has the features I need to get started on other stuff.

You'll notice MulletDB is AGPL and it's also written in C++ with Python APIs. I may release the Python driver as MIT licensed later when it's actually stable.

So far it can do the following:

I had to fix some last remaining bugs in Lamson, and then just decided to release it 1.0 and give it a BSD license. You can read the new license in the timeline which is now BSD and GPLv3. You are now free to rip me off all you want, but it'd be nice if you didn't.

This release has a few minor bug fixes and has a hard dependency on python-daemon 1.5.5. It turns out the new python-daemon package has a broken dependency on lockfile 0.9, which isn't available through the usual package installers. Also there's a few platforms that seem to explode from weird code in lockfile (complaining about metaclasses).

When you upgrade to Lamson 1.0, if you have a problem make sure to zap any install of python-daemon and lockfile from your system first. Hopefully you've got a virtual env setup so this is easier.

About Python And The GPLv3

Last night I finished up enough of NoNoSQL that it's useful to use on crapmania.com including writing the Python driver for it. I based the Python API and how you interact with NoNoSQL on the web.py database API and it worked out great. The Python library has is currently only 114 lines of very minimal code to implement a full SQL, disk key=value, and memory key=value storage system.

Of course this will need to expand as I implement things like better robustness, better error reporting, and a tighter better defined protocol for NoNoSQL. I'm also planning on adding the ability to interact with NoNoSQL in an async model, where you blast out requests and collect them later if you like.

But, if I'm going to make it useful I need to use it on a project, so I've decided to try using it on Crapmania. Hopefully by keeping to this API and using a good unit test suite I can later swap it out if NoNoSQL proves too difficult to use in practice. I'm anticipating though that it'll be very useful for the kind of application crapmania ends up being.

Crapmania First Design

I got bored recently so I decided to create my NoSQL entrant. Those seem to be pretty hot projects and I had some ideas, so I hacked one up in a few days to try it out. Originally it was just gonna be a joke, but actually I kind of like this thing. I'm gonna screw around with it and try it out on Crapmania! to see if it'll work.

I'm calling it NoNoSQL for lack of a better name. If it gets more serious then I'll pick something else. The man part of the joke is that it's a "NoSQL" server, but it has SQL in it, but you don't use SQL to do the SQL. Let me explain.

What I did was combine 0mq with sqlite3, tokyo cabinet, and finally, the filesystem, to create a server that lets you do all four at once using the 0mq scalability and speed building blocks. In theory this thing should be dead easy to scale out, shard, cluster, whatever you need to do. Right now of course it doesn't but the potential is there.

However, I took things one step further by making most everything you send and receive JSON. When you build a SQL query, it's just a JSON abbreviated format:

I've been having a blast writing Learn Python The Hard Way and I'm already up to Exercise 16. I just finished closing tons of tickets and made all the edits people submitted so I decided to bump the verion number up to 0.2.

This release you can get the PDF from http://learnpythonthehardway.com/ and I've close quite a few tickets but keep putting them in.

Here's the big list of things I've fixed:

• Officially changed the '#' character from pound to octothorpe.
• Cleaned up a ton of little edits and fixes (thanks folks!).
• Finally got the exercise numbering and other number straight.
• Wrote a placeholder for exercise 27 where I start to introduce logic.
• Got the exercises to bring the reader to where they're finally reading and writing files.
• Created the book's project web site using fossil to host it.
• Declared the book free to distribute as long as you don't charge for it or alter it.

I got the learnpythonthehardway.org site up and working good. You can get the latest book there from now on at:

http://learnpythonthehardway.org/static/LearnPythonTheHardWay.pdf

Which I update frequently. The site also supports the following for people interested in contributing:

1. It's a full fossil repository, so that means you can grab the whole site for yourself, including all the sphinx code. No license decided yet so just assume I own Copyright and you can redistribute for non-commercial purposes.
2. There's a wiki where you can write a proposed exercise if you think of one. Make sure it's good in case someone comes along on accident and doesn't get that the wiki is only a testing ground.
3. You can enter in tickets if you find problems in the book.

I finally got a fossil repository/wiki up for people who want to help writing exercises. It's just up temporarily until I can get the full site going, but gotta strike while the Iron is hot.

You can go to:

http://zedshaw.com:8080/home

For now and after logging in as anonymous you should be able to edit the Proposed Exercises wiki page. Make sure you read the home page and the Example Exercise to get an idea of what it's all about.

I'm quickly working on Learn Python The Hard Way and should have a fossil repo up soon so others can contribute. I've also grabbed the domain where I'll try to host the book repo today. I think when the book is further along I'll try to do up a site to go with the book, but right now the book is enough work.

I really quickly wanted to get the word out about two books that are so far looking to be great beginner Python books.

1. How To Think Like A Computer Scientist is actually used as the curriculum at a few universities in the US and maybe in other locations. Despite the title it is actually good introduction to the language and to programming.
2. A Byte Of Python is also free and has been translated to [a huge number of languages](http://www.swaroopch.com/notes/Python_en:Translations and also seems to a good first book.

My plan is to mention these two books as next steps after my book, as well as the Django Book for after they finish one of those two.

I've had an idea for an introductory book on programming that follows the model of "trainer" books for learning a musical instrument. Most of these books are organized like this:

1. There's a bunch of exercises.
2. Each exercise is 1 or 2 pages.
3. There's only a little bit of prose.
4. You do each exercise exactly, then move on.

It's a simple concept, with the idea that a big part of music is learning to do things with your instrument, and that's best accomplished by fairly repetitive progressively difficult exercises.

There's a few books that do this for other languages, like the wonderful Little Schemer, Seasoned Scheme, Reasoned Schemer series. I think those books have the right idea, but again I didn't like the format.

I thought I'd do a little explanation about the idea behind sheddingbikes.com, oppugn.us, fretwar.com and the new setup of zedshaw.com as a more thoughtful non-rant post on what I'm thinking these days.

Once a year I like to change up my blog and online writing style. I find that the Internet is impermanent and fluid, but too often people keep their blogs exactly the same for decades. I'm not just talking the design but the writing style, topics, and how it's themed. In general I tend to stick to rants and thoughtful pieces, but lately I've been posting music on Fret War and wanted to roll that in.

Herein lies the problem: these three types of expression don't go together. The music is mostly guitar music with some oddball theme that probably only interests musicians, not coders. Fret War is also a strange concept of a "guitar competition" so even guitarists might find it weird. Posting my music to zedshaw.com wouldn't fit.

Then there's the problem of the rants vs. the essays. I'm known as the rant guy, for very good reasons. But, I also like to write more thoughtful less ranty pieces that take me much longer to compose. Those essays were getting drowned out by the fun of the rants. The rants are totally like candy for programmers, but the essays are like their veggies. People just don't like eating candy and veggies at the same time.

For the first post to Shedding Bikes I thought I'd voice my real observation about how Apple screwed up with section 3.3.1 of their SDK license. Unlike the rant I wrote at oppugn.us this is a simple observation not meant to be hyperbolic in any way.

First, here's the section once again so you can review:

3.3.1 Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited)

With the key part being that software needs to be "originally" written in one of the mentioned languages. It's this word "originally" that has caused all the problems, and I guess is actually theoretically impossible to enforce technically. I'm actually thinking that Apple will probably strike this word from the license by the end of the week, but not for any particular business reason.

I've started Shedding Bikes as part of my recent trend of expanding out my blog to other sites. Just like "oppugn.us for rants" and the other sites I'm doing with focused topics, this site will be non-rants about programming culture and philosophy. The range of topics will be observations about programmers, the things they create, and how that relates to their thinking and behavior.

The rules here are to not have any cursing, and to keep it clean. This of course doesn't mean it'll all be puppy dogs and rainbows because a huge part of programmer culture is disagreeing with the way things currently are done. But at sheddingbikes.com this disagreement will be measured and well thought out, not rants for the sake of poking fun at people and whipping the internet froth.

Just like oppugn.us you can post to sheddingbikes.com and if your post is well written then you'll be included in the roll. Your full name from your email address (the "Joe Frank" part) will be shown unless you don't set your name then your email address gets shown.

You'll also notice there are no comments on this blog. If you have something to say, write it as a post and put some effort into it.