Dinosaurs and Programming

Ever since I can remember I've had a love for dinosaurs. Even now, I sometimes secretly dream of flying off into the African sunset on a Pterodactyl, gunning down antelope with a .50 caliber machine gun (see http://theoatmeal.com/comics/ptero). If it wasn't for computers (or computer games actually) I would probably have studied Archaeology.

Now, at this point in time you may be asking: "What does dinosaurs have to do with programming or Whaleventures for that matter?" Well ... a lot actually. Let me set the scene for you: I was 9 years old and in standard 1 (we still referred to it as standard 1 back in the day) when Jurassic Park was released. For a 9-year old dinosaur freak, this was the biggest thing since chocolate coated vanilla ice-cream. Every time I got a glimpse of a trailer or poster somewhere I would nag my dad about it. Finally; I think out of sheer desperation to shut me up; my dad took me to see Jurassic Park. However, there was one technical difficulty: Jurassic Park's minimum age restriction was 10, while I was obviously 9 years old. With a suspicious look from the lady at the counter, and a white lie or two, my dad got me into the theater. I was in for an awesome action-packed 2-hour roller-coaster ride that involved a T. Rex eating a man straight off the toilet, Velociraptors raining down death and destruction on the main cast and Brachiosaurus sneezing a ball of snot in a girl's face.

At the time, I din't notice it, but years later when I watched Jurassic Park again, something interesting caught my attention. One of the characters in the movie is a computer programmer by the name of Dennis Nedry (or Ned). If you didn't watch the movie, note: some spoilers are up ahead. He is basically the guy that was responsible for the downfall of the entire park. One set of scenes (right after the shit starts to hit the fan), is of particular interest where Samual L. Jackson (he goes by the name Ray Arnold in the movie) tries to get the security system of the park up and running again. Here is an outline of what sort-of happened:

In the abovementioned set of scenes we get to see a strange operating system, with some lines of code in a strange (or not-so-strange) programming language.

Thus, to kick off part 1 of our: "Identify the programming language / operating system in the movie / television program"-series the first challenge (which is not that difficult) is to identify the programming language in which the Jurassic Park system is coded. The first one to provide an answer (do not cheat!, I'm watching you) will get 500 internets for his/her efforts. Here is a hint (more or less) of what you'll be looking for when watching the movie:

The second challenge is to identify the operating system that is running Jurassic Park. Lex (the hacker girl) is also using it when she tries to lock the Velociraptors out during a chase scene much later in the movie. If you can identify this (without cheating) you'll be really hardcore and you'll win 20000 internets (seriously).

So, go to your nearest video store this weekend, and rent this excellent movie. You'll be amazed by the special effects (given that it was 1993), the action scenes, and more importantly: the DINOSAURS!!!

Managing callback "spaghetti" in node.js

Although node.js' evented I/O model works really, really great for writing tcp/http servers, which are inherently event driven domains, sometimes you just want to execute a number of steps in sequence, wait for them all to complete, and then take some action.

Let's take the example of adding some sample data into a test mongo database. This is a pattern you see a lot. Some test framework, be it mocha/vows/nodeunit whatever, it doesn't mater. But there is a before hook, and you want to populate the database with a nice set of initial data for your test.

Now normally, you would end up with what I call callback "spaghetti". A bunch of async calls, taking callbacks, being called in sequence, each taking the next step as a callback, until finally you end up with a heavily-spaghetti, difficult-to-follow piece of code. And it's always the same pattern over and over again. To give an example, here is some demo code using mongojs to populate a mongo db for testing:

Now as you can clearly see, this very quickly gets unwieldy and annoying to work with. Luckily, there is a pattern to get around this. It is mentioned in the book Secrets of the Javascript Ninjas by John Resiq (thanks Dawid for the tip). On a basic level, it has the following form:

So basically, you have an array of functions. These functions each do their work, and call the shared "callback" function. The "shared" callback function simply pops the first function in the array, and executes it. Next time, it again pops the first item, and executes it. Until the array is empty. At this point your sequence of steps is completed.

This is a very nice pattern, and something that I know I've come up with in some form of another a couple of times already. But I always love it when someone actually comes along and "solidifies" the pattern. I don't know what the pattern is called, I just know that I like it.

So, I thought, what about trying to make it even more generic? What if you can have some function, that takes a "done" functions, and a list of "step" functions (each taking a callback) which do their async work, and then calls the callback, that was passed in to them. So, I came up with this:

This node module exports a single function, waitForAll. This function takes a variable number of arguments (as is natural with any JS function). The first argument is the "done" function - this gets called upon completion of all the async steps. All the other arguments are then treated as async step functions. So inside the function, the list of actions, and the in-between callback function gets created. The in-between callback function is then partially applied as the first argument to all the action step functions inside the preparation code. This means that all functions passed in already have their first argument bounded to the internally created callback function. All that is left then, is to capture inside the internal shared callback function, the arguments that were passed to that function, and apply them to the partially applied function from the first step. This allows us to pass data from the one async step to the next. Consider the following test:

As you can see, the steps are now simply arguments passed to the waitForAll function. The first argument being the done function. Each step gets the callback, and all parameters passed in from the previous step. It is the single duty of each step to ensure that the callback() function gets called when work is done.

This means that, the first example becomes much much simpler to comprehend/manage:

A big win for readability in my opinion.

Raspberry Pi vs cstick Cotton Candy

Two similar ideas with very different prices :)

Raspberry Pi launched two models of their "mini-computer" (micro-computer?) today. Model A costs $25 while Model B costs $35.

FXI Technologies also opened the cstick Cotton Candy for pre-order and it costs $199.

Deep down, you know that IDEs suck

Let's not kid ourselves, most IDEs are pretty good in general. There is however, an area where they are particularly bad. That just happens to be the editor - the main component every developer works with!

Now, an editor is the main tool every programmer uses. Without a good editor, you are simply wasting your time. Most people will probably say that IDE editors are great because they have some form of code generation and type checking.

While that may be slightly true, the editor still is the most un-feature-rich part of all IDEs. The refactoring support may be useful, but it does not aid you in writing the text that is your logic. The IDE is nothing more than a simple text editor that has some hooks allowing you to see type errors / syntax problems / etc.

So what can we do about it? Sadly not too much :P The IDE frameworks are generally monstrous and do not allow for much customization. It seems strange to say, but Emacs got it right. An editor that allows the user to customize it completely. Similar efforts are going into Sublime Text 2 (which has Python at the core) and obviously VIM (with VIMScript). These editors are what people should be using to make themselves more productive. If the editor is missing something like a snippet expansion etc, the person using the editor can just implement it themselves and more importantly, share it with the world on GitHub :)

So what does an awesome editor, something like Emacs, not have? This is what IDEs have tried to solve, but mostly very badly. It simply needs the interaction with the compiler and/or linting system. One technology that enables this is the SWANK protocol. The SLIME mode in Emacs uses it and the Clojure guys have managed to use it to create very nice tooling for Clojure.

ENSIME is another gem, it is a server (and Emacs mode) that caters for Scala and to a lesser degree Java. This means that the SWANK protocol is all that is needed for an editor to get these features (along with the editor actually being able to use this data in some way. ENSIME has very advanced Emacs support and integration is currently underway for Sublime Text 2 and for Vim. Very good stuff.

The point is, people find reasons to blame the IDE. Agreed, its a valid complaint and you have to wait for a while for the fix to propagate into the next release. Unfortunately, software like eclipse is, simply put, very, very poor. I don't care about the notion of a "workspace" and neither should the IDE / editor. Let me write my programs effectively, thats all I as a "professional" ask for. Let the stupidity die and lets really move forward!

I would like to have Funtoo

I am a gentoo fanboi, but I heard about Funtoo a while ago which is a different variant of gentoo, like Sabayon and also Google's ChromeOS according to http://www.internetnews.com/skerner/2010/02/google-goes-with-gentoo-portag.html. Both are from the same creator, Daniel Robbins. To me it seems that the major difference is that the portage tree is stored in git of which I am also a fanboi. I will be getting a new laptop soon and I am considering making the switch to Funtoo. I will post my experiences here.

First message in gmail

So for some reason today, I wanted to find out when I registered my gmail account. I couldn't see any direct place on the UI or whatever, so I found out that if you go to https://mail.google.com/mail/#search//p99999999 it shows you the first message ever received.

Mine is my "Gmail is different" e-mail, which I'm assuming is like your welcome e-mail. It's dated 2004-09-01

Whats yours?

Spatial Competitive Evolution

Okay, time to post some (semi-)tangible stuff.  The philosophical discussions have their place, sure... but that is mostly based on opinions and biases from past experiences.  Experience and empirical observation is, in my opinion, what this blog should be all about. (There I go again with philosophy... sorry, dear reader.) 

Ok, I am going to assume that you are familiar with Evolutionary Algorithms (EAs).  If not, where have you been? :-) No worries, have a look at http://en.wikipedia.org/wiki/Evolutionary_algorithm.

Some of the problems that I currently experience with the run-of-the-mill EA includes the following:

1. The populations are simply too small.  Evolution does not work effectively with anything under a 1,000 individuals. The biological metaphor on which EAs are based, operates on hundreds of thousands or even millions of individuals.  Small populations tend to cause a dramatic drop in diversity with a high occurrence of genetic drift.  Representing a large population (say, 100k+) tends to be too expensive in terms of processor and memory requirements and problems which do allow large populations are typically simple, trivial problems.
2. Selection tends to operate on the entire population space.  Again, this is unnatural and tends to produce algorithms with a higher selective pressure.  Do you really date someone from Japan, Jamaica and Portugal at the same time? ...and if you do, how do address the language barrier? 
3. Most fitness functions are one-dimensional in nature.  Several characteristics are measured on a like-for-like basis (very similar to the zero-sum games of classical AI.)  What I mean by this is that a feature is identified (by the creator!) and individuals are measured in exactly the same way against that feature.  The evaluation is therefore rigid and static - and limited by some predefined bias.  This, in my opinion, is the true ball-and-chain of classical EAs. This often result in an "unfit" individual, with a small, yet break-through component in its genetic make-up (the super-genius in the wheel-chair) to become extinct.  Furthermore, a high selective pressure coupled with this, short-sighted fitness function really tends towards a hill-climbing rather than true evolution.

I am fully aware that many people with disagree with me on many of these points.  Thank you, you're more than welcome (please share why)... I am currently experimenting with a new idea (I think) that I call "SCE" or "Spatial Competitive Evolution", which tries to address these issues to some extent.  It has not been as successful (yet!) as I hoped, but I am continuing fiddling with it.  Here's the main ideas:

1. The algorithm operates on a 2d grid (for now), where each cell represents an individual. Individuals can only reproduce and compete with its eight neighbouring cells (extending this beyond is of course also possible). This increases diversity remarkably well and take-over time takes much longer.  Unfortunately, genetic drift still occurs and larger populations are still required.  This is very similar to the "lbest" algorithm of Particle Swarm Optimisation.
2. The fitness function is relative-based (that is, in competition with its neighbours).  The function varies in that only a specific evaluation region of the fitness function is applied during evaluation.  The fitness function itself evolves with the individual and has the ability to include all evaluation regions.  This allows different sub-sets of the population to excel at different aspects then elsewhere on the grid.  Eventually take-over will occur and the fitness function will evaluate all regions.  This is true evolution, where the environment change (the lion runs faster) and you have to adapt (as the gazelle, you better learn some camouflage... and if the trees disappear? ...well then you better learn to run faster!).
3. Except for the fitness function and population representation, the standard EA is followed with standard mutation and cross-over functions.  Some changes come into play with the 2D grid, but these are rather intuitive.

Results... well... not that well at this point.  I've used the algorithm to evolve pieces of text that resembles a predefined line of text (text-writing Shakespearean monkeys comes to mind).  The fitness function starts off by evaluating a single letter and this then expands over time with the individual.  It takes significantly longer to find the final solution (compared to a vanilla GA), but it does eventually get there...

I often keep things like this secret... but I realise that sharing it on this blog will help in determining if this approach has any merit (especially with all these smart people on this blog!).  I will submit what I have on Github soon (and details to follow on this blog).  You're welcome to clone the repository, play with it and let us know if you find a working set of parameters.