Multiuser Programming

When I was in High School, LambdaMOO opened to the public. I spent a lot of time in high school and college hanging out there, programming, and even did my first web application development on E_MOO, which had an HTTP server implemented in moocode. After E_MOO went down, I decided that I wanted to recreate the MOO experience as a graphical multiuser networked programming environment. The ability to log in to a machine, edit some code, manipulate the "object" whose code you just edited, and hand it to your friend halfway around the world for debugging is a very compelling experience. I hadn't seen it recreated to my satisfaction and I wanted to do it.

I spent a few years coding, but it was slow going. Both network applications and GUI applications require a different mindset than the batch process style of programming which is what you learn in computer science classes. Event driven programming takes a lot of getting used to, and even when you understand what you are doing, it can be difficult to write and debug event-driven in C or C++, the language of choice in the mid 90s.

In 1999, I was doing a lot of Flash work and saw how it might be possible to tie a Flash front end to a multiuser back end server. When Flash 5 came out, I decided to revive the idea of a graphical front end to a multiuser programing environment. I wrote a simple HTML page which contained some JavaScript DHTML code I put together for mutating the DOM and embedded a small, wide Flash movie towards the bottom of the page. The movie contained an input text box and some ActionScript which sent the contents of the box over an XMLSocket when the user pressed return.

Because the XMLSocket used null bytes to delineate each message sent across the wire, I hacked the C source of the MOO server (running a MOO core my friend Pictwe and I had been running since 1995 -- MOOf) to send null bytes between each "line" of output from the server. Because of restrictions Flash placed on the hosts and ports to which you could open an XMLSocket, I had to create an application server capable of both serving the HTML, JavaScript, and Flash files, and proxying the XMLSocket connection to the actual MOO server. I had done a few simple CGIs in perl and thought about using that, but perl sucked. I did a bit of web searching and discovered Python.

After reading the socket and threading documentation (old habits die hard) I had a simple server working. You could load the web page, type commands in a text box, press return, and your command would be sent to the server where it was executed like it came from any other client being used to access the MOO. Any time the MOO generated any output on your connected socket, the intermediary server would push it into the browser over the XMLSocket, the ActionScript would unpack it and send it to JavaScript using LiveConnect, and the JavaScript would use DHTML to change the page. Sound like AJAX? :-)

Shortly thereafter I got a full-time job writing Python web applications. After developing for a few years in Webware and Zope, I decided to play around with some ideas I had for an easier to use templating system. This led to the creation of DOMTemplate, Woven, and Nevow, and it turned out to be a longer process than I had hoped. Sometime in late 2002 I decided to revisit the idea of performing out-of-band communications with a server to allow the web application running serverside to push and pull information into and out of the browser. I called it LivePage, and it was based on a highly transparent Model-View-Controller design, where controllers received events from the browser, updated models, and views automatically re-rendered themselves based on model dependencies. The results were then shipped to the browser and some DOM hackery was employed to replace the old DOM fragment with the newly rendered view.

The original Woven implementation of LivePage used Flash XMLSocket as the out-of-band event conduit. XMLHttpRequest was around at the time, and I did attempt to use it, but it reeked of MSIE nastiness and wasn't very standard cross-browser. XMLHttpRequest also has the distinct disadvantage of being based on HTTP. XMLSocket is a persistent, two-way, asynchronous socket architecture that made it delightfully easy to implement both Client-to-Server events and Server-to-Client events. XMLHttpRequest can only send data to the server once, and if you repeatedly send data from the server to the client and handle it incrementally, the document will consume memory without bound.

People thought I was crazy to even try to make web browsers do the things I wanted to do. It'll never work, it'll never be cross-browser, browsers can't handle it, people won't understand it... But I persevered. Browser stability was a real problem. After leaving a live page open for a while, with lots of changes being sent, browsers would leak all over the place and eventually crash. Both IE and Mozilla crashed like crazy. Eventually, after Gmail came out, the browser vendors seemed to get things under control and DOM mutation is relatively reliable now.

In October of 2003 when I sat down to spend a bit of hacking to try to come up with ways to simplify Woven, Nevow went from proof-of-concept to ready to use in a weekend. It wasn't until Gmail was in private beta that I decided it was time to revisit LivePage. This time, the aim was to keep the implementation as short and easy to understand as possible, with the hopes that other people would be able to figure out how and why to use it. I also decided to use XMLHttpRequest as an experiment, to reduce the number of dependencies. XMLHttpRequest makes some aspects of the architecture more difficult (Server-to-Client events) but in some ways makes other parts of the implementation much easier.

Since I only get to work on LivePage in my spare time, it has been slow going. However, the implementation as it is in Nevow is now almost ready to tackle the task of constructing very dynamic, complex applications. There are some implementation details which, after being exposed to the real world for a while, need refactoring. I have a few ideas on how to make it even easier to use from an end-user programmer's perspective. Finally, a coherent idea of how error handling occurs between client and server is starting to take shape. In the latest livepage branch which includes some unfinished improvements, which I hope to merge in the next week or so, the server can reliably handle an exception which occurs on the client. That's right, Python code can be written to handle client-side JavaScript exceptions. Also, LivePage as it exists in the 0.4.1 release has very robust "User has left the page" notification support. If the user clicks the back button, closes the browser, or even crashes their machine, the server is guaranteed to know about it no more than a minute or so later.

I have spent 10 years of my life, on and off, working towards this dream I have. Nevow and LivePage are the latest incarnations of puzzle pieces which fit into the mosaic of this dream. Hopefully someday very soon I will start fitting the pieces together and start to see what a true multiuser web application looks like. We already have a taste for what collaboration can enable; sharing information using weblogs, instant messengers, wikis, file sharing applications, and tools like SubEthaEdit allows to more rapidly fine-tune our idea of what others are thinking. Only a few pieces are missing; a more graphical, spatial ability to arrange information, where information can be picked up easily and carried from place to place in order to sort and categorize it; a more real-time sense of community where presence information gives us a sense of who we are near and allows us to easily communicate with them; and a more immediate programming model where changes can be made to live applications and applied immediately, with results available for all to see. All the pieces are there, they just need to be put together to complete the puzzle.