5/18/2005

Multiuser Programming

When I was in High School, LambdaMOO opened to the public. I spent a lot of time in high school and college hanging out there, programming, and even did my first web application development on E_MOO, which had an HTTP server implemented in moocode. After E_MOO went down, I decided that I wanted to recreate the MOO experience as a graphical multiuser networked programming environment. The ability to log in to a machine, edit some code, manipulate the "object" whose code you just edited, and hand it to your friend halfway around the world for debugging is a very compelling experience. I hadn't seen it recreated to my satisfaction and I wanted to do it.



I spent a few years coding, but it was slow going. Both network applications and GUI applications require a different mindset than the batch process style of programming which is what you learn in computer science classes. Event driven programming takes a lot of getting used to, and even when you understand what you are doing, it can be difficult to write and debug event-driven in C or C++, the language of choice in the mid 90s.



In 1999, I was doing a lot of Flash work and saw how it might be possible to tie a Flash front end to a multiuser back end server. When Flash 5 came out, I decided to revive the idea of a graphical front end to a multiuser programing environment. I wrote a simple HTML page which contained some JavaScript DHTML code I put together for mutating the DOM and embedded a small, wide Flash movie towards the bottom of the page. The movie contained an input text box and some ActionScript which sent the contents of the box over an XMLSocket when the user pressed return.



Because the XMLSocket used null bytes to delineate each message sent across the wire, I hacked the C source of the MOO server (running a MOO core my friend Pictwe and I had been running since 1995 -- MOOf) to send null bytes between each "line" of output from the server. Because of restrictions Flash placed on the hosts and ports to which you could open an XMLSocket, I had to create an application server capable of both serving the HTML, JavaScript, and Flash files, and proxying the XMLSocket connection to the actual MOO server. I had done a few simple CGIs in perl and thought about using that, but perl sucked. I did a bit of web searching and discovered Python.



After reading the socket and threading documentation (old habits die hard) I had a simple server working. You could load the web page, type commands in a text box, press return, and your command would be sent to the server where it was executed like it came from any other client being used to access the MOO. Any time the MOO generated any output on your connected socket, the intermediary server would push it into the browser over the XMLSocket, the ActionScript would unpack it and send it to JavaScript using LiveConnect, and the JavaScript would use DHTML to change the page. Sound like AJAX? :-)



Shortly thereafter I got a full-time job writing Python web applications. After developing for a few years in Webware and Zope, I decided to play around with some ideas I had for an easier to use templating system. This led to the creation of DOMTemplate, Woven, and Nevow, and it turned out to be a longer process than I had hoped. Sometime in late 2002 I decided to revisit the idea of performing out-of-band communications with a server to allow the web application running serverside to push and pull information into and out of the browser. I called it LivePage, and it was based on a highly transparent Model-View-Controller design, where controllers received events from the browser, updated models, and views automatically re-rendered themselves based on model dependencies. The results were then shipped to the browser and some DOM hackery was employed to replace the old DOM fragment with the newly rendered view.



The original Woven implementation of LivePage used Flash XMLSocket as the out-of-band event conduit. XMLHttpRequest was around at the time, and I did attempt to use it, but it reeked of MSIE nastiness and wasn't very standard cross-browser. XMLHttpRequest also has the distinct disadvantage of being based on HTTP. XMLSocket is a persistent, two-way, asynchronous socket architecture that made it delightfully easy to implement both Client-to-Server events and Server-to-Client events. XMLHttpRequest can only send data to the server once, and if you repeatedly send data from the server to the client and handle it incrementally, the document will consume memory without bound.



People thought I was crazy to even try to make web browsers do the things I wanted to do. It'll never work, it'll never be cross-browser, browsers can't handle it, people won't understand it... But I persevered. Browser stability was a real problem. After leaving a live page open for a while, with lots of changes being sent, browsers would leak all over the place and eventually crash. Both IE and Mozilla crashed like crazy. Eventually, after Gmail came out, the browser vendors seemed to get things under control and DOM mutation is relatively reliable now.



In October of 2003 when I sat down to spend a bit of hacking to try to come up with ways to simplify Woven, Nevow went from proof-of-concept to ready to use in a weekend. It wasn't until Gmail was in private beta that I decided it was time to revisit LivePage. This time, the aim was to keep the implementation as short and easy to understand as possible, with the hopes that other people would be able to figure out how and why to use it. I also decided to use XMLHttpRequest as an experiment, to reduce the number of dependencies. XMLHttpRequest makes some aspects of the architecture more difficult (Server-to-Client events) but in some ways makes other parts of the implementation much easier.



Since I only get to work on LivePage in my spare time, it has been slow going. However, the implementation as it is in Nevow is now almost ready to tackle the task of constructing very dynamic, complex applications. There are some implementation details which, after being exposed to the real world for a while, need refactoring. I have a few ideas on how to make it even easier to use from an end-user programmer's perspective. Finally, a coherent idea of how error handling occurs between client and server is starting to take shape. In the latest livepage branch which includes some unfinished improvements, which I hope to merge in the next week or so, the server can reliably handle an exception which occurs on the client. That's right, Python code can be written to handle client-side JavaScript exceptions. Also, LivePage as it exists in the 0.4.1 release has very robust "User has left the page" notification support. If the user clicks the back button, closes the browser, or even crashes their machine, the server is guaranteed to know about it no more than a minute or so later.



I have spent 10 years of my life, on and off, working towards this dream I have. Nevow and LivePage are the latest incarnations of puzzle pieces which fit into the mosaic of this dream. Hopefully someday very soon I will start fitting the pieces together and start to see what a true multiuser web application looks like. We already have a taste for what collaboration can enable; sharing information using weblogs, instant messengers, wikis, file sharing applications, and tools like SubEthaEdit allows to more rapidly fine-tune our idea of what others are thinking. Only a few pieces are missing; a more graphical, spatial ability to arrange information, where information can be picked up easily and carried from place to place in order to sort and categorize it; a more real-time sense of community where presence information gives us a sense of who we are near and allows us to easily communicate with them; and a more immediate programming model where changes can be made to live applications and applied immediately, with results available for all to see. All the pieces are there, they just need to be put together to complete the puzzle.




7 comments:

seth goldstein said...

Donovan, what are your ideas for the kinds of spatial arrangement metaphors that could enable with this? Google/Ajax all seem to be pretty flat, even the best flicker apps move in one dimension across the page. How can you enable screen depth for personal information navigation? Have you seen omniture, hitbox, urchin, visual sciences... anything comparable?

Thomas Herve said...

Nevow and LivePage are really great pieces of work, and a joy to use for every Python developper. I think that what Nevow needs now (and I don't see it without LivePage) is a killer-app using it. Quotient could be it, but I think one more specific program would be better (instant messaging, blog.. webmail ? :))

Garth T Kidd said...

I'm able to re-set my document title fifty times a second. Not bad! It only seems to work in FireFox, though. Any chance we can make this work in MSIE?

Donovan Preston said...

Garth, LivePage works just fine in IE. It also works in Safari, although there was a trivial bug that prevented it from working in the 0.4.1 release, which will be fixed soon.
I'm not sure what problem you are having... it must have to do with cross-browser javascript incompatibilities with the way you are setting the title.
Seth, I'm whipping up a prototype to give people a better idea of what I am talking about. It should be ready in a week or so.

Ed said...

On another topic. Donovan, at a past Python group meeting, you indicated you would post a response to the Ruby on Rails/Python Subway hysteria. Any update?

Sean Lynch said...

Donovan, I've been thinking along the lines of multi-user web programming as well. I, too, was inspired by LambdaMOO, and I was involved with the Cold project for a while.
I also think the appropriate client for a multiuser environment is a web browser, and something like LivePage is the right thing for communicating with the browser. The biggest problem I keep running into is how to allow users to execute code on the server without stepping all over one another. I have yet to find decent support in a real programming language (rather than a mud-specific one) for doing this sort of thing with any real security. I have played a little bit with embedding JavaScript in Python, but I really don't know JavaScript that well so I don't know whether it's the answer.
The real answer is probably PyPy, but that's not yet really in a state that I can play with it for this sort of purpose.
Anyway, feel free to drop me an email or something if you want to chat more about this. My AIM ID is kg6cvv and my Jabber ID is seanl@masten-space.com.

justizin said...

I think seth is right about seeing a truly killer app built on LivePage. We've been trying to haxx0r ajax into the Plone CMS and have made some progress, but are probably a long way from anything like handling js exceptions in Python - a freaking cool idea, at that. Maybe a simple Archetypes-based CMS could be built on LivePage. Archetypes is truly the heart of Plone and there seems to be a lot of similar goals inbetween twisted and zope - esp z3 development. As we move toward z3 view classes, events, etc.. in Plone, perhaps the barrier to doing such a thing is even becoming lower.
I haven't touched you guys' stuff for a while, but I remember that I was getting into twisted, woven, and nevow around the same time that I learned about Plone. I'm thinking it was too low-level for me at the time, but a lot of our time in Plone team is being spent fighting with Zope2. Even if just some proof-of-concept hybrid system could be built tying in more than just the reactor from twisted along with z3 components, we might start making a lot of headway in this area.
-=$0.02