ptth (Reverse HTTP) implementation in a browser using Long Poll COMET

ptth is an idea I have planning on implementing for a few years now. The basic idea is that you take normal HTTP semantics and reverse them, meaning that the client (from the TCP perspective) acts like a server (from the application perspective), and the server (from the TCP perspective) acts like a client (from the application perspective) and makes requests on the client whenever it feels like it. This is distinguished from most normal COMET semantics in that ptth retains all of http's characteristics even though the underlying transport is radically different looking at the TCP level.

When I was at Linden Lab, I advocated using this technique in the Second Life Viewer as a refinement of the Plain Old COMET implementation currently in use (which I also helped implement). I wrote a wiki page describing how the http Upgrade: header can be used to initiate a ptth connection, effectively turning a socket that the client opened to the server around, allowing the server to make requests on the client as if the server had opened a connection to the client (even though it didn't). I even did an implementation in Python showing how once the Upgrade: has been performed the semantics are exactly the same as normal http. This means with a little hackery it's possible (and in the Python case, almost trivial) to reuse existing http client and server libraries. All you have to mess around with is the setup of the socket; once both sides have an open socket and have agreed to Upgrade:, you just grab the underlying socket and pass it to the client or server library and away you go.

Even though I didn't get the chance to implement and deploy this technique in the Second Life Viewer and Server before I left Linden for Mochi, I still hope this gets implemented someday, as I think it is a very elegant and efficient technique. While implementing the real ptth Upgrade: in C++ will be more challenging than doing a quick Python prototype, once the dirty business of extracting sockets and injecting them into the client and server libraries used is complete, it should be a very reliable technique since at that point everything is exactly the same as normal http.

However, it won't be possible to do these type of Upgrade: shenanigans when we are in the browser's Javascript environment and don't have access to low level details like socket APIs. Therefore, I also specced out what ptth would look like running over a Plain Old COMET Long Poll style transport. The wiki page describes encoding the reverse request and response as JSON for ease of parsing and generating in Javascript, but other content-types could be used (application/x-http-request and application/x-http-response perhaps, or maybe the message/http mime type could simply be used or modified to be message/http+request and message/http+response?)

On Saturday the 6th we had a Mochi Hack Day at our office, and I was hacking on my perpetual hacking project, Pavel. If you don't know me personally and haven't heard me talk about Pavel, someday I'll flesh out the ideas behind it more fully in a series of web pages, but for now you can read this old blog post to get a rough idea of what it is. The post uses the term "graphical multiuser networked programming environment" to describe the basic idea. From the very beginning I conceived ptth as a vehicle for driving updates of the user interface to Pavel, so I decided to get down to it and actually implement it. Since I have implemented so many COMET servers at this point that I have lost count, it turned out to be almost trivially easy, and I had something working in a few hours.

And now the part everyone has been waiting for: the demo. The demo takes place in firebug, where you can see the Javascript side of the Long Poll operating, and in the eventlet backdoor running inside of a terminal. The eventlet backdoor gives me a Python interactive REPL into the process which is serving the server side of the Long Poll, and allows me to manually inject ptth messages into the system which then get delivered to the browser, which then responds to the request. The first thing you see me doing is building a simple ptth request by hand, encoded in JSON. I then inject this message into the ptth system, copying and pasting the uuid of the user who is connected via the Firefox browser in the background. You can see the debug printing in Firebug showing that the request was delivered to the browser, and the result in the backdoor's REPL is the response that was generated in Javascript and sent back to the server.

This means that I now need to implement some sort of web framework in Javascript. I know Dojo has already done this and I'm sure other people will start to experiment with this idea as well, but for my purposes I'll probably come up with something super simple. The idea that immediately came to my mind is to have URIs represent XPath into the html document, and PUT replacing the selected node with the content fragment from the request body of the PUT.