Python OSA Component

I started work on embedding Python in an OSA component again recently. It's not that it is conceptually hard, although the available documentation is baroque and largely out of date, but that it is time consuming and boring to be writing C code when you are used to writing Python. With that in mind, my plan is to make the call into Python as soon as humanly possible, and do all the work from there with the help of Jack's recent bgen wrapping of OSA.h. So far I have been largely successful, writing code which gets the text of your script into Python by way of a Python AEDesc wrapper wrapped with a call to AEDesc_New. Unfortunately, while it works in Script Editor 2.0 (the opposite of the OSAShell effect), it segfaults osacompile. I'm sure there's something simple I'm just not implementing yet, since the code is still less than 100 lines.

In other news, Bob is busy working on his refactor of aeve now that he knows a lot more about AppleEvents than he used to. Hopefully he will get it to the point where he will give me SVN access to the code right around the time I finish embedding Python in an OSA component, and I can start helping him out by really banging on it and making sure it is possible to do everything that needs to be done.

Python and AppleEvents Slides

Better late than never, here is a link to a PDF of my slides from my OSCON talk. They look a little funny because the drop shadows from OmniGraffle didn't translate over perfectly, but the content is all fine.

http://www.soundfarmer.com/content/slides/PythonAppleEvents.pdf

Apache, ProxyPass and Twisted

One of the ongoing problems with writing web applications in Woven, my Twisted web app server and templating system, is creating links to other web pages in the system without creating bugs in those links.

Twisted stores the 'host' header sent by the browser in the request, and it stores the path segments that were used to locate the current Resource object in request.prepath. request.prepath is a list of strings indicating the URL segments leading up to the current Resource object. request.postpath is a list of path segments that have not yet been handled, but this is usually not terribly interesting.
So, it is possible, for example, to construct sibling urls, parent urls, and child urls by making a copy of request.prepath and performing list operations on it:
new = request.prepath[:]
new[-1] = 'someSibling.html'
Now that we have a list of path segments leading to a sibling Resource, we can construct a URL:
url = 'http://%s/%s' % (request.getHeader('host'), '/'.join(new))
While this works for simple cases, it breaks down when things get more complex. For example, the browser could be talking to the server over https, and while it is possible, it is non-trivial to detect this and construct a URL with the proper method. Duplicating this code throughout user-level code leads to many buggy implementations, with subtle problems that don't show up unless under very specific configurations.
For example, it is often desirable to run twisted.web behind a reverse proxy, such as using apache and the ProxyPass directive. (Someone please correct me if this is not the correct terminology.) In this case, the user browses to a URL served by the Apache server, and Apache forwards the request to the port upon which Twisted Web is listening. However, while it is doing this, it changes the "host" header from the header originally sent by the browser, to the host upon which Apache is running. Which means if we rely on the host header to construct our new URLs, they will reference the private proxy server rather than the public Apache server.
When I first began working on the Twisted project, one of the classes I wrote was called PathReferenceAcqusitionContext. It had this long name for historical reasons, and I shall simply refer to it as PathRef. One of the abilities of this object, which you could construct by calling the pathRef method on the Request, was to generate other PathRef objects by calling methods such as child, sibling, parent, etc. Then, once you had navigated to the conceptual URL location you desire, you could convert it to a URL string.
This implementation of PathRef was never documented, and had some other, unfortunate, unrelated properties which made it a perpetual pain in the ass. Glyph, finally fed up with the unused and problem-causing PathRef, removed it recently, and it is no longer in Twisted 1.0.6. Which is a good thing.
However, Glyph, finally seeing the need for a way to talk about URLs abstractly in a secure and convenient way, has kindly written the very good (in my opinion) Twisted/twisted/sandbox/paths.py, an implementation of all that was good about PathRef. I hope this gets moved into Twisted fairly soon, and a method for generating a URLPath representing the current Resource is added to the Request.
However, this method needs to be able to understand that the 'host' header is not necessarily the address to which the outside world refers. I decided to do a little experiment to see what Apache would do when performing a ProxyPass, and if we could determine where the outside world considers our application to live. When I was running Apache on port 8081 on my machine, and it was set up to forward requests to Twisted Web, I was able to observe that Apache had inserted an additional header in the process of forwarding the request:
'x-forwarded-host': 'localhost:8081'
The factory method on the request which generates a URLPath object should check to see if 'x-forwarded-host' has been set, and compensate accordingly. This should be a good solution to a long standing problem with using Twisted Web.

Generators and Python/AppleEvents

I started writing some code in aetypes that adds iteration support to DelayedComponentItem today. Basically the idea is you would be able to do this:

import mail
m = mail.mail()

for msg in m.messages:
print msg.subject

That's just pseudocode, but you get the idea. All of the necessary moving parts are already in place to do this, as Jack Jansen noted in this message from 1999:

http://mail.python.org/pipermail/pythonmac-sig/1999-August/001245.html

The bits that I'm doing differently are thanks to Python 2.2's generators, which make it very easy to write support for things like "for foo in bar".

Here's what I'm doing:

I added a __getattr__ to TalkTo that uses the _moduleName attribute I added last year to look for the given name in the correct module. If one is found, I then return a DelayedComponentItem with an additional parameter passed -- self, or the TalkTo instance itself. This allows the DelayedComponentItem to call count, which is a generic AppleEvent which returns the number of items of that type in the specified container (which can be None, to mean the application itself)

Then, I added a __len__ method to DelayedComponentItem that calls count to see how many items of that type there are.

Finally, I added an __iter__ method that does len(self) and returns actual ComponentItem instances for each item in the container. It works perfectly!

The only thing that tripped me up was the fact that AppleEvents like to count things starting with 1 and Python counts starting with 0. So the first ComponentItem returned won't actually be useful for anything and will generate an error if you try to use it. I realized this while I was driving home, and I look forward to fixing it tomorrow morning!

Scripting Mac OS X with Python using AppleEvents is accepted

Well, my OSCON proposal about scripting Mac OS X applications was accepted. Coincidentally, just yesterday I wrote a long piece about the discoveries I had made over the past year that would improve gensuitemodule to the MacPython mailing list. I think the ball is really going to start rolling this year. A lot more people are interested in Python, Mac OS X, and scripting, and Python has huge untapped potential in this area. I think the lack of documentation of the Python AppleEvent modules along with the generally arcane AppleEvent architecture prevented all but the most daring (me) from even attempting to do anything.

Good things are afoot.

Switched to MoveableType

Well, since MoveableType for my wife on our web host, I figured I might as well switch pyx to it -- at least until I finish Blog enough to want to use it for real. It's pretty close with all the features I added a couple weekends ago, but not close enough yet. It seems like working in concentrated spurts is a good way for me to get things done. I'll think about how I want things to be for a while, and when I get around to doing it I'll just whip it out really fast. Developing in Woven is turning out to be a real joy -- the code is clean enough that I can leave it lying around for months and come back to it with no problem, adding features easily.

Woven models and the request object

It has irritated me for a while now that the interfaces on IModel don't take the request object as the first argument. When I originally wrote IModel, it was written in the context of "here is a general MVC module, and Woven will build upon this", but that turned out to be more trouble than it was worth -- there's no sense separating things out when you will never use MVC standalone. So the other interfaces, IView and IController, which were already considered web-aware, had access to the request, but not IModel.

I have been aware for some time now that IModel is an interface that is about preparing a model for display in a web browser -- much like Zope3's IView. It converts from whatever raw object interface is present on the original model, to an interface which exposes data suitable for traversal and insertion into a web template file.

So, tonight I went through and added the resource object as the first parameter to lookupSubmodel, getSubmodel, setSubmodel, getData, and setData. Changing the signature of methods which are widely used in a wide variety of situations is painful -- especially when you're changing an interface which is as fundamental to a system as IModel is to woven. However, thanks to the inspect module and some fancy use of positional optional arguments, I was able to come up with a solution that logs DeprecationWarnings whenever a method doesn't expect the request, and passes the request when a method does.

Whoops. I left out a

Whoops. I left out a step that is required to get FlashConduit working properly in the post below. You need to pass --flashconduit 4321 to mktap web to make the FlashConduit server side component listen on port 4321, like this:

mktap web --path=/path/to/web/files --flashconduit=4321

I'll edit the post below to mention this.

For people who are curious

For people who are curious about the bleeding edge of Woven development, I have been working on LivePage a bit more recently. LivePage allows asynchronous events both from the browser, and to the browser after the page has already been loaded. The current implementation in CVS requires Flash, but it is also technically possible to get this to work with an IFrame connection that the server never closes as the Output Conduit. I whipped up a little example so you can get a feel for where I am going with this, and give me feedback if you desire.

If you want to try this, save it as an rpy and view it with twisted.web using a tap created like this: (Latest CVS is probably required)

mktap web --path=/path/to/web/resources --flashconduit 4321

Make sure you put a slash after the .rpy, so the relative urls spit out by the webConduitGlue resolve correctly. For example, http://localhost:8080/simpleLivePageExample.rpy/

from twisted.web.woven import model, page, input, widgets


class ANumber(model.AttributeModel):
def __init__(self):
model.AttributeModel.__init__(self)
self.value = 0


class AWidget(widgets.Widget):
def setUp(self, request, node, data):
self.add(widgets.Text(data))


class HandleInput(input.Anything):
def initialize(self):
self.view.addEventHandler("onclick", self.clicked)

def clicked(self, request, widget):
print "CLICKED!"
self.model.setData(self.model.getData() + 1)
self.model.notify({'request': request})


resource = page.LivePage(ANumber())

resource.setSubviewFactory('aView', AWidget)
resource.setSubcontrollerFactory('aController', HandleInput)

resource.template = """<html>
<body>
<div model="value" view="aView">
stuff
</div>

<img src="http://www.google.com/images/logo.gif" model="value" view="Widget" controller="aController" />

<div view="webConduitGlue" />
</body>
</html>"""

Learning about the twisted trial unittesting framework

I used to run the woven tests by themselves a lot using the -file switch of admin/runtests. Since the trial unittest framework has been checked in, this no longer works. It appears the way to run a single test is to use the -m switch to specify the module that contains the tests:

bin/trial -m twisted.test.test_woven

Do not read this.

Do not read this.

from __future__ import generators


class Text(str):
def __init__(self, it):
self.it = it
def __str__(self):
return self.it


def coerceBasic(item):
if isinstance(item, str):
item = Text(item)
elif isinstance(item, tuple):
item = lmx2(item)
assert isinstance(item, Text) or
isinstance(item, lmx2), "wtf did you add %s to me for?" % item
return item


class lmx2(object):
def __init__(self, argument, **kw):
if isinstance(argument, str):
self._name = argument
self._children = []
self._attributes = kw
else:
assert not kw, "Either use a fourple or a tag name and set of keyword arguments"
self._name, self._attributes, self._children, spare = argument
def __lshift__(self, other):
self._children.append(other)
return self
def __rshift__(self, other):
other.__dict__ = self.__dict__
other.__class__ = self.__class__
return other
def _gen(self):
for item in self._children:
yield coerceBasic(item)
def __iter__(self):
return self._gen()
def __getitem__(self, item):
if isinstance(item, int):
return coerceBasic(self._children[item])
return self._attributes[item]
def __setitem__(self, item, value):
if isinstance(item, int):
self._children[item] = value
else:
self._attributes[item] = value
def __delitem__(self, item):
if isinstance(item, int):
del self._children[item]
else:
del self._attributes[item]
def __getattr__(self, name):
if name[0] == '_':
raise AttributeError("no private attrs")
return lambda **kw: self._children.append(lmx2(name, **kw))
def __str__(self):
return ('<%s ' % self._name) + ' '.join(['%s="%s"' % (key, value)
for key, value in self._attributes.items()]) + ('>...' % self._name)

Man, I _told_ you not to read this. Why you always trippin like that?

Twisted logging

In the CVS version of twisted, there have been some modifications to make the default logging less intrusive (it won't write to stdout by default). If you just use twistd to run your apps, you shouldn't notice any difference, but if you like to write scripts that you run directly that start the reactor directly, you might be wondering what happened to logging. Here's a little recipe to turn on logging again.

import sys
from twisted.python import log
log.startLogging(sys.stdout, 0)

If you wish, you can also use any file in place of sys.stdout.