Better late than never, here is a link to a PDF of my slides from my OSCON talk. They look a little funny because the drop shadows from OmniGraffle didn't translate over perfectly, but the content is all fine.
http://www.soundfarmer.com/content/slides/PythonAppleEvents.pdf
9/04/2003
6/28/2003
Apache, ProxyPass and Twisted
One of the ongoing problems with writing web applications in Woven, my Twisted web app server and templating system, is creating links to other web pages in the system without creating bugs in those links.
Twisted stores the 'host' header sent by the browser in the request, and it stores the path segments that were used to locate the current Resource object in request.prepath. request.prepath is a list of strings indicating the URL segments leading up to the current Resource object. request.postpath is a list of path segments that have not yet been handled, but this is usually not terribly interesting.
So, it is possible, for example, to construct sibling urls, parent urls, and child urls by making a copy of request.prepath and performing list operations on it:
new = request.prepath[:]
new[-1] = 'someSibling.html'
Now that we have a list of path segments leading to a sibling Resource, we can construct a URL:
url = 'http://%s/%s' % (request.getHeader('host'), '/'.join(new))
While this works for simple cases, it breaks down when things get more complex. For example, the browser could be talking to the server over https, and while it is possible, it is non-trivial to detect this and construct a URL with the proper method. Duplicating this code throughout user-level code leads to many buggy implementations, with subtle problems that don't show up unless under very specific configurations.
For example, it is often desirable to run twisted.web behind a reverse proxy, such as using apache and the ProxyPass directive. (Someone please correct me if this is not the correct terminology.) In this case, the user browses to a URL served by the Apache server, and Apache forwards the request to the port upon which Twisted Web is listening. However, while it is doing this, it changes the "host" header from the header originally sent by the browser, to the host upon which Apache is running. Which means if we rely on the host header to construct our new URLs, they will reference the private proxy server rather than the public Apache server.
When I first began working on the Twisted project, one of the classes I wrote was called PathReferenceAcqusitionContext. It had this long name for historical reasons, and I shall simply refer to it as PathRef. One of the abilities of this object, which you could construct by calling the pathRef method on the Request, was to generate other PathRef objects by calling methods such as child, sibling, parent, etc. Then, once you had navigated to the conceptual URL location you desire, you could convert it to a URL string.
This implementation of PathRef was never documented, and had some other, unfortunate, unrelated properties which made it a perpetual pain in the ass. Glyph, finally fed up with the unused and problem-causing PathRef, removed it recently, and it is no longer in Twisted 1.0.6. Which is a good thing.
However, Glyph, finally seeing the need for a way to talk about URLs abstractly in a secure and convenient way, has kindly written the very good (in my opinion) Twisted/twisted/sandbox/paths.py, an implementation of all that was good about PathRef. I hope this gets moved into Twisted fairly soon, and a method for generating a URLPath representing the current Resource is added to the Request.
However, this method needs to be able to understand that the 'host' header is not necessarily the address to which the outside world refers. I decided to do a little experiment to see what Apache would do when performing a ProxyPass, and if we could determine where the outside world considers our application to live. When I was running Apache on port 8081 on my machine, and it was set up to forward requests to Twisted Web, I was able to observe that Apache had inserted an additional header in the process of forwarding the request:
'x-forwarded-host': 'localhost:8081'
The factory method on the request which generates a URLPath object should check to see if 'x-forwarded-host' has been set, and compensate accordingly. This should be a good solution to a long standing problem with using Twisted Web.
Twisted stores the 'host' header sent by the browser in the request, and it stores the path segments that were used to locate the current Resource object in request.prepath. request.prepath is a list of strings indicating the URL segments leading up to the current Resource object. request.postpath is a list of path segments that have not yet been handled, but this is usually not terribly interesting.
So, it is possible, for example, to construct sibling urls, parent urls, and child urls by making a copy of request.prepath and performing list operations on it:
new = request.prepath[:]
new[-1] = 'someSibling.html'
Now that we have a list of path segments leading to a sibling Resource, we can construct a URL:
url = 'http://%s/%s' % (request.getHeader('host'), '/'.join(new))
While this works for simple cases, it breaks down when things get more complex. For example, the browser could be talking to the server over https, and while it is possible, it is non-trivial to detect this and construct a URL with the proper method. Duplicating this code throughout user-level code leads to many buggy implementations, with subtle problems that don't show up unless under very specific configurations.
For example, it is often desirable to run twisted.web behind a reverse proxy, such as using apache and the ProxyPass directive. (Someone please correct me if this is not the correct terminology.) In this case, the user browses to a URL served by the Apache server, and Apache forwards the request to the port upon which Twisted Web is listening. However, while it is doing this, it changes the "host" header from the header originally sent by the browser, to the host upon which Apache is running. Which means if we rely on the host header to construct our new URLs, they will reference the private proxy server rather than the public Apache server.
When I first began working on the Twisted project, one of the classes I wrote was called PathReferenceAcqusitionContext. It had this long name for historical reasons, and I shall simply refer to it as PathRef. One of the abilities of this object, which you could construct by calling the pathRef method on the Request, was to generate other PathRef objects by calling methods such as child, sibling, parent, etc. Then, once you had navigated to the conceptual URL location you desire, you could convert it to a URL string.
This implementation of PathRef was never documented, and had some other, unfortunate, unrelated properties which made it a perpetual pain in the ass. Glyph, finally fed up with the unused and problem-causing PathRef, removed it recently, and it is no longer in Twisted 1.0.6. Which is a good thing.
However, Glyph, finally seeing the need for a way to talk about URLs abstractly in a secure and convenient way, has kindly written the very good (in my opinion) Twisted/twisted/sandbox/paths.py, an implementation of all that was good about PathRef. I hope this gets moved into Twisted fairly soon, and a method for generating a URLPath representing the current Resource is added to the Request.
However, this method needs to be able to understand that the 'host' header is not necessarily the address to which the outside world refers. I decided to do a little experiment to see what Apache would do when performing a ProxyPass, and if we could determine where the outside world considers our application to live. When I was running Apache on port 8081 on my machine, and it was set up to forward requests to Twisted Web, I was able to observe that Apache had inserted an additional header in the process of forwarding the request:
'x-forwarded-host': 'localhost:8081'
The factory method on the request which generates a URLPath object should check to see if 'x-forwarded-host' has been set, and compensate accordingly. This should be a good solution to a long standing problem with using Twisted Web.
3/06/2003
Generators and Python/AppleEvents
I started writing some code in aetypes that adds iteration support to DelayedComponentItem today. Basically the idea is you would be able to do this:
import mail
m = mail.mail()
for msg in m.messages:
print msg.subject
That's just pseudocode, but you get the idea. All of the necessary moving parts are already in place to do this, as Jack Jansen noted in this message from 1999:
http://mail.python.org/pipermail/pythonmac-sig/1999-August/001245.html
The bits that I'm doing differently are thanks to Python 2.2's generators, which make it very easy to write support for things like "for foo in bar".
Here's what I'm doing:
I added a __getattr__ to TalkTo that uses the _moduleName attribute I added last year to look for the given name in the correct module. If one is found, I then return a DelayedComponentItem with an additional parameter passed -- self, or the TalkTo instance itself. This allows the DelayedComponentItem to call count, which is a generic AppleEvent which returns the number of items of that type in the specified container (which can be None, to mean the application itself)
Then, I added a __len__ method to DelayedComponentItem that calls count to see how many items of that type there are.
Finally, I added an __iter__ method that does len(self) and returns actual ComponentItem instances for each item in the container. It works perfectly!
The only thing that tripped me up was the fact that AppleEvents like to count things starting with 1 and Python counts starting with 0. So the first ComponentItem returned won't actually be useful for anything and will generate an error if you try to use it. I realized this while I was driving home, and I look forward to fixing it tomorrow morning!
import mail
m = mail.mail()
for msg in m.messages:
print msg.subject
That's just pseudocode, but you get the idea. All of the necessary moving parts are already in place to do this, as Jack Jansen noted in this message from 1999:
http://mail.python.org/pipermail/pythonmac-sig/1999-August/001245.html
The bits that I'm doing differently are thanks to Python 2.2's generators, which make it very easy to write support for things like "for foo in bar".
Here's what I'm doing:
I added a __getattr__ to TalkTo that uses the _moduleName attribute I added last year to look for the given name in the correct module. If one is found, I then return a DelayedComponentItem with an additional parameter passed -- self, or the TalkTo instance itself. This allows the DelayedComponentItem to call count, which is a generic AppleEvent which returns the number of items of that type in the specified container (which can be None, to mean the application itself)
Then, I added a __len__ method to DelayedComponentItem that calls count to see how many items of that type there are.
Finally, I added an __iter__ method that does len(self) and returns actual ComponentItem instances for each item in the container. It works perfectly!
The only thing that tripped me up was the fact that AppleEvents like to count things starting with 1 and Python counts starting with 0. So the first ComponentItem returned won't actually be useful for anything and will generate an error if you try to use it. I realized this while I was driving home, and I look forward to fixing it tomorrow morning!
3/05/2003
Scripting Mac OS X with Python using AppleEvents is accepted
Well, my OSCON proposal about scripting Mac OS X applications was accepted. Coincidentally, just yesterday I wrote a long piece about the discoveries I had made over the past year that would improve gensuitemodule to the MacPython mailing list. I think the ball is really going to start rolling this year. A lot more people are interested in Python, Mac OS X, and scripting, and Python has huge untapped potential in this area. I think the lack of documentation of the Python AppleEvent modules along with the generally arcane AppleEvent architecture prevented all but the most daring (me) from even attempting to do anything.
Good things are afoot.
Good things are afoot.
Switched to MoveableType
Well, since MoveableType for my wife on our web host, I figured I might as well switch pyx to it -- at least until I finish Blog enough to want to use it for real. It's pretty close with all the features I added a couple weekends ago, but not close enough yet. It seems like working in concentrated spurts is a good way for me to get things done. I'll think about how I want things to be for a while, and when I get around to doing it I'll just whip it out really fast. Developing in Woven is turning out to be a real joy -- the code is clean enough that I can leave it lying around for months and come back to it with no problem, adding features easily.
Subscribe to:
Posts (Atom)