The last release of eventlet was 0.2, which we did when we re-open-sourced the fork of eventlet I worked on while I was at Linden Lab. 0.2 was released quite a while ago, and eventlet has seen significant improvement in the meantime.
The main change in this release is the ability to use libevent as the multiplexing api instead of raw select or poll. If libevent and the Python wrapping are not installed, eventlet will still fall back, first checking for the presence of poll and falling back to select if it is not available.
Another major change in this release is a much improved eventlet.wsgi server. The wsgi server now supports Transfer-Coding: chunked as well as Expect: 100 Continue, and is quite fast. I tested it against an eventlet based wsgi server I wrote which uses wsgiref (from the Python 2.5 standard library) and my informal tests showed eventlet.wsgi being several hundred requests a second faster at serving a "Hello, World!" wsgi application.
This release also features significant refactoring, cleaner code, support for cooperative operations on pipes (and unix domain sockets) as well as sockets, more tests, and docstrings for pretty much everything. The documentation, which was non-existant before, is now pretty comprehensive.
To install, just "easy_install eventlet" and start hacking!
- PyPI page: http://pypi.python.org/pypi/eventlet/0.5
- Overview: http://wiki.secondlife.com/wiki/Eventlet
- Documentation: http://wiki.secondlife.com/wiki/Eventlet/Documentation
- Mercurial Repository: http://donovanpreston.com:8888/eventlet
12 comments:
I haven't looked at the code yet, but I have to ask: how hard would it be to build a high capacity Comet server on eventlet.wsgi?
What's the WSGI server like? Threaded? wsgi.input hooked up to the socket, or does it pre-load the input? With a paste deploy entry point people could more easily try it out with just a line of config (http://pythonpaste.org/deploy/#paste-server-factory)
@Simon Willison: Very, very easy. Comet servers are why I have been working on networking libraries for Python for so long :-)
There are already two experimental Comet servers in mulib, based on the older eventlet.httpd (which is deprecated; wsgi is better(tm)).
mulib.eventrouter is a Comet server that supports async message-passing (no return values) from javascript to python and from python to javascript. I don't really like this approach any more and won't be working on it further.
mulib.pantheon is an interesting experiment; javascript clients register for server-side event callbacks using the observe function like this: observe('foo/bar', function(...) { ... }). Then, assuming the Pantheon app is installed at /, other clients can trigger the function by doing PUT or DELETE to /foo/bar. I'm very happy with these semantics and hope to explore this approach further again in the future.
Another thing I should definitely do is a Bayeux implementation on top of eventlet. It would be trivial and useful to do, I just haven't made the time to do it since I don't have any real-world use for a Comet server at the moment and I obviously like experimenting with Comet semantics. Are there any other Comet standardizations besides Bayeux that I should be aware of?
Thanks for the comment!
@Ian Bicking: eventlet is a microthreading library using greenlet, so socket calls cooperatively switch to another greenlet that's ready to run. The event hub uses libevent, poll, or select to decide what's ready to run. So the semantics are like threading without the disadvantages of threading. eventlet's goal is to eventually monkey-patch all python operations which would block to cooperate instead. eventlet already supports monkey-patching the standard socket module, along with operations on pipes and unix domain sockets, and has code for doing blocking database operations in a threadpool (but no transparent monkey-patching yet for database calls; just requires doing it for various popular database libraries).
wsgi.input is hooked up to the socket; reading from it causes a yield until the data is ready to read. There's also a guard around the raw socket object preventing you from reading more than Content-Length from the socket.
I actually have already written a Paste Deploy entry point as an experiment (it works) but haven't released it yet. The server I am going to be deploying to replace the default Paste web server is multi-process to take advantage of multiple cores or processors (as well as each process using cooperative multiplexing) so I just need to do some more digging to figure out how to get the information I need out of the entry point to spawn new processes.
While I'm talking about it, the server also supports graceful reloading on SIGHUP by starting up new processes immediately to handle new requests but allowing the old processes to finish handling whatever requests are outstanding before dying. I will definitely be releasing it open source soon but it just needs more cleanup to be general enough to release.
Thanks for commenting, and thanks for Paste and blazing the trail for wsgi for so long! I really should have been using wsgi for the last couple of years, but I was just too lazy to convert all my old code over.
Eventlet's really slick... it's fast to program in, and very straightforward. It makes asynchronous programming a dream. Deferreds are a pain to deal with and obscure your code flow, but with Eventlet you write almost exactly as you would in a single-threaded program, and it just works.
Now that Donovan's added this WSGI support, it opens up a whole new world of possibilities. I sat down today to create a REST interface to one of my programs; I just created a tiny WSGI app, threw WebOb in there to make life easy, and plugged it into Eventlet's WSGI server. All the benefits of asynchronous code, without the hassle... and I'll be able to come back to that code in another month and still know exactly how the code flows.
Donovan's been twittering about integrating pyprocessing with eventlet in some way (Donovan, if you have more info about that, I'd love to hear your progress.)... talk about real possibilities. No question, this is great stuff.
@Marcus Cavanaugh: Thanks for the praise :-) It's been a couple of years now since I have been heavily using coroutines with greenlet, and I have been really impressed with just how easy it is to program with coroutines and how fast greenlet is. You do have to be careful to write code that doesn't block, but if you control all the code and monkey-patch anything out of your control that blocks with a cooperating implementation, it's very easy to write correct code. It's easy to reason about what a piece of code is going to do without having to worry about preemption, and if you want to run some code that you don't trust or is going to block it's easy to use another process or thread for that.
Don't forget to check out mulib if you are interested in mixing REST in with wsgi. It provides REST adapters for basic python types and makes it easy to expose your existing Python classes by registering a handler for your class, which is just a wsgi application callable. Mulib is still much more experimental than eventlet and is subject to change, but I think it's getting pretty useful.
Clone the mercurial repository to get the wsgi-aware mulib: http://donovanpreston.com:8888/mulib
http://wiki.secondlife.com/wiki/Mulib/Documentation
As far as PyProcessing integration goes, the eventlet.util.wrap_pipes_with_coro_pipes function is the result of my initial attempt to monkey-patch enough functions to get PyProcessing to work. Unfortunately, PyProcessing uses blocking operations in a C extension module to do inter-process communication, so it does not currently work with eventlet. I do plan on re-implementing the protocol with pure python and monkey-patching PyProcessing to get it to work, but I haven't gotten around to it.
With wrap_pipes_with_coro_pipes you can however use standard fork/exec/popen style multi-process programming. After calling wrap_pipes_with_coro_pipes, calls to os.read and os.write on a pipe filedescriptor will cooperate, and calling os.fdopen will give you a GreenPipe object whose read/write/etc methods will also cooperate. There is also eventlet.processes.Process which is an easy to use wrapper around popen.
Thanks for checking out eventlet, and be sure to tell me if you have any feature requests or find any bugs!
easy_install eventlet failed on the greenlet dependency. Any news on a greenlet binary that works with OS X 10.5?
@Simon Willison: How did it fail? It worked for me. Do you have Xcode installed? gcc isn't installed unless you do.
However, I would like to do another release of greenlet as an egg. If I do I will make sure to provide a binary for OS X 10.5.
Also, eventlet will automatically detect and use the greenlet that comes with the py.lib, but I dunno if there are 10.5 binary eggs for that either.
$ sudo easy_install eventlet
Best match: eventlet 0.5
Processing eventlet-0.5-py2.5.egg
eventlet 0.5 is already the active version in easy-install.pth
Using /Library/Python/2.5/site-packages/eventlet-0.5-py2.5.egg
Processing dependencies for eventlet
Searching for greenlet
Reading http://pypi.python.org/simple/greenlet/
Reading http://undefined.org/python/#greenlet
Best match: greenlet 0.1
Downloading http://pypi.python.org/packages/source/g/greenlet/greenlet-0.1.tar.gz#md5=b9536375126dd373abdd4a43cb70136f
Processing greenlet-0.1.tar.gz
Running greenlet-0.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-UR4hWT/greenlet-0.1/egg-dist-tmp-zrr98e
zip_safe flag not set; analyzing archive contents...
No eggs found in /tmp/easy_install-UR4hWT/greenlet-0.1/egg-dist-tmp-zrr98e (setup script problem?)
error: Could not find required distribution greenlet
@Simon Willison: Uh. Hmm. Maybe try easy_install greenlet first? That's what I did and it worked for me(tm). I don't know too much about eggs and dependencies, but I guess I'll learn if I have to.
You could also just download the greenlet source tarball separately and 'python setup.py install' in there.
I was wanting to give eventlet a try but when I went to install greenlet, I get an error. This is on Mac OS X 10.5.5 with the developer tools installed.
greenlet.c:295:2: error: #error "greenlet needs to be ported to this platform, or teached how to detect your compiler properly."
greenlet.c: In function �initgreenlet�:
greenlet.c:897: error: �slp_switch� undeclared (first use in this function)
greenlet.c:897: error: (Each undeclared identifier is reported only once
greenlet.c:897: error: for each function it appears in.)
switch_amd64_unix.h: In function �slp_switch�:
switch_amd64_unix.h:39: error: PIC register �rbx� clobbered in �asm�
switch_amd64_unix.h:51: error: PIC register �rbx� clobbered in �asm�
lipo: can't open input file: /var/folders/CV/CVPPvz5lEMGto97R0w7NKE+++TI/-Tmp-//ccyG7ZDO.out (No such file or directory)
error: command 'gcc' failed with exit status 1
Any advice on what to do? Thanks.
@Greg: Hmm. Greenlet definitely works on OS X since it's what I use. Did you run "easy_install greenlet"? I've also compiled greenlet from source without any problems... I'm not sure why you are having problems. Maybe you have another version of Python installed such as the one from Fink or MacPorts? I'm not sure why it wouldn't just work there either, though.
Post a Comment