Personal Web Proxy

I wrote this as a comment on 0xDECAFBAD, and thought I would duplicate it here:

Personally, I am very interested in developing and using the PersonalWebProxy. The first thing I thought about writing was a look-ahead accellerator -- when you visit a page, the proxy fetches that page and all the pages that are linked from that page, so when you click a link the page is already downloaded.

Another thing I'd like is a permanent history. Whenever I use any browser to browse through the proxy, the PWP should keep track of where I have been, and allow me to review this history chronologically, and also perhaps automatically sort things by keyword. I'd also like this history to be (once I have set up PWP to do so) synchronized with a remote server, so for example, when I am browsing at home, my history file gets synchronized with my web host, and when I am browsing at work, the same history file on my web host is updated.

Something that goes along with this idea is a permanent bookmarks interface. Visiting the PWP application on a certain web port should show me a list of bookmarks and allow me to add to them, categorize them, etc. The big advantage here would be the syncronization I just talked about with my web host space. My bookmarks would then be available from any host on the internet; no matter what machine I am using (as long as I'm running PWP), my bookmarks would also be locally available.

Another idea I have been thinking about ever since you started talking about the PersonalWebProxy project is something like a mail proxy. The way it would work is, you would set up the PWP to connect to your mail account as a client. It would then download certain mail messages (more about that later) and re-serve them as a local mail server. You would then set up your email client to talk to the mail server running inside PWP. Thus, the same sort of transformation that occurs while you're browsing web pages occurs to your email.

The first thing I would do with this is set up the server only to download email from email addresses in a certain list. Then I would allow rules based sorting. Of course, this functionality is all present in a basic email client, but once you're relatively sure you're not downloading spam you can start doing more interesting things. (The idea of only downloading email messages from people in a list is similar to only accepting instant messages from people in your buddy list)

So things start to get more interesting: We index email for searching, and synchronize email with the email repository on my web host; we parse email messages from friends looking for URLs and have the web proxy immediately download the pages so they are on the hard drive for later, instant viewing; we can send ourselves commands in a simple command language from anything that can send email. Commands could be things like "Bookmark this URL that I don't have time to look at right now"; "Delay delivery of this reminder until the event is near"; etc.

Anyway, this is all just wild pie-in-the-sky dreaming, which I don't normally do. But the prospects of improving our daily internet experience with these ideas, using an application built on Twisted, is very exciting, and very possible.