Over the past few years, I’ve made a number of little Web applications that are actually just HTML pages.

Building things this way is really fun and really simple. It’s easy to understand and remix because there’s no custom server-side infrastructure to complicate matters. In some ways, it’s just like writing my first Web pages in the 1990′s, only now I can use JavaScript for more than just image rollovers.

I’ve always run into a roadblock, though, when I’ve wanted to make one of my apps social. There aren’t any Ajax APIs I know of that can be used to share information in a way that isn’t vulnerable to spammers and other kinds of misuse, so I made one.

Twitblob is a simple cross-origin RESTful API that gives all Twitter users a small amount of public JSON blob storage on a host server. It also handles three-legged Twitter OAuth key exchange, allowing web pages to add social functionality with minimum fuss.

In a lot of ways, Twitter is a spam-resistant identity provider: its staff deals with the incredible hassle of keeping out malicious users, which makes it possible for me to offer 20k of storage to every Twitter user across any page on the Web without having to worry too much about misuse. So far I’ve used my Twitblob endpoint for a simple voting booth for my local Awesome Foundation chapter, as well as a prototype scheduling application for the 2010 Mozilla Summit. It’s been nice for adding social functionality to a page in a pinch.

Since learning JavaScript over a year ago, it’s become one of my favorite dynamic programming languages alongside Python. And as I’ve mentioned before, I think the two languages actually complement each other pretty well.

Python, at its heart, is a platform that’s built to be extended. The evidence for this is plentiful: there’s modules and packages out there that offer practically any functionality you want, from web servers to 3D game engines to natural language processing toolkits and more, all instantly accessible through a simple command or an installer download. Yet one of the costs of all this generativity has been the fact that Python doesn’t really have much of a security model to speak of: any Python program has as much access to the underlying system as the current user does, which, compared to the Web, is basically omnipotence. Creating programs that obey the principle of least privilege is pretty hard.

JavaScript, on the other hand, has many of the opposite problems. For one thing, it’s really built for embedding: until the very recent advent of CommonJS and Narwhal, for instance, the language has always lacked a general-purpose platform and standard library. A Pythonic way of saying this is that the language doesn’t come with “batteries included”, but this can actually be a good thing from a security standpoint: because the simplest possible embedding has no privileges and needs to be explicitly given all its capabilities by its embedder, it’s very easy to follow the principle of least privilege. Recent work on membranes and capability models puts JavaScript way ahead of many other languages in the security realm, yet the lack of a mature general-purpose platform has meant that anyone who’s wanted to leverage these strengths has always had to muck around in C/C++ to create the kind of embedding they wanted.

Well, to an extent. One of the many aspects of Java that I’ve frequently been envious of has been Rhino, a JavaScript engine written entirely in Java, which allows anyone who knows Java to create their own embedding solution that leverages Java’s strengths. But I prefer Python to Java, and moreover, the engine itself isn’t worked on with as much intensity as the JS engines that power real-world consumer products like V8 and SpiderMonkey—so new language features are slow to be implemented and performance isn’t great.

I’d briefly tried resurrecting John J. Lee’s Python-Spidermonkey last year but I soon discovered that it wasn’t really what I wanted. For instance, JS objects were copied into Python approximations as they crossed the language boundary, which resulted in a “lossy” transfer and prevented features like identity perseverance. It was essentially a high-level wrapper created to solve a specific problem, rather than a low-level tool intended to enable any kind of wrapping based on context (e.g., how trusted the JS code is).

Introductions

In part because of all this, and in part because I’d always wanted to write a Python C extension from scratch, I’ve decided to create a new Python-Spidermonkey bridge: Pydermonkey.

Pydermonkey’s mission is pretty simple and straightforward: it’s just meant to wrap Spidermonkey’s C API as faithfully as possible—including its debugging API—while enforcing the memory safety that Python is known for. This makes it awfully low-level for casual programmers, but thanks to Python’s awesome support for magic methods, it’s not hard to create high-level wrappers that provide much more convenient bridging between JavaScript and Python code.

Where It’s At

Pydermonkey is currently at version 0.0.6; its API supports a decent subset of the Spidermonkey C API, but it’s still quite lacking in places: operation callbacks will allow you to run untrusted code that runs in infinite loops, throw hooks allow for full Python-esque stack tracebacks of JS code, yet property catchalls haven’t yet been implemented, which means that security is constrained to conventional sandboxing (membranes and object capabilities aren’t currently possible). There’s also the nasty problem of not being able to detect reference cycles that cross language boundaries, which means that such cycles need to be broken manually for now.

Getting It

Pydermonkey is available at the Python Package Index in source form, and as a precompiled binary for the few platforms that I happen to have access to at the moment.

You should be able to type easy_install pydermonkey at the command line and everything should “just work”: I’ve set up the Paver build script such that the Spidermonkey source code is automatically downloaded and built before the C extension if you’ve got the compiler toolchain on your system, though there are a few snags on Windows to circumnavigate. For more information, read the Pydermonkey documentation. And please feel free to file a bug if you run into one!

Where To Go From Here

If you’d like to see an example of a high-level wrapper, check out my Pydertron experiment. It provides a simple interface to expose untrusted JS functionality to Python code and also contains a CommonJS-compliant implementation of the SecurableModule standard. I’m also playing around with creating a Pydermonkey engine for Narwhal on github; contributions to any of these codebases are more than welcome, and there’s some low-hanging fruit in Pydermonkey that would be perfect for students or first-time contributors.

Finally, if you do anything interesting with Pydermonkey, I’d love to know about it.

We want to make Ubiquity awesome at reporting errors. In our original release, a transparent message with JavaScript exception information was displayed, which wasn’t very useful to the average user, and was downright annoying when dozens of exceptions were logged in the same instant.

At present, running a command that raises an error just results in that message being logged to the JS Error console, which very few people know how to access—so most people are left scratching their heads and wondering why their command is taking so long to run.

For the next release of Ubiquity, we’re going to be trying something more user-friendly: if a command encounters an error, a transparent message will be displayed telling the user that it didn’t work. The message will also recommend using the “report-bug” command to send information about the bug to the Ubiquity team. If the user decides to run this command, a page is opened that looks like this:

Aside from inviting the user to describe their problem, a lot of information is included about their system: what OS they’re using, what extensions and plugins they have installed, what recent exceptions were thrown, and so forth. We’re hoping this will lower the barrier to entry both for receiving and providing technical support, since most of the information needed to describe and investigate a problem is contained in a single link.

We don’t yet have an interface for browsing existing bugs, but we do have a display for viewing them. It looks pretty similar to the page for submitting a bug:

One interesting aspect of our bug reporting system is that we’re not using numbers to identify bug reports: they get big fast as many reports are submitted, and big numbers are hard to remember. Instead, we’re mashing two random words from the dictionary together. For instance, the first bug I reported using this system was called anaphorically-spinach.

Under The Hood

The bug reporting system is pretty lightweight: mostly it’s just static HTML/JavaScript code that talks to a web service that’s implemented in under 100 lines of Python. The actual bug report is just a JSON object, and is deposited into a CouchDB server.

The big advantage of using CouchDB here is that we’ll be able to easily create really rich queries using plain old JavaScript. For instance, here’s a query that shows all reported error messages that contain the text “Invalid chrome URI”. It won’t be hard to create complex queries that, for instance, give us all the bug reports in which the user had a certain extension installed and had a command crash at a particular line in a particular file.

A Public Asset

Right now all reports are submitted to the public domain, and as such the report database is a public asset; users are informed of this before they submit the bug, and encouraged to look at the additional data that’s being sent with their report to ensure that there’s nothing sensitive in there. In the future, it’d be nice to allow the user to click on any parts of the data that are personally identifying, so that they can submit a version of the report that masks out the sensitive information.

Reuse

The bug report system has been designed to be decoupled from Ubiquity itself. For instance, the report viewing application is designed as a reusable JavaScript component, so it should ultimately be easy to embed into any web page. In other words, it should be easy to use as a bug reporting mechanism for any Firefox extension—and possibly for any web application in general. If you’re interested in adding the component to your own project, please let us know; the code is still a work-in-progress, and any contributions or comments are appreciated.

Yesterday Tempura left a good question as a comment on my blog post concerning Ubiquity’s experimental support for Python:

What about http://pyxpcomext.mozdev.org ? They bring Python in an xpi for all major plattforms, without the need of an local installed interpreter.

Using PyXPCOM was actually a potential option we had considered, but we ended up going with jsbridge for a number of reasons:

• PyXPCOM uses XPCOM as its means of communication between Python and the Mozilla platform. Unfortunately, XPCOM is super heavyweight and it’s a particularly difficult hurdle to get through when all you really want is for two very dynamic languages to communicate with each other. For instance, Ubiquity itself implements very few XPCOM interfaces, and most of its functionality is exposed through JS Modules; jsbridge allows us to access these JS Modules directly, while accessing them from PyXPCOM would require us to create verbose XPCOM wrappers around them—a particularly egregious amount of work when you think about how similar JavaScript and Python are.
• One advantage of using the local system’s Python installation is that we inherit whatever packages come with it, which means that on OS X 10.5, for instance, we automatically get access to Python’s bindings to Objective C and Cocoa, whereas with PyXPCOM we really have to manage the user’s distribution manually. It could similarly be said that PyXPCOM takes an embedding approach, while jsbridge takes an extending approach; a very good analysis between these two approaches has been written by Glyph Lefkowitz here.
• In regards to Windows, putting a distribution of Python on the end-users system that any other application can leverage is a good thing—much better than requiring the user to download a big package and then only allowing Firefox to use it. This gets into the idea of Python as Platform that I’ve written about before.
• In regards to OS X and Linux, using the pre-existing installation of Python is obviously nice because it doesn’t require the end-user to download anything (aside from mozrunner and jsbridge, which are pretty small).
• jsbridge has no binary components, which makes it a lot easier to maintain and experiment with than PyXPCOM, which is difficult to compile and extend.

PyXPCOM is definitely useful for a lot of things, though—for instance, manipulating the DOM of a page—so I think there’s still some things we can leverage there. And jsbridge certainly presents its own hurdles and limitations, too, but hopefully this post sheds some light on the thought processes that went into deciding which mechanism to use.

A few weeks ago I wrote about Ubiquity Feed Plugins, which are basically just a way of separating the user interface of subscribing to a new feature from the implementation of the feature itself.

As I’ve written about before, one of the things I’ve missed about the Mozilla development environment is its support for the Python programming language. Aside from being humane and having a great community, it has functionality that could complement the Mozilla platform quite nicely. So we’ve whipped up a quick proof-of-concept Python Feed Plugin for Ubiquity to explore this possibility.

Here’s a really simple command feed written in Python:

def cmd_sample_python_command(ubiquity):
    ubiquity.displayMessage('Hello from Python %s.' % sys.version)

cmd_sample_python_command.preview = 'a sample python command!'

For anyone who knows Python and has read the Ubiquity Author Tutorial, this should be fairly self-explanatory. The ubiquity argument is just passed in to the command execution function to make unit testing as easy as possible.

As with any Ubiquity feed, the file containing the above code needs to be referenced in the <HEAD> element of a web page:

<LINK REL="python-commands" HREF="sample_command.py">

In this case the python-commands value of the REL attribute is what tells Ubiquity to process this feed using the Python feed plugin.

Visiting the web page containing this tag results in the same subscripton panel that shows up with any page that has a Ubiquity command feed. Because Python code executes at the same privilege level as default Ubiquity feeds, the Python feed plugin really ought to display the same warning page that default Ubiquity feeds show when the user clicks “Subscribe”; but because this is just a proof of concept, we skipped that implementation and currently only allow Python feeds to be used when they already exist on the user’s local filesystem—in other words, when the feed is at a file: URL.

The first time an end-user subscribes to a Python command feed, they’re presented with this non-intrusive message:

What happens behind-the-scenes is a bit more complex: Ubiquity uses virtualenv to bootstrap a sandboxed Python virtual environment inside the user’s profile directory. It then uses easy_install to install jsbridge, and then sets up a “feed server” process on the user’s system, which Ubiquity communicates with over jsbridge to execute Python code:

Pretty Frankenstein, but it works on OS X and Linux. Because Windows doesn’t come with Python built-in like the other two operating systems do, we’ll have to bootstrap it first, which we haven’t gotten around to yet.

So, if you’ve got OS X or Linux and like Python, feel free to play around with this; a sample Python command feed is included with the latest Ubiquity beta. If this is something that seems interesting to you, we’d love to know—you’re welcome to leave a comment on this post, drop us a line on the ubiquity-firefox Google Group, or say hello on #ubiquity on irc.mozilla.org.

In my recent post on Trusting Functionality I alluded to a socially-based framework for trust that would allow software to be generative and safe at the same time.

When trying to figure out a solution to this problem, I realized that there are already communities on the internet that have built-in social mechanisms for trust. Python, for example, is a language notorious for its lack of protection against untrusted code. Yet we don’t see much concern that a Python script may contain malicious code, even though it has the ability to do whatever it wants to our computer. Why is this?

One obvious answer has to do with the users of Python code: because there’s relatively few of them and they tend to be quite technically skilled, there’s a high risk involved in creating malicious Python code, and little economic gain to be made from it.

But another answer, I think, has something to do with community. A open-source community is a lot like a corporation with a few key differences: its processes are largely transparent, and the software itself is almost guaranteed to be developed by its users. The former allows for peer review and accountability, while the latter helps ensure that the software’s features are always in the best interests of the users. These are pre-existing social mechanisms that help create trust between an open-source community and its stakeholders.

So ultimately, a project’s community can be at least as reliable as a corporate entity: the economic incentive structure of the latter is replaced by a social incentive structure in the former. I trust that the Python SVN repository won’t contain malicious code because I trust the community to only grant commit privileges to those it trusts, and I trust the community’s server administrators to ensure that the python.org domain won’t be hacked.

Another advantage of the transparency of open communities is the fact that—since well before the advent of Facebook and MySpace—they have meaningful social networks embedded in them. One merely has to take a look at the Subversion commit logs for the Python code repository to see who its most highly-respected members are, for instance, and public newsgroups and forums can be mined to discover other relationships. What this means is that it’s possible for us to infer—or be told explicitly—what individuals a community trusts.

So, individuals trust certain open communities and vice versa. This information can be leveraged to create a relatively low-cost web of trust which can be used in a variety of ways.

For instance, let’s assume for a moment that Mozilla’s source control system at hg.mozilla.org supported OpenID and had a simple web API that allowed a remote service to query whether or not a particular user has commit privileges. This would allow Ubiquity‘s source control system—which is hosted here on Toolness—to instantly inherit the permission system from Mozilla: anyone trusted enough to have commit privileges to Mozilla’s code repository would instantly be able to commit to Ubiquity. (This is, by the way, the reason that Ubiquity isn’t currently hosted on hg.mozilla.org: it forces us to think of ways to decentralize the Mozilla community.)

There’s at least one existing web of trust that we can draw from, too: the open-source social networking site Ohloh uses a ranking system called Kudos, which could be used as a rough measure of trust, to make inferences about whether an arbitrary piece of code from a known programmer can be trusted.

Such webs of trust would be useful for things other than code, too; it could be used as an alternative to spam-filtering to determine whether content can be trusted. Imagine a workflow in which blog software draws from publicly-available social history to see if a comment posted by an OpenID-logged-in user is spam. This means, for instance, that my Wikipedia history and my Yelp standing could be used to infer that a comment I leave on a blog isn’t spam, obviating the need for frustrating and error-prone captchas.

Inter-community trust doesn’t have to be the only form of trust we use to make our decisions, either; it can easily be used in conjunction with the hierarchical trust system that the web currently uses, for instance; or individual webs of trust can be leveraged from existing social network sites like LinkedIn, as long as the final solution doesn’t inconvenience the end-user.

This is just one potential social solution to the problem of participating in a digital ecosystem with bad actors who have economic incentives to hurt others. If you have any ideas for other solutions to the trust problem, or know of any existing ones, I’d love to hear them.

Over the past few days my friends Ben Collins-Sussman and Jim Blandy and I have been having an interesting conversation about the use of Mercurial for development collaboration. Eventually one of my email responses got so long-winded that I figured it’d be best to make the conversation public.

So, here’s my take on Mercurial, and some reasons for why a HG birds-of-a-feather session at the Mozilla Summit coming up next week would be very useful for me.

To begin with, from a purely social standpoint, the concept of distributed revision control is amazing to me because of how it removes many technological barriers to collaboration, providing software projects with an enormous amount of freedom on how their development process is structured. For any readers who aren’t familiar with it, check out Chapter 1 of the HG Book.

But with all this additional freedom comes additional responsibilities. To quote the MDC Mercurial basics: this gun is loaded.

I’m used to working on small-to-medium sized projects with relatively small teams. Subversion was great for this, because when we were working on things simultaneously, we rarely ran into situations where we were editing the same file, much less the same part of the same file. There were rarely reasons to need to create SVN branches—though we all knew how to do it, and did it when necessary. But as a result, merges were very rare, and when we had to merge, we were extremely careful and diligent about it.

I’m still working on relatively small-to-medium-sized projects (e.g. Weave and Ubiquity) and the forced merging that HG makes us do almost every time we push is a world of pain, relatively speaking. With SVN, I’d just svn commit and see if SVN rejected our commit because someone else committed a change to the same file while we were editing it—this happened rarely, and when it did, we were careful about ensuring that our changes gelled. In this sense, SVN was really humane; 90% of the time things “just worked”, and when things didn’t just work, it was for very good reasons.

But HG almost never “just works”. If I edit a.py and my friend edits b.py and pushes it before I’ve pushed my changes, I have to make a merge commit and manually ensure that nothing bad happened. The end result of this is a huge burden on each programmer compared to SVN, as they have to do a separate merge commit for nearly every push they make, which essentially encourages people to either (A) not push often or (B) ignore their merge commits (a practice which is encouraged by the use of hg fetch). The disadvantages of the first approach are nicely explained by Ben’s post on Programmer Insecurity; the latter approach is bad for obvious reasons.

This is basically the axis around which all my woes with HG revolve. With SVN, it’s really easy to see how code has changed, but because of the constant merging of tiny branches in HG, the whole code history becomes obfuscated and it’s hard to tell what’s happened to it. In fact, several weeks ago a friend somehow mis-merged his commits to Weave, which undid a major refactoring I did, and the really scary thing is that it was somehow impossible for me to tell this had happened from looking at the diff logs alone. I looked at them for a good half-hour or so and was still scratching my head. Needless to say, my inability to understand what had happened to the code by looking at the logs drastically reduced my faith in the tool.

While everyone I know understands the basics of HG and the philosophy behind distributed VCS, it’s the particulars of actually “working in the wild” that many are finding very confusing. So a HG BoF at the Mozilla Summit would be extremely useful.

One of the major challenges we face with the design of our new linguistic command-line project is that of trust. As Zittrain mentions in The Future of the Internet, this is really the fundamental problem of generative systems, and also their most valuable asset: the ability for a user to run arbitrary code is simultaneously what gives the personal computer its revolutionary power, but it’s also its greatest vulnerability.

At present, because our project is still in the prototyping stage, we’re opting for freedom of expressiveness and experimentation over security. That means that all the various verbs we write, while written in JavaScript, are always executed with Chrome privieges, meaning that they’re capable of doing whatever they want to the end-user’s computer.

So the particular dilemma that needs to be solved here is: how can an end-user trust that a verb won’t do anything harmful to their data or privacy—be it intentional or accidental—while still providing a low barrier of entry for aspiring authors to write and distribute their own verbs?

We’ve considered some technical options so far. One is the idea of “security manifests” that come with verbs, specifying what a verb is capable of doing. For example, the “email” verb mentioned in my last post could specify in a manifest that it needed access to the user’s email service and their current selection. This information could then be presented to a user when they choose to install a verb. At an implementation level, the code could run in a specially-constructed sandbox to ensure that the verb code never steps outside the bounds prescribed by its manifest. Alternatively, or in addition to this, an object-capabilities subset of JavaScript like Caja can be used. Such mechanisms ensure that untrusted code can only go as far as the end-user lets it—which, unfortunately, also puts a burden on said end-user. While I don’t personally mind having such a burden myself, I know I wouldn’t want to put it on my friends and family.

Digital certificates are another component of potential solutions, but they too have their own problems. While they’re easy for centralized corporations to deal with, they’re problematic for more distributed operations, and the monetary cost involved in obtaining one significantly increases the barrier to entry for individual software authors. And even signed code doesn’t prevent the more privacy-invasive—but not outright malicious—classes of software like spyware.

As I’ve indicated earlier, this general issue of trust isn’t a new problem, or even just an important issue for an experimental Firefox addon. It’s what Zittrain believes is at the core of the future of the internet and the PC, and the solutions we create—or don’t create—will determine whether their future is one that is sterile or generative. Windows Vista, being the most frequently exploited operating system as a natural result of its widespread use, is the harbinger of a future that relies entirely on technology and corporate trust heirarchies without taking any kind of social mechanisms into account; the result is a notoriously hard-to-use user interface that places an enormous burden on the end-user to constantly make informed security decisions about their computing experience. I don’t think that the answer is merely a “less extreme” version of Vista—which is the model that most other operating systems and extensible applications seem to be following—but rather that a more effective solution is primarily a social one that is supported by technological tools.

I have a particular solution in mind that I’ll be writing about soon. That said, I’d still love to hear any thoughts that anyone has on this topic.

Last week, Scott Petersen from Adobe gave a talk at Mozilla on a toolchain he’s been creating—soon to be open-sourced—that allows C code to be targeted to the Tamarin virtual machine. Aside from being a really interesting piece of technology, I thought its implications for the web were pretty impressive.

Before reading this post, readers who aren’t familiar with Tamarin may want to read Frank Hecker’s excellent Adobe, Mozilla, and Tamarin post from 2006 for some background on its goals and why it’s relevant to Mozilla and the open-source community in general.

If I followed his presentation right, Petersen’s toolchain works something like this:

1. A special version of the GNU C Compiler—possibly llvm-gcc—compiles C code into instructions for the Low Level Virtual Machine.
2. The LLVM instructions are converted into opcodes for a custom Virtual Machine that runs in ActionScript, a variant of ECMAScript and sibling of JavaScript.
3. The ActionScript is automatically compiled into Tamarin bytecode by Adobe Flash, which may be further compiled into native machine language by Tamarin’s Just-in-Time (JIT) compiler.

The toolchain includes lots of other details, such as a custom POSIX system call API and a C multimedia library that provides access to Flash. And there’s some things that Petersen had to add to Tamarin, such as a native byte array that maps directly to RAM, thereby allowing the VM’s “emulation” of memory to have only a minor overhead over the real thing.

The end result is the ability to run a wide variety of existing C code in Flash at acceptable speeds. Petersen demonstrated a version of Quake running in a Flash app, as well as a C-based Nintendo emulator running Zelda; both were eminently playable, and included sound effects and music.

So, once Petersen’s modifications to Tamarin make their way into the next version of Adobe Flash, we can expect to see older commercial games running in the browser. Even more impressive, though, is the sheer volume of existing code that can be made to run inside the browser: Petersen showed us the C-compiled versions of Lua, Ruby, Perl, and Python all running on the web in secure Flash sandboxes.

What this means for Python

The potential implications this has for Python are particularly interesting to me. The ability to run Python on the web is exciting, to say the least; also interesting is the fact that by sandboxing CPython in a virtual machine, we solve a lot of the security issues that currently face the language when it comes to running untrusted code.

Petersen’s work also resonates with a few goals of another project called PyPy. I’m going to try to explain the idea behind PyPy in a later post; for the time being, the slides from my April 2007 ChiPy presentation on PyPy may serve as a passable introduction.

In a nutshell, the difference in mindset between PyPy and Petersen’s work is that the former is radically innovative in scope and mission, while the latter is pragmatic. PyPy’s goal is essentially to move the canonical implementation of Python from C to Python itself, and then use a pluggable toolchain to translate the Python interpreter to any platform with a configurable set of language and implementation features. In one fell swoop, this modularizes the composition of the Python interpreter in such a way that innovating and maintaining different ports and variants of Python like IronPython, Jython, and Stackless no longer requires either writing an entire copy of the same interpreter in a different language or branching the CPython source code and making pervasive changes to it.

Rather than focusing on innovation, Petersen’s work focuses on code reuse. Instead of moving a canonical interpreter implementation from C to a dynamic language, his strategy is to simply compile the existing C code to run in a virtual machine that’s implemented in a dynamic language. Both approaches aim to obviate the necessity of “ports” of interpreters to different platforms, and as such their purposes intersect at a common subset of functionality. But Petersen’s work can’t be used to facilitate the innovation of the Python language and its implementation, while PyPy offers few or no tools to reuse existing non-Python code. Perhaps it’s possible to combine the best of both worlds by taking PyPy’s generated C interpreter and using Petersen’s toolchain to allow it to be usable on the web and other places that Tamarin runs.

What this means for the Open Web

To be honest, I’m not quite sure where the dividing line is between what of Petersen’s work is Flash-specific and what can be reused to benefit the Open Web. Since ActionScript is a sibling language to JavaScript, it’s possible that the custom VM he created can be run in a browser with relatively few modifications—albeit much more slowly in Firefox at the time being, since SpiderMonkey-Tamarin integration is not yet complete. Once that’s further along, though, I imagine it should be possible to create C “libraries” that can be used in the toolchain to allow sandboxed C code to interact with web pages rather than Flash apps. Should this be feasible, I think it will possibly be the ultimate in a relatively recent string of next-generation Javascript virtual machines that allow existing code to run safely in browsers.

Also, in the context of the web, download size is a significant concern because applications are essentially streamed to clients. While Petersen’s toolchain means that it’s possible to instantly inherit most of CPython’s benefits on the web, it also means that we get all of its flaws along with it—such as the fact that the standard CPython distribution is a few megabytes large. But there’s ways to get around this.

In any case, I’m really excited to see how both Petersen’s work and PyPy proceed. I just hope I haven’t mis-represented either one of them here due to a lack of understanding; I’ll try to correct this blog post as I become aware of my mistakes.

Yesterday I found John J. Lee’s old Python-SpiderMonkey code from 2003, which creates a bridge between the Python language and Mozilla’s SpiderMonkey JavaScript engine—the same engine that powers Firefox. Lee mentioned on his website that it’s not currently maintained, and after downloading it and trying to compile it, I found that SpiderMonkey had changed a bit since the code had been written, so I made some fixes and, after discussing things with him, set up a new Google Code project for it at http://code.google.com/p/python-spidermonkey.

I also revamped the documentation, wrote a basic tutorial, and removed some dependencies for building it.

Interoperating between Python and C

Python-SpiderMonkey still isn’t particularly easy to build, though: while you no longer need to get Pyrex or compile SpiderMonkey yourself, you do need to download the XULRunner SDK and have a C compiler on your system so the C Python extension can be created. Ideally, this shouldn’t have to be done: I’ve written some example code that uses ctypes to dynamically load the SpiderMonkey shared library file at runtime, which is proof that all that should be required by Python-SpiderMonkey are Python and either Firefox or the XULRunner SDK.

On the other hand, Python-SpiderMonkey currently uses Pyrex—a dialect of Python that understands C data types—to do all its communication with SpiderMonkey. This was my first time using Pyrex, and I’m quite impressed: without even reading a tutorial, it was easy for me to understand and change what Lee’s Pyrex code was doing. Having written “from scratch” C Python extensions, SWIG extensions, and ctypes code, I think that my (admittedly brief) encounter with Pyrex makes it my favorite of all the options from a coding standpoint: it was very easy for me to write (and read!) code that interoperated between Python and C. On top of that, tracebacks into Pyrex-generated code were eminently useful, which made debugging very easy—and almost as much fun as writing pure Python code, if it weren’t for the recompilation step (which makes me wonder if there’s an importer that transparently re-compiles Pyrex code on-the-fly).

The only downside of Pyrex, though, is that it compiles to a C Python extension, which makes it difficult for other Python users to use. It’d be really cool to see an optional backend for Pyrex that compiled its extension down to ctypes-wielding Python code instead, but I’m not sure how possible that is. I suspect that the issue of calling C macros from Python code would be an annoying one to get around.

Why did I do this?

To begin with, I should make it clear that I didn’t actually do much—this is still almost exclusively John J. Lee’s code. I just added some documentation, made the build process a bit easier, and got it to work on modern builds of SpiderMonkey.

One reason I did this was just for personal education: I wanted to see what Pyrex was like, and I also wanted to get my hands dirty with the rather well-documented SpiderMonkey C API. In doing that, I actually found a potential bug in SpiderMonkey that I submitted a test case for.

The other main reason I did this may be a bit controversial.

Since becoming more acquainted with web programming, I’ve started to really like JavaScript. Aside from the fact that it’s on almost everyone’s computer and its applications in web-content-space are incredibly easy to deploy, it’s just a fun and powerful language to use, once you get around some of its wrinkles. It’s in this web-content-space where the language is used by an enormous community of developers and new tools and better ways of doing things are invented every day.

The thing is, a lot of Firefox itself is implemented in JavaScript, and that’s not really web-content-space; Mozilla calls it chrome-space, because it has entry to a variety of software components—via a rather obtuse and verbose Microsoft-inspired system called XPCOM—that provide access to the local system. Think file I/O, sqlite storage, the sorts of things that the Python standard library does.

The only problem is that, perhaps because chrome-content JavaScript and the associated XPCOM componentry is only really used by Mozilla, Firefox extension developers, and a handful of smaller projects, the development environment isn’t nearly as mature and as much fun to program in as web-content JavaScript and Python. There’s no Firebug, for instance, and there’s nothing like nose; there are very few development and testing tools in general, and they’re not terribly robust. Perhaps most importantly for newcomers to chrome-content like myself—and in stark contrast to the Python standard library and nearly every major Python library in existence—many XPCOM components don’t even have any documentation. Just accessing the filesystem to open a file can involve a nontrivial amount of XPCOM sorcery, unless you happen to find the right page on MDC or talk to the right person on IRC. Development tools often fall into disrepair, quickly becoming nonfunctional because the trunk changed, the original developer stopped maintaining the code and no one was interested in picking it up.

This isn’t to say that it’s impossible to do things in chrome-content. Almost nothing is impossible for a programmer; it’s rather that things just aren’t fun, especially when compared to well-documented platforms with diverse, vibrant communities like web-content JavaScript and Python.

So, I’ve been spending some of my time trying to connect Python to JavaScript, so that, at least when it comes to writing unit and system tests, chrome-content JS can leverage some of Python’s mature, well-documented, easy-to-use tools. I don’t have any really concrete ideas just yet, but I think that building a bridge between Python and JavaScript is the first step. There’s also another somewhat neglected package called PyXPCOM that I’d like to make a little easier to use and possibly integrate with Python-SpiderMonkey—but that’s a topic for another post.

Aside from the world of chrome-content JavaScript, there are a few other places where I think this kind of bridge could be useful:

• As I mentioned in my overview of Weave, because the client’s data is decrypted client-side, almost all synchronization logic needs to be done there. This logic needs to be rock-solid in all client implementations, or else the user’s data can become corrupted. So one way of creating a Python client may be to simply reuse the sync logic from the Weave extension, which is written in JavaScript 1.7.
• It seems as though web applications need to duplicate logic on both the client and server side: for example, client-side code in a web application may want to check to see if a user’s credit card number is valid, and warn them before they submit the form. The same logic may need to be duplicated on the server-side; one way to do this is by allowing Python-based servers to access some of the JavaScript code that’s being sent to the client, to prevent the duplication of code in multiple languages.

I’ve talked to Ian Bicking and Kumar McMillan about this too, and they had some pretty interesting ideas of things that could be done with Python-SpiderMonkey. I’m interested in hearing any other thoughts people have on this.