Couches in Browsers

Published
by
Atul
on April 16, 2009
in Coding and Mozilla
. 10 Comments

A little while ago, Vladimir Vukićević wrote an excellent blog post outlining the reasons why he’s not a fan of exposing a specific implementation of SQL to Web Content.

I agree with everything he says in his post; I’ve also been a fan of CouchDB for some time. A CouchDB-like API seems like a nice solution to persistent storage on the Web because so many of its semantics are delegated out to the JavaScript language, which makes it potentially easy to standardize, as well as easy to learn for Web developers. Furthermore, CouchDB’s MapReduce paradigm also naturally takes advantage of multiple processor cores—something that is increasingly common in today’s computing devices.

To explore the possibility, I decided to spend some time prototyping a JavaScript implementation of CouchDB, which I’ve dubbed BrowserCouch. It’s intended to work across all browsers, gracefully upgrading its functionality when support for features like Web Workers and DOM Storage are detected.

Right now this is very much a work-in-progress and there isn’t anything particularly shiny to see; just the test suite and the semi-large data set test. In the future, it’d be great to make CouchDB’s Futon client work entirely using BrowserCouch as its backend instead of a CouchDB server, but that’s a ways away.

If you’d like to see some code samples of what the BrowserCouch API currently looks like, check out the annotated source code for the test suite. You can also read the primary source code documentation for more on BrowserCouch’s implementation. And if you’re interested in hacking on the code, the Mercurial repository is right here.

Design Challenge Tutorials

Published
by
Atul
on April 4, 2009
in Coding, Mozilla and Usability
. 3 Comments

Over the last two weeks, I gave two tutorials to our Design Challenge students.

The first was called Engineering Prototypes, and centers on the most challenging part of working on prototypes for me, which is the balance between expediency of implementation and robustness. Prototyping involves prioritizing the former over the latter, but it’s unwise to throw engineering principles out the door: for instance, a prototype that constantly crashes or runs slowly may not be usable enough to dogfood, and one whose implementation is poorly designed can be difficult to iterate and evolve. My tutorial attempts to present some of the factors one should take into account to produce prototypes that are both quick to implement and robust enough to dogfood.

The other session was called Prototyping with jQuery but it included a heavy dose of Firebug as well.

For the second session, I created a prototype of something that I’ve wanted to make for a while: Open Web Challenges.

These are essentially a series of interactive web-based problems that require “hacking the page” using real-world tools to solve. They’re inspired by a number of my favorite pedagogical dilemmas, such as the time someone in LambdaMOO made me program my way out of a paper bag; the inventive exercises from Graham Nelson’s Inform Designer’s Manual 4th Edition; the labs from Bryant and O’Hallaron’s Computer Systems: A Programmer’s Perspective; and the mathematical proofs from Carol Schumacher’s Chapter Zero. Obviously, being a one-day hack, these Open Web Challenges pale in comparison, but the prototype was fun to build and I’d like to continue creating more interactive exercises like this.

All in all, I thought the Design Challenge was an awesome opportunity for students around the world to learn more about open-source design and development, as well as a great way for Mozilla folks to get a chance to talk to students, teach, and obtain a better understanding of what we need to do to make the Web as a platform easier to learn. If you’re interested, I recommend checking out the rest of the tutorials at design-challenge.mozilla.com/spring09.

I’d also like to thank Pascal Finette for putting the Design Challenge together—it’s unquestionably a success and I’m looking forward to participating in more Mozilla Education projects in the future.

Redesigning Planets and Project Dashboards

Published
by
Atul
on March 3, 2009
in Coding and Mozilla
. 12 Comments

The Ubiquity project has been moving pretty quickly and despite the fact that I spend most of my time working on it, I actually have a hard time keeping track of its progress. At Labs we’ve talked about the idea of having “project dashboards” that present the latest developments on our projects, so I thought it might be a good opportunity to play around with new ways of visualizing community activity.

Most of the information I actually need to know about where Ubiquity’s at is actually contained in Planet Ubiquity; the problem for me is that some of the most important information I need, such as recent code commits, is actually contained way down in the sidebar of the page. And while I want different types of information to be spatially separated—so that I can just look at code commits without having to see blog posts, for instance—I also want them to be chronologically grouped so that it’s possible for me to see what’s happened with the project in the last 8 hours.

Here’s a screenshot of what I’ve got so far:

This visualization presents a grid whose columns represent different types of information for the project, and whose rows represent chronological “bins”. Each row is further colored to provide visual indication of how old the information is. The actual aesthetics here aren’t fully-formed yet—they’re actually just a modification of the stylesheet for my about:mozilla redesign—but the general grid concept is something that’s been simmering in my head for a while. As an active project contributor, it works well for me, though I’m not sure how suitable it is for the casual passerby who’s curious about where the project’s going.

Under the hood, everything’s implemented using client-side JavaScript and Google’s AJAX Feed API. Each column can include information from more than one feed; for instance, the “Code” column displays a colored square next to each HG commit indicating the result of the automated Buildbot tests for that revision.

Although this implementation is merely a prototype, I’ve tried making it decoupled enough to be reused as a general dashboard for any project; I’ve already spoken about this to Dan Mills, so we should have a Weave project dashboard soon.

To see the prototype in action, check out the Planet Ubiquity Redesign. I’ve tried it out in Firefox and Safari and it seems to work fine, but I wouldn’t be surprised if MSIE barfs on it at the moment. And because everything’s done on the client-side, JavaScript needs to be enabled.

I’ve also documented the code for anyone who’s interested.

An Experiment in Redesigning about:mozilla

Published
by
Atul
on March 1, 2009
in Mozilla
. 13 Comments

With Deb Richardson’s recent posts on the evolution of the about:mozilla newsletter, I decided to try my hand at a new layout for the existing issues, in the hopes that experimenting with new designs could help shed some light on the situation.

For reference, this is how the February 24th issue looks:

One of the notable things about this layout is that there isn’t actually much useful information in the first screen: there’s a table of contents that has headlines, but the user needs to click on a headline or scroll down to learn more. The length of each line is also quite long: at over 130 characters, it can be difficult for the reader’s eye to move from the end of one line to the beginning of another.

Here’s my redesign of the same issue:

This layout takes advantage of the fact that all the articles in about:mozilla are pretty short—short enough to fit on most PC/laptop screens in a narrow column without forcing the user to scroll vertically.

Since most computer screens these days are much wider than they are tall, I thought it’d be useful to have the articles placed horizontally next to one another rather than vertically: that way the horizontal space is used efficiently, and each article remains easy to read because it’s in a column about 40 characters wide. It also places the headlines in the same visual line, so that it’s easy for readers to scan through the issue and filter out articles that they’re not interested in. The top and bottom margins of the page are black so as to focus the user’s eyes on the content (the same reason a theater’s lights are turned off during a show).

This design was inspired in part by the Swiss Grid System. In an attempt to make it somewhat resolution-independent, the CSS doesn’t use pixels as a measurement unit for anything other than borders; everything else is specified in points and ems.

I’ve also hooked up this layout to pull its content from the Atom feed for the newsletter using Google’s AJAX Feed API; if you’re interested in reading the latest issues of about:mozilla using this alternative layout, you’re welcome to use this page. Although there isn’t currently an interface for viewing back-issues, you can change the number at the end of the URL to read them if you need to.

Automatic Bug Reporting for Firefox Extensions

Published
by
Atul
on February 27, 2009
in Coding, Mozilla, Python and Usability
. 4 Comments

We want to make Ubiquity awesome at reporting errors. In our original release, a transparent message with JavaScript exception information was displayed, which wasn’t very useful to the average user, and was downright annoying when dozens of exceptions were logged in the same instant.

At present, running a command that raises an error just results in that message being logged to the JS Error console, which very few people know how to access—so most people are left scratching their heads and wondering why their command is taking so long to run.

For the next release of Ubiquity, we’re going to be trying something more user-friendly: if a command encounters an error, a transparent message will be displayed telling the user that it didn’t work. The message will also recommend using the “report-bug” command to send information about the bug to the Ubiquity team. If the user decides to run this command, a page is opened that looks like this:

Aside from inviting the user to describe their problem, a lot of information is included about their system: what OS they’re using, what extensions and plugins they have installed, what recent exceptions were thrown, and so forth. We’re hoping this will lower the barrier to entry both for receiving and providing technical support, since most of the information needed to describe and investigate a problem is contained in a single link.

We don’t yet have an interface for browsing existing bugs, but we do have a display for viewing them. It looks pretty similar to the page for submitting a bug:

One interesting aspect of our bug reporting system is that we’re not using numbers to identify bug reports: they get big fast as many reports are submitted, and big numbers are hard to remember. Instead, we’re mashing two random words from the dictionary together. For instance, the first bug I reported using this system was called anaphorically-spinach.

Under The Hood

The bug reporting system is pretty lightweight: mostly it’s just static HTML/JavaScript code that talks to a web service that’s implemented in under 100 lines of Python. The actual bug report is just a JSON object, and is deposited into a CouchDB server.

The big advantage of using CouchDB here is that we’ll be able to easily create really rich queries using plain old JavaScript. For instance, here’s a query that shows all reported error messages that contain the text “Invalid chrome URI”. It won’t be hard to create complex queries that, for instance, give us all the bug reports in which the user had a certain extension installed and had a command crash at a particular line in a particular file.

A Public Asset

Right now all reports are submitted to the public domain, and as such the report database is a public asset; users are informed of this before they submit the bug, and encouraged to look at the additional data that’s being sent with their report to ensure that there’s nothing sensitive in there. In the future, it’d be nice to allow the user to click on any parts of the data that are personally identifying, so that they can submit a version of the report that masks out the sensitive information.

Reuse

The bug report system has been designed to be decoupled from Ubiquity itself. For instance, the report viewing application is designed as a reusable JavaScript component, so it should ultimately be easy to embed into any web page. In other words, it should be easy to use as a bug reporting mechanism for any Firefox extension—and possibly for any web application in general. If you’re interested in adding the component to your own project, please let us know; the code is still a work-in-progress, and any contributions or comments are appreciated.

Ubiquity 0.1.6 and Release Scheduling

Published
by
Atul
on February 17, 2009
in Mozilla
. 16 Comments

As we’ve mentioned before, Ubiquity 0.2 has fairly broad, visionary goals that won’t be fully satisfied for some time. So we’re going to be pushing its changes to the 0.1 line at more regular intervals as we continue to develop it.

By “pushing its changes” we mean that we’ll effectively be disguising our work-in-progress 0.2 as a 0.1.x release. For instance, Ubiquity 0.1.5, which we released about a month ago, is essentially the same thing as 0.2pre7; similarly, Ubiquity 0.1.6 will basically be the same thing as 0.2pre13, only with our more experimental features—such as locked-down feeds and python feeds—disabled by default and unadvertised on Ubiquity’s front page.

There’s a couple reasons for this:

  • Ensuring that the general public is using the same codebase as testers and developers means that our code gets to be used by “the real world” more quickly, and means we’ll know about bugs closer to the time that they were introduced into the software.
  • Releasing new public updates at a regular interval is a direct way of letting our users know that Ubiquity is still alive and kicking, and getting better all the time.
  • We can get feedback from the broad public more quickly, so that if we’re really going the wrong direction with something, we can know about it before we’ve invested too much time and effort into it.

So we’re aiming to release a new public update to Ubiquity at least once every two weeks.

Things you can do to help

Ubiquity is a community effort, and we welcome contributions!

  • Testing. Please try out the release candidate and bang on it! Try it out on every operating system that’s convenient for you to try it out on, and on every version of Firefox too (3.0, 3.1 beta, or 3.2 alpha). Any comments to this post about where it works or doesn’t will be very appreciated.
  • Release Notes Documentation. The 0.1.6 release notes can always use more help; please feel free to edit that page and fix a TODO if you enjoy technical writing, and leave a comment here or on IRC if you have any questions. Keep in mind that the wiki tracks all changes, so even if you mess up, we can always revert the changes. Experimentation is good!
  • Ubiquity Tutorial Documentation/Testing. Feel free to try out the user tutorial or the author tutorial with the release candidate and make sure everything still works. Regardless of the outcome, post back here to let us know what you found!

Thanks for reading this. If you have any questions, please don’t hesitate to reply to this post or poke us on #ubiquity on irc.mozilla.org.

PyXPCOM vs. jsbridge

Published
by
Atul
on February 11, 2009
in Coding, Mozilla and Python
. 3 Comments

Yesterday Tempura left a good question as a comment on my blog post concerning Ubiquity’s experimental support for Python:

What about http://pyxpcomext.mozdev.org ? They bring Python in an xpi for all major plattforms, without the need of an local installed interpreter.

Using PyXPCOM was actually a potential option we had considered, but we ended up going with jsbridge for a number of reasons:

  • PyXPCOM uses XPCOM as its means of communication between Python and the Mozilla platform. Unfortunately, XPCOM is super heavyweight and it’s a particularly difficult hurdle to get through when all you really want is for two very dynamic languages to communicate with each other. For instance, Ubiquity itself implements very few XPCOM interfaces, and most of its functionality is exposed through JS Modules; jsbridge allows us to access these JS Modules directly, while accessing them from PyXPCOM would require us to create verbose XPCOM wrappers around them—a particularly egregious amount of work when you think about how similar JavaScript and Python are.
  • One advantage of using the local system’s Python installation is that we inherit whatever packages come with it, which means that on OS X 10.5, for instance, we automatically get access to Python’s bindings to Objective C and Cocoa, whereas with PyXPCOM we really have to manage the user’s distribution manually. It could similarly be said that PyXPCOM takes an embedding approach, while jsbridge takes an extending approach; a very good analysis between these two approaches has been written by Glyph Lefkowitz here.
  • In regards to Windows, putting a distribution of Python on the end-users system that any other application can leverage is a good thing—much better than requiring the user to download a big package and then only allowing Firefox to use it. This gets into the idea of Python as Platform that I’ve written about before.
  • In regards to OS X and Linux, using the pre-existing installation of Python is obviously nice because it doesn’t require the end-user to download anything (aside from mozrunner and jsbridge, which are pretty small).
  • jsbridge has no binary components, which makes it a lot easier to maintain and experiment with than PyXPCOM, which is difficult to compile and extend.

PyXPCOM is definitely useful for a lot of things, though—for instance, manipulating the DOM of a page—so I think there’s still some things we can leverage there. And jsbridge certainly presents its own hurdles and limitations, too, but hopefully this post sheds some light on the thought processes that went into deciding which mechanism to use.

Ubiquity’s Python Feed Plugin

Published
by
Atul
on February 10, 2009
in Coding, Mozilla and Python
. 20 Comments

A few weeks ago I wrote about Ubiquity Feed Plugins, which are basically just a way of separating the user interface of subscribing to a new feature from the implementation of the feature itself.

As I’ve written about before, one of the things I’ve missed about the Mozilla development environment is its support for the Python programming language. Aside from being humane and having a great community, it has functionality that could complement the Mozilla platform quite nicely. So we’ve whipped up a quick proof-of-concept Python Feed Plugin for Ubiquity to explore this possibility.

Here’s a really simple command feed written in Python:

def cmd_sample_python_command(ubiquity):
    ubiquity.displayMessage('Hello from Python %s.' % sys.version)

cmd_sample_python_command.preview = 'a sample python command!'

For anyone who knows Python and has read the Ubiquity Author Tutorial, this should be fairly self-explanatory. The ubiquity argument is just passed in to the command execution function to make unit testing as easy as possible.

As with any Ubiquity feed, the file containing the above code needs to be referenced in the <HEAD> element of a web page:

<LINK REL=”python-commands” HREF=”sample_command.py”>

In this case the python-commands value of the REL attribute is what tells Ubiquity to process this feed using the Python feed plugin.

Visiting the web page containing this tag results in the same subscripton panel that shows up with any page that has a Ubiquity command feed. Because Python code executes at the same privilege level as default Ubiquity feeds, the Python feed plugin really ought to display the same warning page that default Ubiquity feeds show when the user clicks “Subscribe”; but because this is just a proof of concept, we skipped that implementation and currently only allow Python feeds to be used when they already exist on the user’s local filesystem—in other words, when the feed is at a file: URL.

The first time an end-user subscribes to a Python command feed, they’re presented with this non-intrusive message:

What happens behind-the-scenes is a bit more complex: Ubiquity uses virtualenv to bootstrap a sandboxed Python virtual environment inside the user’s profile directory. It then uses easy_install to install jsbridge, and then sets up a “feed server” process on the user’s system, which Ubiquity communicates with over jsbridge to execute Python code:

Pretty Frankenstein, but it works on OS X and Linux. Because Windows doesn’t come with Python built-in like the other two operating systems do, we’ll have to bootstrap it first, which we haven’t gotten around to yet.

So, if you’ve got OS X or Linux and like Python, feel free to play around with this; a sample Python command feed is included with the latest Ubiquity beta. If this is something that seems interesting to you, we’d love to know—you’re welcome to leave a comment on this post, drop us a line on the ubiquity-firefox Google Group, or say hello on #ubiquity on irc.mozilla.org.

Beautifully Documented Code

Published
by
Atul
on January 7, 2009
in Coding and Mozilla
. 26 Comments

When searching for a documentation system to use for Ubiquity, we looked at a number of tools. None of them particularly satisfied me; all the ones I saw required a build step to convert raw source code into formatted documentation, and I wasn’t very pleased with the typography of the generated content—though obviously the aesthetics were customizable through CSS, none of the default stylesheets left me dying to read the documentation I created.

I’ve always had these kinds of hangups when it comes to design; a lack of it repels me from some software that might otherwise be interesting or useful, while good aesthetics is part of the reason I’m drawn to a handful of projects like TeX that are truly elegant out-of-the-box. With this in mind, I decided to play around with creating my own simple code documentation system.

The result so far is here: a single-page JavaScript application that can be served from static files, which takes in raw JavaScript files and dynamically renders the documentation alongside the code in the reader’s browser.

The raw source code for the file being documented above just has chunks of comments that are marked-up in WikiCreole; when the parser runs into such a chunk, it renders it alongside the code it annotates using Ivan Fomichev and Chris Purcell’s JavaScript Creole 1.0 Parser.

The idea of using the documentation to annotate the code—or the code to annotate the documentation, depending on how you look at it—was inspired in part by some of the typography presented in Ellen Lupton’s excellent Thinking with Type, which I recommend to anyone interested in the field.

It’s nice not having a separate documentation build step: aside from making the process of writing and editing documentation quicker, it also lowers the barrier for entry to contributors since they don’t need to setup a toolchain. It also means that we get versioned documentation in every commit for free.

Right now there isn’t much else to the system; the only other feature I added is the auto-generation of quasimodal popup menus that link the names of XPCOM interfaces to their entries on XUL Planet and MDC. It’d be nice to have more features like this; other niceties would be an automatically-generated table of contents, JavaScript Doctests, search functionality, cross references to other code, and even the ability to fix code formatting errors in-page. For now, though, I need to focus on actually using this tool to document more code.

Speaking of which, Ubiquity code could definitely use more documentation—especially the APIs for command authors, which many developers would love to see. If you’re interested in helping out, please feel free to join us on ubiquity-firefox, #ubiquity on irc.mozilla.org, or email me directly at my first name at mozilla.com.

EDIT: Since posting this, I’ve spun-off this code into a small new project called Code Illuminated. Contributions are welcome; I’d specifically like to thank Sander Dijkhuis for his excellent work so far. If you’re interested in helping out, feel free to join the mailing list.

Ubiquity 0.2 Preview Release: Feed Plugins!

Published
by
Atul
on December 31, 2008
in Coding and Mozilla
. 1 Comment

I’ve just released the first preview release of Ubiquity 0.2, which implements some of the functionality outlined in the Ubiquity 0.2 Architecture Proposal.

In particular, we now have support for something called Feed Plugins, which makes it possible for Ubiquity to draw from a much wider range of functionality: imagine, for instance, if end-users saw Greasemonkey and CoScripter scripts no differently from standard Ubiquity command feeds, and used the exact same interface to subscribe to and use functionality that’s been implemented with any number of web technologies.

Until now, including a command script on a page has involved putting the following in the <HEAD> element of a web page:

<LINK REL=”commands” HREF=”myfeed.js”>

A feed plugin is triggered by the text in the REL attribute above; in this case, the value commands tells Ubiquity to use what we’re now calling the Default Feed Plugin (DFP), which is the one that Ubiquity users and developers are familiar with. But with version 0.2, we’ll be able to create all sorts of new feed types, which will allow us to not only hook into pre-existing scripting technologies like Greasemonkey, but also experiment with a variety of security models.

As I’ve written about before, one of the big issues confronting users of Ubiquity and other generative tools like GreaseMonkey is that of how to trust functionality. I don’t know exactly what the answer is, but I think that one great way to find out is by empowering people with the tools they need to discover it together.

As a proof of concept, I spent a few hours today creating a Locked-Down Feed Plugin (LDFP), which has syntax reminiscent of the DFP but is intended to be completely secure. While DFP feeds have the ability to do whatever they want to your computer, LDFP feeds actually have less freedom than web pages: they can’t even make outbound network connections. While they can effectively modify your current HTML selection, any HTML they inject into a page is decontaminated by the excellent HTML sanitizer created by the Caja project to ensure that cross-site scripting attacks aren’t possible. I’ve written a sample LDFP feed that showcases some of its security and functionality; if you’re familiar with JavaScript, take a look at its source code.

Since LDFP feeds are capable of so little, clicking on the Subscribe button when visiting a command feed doesn’t result in the terrifying warning of doom that’s shown when you subscribe to a DFP feed from an untrusted domain. In fact, clicking the button just results in an unobtrusive notification message confirming your subscription. This is clearly a win for the end-user, since it makes subscribing to functionality as painless as visiting a webpage.

Of course, with this security comes less freedom, so it’s not possible to be nearly as creative with LDFP as the Ubiquity community has been with all the DFP feeds it’s created over the past few months. My guess is that the real solution to the problem will be at least twofold: strong technical security—perhaps an object-capability model as some have suggested—to help individuals with technical expertise make decisions about what functionality to trust, combined with a social web-of-trust model that helps both technical and non-technical users delegate their decision-making to individuals that they have a high degree of faith in. It’s my hope that the Feed Plugin mechanism will help us explore this and many other ideas.

The Feed Plugin API is still in flux, however, so we haven’t yet written any documentation for it. The source code for the Locked-Down Feed Plugin itself is fairly short, though familiarity with JavaScript security and XPConnect wrappers can help one understand it, but feel free to take a look at it if you’re really eager. Look for a solidified, documented API and a final release of Ubiquity 0.2 in the next few weeks.

In he meantime, if you’d like to try it out, feel free to download the latest Ubiquity 0.2 preview release; if you’d like something more stable, we just released Ubiquity 0.1.3 yesterday.

As always, you’re welcome to drop by #ubiquity on irc.mozilla.org or visit the ubiquity-firefox Google group if you’d like to contribute or ask any questions.