Lately some of us at Mozilla Labs have been experimenting with graphical keyboard user interfaces in Firefox. Our current work-in-progress is something that we’re calling Ubiquity for the time being, though the name is by no means set in stone.
This project is heavily informed by Enso, a software product developed by me and my colleagues at Humanized from 2005-07. Aside from the benefits outlined in Alex Faaborg’s blog post entitled The Graphical Keyboard User Interface, this experiment is intended to solve few other problems, one of which I’ll address in this post.
Web applications, much the same as desktop applications, are a bit like isolated cities: it’s difficult for an end-user to arbitrarily share data and functionality between them. This is alleviated to some extent by creations like Firefox Add-ons that add toolbars or sidebars to Firefox’s UI, Bookmarklets, and Greasemonkey, but while all of these solutions are powerful, each comes with its own set of problems. The buttons and bars of many Firefox add-ons don’t scale well because of the valuable screen real-estate they consume; Bookmarklets are restricted in scope because they only have the access privileges of the website they’re running on; and Greasemonkey doesn’t prescribe any kind of interaction model, which makes it difficult to reuse the functionality of a script in a context other than the ones it was expressly designed for.
Our new project attempts to alleviate all of these problems by allowing end-users to apply textual commands, or verbs, to whatever they’re looking at. For instance, let’s assume that I’ve found a typo on a friend’s blog, and I want to let him know about it. I can first select the typo on the page by dragging my mouse over it:
Then, I can activate what we’re calling the “command entry mode”. At the moment, this is done via a hotkey that displays a popup window, though this particular interface is only temporary (more on that later):
Now I type “highlight”, because that’s what I want to do to the typo, to emphasize the location of the typo to my friend:
I then press enter, which highlights the text, like so:
Now I’d like to email part of the sentence containing the highlighted typo to my friend, so that he has some context for where the typo is located. I do this by highlighting part of the sentence, and issuing the “email” command. This causes Firefox to switch to my Gmail tab and compose an email for me with some pre-filled content:
I can now modify the email as necessary to tell my friend about his typo, and then send it.
Our current code is capable of performing this workflow, though Gmail is the only supported mail application at the moment. The hotkey interface is also something of a placeholder, because it was the most expedient to implement while making the project dogfoodable. Ultimately, it would be ideal for verbs to be exposed to end-users through interfaces that already exist in Firefox, such as the AwesomeBar or contextual menus.
Exactly what the user needs to type in order to invoke a command is also under contention, and Jono has been doing some really interesting work on this front. There’s a number of additional aspects of the experiment that I’d like to write about, so keep reading Toolness for more updates.
We’re currently sprinting to have something more presentable by the Firefox Summit—hopefully a 0.1 release, if all goes well. In the meantime, if you’re interested, feel free to join the Google Group, and even check out the source code.