Lately some of us at Mozilla Labs have been experimenting with graphical keyboard user interfaces in Firefox. Our current work-in-progress is something that we’re calling Ubiquity for the time being, though the name is by no means set in stone.
This project is heavily informed by Enso, a software product developed by me and my colleagues at Humanized from 2005-07. Aside from the benefits outlined in Alex Faaborg’s blog post entitled The Graphical Keyboard User Interface, this experiment is intended to solve few other problems, one of which I’ll address in this post.
Web applications, much the same as desktop applications, are a bit like isolated cities: it’s difficult for an end-user to arbitrarily share data and functionality between them. This is alleviated to some extent by creations like Firefox Add-ons that add toolbars or sidebars to Firefox’s UI, Bookmarklets, and Greasemonkey, but while all of these solutions are powerful, each comes with its own set of problems. The buttons and bars of many Firefox add-ons don’t scale well because of the valuable screen real-estate they consume; Bookmarklets are restricted in scope because they only have the access privileges of the website they’re running on; and Greasemonkey doesn’t prescribe any kind of interaction model, which makes it difficult to reuse the functionality of a script in a context other than the ones it was expressly designed for.
Our new project attempts to alleviate all of these problems by allowing end-users to apply textual commands, or verbs, to whatever they’re looking at. For instance, let’s assume that I’ve found a typo on a friend’s blog, and I want to let him know about it. I can first select the typo on the page by dragging my mouse over it:
Then, I can activate what we’re calling the “command entry mode”. At the moment, this is done via a hotkey that displays a popup window, though this particular interface is only temporary (more on that later):
Now I type “highlight”, because that’s what I want to do to the typo, to emphasize the location of the typo to my friend:
I then press enter, which highlights the text, like so:
Now I’d like to email part of the sentence containing the highlighted typo to my friend, so that he has some context for where the typo is located. I do this by highlighting part of the sentence, and issuing the “email” command. This causes Firefox to switch to my Gmail tab and compose an email for me with some pre-filled content:
I can now modify the email as necessary to tell my friend about his typo, and then send it.
Our current code is capable of performing this workflow, though Gmail is the only supported mail application at the moment. The hotkey interface is also something of a placeholder, because it was the most expedient to implement while making the project dogfoodable. Ultimately, it would be ideal for verbs to be exposed to end-users through interfaces that already exist in Firefox, such as the AwesomeBar or contextual menus.
Exactly what the user needs to type in order to invoke a command is also under contention, and Jono has been doing some really interesting work on this front. There’s a number of additional aspects of the experiment that I’d like to write about, so keep reading Toolness for more updates.
We’re currently sprinting to have something more presentable by the Firefox Summit—hopefully a 0.1 release, if all goes well. In the meantime, if you’re interested, feel free to join the Google Group, and even check out the source code.
My name is Atul Varma, and I'm the co-founder of a small Chicago company called 
Hi,
are you aware the Inky project by Rob Miller’s team at MIT ?
In case you didn’t hear about it :
“Inky (short for Internet Keywords) is a Firefox extension that provides a sloppy command line for the web.”
http://groups.csail.mit.edu/uid/inky/index.html
The UIST 2008 paper (probably not the final version) :
http://groups.csail.mit.edu/uid/projects/inky/uist08-inky.pdf
an older web page :
http://groups.csail.mit.edu/uid/projects/inky/
Regards
Aurélien
There is PodiPodi too, very similar:
http://www.podipodi.com/
Sorry for bad english,
from Brazil, Marco Gomes
CTO of the boo-box team
http://boo-box.com
Gnome Do is a contextual launcher utility; this seems like a browser specific complement. I’d be interested to see the various forms of context that can be captured or acted on; the example here focuses around actions on selected text, and I’m wondering if it goes deeper.
I think a screencast would be easier to understand than reading along.
Great idea. I would love to use it in a future version. It will be very powerful when there is some kind of plugin-system for commands so everybody can extend it.
I love the idea, but on FF 2.0.0.6 with OS X Server 10.4.11 from command space I get a uniquity pop-up menu and no command line (though I get a black box at the top of the screen).
Isn’t this more or less exactly what the vimperator plugin does? Except based on a pre-existing well known set of commands (vi commands), and with extreme configurability.
This is a really interesting pursuit and leads deep into the waters of ontology. I have been working on this idea of verbs and nouns through my project truthstructures. Its a much higher level of what computer “use” might be when it becomes a collabortative and simultaneous act. Please do read more about it here, I think we have a lot of common pursuits:
http://docs.google.com/Doc?docid=acv93nf7c8gv_12fp8hfw&hl=en
i can exactly do the same with custom keywords and bookmarklets… but it’s a great idea anyway! the only thing i don’t like it’s the name of the project
with the other mozilla labs projects such as snowl, prism, weve, this ubiquity seems something… ugly for me 
Very interesting idea. But I was wondering whether internationalization or localization issues have been considered in the interface. Will the commands be in Japanese, Chinese, Dutch or other languages?
Have you looked at Conkeror? Like Emacs, it uses M-x for a sort of “command line”. There is even a GMail major-mode, although I don’t think it does anything like this (yet).
It’s probably not a replacement for where you want to go with Ubiquity, but I think it is maturing into a useful complement for the sort of users who prefer Emacs (or other Unix-geeky things).
Seems like everything old really is new again
Looks like there are a collection of comparable command line efforts. If you look at Google Project 0 and IBM’s sMash, you’ll see that they have a palette and a drag-and-drop/wire-together interface. A similar interface here might be useful. If you allow command line would it be useful to allow a certain level of nesting? e.g. email {map Pepsi Center Denver, CO} foo@bar.com
Seems like Quicksilver for Firefox?
Would be a great idea since I love Quicksilver and using awesome keyboard shortcuts to do everything in FF would be good.
I hope that these new ways of interacting with our computers don’t get corrupt by the unix mentality of cryptic verb words, that are only one to five characters long. There needs to be a balance of speed verses usability verses new uses verses old people who take the time to spell out full words.
Hi,
Awsome initiave. I personally would LOVE to see one feature added:
A pasted text from a site (URL or not) gets force executed as “Go to address in FF”.
I’d love to see better support for Microformats. For example when I type ‘add-to-calendar’ on a page with calendar microformats, it would be brilliant if they appeared in the suggestion list.
You could do similar things for places and maps, and for people and contacts.
I think I saw you tweet about needing a text to speech API?
Check out ours. We have a free version you can use too.
Feel free to contact us!
Best,
Heath