Building Bridges Between GUIs and Code With Markup APIs

Recently the Twitter Bootstrap documentation gave a name to something that I’ve been excited about for a pretty long time: Markup API.

Markup APIs give superpowers to HTML. Through the use of class attributes, data attributes, X-Tags, or other conventions they effectively extend the behavior of HTML, turning it into a kind of magic ink. Favorite examples of mine include Twitter Bootstrap, Wowhead Tooltips, and my own Instapoppin.

The advantages of a markup API over a JavaScript API are numerous:

  • They mean that an author only needs to know HTML, whose syntax is very easy to learn, rather than JavaScript, whose syntax is comparatively difficult to learn.
  • Because the API is in HTML rather than JavaScript, it’s declarative rather than imperative. This makes it much easier for development tools to intuit what a user is trying to do—by virtue of a user specifying what they want rather than how to do it. And when a development tool has a clearer idea of what the user wants, it can offer more useful context-sensitive help or error messaging.
  • Because of HTML’s simple and declarative structure, it’s easy for tools to modify hand-written HTML, especially with a library like Slowparse, which help ensure that whitespace and other formatting is preserved. Doing the same with JavaScript, while possible with libraries like esprima, can be difficult because the language is so complex and dynamic.

These advantages make it possible to create GUI affordances atop hand-coded HTML that make it much easier to write. As an example of this, I hacked up prototype slideshow demo and physics demo in July of last year. Dragging an element with the class thimble-movable in the preview pane changes (or adds) CSS absolute positioning properties in the source code pane in real-time, and holding down the shift key modifies width and height. This allows users to size and position elements in a way that even a professional developer would find far more preferable to the usual “guess a number and see how it looks” method. Yet this mechanism still places primacy on the original source code; the GUI is simply a humane interface to change it.

This is the reverse of most authoring tools with an “export to HTML” feature, whereby an opaque internal data model is compiled into a blob of HTML, CSS, and JavaScript that can’t be re-imported into the authoring tool. Pedagogically, this is unfortunate because it means that there’s a high cost to ever leaving the authoring tool—effectively making the authoring tool its own kind of “walled garden”. Such applications could greatly facilitate the learning of HTML and CSS by defining a markup API for their content and allowing end-users to effortlessly switch between hand-coding HTML/CSS and using a graphical user interface that does it for them.

Learning and Grammatical Forgiveness

HTML is a very interesting machine language because, like human languages, most things that interpret it are very forgiving.

For instance, did you know that the following HTML is technically invalid?

  <source src="movie.mp4"></source>

It’s invalid because <source> is a so-called void element: since it can’t have any content inside it, you simply don’t need a closing tag for it. The <img> tag works the same way. The technically correct way to write the above HTML snippet is as follows:

  <source src="movie.mp4">

However, in practice, all Web browsers will interpret both of these snippets the exact same way. When a browser sees the closing </source> tag on the first snippet, it realizes the “mistake” the author has made, and simply pretends it isn’t there.

What’s interesting to me is the way this mirrors human languages, and what it means for teaching. For instance, the following sentence is grammatically incorrect:

The dog loves it's owner.

However, no one who knows English will actually be confused by the meaning of the statement.

When I was trained as an adult literacy tutor several years ago, one of the most important principles we were taught was that fostering a love for writing was vastly more important than grammatical correctness. The “red pen” commonly used by school teachers for correcting grammatical errors was seen as anathema to this: when we found a grammatical error in a novice writer’s work, we were encouraged to ignore it unless it actually made the piece confusing or ambiguous for readers in a way that the author didn’t intend. Otherwise, the novice writer would become quickly distracted and discouraged by all their “mistakes” and view writing as a minefield rather than a way to communicate their thoughts and ideas.

We’re running into similar issues in the design of the Webpage Maker. On one hand, the fact that Web browsers are so forgiving when interpreting HTML enables us to follow a similar philosophy as that of progressive adult literacy tutors.

But sometimes, the forgiving nature of Web browsers backfires: they actually render a document that is vastly different from the author’s intent, which is just as frustrating as a pedantic nitpicker. We’ve created a library called Slowparse—soon to be renamed—which attempts to assist with this, providing the logic needed for a user interface to gently inform users of potential ways their HTML and CSS code might be misinterpreted by machines. A full specification of errors and warnings is also available, as is an interactive demo that uses the library to provide real-time feedback to users.

It’s been interesting to see how different Slowparse is from a HTML/CSS validator, whose goal is not one of learning, but of ensuring conformance to a specification. From a learning perspective, a validator is like the pedantic teacher who loves their red pen: some of its feedback is quite useful, but the remainder is likely to confuse and intimidate a newcomer.

Partly as a result of its learning goals, Slowparse actually “warns” the user of things that are technically valid HTML/CSS, but which likely don’t reflect the intent of the author. One current example of this is in regards to the use of unquoted attributes in HTML5, though that particular example is still subject to change.

At this point, I think the challenge will be to work with our learning team and user test our interface to the point that we achieve a good balance between being a pedantic nitpicker and providing useful feedback that helps users as quickly as possible. In my opinion, if we do things right, we’ll help people develop a love for HTML and CSS—even if what they write may technically be “grammatically incorrect.”

Storything Interactive Prototype

Last week I merged the Webmaking Tutorial Prototype with the Webmaking 101 for Journalists prototype we made during our three day sprint in NYC in February.

The result, which is code-named Storything, is currently hosted at Give it a try!

The design for this prototype is based on Jess Klein’s instructional overlay mockups. A separate two-pane editor for example snippets is included in the movie frame; my hope here is that by setting the tutorial movies in an actual editing environment, users will obtain a better understanding of how to use our tool. This is further aided by the ability for the tutorial movies to highlight parts of the user interface outside the movie frame.

At present, the prototype awards badges for a few simple things, like creating your first paragraph and header elements. These badges aren’t necessary to advance the tutorial, however; this is partly because we’d like the tutorial to be approached in a non-linear way, and also because the assessment algorithms for the badges likely can’t capture every possible edge case in user input.

The prototype’s source code can be forked at toolness/storything on Github.

Webmaker Tutorial Prototyping

Recently I’ve been playing around with creating interactive tutorials that teach people how to create things on the Web.

Check out this prototype. At the end of a movie-like tutorial, you’ll be given a challenge to write your first bit of HTML. At any time, you can use the scrubber at the bottom-right to review any part of the tutorial; anything you’ve typed so far in the challenge is undone while you’re scrubbing, and is automatically re-applied once you’re back at the challenge.

This prototype is an evolution of an idea that Jess Klein wrote about regarding instructional overlays for our webmaking tools.

How It Works

The tutorial uses Popcorn to construct a “movie” that automates the user interface of a two-paned HTML editor with an instructional overlay. I made an ad hoc mini-framework called tutorial.js to make it easy to script these sorts of experiences; for example, here’s what the beginning of the prototype’s tutorial looks like:

  .dialogue("Hi! I am Mr. Love Bomb and will teach you how to be a webmaker now.", 0)
  .typechars("you're really cool.")
  .dialogue("Check it out, this is HTML source code—the language of the Web.", 0)

The dialogue() method just makes the tutorial avatar—in this case, Mr. Lovebomb—say something. typechars() types some characters into the HTML editor, and spotlight() draws the user’s attention to part of the page. Each action is queued to occur serially, in a style inspired by the chaining API of soda.

The full tutorial script is at tutorial-script.js, and the full tutorial source is available on Github.

Playfulness and Learning

Michelle Levesque recently wrote a post about the importance of play in learning.

We need to change people’s mindsets to make them comfortable fooling around, making things, breaking things, and playing on the web.

I totally agree. This is one of the design goals of the Hackasaurus tools and events, actually—it’s a combination of stylistic touches and emotional design to help people feel that what they’re doing is fun, along with humane functionality that makes experimentation easier, such as infinite undoability.

This is something I feel that Apple has managed to do with their products, too: I see non-technical people like my father who are typically terrified of using their personal computer take joy in installing apps on their iPad and playing with them in a way that they never would have dared to do on their PC. Partly it’s due to concrete features, like the fact that it’s impossible for an app to impair the behavior of another app—but it’s also partly due to stylistic touches, like all the device’s glee-inspiring animations.

It’s ironic that tech-savvy folks like me berate the iPad for its lack of generativity and hackability, yet it manages to orient its users towards a sense of playfulness and empowerment in a way that other tools rarely have.

Hacking The Web With Interactive Stories

I recently made The Parable of The Hackasaurus, which is a game-like attempt to make web-hacking easy to learn through a series of simple puzzles in the context of a story.

The parable is really more of a proof-of-concept that combines a bunch of different ideas than an actual attempt at interactive narrative, though. The puzzles don’t actually have anything to do with the story, for instance. But I wanted an excuse to do something fun with the vibrant art that Jessica Klein has made for the project, while also exploring possibilities for the Hack This Game sprint and giving self-directed learners a path to understanding how the Hackasaurus tools work.

If you know HTML and CSS, you’re welcome to remix this and make your own “web hacking puzzle” where the goggles are the controller. Just fork the repository on GitHub and modify index.html and bugs.js as you see fit. Almost all the puzzles, achievements, and hints are data-driven and explained in documentation, so you don’t actually need to know much JavaScript.

For more information on my motivations in creating the parable, check out the experiment’s README. Jess is also thinking about making an interactive comic that teaches web-hacking, which I’m looking forward to.

The Decline and Fall of The URL

The URL is a very powerful concept; it represents a universal way to access any resource anywhere in the world. Here’s one of them, as it appears in Firefox 5’s address bar:

Address bar containing

The first few letters before the colon are called the protocol, which tells the computer how to interpret the rest of the URL. The http protocol is the most common and specifies a resource on the World Wide Web, while the tel protocol specifies a telephone number, and https specifies a resource on the Web transferred over a secure channel that can’t be eavesdropped. Those are just a few; there’s lots of other ones.

Many user interface designers for browsers believe that most users don’t understand what a protocol is, which is probably accurate. Google Chrome’s solution is to hide the protocol when it’s http, but to display the protocol in all other cases. Firefox is now adopting the same behavior.

There’s a number of things that trouble me about this approach. I’ve already written about the behavioral impacts, whereby user expectations of copy-paste are broken and a confusing mode is introduced. Furthermore, because protocol information is still displayed for any non-http resource, understanding how to read the address bar is (ironically) made more complex.

Aside from those concerns, however, there’s something else I’m worried about. The main argument I’ve heard against exposing protocol information to end-users is that, if we present it, we might as well present all kinds of other information about the TCP connection, CPU registers, and other obscure technical statistics.

Now, I know I’m biased because I’m on the Hackasaurus team and trying to teach people the basics of HTML and CSS, but browsers have historically been very friendly to learning web-making, in part because they keep protocol information in the address bar. My guess is that removing the http:// neither helps nor hinders someone from using the basics of the web—but it definitely makes it harder to learn what hypertext is.

Understanding technology is relative. Someone can know the basics of writing an anchor tag without knowing what TCP/IP is, and it’s still quite empowering, in much the same way that it’s empowering to know how to grill vegetables without necessarily knowing everything about the chemical reactions taking place underneath.

Doing little things in the interface that promote transparency and help people move from being a web-user to a web-maker is important, so long as we don’t make things difficult for the people that just want to be users. I’ve never found my parents or other non-technical users to be confused by the presence of http://, which is part of why I don’t see much gain in removing it—especially given the behavioral shortcomings of this change. Far more exciting to me is the exact opposite approach: designing experiences to help users understand what the URL in their address bar means, and encouraging them to create things on the Web instead of just browse.


I’ve been reading Eli Pariser’s book The Filter Bubble and was fascinated by his description of how data collection companies operate. Independently of that, David Ascher suggested that I add a feature to the Hackasaurus goggles which helps learners understand how cookies and tracking works.

I actually didn’t know a lot about tracking myself, so I whipped up a Firefox add-on called Collusion to help me visualize it better. The results were a little unsettling.

I’ve put a demonstration up at, which takes you through five popular websites and visualizes the data collection companies that track you across them. From there, you can download the add-on if you want to see the tracking visualization of your own browsing behavior evolve in real-time.

Special thanks to the Mozilla Add-on SDK team for making a great foundation to build on. This experiment also gave me a chance to play around with d3.js, which is a fantastic successor to Protovis. And thanks to PrivacyChoice for their excellent tracker list, which I’m sort of using without their permission. I hope that is okay.

I’m also not really a privacy expert, so I’m not sure if everything I say in the demonstration is completely true. If you find any inaccuracies, please let me know.

Finally, if you need the source code, it’s all at I’m particularly interested in seeing better visualizations than the force-directed graph I’m using, which regrettably requires a lot of user interaction to explore and understand.

Making Hackasaurus Remix Easier

I’ve been working on an experimental iteration in the X-Ray Goggles which addresses some problems we observed kids having with the Compose A Replacement (aka “remix”) dialog at the TEDx Kids Brussels 2011 event and other hack jams.

The current remix dialog looks like this when focused on a Facebook sidebar on the New York Times website:

There’s a number of problems with this dialog:

  • The HTML Source Code column isn’t formatted in a way that helps the reader understand much about the structure of a Web page.
  • Despite the syntax highlighting provided by Ace, it’s very easy for newcomers to struggle with HTML syntax, missing out on more important concepts (and more fun experimentation).
  • The What Your Browser Sees column is frequently overlooked by users; as such, it doesn’t effectively communicate the idea of a DOM hierarchy, and it also takes up lots of screen space.
  • The What You See column doesn’t inherit the CSS styling of the page being remixed, which is very confusing.

The Easy Remix Dialog is an attempt to resolve the above issues, inspired by the interfaces of tools like Scratch and Yahoo Pipes. Here’s what the new design looks like:

You can see it in action in the following silent screencast.

Continue reading “Making Hackasaurus Remix Easier”

Three Lessons Learned From Hack Jams

The first five Hackasaurus hack jams taught us a lot. Here are a few lessons we learned from them.

  1. Know How Much Freedom You Have.

    At our hack jams in the New York Public Library, search we discovered that their publicly-accessible Windows XP machines were so locked-down that we were unable to run Firefox or any other programs off a USB stick. Our prototype Web X-Ray Goggles don’t currently work with the built-in Internet Explorer 8, so we weren’t able to use them. Even more distressing was that we couldn’t even show kids the power of View Source, because that feature was actually disabled on the library’s computers.

    This gets to the heart of what Jonathan Zittrain has described as Freedom At The Endpoints, because such a thoroughly locked-down computer—particularly one with an aging browser that doesn’t support many of the Web’s greatest benefits—has relatively little generative potential.

    We still need to figure out how to make it accessible for people to experience the full awesomeness of the Web in public spaces, but in the meantime, we’ve learned to bring our own computers for kids to use when necessary.

  2. Digital Natives Know Less Than You Think.

    When I was a kid, computers were all the rage at schools around the country. Throughout elementary and middle school, my classmates and I were taught Logo, BASIC, HyperCard, touch typing, and more.

    These days, however, kids are apparently assumed by the school system to be “Digital Natives” and not in need of any technology education: as a result, many of the kids in our hack jams had never done any programming in their life, and few of them knew how to touch type. Aside from being a distressing discovery, this meant that we had to “calibrate” the jams to the experience level of our students. It varied a lot from one jam to another, since we couldn’t rely on the school system to give them a firm grounding in basic computer use.

  3. Engage Your Audience’s Interests.

    One of the broad goals of Hackasaurus is to help kids see themselves as hackers—especially girls and minorities who are traditionally under-represented in technical communities.

    The first four hack jams we did in New York City and Chicago were promoted as Hackasaurus events: the main reason to attend was to learn how to “hack the Web”. Unsurprisingly, we only got kids who self-identified as potential hackers, which meant that nearly everyone who showed up was male.

    In contrast, the fifth hack jam, held in San Francisco at the Bay Area Video Coalition, was actually a Web Made Movies Workshop where Ben Moskowitz decided to deploy the Hackasaurus tools. This was a perfect pairing, as understanding the basics of HTML and how the Web works are closely related to making movies with it.

    Since the event was promoted as a filmmaking workshop, however, we didn’t get kids who self-identified as potential hackers. They picked up hacking just as quickly as the kids from previous jams, and they enjoyed it just as much; but by teaching them about technical concepts that were directly related to things that they were already passionate about, we were able to make them really excited about the Open Web. Aside from attracting a more diverse group of kids, we were also able to engage them more directly.

    So, future Hackasaurus hack jams might not even be promoted as such. Right now we’re thinking about developing many different “genres” of jams that engage kids on non-technical issues that they’re already excited about. If you have any suggestions, we’d love to hear them!