The Social Constraints of Bettering The Web, Part I
I’ve recently been proud and inspired to see two new features land in the latest Firefox 4 betas: Web developers can now access the raw audio data in <audio> and <video> elements, and Firefox Panorama helps users manage their tabs.
In his excellent post Experiments with audio, conclusion, Dave Humphrey mentions the following Tweet from Joe Hewitt:
Bottom line: we can currently only move as fast as employees of browser makers can go, and our imagination is limited by theirs. @joehewitt
Dave goes on to say that not one person who did the audio work in Firefox 4 was an employee of the Mozilla corporation. He also mentions the following:
As much as Mozilla made it possible for me to experiment, they also made sure that what got accepted was of the highest quality. I haven’t blogged about audio much over the past three months, mostly because we’ve been too busy getting the patch fixed up based on reviews. Before it could land we had to think about testing, security, JS performance, DOM manipulations, memory allocation, etc. To get this landed we needed lots of advice from various people, who have been generous with their time and knowledge.
As far as I can tell from my discussions with individuals in the trenches at Mountain View and throughout the Mozilla community, this process of getting advice and approval from various people is the biggest bottleneck of innovation in Firefox. Some of my talented coworkers created an excellent feature called account manager that won’t be making its way into Firefox 4 because the feature’s patches weren’t able to be reviewed in time for Firefox 4′s feature freeze. The core work was done, but the team couldn’t find the human resources to make sure it passed Firefox’s technical standards.
Other colleagues of mine are working on new built-in developer tools for Firefox, but as I understand it, there are only one or two people in the world who can review their patches and approve them for landing in Firefox’s code repository.
Within Mozilla, I see my coworkers vie for the attention of this tiny handful of gatekeepers. People in charge need convincing; the clever social engineer has a lot of power when it comes to navigating this landscape. I don’t know how much of this happens publicly via IRC, newsgroups, demos, blogs, bugs, wikis, or other backwaters of the Mozilla megaverse, as opposed to private conversations. I’m not sure how much the distinction matters, really; it’s just unfortunate that after working at Mozilla for two and a half years, I still have no idea how high-level decisions are made, who the gatekeepers are, and how they are to be convinced.
Yet I’m also sure that convincing the people who hold the keys to change a product used by hundreds of millions of people is non-trivial at any organization, and probably with good reason.
At a broad level, Hewitt’s tweet was about the notion that there exists some kind of bottleneck between people’s imagination and the ability of browser makers to keep up with it. Dave Humphrey pointed out that the bottleneck has nothing to do with paid employees of Mozilla, since anyone can contribute. My observations indicate that the biggest choke-point at Mozilla right now is a social problem that Clay Shirky might call fame: an imbalance between inbound attention and outbound attention. Lots of people want to get great features into Firefox, but a very limited set of people are actually able to pay attention to them and get those features out the door—much less determine which features are actually appropriate for the hundreds of millions the product is broadcast to.
That’s all I’ve got for now; I’ll write about some possible solutions to this in Part II.
August 31st, 2010 at 1:00 pm
[...] Atul mentioned in “The Social Constraints of Bettering The Web” [toolness.com], Account Manager will likely not make its way into Firefox 4. He points out [...]
August 31st, 2010 at 1:24 pm
Using Account Manager as an example here is misleading, and I suspect that if you ask anyone involved in that project, they’ll help you to understand why.
I’m sorry that you’re having trouble seeing how these things, and even more sorry that this is the conclusion you’ve reached from talking to people “in the trenches” at Mozilla. As one of the people involved in prioritization, focusing, and helping to align resources, that reflects a failure on my part.
I do wonder, however, how integrated you’ve attempted to become. I never see you in any of the IRC channels in which these decisions are being discussed and made day-by-day, hour-by-hour. I don’t see you commenting in the bugs. I don’t see you commenting in any of the newsgroups. I don’t see you at the meetings.
Please help me understand how we can and should communicate these decisions and constraints more broadly. I want to improve.
August 31st, 2010 at 1:50 pm
@beltzner
I don’t think you meant it to come off this way but your comment makes it sounds like the avenue to getting code in to Firefox is to be as entrenched in the day-to-day process as you are. That clearly isn’t a viable barrier to adoption so I’m sure I must not be understanding your comment.
If i have code for a feature that is likely to be accepted in to Firefox what are the barriers I face getting that code in?
Who do I talk to? Where do I start?
If the answer is jumping in to the entire Firefox release process just to get a patch that’s not acceptable in my opinion.
August 31st, 2010 at 2:10 pm
One thing I’ve noticed with at least the web dev tools is that the code is doing tons of things different to the rest of our code base. Like, using HTML instead of XUL, localizing late, what’s RTL, etc. I didn’t have the pleasure to look at the account manager code, so I can’t comment on that. Panorama code has been similar. Like, there are hard-coded strings in there still just because things don’t go together well.
For at least the dev tools and panorama code, it’s some seriously tough shit to review, and you’re not able to reuse coding patterns one is used to from the majority of our code. Bridging that gap is hard. And making the new code patterns integrate with the main code base is challenging, if possible at all ends.
August 31st, 2010 at 2:46 pm
@mikeal, with a release cycle that’s about a year long, I don’t think that getting a major new feature landed during the last month or so of that process should be a whole lot easier than “jumping in to the entire Firefox release process”.
The fact that it’s possible at all to get any new features landed during that extremely late stage is thanks in big part to the heroic efforts of all of the people involved in that “entire Firefox release process”.
August 31st, 2010 at 2:55 pm
@atul said “I don’t know how much of this happens publicly via IRC, newsgroups, demos, blogs, bugs, wikis, or other backwaters of the Mozilla megaverse, as opposed to private conversations.”
I think Beltzner’s pretty much answered it. It’s something like: “in the IRC channels… in the bugs… in the newsgroups… at the meetings” none of which I’d consider “backwaters.”
Those are actually the primary channels for the major Mozilla projects and have been for more than a decade.
August 31st, 2010 at 3:13 pm
@mikeal certainly not entrenched, no. What I meant to imply is that all of these avenues are open, and generally speaking, I and others make ourselves highly available to those who have questions about how to get things landed. We are unlike pretty much any other project I know about these days.
I am also being extremely genuine when I say that I would love to know how to improve. Who’s doing this better than us, and how are they doing it better? We know our tools can improve, and perhaps our documentation can improve (I forgot to list MDN!), but to do so we need help and feedback on what we could be doing better.
August 31st, 2010 at 3:28 pm
@asa to be clear, the Sync and Tab Candy guys were working – as were the hackers – for months before those features landed, getting early feedback, reviews, and asking questions to understand what they needed to do in order get things into the tree. It was more than just getting it in at the 11th hour; these are just the costs of making software.
It looks easy, it’s actually very, very hard.
My feeling is that a lot of this has to do with some of the particularities of our codebase, and the fact that many people feel ashamed or embarrassed about asking questions or doing something wrong. There’s been more and more work that comes in large bunches which is harder to review, understand and reason about than smaller, incremental changes, or changes that were tempered with more frequently reviews starting at a higher level. The feedback? flag was added to help with this, and help it has!
Ironically, some of the reason it’s hard to get things into our codebase is *because* of the rate of change, and how that affects dependencies in the tree. Another reason is that we’ve added so much in the past few years, and the agents have change have moved on to change other things, and we’re still fixing bugs from those initial changes, limiting our ability to focus on the next, new change.
August 31st, 2010 at 6:41 pm
>People in charge need convincing; the clever social engineer
>has a lot of power when it comes to navigating this landscape.
>I don’t know how much of this happens publicly
Actually the public itself often ends up as a source of power that one can wield in the negotiation (especially if you are a very clever social engineer). If you announce a feature will be in Firefox before anyone in charge has fully signed off, you can create a leveraged situation where the press will say we didn’t meet expectations if your project doesn’t land. Going rouge publicly is a very effective strategy for one to acquire limited developer resources.
In some ways, this is an efficient and darwinian way for us to operate, features that gain political capital (regardless of the tactics employed) get to spend it. However things will fall apart pretty quickly if everyone tries to go rouge simultaneously.
Just one more complexity of doing everything in the open.
August 31st, 2010 at 7:29 pm
In terms of feedback I think it’s pretty clear what Atul was suggesting.
@beltzner said about sync etc “getting early feedback, reviews, and asking questions to understand what they needed to do in order get things into the tree”
Atul’s feedback was very specific. There are not enough people to give the feedback, perform the reviews, or answer the questions to cover everyone who would like to contribute. To oversimplify – the people who are capable of doing this could be devoting more of their time to this, alternately more people capable of doing this are needed.
The point about where this happens was, as I read it, secondary.
August 31st, 2010 at 9:56 pm
@Lucy I’m not sure why atul thinks that only one or two people can review the dev tools stuff. The stuff in toolkit can have any of the people in “peers” review it, as well as the module owner:
http://www.mozilla.org/projects/toolkit/review.html
That makes for ten possible people to review that code. The stuff in browser is a bit more constrained (and this page appears to be out of date), but peers and module owner again:
http://www.mozilla.org/projects/firefox/review.html
Justin Dolske should also be listed as a peer there, which brings the total number of people who can review those changes to four.
In time, the people hacking on developer tools will be able to review code for their team, but they are still learning the requirements of the firefox and toolkit modules. Yes, that means reviews are going to be resources constrained for a bit, but we essentially doubled the number of people working full time at least on the firefox module with just the dev tools team. Then we’ve also got Sync and Panorama which also steal time from these peers, and we’ve probably tripled or quadrupled the number of people hacking in the firefox module. And that’s just in the past two or three months.
It doesn’t help that during the run-up to a freeze, the review load increases (this is my perception, I don’t actually have data to back it up as of yet). Yes, we need more peers, but we need to train people for that position. Asking peers to only do reviews is also not fair to the work they contribute.
August 31st, 2010 at 9:57 pm
Atul, shouldn’t all of these features start out as addons, get feedback from real (ok may be extra-motivated) users, demonstrate compatibility with the platform, then be reviewed for coding details? That is more or less the story of Sync as I understand it, not so much for some of the other examples. Code review for an addition that users don’t want or can’t use wastes the most precious resources on the project.
September 1st, 2010 at 7:42 pm
@sdwilsh I agree with what you said, however that doesn’t change Atul’s point about it being a bottleneck. While you pointed out some good reasons why things get congested especially at release time, you also did point out some things that can be improved – eg keeping those pages up to date.
A wise man once said “love every idea for 5 minutes.” Maybe Atul hasn’t been participating in the places Beltzner listed, and maybe he’s wrong about the toolkit reviews, but what is he right about?
- Do you have to be in IRC or newsgroups to understand how the decision making process works? If yes, this should be fixed. There should be a document that tells you to go to IRC and the newsgroups. If there is already such a document someone should link it so people like Atul who want to know can learn.
- Are there enough people to properly assist community contributors? The answer seems to be “not at release time” which is perfectly fine. On the other hand, if that’s the case there should be an outreach to those people at the start of the next development cycle when devs do have time.
I don’t think anyone would say Mozilla is always perfect. Atul is just trying to help fix something he perceives to be a problem. Though I also learned the hard way release time is _not_ a good time to criticize devs about anything
Looking forward to part II.
September 2nd, 2010 at 2:27 pm
There’s a tremendous amount of things going on right now, and already when the Firefox 4 plan was presented, I expected that some of the things in the plan might need to be cut (and to Beltzner’s rescue, he had even marked some as possibly making it and possibly not), I knew if the full plan with all its options was to make it in that time, this community would be not just awesome (which is surely is) but have almost god-like powers.
The reason I’m saying this here is because that incredible pace the Firefox team is working on right now must naturally have some fallout, and among that is forking Firefox stuff away from toolkit stuff more than usual (which hurts non-FF projects to some degree, depending on what it is) and growing constraints on review time, as usually the best reviewers are also awesome coders, and unfortunately we can’t multiply them at will – it’s a well-known problem in our community that too few implementations of nsIClonableDeveloper exist.
The topic of reviews is a recurring topic, and we need to continue to think about it, how we can improve things for developers, but we need to keep in mind that it’s incredibly easy to write some pieces of code that seem awesome but have serious flaws and are not well-thought-through – or aren’t stable enough to be in shipping code, all of which are things that usually come up in review and need revisions. We have been burned with suboptimal-quality code before, esp. from large projects that had little review before landing, if we want to keep going at high quality, we need good reviews – but that also means not rushing in things shortly before a release just because they might be cool, but going for landing earlier in the release cycles if possible. Sure, high code quality and thorough review and shipping a large amount of cool new stuff might be a tradeoff when personnel resources are limited, but one that we need to carefully think about and not one to judge too fast.
September 5th, 2010 at 2:21 am
Hey Atul…Really a nice post…planet Mozilla really shows how large and complex software is built…and how difficult it is to manage things (sometime “doing it” easier then “manage it”).
Regarding reviewer not available issue, I suggest one solution (I am not mozilla committer, so please ignore if its too naive.)
How about building community of reviewer. I am sure with such a large developer base, There are many who can do this job (May be not from Mozilla corp). So only issue i see here is they are not identified. How about some ranking system which just grades reviewer based on his past reviews. This way we can reduce load on final reviewer.
Just my 2 bits..!
October 4th, 2010 at 12:18 pm
[...] I mentioned in my post on The Social Constraints of Bettering The Web, finding a code reviewer can be difficult in Mozilla projects. At least, it’s definitely the [...]
November 8th, 2010 at 9:31 am
[...] several days, weeks, or even months—and that’s not even including the implicit task of finding a reviewer in the first place. All this can sometimes make the pace of development slow to a crawl: imagine a [...]
January 16th, 2011 at 10:21 pm
I’m looking at possibly getting into contributing for mozilla. Business is slow, and I’d like some form of creative satisfaction. It’s interesting to read how someone involved in the process first hand perceives the management.