Meritocrats, cluebats and the open development method: an interview with Justin Erenkrantz

by Paul Anderson, Intelligent Content on 6 January 2009 , last updated 15 February 2012

Introduction

If you are reading this on the Web then there is a better than evens chance that the pages were delivered to you using an Apache web server, one of the computer industry’s most famous open source projects. Since its re-launch from the ashes of earlier web development work at the National Center for Supercomputer Applications (NCSA) in Illinois, the Apache HTTP Server Project (to give it its full name) has regularly clocked up market share figures in the order of between 60% and 70% and it has long been regarded as a bridge head for the wider acceptance of open source software within corporate businesses and public organisations.

This venerable status means that those associated with Apache code development and the not-for-profit foundation that now oversees it are held in high regard. Perhaps none more so than Justin Erenkrantz, who is the Apache Software Foundation’s president¹. Despite a laid-back, southern Californian personal style, Erenkrantz is as busy as one of the Foundation’s most hyperventilating servers, managing to combine his voluntary Foundation duties with work as a software designer at European online TV start-up Joost, and undertaking a PhD at the University of California, Irvine (UCI), plus a stint helping open source development at Google. As he quietly understates: ‘I have a number of different hats.’

Despite this frantic pace (he flew to Europe twelve times last year) he still found time to come to Oxford and help OSS Watch with an educational mentoring project. While he was here I caught up with him for a one-to-one interview on Apache and the importance of its open community development method.

It all began back in late 2000, when Justin was finishing his undergraduate studies at the University of California, Irvine, a college with a strong background in Internet-related software research. Justin says: ‘I ran across a guy called Roy Fielding. Roy was completing his PhD at UCI and he had just started working for a company called e-Built. We kept in touch. I kind of got started working for different clients and Roy was one of the technical leaders and said: “Oh we are going to do an Apache module.” And this was my formal introduction to the Apache web server. It was [the] tail-end of the Version 2.0 development process—it kind of worked but didn’t have that coat of polish.’ Justin got involved in writing patches for the server, working the associated mailing-lists and interacting with the key Apache developers. This served as his introduction to the development community and led to a deeper commitment, and, eventually, to his current position.

The open development method

The Apache Software Foundation (ASF) is an umbrella organisation that supports the HTTP Server Project and a growing number of other open source development projects. It operates through a number of Project Management Committees (PMC), each of which manages the projects that are currently in development. Ultimately each PMC reports to the Board, which carries out general management and oversight and is answerable to the membership and the wider ASF community as a whole. The ASF’s structure is designed to be as open as it possibly can be, with pretty much everything going on in public, and the clue to understanding why this is so important lies in the history of how the ASF came about.

The ASF’s original, flagship product is the web server, a piece of software that handles the interaction between a client’s browser and the server that stores the webpages. The Apache code is based on original work that was undertaken on a Unix-based HTTPd daemon which was developed by Rob McCool between 1993 and 1994 at the NCSA. This was during the heady days of the early web, and McCool soon left to work with the newly founded Netscape. This meant that the web server was left unsupported until a group of concerned users including Brian Behlendorf and Roy Fielding began to informally share problems and fix bugs by e-mail. Justin says: ‘They started trading patches and stuff. Independently all these guys were trying to maintain this kind of ball of mud. So they said let’s work together so that we can make something better than each one of us individually could make on our own’. Their modifications were known as ‘patches’ and was punned to ‘apache’ as the group coalesced into a more formal structure.

So the motivation for this way of working came, in part, from the circumstances of the organisation’s inception. Justin recalls: ‘If you look at how it started you had these eight guys who had the code dropped in their lap because the guys who were maintaining it had walked away. They were kind of motivated to not see this happen again. This was the strength of open source.’

But it was not only open source, but also an open development method that helped with the success of the project. Decisions were and continue to be made in public and it is seen as very important that projects and their PMCs, who are responsible for their own actions, should continue this tradition. Justin notes that: ‘Apache is open development. We are very aggressive on that, making sure it is consistent across all the Apache projects.’ Although it doesn’t happen very often Justin admits that: ‘every once in a while one of the projects starts to do things in private and we are like “Oh no no no!” And we kind of go in there with the cluebat² [and say] no, you’re going to do everything in public and we will not stand for this!’

Open source versus open development

Justin believes open development is crucially important for long-term success and notes that two ‘flavours’ of open source are emerging. He says: ‘We are seeing a difference between open source and open development. So what we are seeing is some projects and entities who are saying here is the source code, it is under Apache licence or GPL or whatever but the decisions about how the code got to that state are behind some wall—it is not in public’. This means that a developer has all the requisite rights to edit, modify and redistribute the code, but doesn’t have any real understanding as to how or why the code developed that way in the first place. Justin says: ‘So you run into some bug and you are saying why is the code this way? There is no historical context or archiving or rationale…So, [with the open development method] what we are seeing is, that in addition to “here is the code” we are seeing “here are all the decisions being made in public”.’

The other strength of the open development method is the business-like way the ASF develops software and the peer review processes that are in place. As Justin notes, ‘We see here with the establishment of the ASF, all the work [is] done in the open, everything is peer-reviewed and this [has] led to very high quality’. This focus on the work of the developer and the need to perform to the very highest standards, in public, is reflected in the structure of the organisation.

The very first stage for a developer in the Apache development model is to be a simple ‘User’—making use of the software and offering feedback, bug reports and ideas. Evolution to the next stage is by proving your worth and getting more involved in the development process. Justin describes it as: ‘Proving over time. Being able to say you can work with the community, you can send in patches, you can add to the documentation or whatever it is.’ People at this level are known as ‘developers’ and they can suggest modifications to the code. They cannot, however, directly edit the core code in the software repository, a job that is allocated to those with more experience who are known as ‘committers’. Finally, after having been elected by others, you can become a ‘member’. This entails becoming a ‘shareholder’ of the not-for-profit ASF charitable foundation and acquiring voting rights on new members and the Board of Directors.

As a way of providing some idea of the scale of the organisation Justin says: ‘There are around 260 or so members of the Foundation and 1,700 committers who can contribute directly to some part of the code base³. If these guys stick with it then at some point they will become members.’

Loyalty, talent and sheer hard work are core Apache traits and the ASF website was recently retagged with the slogan Meritocracy in Action. In actual fact, the term ‘meritocracy’ was originally coined by British social theorist Michael Young in a famous book, The Rise of the Meritocracy. His vision of a society based purely on merit was actually a dystopian warning. When I point this out, Justin laughs loudly and says: ‘I believe Roy Fielding, who was one of the first chairs of Apache, has a background in social science and was keenly aware of that.’ Justin argues that the term neatly sums up their overall approach, saying: ‘Actually it is a good representative and [is] basically saying “If you want to do something and you have demonstrated to the community that you are trustworthy and you have this merit then we will try and give you the environment to let you do the things you want to do”.’

Pragmatic open source

Another distinction between open source and open development comes to light when Justin talks about proprietary software. This touches on a hotly debated bone of contention between the ‘free’ (as exemplified by Richard Stallman and the Free Software Foundation, or FSF) and ‘open’ software movements, and manifests itself in the different approaches taken to the development of code.

For example, the FSF is quite clear that ‘free’ software respects an individual user’s freedom. A program can only be classified as free software if its licence provides the user with four essential freedoms: the freedom to run the program as they wish; the freedom to study how the program works and adapt it; the freedom to redistribute copies; the freedom to improve the program and release improvements to the public. The Apache License Version 2.0, however, makes it clear that the source code can be modified and then incorporated into a proprietary product. This means that while Apache-licensed code qualifies as free software, products built using that code may not be free software.

Justin argues that this is simply being pragmatic and reflects what the developers want, and says: ‘By and large you’ll see that Apache projects tend to be focused on developers. These developers are primarily our target audience. One of the things with Richard Stallman’s view is that he wants the users to be free. Our attitude within Apache is that we are looking for the developers to have the freedom. They are saying: “You know what, if you want to make this into a closed source thing or if you want to make it into an open source thing then that’s fine”.’ And he adds: ‘We don’t really care necessarily how or where you use it as long as you abide by our licence, that you give us credit etc. But we are not necessarily going to say “hey you have to use it in a free and open source product”.’

Some of these differences centre around the role of software patents, an issue that remains controversial in the software industry. Justin says that ‘the FSF said “we don’t even talk about patents”. Apache License Version 2.0 has kind of set the standard in a lot of ways for how you think about patent protection and that kind of stuff. Our attitude is patents are going to be here and we have to deal with them and we have to be pragmatic. It was something that some of the other licences had done, but I think that as far as a large open source foundation [is concerned] we were very much on the cutting edge by saying that we are going to deal with this.’

There has also been much discussion in the software development community over the increasing use of Web 2.0-style software services, otherwise known as Software as a Service (SaaS) or the ‘Service Cloud’. The Free Software Foundation has responded to concerns over how one goes about licensing open source code when it is running on a remote server and providing a service through a browser client, by working on a new licence, called Affero.

Justin links this to the different ways in which the ASF and FSF view distribution: ‘It gets into [the question of] “what is distribution”. The challenge is: am I giving you the software by exposing it on a website? And the Affero exception, or however it gets phrased nowadays … [says that] putting it on a website is distributing and so you need to give the source.’ This is certainly how Richard Stallman sees it, but Justin says: ‘That doesn’t generally tend to be the way most people view distribution. They kind of realise, “Oh I am running the code on Google Docs but it doesn’t necessarily mean that I receive the software, I’m just using it.”’ Apache therefore has no plans to work on an Affero-style licence.

Open source in education

Such pragmatism also manifests itself in Justin’s attitude to the role of open source software in educational settings like schools and colleges. In contrast to Stallman, who believes schools and colleges should resist using proprietary software, Justin says: ‘Schools should use the software that is appropriate for the task. Our hope, our wish, is that open source fits those goals but that might not be the case.’ Justin feels that what is more important is that schools and colleges expose students to a wide range of different systems, both closed and open, saying: ‘One of the things about university is that it is supposed to expose people to different ideas. It shouldn’t be just pigeon-holed so that it is said, “Oh everyone here will use Microsoft” and it becomes the default. Expose students to the alternatives. Think outside the box.’

The future

Finally, we turn to the future and where next for Apache. Justin outlines the work of the Apache incubator, a formal process for setting up and incorporating new ideas and projects. He views this work as very important to the long-term development of the Foundation and an example of how important the concept of a developing community is. He says: ‘We call these [embryonic projects] podlings, and after a time they graduate and hatch and become top-level projects. There are 20 or 30 in the pipeline. They may not all succeed, they may not all attract the community, but this is how projects come into Apache.’

And what of his PhD? He argues that the research work is relevant to Apache’s long-term development of the web as he is looking at conceptual models that can describe the way different entities (clients, servers etc.) communicate and interact on the web. He is being supervised by Professor Richard Taylor(Director of the Institute for Software Research) who also supervised Roy Fielding’s well-known PhD, which first presented the Representational State Transfer (also known as REST) architectural style. This focuses on describing, in a formal way, the manner in which web-based document delivery works through the process of HTTP and URIs.

Today’s web, however, has an abundance of emerging technologies and uses what might be described as more computational ways of working, involving scripts such as AJAX, Web 2.0, data mash-ups and Web Services. What Justin is working on is something he calls CREST – Computational REST – which involves coming up with new ways of formally describing this new way of working on the web that will scale. According to Justin: ‘REST kind of said this is what the web is. It’s been that way for 10 years and this [mine] will say what the web is going to be in another 10 years.’

All in all, it’s a heady mix and Justin admits he’s very busy. The mentoring and development of software communities is a time-consuming task and he notes: ‘As Apache [Foundation] has grown from five or six projects to 70 top-level projects or so, running [it] takes a lot more [effort].’ One of the advantages of Apache’s meritocractic system is that there are plenty of talented people to draw on and Justin admits that he has recently been grateful for offers of help to delegate some of his role. ‘It is really about harnessing the energy’, he says.

At the end of the day, though, what continues to drive him is the way in which ideas about the way the web should work can be incorporated, at great speed, into the Apache code and so be made available to perhaps 60% or 70% of the servers on the web. To back this up he recounts an anecdote about Roy Fielding’s PhD viva, in which the candidate was asked how he knew that his stuff worked. The answer came back: ‘The World Wide Web.’ The examiners thought for a moment and then replied: ‘OK, we’ll accept that.’ For Justin, continuing to try to come up with methods that will handle this kind of scaling is the nub of the challenge that keeps him going. As he says: ‘This is the nice aspect of my involvement with open source.’

OSS Watch provides unbiased advice and guidance on the use, development, and licensing of free software, open source software, and open source hardware.