Exim: a case study in sustainability

by Philip Hazel, University of Cambridge Computing Service on 5 June 2007 , last updated

Introduction

In the latter half of 2006, the Joint Information Systems Committee (JISC) commissioned a study via its Teaching and Learning committee to examine the issues surrounding sustainability of open source software. The resulting report drew together seven case studies of successful but very different open source projects and examined each project’s sustainability model. Each of these case studies has been told from the point of view of the lead developer or one of the key personnel and gives a fascinating insight into the factors that have determined the success of each project. These case studies are now presented by OSS Watch as stand alone documents in a series.

This case study, examining the Exim project, has been written by Philip Hazel, University of Cambridge Computing Service.

Brief description

Exim is a Mail Transfer Agent (MTA), a program that delivers electronic mail. It is an extremely robust, enterprise-level application, released under the GNU General Public License (GPL) and has been adopted as the default MTA used by Debian. It runs on most Unix-like operating systems, replacing the traditional Sendmail program, and can also be run under Cygwin on Windows systems. A complete, up-to-date reference manual is included with each release.

I am the original author of Exim. So far, the code development part of the project has largely been sustained by my almost full-time contribution sanctioned by my employer, the University of Cambridge. Many people have sent in patches to fix bugs and code for enhancements, and several major portions of the code have been sent in by, and are maintained by, other contributors. The peripheral support apparatus (website, mailing lists, etc.) is maintained by volunteers, partly in their own time, and partly with the blessing of their employers. Nobody is paid for working on the Exim project. The project does not exist in any formal sense, and therefore has no means of acquiring money or paying for anything.

The University of Cambridge has played a major role in sustaining the Exim project and I have had nothing but positive support from my management and colleagues in this endeavour.

Introduction

MTAs are the postmen of the Internet, transporting e-mail messages from one host to another, and ultimately delivering them into mailboxes. The users of Exim, in the sense of those who choose to run it instead of one of the other available MTAs, are system administrators and those who choose default MTAs for packaged operating system distributions.

The distinguishing characteristic of Exim is its flexibility. Its configuration facilities are more like a kit of parts than a set of preconfigured solutions. It is able to use data from external sources such as LDAP or SQL databases as well as from files, sockets, and the Domain Name Service (DNS). Although Exim was designed for moderate-sized Internet e-mail servers, it has also found favour elsewhere, and has been used in many different environments, from personal computers to very large servers handling millions of accounts.

From a sustainability point of view Exim started as a small, one-man project and has grown to become a significant player in its field. Although it could still be considered a one-man project for most of the basic code modules and the reference manual, it has always needed the support of others to keep it going and to help it grow and develop. To all intents and purposes Exim is run as a ‘benevolent dictatorship’ but this may have to change when I retire, in three years’ time.

Project history

In 1995, the central servers at the University of Cambridge were running a variety of MTAs, including Sendmail, Smail 3, and PP. At that time, the Internet was a fairly friendly place and there was little need to take many precautions against hostile acts; most sites ran open mail relays, for example. It was clear, however, that this situation was changing and that new requirements were arising. I had implemented some modifications to Smail, but by then it was eight-year-old code, written in pre-standard C, and originally designed for use in a very different environment. I therefore decided to see if I could build a new MTA from scratch, taking the basic philosophy of Smail and extending it, and I began Exim as a one-man experimental project early in 1995. As I wasn’t exactly sure what the outcome would be, I called it EXperimental Internet Mailer (Exim).

In November 1995, one of my colleagues found out about what I was doing and, after having begged an evaluation copy, put it into service in the Computer Science Department at Cambridge. He started telling others about it so I began putting releases on an FTP site and answering e-mail about it. The early releases were never formally announced, details of them just spread by word of mouth.

After about a year I had a working program, and in June 1996 I gave a talk at a ‘Campus Mail Day’ in Aberdeen. When people heard about it they started using it, and this increased usage has led to Exim’s growth and created an interest in seeing it develop further.

Growth and development

The original releases of Exim used a regular expression library that provided basic features only. I could not find a library that supported anything close to the powerful, regular expressions supported by Perl, so in the summer of 1998 I decided to write my own. This became Perl-Compatible Regular Expressions (PCRE), which I released as a separate open source application.

It seemed that I had unwittingly filled a widely felt need, and soon PCRE was being used by many projects of all kinds, and not only on Unix-like systems. Many of the projects that use PCRE are well-known applications, such as Apache, PHP, and Mathematica. As a consequence, PCRE has needed a lot more development than I originally anticipated, and the project is still active. Unlike Exim, the core PCRE C library has remained a one-person project, though the C++ wrapper was built by a programmer from Google Inc. and other Google programmers have subsequently maintained it.

A year or two after Exim came into existence I was asked by a UK ISP to run an in-house training course for their e-mail team. This took place in the spring of 1999, and in the summer I ran it again, but this time for Planet Online, another UK-based ISP. At about the same time there were requests on the mailing list for Exim training, so in September 1999 we ran the first course at a college in Cambridge. We had expected 30—40 attendees at the most, but ended up with 120. This was a one-day course, and there was a subsequent ‘advanced’ course the following spring. After that we settled into a pattern of running the course once a year, in late June or July.

In 2002 the training course was expanded to two days, and a couple of slots were filled by external speakers, who were invited to give case study talks. In 2005 we extended the course to three days because, with the addition of the content-scanning features to Exim, the subject matter would no longer fit into two days. As part of this extension, two sessions of practical exercises were added. As we have no income streams or sponsorship, the costs of the courses are met by charging the attendees: the accommodation is at a Cambridge college, and the charges are just passed on. So far we have not made a loss.

In September 1997 Planet Online volunteered to run a website and mailing list. Although Planet Online no longer exists, Nigel Metheringham, who was working for them at the time, continues to work on it. He maintains the mailing list and has also contributed to the code.

When I wrote Exim, I envisaged it being used by experienced Unix system administrators who would be installing it from source. Then Debian decided to use Exim as their default MTA and things changed. Debian 2.1, codenamed Slink, was released in March 1999, and this was the first Debian stable release that had Exim as its default MTA. This brought into existence a whole new class of Exim user, many of whom had little experience of running a Unix-like system, and even less of running a mail server. There was an enormous increase in the number of ‘newbie’ questions on the mailing lists but fortunately there were a number of people subscribed to the list who were able and willing to jump in and answer newbie-type questions. So it didn’t all collapse, but it changed the nature of things a bit and ultimately led to an Exim developers’ mailing list splitting off from the Exim users’ list.

Just to complicate this a little bit more, Debian adopted Exim 3 rather than Exim 4, which meant that each new user of Debian became a new user of Exim 3 and, in fact, it was several years before Debian’s next major release came out with Exim 4. This wasn’t the best thing, as all the experienced Exim users were on Exim 4 and were rapidly forgetting Exim 3. In the end, the Exim project had to just forge ahead, and Debian had to take responsibility for things like back-porting security patches from Exim 4 to Exim 3. The load on Exim maintainers therefore became the need to continually refer Debian users back to Debian.

I wasn’t involved in Debian’s formal discussions around adopting Exim, but, with hindsight, having advance warning about a sudden influx of new users with much less general experience than the existing community would have been helpful. If you had the resources and if you knew something like this was coming, then getting somebody to write introductory and teaching material in advance would be the sensible thing to do. Nowadays there’s a lot more information around in the wiki but back then we didn’t have a wiki. In addition, the documentation provided with Exim was (and is) very much reference material intended for the original readership of experienced Unix system administrators. When less experienced people started trying to administer Exim, the need for more introductory documentation became acute. To address this I wrote a book, Exim: the Mail Transport Agent, as a kind of tutorial. This was published by O’Reilly in 2001; a revised version, The Exim SMTP Mail Server: Official Guide for Release 4 was published in 2003.

Project structure: sustainability

Over the years Exim has grown into a major project. The original website and mailing list were run on Planet Online’s server for several years, but a number of factors, such as the hardware getting old and staff changes, meant that another solution had to be found. As the University of Cambridge Computing Service had formally agreed to support Exim, and potentially other open source projects, they installed a dedicated host machine in May 2004, and services were transferred to it that summer. The website, mailing lists, Wiki, and Bugzilla are now hosted there, and the primary FTP site is the University’s dedicated FTP server.

As mentioned before, nobody is paid for working on the Exim project, and the project has no means of acquiring money or paying for anything.

Key participants

I still maintain the majority (but not all) of the code that comprises Exim and its utilities1. However, some sections of the code, some utility programs, and all of the peripheral support, such as the website, mailing lists, CVS repository, Wiki, and Bugzilla are maintained by other key participants. Tom Kistner implemented and maintains the content-scanning extension to Exim. Michael Haardt implemented and maintains the Sieve extension. Steve Campbell took over and continues to maintain the eximstats utility. John Jetmore implemented and maintains the exipick utility. Nigel Metheringham maintains the mailing lists and has contributed to the code. Two Cambridge colleagues handle all the arrangements for the annual Exim training course.

Roles are what people make for themselves—many, of course, just submitted one fix/addition to solve a problem that they had encountered. My acknowledgments file contains 150 names, and I know it is not complete.

Project structure: process and governance

Because this project ‘just grew’, there is no formal governance model that can be said to be behind it. So far, the code development part of the project has been sustained to a large extent by my almost full-time contribution, sanctioned by my employer. Many people have sent in patches to fix bugs and code for enhancements, which I review before adding them to the sources. In most cases the explanation for why people participate would be “I like Exim but it doesn’t quite do what I need: here’s a patch”. Some of the bigger contributors then got involved in keeping the whole thing going. But people do drop in and drop out from time to time.

Apart from just using Exim, many of the worldwide community of Exim users participate by contributing code and ideas, which are discussed on the mailing lists. This includes both bug fixes and program extensions (sometimes substantial). For the most part, the atmosphere on the mailing lists is friendly and co-operative, something I particularly welcome.

A large number of people have contributed additions of various sizes, but do not formally retain responsibility for them. Some of the larger ones are interfaces to various databases (e.g. LDAP, MySQL, PostgreSQL) and different authentication methods (e.g. Cyrus SASL, Dovecot).

Up to now, I have taken most of the major decisions in consultation with other developers and those on the mailing lists. However, recently, I have been devolving more and more of the project in preparation for my retirement in a few years’ time. When that happens, there may well have to be more formality about the governance structure.

Reflections and future

E-mail is one of the major Internet services. Its use is still expanding rapidly. The requirement for software to process e-mail is going to be with us for the foreseeable future. However, when I first wrote Exim, back in 1995, I assumed that it would be ‘finished’ after a year or two—after all, mail reception and delivery was surely a well-understood process, and once Exim was doing it properly, that would be that. How wrong I was! There has been a continuous stream of requirements for new functionality, caused partly by the explosion of Internet usage and the growth of e-mail abuse, and partly because people keep coming up with new ways of processing e-mail. I now suspect that this will continue for some years and if Exim is to remain viable, it will have to continue to develop. However, I’m also aware that in time software ‘wears out’ and people move on to something else. Exim has had over ten years now, and it may be that there is a better ‘mousetrap’ waiting to burst onto the scene.

At the time of writing, the Exim project is approaching a watershed, because at some time in the next three years I will retire. Although I do not intend to disappear completely, I do not want to continue working on Exim after retirement other than in an occasional, advisory capacity. There are a number of people who understand the code of Exim quite well, and I hope that there is now enough momentum to keep the development process going, albeit perhaps at a slower pace.

However, I suspect that some more formal governance procedures will need to be devised, and also some working rules for decisions, such as when to create a new release and whether or not to include a particular patch. My feeling is that when one person doing most of the work has to be replaced by lots of people doing smaller parts, there will need to be more formality in order to avoid too much disorder.

Project details

The Exim website is http://www.exim.org/, which is mirrored at a number of sites around the world. There is no separate development site. The project’s files are kept in a CVS repository, but at present there is no anonymous access. However, a nightly snapshot of the code and test suite is automatically uploaded to an FTP site for the benefit of those who do not have accounts on the CVS system. The reference manual is online at the website, as well as being included as a text file in the Exim distribution. It is also downloadable in PostScript, PDF, HTML, and Texinfo formats.

There are now three mailing lists of interest to Exim users: exim-users, a general user discussion list; exim-dev, a discussion of development issues; exim-announce, a low-traffic announcements list. These lists are open to all but only subscribers may post to the first two, and the third is restricted to a few authorised posters.

There are two further lists, of interest to developers. The exim-cvs list receives commit messages from the CVS system. Anyone who wants to keep a close watch on the developing source may subscribe (by e-mail). The exim-maintainers list is a hand-maintained list of those who have a CVS commit privilege. Nigel Metheringham set up the mailing lists and maintains them. He also proposed and implemented the policies.

Subscription is handled by a web form.

The website also hosts a Wiki at http://www.exim.org/eximwiki/ and a Bugzilla at http://www.exim.org/bugzilla/.

Current status

The following status update was kindly provided by Nigel Metheringham in December 2008:

Philip Hazel retired from the University of Cambridge in September 2007, having headed up the development of Exim throughout its history. In the run up to his retirement a number of changes were made to the way the project works, with a publicly visible (with a small number of committers) CVS archive of source and documentation made available on a University of Cambridge system as well as a publicly visible bug tracking system. Exim releases after 4.50 (Early 2005) had a number of changes committed directly by various authors (around 6 people in total) into CVS, although the majority of changes were still committed by Philip.

There has been one release (version 4.69) since Philip retired, although there are a significant number of bug fixes or enhancements currently within the source repository. We expect there to be a further release in the next few months. However release management is a significant task, and involves considerable work on the part of those involved - this means that releases are likely to be infrequent unless someone takes on that role.

At this time there is no single Project Leader managing the project, but instead a group of a few people that have been involved with Exim for several years. One of the particular problems is that many of those involved are no longer involved in managing mail systems for the majority of their time, and so have less immediate interest in pushing further development. Exim is also a mature project - it has wide functionality and is fairly feature complete. Many of the requests for additional functionality tend to be addressing niches which have not attracted development effort. Email handling issues, protocols and solutions are also moving along relatively slowly compared to the changes when the project started.

The mailing lists are still well used for support of Exim, with a combination of new and experienced users posting on the lists. The versions of Debian supplying Exim version 3 have mostly become obsolete, so meaning that versions of Exim before version 4 can now be realistically considered obsolete.

At present, in my role as non-leader, I expect Exim to continue in a maintenance role for a number of years, unless and until significant new email handling functionality is required, more people join as active members of the project team, or another set of mail handling software takes mindshare from Exim.

As of December 2011 the current version of Exim is 4.82. The mailing lists continue to see activity.

Further reading

Links:

Related information from OSS Watch:

Acknowledgements

The sustainability study from which this case study is taken was commissioned by the JISC Learning and Teaching committee and funded from HEFCE’s IT Infrastructure funds. The Learning and Teaching committe is responsible for supporting the learning and teaching community by helping institutions to promote innovation in the use of ICT to benefit learning and teaching, research and the management of institutions.

The sustainability study was edited by Gaynor Backhouse of IntelligentContent and her editorial guidance has contributed in large part to the excellent result.

  1. Philip Hazel has now retired, see the current status section for more information. ↩