Free/open source programming, and software engineering

by Sebastian Rahtz on 1 August 2003


Free/open source programming, and software engineering Sebastian Rahtz Information Manager Oxford University Computing Services (also JISC Open Source Software Advisory Service) August 2003


  • What is free/open source software?
  • What is software engineering?
  • Does free/open source imply a mode of programming?
  • Do free/open source methodologies develop from, turn into, or co-exist with changes in software engineering?
  • What is the future for computer programming?

What is Free/Open Source Software?

Software for which:

  • the source code is available to the end-user;
  • the source code can be modified by the end-user;
  • the licensing conditions are intended to facilitate continued re-use and wide availability of the software, in both commercial and non-commercial contexts;
  • the cost of acquisition to the end-user is often minimal.Open Source is a development methodology; Free Software is a social movement.

Virtues of free and open source software?

  • has no secrets: the innards are available for anyone to inspect
  • is not privately controlled: so likely to promote open rather than proprietary formats
  • is typically maintained by communities rather than corporations: so bug fixes and enhancement are often frequent and free
  • is usually distributed free of charge (developers make their money from support, training, and specialist add-ons; not marketing)

Clearing up misunderstandings

  • Free software uses the free from freedom, not the one from free beer. Open source software may or may not cost money
  • The cost of ownership often bears little relation to the cost of acquiring a piece of software
  • Public domain is something different. Open source software has a copyright holder and conditions of legal use
  • Open source software does not mandate exclusivity. You can use open source programs under Windows
  • People do not choose software solely on the basis of open source. Interoperability and open standards for data are equally important


  • The father, high priest, best emissary, and bête noire, of free software is Richard Stallman

  • The man who made it mainstream by creating Linux, the Gandhi of Finland, is Linus Torvalds

The cathedral and the bazaar

F/OSS projects tend to be classified into two extreme camps:

  • A genius programmer creates something which others like. (S)he controls pace and scope of future development.
  • A group of people get together to fill a gap and take on different tasks. As time goes by, some drift away and others join in. There is always someone to take over. Most projects fall in between. Most of them have a recognised leader, to whom other members defer. A gentle system of leadership challenge and deposition assures the health of the herd is kept up. (see Garcia and Steinmueller, paper 92, for much more on this)

F/OSS groupstyle creation characteristics (1)

Free and open source software tends to have the following positive development characteristics:

  • Programmer commitment, because the programmer is also the user
  • Rapid change, because programmers want to see results
  • Unconstrained specifications, because there is no external client
  • Collective ownership of code
  • Response to change, dictated by (perhaps unexpected) users

F/OSS groupstyle creation characteristics (2)

… and the following negative characteristics:

  • Unrealistic expectations, because there is no client controlling the specification
  • Uneven development rates, because programmers work on what they want
  • Flat managerial tree, because programmers are not working to contract, so no control or direction
  • Changing resources (no guarentee of work hours)
  • Possibly conflicting goals or aspirations
  • It does not have to succeed_a bit like a rock band?_

F/OSS groupstyle creation characteristics (3)

But the following do not necessarily apply:

  • Poor quality code: likely or as unlikely as in traditional programming
  • Slow development: depends on who takes an interest and how hard they work
  • It isn’t as polished as commercial software: much commercial software is riddled with problems
  • No-one supports it: most projects have a large group of “hangers on” who do not directly develop, but do help others

Software development vocabulary

  • a process is defined that has milestones
  • activities are planned, to be carried out by teams
  • teams are formed by people who take roles and have skills
  • activities lead to products by the use of techniques
  • techniques can be supported by tools
  • products have to adhere to standards
  • products have to satisfy quality benchmarks

Sofware methodologies describe

  • Methodology size and weight
  • Ceremony, the amount of precision, tighteness, tolerance
  • Problem size
  • System criticality
  • Stability, the likelihood of change

Traditional software development

Relies on sequential phases of development; it makes several dangerous assumptions:

  • the existence of a set of requirements that is defined beforehand and is maintained unchanged during development;
  • the assumption that all code is designed from scratch, making no allowance for the reuse of existing software components
  • no repeating previous phases of the development without having to repeat the whole sequence of steps for the entirety of the system.
  • development as a linear process.

Iterative and incremental development methods

Incremental development:

  • proceeds from an initial subset of the requirements to more and more complete subsets, until the whole system is addressed
  • partitions the desired functionality
  • supports parallel development The final product results from the total integration of the partitions.

Lighter methodologies

Over the last 3-4 years, software theorists have tried to deal with issues such as:

  • Software must change more and more quickly these days
  • Software keep being delivered late
  • The customer is not involved with what she is buying
  • Programmers lack motivation
  • We need new software all the time The industry needed the equivalent of fast food: predictable quality delivered quickly.

A methodology: Extreme Programming

  • The Planning Game
  • Small Releases
  • System Metaphor (a story)
  • Simple Design
  • Continuous Testing
  • Refactoring
  • Pair Programming
  • Collective Code Ownership
  • Continuous Integration
  • 40-Hour Work Week
  • On-site Customer
  • Coding Standards

Extreme: the details

  1. The Planning Game: Business and development cooperate to produce the maximum business value as rapidly as possible. The planning game happens at various scales, but the basic rules are always the same:
  • Business comes up with a list of desired features for the system. Each feature is written out as a User Story, which gives the feature a name, and describes in broad strokes what is required. User stories are typically written on 4x6 cards.
  • Development estimates how much effort each story will take, and how much effort the team can produce in a given time interval (the iteration).
  • Business then decides which stories to implement in what order, as well as when and how often to produce a production releases of the system.

Extreme: the details (2)

  1. Small Releases: Start with the smallest useful feature set. Release early and often, adding a few features each time.

  2. System Metaphor: Each project has an organizing metaphor, which provides an easy to remember naming convention.

  3. Simple Design: Always use the simplest possible design that gets the job done. The requirements will change tomorrow, so only do what’s needed to meet today’s requirements.

Extreme: the details (3)

  1. Continuous Testing: Before programmers add a feature, they write a test for it. When the suite runs, the job is done. Tests in XP come in two basic flavors.
  • Unit Tests are automated tests written by the developers to test functionality as they write it. Each unit test typically tests only a single class, or a small cluster of classes. Unit tests are typically written using a unit testing framework, such as JUnit.
  • Acceptance Tests (also known as Functional Tests) are specified by the customer to test that the overall system is functioning as specified. Acceptance tests typically test the entire system, or some large chunk of it. When all the acceptance tests pass for a given user story, that story is considered complete. At the very least, an acceptance test could consist of a script of user interface actions and expected results that a human can run. Ideally acceptance tests should be automated, either using the unit testing framework, or a separate acceptance testing framework.
  1. Refactoring: Refactor out any duplicate code generated in a coding session. You can do this with confidence that you didn’t break anything because you have the tests.

Extreme: the details (4)

  1. Pair Programming: All production code is written by two programmers sitting at one machine. Essentially, all code is reviewed as it is written.

  2. Collective Code Ownership: No single person “owns” a module. Any developer is expect to be able to work on any part of the codebase at any time.

  3. Continuous Integration: All changes are integrated into the codebase at least daily. The tests have to run 100% both before and after integration.

  4. 40-Hour Work Week: Programmers go home on time. In crunch mode, up to one week of overtime is allowed. But multiple consecutive weeks of overtime are treated as a sign that something is very wrong with the process.

Extreme: the details (5)

  1. On-site Customer: Development team has continuous access to a real live customer, that is, someone who will actually be using the system. For commercial software with lots of customers, a customer proxy (usually the product manager) is used instead.

  2. Coding Standards: Everyone codes to the same standards. Ideally, you shouldn’t be able to tell by looking at it who on the team has touched a specific piece of code.

A movement: Agile

Agile is a conscious revolutionary movement: It prefers:

Individuals over processes and tools
Working software over full documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Agile software engineering

Traditional engineering | Agile engineering

passive client | active client minimal changes | encourage change process rules | developer rules long iterations | short iterations static process | dynamic process

The Agile Manifesto

  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Business people and developers must work together daily throughout the project.
  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The Agile Manifesto (2)

  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Working software is the primary measure of progress.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity—the art of maximizing the amount of work not done—is essential.
  • The best architectures, requirements, and designs emerge from self-organizing teams.
  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

What does open source add to the mix?

Some open source projects make a virtue of:

  • Space, time and culture separation of team members
  • Semi-professional programmers having fun
  • Publicity and (friendly) competition
  • Co-operating systems (supply of components)
  • Experience with virtual communities
  • Promotion of open standards
  • Allowance of failure Others are traditional programming teams in big companies.

What does free software add to the mix?

  • Free as in beer
  • Concept of the public good
  • Identification of software as unpatentable human heritage
  • Anti-establishment ethos
  • Missionary conversion of others

Open standards meets free/open source?

Which is better?

  • Commercial software which uses an XML data format and can be accessed using web service protocolsor

  • An free/open source program which uses its own binary data format, its own interface, and its own programming language Open data and open communication between different components of your IT system gives you better interchangeability of components.

Sociological perspectives on F/OSS

  • An essential characteristic: Novel use of IPR to distribute/publish software under public domain-like conditions
  • Almost essential: _Distributed mode of creating (producing) an information-good: software _
  • Common but not necessary: Extensive voluntary participation by communities of skilled and neophyte software developers
  • Likely: Dependence of the production mode upon the non-proprietary distribution regime
  • True across the board: Critical role of computer-mediated communications (CMC) for this production system
  • Always desirable: Self-documenting nature of the process

Overlapping areas of interest


  • The open source movement is another element of the change to human-oriented lightweight software development
  • The ways in which it differs relate to leadership and trust, and to IPR
  • The commonly-used open source development system (voluntary distributed communities) is not a precondition for free software
  • Free software is a social movement independent of open source