Free/open source programming, and software engineering

by Sebastian Rahtz on 1 August 2003

Introduction

Free/open source programming, and software engineering Sebastian Rahtz Information Manager Oxford University Computing Services (also JISC Open Source Software Advisory Service) August 2003

Questions

What is free/open source software?
What is software engineering?
Does free/open source imply a mode of programming?
Do free/open source methodologies develop from, turn into, or co-exist with changes in software engineering?
What is the future for computer programming?

What is Free/Open Source Software?

Software for which:

the source code is available to the end-user;
the source code can be modified by the end-user;
the licensing conditions are intended to facilitate continued re-use and wide availability of the software, in both commercial and non-commercial contexts;
the cost of acquisition to the end-user is often minimal.Open Source is a development methodology; Free Software is a social movement.

Virtues of free and open source software?

has no secrets: the innards are available for anyone to inspect
is not privately controlled: so likely to promote open rather than proprietary formats
is typically maintained by communities rather than corporations: so bug fixes and enhancement are often frequent and free
is usually distributed free of charge (developers make their money from support, training, and specialist add-ons; not marketing)

Clearing up misunderstandings

Free software uses the free from freedom, not the one from free beer. Open source software may or may not cost money
The cost of ownership often bears little relation to the cost of acquiring a piece of software
Public domain is something different. Open source software has a copyright holder and conditions of legal use
Open source software does not mandate exclusivity. You can use open source programs under Windows
People do not choose software solely on the basis of open source. Interoperability and open standards for data are equally important

Personalities

The father, high priest, best emissary, and bête noire, of free software is Richard Stallman
The man who made it mainstream by creating Linux, the Gandhi of Finland, is Linus Torvalds

The cathedral and the bazaar

F/OSS projects tend to be classified into two extreme camps:

A genius programmer creates something which others like. (S)he controls pace and scope of future development.
A group of people get together to fill a gap and take on different tasks. As time goes by, some drift away and others join in. There is always someone to take over. Most projects fall in between. Most of them have a recognised leader, to whom other members defer. A gentle system of leadership challenge and deposition assures the health of the herd is kept up. (see Garcia and Steinmueller, paper 92, for much more on this)

F/OSS groupstyle creation characteristics (1)

Free and open source software tends to have the following positive development characteristics:

Programmer commitment, because the programmer is also the user
Rapid change, because programmers want to see results
Unconstrained specifications, because there is no external client
Collective ownership of code
Response to change, dictated by (perhaps unexpected) users

F/OSS groupstyle creation characteristics (2)

… and the following negative characteristics:

Unrealistic expectations, because there is no client controlling the specification
Uneven development rates, because programmers work on what they want
Flat managerial tree, because programmers are not working to contract, so no control or direction
Changing resources (no guarentee of work hours)
Possibly conflicting goals or aspirations
It does not have to succeed_a bit like a rock band?_

F/OSS groupstyle creation characteristics (3)

But the following do not necessarily apply:

Poor quality code: likely or as unlikely as in traditional programming
Slow development: depends on who takes an interest and how hard they work
It isn’t as polished as commercial software: much commercial software is riddled with problems
No-one supports it: most projects have a large group of “hangers on” who do not directly develop, but do help others

Software development vocabulary

a process is defined that has milestones
activities are planned, to be carried out by teams
teams are formed by people who take roles and have skills
activities lead to products by the use of techniques
techniques can be supported by tools
products have to adhere to standards
products have to satisfy quality benchmarks

Sofware methodologies describe

Methodology size and weight
Ceremony, the amount of precision, tighteness, tolerance
Problem size
System criticality
Stability, the likelihood of change

Traditional software development

Relies on sequential phases of development; it makes several dangerous assumptions:

the existence of a set of requirements that is defined beforehand and is maintained unchanged during development;
the assumption that all code is designed from scratch, making no allowance for the reuse of existing software components
no repeating previous phases of the development without having to repeat the whole sequence of steps for the entirety of the system.
development as a linear process.

Iterative and incremental development methods

Incremental development:

proceeds from an initial subset of the requirements to more and more complete subsets, until the whole system is addressed
partitions the desired functionality
supports parallel development The final product results from the total integration of the partitions.

Lighter methodologies

Over the last 3-4 years, software theorists have tried to deal with issues such as:

Software must change more and more quickly these days
Software keep being delivered late
The customer is not involved with what she is buying
Programmers lack motivation
We need new software all the time The industry needed the equivalent of fast food: predictable quality delivered quickly.

A methodology: Extreme Programming

The Planning Game
Small Releases
System Metaphor (a story)
Simple Design
Continuous Testing
Refactoring
Pair Programming
Collective Code Ownership
Continuous Integration
40-Hour Work Week
On-site Customer
Coding Standards

Extreme: the details

The Planning Game: Business and development cooperate to produce the maximum business value as rapidly as possible. The planning game happens at various scales, but the basic rules are always the same:

Business comes up with a list of desired features for the system. Each feature is written out as a User Story, which gives the feature a name, and describes in broad strokes what is required. User stories are typically written on 4x6 cards.
Development estimates how much effort each story will take, and how much effort the team can produce in a given time interval (the iteration).
Business then decides which stories to implement in what order, as well as when and how often to produce a production releases of the system.

Extreme: the details (2)

Small Releases: Start with the smallest useful feature set. Release early and often, adding a few features each time.
System Metaphor: Each project has an organizing metaphor, which provides an easy to remember naming convention.
Simple Design: Always use the simplest possible design that gets the job done. The requirements will change tomorrow, so only do what’s needed to meet today’s requirements.

Extreme: the details (3)

Continuous Testing: Before programmers add a feature, they write a test for it. When the suite runs, the job is done. Tests in XP come in two basic flavors.

Unit Tests are automated tests written by the developers to test functionality as they write it. Each unit test typically tests only a single class, or a small cluster of classes. Unit tests are typically written using a unit testing framework, such as JUnit.
Acceptance Tests (also known as Functional Tests) are specified by the customer to test that the overall system is functioning as specified. Acceptance tests typically test the entire system, or some large chunk of it. When all the acceptance tests pass for a given user story, that story is considered complete. At the very least, an acceptance test could consist of a script of user interface actions and expected results that a human can run. Ideally acceptance tests should be automated, either using the unit testing framework, or a separate acceptance testing framework.

Refactoring: Refactor out any duplicate code generated in a coding session. You can do this with confidence that you didn’t break anything because you have the tests.

Extreme: the details (4)

Pair Programming: All production code is written by two programmers sitting at one machine. Essentially, all code is reviewed as it is written.
Collective Code Ownership: No single person “owns” a module. Any developer is expect to be able to work on any part of the codebase at any time.
Continuous Integration: All changes are integrated into the codebase at least daily. The tests have to run 100% both before and after integration.
40-Hour Work Week: Programmers go home on time. In crunch mode, up to one week of overtime is allowed. But multiple consecutive weeks of overtime are treated as a sign that something is very wrong with the process.

Extreme: the details (5)

On-site Customer: Development team has continuous access to a real live customer, that is, someone who will actually be using the system. For commercial software with lots of customers, a customer proxy (usually the product manager) is used instead.
Coding Standards: Everyone codes to the same standards. Ideally, you shouldn’t be able to tell by looking at it who on the team has touched a specific piece of code.

A movement: Agile

Agile is a conscious revolutionary movement: http://agilemanifesto.org/. It prefers:

Individuals	over	processes and tools
Working software	over	full documentation
Customer collaboration	over	contract negotiation
Responding to change	over	following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Agile software engineering

Traditional engineering | Agile engineering

The Agile Manifesto

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The Agile Manifesto (2)

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances agility.
Simplicity—the art of maximizing the amount of work not done—is essential.
The best architectures, requirements, and designs emerge from self-organizing teams.
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

What does open source add to the mix?

Some open source projects make a virtue of:

Space, time and culture separation of team members
Semi-professional programmers having fun
Publicity and (friendly) competition
Co-operating systems (supply of components)
Experience with virtual communities
Promotion of open standards
Allowance of failure Others are traditional programming teams in big companies.

What does free software add to the mix?

Free as in beer
Concept of the public good
Identification of software as unpatentable human heritage
Anti-establishment ethos
Missionary conversion of others

Open standards meets free/open source?

Which is better?

Commercial software which uses an XML data format and can be accessed using web service protocolsor
An free/open source program which uses its own binary data format, its own interface, and its own programming language Open data and open communication between different components of your IT system gives you better interchangeability of components.

Sociological perspectives on F/OSS

An essential characteristic: Novel use of IPR to distribute/publish software under public domain-like conditions
Almost essential: _Distributed mode of creating (producing) an information-good: software _
Common but not necessary: Extensive voluntary participation by communities of skilled and neophyte software developers
Likely: Dependence of the production mode upon the non-proprietary distribution regime
True across the board: Critical role of computer-mediated communications (CMC) for this production system
Always desirable: Self-documenting nature of the process

Overlapping areas of interest

Conclusions

The open source movement is another element of the change to human-oriented lightweight software development
The ways in which it differs relate to leadership and trust, and to IPR
The commonly-used open source development system (voluntary distributed communities) is not a precondition for free software
Free software is a social movement independent of open source