layout: article title: “Free/open source programming for the JISC”

date: 2004-05-01 author: Stuart Yeates and Sebastian Rahtz permalink: jiscoss.html firstpub: 2004-05-01 reviewed: status: live —Free/open source programming for the JISCStuart YeatesSebastian Rahtz JISC Open Source Software Advisory ServiceMay 2004

Our programme today

Let’s talk about:

• Why open source is undermining decent software
• Why open source folks just steal good ideas and make monkey copies
• Why open source software is full of bugs and security loopholes
• Why it is always best go with a single large provider of software
• Why Microsoft Word is a standard
• Why you can’t trust stuff written by Russian teenagers

The real programme for the afternoon

We want to:

• Tell you what OSS Watch does
• Talk about open source in theory
• Discuss practical decisions like licensing, repositories etc
• (most importantly) facilitate your discussion about working plans for the coming months

Depending on your viewpoint

Open source is

• Just politics, it’ll go away
• A legal and cultural affair, which will take years to follow through
• A matter of software engineering
• Just trivia for sandal-wearing bigots to worry about

What was that about OSS Watch?

We are here to advise UK HE/FE about issues around open source software:

• funded by JISC for two years from 1st July 2003
• part of JISC’s Information Environment
• working in partnership with other JISC services, eg CETIS and the Mirror Service
• serving FE and HE equally (not just research universities)
• visible at http://www.oss-watch.ac.uk We have a staff of 1.25 FTE, comprising one manager (0.25), one communicator (0.5) and two researchers (0.2 and 0.3).

What OSS Watch does

• Offers a neutral and practical web site
• Runs at least two open meetings a year
• Runs two focus groups year, and writes analyses
• Engages in understanding institutional processes
• Advises IT managers, project developers, and users
• Gives advice on open source at any UK/FE forum — and makes all its material available under the GNU Free Documentation License

What OSS Watch does notdo

• Try to persuade people to adopt open source
• Run a software repository
• Help people with their Open Office problems
• Compete with FLOSSIE, freshmeat, slashdot etc
• Provide definitive legal advice
• Be a debating forum for hairy sandal-wearing geeks

Conference: Support and training

A one day conference to be held on June 3rd in London, to look at institutional commitment to open source software in the back office

• What are the support models for OSS?
• How do commercial vendors work with OSS?
• How can institutions combine together to support each other?
• What is the impact on staff training needs? This matters to you! You have to think about future support plans!

Study: Interoperability and migration

A workshop to be held in September, with the aim of writing a road map for deployment

• Which types of software can work in a mixed economy?
• What is the impact on legacy data?
• What is the cost of migration?
• Do open standardsmean anything? This matters to you! Your software will not be working in a vacuum!

Conference: the OSS Watch institutional briefing

A conference to be held in December at which we plan to discuss institutional IT strategies and how they may be impacted by government and international developments.

Questions we might answer today

• What is free/open source software?
• How does it relate to software engineering?
• Does free/open source imply a mode of programming?
• Do free/open source methodologies develop from, turn into, or co-exist with changes in software engineering?
• What work practices can help the JISC framework programme?
• What must the JISC mandate to get good results?

What is Free/Open Source Software?

Software for which:

• the source code is available to the end-user;
• the source code can be modified by the end-user;
• there are no restrictions on redistribution or use;
• the licensing conditions are usually intended to _facilitate _ continued re-use and wide availability of the software, in both commercial and non-commercial contexts;
• the cost of acquisition to the end-user is often minimal.Open Source is a development methodology; Free Software is a social movement.

Virtues of free and open source software?

• has no secrets: the innards are available for anyone to inspect
• is not privately controlled: so likely to promote open rather than proprietary formats
• is typically maintained by communities rather than corporations: so bug fixes and enhancement are often frequent and free
• is usually distributed free of charge (developers make their money from support, training, customisation and specialist add-ons; not marketing)

Open Source — The Bad News

• It is not a silver bullet
• Software engineering laws still apply—The Mythical Man Month is still in print!
• It doesn’t guarantee hoards of OS hackers to help you
• It doesn’t guarantee defect-free code

Open Source — The Good News

• It can reduce the likelihood that software will be abandoned at the end of the project
• If it takes off, it reduces maintainance load on originators

• Non-source contributions are also likely:

  • translations (Welsh?)
  • documentation
  • FAQs
  • body of knowledge in the mailing lists

Clearing up misunderstandings

• Free software uses the freefrom freedom, not the one from free beer. Open source software may or may not cost money
• The cost of ownership often bears little relation to the cost of acquiring a piece of software
• Public domain is something different. Open source software has a copyright holder and conditions of legal use
• Open source software does not mandate exclusivity. You can use open source programs under Windows (TheOpenCD)
• People do not choose software solely on the basis of open source. Interoperability and open standards for data are equally important

An open source guidance framework

Some of the areas in which JISC needs to provide guidance:

• IPR Ownership
• Licensing
• Trademarks
• Software Dependencies
• Storage
• Quality assurance
• Version Control
• Sustainability
• Documentation
• Open Standards
• Software Development Methodology

1. IPR Ownership

Copyright ownership of software documentation, manuals, user interface and source code must be recorded, may be vested with a JISC-appointed body and should be released under an OSI-compliant open source licence. It must be licenced and publicly available at zero financial cost throughout UK higher and further education.

The ownership of code developed in joint projects must be established before work begins.

Software is property. In law, property belongs to your employer unless your contract explicitly says not. The documentation for your software is part of its design, and is property too.

Why divest yourself of IPR?

There are three good reasons for giving copyright to a neutral third party:

• Many projects are partnerships between several educational institutions and commercial companies. It is often hard to record where precisely property has been created.
• Small businesses or educational establishments do not have the resources to pursue legal claims. Don’t make claims you are not prepared to back up!
• The copyright owner is not bound by the license. A university can release version 1 under the GPL, but commercialize version 2. If the sources is notionally licensed from a third party, it keeps them honest.

2. Licensing

The three basic license models:

GPL

If you extend a GPL program, or create a composite with it, you cannot release your work except under the GPL and must release offer the source code to the full program to everyone you give the program to

LGPL

If you extend an LGPL program, as with GPL. If you create composites with LGPL code you must offer the source code of the LGPL portion, but may retain the source code to your portions

BSD

You must continue to display the copyright notice on any derived work

If a system is built of two parts communicating via SOAP, they’re considered separate programs.

Some software history

In the beginning (well, 30 years ago):

• Hardware cost serious money
• Operating system software was usually patched locally
• Application software was thin on the ground
• Users were highly computer-literate and were expected to develop their own programs so who cared about whether software was free?

…and then we had the PC revolution…

• Desktop computers had to be sold cheaply
• Ordinary people got computers
• Some clever folks wrote killer applications
• Bill Gates saw the value of a standard operating system and lo! a market was created and all was well in the world.

Well, almost

• people got greedy and wanted free software
• some big companies woke up to the value of the assets they had previously given away
• a community of amateur programmers was unleashed on the world
• a few people realized that there was a danger of being locked out of developments entirely

Red herrings

We had a curious interregnum where:

• Lots of people stole (pirated) software
• Some people tried selling on trust (shareware)
• Other people thought public domain would solve everything
• Yet other people thought a vague don’t use it to make money or war condition would be binding but now we have largely settled into a stable mixed economy.

Typical licensing issues

• you must (sometimes) redistribute the software intact with source code
• if you integrate it with your own system and then redistribute that, your software may covered by the license too
• most OSS licenses are perpetual but the author is not bound to release revised versions under the license
• if you do not submit patches back to the original, expect to have maintenance issues in the future, particularly when security patches are released
• your patches are copyrighted works in themselves, and licensing issues apply to them

The GNU license question

50-80% of OSS uses the GNU General Public License.

The GPL does not require you to release your modified version.

• You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it internally without ever releasing it outside the organization.
• If you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program’s users, under the GPL. Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.

GPL urban myths exploded

• You can sell GPLed programs with no problem
• You can sell support, training and certification for GPL programs
• The GPL does not apply to things you create with GPLed tools
• You can change the programs and keep the changes to yourself; so long as you don’t distribute them
• GPL software does not have any stamp of approval or follow any special standards
• The copyright owner is not bound by the license

Does the GPL limit you?

It limits you if plan to make money selling modified versions of GPL programs, or incorporate them into your own.

But many people make a living from GPLed software.

Companies mixing OSS with business

• MySQL
• RedHat and SuSE (suppliers of Linux)
• IBM
• Sun
• Novell
• Apple
• …and many small independent consultants All these see a place for licensing some software under open source terms as part of their business.

What about dual licensing?

Some companies offer their software under more than one license. MySQL is a good example. They are a reasonable-sized company with three main sources of revenue:

• Online support and subscription services sold globally over the MySQL.com website to all users of the MySQL server.
• Sales of commercial MySQL licenses to users and developers of software products and of products that contain software.
• Franchise of MySQL products and services under the MySQL brand to value-added partners. MySQL is available under GPL or commercial license

License conclusion

Use a simple, well-understood license. Unless you have a free lawyer, don’t try to write your own. We believe that the LGPL license is the best for toolkits, primarily because it involves an ongoing relationship between the original authors and the subsequent developer community.

The BSD-like licenses are more one-sided, allowing for unlimited use by everyone, but less protection for the community.

Do not try to limit your license to the UK educational community. You will cut yourself off from distributions and developers. You will not be open source.

3. Trademarks

The JISC will make no claim to trademarks. Projects are encouraged to trademark their brands to build their reputations, and provide a route to exploitation of their knowledge.

4. Dependencies

Projects must maintain an IPR register, listing all contributors to their software, who owns the copyright on contributions (differs between contractors and employees).

Projects must state what build-time and run-time dependencies software has, dependencies include the operating systems, compilers, development enviroments, web servers and similar software needed to build and or run the software. Projects should state which specific versions of the dependencies are required, the range tested against and the licences applicable to these dependencies.

5. Storage

All electronic software outputs must be archived in a repository with a written preservation stragegy. Source code should be stored in a version control system; where a version control system is not used the institution will provide an alternative method of performing due diligence on the source code.

6. Quality assurance

Projects which develop software are must have appropiate testing systems in place. Automated testing should be used. Implemented standards mustbe tested against. Peer evaluation and demonstrator contexts are necessary parts of testing.

7. Version Control

Version control must be used. Projects with more than 3 contributors or 2 institutions must use a version control system that identifies who made each change. The full change history must be contained and it must be archived as above.

8. Sustainability and communities

Projects must state whether they foresee the project continuing beyond the timespan of the funding and if so whom they see participating in the project. Projects are should engage with end users and other parties to encourage and build self-sustaining communities.

A Sense of Community

• What is the difference between licencing and community?
• The cathedral vs the bazaar
• The bazaar model engages user communities
• The cathedral model is best if you already know everything…

The cathedral and the bazaar

F/OSS projects tend to be classified into two extreme camps:

• A genius programmer creates something which others like. (S)he controls pace and scope of future development. The genius has a complete idea of what the finished software will look like. Changes to the design during the build process are very costly, but the resulting software very polished and uniform.
• A group of people get together to fill a gap and take on different tasks. As time goes by, some drift away and others join in. There is always someone to take over. Design is by consensus, with people working on topics that motivate them. Design changes are relatively cheap but with no over-arching the finished program can be patchy. Most projects fall in between. Most of them have a recognised leader, to whom other members defer. A gentle system of leadership challenge and deposition assures the health of the herd is kept up.(see Garcia and Steinmueller, paper 92, for much more on this)

cf http://www.oreilly.com/catalog/cathbazpaper/(Eric Raymond’s The Cathedral and the Bazaar)

Example 1: Apache

A charitable consortium which

• guards IPR
• sets technical standards
• encourages related projects which match their interests
• launders money The core Apache server does not get abused because it is in no-one’s interests to do so.

Example 2: Exim

An individual employed at Cambridge to do good stuff. He

• decides what and when to release
• accepts patches
• may or may not hand over to someone else one day

Example 3: MySQL

The classic dual-licensing system. The MySQL company

• own the code and the name
• release it under the GPL to anyone
• license it for money to people who want to embed it
• hire the people who understand the code best
• stream of new programmers come on board from working with the GPL version

Example 4: Bodington

The cottage industry example

• built up from personal code
• going into Sourceforge slowly
• not very well documented
• building a community post hoc

Example 5: OpenOffice

The classic big brother project

• Core is developed by Sun programmers
• XML format defined by OASIS
• Interested groups work on extensions
• Open sourceness largely theoretical. Free beer and open file formats is much more important!

Example 6: Text Encoding Initiative

OSS is not just about programming. The TEI defines guidelines for marking up text.

• Very traditional cathedral model with closed license
• Now switching to open source license (GPL) and public development on Sourceforge
• Facing a major cultural shift for all concerned The TEI move is largely driven by a desire to be distributed and used widely.

Example 7: Urchin

The JISC-funded open source project to do RSS aggregation

• Copyright (C) Higher Education Funding Council for England (HEFCE)
• Lives on Sourceforge
• Ongoing engagement with RDF research community
• Provides toolkit and some tools
• Waiting to be packaged…still nerd-oriented

Example 8: TeX

The story from hell. A very well-respected academic writes a typesetting system. You can do what you like with it, but if you call it TeX it must always produce the same result.

• the devil and deep blue sea: you can put a changed version out under another name, but then no-one knows what it is
• the owner of the copyright has no interest in change, but would not release copyright
• there is an accretion of associated software which has no overall direction
• users groups set standards for themselves but cannot enforce them But it remains in widespread use.

9. Documentation

Projects which use IRC, mailing lists, forums or similar media for coordinating software development, reporting bugs or answering user queries must archive these and should store them in a format readibly accessible to search engines. Documentation must be releaesed in a form free from (DRM) Digital Rights Management restrictions.

10. Open Standards

Projects should use open standards. Projects in fields with widely used or ubiquitous open standards must implement those standards as they apply to the project unless an explicit argument is made in a project bid as to why this is unsuitable.

Open standards and documentation

Open standards are not just for software interchange languages like SOAP or WSRP. They are also for

• Documentation — Word .doc files are not an open standard! Nor is RTF
• Web sites — banish that mal-HTML now
• Data, graphic, multimedia etc — remember you want everyone to access your material
• Accessibility — rules apply in your technical site too Put everything on the web where Google will find it (except for personal information).

11. Software Development

Projects should release incremental milestone releases of working but incomplete software. Projects must demonstrate their software working with third party software, data, documents, processes and/or environments, as appropriate. Projects should demonstrate third party software working with their software, data, documents, processes and/or environments, as appropriate. During demonstrations projects must make clear which components have been solely developed by the project, which the project has contributed to and are entirely third party. Demonstrations must be to independent peers, should be accessible on the web and may be freely downloadable or avaliable as live CDs.

Release early, release often, show people.

All open source programs are equal, but some are more open then others

There is no correlation between:

• How well defined the task is, and thus how precise the specification and the goals are
• What the license is and who owns the copyright
• Whether you develop in public with a crowd of people looking over your shoulder, or in a glass box
• How you gather a community of users
• Whether you’ll find other programmers building around you

F/OSS groupstyle creation characteristics (1)

Free and open source software tends to have the following positive development characteristics:

• Programmer commitment, because the programmer is also the user
• Rapid change, because programmers want to see results
• Unconstrained specifications, because there is no external client
• Collective ownership of code
• Response to change, dictated by (perhaps unexpected) users

F/OSS groupstyle creation characteristics (2)

… and the following negative characteristics:

• Unrealistic expectations, because there is no client controlling the specification
• Uneven development rates, because programmers work on what they want
• Flat managerial tree, because programmers are not working to contract, so no control or direction
• Changing resources (no guarentee of work hours)
• Possibly conflicting goals or aspirations
• It does not have to succeed_a bit like a rock band?_

F/OSS groupstyle creation characteristics (3)

But the following do not necessarily apply:

• Poor quality code: likely or as unlikely as in traditional programming
• Slow development: depends on who takes an interest and how hard they work
• It isn’t as polished as commercial software: much commercial software is riddled with problems
• No-one supports it: most projects have a large group of “hangers on” who do not directly develop, but do help others

Software development vocabulary

• a process is defined that has milestones
• activities are planned, to be carried out by teams
• teams are formed by people who take roles and have skills
• activities lead to products by the use of techniques
• techniques can be supported by tools
• products have to adhere to standards
• products have to satisfy quality benchmarks

Sofware methodologies describe

• Methodology size and weight
• Ceremony, the amount of precision, tighteness, tolerance
• Problem size
• System criticality
• Stability, the likelihood of change

Traditional software development

Relies on sequential phases of development; it makes several dangerous assumptions:

• the existence of a set of requirements that is defined beforehand and is maintained unchanged during development;
• the assumption that all code is designed from scratch, making no allowance for the reuse of existing software components
• no repeating previous phases of the development without having to repeat the whole sequence of steps for the entirety of the system.
• development as a linear process.

Iterative and incremental development methods

Incremental development:

• proceeds from an initial subset of the requirements to more and more complete subsets, until the whole system is addressed
• partitions the desired functionality
• supports parallel development The final product results from the total integration of the partitions.

Lighter methodologies

Over the last 3-4 years, software theorists have tried to deal with issues such as:

• Software must change more and more quickly these days
• Software keep being delivered late
• The customer is not involved with what she is buying
• Programmers lack motivation
• We need new software all the time The industry needed the equivalent of fast food: predictable quality delivered quickly.

A methodology: Extreme Programming

• The Planning Game
• Small Releases
• System Metaphor (a story)
• Simple Design
• Continuous Testing
• Refactoring
• Pair Programming
• Collective Code Ownership
• Continuous Integration
• 40-Hour Work Week
• On-site Customer
• Coding Standards

Extreme: the details

1. The Planning Game: Business and development cooperate to produce the maximum business value as rapidly as possible. The planning game happens at various scales, but the basic rules are always the same:

• Business comes up with a list of desired features for the system. Each feature is written out as a User Story, which gives the feature a name, and describes in broad strokes what is required. User stories are typically written on 4x6 cards.
• Development estimates how much effort each story will take, and how much effort the team can produce in a given time interval (the iteration).
• Business then decides which stories to implement in what order, as well as when and how often to produce a production releases of the system.

Extreme: the details (2)

1. Small Releases: Start with the smallest useful feature set. Release early and often, adding a few features each time.

2. System Metaphor: Each project has an organizing metaphor, which provides an easy to remember naming convention.

3. Simple Design: Always use the simplest possible design that gets the job done. The requirements will change tomorrow, so only do what’s needed to meet today’s requirements.

Extreme: the details (3)

1. Continuous Testing: Before programmers add a feature, they write a test for it. When the suite runs, the job is done. Tests in XP come in two basic flavors.

• Unit Tests are automated tests written by the developers to test functionality as they write it. Each unit test typically tests only a single class, or a small cluster of classes. Unit tests are typically written using a unit testing framework, such as JUnit.
• Acceptance Tests (also known as Functional Tests) are specified by the customer to test that the overall system is functioning as specified. Acceptance tests typically test the entire system, or some large chunk of it. When all the acceptance tests pass for a given user story, that story is considered complete. At the very least, an acceptance test could consist of a script of user interface actions and expected results that a human can run. Ideally acceptance tests should be automated, either using the unit testing framework, or a separate acceptance testing framework.

1. Refactoring: Refactor out any duplicate code generated in a coding session. You can do this with confidence that you didn’t break anything because you have the tests.

Extreme: the details (4)

1. Pair Programming: All production code is written by two programmers sitting at one machine. Essentially, all code is reviewed as it is written.

2. Collective Code Ownership: No single person owns a module. Any developer is expect to be able to work on any part of the codebase at any time.

3. Continuous Integration: All changes are integrated into the codebase at least daily. The tests have to run 100% both before and after integration.

4. 40-Hour Work Week: Programmers go home on time. In crunch mode, up to one week of overtime is allowed. But multiple consecutive weeks of overtime are treated as a sign that something is very wrong with the process.

Extreme: the details (5)

1. On-site Customer: Development team has continuous access to a real live customer, that is, someone who will actually be using the system. For commercial software with lots of customers, a customer proxy (usually the product manager) is used instead.

2. Coding Standards: Everyone codes to the same standards. Ideally, you shouldn’t be able to tell by looking at it who on the team has touched a specific piece of code.

A movement: Agile

Agile is a conscious revolutionary movement: http://agilemanifesto.org/. It prefers:

That is, while there is value in the items on the right, we value the items on the left more.

Agile software engineering

Traditional engineering | Agile engineering

passive client | active client minimal changes | encourage change process rules | developer rules long iterations | short iterations static process | dynamic process

The Agile Manifesto

• Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Business people and developers must work together daily throughout the project.
• Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The Agile Manifesto (2)

• The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
• Working software is the primary measure of progress.
• Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.
• Continuous attention to technical excellence and good design enhances agility.
• Simplicity—the art of maximizing the amount of work not done—is essential.
• The best architectures, requirements, and designs emerge from self-organizing teams.
• At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

What does open source add to the mix?

Some open source projects make a virtue of:

• Space, time and culture separation of team members
• Semi-professional programmers having fun
• Publicity and (friendly) competition
• Co-operating systems (supply of components)
• Experience with virtual communities
• Promotion of open standards
• Allowance of failure Others are traditional programming teams in big companies.

What does free software add to the mix?

• Free as in beer
• Concept of the public good
• Identification of software as unpatentable human heritage
• Anti-establishment ethos
• Missionary conversion of others

Open standards meets free/open source?

Which is better?

• Commercial software which uses an XML data format and can be accessed using web service protocolsor

• An free/open source program which uses its own binary data format, its own interface, and its own programming language Open data and open communication between different components of your IT system gives you better interchangeability of components.

Sociological perspectives on F/OSS

• An essential characteristic: Novel use of IPR to distribute/publish software under public domain-like conditions
• Almost essential: _Distributed mode of creating (producing) an information-good: software _
• Common but not necessary: Extensive voluntary participation by communities of skilled and neophyte software developers
• Likely: Dependence of the production mode upon the non-proprietary distribution regime
• True across the board: Critical role of computer-mediated communications (CMC) for this production system
• Always desirable: Self-documenting nature of the process

Overlapping areas of interest

So…

• The open source movement is another element of the change to human-oriented lightweight software development
• The ways in which it differs relate to leadership and trust, and to IPR
• The commonly-used open source development system (voluntary distributed communities) is not a precondition for free software
• Free software is a social movement independent of open source

Summary

IPR Ownership

Decide now. Not at the end.

Licensing

Use the LGPL if possible

Trademarks

Use your name to establish a presence

Software Dependencies

Know the implications of what you are doing

Storage

Due diligence matters

Quality assurance

Er, test it!

Version Control

Use version control. Always. For everything.

Sustainability

Identify with your community at the start

Documentation

Remember the totality of written material

Open Standards

Use them, that’s what they are there for. And don’t use Word to document things…

Software Development Methodology

Don’t lock your programmer in a glass box