SANE 2004 Conference Report

by Ray Miller on 1 October 2004 , last updated 23 September 2008

Introduction

September 30th 2004 saw the opening of the 4th International System Administration and Network Engineering Conference (SANE) at Amsterdam’s RAI conference centre. The conference was organized by the Netherlands UNIX User Group (NLUUG), co-sponsored by Stichting NLnet, with cooperation from USENIX, the Advanced Computing Systems Association.

A SANE conference has been held every two years since the first was organized in 1998 …to strengthen the European ties between the National UNIX User Groups and their members, in the spirit of the former EUUG/EurOpen. I had attended SANE 2000, held in Maastricht, so was delighted to receive an invitation from NLUUG to represent UKUUG at SANE 2004.

The conference itself was preceded by three days of tutorials - a very strong programme with five parallel streams throughout the three days. Topics ranged from networking (IPv6, firewalls, wireless, IP telephony), through operating systems (FreeBSD 5.2 code walkthrough, Linux 2.6 process management), to popular applications (MySQL, Postfix, OpenLDAP, Samba). Every SANE conference has also featured a Black Hats Session, which is obviously popular: this year’s (Black Hats Session IV: Developments in Security) was run on Monday and repeated on Tuesday.

Work pressures prevented me from attending the tutorials, but I arrived at RAI on Wednesday evening just as Richard Stallman was finishing his presentation, The Danger of Software Patents. Stallman had travelled to Amsterdam earlier in the day and joined in the demonstration for innovation without software patents held in Amsterdam’s Dam Square. The demonstration was organized to coincide with a high-level EU conference on future ICT policy in Europe (initiated by the Dutch government in their 2004 Presidency of the EU), also being held in Amsterdam. Enough politics (for now).

Wednesday evening also saw the SANE Free Software Bazaar, a free event open to non-delegates. Here you could meet and chat informally with developers from the Debian project, OpenBSD, FreeBSD, CAcert, and many others. Birds-of-a-feather sessions covering Samba, KDE, MMBase, KeyWorx, and VIM were also held on Wednesday evening.

The conference proper started on Thursday morning with a keynote by Paul Kilmartin of eBay, Inc, eBay through the eyes of the Systems Administrator. This was a very interesting talk about the challenges of managing the IT infrastructure behind a (rapidly) growing company, where downtime means losing real money (eBay currently transacts business worth more than USD 1000/second). The most important point I came away with was this: when you are planning for high availability, you do not want to be at the bleeding edge, you want to be doing what other HA sites are doing. Unfortunately for eBay, this is not always possible: they are, after all, one of the world’s largest online retailers.

Another important point from Kilmartin’s talk was that they are never under the illusion of having solved a problem: while a new system might handle today’s workload, eBay’s growth is such that the lifetime of any solution is strictly limited. Kilmartin ended his talk with a section entitled Why I Hate Vendors. Anyone who has dealt with a vendor support desk more interested in closing a trouble ticket than actually solving a problem will have a lot of sympathy with him.

After the keynote, the conference split into two streams: refereed papers, and invited speakers. I stayed with the invited speakers for the rest of the morning.

The first of these was Arjen Lentz of MySQL AB, with MySQL Roadmap - What we have now and where we are heading. He covered some history of the MySQL project, their development procedures and release schedule, and MySQL’s current (and planned) features. Whenever he was talking about a feature, he said a few words about the developer behind it: their background, where they are in the world, and how they came to be involved with the project. This added a personal dimension to what might otherwise have been a dull list of features, and also emphasized the global bazaar nature of MySQL development.

Next was Wietse Venema’s Open Source Security Lessons. He began his talk with some history, taking us back to the time when Eindhoven University in the Netherlands was first connected to the Internet. One unofficial user of their systems was causing problems for system administrators: they cleaned up after their activities with rm -rf /. In an effort to track down this intruder, Venema wrote the first version of what we now know as TCP wrappers.

He went on to talk about the press response to his and Dan Farmer’s release of SATAN, the network security vulnerability scanner: It’s like distributing high-powered rocket launchers throughout the world, free of charge, available at your local library or school (San Jose Mercury). As it turned out, the release of SATAN did not result in an increase in reports of computer break-in activity, and SATAN proved a useful addition to the system administrator’s toolbox for many years.

He then talked about Postfix, and the role its release had in bringing open source software to the attention of IBM’s senior management. Finally, he came to the debate about open versus closed source software and security, where he thinks the protagonists are missing the point: …when a system isn’t built to be secure, then it will be like Swiss cheese no matter how many security patches you apply. He pointed out that this is not a new insight, and quoted a 30-year-old paper saying essentially the same thing.

After lunch, I moved to the other lecture room for the refereed papers: Lambda Networking in NetherLight by Erik Radius of SURFnet; then Traffic shaping for large-scale web services by Angelos Varvitsiotis of the Greek Research and Technology Network.

The first of these was a technical talk about using different wavelengths of light (lambdas) to transmit multiple data channels over a single optical fibre (dense wavelength division multiplexing). As well as the technical aspects, Radius talked about NetherLight’s global connectivity (which includes StarLight in Chicago, and UKLight in London), and potential uses for the technology (for example, high-bandwidth GRID computing).

From high bandwidth to low: Varvitsiotis’s talk was about traffic shaping for web servers with an uplink bottleneck. He used an Apache module, mod_mimetos, to set the IP type-of-service value according to the MIME type, file size, directory, etc. of the content being delivered, in conjunction with a class-based queuing (CBQ) scheme and a set of filters to map ToS values to particular queues, implemented using the Linux kernel’s advanced routing and traffic control mechanisms. He also updated Apache’s mod_mime_magic module to bring it into line with the latest file code.

Varvitsiotis then used data gathered from his University’s cache logs to generate driver data for a simulation, and ran different workloads against an uplink-throttled web server. The results of these experiments are detailed in his paper.

The next refereed paper, TCG 1.2 - fair play with the Fritz chip?, was presented by Rudiger Weis of Vrije University. This was an entertaining (but nevertheless worrying) look at the latest proposal from Microsoft and other members of the Trusted Computing Group (TCG).

The concept of trusted computing is to place an especially trusted observer, or Fritz chip, into information-handling devices, to prevent even the device owner from carrying out certain operations: the owner gives up some control of their device in return for the ability to verify a device’s trustworthiness.

While the proposed architecture will offer only limited protection against worms and viruses, it offers a lot of features that can be used to protect a personal computer against its owner, especially in the field of Digital Restrictions Management (in the words of Ron Rivest, …you are putting a virtual set-top box inside your PC. You are essentially renting out part of your PC to people you may not trust).

Cryptographers and privacy organizations have pressurized the TCG into modifying their proposals, and the recent TCG 1.2 specification does address some of their concerns. There are, however, still worries about backdoors, potential compatibility problems between Trusted Computing and Free (GPL-licensed) Software, and patent issues (an official Microsoft statement reads …Much of the next-generation secure computing base architecture design is covered by patents, and there will be intellectual property issues to be resolved. It is too early to speculate on how those issues might be addressed.).

The final talk of the day was a choice between the invited speaker, John Nelson on Special Effects on the Movie ‘I, Robot’, and Clifford Wolf’s refereed paper on Distributed Software Development using Subversion and SubMaster. I opted for the latter.

Some of you will already know Clifford Wolf as the project leader for ROCK Linux. Just over a year ago, the ROCK Linux project decided to switch from CVS to Subversion. In the first half of his talk, Wolf covered the basics of revision control systems and introduced Subversion itself. He then moved on to discuss SubMaster, and it was here that the talk started to get interesting.

Like CVS, Subversion is a centralized revision control system, where only privileged project members have commit access to a central repository. Other developers must submit patches via a mailing list, where they can easily be overlooked.

SubMaster, developed by the ROCK Linux project, is an attempt to address this problem and provide for a distributed development model. SubMaster provides scripts that make it easy for developers to create and manage their own branches (in their own local Subversion repository), keep them synchronized with the central repository, and send patches upstream. It also provides a CGI script to manage patch submission, collect feedback, make regression tests, and apply patches to the main tree.

But a conference is about more than just technical talks, and SANE is no exception. There are opportunities to chat informally with peers during the refreshment breaks, but there’s nothing like being thrown together on a boat with an unlimited supply of beer to break the ice.

The SANE 2004 social event on Thursday evening began as something of a mystery tour, with three bendy busses setting off across the city, attempting a three-point turn on a dual carriageway, then dropping us in the middle of nowhere. After a short walk through a residential then industrial area, we arrived at a boat yard and boarded a boat for the evening’s cruise. Entertainment was provided by the Bucket Big Band (I counted seven saxophones, a clarinet, trombone, two trumpets, two guitars, a drummer, and a very energetic conductor). As well as unlimited drinks, a buffet provided plenty of Indonesian food, making for a very enjoyable evening. Better still, by the time we docked, the bus drivers had found the boat yard, so there was no need to repeat the walk.

The first invited speaker on Friday morning was Geoff Halprin of The SysAdmin Group, with The Changing Face of System Administration. Halprin discussed the challenges facing modern-day system administrators and the often conflicting priorities: troubleshooting, user support, infrastructure projects, keeping our skills up-to-date. He stressed the importance (to system administrators as well as managers) of measuring how much time is spent on each of task, and of maintaining the correct balance (learning and infrastructure projects should not lose out to short-term objectives).

I switched to the refereed papers stream for the next two talks, High Available Loadsharing with OpenBSD by Marco Pfatschbacher, then Deployment of Worldwide IDS Networks by Matthias Hofherr. Both of these speakers work for GeNUA mbH, a German IT security consultancy.

Pfatschbacher presented a paper describing work carried out as part of his diploma thesis about High Availability VPNs. In a traditional load balancing setup, the load balancer is a single point of failure unless a second, redundant, load balancer is introduced. As with many HA solutions, this introduces extra complexity. Pfatschbacher came up with a nifty idea to provide HA and load balancing without this complexity.

He implemented a new kind of network interface in OpenBSD, a virtual Ethernet interface, or veif. The veif can be assigned an arbitrary MAC address, effectively providing two network interface cards in one. Thus two hosts on the same network can share a common MAC and IP address without changing the MAC addresses of their physical interfaces. Each host remains individually addressable, while packets sent to the common address are seen by both hosts.

Of course, this presents problems on a switched network, so his next trick is to make a switch behave like a hub. To achieve this, veif never sends any packets with its virtual MAC as a source address (think proxy ARP), so the switch never learns the whereabouts of the common MAC address.

The next step is to ensure that, although all packets are seen by both hosts, each packet is only processed by one host. Pfatschbacher introduced an option to OpenBSD’s pf to filter packets based on a hash of the source and destination IP addresses and ports. One host is configured to drop all packets in one half of the hash space, and the other host to drop all packets in the opposite half.

OpenBSD 3.5 introduced support for CARP (Common Address Redundancy Protocol), which utilizes virtual MAC addresses to enable multiple machines on the same local network to share a set of IP addresses, while ensuring that these addresses are always available. Pfatschbacher used CARP for monitor and failover of the pf-hash configuration: if one host fails, its hash range is migrated to one of the remaining CARP hosts.

In the next talk, Deployment of worldwide IDS networks, Hofherr presented a case study featuring a fictional company, BigCorp, who wanted to employ a network intrusion detection system in their offices across the globe.

Hofherr described a hierarchical solution, with IDS sensors analyzing traffic and generating alerts that are fed upstream to a Central. The sensors and the central communicate over a dedicated management network, both to lessen the burden on the production network, and to reduce the likelihood of an attacker analyzing the IDS data. The solution was based on the open source IDS Snort, with a central server running PostgreSQL. Administration is over https to an Apache server, using client certificates for authentication.

Hofherr discussed the different possibilities for traffic capture, their chosen solution (Ethernet Tap devices), the problems this introduced for Snort (and how they solved them), and the protocol for communication between the sensors and central servers. He also discussed security, availability, and monitoring of the IDS infrastructure itself.

He concludes that, although installation of a single network intrusion detection system is well understood and documented, implementing a distributed IDS presents new problems. While there are no out-of-the-box open source solutions, the software components do exist and the challenge is in coming up with a robust, secure, and conclusive design.

A meeting of national Unix User Group board members had been called for Friday lunchtime. The Netherlands (NLUUG), Norway (NUUG), Denmark (DKUUG), United Kingdom (UKUUG), and Croatia (HrOpen) were all represented here. Discussion focused on how the national groups might work together, for example, reciprocal agreements enabling members to attend national UUG events at the local members’ rate. DKUUG is planning to revitalize the defunct EUUG/EurOpen and put the content of old EUUG magazines online, and NUUG has digital video footage of some of its talks available.

It was interesting to meet with the other UUG board members and to see the common challenges we are facing. The meeting engendered an excellent spirit of cooperation, and I came away feeling quite optimistic. The challenge remains in turning ideas into concrete actions, and following through on those actions.

I returned to the invited speakers for the remainder of the conference. This stream started off after lunch with a talk on Dutch Law Enforcement vs High Tech Crime by Pascal Hetzscholdt, a policy advisor to the Dutch National Police Agency. Hetzscholdt is currently involved in setting up a High Tech Crime Centre in the Netherlands.

He talked about the challenges faced by the police in tackling the new cyber crime, and the links between high tech crime (phishing, fraud) and organized gangs often involved in drug trafficing and arms trading. These links can make it hard to decide which agency should tackle the problem: fraud investigators, because of the financial aspects of phishing? cybercops for their technical expertise? drug enforcement agencies when the money is used for drug trafficing?

Fighting IT crime is not seen as a cool thing - sitting in front of a computer screen is not as exciting as a high-speed car chase. And shouldn’t priority be given to more shocking crimes like murder, rape, kidnapping? In the Netherlands, these priorities are decided by the public prosecutor who often does not recognize the significance of computer crime, but knows that it can be costly to find the IT expertise required to fight it.

Hetzscholdt appealed to the system administrators and Internet service providers in the audience for their help: the police need our expertise. But he was not given an easy time during audience questioning: many are unhappy with legal requirements imposed on ISPs to collect logs and data about their users activities and meet the costs of storing this for long periods of time.

Next came my favourite talk of the conference, Sjoera Nas of Bits of Freedom on The Multatuli Project: ISP Notice & Take Down. Under the European directive on electronic commerce, Internet service providers risk liability for hosting apparently illegal content from their customers. This is quite different from the situation in the United States, where the DMCA provides a safe harbour for service providers.

In 2003, three researchers from the Oxford Centre for Socio-Legal Studies conducted a small experiment with notice and take-down, to see if the different legal frameworks made any difference in practice. They published an article (an extract from John Stuart Mill’s On Liberty, about freedom of speech) on a homepage in the UK and one in the USA. This was clearly marked as dating from 1869, and belonging to the public domain.

They then sent a fake complaint to the two ISPs, using an anonymous Hotmail address. The UK provider removed the homepage within 24 hours, while the US provider insisted that the complainant declare they were acting in good faith (this is one of the safe harbour provisions in the DMCA). Not wanting to risk the next (fraudulent) step, the researchers stopped there.

Bits of Freedom organized a similar experiment this summer, involving ten Dutch ISPs. They uploaded some text by the famous author Multatuli (Eduard Douwes Dekker), dating from 1871. Again, their homepage clearly attributed the text and stated that it was in the public domain.

Seven of the ten providers took down the homepage, one within 3 hours of receiving the fake complaint. Only one provider showed any distrust about the origin of the complaint, and only one demonstrated that they had actually looked at the page in question. In one case, the customer was not even informed of the complaint, and in another, the customer’s personal details were forwarded to the complainant. Two of the ISPs did not reply at to the email sent to their official abuse addresses.

Nas concludes It only takes a Hotmail account to bring a website down, and freedom of speech stands no chance in front of the Texan-style private ISP justice.

The final talk of the conference was by Peter H. Salus, the famous USENIX bookworm. His talk UNIX and the ARPAnet/Internet at 35; Linux a teenager; still in court, gave a historical perspective on the SCO Group’s attack on Linux through the court system. Salus interspersed his many slides of penguin photos with copies of legal documents from the SCO Group court cases, giving a light-hearted view of the proceedings.

Throughout the conference, more than a dozen technical posters were on display in the lobby: an alternative method for authentication, authorization and accounting for Windows 2000/XP systems; PPTP must die; CAcert; and more. The prize for best poster was awarded to John Borwick of Wake Forest University for his poster on LDAP for Systems and Network Engineering. This described a method for storing DNS and DHCP configuration data in an LDAP database, and using Perl scripts to retrieve the data and generate configuration files.

There was also a prize for best paper, which was awarded to Luca Deri for his paper Improving Passive Packet Capture: Beyond Device Polling. Deri proposes a new approach to passive packet capture which, combined with device polling, allows packets to be captured and analyzed at (almost) wire speed on Gbit networks using a legacy PC.

After presentation of the prizes and thank you’s to the many volunteers who helped to make the conference run so smoothly, Quiz Master Kevlin Henney took over with the inSANE quiz. Two teams were drawn completely at random from the business cards solicited earlier in the day, and pitted against each other and the Quiz Master’s completely fair scoring.

You really had to know your geek culture to do well in this quiz - but that alone was not enough. There was audience participation too, with each team having to guess how the audience would respond to yellow or green questions. For example, the Quiz Master would shout Yellow - Python, Green - Perl, the teams would have to write down their answers (yellow or green) before the audience voted by holding coloured cards in the air.

After one team had been eliminated, the three members of the remaining team contended with each other for prizes of books, posters and T-shirts. The quiz was a fun way to end a very enjoyable conference.

I was impressed both by the professionalism of the organization, the quality of the talks, and the smooth running of the event. RAI offered excellent facilities, and the organizers had provided wireless networking throughout the conference area, as well as a terminal room with Internet access for those of us traveling without laptops.

Congratulations, NLUUG, on another excellent conference! I am looking forward already to SANE 2006, and heartily recommend it to anyone else with an interest in network or system administration. You can find out more about past and future SANE conferences at http://www.sane.nl/.

Ray Miller is a director of UKUUG, the UK’s Unix and Open Systems User Group, and Chairman of UKUUG Council. He works as a Unix Systems Programmer at the University of Oxford, where he leads the Systems Development and Support team in the University’s Computing Services.

This article is available on the author’s home page at http://users.ox.ac.uk/~raym/writing/sane2004.html.