http://www.sane.nl/sane2006/

September 30th 2004 saw the opening of the 4th International System
Administration and Network Engineering Conference (SANE) at Amsterdam's
RAI conference centre. The conference was organized by the Netherlands
UNIX User Group (NLUUG), co-sponsored by Stichting NLnet, with
cooperation from USENIX, the Advanced Computing Systems Association.

A SANE conference has been held every two years since the first was
organized in 1998 '...to strengthen the European ties between the
National UNIX User Groups and their members', in the spirit of the
former EUUG/EurOpen. I had attended SANE 2000, held in Maastricht, so
was delighted to receive an invitation from NLUUG to represent UKUUG at
SANE 2004.

The conference itself was preceded by three days of tutorials - a very
strong programme with five parallel streams throughout the three days.
Topics ranged from networking (IPv6, firewalls, wireless, IP
telephony), through operating systems (FreeBSD 5.2 code walkthrough,
Linux 2.6 process management), to popular applications (MySQL, Postfix,
OpenLDAP, Samba). Every SANE conference has also featured a Black Hats
Session, which is obviously popular: this year's ('Black Hats Session
IV: Developments in Security') was run on Monday and repeated on
Tuesday.

Work pressures prevented me from attending the tutorials, but I arrived
at RAI on Wednesday evening just as Richard Stallman was finishing his
presentation, 'The Danger of Software Patents'. Stallman had
travelled to Amsterdam earlier in the day and joined in the
demonstration for innovation without software patents held in
Amsterdam's Dam Square. The demonstration was organized to coincide
with a high-level EU conference on future ICT policy in Europe
(initiated by the Dutch government in their 2004 Presidency of the EU),
also being held in Amsterdam. Enough politics (for now).



Wednesday evening also saw the SANE Free Software Bazaar, a free event
open to non-delegates. Here you could meet and chat informally with
developers from the Debian project, OpenBSD, FreeBSD, CAcert, and many
others. Birds-of-a-feather sessions covering Samba, KDE, MMBase,
KeyWorx, and VIM were also held on Wednesday evening.

The conference proper started on Thursday morning with a keynote by
Paul Kilmartin of eBay, Inc, 'eBay through the eyes of the Systems
Administrator'. This was a very interesting talk about the challenges
of managing the IT infrastructure behind a (rapidly) growing company,
where downtime means losing real money (eBay currently transacts
business worth more than USD 1000/second). The most important point I
came away with was this: when you are planning for high availability,
you do not want to be at the bleeding edge, you want to be doing what
other HA sites are doing. Unfortunately for eBay, this is not always
possible: they are, after all, one of the world's largest online
retailers.

Another important point from Kilmartin's talk was that they are never
under the illusion of having solved a problem: while a new system might
handle today's workload, eBay's growth is such that the lifetime of any
solution is strictly limited. Kilmartin ended his talk with a section
entitled 'Why I Hate Vendors'. Anyone who has dealt with a vendor
support desk more interested in closing a trouble ticket than actually
solving a problem will have a lot of sympathy with him.

After the keynote, the conference split into two streams: refereed
papers, and invited speakers. I stayed with the invited speakers for
the rest of the morning.

The first of these was Arjen Lentz of MySQL AB, with 'MySQL Roadmap -
What we have now and where we are heading'. He covered some history
of the MySQL project, their development procedures and release
schedule, and MySQL's current (and planned) features. Whenever he was
talking about a feature, he said a few words about the developer behind
it: their background, where they are in the world, and how they came to
be involved with the project. This added a personal dimension to what
might otherwise have been a dull list of features, and also emphasized
the global bazaar nature of MySQL development.

Next was Wietse Venema's 'Open Source Security Lessons'. He began
his talk with some history, taking us back to the time when Eindhoven
University in the Netherlands was first connected to the Internet. One
'unofficial' user of their systems was causing problems for system
administrators: they cleaned up after their activities with 'rm -rf
/'. In an effort to track down this intruder, Venema wrote the first
version of what we now know as 'TCP wrappers'.

He went on to talk about the press response to his and Dan Farmer's
release of SATAN, the network security vulnerability scanner: 'It's
like distributing high-powered rocket launchers throughout the world,
free of charge, available at your local library or school' (San Jose
Mercury). As it turned out, the release of SATAN did not result in an
increase in reports of computer break-in activity, and SATAN proved a
useful addition to the system administrator's toolbox for many years.

He then talked about Postfix, and the role its release had in bringing
open source software to the attention of IBM's senior management.
Finally, he came to the debate about open versus closed source software
and security, where he thinks the protagonists are missing the point:
'...when a system isn't built to be secure, then it will be like
Swiss cheese no matter how many security patches you apply'. He
pointed out that this is not a new insight, and quoted a 30-year-old
paper saying essentially the same thing.

After lunch, I moved to the other lecture room for the refereed papers:
'Lambda Networking in NetherLight' by Erik Radius of SURFnet; then
'Traffic shaping for large-scale web services' by Angelos
Varvitsiotis of the Greek Research and Technology Network.

The first of these was a technical talk about using different
wavelengths of light (lambdas) to transmit multiple data channels over
a single optical fibre (dense wavelength division multiplexing). As
well as the technical aspects, Radius talked about NetherLight's global
connectivity (which includes StarLight in Chicago, and UKLight in
London), and potential uses for the technology (for example,
high-bandwidth GRID computing).

>From high bandwidth to low: Varvitsiotis's talk was about traffic
shaping for web servers with an uplink bottleneck. He used an Apache
module, mod_mimetos, to set the IP type-of-service value according to
the MIME type, file size, directory, etc. of the content being
delivered, in conjunction with a class-based queuing (CBQ) scheme and a
set of filters to map ToS values to particular queues, implemented
using the Linux kernel's advanced routing and traffic control
mechanisms. He also updated Apache's mod_mime_magic module to bring it
into line with the latest 'file' code.

Varvitsiotis then used data gathered from his University's cache logs
to generate driver data for a simulation, and ran different workloads
against an uplink-throttled web server. The results of these
experiments are detailed in his paper.

The next refereed paper, 'TCG 1.2 - fair play with the Fritz
chip?', was presented by Rudiger Weis of Vrije University. This was
an entertaining (but nevertheless worrying) look at the latest proposal
from Microsoft and other members of the Trusted Computing Group (TCG).

The concept of trusted computing is to place an especially trusted
observer, or 'Fritz chip', into information-handling devices, to
prevent even the device owner from carrying out certain operations: the
owner gives up some control of their device in return for the ability
to verify a device's 'trustworthiness'.

While the proposed architecture will offer only limited protection
against worms and viruses, it offers a lot of features that can be used
to protect a personal computer against its owner, especially in the
field of Digital Restrictions Management (in the words of Ron Rivest,
'...you are putting a virtual set-top box inside your PC. You are
essentially renting out part of your PC to people you may not
trust').

Cryptographers and privacy organizations have pressurized the TCG into
modifying their proposals, and the recent TCG 1.2 specification does
address some of their concerns. There are, however, still worries about
backdoors, potential compatibility problems between Trusted Computing
and Free (GPL-licensed) Software, and patent issues (an official
Microsoft statement reads '...Much of the next-generation secure
computing base architecture design is covered by patents, and there
will be intellectual property issues to be resolved. It is too early to
speculate on how those issues might be addressed.').

The final talk of the day was a choice between the invited speaker,
John Nelson on 'Special Effects on the Movie 'I, Robot'', and
Clifford Wolf's refereed paper on 'Distributed Software Development
using Subversion and SubMaster'. I opted for the latter.

Some of you will already know Clifford Wolf as the project leader for
ROCK Linux. Just over a year ago, the ROCK Linux project decided to
switch from CVS to Subversion. In the first half of his talk, Wolf
covered the basics of revision control systems and introduced
Subversion itself. He then moved on to discuss SubMaster, and it was
here that the talk started to get interesting.

Like CVS, Subversion is a centralized revision control system, where
only privileged project members have commit access to a central
repository. Other developers must submit patches via a mailing list,
where they can easily be overlooked.

SubMaster, developed by the ROCK Linux project, is an attempt to
address this problem and provide for a distributed development model.
SubMaster provides scripts that make it easy for developers to create
and manage their own branches (in their own local Subversion
repository), keep them synchronized with the central repository, and
send patches upstream. It also provides a CGI script to manage patch
submission, collect feedback, make regression tests, and apply patches
to the main tree.

But a conference is about more than just technical talks, and SANE is
no exception. There are opportunities to chat informally with peers
during the refreshment breaks, but there's nothing like being thrown
together on a boat with an unlimited supply of beer to break the ice.

The SANE 2004 social event on Thursday evening began as something of a
mystery tour, with three 'bendy busses' setting off across the
city, attempting a three-point turn on a dual carriageway, then
dropping us in the middle of nowhere. After a short walk through a
residential then industrial area, we arrived at a boat yard and boarded
a boat for the evening's cruise. Entertainment was provided by the
Bucket Big Band (I counted seven saxophones, a clarinet, trombone, two
trumpets, two guitars, a drummer, and a very energetic conductor). As
well as unlimited drinks, a buffet provided plenty of Indonesian food,
making for a very enjoyable evening. Better still, by the time we
docked, the bus drivers had found the boat yard, so there was no need
to repeat the walk.

The first invited speaker on Friday morning was Geoff Halprin of The
SysAdmin Group, with 'The Changing Face of System Administration'.
Halprin discussed the challenges facing modern-day system
administrators and the often conflicting priorities: troubleshooting,
user support, infrastructure projects, keeping our skills up-to-date.
He stressed the importance (to system administrators as well as
managers) of measuring how much time is spent on each of task, and of
maintaining the correct balance (learning and infrastructure projects
should not lose out to short-term objectives).

I switched to the refereed papers stream for the next two talks,
'High Available Loadsharing with OpenBSD' by Marco Pfatschbacher,
then 'Deployment of Worldwide IDS Networks' by Matthias Hofherr.
Both of these speakers work for GeNUA mbH, a German IT security
consultancy.

Pfatschbacher presented a paper describing work carried out as part of
his diploma thesis about High Availability VPNs. In a traditional load
balancing setup, the load balancer is a single point of failure unless
a second, redundant, load balancer is introduced. As with many HA
solutions, this introduces extra complexity. Pfatschbacher came up with
a nifty idea to provide HA and load balancing without this complexity.

He implemented a new kind of network interface in OpenBSD, a virtual
Ethernet interface, or veif. The veif can be assigned an arbitrary MAC
address, effectively providing two network interface cards in one. Thus
two hosts on the same network can share a common MAC and IP address
without changing the MAC addresses of their physical interfaces. Each
host remains individually addressable, while packets sent to the common
address are seen by both hosts.

Of course, this presents problems on a switched network, so his next
trick is to make a switch behave like a hub. To achieve this, veif
never sends any packets with its virtual MAC as a source address (think
proxy ARP), so the switch never learns the whereabouts of the common
MAC address.

The next step is to ensure that, although all packets are seen by both
hosts, each packet is only processed by one host. Pfatschbacher
introduced an option to OpenBSD's pf to filter packets based on a hash
of the source and destination IP addresses and ports. One host is
configured to drop all packets in one half of the hash space, and the
other host to drop all packets in the opposite half.

OpenBSD 3.5 introduced support for CARP (Common Address Redundancy
Protocol), which utilizes virtual MAC addresses to enable multiple
machines on the same local network to share a set of IP addresses,
while ensuring that these addresses are always available. Pfatschbacher
used CARP for monitor and failover of the pf-hash configuration: if one
host fails, its hash range is migrated to one of the remaining CARP
hosts.

In the next talk, 'Deployment of worldwide IDS networks', Hofherr
presented a case study featuring a fictional company, BigCorp, who
wanted to employ a network intrusion detection system in their offices
across the globe.

Hofherr described a hierarchical solution, with IDS sensors analyzing
traffic and generating alerts that are fed upstream to a 'Central'.
The sensors and the central communicate over a dedicated management
network, both to lessen the burden on the production network, and to
reduce the likelihood of an attacker analyzing the IDS data. The
solution was based on the open source IDS Snort, with a central server
running PostgreSQL. Administration is over https to an Apache server,
using client certificates for authentication.

Hofherr discussed the different possibilities for traffic capture,
their chosen solution (Ethernet Tap devices), the problems this
introduced for Snort (and how they solved them), and the protocol for
communication between the sensors and central servers. He also
discussed security, availability, and monitoring of the IDS
infrastructure itself.

He concludes that, although installation of a single network intrusion
detection system is well understood and documented, implementing a
distributed IDS presents new problems. While there are no
out-of-the-box open source solutions, the software components do exist
and the challenge is in coming up with a robust, secure, and conclusive
design.

A meeting of national Unix User Group board members had been called for
Friday lunchtime. The Netherlands (NLUUG), Norway (NUUG), Denmark
(DKUUG), United Kingdom (UKUUG), and Croatia (HrOpen) were all
represented here. Discussion focused on how the national groups might
work together, for example, reciprocal agreements enabling members to
attend national UUG events at the local members' rate. DKUUG is
planning to revitalize the defunct EUUG/EurOpen and put the content of
old EUUG magazines online, and NUUG has digital video footage of some
of its talks available.

It was interesting to meet with the other UUG board members and to see
the common challenges we are facing. The meeting engendered an
excellent spirit of cooperation, and I came away feeling quite
optimistic. The challenge remains in turning ideas into concrete
actions, and following through on those actions.

I returned to the invited speakers for the remainder of the conference.
This stream started off after lunch with a talk on 'Dutch Law
Enforcement vs High Tech Crime' by Pascal Hetzscholdt, a policy
advisor to the Dutch National Police Agency. Hetzscholdt is currently
involved in setting up a High Tech Crime Centre in the Netherlands.

He talked about the challenges faced by the police in tackling the new
'cyber crime', and the links between high tech crime (phishing,
fraud) and organized gangs often involved in drug trafficing and arms
trading. These links can make it hard to decide which agency should
tackle the problem: fraud investigators, because of the financial
aspects of phishing? 'cybercops' for their technical expertise?
drug enforcement agencies when the money is used for drug trafficing?

Fighting IT crime is not seen as a 'cool thing' - sitting in front
of a computer screen is not as exciting as a high-speed car chase. And
shouldn't priority be given to more shocking crimes like murder, rape,
kidnapping? In the Netherlands, these priorities are decided by the
public prosecutor who often does not recognize the significance of
computer crime, but knows that it can be costly to find the IT
expertise required to fight it.

Hetzscholdt appealed to the system administrators and Internet service
providers in the audience for their help: the police need our
expertise. But he was not given an easy time during audience
questioning: many are unhappy with legal requirements imposed on ISPs
to collect logs and data about their users activities and meet the
costs of storing this for long periods of time.

Next came my favourite talk of the conference, Sjoera Nas of Bits of
Freedom on 'The Multatuli Project: ISP Notice & Take Down'. Under
the European directive on electronic commerce, Internet service
providers risk liability for hosting apparently illegal content from
their customers. This is quite different from the situation in the
United States, where the DMCA provides a safe harbour for service
providers.

In 2003, three researchers from the Oxford Centre for Socio-Legal
Studies conducted a small experiment with notice and take-down, to see
if the different legal frameworks made any difference in practice. They
published an article (an extract from John Stuart Mill's 'On
Liberty', about freedom of speech) on a homepage in the UK and one in
the USA. This was clearly marked as dating from 1869, and belonging to
the public domain.

They then sent a fake complaint to the two ISPs, using an anonymous
Hotmail address. The UK provider removed the homepage within 24 hours,
while the US provider insisted that the complainant declare they were
acting in good faith (this is one of the safe harbour provisions in the
DMCA). Not wanting to risk the next (fraudulent) step, the researchers
stopped there.

Bits of Freedom organized a similar experiment this summer, involving
ten Dutch ISPs. They uploaded some text by the famous author Multatuli
(Eduard Douwes Dekker), dating from 1871. Again, their homepage clearly
attributed the text and stated that it was in the public domain.

Seven of the ten providers took down the homepage, one within 3 hours
of receiving the fake complaint. Only one provider showed any distrust
about the origin of the complaint, and only one demonstrated that they
had actually looked at the page in question. In one case, the customer
was not even informed of the complaint, and in another, the customer's
personal details were forwarded to the complainant. Two of the ISPs did
not reply at to the email sent to their official abuse addresses.

Nas concludes 'It only takes a Hotmail account to bring a website
down, and freedom of speech stands no chance in front of the
Texan-style private ISP justice'.

The final talk of the conference was by Peter H. Salus, the famous
USENIX bookworm. His talk 'UNIX and the ARPAnet/Internet at 35; Linux
a teenager; still in court', gave a historical perspective on the SCO
Group's attack on Linux through the court system. Salus interspersed
his many slides of penguin photos with copies of legal documents from
the SCO Group court cases, giving a light-hearted view of the
proceedings.

Throughout the conference, more than a dozen technical posters were on
display in the lobby: an alternative method for authentication,
authorization and accounting for Windows 2000/XP systems; PPTP must
die; CAcert; and more. The prize for best poster was awarded to John
Borwick of Wake Forest University for his poster on 'LDAP for Systems
and Network Engineering'. This described a method for storing DNS and
DHCP configuration data in an LDAP database, and using Perl scripts to
retrieve the data and generate configuration files.

There was also a prize for best paper, which was awarded to Luca Deri
for his paper 'Improving Passive Packet Capture: Beyond Device
Polling'. Deri proposes a new approach to passive packet capture
which, combined with device polling, allows packets to be captured and
analyzed at (almost) wire speed on Gbit networks using a legacy PC.

After presentation of the prizes and thank you's to the many volunteers
who helped to make the conference run so smoothly, Quiz Master Kevlin
Henney took over with the inSANE quiz. Two teams were drawn
'completely at random' from the business cards solicited earlier in
the day, and pitted against each other and the Quiz Master's
'completely fair scoring'.

You really had to know your geek culture to do well in this quiz - but
that alone was not enough. There was audience participation too, with
each team having to guess how the audience would respond to 'yellow
or green' questions. For example, the Quiz Master would shout
'Yellow - Python, Green - Perl', the teams would have to write down
their answers ('yellow' or 'green') before the audience voted
by holding coloured cards in the air.

After one team had been eliminated, the three members of the remaining
team contended with each other for prizes of books, posters and
T-shirts. The quiz was a fun way to end a very enjoyable conference.

I was impressed both by the professionalism of the organization, the
quality of the talks, and the smooth running of the event. RAI offered
excellent facilities, and the organizers had provided wireless
networking throughout the conference area, as well as a terminal room
with Internet access for those of us traveling without laptops.

Congratulations, NLUUG, on another excellent conference! I am looking
forward already to SANE 2006, and heartily recommend it to anyone else
with an interest in network or system administration. You can find out
more about past and future SANE conferences at http://www.sane.nl/.

Ray Miller is a director of UKUUG, the UK's Unix and Open Systems User
Group, and Chairman of UKUUG Council. He works as a Unix Systems
Programmer at the University of Oxford, where he leads the Systems
Development and Support team in the University's Computing Services.

This article is available on the author's home page at
http://users.ox.ac.uk/~raym/writing/sane2004.html.