Using Linux virtual bridging, User-Mode-Linux and Zebra for IP routing exercises

Christian Hammers <>
v1.0 - last modified at 2003-06-21


  1. Introduction
  2. Description of my test setup
  3. Preperation of UML
  4. Basic Networking
  5. Setting up Zebra
  6. OSPF
  7. Benchmarking UML
  8. Sample config files
  9. References and further literature


Shameless self advertising: A similar article [25] by me was printed in the 01/2004 issue of the German "Linux-Magazin". Go and read it, too! :-)

This tutorial shows how to create several, via OSPF interconnected, IP over Ethernet networks consisting of up to approx. ten hosts (depending on the available hardware) without additional hardware by using simulated hosts and networks.
Only open and free software is used: GNU/Linux for the host system, Linux Bridging to for the virtual network multiport hub/switch, TUN/TAP for the virtual network interfaces, User-Mode-Linux to host the virtual systems and Zebra as OSPF and optionally BGP routing software.

Other projects/howtos that aim at similar goals:

"UML-based pseudo-dedicated hosting service" [23]
Describes how to set up commercial hosting services with uml as compromise between dedicated hosting and virtual servers.
Sharing a hosted server at [24]
An article about using UML to share an ISP-hosted server between several people to save money.

About User-Mode-Linux

User-Mode-Linux (short "UML") [2] is a project that enhances the standard linux kernel [3] in a way that allowes it to run as an executable program inside a normal linux environment. Unlike chroot which only jails one into another file system hierarchy, uml also has a completely seperated process list, devices, process sheduling memory and what is important for this scenario its own IP stack.
More resources can also be found at the UML community site [18] and at yet another UML fan site [19].

About Linux Bridging and TUN/TAP

The Linux Bridging [11] Code simulates the behaviour of a multiport Ethernet switch or bridge. The main part is already included into the Linux kernel code and has to be enabled there.
The TUN/TAP software [12] provides ethernet network interfaces that can be configured for an unprivileged user and attached to a virtual bridge.

About Zebra

The Zebra [9] routing suite is a modular application for RIP, OSPF and BGP routing. Zebra has a Cisco ® [10] like configuration style which makes it especially usefull for practicing. It is stable and capable enough to run in production at various ISPs.
More info about Zebra can be found at the "Unofficial GNU Zebra FAQ" [20] and at zebra-pj [21] which is the unofficial successor to the somewhat orphaned Zebra.

About OSPF and BGP

OSPF is a link-state intranet routing protocol for IP networks. If you read this, you should already know this. Some good information can be found in the references [4], [5] and [6]. OSPF is used as example for this scenario because it is the most commonly used IP over Ethernet routing protocol and uses Ethernet specific features like multicast which are especially interesting to test the capabilities of the simulated network.
BGP [7] was also tried successfully but is not further discussed in this text.
Another ospf testing project can be found at [22].

Description of my test setup

network diagram The setup to which this document references consists of one real host called "app109" and a virtual network with a triangular shape that consists of three virtual UML hosts called "uml1", "uml2" and "uml3" which are connected to each other by three virtual networks.

Click on the picture below to enlarge it and here for the Dia [16] source.

The "networks" are in fact virtual bridges. They are called "brx" and "bra" to "brc". The "brx" bridge connects all of the three virtual hosts and the real host and is used for maintenance like FTP updates and DNS service which is provided by the real host.

The three hosts all talk OSPF and are all connected via eth0 to the real system (for maintenance like FTP updates and DNS service) and with eth1 and eth2 to the other virtual hosts. The three interface have their counterparts on the real host where they are called "umlx1", "umlc3" etc. where the "umlc3" means host "uml3" on bridge "brc". As bridges need different MAC addresses for each interface to work, the virtual ones get theirs, too. They are created so that "00:00:00:00:03:01" means host "uml1" on the third bridge ("brc"). This eases debugging.

The relation between the interface names as well as the invented MAC addresses for the virtual interfaces is defined in the uml start scripts.

Preperation of UML

Creating the simulation is basically done in the following steps:

The host system can be any common GNU/Linux (maybe others, too) system. This tutorial uses the Debian [8] distribution as it is freely available and commonly used at ISPs.
Hardware requirements are depending on the number of hosts that should be simulated. An AMD Athlon 750MHz with 256MB RAM running a memory hungry X11 Gnome/KDE desktop in the background is sufficient for at least three hosts.

Configuring Linux virtual Bridging and Interfaces
Setting up the virtual network is probably the hardest part as the software is not documented very well. A sample configuration file is given in the appendix. Basically the following steps were made:
Several dummy interfaces have to be created. One for each network. Note that you have to use the "-o" option to enable more than three interfaces!
The bridge is created with brctl and set to state up with the "ip" command.
The dummy interface, which is the connection between the host system and the virtual network is set to state "up", too and then attached to the bridge.
All other interfaces on the virtual network are TAP devices which get set up by "tunctl" and "ip" before adding them to the bridge.
Deconfiguring works the other way around. Be warned that this code is not very tolerant to senseless commands. Killing a bridge which still has devices on it or similar stupid things will lock up the host system!
The networking script has of course to be started before any of the UML hosts!
More information can be found on the TUN/TAP [13] and UML [14] mailing lists and in the UML network docs [15].

Preparing a chroot environment
Every UML host needs its own root file system to make it as autarchic as possible. The file system is created inside a normal file using the loopback filesystem. After that, the debootstrap utillity is used to create a plain Debian GNU/Linux installation on this new file system.
We create a file large enough to hold all the file system data.
dd if=/dev/zero of=uml1.loop bs=1M count=250
losetup makes the file appear like a normal block device.
losetup /dev/loop0 uml1.loop
We create a file system there,
mkfs.ext3 -j /dev/loop0
mount it,
mount -t ext3 /dev/loop0 /mnt/uml1
and finally the Debian GNU/Linux system is created inside the loop file. You can add a mirror of your choice as last argument.
debootstrap woody /mnt/uml1
Further configuration to the system can be made by chrooting into it. They can of course also be made later when starting the chroot with UML.
chroot /mnt/uml1
Before using the file system with UML it has to be unmountet.
umount /mnt/uml1

Reproducing the file system images
At this point, after the final configuration when setting up additional hosts and generally every now and then a backup of the UML file systems should be made. This is actually quite easy the systems are plain files which can be replicated the fastest by using rsync which only copies the differences between two files.
fsck.ext3 -v -f uml1.loop
rsync --archive --verbose --progress uml1.loop uml1.2003-05-25.loop

Installing UML
User-Mode-Linux is available as Debian package and should be installed as such. For basic operation no kernel patch is needed. Only compiling an UML kernel requires patching. The kernel can be tested by simply typing "linux" which should output the well known Linux boot up text and stop with "VFS: Unable to mount root fs" which is where our chroot device comes into play.

Configuring UML
An UML instance is started at the command line as followes:
linux \
        umid=uml1 \
        ubd0=uml1.loop \
        mem=128M \
        con0=fd:0,fd:1 con=xterm \
        eth0=tuntap,umlx1,00:00:00:00:00:01, \
        eth1=tuntap,umla1,00:00:00:00:01:01, \
The first parameter is the unique id of this host. It has no relation to the hostname which can be freely configurable for every system.
ubd0 is the first user mode block device, the one where the root file system is searched on, i.e. the just generated loop file.

How much memory a router depends depends mainly on the number of routes it has to deal with. In this test scenario 48MB would probably be sufficient, too.
The last option opens a new xterm which is the tty0 console of the booted system where the dmesg text and the login prompt will appear.
As User-mode-linux is a normal application it can be started by any arbitrary unprivileged user and can only do things on the hosting system that this use could do, too, of course. Starting it as root is discouraged for security reasons.

The network interface are only usable after configuring the virtual network, of course. The second argument in each line is the interface after which followes an arbitrary choosen but unique MAC address and the IP address you want to configure inside the uml system later.
Interestingly configuring the same IP address that was entered on the command line results in a warning:
The tap IP address and the UML eth IP address must be different
It is debatable if my setup is wrong or not, at least it is the only working one I am aware of.
FIXME: If you know a better solutions, please mail me!
While starting up the following line should apper and be checked: Netdevice 0 (00:00:00:00:00:01) : TUN/TAP backend - IP =
You can do this later with the "dmesg(8)" command inside the uml host.

After starting up the uml linux should appear in the host's process list like this:
ch        1642  0.0  0.9  6696 2384 ?        S    17:38   0:00 xterm
ch        1643  0.0  0.6  4420 1584 pts/2    S    17:38   0:00  \_ bash
ch        1665  0.0  0.4  4244 1172 pts/2    S    17:38   0:00      \_ /bin/sh ./
ch        1666  1.3  6.1 25696 15904 pts/2   S    17:38   0:04          \_ linux (uml2) [(Unknown)]                                                    
ch        1668  0.3  0.3  1748  928 pts/2    T    17:38   0:00              \_ [linux]
ch        1673  0.0  6.1 25696 15904 pts/2   S    17:38   0:00              \_ linux (uml2) [(Unknown)]                                                
ch        1674  0.0  6.1 25696 15904 pts/2   S    17:38   0:00              \_ linux (uml2) [(Unknown)]                                                
ch        1695  0.0  0.9  6692 2364 pts/2    S    17:38   0:00              \_ x-terminal-emulator -T Virtual Console #1 (uml2) -e /usr/lib/uml/port-he
ch        1696  0.0  0.1  1180  296 ?        S    17:38   0:00              |   \_ /usr/lib/uml/port-helper -uml-socket /tmp/xterm-pipeHqldhC
ch        1697  0.0  2.3 25696 6004 pts/4    S    17:38   0:00              \_ linux (uml2) [(Unknown)]  
Notice that this snapshot was made with mem=25M and that the process always uses exactly this amount of memory on the host.

Inside the uml host the network has also to be configured via /etc/network/interfaces. The file should contain a stanca like this:
auto eth0
iface eth0 inet static
The interface is then controlable with ifdown eth0 and ifup eth0 or the common ifconfig(8) or better the ip(8) command.

Installing Zebra
Inside the chroot the Zebra software can be installed just like any other Debian packet with apt-get -u install zebra/ or downloaded as source and build in /usr/local/. Important: The current Debian stable release 3.0 "woody" ships the 0.92a version of Zebra as it has a bit more changes as written on the Zebra web page. An upgrade to 0.93b which is e.g. in unstable is adviced.
When installed as Debian package the servers can be configured by editing /etc/default/zebra and the files in /etc/zebra/. There is one file for every protocol daemon, zebra.conf for the main zebra daemon who only gathers the routes from the different protocols and mixes them with the kernel routing table and finally the daemons where protocol daemons can be en- or disabled.
After the initial configuration the servers can be restarted by typing /etc/init.d/zebra restart and then configured like a normal router on the command line interface (cli). To get access to the cli telnet to the daemons special port number (/etc/services knows them): telnet localhost ospfd. The typical copy running-config startup-config works in Zebra and saves the config to /etc/zebra/ (Debian) or /usr/local/etc/ (self compiled)
The ip adresses and interfaces whould be configured in the zebra.conf as well as in the standard /etc/network/interfaces file which is also the right location to setup MTU sizes and similar.

Basic Networking

Once the uml host was correctly configured and has booted the network should be immediately be usable. This can be checked by pinging the real host (app109) or the neighboring hosts, if they are already up. After pinging their MAC addresses should be visible in the arp table which can be displayed by "arp -n".
Hint: If nothing works ensure that all firewall rules are turned of and the bridge initializing script was started before booting the first virtual host.
I suggest to install bind and apt-proxy on the real host and make it accessable by the virtual UML hosts. This way you can use easy to remember hostname and install Debian packages without connecting the test networks to the real network. Sample BIND configuration files can be found in the config section below.

Setting up Zebra

The Zebra daemon itself needs no real configuration. It just collects the routes from the various routing protocol daemons like ospfd and bgpd and merges them with the kernel routing table.
As telnetting to the Zebra daemon with "telnet localhost zebra" is usefull to debug, at least the passwords should be set although. Interface descriptions can be usefull, too.
A typical Zebra debug command is:

uml1-zebra# show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       B - BGP, > - selected route, * - FIB route

O [110/10] is directly connected, eth1, 00:07:11
C>* is directly connected, eth1
O>* [110/20] via, eth1, 00:07:01
O [110/10] is directly connected, eth2, 00:07:11
C>* is directly connected, eth2
C>* is directly connected, lo
O [110/20] via, eth2, 00:07:00
C>* is directly connected, eth0

Warning: If one of the Zebra daemons crashes, all daemons should be killed, "ip route flush proto zebra" be issued on the command line and Zebra be restarted. Although restarting only the crashed daemon seems to work, those kernel routing table entries that this daemon caused will eventually never be deleted! This can be very hard to debug :-(
My Zebra setup for this environment on host uml1 is below. The other configs are in the config file section further down this text.

! Zebra configuration saved from vty
!   2003/02/02 23:39:57
hostname uml1-zebra
password q
enable password q
log syslog
service terminal-length 0
interface lo
 description Loopback 
interface eth0
 description umlx1 
interface eth1
 description umla1 
interface eth2
 description umlb1 
interface dummy0
 ip address
smux peer zebra
line vty


The OSPF config can be as complicated as you like :-) For this environment I configured the following:

The ospf daemon can be configured either by editing the config file and restarting it or by simply "telnet localhost ospfd".
The final config for host uml1 is below. The others in the config file section of this text.

! Zebra configuration saved from vty
!   2003/02/02 23:12:23
hostname uml1-ospfd
password q
enable password q
service terminal-length 0
interface lo
interface eth1
interface eth2
interface eth0
router ospf
 ospf router-id
 redistribute connected
 redistribute static
 network area 0
 network area 3
smux peer zebra_ospfd
line vty

A typical OSPF setup looks like this:

uml1-ospfd# show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface           RXmtL RqstL DBsmL       1   Full/DR         00:00:32        eth1:       0     0     0       1   Full/DR         00:00:32        eth2:       0     0     0

And finally some pretty pictures, made with Ethereal [17], from lonely HELO packets flying across the ether trying to mate...
ospf_image1Host1 sending initial HELO   ospf_image2Host2 sending initial HELO   ospf_image3Host1 saw Host2

After OSPF sessions are established you should be able to ping every ip address. A typical IP configuration then looks like this:

uml1:~# ip address
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet scope host lo
2: eth0: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth0
3: eth1: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:00:00:01:01 brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth1
4: eth2: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:00:00:03:01 brd ff:ff:ff:ff:ff:ff
    inet brd scope global eth2

uml1:~# ip neighbor dev eth0 lladdr 00:00:00:00:00:02 nud reachable dev eth0 lladdr 00:00:00:00:00:03 nud reachable dev eth0 lladdr 00:00:00:00:00:00 nud stale dev eth1 lladdr 00:00:00:00:01:02 nud stale dev eth2 lladdr 00:00:00:00:03:03 nud stale

uml1:~# ip route dev eth1  proto kernel  scope link  src via dev eth1  proto zebra  metric 20 dev eth2  proto kernel  scope link  src dev eth0  proto kernel  scope link  src

Benchmarking UML

The following benchmarks were made:

kernel compile and "time tar xjf linux-2.4.20.tar.bz2"
It's my standard stability test for new systems and usefull to get some comparable data of a real world application. Memory, CPU and I/O are heavily used.
"openssl speed" and "john" cracking the DES encrypted string "626"
These tests are useful to measure pure CPU performance.
My own benchmark programm
Is written in C, precalculates 100MB of random data and then writes and reads it from disk so that I'm sure that nothing else but fread/fwrite is involved. It can be found here.

The results can be found here. My preliminary conclusion is that the CPU is not much affected by the User-Mode-Linux but I/O performance is quite heavily decreasing (writing more than reading). This test did not measure the performance in relation to the uml or host "load".

Sample Config Files

The following config files can be used to reproduce the setup described in this text:
This script openes my personal firewall for virtual bridging. You probably don't need this.
This is where the black magic starts. At least it works. Get it here.
The start scripts for the individual uml hosts., and
And a sample /etc/network/interfaces file, here from uml1:
BIND configuration files
apt-proxy configuration files
app109:/etc/apt-proxy/apt-proxy.conf (excerpt)
app109:/etc/inetd.conf (excerpt)
Zebra configuration files
for host uml1: zebra.conf and ospfd.conf
for host uml2: zebra.conf and ospfd.conf
for host uml3: zebra.conf and ospfd.conf

References and further literature

[1] The permanent location of this text is
[2] The User-Mode-Linux homepage at
[3] The Linux Kernel homepage at
[4] OSPF, Anatomy of an Internet Routing Protocol, John T. Moy, 1998, Addison Wesley Longman, ISBN: 0201634724
[5] Cisco's "OSPF Design Guide" at
[6] The OSPFv2 RFC at
[7] The BGP4 RFC at (and others)
[8] The Debian Project at
[9] The Zebra routing suite at
[10] Cisco Systems at
[11] Linux Bridging at
[12] Linux TUN/TAP driver at
[13] Linux TUN/TAP mailing list at
[14] User-Mode-Linux mailing list at
[15] User-Mode-Linux network docs at
[16] Gnome Dia at
[17] Ethereal at
[18] User-Mode-Linux Community Site at
[19] Another UML Fan site at
[20] The unofficial GNU Zebra FAQ at
[21] Zebra Patched (zebra-pj) at
[22] OSPFD Routing Software Resources at
[23] UML for hosting services at
[24] UML for shared hosting at
[25] Schein-Netz - Simulierte Netze mit User Mode Linux als Basis für Firewall-Tests at

Valid HTML 4.01! Valid CSS! Powered by Zebra ad:vim ad:debian