Christian Hammers <ch@lathspell.de>
v1.0 - last modified at 2003-06-21
This tutorial shows how to create several, via OSPF interconnected, IP over
Ethernet networks consisting of up to approx. ten hosts (depending on the
available hardware) without additional hardware by using simulated hosts and
networks.
Only open and free software is used: GNU/Linux for the host system, Linux
Bridging to for the virtual network multiport hub/switch, TUN/TAP for the
virtual network interfaces, User-Mode-Linux to host the virtual systems and
Zebra as OSPF and optionally BGP routing software.
Other projects/howtos that aim at similar goals:
User-Mode-Linux (short "UML") [2] is a project that enhances the
standard linux kernel [3] in a way that allowes it to run as an
executable program inside a normal linux environment. Unlike chroot which only
jails one into another file system hierarchy, uml also has a completely
seperated process list, devices, process sheduling memory and what is important
for this scenario its own IP stack.
More resources can also be found at the UML community site [18] and
at yet another UML fan site [19].
The Linux Bridging [11] Code simulates the behaviour of a
multiport Ethernet switch or bridge. The main part is already included into the
Linux kernel code and has to be enabled there.
The TUN/TAP software [12] provides ethernet network interfaces
that can be configured for an unprivileged user and attached to a virtual bridge.
The Zebra [9] routing suite is a modular application for RIP, OSPF and BGP routing.
Zebra has a Cisco ® [10] like configuration style which makes it especially
usefull for practicing. It is stable and capable enough to run in production at various ISPs.
More info about Zebra can be found at the "Unofficial GNU Zebra FAQ" [20] and
at zebra-pj [21] which is the unofficial successor to the somewhat orphaned Zebra.
OSPF is a link-state intranet routing protocol for IP networks. If you read
this, you should already know this. Some good information can be found in the
references [4], [5] and [6]. OSPF
is used as example for this scenario because it is the most commonly used IP
over Ethernet routing protocol and uses Ethernet specific features like
multicast which are especially interesting to test the capabilities of the
simulated network.
BGP [7] was also tried successfully but is not further discussed
in this text.
Another ospf testing project can be found at ospf.org [22].
|
The setup to which this document references consists of one real host called "app109"
and a virtual network with a triangular shape that consists of three virtual UML hosts
called "uml1", "uml2" and "uml3" which are connected to each other by three virtual
networks. Click on the picture below to enlarge it and here for the Dia [16] source. The "networks" are in fact virtual bridges. They are called "brx" and "bra" to "brc". The "brx" bridge connects all of the three virtual hosts and the real host and is used for maintenance like FTP updates and DNS service which is provided by the real host. |
The three hosts all talk OSPF and are all connected via eth0 to the real system (for
maintenance like FTP updates and DNS service) and with eth1 and eth2 to the other virtual
hosts. The three interface have their counterparts on the real host where they are called
"umlx1", "umlc3" etc. where the "umlc3" means host "uml3" on bridge "brc".
As bridges need different MAC addresses for each interface to work, the virtual
ones get theirs, too. They are created so that "00:00:00:00:03:01" means host "uml1"
on the third bridge ("brc"). This eases debugging.
The relation between the interface names as well as the invented MAC addresses
for the virtual interfaces is defined in the uml start scripts.
Creating the simulation is basically done in the following steps:
#!/bin/sh
linux \
umid=uml1 \
ubd0=uml1.loop \
mem=128M \
con0=fd:0,fd:1 con=xterm \
eth0=tuntap,umlx1,00:00:00:00:00:01,172.16.42.1 \
eth1=tuntap,umla1,00:00:00:00:01:01,10.0.1.1 \
eth2=tuntap,umlc1,00:00:00:00:03:01,10.0.3.1
The first parameter is the unique id of this host. It has no relation to the
hostname which can be freely configurable for every system.ch 1642 0.0 0.9 6696 2384 ? S 17:38 0:00 xterm ch 1643 0.0 0.6 4420 1584 pts/2 S 17:38 0:00 \_ bash ch 1665 0.0 0.4 4244 1172 pts/2 S 17:38 0:00 \_ /bin/sh ./uml2.sh ch 1666 1.3 6.1 25696 15904 pts/2 S 17:38 0:04 \_ linux (uml2) [(Unknown)] ch 1668 0.3 0.3 1748 928 pts/2 T 17:38 0:00 \_ [linux] ch 1673 0.0 6.1 25696 15904 pts/2 S 17:38 0:00 \_ linux (uml2) [(Unknown)] ch 1674 0.0 6.1 25696 15904 pts/2 S 17:38 0:00 \_ linux (uml2) [(Unknown)] ch 1695 0.0 0.9 6692 2364 pts/2 S 17:38 0:00 \_ x-terminal-emulator -T Virtual Console #1 (uml2) -e /usr/lib/uml/port-he ch 1696 0.0 0.1 1180 296 ? S 17:38 0:00 | \_ /usr/lib/uml/port-helper -uml-socket /tmp/xterm-pipeHqldhC ch 1697 0.0 2.3 25696 6004 pts/4 S 17:38 0:00 \_ linux (uml2) [(Unknown)]Notice that this snapshot was made with mem=25M and that the process always uses exactly this amount of memory on the host.
auto eth0
iface eth0 inet static
netmask 255.255.255.0
network 172.16.42.0
address 172.16.42.1
broadcast 172.16.42.255
The interface is then controlable with ifdown eth0 and ifup eth0
or the common ifconfig(8) or better the ip(8) command.
Once the uml host was correctly configured and has booted the network should
be immediately be usable. This can be checked by pinging the real host (app109)
or the neighboring hosts, if they are already up. After pinging their MAC addresses
should be visible in the arp table which can be displayed by "arp -n".
Hint: If nothing works ensure that all firewall rules are turned of and the
bridge initializing script was started before booting the first virtual host.
I suggest to install bind and apt-proxy on the real host and make it accessable
by the virtual UML hosts. This way you can use easy to remember hostname and
install Debian packages without connecting the test networks to the real network.
Sample BIND configuration files can be found in the config section below.
The Zebra daemon itself needs no real configuration. It just collects the routes
from the various routing protocol daemons like ospfd and bgpd and merges them with
the kernel routing table.
As telnetting to the Zebra daemon with "telnet localhost zebra" is usefull
to debug, at least the passwords should be set although. Interface descriptions can
be usefull, too.
A typical Zebra debug command is:
uml1-zebra# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
B - BGP, > - selected route, * - FIB route
O 10.0.1.0/24 [110/10] is directly connected, eth1, 00:07:11
C>* 10.0.1.0/24 is directly connected, eth1
O>* 10.0.2.0/24 [110/20] via 10.0.1.2, eth1, 00:07:01
O 10.0.3.0/24 [110/10] is directly connected, eth2, 00:07:11
C>* 10.0.3.0/24 is directly connected, eth2
C>* 127.0.0.0/8 is directly connected, lo
O 172.16.42.0/24 [110/20] via 10.0.3.3, eth2, 00:07:00
C>* 172.16.42.0/24 is directly connected, eth0
Warning: If one of the Zebra
daemons crashes, all daemons should be killed, "ip route flush proto zebra"
be issued on the command line and Zebra be restarted. Although restarting only
the crashed daemon seems to work, those kernel routing table entries that this daemon
caused will eventually never be deleted! This can be very hard to debug :-(
My Zebra setup for this environment on host uml1 is below. The other configs are
in the config file section further down this text.
! ! Zebra configuration saved from vty ! 2003/02/02 23:39:57 ! hostname uml1-zebra password q enable password q log syslog service terminal-length 0 ! interface lo description Loopback ! interface eth0 description umlx1 ! interface eth1 description umla1 ! interface eth2 description umlb1 ! interface dummy0 ip address 192.168.1.1/30 ! smux peer 1.3.6.1.4.1.3317.1.2.1 zebra ! line vty !
The OSPF config can be as complicated as you like :-) For this environment I configured the following:
The ospf daemon can be configured either by editing the config file and restarting
it or by simply "telnet localhost ospfd".
The final config for host uml1 is below. The others in the config file section of
this text.
! ! Zebra configuration saved from vty ! 2003/02/02 23:12:23 ! hostname uml1-ospfd password q enable password q service terminal-length 0 ! ! ! interface lo ! interface eth1 ! interface eth2 ! interface eth0 ! router ospf ospf router-id 172.16.42.1 redistribute connected redistribute static network 10.0.1.0/24 area 0 network 10.0.3.0/24 area 3 ! smux peer 1.3.6.1.4.1.3317.1.2.5 zebra_ospfd line vty !
A typical OSPF setup looks like this:
uml1-ospfd# show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface RXmtL RqstL DBsmL 172.16.42.2 1 Full/DR 00:00:32 10.0.1.2 eth1:10.0.1.1 0 0 0 172.16.42.3 1 Full/DR 00:00:32 10.0.3.3 eth2:10.0.3.1 0 0 0
And finally some pretty pictures, made with Ethereal [17], from
lonely HELO packets flying across the ether trying to mate...
Host1 sending initial HELO
Host2 sending initial HELO
Host1 saw Host2
After OSPF sessions are established you should be able to ping every ip address. A typical IP configuration
then looks like this:
uml1:~# ip address
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff
inet 172.16.42.1/24 brd 172.16.42.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:01:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.1/24 brd 10.0.1.255 scope global eth1
4: eth2: <BROADCAST,MULTICAST,ALLMULTI,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:00:00:00:03:01 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global eth2
uml1:~# ip neighbor
172.16.42.2 dev eth0 lladdr 00:00:00:00:00:02 nud reachable
172.16.42.3 dev eth0 lladdr 00:00:00:00:00:03 nud reachable
172.16.42.254 dev eth0 lladdr 00:00:00:00:00:00 nud stale
10.0.1.2 dev eth1 lladdr 00:00:00:00:01:02 nud stale
10.0.3.3 dev eth2 lladdr 00:00:00:00:03:03 nud stale
uml1:~# ip route
10.0.1.0/24 dev eth1 proto kernel scope link src 10.0.1.1
10.0.2.0/24 via 10.0.1.2 dev eth1 proto zebra metric 20
10.0.3.0/24 dev eth2 proto kernel scope link src 10.0.3.1
172.16.42.0/24 dev eth0 proto kernel scope link src 172.16.42.1
The following benchmarks were made:
The results can be found here. My
preliminary conclusion is that the CPU is not much affected by the
User-Mode-Linux but I/O performance is quite heavily decreasing (writing more than
reading). This test did not measure the performance in relation to the uml or host "load".
The following config files can be used to reproduce the setup described in this text:
