Written by {"display_name"=>"Joerg Moellenkamp", "login"=>"Joerg Moellenkamp", "email"=>"weblog@joerg.moellenkamp.org"}
on January 01, 2005

First Cut: Crossbow LKSF

At the moment ZFS, DTrace or Zones are the poster box features of Solaris. But there will be a fourth feature soon. Since Build 105 it´s integrated (many people will already know which feature i want to describe in this artcle) into Solaris. This feature has the project name Crossbow. It´s the new TCP/IP stack of Opensolaris and was developed with virtualisation in mind from ground up.

Concepts of Crossbow

The Virtualisation part

This part is heavily inspired by this blog entry of Ben Rockwood, but he ommited some parts in the course of his article, so i extended it a little bit. It´s really easy to create etherstubs and virtual nics.

At first we create two virtual switches. They are called etherstub0 and etherstub1

# dladm create-etherstub etherstub0<br />
# dladm create-etherstub etherstub1

Okay, now we create virtual nics that are bound to the virtual switch etherstub0. These virtual nics are called vnic1 and vnic0.

# dladm create-vnic -l etherstub0 vnic1<br />
# dladm create-vnic -l etherstub0 vnic2

Now we do the same with our second virtual switch:

# dladm create-vnic -l etherstub1 vnic3<br />
# dladm create-vnic -l etherstub1 vnic4

# dladm showlink<br />
LINK        CLASS    MTU    STATE    OVER<br />
ni0         phys     1500   unknown  --<br />
etherstub0  etherstub 9000  unknown  --<br />
etherstub1  etherstub 9000  unknown  --<br />
vnic1       vnic     9000   up       etherstub0<br />
vnic2       vnic     9000   up       etherstub0<br />
vnic3       vnic     9000   up       etherstub1<br />
vnic4       vnic     9000   up       etherstub1<br />
vnic0       vnic     1500   up       ni0

Yes, that´s all … but what can we do with it? For example simulating a complete network in your system. Let´s create a network with two networks, a router with a firewall and nat and a server in each of the network. Obviously we will use zones for this.

Prerequisites

A template zone

At first we create a template zone. This zone is just used for speeding up the creation of other zones. To enable zone creation based on ZFS snapshots, we have to create a filesystem for our zones and mount it at a nice position in your filesystem:

# zfs create rpool/zones<br />
# zfs set compression=on rpool/zones<br />
# zfs set mountpoint=/zones rpool/zones

Now we prepare a command file for the zone creation. The pretty much the standard for a sparse root zone. We don´t configure any network interfaces, as we never boot or use this zone. It´s just a template as the name alreay states.

create -b<br />
set zonepath=/zones/template<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
commit

Now we create the zone.

# zonecfg -z template -f template<br />
# zoneadm -z template install<br />
A ZFS file system has been created for this zone.<br />
Preparing to install zone <template2>.<br />
Creating list of files to copy from the global zone.<br />
Copying <3488> files to the zone.<br />
Initializing zone product registry.<br />
Determining zone package initialization order.<br />
Preparing to initialize <1507> packages on the zone.<br />
Initialized <1507> packages on zone.<br />
Zone <template2> is initialized.<br />
The file </zones/template2/root/var/sadm/system/logs/install_log> contains a log of the zone installation.<br />
#

We will not boot it as we don´t need it for our testbed.

site.xml

While waiting for the zone installation to end we can create a few other files. At first you should create a file called site.xml. This files controls which services are online after the first boot. You can think of it like an sysidcfg for the Service Management Framework. The file is rather long, so i won´t post it in the article directly. You can download my version of this file here.

Zone configurations for the testbed

At first we have to create the zone configurations. The files are very similar. The differences are in the zonepath and in the network configuration. The zone servera is located in /zones/servera and uses the network interface vnic2

create -b<br />
set zonepath=/zones/serverA<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=vnic2<br />
end<br />
commit

The zone serverb uses the directory /zones/serverb and is configured to bind to the interface vnic4

create -b<br />
set zonepath=/zones/serverB<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=vnic4<br />
end<br />
commit

The configuration of the router zone is a little bit longer as we need more network interfaces:

create -b<br />
set zonepath=/zones/router<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=vnic5<br />
end<br />
add net<br />
set physical=vnic1<br />
end<br />
add net<br />
set physical=vnic3<br />
end<br />
commit

sysidcfg files

To speed up installation we create some sysidconfig files for our zones. Without this files, the installation would “go interactive” and you would have to use menus to provide the configuration informations. When you copy place such a file at /etc/sysidcfg the system will be initialized with the information provided in the file. I will start with the sysidcfg file of router zone:

system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=vnic5 {primary hostname=router1 ip_address=10.211.55.10 netmask=255.255.255.0 protocol_ipv6=no default_route=10.211.55.1}<br />
network_interface=vnic1 {hostname=router1-a ip_address=10.211.100.10 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
network_interface=vnic3 {hostname=router1-b ip_address=10.211.101.10 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central

After this, we create a second sysidconfig file for our first server zone. I store the following content into a file called servera_sysidcfg:

system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=vnic2 {primary hostname=server1 ip_address=10.211.100.11 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central

When you look closely at the network_interface line you will see, that i didn´t specified a default route. Please keep this in mind. In a last step i create serverb_sysidcfg. It´s the config file for our second server zone

system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=vnic4 {primary hostname=server2 ip_address=10.211.101.11 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central<br />

Firing up the zones

After creating all this configuration files, we use them to create some zones. The procedure is similar for all zone. At first we do the configuration. After this we clone the template zone. As we located the template zone in a ZFS filesystem, the cloning takes just a second. Before we boot the zone, we place our configuration files we prepared while waiting for the installation of the template zone.

<br />
# zonecfg -z router -f router<br />
# zoneadm -z router clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone3<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp router_sysidcfg /zones/router/root/etc/sysidcfg<br />
# cp site.xml /zones/router/root/var/svc/profile<br />
# zoneadm -z router boot</blockquote>
</code><br />
We repeat the steps for <code>servera</a>.
<blockquote><code># zonecfg -z servera -f serverA<br />
# zoneadm -z servera clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone3<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp serverA_sysidcfg /zones/serverA/root/etc/sysidcfg<br />
# cp site.xml /zones/serverA/root/var/svc/profile<br />
# zoneadm -z servera boot</blockquote>
</code><br />
At last we repeat it for our zone <code>serverb</code>:
<blockquote><code># zonecfg -z serverb -f serverB<br />
# zoneadm -z serverb clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone3<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp serverb_sysidcfg /zones/serverB/root/etc/sysidcfg<br />
# cp site.xml /zones/serverB/root/var/svc/profile<br />
# zoneadm -z serverb boot</blockquote>
</code><br />
After completing the last step, we display the existing zones. 

<!-- Migration Rule 4 --> 

<figure class="highlight"><pre><code class="language-plaintext" data-lang="plaintext"># zoneadm list -v
  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
  13 router           running    /zones/router                  native   excl  
  15 servera          running    /zones/serverA                 native   excl  
  19 serverb          running    /zones/serverB                 native   excl </code></pre></figure>


All zones are up and running.
<h4>Playing around with our simulated network</h4>
<blockquote><code># ifconfig vnic2 plumb<br />
vnic2 is used by non-globalzone: servera<code></blockquote>
Before we can play with our mini network, we have to activate forwarding and routing on our new router. Since Solaris 10 this is really easy. There is a command for it: 
<blockquote><code># routeadm -e ipv4-forwarding<br />
# routeadm -e ipv4-routing<br />
# routeadm -u<br />
# routeadm<br />
              Configuration   Current              Current<br />
                     Option   Configuration        System State<br />
---------------------------------------------------------------<br />
               IPv4 routing   enabled              enabled<br />
               IPv6 routing   disabled             disabled<br />
            IPv4 forwarding   enabled              enabled<br />
            IPv6 forwarding   disabled             disabled
           Routing services   "route:default ripng:default"
Routing daemons:
                      STATE   FMRI<br />
                   disabled   svc:/network/routing/zebra:quagga<br />
                   disabled   svc:/network/routing/rip:quagga<br />
                   disabled   svc:/network/routing/ripng:default<br />
                   disabled   svc:/network/routing/ripng:quagga<br />
                   disabled   svc:/network/routing/ospf:quagga<br />
                   disabled   svc:/network/routing/ospf6:quagga<br />
                   disabled   svc:/network/routing/bgp:quagga<br />
                   disabled   svc:/network/routing/isis:quagga<br />
                   disabled   svc:/network/routing/rdisc:default<br />
                     online   svc:/network/routing/route:default<br />
                   disabled   svc:/network/routing/legacy-routing:ipv4<br />
                   disabled   svc:/network/routing/legacy-routing:ipv6<br />
                     online   svc:/network/routing/ndp:default<br />

Now let´s look into the routing table of one of our server:

# netstat -nr
Routing Table: IPv4<br />
  Destination           Gateway           Flags  Ref     Use     Interface<br />
-------------------- -------------------- ----- ----- ---------- ---------<br />
default              10.211.100.10        UG        1          0 vnic2<br />
10.211.100.0         10.211.100.11        U         1          0 vnic2<br />
127.0.0.1            127.0.0.1            UH        1         49 lo0

Do you remember, that i´ve asked you to keep in mind, that we didn´t specified a default route in the sysidcfg? But why have we such an defaultrouter. There is some automagic in the initial boot routine. When a system with a single interfaces comes up without an defaultroute specified in /etc/defaultrouter or without being a dhcp client it automatically starts up the router discovery protocol as specified by RPC 1256. By using this protocol the hosts adds all available routers in the subnet as a defaultrouter. The rdisc protocol is implemented by the in.routed daemon. It implements two different protocols. The first one is the already mentioned rdisc protocol. But it implements the RIP protocol as well. The RIP protocol part is automagically activated when a system has more than one network interface.

# ping 10.211.100.11<br />
10.211.100.11 is alive<br />
# traceroute 10.211.100.11<br />
traceroute to 10.211.100.11 (10.211.100.11), 30 hops max, 40 byte packets<br />
 1  10.211.101.10 (10.211.101.10)  0.285 ms  0.266 ms  0.204 ms<br />
 2  10.211.100.11 (10.211.100.11)  0.307 ms  0.303 ms  0.294 ms<br />
#

Building a more complex network

Let´s extend our example a little bit…

<br />
# dladm create-etherstub etherstub10<br />
# dladm create-vnic -l etherstub1 routerb1<br />
# dladm create-vnic -l etherstub10 routerb10<br />
# dladm create-vnic -l etherstub10 serverc1<br />
# dladm create-vnic -l etherstub1 routerc1<br />
# dladm create-vnic -l etherstub10 routerc2

As you see, you are not bound to a certain numbering scheme. You can call a vnic as you want, as long it´s beginning with letters and ending with numbers.

create -b<br />
set zonepath=/zones/routerB<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=routerb1<br />
end<br />
add net<br />
set physical=routerb10<br />
end<br />
commit

We don´t have to configure any default router in this sysidcfg. The system boots up with a router and will get it´s routing tables from the RIP protocol.

system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=routerb1 {primary hostname=routerb ip_address=10.211.101.254 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
network_interface=routerb10 {hostname=routerb-a ip_address=10.211.102.10 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central

Okay, we can fire up the zone.

# zonecfg -z routerb -f routerb<br />
# zoneadm -z routerb clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone4<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp routerb_sysidcfg /zones/routerb/root/etc/sysidcfg<br />
# cp site.xml /zones/routerB/root/var/svc/profile/<br />
# zoneadm -z routerb boot

Okay, the next zone is the routerc zone. We bind it to the matching vnics in the zone configuration.

create -b<br />
set zonepath=/zones/routerc<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=routerc1<br />
end<br />
add net<br />
set physical=routerc2<br />
end<br />
commit

The same rules as for the routerb apply to the routerc</a>. We will rely on the routing protocols to provide a defaultroute, so we can just insert NONE into the sysidcfg

# cat routerc_sysidcfg<br />
system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=routerc1 {primary hostname=routerb ip_address=10.211.102.254 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
network_interface=routerc2 {hostname=routerb-a ip_address=10.211.100.254 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central

Okay, i assume you already know the following steps. It´s just the same just with other files.

# zonecfg -z routerc -f routerC<br />
# zoneadm -z routerc clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone4<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp routerb_sysidcfg /zones/routerC/root/etc/sysidcfg<br />
# cp site.xml /zones/routerC/root/var/svc/profile/<br />
# zoneadm -z routerc boot

Okay, this is the zone configuration in my tutorial. It´s the zone for serverc

# cat serverC<br />
create -b<br />
set zonepath=/zones/serverC<br />
set ip-type=exclusive<br />
set autoboot=false<br />
add inherit-pkg-dir<br />
set dir=/lib<br />
end<br />
add inherit-pkg-dir<br />
set dir=/platform<br />
end<br />
add inherit-pkg-dir<br />
set dir=/sbin<br />
end<br />
add inherit-pkg-dir<br />
set dir=/usr<br />
end<br />
add inherit-pkg-dir<br />
set dir=/opt<br />
end<br />
add net<br />
set physical=serverc1<br />
end<br />
commit<code></blockquote>
Again ... no defaultroute ... as this is a single-interface system we leave it to the ICMP Router Discovery Protocol to find the routers.
<blockquote><code>system_locale=C<br />
terminal=vt100<br />
name_service=none<br />
network_interface=serverc1 {primary hostname=server2 ip_address=10.211.102.11 netmask=255.255.255.0 protocol_ipv6=no default_route=NONE}<br />
nfs4_domain=dynamic<br />
root_password=cmuL.HSJtwJ.I<br />
security_policy=none<br />
timeserver=localhost<br />
timezone=US/Central

Well … it´s zone startup time again …

# zonecfg -z serverc -f routerC<br />
# zoneadm -z serverc clone template<br />
Cloning snapshot rpool/zones/template@SUNWzone4<br />
Instead of copying, a ZFS clone has been created for this zone.<br />
# cp serverc_sysidcfg /zones/serverC/root/etc/sysidcfg<br />
# cp site.xml /zones/serverC/root/var/svc/profile/<br />
# zoneadm -z serverC boot

Obviously we have to login into the both routing zones and activating forwarding and routing. At first on routerb:

# routeadm -e ipv4-forwarding # routeadm -e ipv4-routing # routeadm -u </blockquote> Afterwards on routerc. The command sequence is identical. # routeadm -e ipv4-forwarding # routeadm -e ipv4-routing # routeadm -u</blockquote> Now lets login into the console of our server: servera# netstat -nr Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 10.211.100.10 UG 1 0 vnic2 default 10.211.100.254 UG 1 0 vnic2 10.211.100.0 10.211.100.11 U 1 0 vnic2 127.0.0.1 127.0.0.1 UH 1 49 lo0 As you see, there are two default routers in the routing table. The host receives router advertisments from two routers, thus it adds both into the routing table. Now let´s have a closer at the routing table of the routerb system. routerb# netstat -nr Routing Table: IPv4 Destination Gateway Flags Ref Use Interface -------------------- -------------------- ----- ----- ---------- --------- default 10.211.101.10 UG 1 0 routerb1 10.211.100.0 10.211.102.254 UG 1 0 routerb10 10.211.101.0 10.211.101.254 U 1 0 routerb1 10.211.102.0 10.211.102.10 U 1 0 routerb10 127.0.0.1 127.0.0.1 UH 1 23 lo0 This system has more than one devices. Thus the in.routed starts up as a RIP capable routing daemon. After a short moment the in.routed has learned enough about the network and adds it´s routing table to the kernel. The Flow part Conclusion

← → Top