The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.
Sunday, March 1. 2009
At the moment ZFS, DTrace or Zones are the well known features of Solaris. But in my opinion there will be a fourth important feature soon. Since Build 105 itīs integrated (many people will already know which feature i want to describe in this artcle) into Solaris. This feature has the project name Crossbow. Itīs the new TCP/IP stack of Opensolaris and was developed with virtualisation in mind from ground up. "Virtualisation in mind" does not only lead to the concept of virtual network interface cards, you can configure virtual switches as well and even more important: You can control the resource usage of a virtual client or managing the load by distributing certain TCP/IP traffic to dedicated CPUs. Iīve already held some talks about Crossbow at different events, thus itīs time to write an article about this topic. I will start with the virtualisation part of Crossbow.
The Virtualisation partThis part is heavily inspired by this blog entry of Ben Rockwood, but he ommited some parts in the course of his article to make a full walk-through out of it, so i extended it a little bit.
Normally a network consists out of switches and networking cards, server and router . Itīs easy to replicate this in a single system. Networking cards can be simulated by VNICS, switches are called etherstubs in the namespace of Crossbow. Server can be simulated by zones of course, and as router are not much more than special-purpose servers, we could simulate them by a zone as well.
A simple networkLet us simulate a simple network at first. Just two servers and a router:
At first we create two virtual switches. They are called
Okay, now we create virtual nics that are bound to the virtual switch
Now we do the same with our second virtual switch:
Okay, letīs look up the configuratio of our network.
Yes, thatīs all ... but what can we do with it? For example simulating a complete network in your system. Letīs create a network with two networks, a router with a firewall and nat and a server in each of the network. Obviously we will use zones for this.
A template zoneAt first we create a template zone. This zone is just used for speeding up the creation of other zones. To enable zone creation based on ZFS snapshots, we have to create a filesystem for our zones and mount it at a nice position in your filesystem:
Now we prepare a command file for the zone creation. The pretty much the standard for a sparse root zone. We donīt configure any network interfaces, as we never boot or use this zone. Itīs just a template as the name alreay states. So at first we create a file called
Now we create the zone. Depending on your test equipment this will take some times.
Got a coffee? The next installations will be much faster.We will not boot it as we donīt need it for our testbed.
site.xmlWhile waiting for the zone installation to end we can create a few other files. At first you should create a file called
Zone configurations for the testbedAt first we have to create the zone configurations. The files are very similar. The differences are in the zonepath and in the network configuration. The zone
We have created both config files for the simulated servers. Now we do the same for our simulated router. The configuration of the
sysidcfg filesTo speed up installation we create some sysidconfig files for our zones. Without this files, the installation would "go interactive" and you would have to use menus to provide the configuration informations. When you copy place such a file at
After this, we create a second sysidconfig file for our first server zone. I store the following content into a file called
When you look closely at the network_interface line you will see, that i didnīt specified a default route. Please keep this in mind. In a last step i create
Firing up the zonesAfter creating all this configuration files, we use them to create some zones. The procedure is similar for all zone. At first we do the configuration. After this we clone the
We repeat the steps for
At last we repeat it for our zone
After completing the last step, we display the existing zones.
All zones are up and running.
Playing around with our simulated networkAt first a basic check. Letīs try to plumb one of the VNICs already used in a zone.
Excellent. The system prohibits the plumbing. Before we can play with our mini network, we have to activate forwarding and routing on our new router. Since Solaris 10 this is really easy. There is a command for it:
This test goes only skin-deep into the capabilities of Solaris in regard of routing. But that is stuff for more than one LKSF tutorial. Now letīs look into the routing table of one of our server:
Do you remember, that iīve asked you to keep in mind, that we didnīt specified a default route in the sysidcfg? But why have we such an defaultrouter now. There is some automagic in the boot. When a system with a single interfaces comes up without an default route specified in
The rdisc protocol is implemented by the
As you can see ... weīve builded a network in a box.
Building a more complex network
As you see, you are not bound to a certain numbering scheme. You can call a vnic as you want, as long itīs beginning with letters and ending with numbers. Now we use an editor to create a configuration file for our
We donīt have to configure any default router in this
Okay, we can fire up the zone.
Okay, the next zone is the
The same rules as for the
Okay, i assume you already know the following steps. Itīs just the same just with other files.
Okay, this is the last zone configuration in my tutorial. Itīs the zone for
Again ... no defaultroute ... as this is a single-interface system we leave it to the ICMP Router Discovery Protocol to find the routers. So create a file called
Well ... itīs zone startup time again ...
So at first we have to make routers out of our routing zones. Obviously we have to login into the both routing zones and activating forwarding and routing. At first on
Now lets login into the console of our server:
As you see, there are two default routers in the routing table. The host receives router advertisments from two routers, thus it adds both into the routing table. Now letīs have a closer at the routing table of the
This system has more than one devices. Thus the in.routed starts up as a RIP capable routing daemon. After a short moment the in.routed has learned enough about the network and adds itīs routing table to the kernel. And after a short moment the routing tables of our router are filled with the routing informations provided by the routing protocols.
ConclusionThis part of the tutorial just covers a small part of Crossbow. In the next part i will talk about the capabilities in regard of managing flows of network traffic with crossbow. The scope of the virtualisation part is wider than just testing. Imagine the following situation: You want to consolidate several servers in a complex networks, but you want or you cant change a configuration file. In regard of the networking configuration you just could simulate it in one machine. And as itīs part of a single operating system kernel it is a very efficent way to do it. You donīt need virtual I/O servers or something like that. Itīs the single underlying kernel of Solaris itself doing this job. Another interesting use case for Crossbow was introduced by Glenn Brunette in his concept for the immutable service containers.
Posted by Joerg Moellenkamp in English, Solaris at 21:20 | Comments (9) | Trackbacks (2)
Related entries by tags:
In "Upcoming Solaris Features: Crossbow - Part 1: Virtualisation" i forgot to provide the site.xml file and got some questions about it. Well ... now you can download it here.
Tracked: Apr 24, 20:21
Tracked: Jan 28, 02:56
Display comments as (Linear | Threaded)
The site.xml could be found where?
#1 Volker on 2009-03-02 10:22
Found a reference thru the lksf book
#1.1 Volker on 2009-06-25 11:03
You mentioned firewall/nat. Is there a new firewall feature build
into Crossbow (Solaris 10 includes IP filter)? As far as I remember
ipf requires some devices (e.g. /dev/ipauth, /dev/ipstate).
Must your firewall/nat zone have access to these devices?
Could you post the reference please to the site.xml file.
I also noticed that the link to site.xml is broken. Could you kindly port that link?
#4 Alejandro on 2009-03-29 22:20
Small error in the text. It says
Okay, now we create virtual nics that are bound to the virtual switch etherstub0. These virtual nics are called vnic1 and vnic0.
# dladm create-vnic -l etherstub0 vnic1
# dladm create-vnic -l etherstub0 vnic2
The 2nd sentence should read "These virtual nics are called vnic1 and vnic2"
I was trying the above setup on OpenSolaris 2009.06 but discovered that routing protocols don't seem to be available (routeadm -e ipv4-routing didn't work) and also within each guest the router discovery isn't happening either. Are these observations expected and is the only workaround SCXE (I presume the above was tested using SCXE) ?
#6 Jeroen on 2009-06-23 07:30
Great article Joerg.
I'm happy to announce that Crossbow was featured in the last JavaOne/Community One conference, It was featured at the keynote speaker.
I'll be making the demo presented during the launch available for the rest of the OpenSolaris community soon.
The LKSF book
The book with the consolidated Less known Solaris Tutorials is available for download here
Joerg M. about Checking VXFS QIO usage
Sun, 20.04.2014 11:27
This is exactly the problem. I t seems even the way, that it' s by default QIO and many peop le think this is all the [...]
David Lange about Checking VXFS QIO usage
Sun, 20.04.2014 00:08
Excuse my errors if present, b ut as I remember you have to d o more than set qio on the mou nt point. The actual DB [...]
George Israel about Actually it's a n-way mirror
Tue, 15.04.2014 17:07
Is 11 the max? Or can N be any number?
Rafal about Oracle VM for SPARC 3.1.1
Fri, 11.04.2014 12:41
Wow, at last. I assume that also FCoE/CNA cards are suppor ted.
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Germany License