English

I wrote at a number of occasions (here or here), that it could be really beneficial to use a different memory allocator for highly-threaded workloads, as the standard allocator is well ... the standard, however not very effective as soon as many threads comes into play.

I didn't wrote about this as it was in my phase of silence but there was some change in the allocator area, Solaris 10 got a revamped mtmalloc allocator in version Solaris 10 08/11 (as described in "libmtmalloc Improvements"). The new memory allocator was introduced to Solaris development by the PSARC case 2010/212.

But what's the effect of this new allocator and how does it works? Rickey C. Weisner wrote a nice article with "How Memory Allocation Affects Performance in Multithreaded Programs" explaining the inner mechanism of various allocators but he also publishes test results comparing Hoard, mtmalloc, umem, new mtmalloc and the libc malloc. Really interesting read and a must for people running applications on servers with a high number of threads.

In the last few days i created a cheat sheet for Solaris 11 ... while it's still a work in progress (it will be surely longer in the future). it's already available on OTN as part of the "Evaluating Oracle Solaris 11"-Collection . You can download the "Oracle Solaris 11 Cheat Sheet General Administration" here.

In Solaris 11 the file /etc/nodename is gone. In order to change you nodename you have alter the configuration of an smf service:

root@fuchikoma:/home/jmoekamp# hostname
tachikoma
root@fuchikoma:/home/jmoekamp# svccfg -s svc:/system/identity:node setprop config/nodename = "fuchikoma"
root@fuchikoma:/home/jmoekamp# svcadm refresh svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# svcadm restart svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# hostname
fuchikoma

You don't have to edit config/loopback

root@fuchikoma:/home/jmoekamp# svcprop  svc:/system/identity:node | egrep "^config"        
config/nodename astring fuchikoma
config/loopback astring fuchikoma

There are quite a number of changes in the procedures to configure some of the networking parameters. Many things have changed, that were just editing of a file in the past, have now command-line based tools in order to change their parameters. Before you ask: The reason for this steps are quite simple.

At first ist much easier to script the execution of a command, instead of the editing a file. I wrote a lot of those scripts in the past and from my experiences people have an infinite wisdom in creating ways and means to edit a file in a way that it breaks your automatic edting scripts. Furthermore putting all this configuration statements into SMF has an interesting consequence. In the past, it wasn't always clear to most people when a change of the configuration got active. At the time of the save, at the restart of the daemon? By using SMF it's clear. As soon as you type in svcadm refresh. Furthermore ifconfig came totally overloaded with parameters with the time. So dladm and ipadm were a very good move from my perspective.

However i know that this is maybe a strange for an long-time user, and to be honest, i was spelling many curses in front of the display, albeit i see the advantages. So i'm putting up my cheatsheet online. My colleague Detlef is planing something similar. As soon his is ready, i will linking to his as well, as i'm sure it's more complete.

With automatisms

In the datacenter it's not that useful, however when you have an Solaris VM for example on your notebook you are using for administrative purposes, it's really nice.

At first we create the network configuration profiles for the datacenter and the office.

root@tachikoma:/home/jmoekamp# netcfg create ncp datacenter
root@tachikoma:/home/jmoekamp# netcfg create ncp office

Now we have to fill both with the configuration data. At first the one for the datacenter. I think the values are speaking for themself.

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp datacenter
netcfg:ncp:datacenter> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> static
ipv4-addr> 192.168.1.27
ipv4-default-route> 192.168.1.1
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> exit
root@tachikoma:/home/jmoekamp#

Now we do the same for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp office
netcfg:ncp:office> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> dhcp
ipv4-default-route>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> exit

Now we can just switch the networking configuration by enabling one of the both profiles:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201000843 mtu 1500 index 6
        inet 192.168.1.27 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Let's switch to the office profile:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201004843 mtu 1500 index 7
        inet 192.168.1.185 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Of course the networking configuration is not just the interface configuration but a lot other stuff as well, link the configuration of the DNS. That options are in the location profiles. However i don't want to activate the profile as well as the location both manually. Thus i configure an automatic activation of such configuration. In my example the datacenter profile configures an IP address of 192.168.1.27, the office profile uses DHCP which results in an IP-address out of the 192.168.1.100-199 range. Thus i'm using this for my automatic location switching. I can configure an activation mode and by choosing conditional-any this location profile will be activated automatically when any of the configured conditions are true. It's pretty simple in my case: If the IP address is 192.168.1.27, i'm in the datacenter, if not i'm in the office.

So, i'm configuring the location profile for the datacenter first. I think this is speaking for itself again, when you take the information into consideration, i gave you in the last paragraph:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc datacenter
Created loc 'datacenter'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is 192.168.1.27
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> datacenter.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:datacenter>
netcfg:loc:datacenter> exit
Committed changes

Now i will configure the one for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc office
Created loc 'office'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is-not 192.168.1.28
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> office.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:office>
netcfg:loc:office> exit
Committed changes
root@tachikoma:/home/jmoekamp# netcfg

Okay, let's try this out. At first we activate the datacenter network configuration profile.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'

In theory, the resolv.conf should now change to nameserver 192.168.1.1. Let's check this.

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  datacenter.c0t0d0s0.org
nameserver      192.168.1.1

Okay, that was successful. Now we will switch to the network configuration profile office.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'

We have a look into the resolv.conf again. The nameserver should have switched to 192.168.1.43:

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  office.c0t0d0s0.org
nameserver      192.168.1.43

Nice.

Without that automatic stuff ....

Configuring all this stuff manually isn't hard as well. Perhaps you have chosen "Automatic" initially at install time in the CUI. So at first we have to get rid of the automatisms.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp defaultfixed
Enabling ncp 'DefaultFixed'

Okay, now we configure an ip interface called net0 and assign an ip address

root@tachikoma:/home/jmoekamp# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             unknown    0      unknown   pcn0
root@tachikoma:/home/jmoekamp# ipadm create-ip net0
root@tachikoma:/home/jmoekamp# ipadm create-addr -T static -a  192.168.1.28/24
net0/v4

Of course we need a defaultrouter. With this command we configure such a persistent default route.

root@tachikoma:/home/jmoekamp# route -p add default 192.168.1.1
add net default: gateway 192.168.1.1
add persistent net default: gateway 192.168.1.1

Configuring the dns server is a little bit more complex:

root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop "config/nameserver = net_address: ( 192.168.1.1 )"
root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop 'config/domain = astring: ("c0t0d0s0.org")'
root@tachikoma:/home/jmoekamp# svccfg -s "name-service/switch" setprop 'config/host = astring: "files dns"'
root@tachikoma:/home/jmoekamp# svcadm refresh name-service/switch
root@tachikoma:/home/jmoekamp# svcadm enable dns/client

Okay, let's check the result:

root@tachikoma:/home/jmoekamp# grep "192.168.1.43" /etc/resolv.conf
nameserver      192.168.1.43
root@tachikoma:/home/jmoekamp# grep "host" /etc/nsswitch.conf
hosts:  files dns

Everything works as designed.

Robin Harris of StorageMojo pointed to an interesting research paper: "Optimizing NAND Flash-Based SSDs via Retention Relaxation". The idea of the method described in that paper is basically: The physics behind the process of writing to NAND-flash allows you to write you faster, when you shorten the retention time. The shorter the retention time, the faster the write. Retention time? That the span of time you have to be capable to read the data from a device.

Obviously I mapped that automatically into the ZFS world: For an separated ZIL device write latency is a really, really important metric.However: The normal live time of the data writen to the sZIL? Just until the transaction group has been committed to the pool. A few seconds. However the need retention time is not that short, because you have address failures. You write something like a ZIL for a reason. In the case of a system failure the data has to survive at least as long until the same or a different system is able to commit the data - that wasn't commited before the failure - that is still on the sZIL to the pool devices. However how long is this? A day … perhaps a week. But surely not the standard of 1year or 10 years. However the standard mechanism writes so it could be read for 1 or 10 years as mandated by some standard bodies. And this is exactly the margin the idea in the paper uses. The model in the paper suggest a write speedup of 2.5 in response time for a TPC-C-similar load (figure 13) when you target for a retention time of two weeks instead of a year.

When you combine sZIL and flash storage capable of such a relaxed retention you could basically use such an optimization for pretty much all data storage needs, as the data is still stored in the pool. Further more you can forget about the refreshment of the data as suggested by the paper for data with longer retention time needs, because by it's nature all data is really short-lived and other processes take care of making the data persistent for the next 10 years. And as a weird idea at the end: You could even use a retention-relaxed SSD as an sZIL device in front of a pool of SSD without relaxed retention.

My colleague Constantin wrote a nice article about using encrypted loopback file based pools for backup purposes for example. And don't forget you can do deduplication or snapshots as well for your backuped data. And in case you don't know how migrate your data into this files ... use shadow migration

I had many interesting discussions in the last few days. However some of them gave me the impression that i should explain one thing. I want to use one example for it: A customer asked me while having some coffee at the Tech Days: "Zones? Isn't that just jails like in FreeBSD?". It's that old question, i get since the introduction of Zones. Just to make it clear: It's not a text in regard of FreeBSD. It's a text about the tendency of people just to pick a single feature and to say "Isn't that feature like …". But it isn't that easy.

I don't want to discuss about that question, if a feature is really like another. However i want to introduce a different tought into this:"A feature is not only a feature when you have an overarching architecture and an overarching idea where the architecture is heading to.It's an enabler. And a feature is often just the next question".

A feature is just the next question? Yes, that right. Because every feature you introduce is just the starting point for the next feature. When we introduced Zones years ago, we had a lot open questions not much later: How do you patch those zones? Especially when you have dozens of them? How to delegate administration? How do you install zones in a fast manner? How to implement bootenvironments, for the OS as well as the Zones? How do you reduce hard-to-find problems of an architecture that shares a kernel, but has several copied userlands? Questions, that have perhaps no technical background, but resulting in technical changes because of operational requirements. Perhaps, at the beginning Jails and Zones may have been similar concepts. But when you look today into the construct, Zones is a lot more. Zones is a large interdepending web of features inside of zones and outsides of zones to enable customers to work with them as efficient as possible.

However: Some of the challenges are just solvable when you have an overarching architecture and the power to decide on the architecture. And this is what i want to say with "a feature is not only a feature". Sometimes it's an enabler for a different feature. For example you have to be capable to say "ZFS is the only filesystem for booting and keeping zone roots" then you have a foundation you can use to implement other features. You can take all the mechanisms of ZFS for example for granted to base other features on it.

An example from the automobile sector? Ever asked why the automatic parking for a VW Golf is that cheap? Well, it just reuses an electro motor that it's there for power steering. You just need some software and a computer to give directly orders to this already existing power steering motor? When you allow all engineering teams to use it's own power steering implementing automatic parking in all vehicles is much more difficult.

Or in Solaris: Fast zone cloning? You need snapshots for it. So go to the ZFS people. Exclusive IP-Stack for a zone? You need a revamped IP stack for it. Ask the Crossbow people. Many things are severely depending on each other? Boot environments like in Solaris 11? Just feasible with a filesystem capable to do snapshots. IPS was invented to a part to have a packaging format that is much more aware of a concept like zones than just post/pre install scripts where zones was just an afterthought. Bandwidth Management? The Crossbow people again. Resource Management for your zones? You could use the foundation already laid out by the people with the Solaris Resource Manager years ago.

And there i'm at the point where i'm saying it doesn't suffice to have a feature, for example like Jails. At the introduction of a feature, a new journey just begins to solve all implication of a new feature. And you have to go all the way to make it really good.

As i got the question on the Solaris 11 Techdays: As reported by the Oracle Tech Network Solaris 11 is now in evaluation under the Operating System Protection Profile using the extended packages Advanced Management, Extended Identification and Authentication, Label Security, and Virtualization at level EAL4+.

In the ZFS Storage Appliance we have little nice feature enabling you to do migrations of data in the background. It's called Shadow Migration. It's a really useful feature. Imagine you have a RAIDZ. After a time you recognize that RAIDZ wasn't a good decision for your workload and RAID10 would be much better choice. But how to transform it into a RAID10 and how to do it with minimal interruption? You can do this with the Shadow Migration feature. With the Shadow Migration feature, you can migrate the data from one local or remote filesystem to another, while you are already accessing the new one to get the data on the old ZFS filesystem. This feature is available in Solaris 11 as well.

For this demonstration we will use two zfs pools consisting out of files. So we have to create the files first:

root@test:/test/brainslug# mkfile 128m source1
root@test:/test/brainslug# mkfile 128m source2
root@test:/test/brainslug# mkfile 128m source3
root@test:/test/brainslug# mkfile 128m source4
root@test:/test/brainslug# mkfile 128m target1 
root@test:/test/brainslug# mkfile 128m target2
root@test:/test/brainslug# mkfile 128m target3
root@test:/test/brainslug# mkfile 128m target4
root@test:/test/brainslug# mkfile 128m target5
root@test:/test/brainslug# mkfile 128m target6

Now the pools are created. At first our RAIDZ pool consisting out of 4 files. It's named source

root@test:/# zpool create source raidz \
 /test/brainslug/source1 \
 /test/brainslug/source2 \
 /test/brainslug/source3 \
 /test/brainslug/source4

The second one is the future target of the shadow migration. It consists out of six "disks"

root@test:/# zpool create target
  mirror /test/brainslug/target1 /test/brainslug/target2 \
  mirror /test/brainslug/target3  /test/brainslug/target4 \
  mirror /test/brainslug/target5 /test/brainslug/target6

When you did a basic install, the tools and daemons needed for shadow-migration are not included. You have to install them and enable the shadowd afterwards:

root@test:/test/brainslug# pkg install shadow-migration
root@test:/test/brainslug# svcadm enable shadowd

Now you should see the shadowd daemon running.

root@test:/test/brainslug# ps -ef | grep "shadow"
    root  3292     1   0 14:32:33 ?           0:03 /usr/lib/fs/shadowd

Okay … to test the shadow migration we create a filesystem in the source pool:

root@test:/test/brainslug# zfs create source/somestuff

Now we have to fill this file with a some data. Let's create some play data.

root@test:/test/brainslug# dd if=/dev/urandom of=myfile bs=1024 count=300000                        
300000+0 records in
300000+0 records out
root@test:/test/brainslug# mkdir demodata
root@test:/test/brainslug# cd demodata
root@test:/test/brainslug/demodata# split -b 128k -a 5  ../myfile

This should yield a significant number of 128k files. Now we copy them to the newly created filesystem source/somestuffWe will copy the files into the zfs filesystem posing as our old filesystem:

root@test:/test/brainslug/demodata# cp * /source/somestuff/
root@test:/test/brainslug/demodata# cd /
root@test:/# zfs list source
NAME    USED  AVAIL  REFER  MOUNTPOINT
source  294M  42,1M  46,4K  /source

Just to have something to compare, you could simply count the files and calculate the md5 checksum of a file.

root@test:/# ls -l /source/somestuff | wc -l
    2345
root@test:/# md5sum /source/somestuff/xaadmd
3fb4a6be2f93c3d93998db52061244aa  /source/somestuff/xaadmd

Shadow migration will only works, when the source filesystem read-only. So we have to put the source filesystem into such a state:

root@test:/# zfs set readonly=on source/somestuff

Okay, now let's configure the shadow migration:

root@test:/# zfs create -o shadow=file:///source/somestuff \
target/newlocationforsomestuff

That's all. The command may take some moments to get back. The migration of data starts right in the moment you create the new filesystem. It runs in the background and starts to copy all data to the new filesystem. Important to know: You can do shadow migration via NFS as well and it can be an UFS filesystem as well. you just have to declare the source of the shadow migration like nfs://fileserver/directory

Okay. With shadowstat we can check the process of migration.

root@test:/# shadowstat    
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  25,5M   -       -       00:01:10

The cool think about shadow migration is: You can already use the new filesystem. Despite the fact that the migration is still running, you will already see all files and when you access one file it will be migrated in the moment you access the file on the new filesystem. You don't have to wait with the access, until the block would be migrated by the normal background migration. When you try to access data, that isn't already migrated, it's migrated in the moment you access it in the new filesystem.

root@test:/# md5sum /target/newlocationforsomestuff/xaadmd               
3fb4a6be2f93c3d93998db52061244aa  /target/newlocationforsomestuff/xaadmd
root@test:/# ls -l /target/newlocationforsomestuff | wc -l                       
    2345

Afterwards it proceeds with the further migration of all data in the pool. You can observe that with the shadowstat command.

root@test:/# shadowstat 
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  97,8M   -       -       00:01:50
target/newlocationforsomestuff  128M    -       -       00:02:00
target/newlocationforsomestuff  147M    -       -       00:02:10
target/newlocationforsomestuff  165M    -       -       00:02:20
target/newlocationforsomestuff  186M    -       -       00:02:30
target/newlocationforsomestuff  202M    -       -       00:02:40
target/newlocationforsomestuff  211M    -       -       00:02:50
target/newlocationforsomestuff  224M    -       -       00:03:00
target/newlocationforsomestuff  236M    -       -       00:03:10
target/newlocationforsomestuff  243M    -       -       00:03:20
target/newlocationforsomestuff  249M    -       -       00:03:30
target/newlocationforsomestuff  256M    -       -       00:03:40
target/newlocationforsomestuff  260M    -       -       00:03:50
target/newlocationforsomestuff  266M    -       -       00:04:00
target/newlocationforsomestuff  272M    -       -       00:04:10
target/newlocationforsomestuff  278M    -       -       00:04:20
target/newlocationforsomestuff  286M    -       -       00:04:30
No migrations in progress

root@test:/#

Successfuly migrated.

Do you want to learn more?

Docs
docs.oracle.com: Migrating ZFS File Systems"
docs.oracle.com: Migrating File System Data to ZFS File Systems

Blogs
blogs.oracle.com: What is Shadow Migration
blogs.oracle.com: Shadow Migration Internals

Brendan Gregg wrote a really interesting article about tracing ZFS: Activity of the ZFS ARC. Really worth a read.

Okay … it's 2012 … and according to some people the world will end this year. However what's really happening? It's the mayan version of the Y2038 problem. While the signed 32 bit integer will send us to the 70ies on 03:14:07 UTC on Tuesday, 19 January 2038, the same happens on 22. December 2012. The mayan calender will send us from 13.0.0.0.0 to 0.0.0.0.0. And sorry, there is more potential in the Y2038 problem to kill us all as in the mayan calendar because nobody used the mayan calendar in embedded systems for nuclear weapons, nuclear power stations, isolation fields for strangelets created in the LHC ( ) , air traffic control. With 32 bit signed integer i'm not so sure

Work in Progress - this entry will change often in the next days and weeks

A few days^H^H^H^Hweeks ago, i wrote about simulating the cloud that is most often tagged with the name "network" or "intranet" and sometimes "internet" . This would not be c0t0d0s0.org without an article to explain how you can configure this. This article will explain how you simulate a complete network on a single host with routers, switches, dynamic routing protocols and so on

Scope

At first i want to set the expectations right. I don't want to simulate a cloud in the sense of cloud computing here. I'm thinking about something more complex:



I'm talking about the simulation of this cloud, that often hides a lot of complexities and traps in architectural diagrams.

In the past i wrote quite often about a thing that i call systemic features, when features start to fit together seamlessly in order to create possibilities more than the sum of the features. One of the systemic features is the simulation of the cloud. I don't talk about that thing that most people connect in mind with the word cloud (the grid with a credit card checkout ), but the cloud-like icon in many architectural diagrams called "Network" or "Internet" that sits between the client and the application that often resembles the "a wonder happens here" box in many architectures.

It's not new: I talked about this mid November at the DOAG conference in Nuremberg. And i've playing around with this at customers an privately for a while now.

Many customers have networks as large and as complex as the internet part of a smaller country perhaps 15 years ago. The interesting question is: How can you test your application for it's resiliency against failures in this cloud shaped icon. How does your application react, when your network is doing its high availability magic.

And interestingly Solaris 11 can help you here. The thoughts behind this are pretty simple.

• A router is a computer that runs an operating environment that is tailormade to do network stuff, but at the end it's a computer with a OS (yeah, i know, hardware offloading makes this a little bit more complex, but at the end it's that way)
• A zone is a virtual operating environment.
• Each zone can have it's own set of routes.
• Each zone can have it's own set of firewall rules.
• Each zone can have it's own set of processes.
• Routing protocols are not more than processes collecting information from the network and configuring the routing table.
• You can install a vast array of dynamic routing protocols on a zone.
• I can have up to 8192 zones (given enough memory)
• In Solaris 11 i can emulate switches (etherstubs)
• I can limit bandwidths in Solaris 11 out-of-the-box with crossbow

When i'm combining all this features i can set up a vast array of zones doing nothing else taking each incoming packet on a interface, routing it on a multitude of ways between each other, and send it out on a outgoing interface. Even when the system in your environment are placed in many separate networks of your network you can still use a system with many networking cards or something called server-on-a-stick (single high-bandwidth connection to a vlan-trunking capable switch and using the switch ports as a fan-out).

So in order to emulate a complex corporate network, all i have to do is configuring a lot of etherstubs, configuring many vnics, replicate the physical bandwidths with the maxbw setting on the vnics, set up a lot of zones, perhaps translate the ACL of the routers into firewall rules for firewall functionality of Solaris, installting the routing daemons and configure it similar to the configuration of the routers (in regard of timeouts and so on).

Now i can test, how my applications react, when the network starts to converge against a new topology because of the failures of some lines. I can test, to which topology my network will converge after an line outage (which is nothing more than a deny-all firewall rule). I can test the impact, when the network converges that way, that my traffic flows over a 2 MBit/s line instead of a 155 MBit/s line. For even more complex failure modes i can even use the htbx driver to introduce additional latencies, packet drop or packet reorderung as shown in this article. In essence you can emulate your complete internal network in a single box and with Zones and Crossbow in Solaris 11 it's so low overhead (at the end it is still just one kernel) that you can really emulate the reality and not a simplyfied view, as you don't have emulate via separate hardware or many independent operating system instances in virtual machines.

At the end you could simply use a single Solaris system, put it between all your test systems and use this solaris system as a emulation device for your corporate network. It's simulating the cloud-shaped icon in your architectural diagrams.

I think with a few words as an introduction and this video everybody should understand Quicksort. They just dance the algorithm:

There is a nice example of the power of boot environment. Boot environments are something like snapshots of your operating system installation made writeable. As you may already assume, they are based on ZFS snapshots and the clone functionality. This is possible due to the usage of ZFS as the root filesystem.

So: Please don't try this at home. Whey you try it, don't try it on any Solaris 11 Express installation of any value. But don't try it. I don't want to hear any story. that you've deleted your ERP system by accident because you used the wrong terminal window. Leave that to trained professional stunt admins with the right equipment (Solaris 11 Express)

Assume you have a system, configured with all your application, everything is running fine. So you think it would be nice to have something like a freezed state of this situation. No problem. This command will do the trick.

# beadm create rescuenet
# init 6

When you reboot your system you will see it as a new entry in the grub menu.



Okay, but boot into the old environment starting "Oracle Solaris ..." first by selecting it in the grub menu (it should be already selected, or you used beadm activate already. Now i will drop the atomic bomb on your installation.

# rm --no-preserve-root -rf /

Essentially we've just nuked the installation. After a moment the system should just freeze. Reset the system and boot again via grub into the boot environment starting with "Oracle Solaris ...":



Okay ... on a normal system this would send you to the tapes. With Solaris 11: Reset the system. Boot into the boot environment "rescuenet" via selecting it in grub.



Tada! Just creating a boot environment with a single command after a config change may safe your butt later .... and btw ... this even works in zones ... they know the concepts of boot environment,too.