Solaris

In Solaris 11 the file /etc/nodename is gone. In order to change you nodename you have alter the configuration of an smf service:

root@fuchikoma:/home/jmoekamp# hostname
tachikoma
root@fuchikoma:/home/jmoekamp# svccfg -s svc:/system/identity:node setprop config/nodename = "fuchikoma"
root@fuchikoma:/home/jmoekamp# svcadm refresh svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# svcadm restart svc:/system/identity:node
root@fuchikoma:/home/jmoekamp# hostname
fuchikoma

You don't have to edit config/loopback

root@fuchikoma:/home/jmoekamp# svcprop  svc:/system/identity:node | egrep "^config"        
config/nodename astring fuchikoma
config/loopback astring fuchikoma

There are quite a number of changes in the procedures to configure some of the networking parameters. Many things have changed, that were just editing of a file in the past, have now command-line based tools in order to change their parameters. Before you ask: The reason for this steps are quite simple.

At first ist much easier to script the execution of a command, instead of the editing a file. I wrote a lot of those scripts in the past and from my experiences people have an infinite wisdom in creating ways and means to edit a file in a way that it breaks your automatic edting scripts. Furthermore putting all this configuration statements into SMF has an interesting consequence. In the past, it wasn't always clear to most people when a change of the configuration got active. At the time of the save, at the restart of the daemon? By using SMF it's clear. As soon as you type in svcadm refresh. Furthermore ifconfig came totally overloaded with parameters with the time. So dladm and ipadm were a very good move from my perspective.

However i know that this is maybe a strange for an long-time user, and to be honest, i was spelling many curses in front of the display, albeit i see the advantages. So i'm putting up my cheatsheet online. My colleague Detlef is planing something similar. As soon his is ready, i will linking to his as well, as i'm sure it's more complete.

With automatisms

In the datacenter it's not that useful, however when you have an Solaris VM for example on your notebook you are using for administrative purposes, it's really nice.

At first we create the network configuration profiles for the datacenter and the office.

root@tachikoma:/home/jmoekamp# netcfg create ncp datacenter
root@tachikoma:/home/jmoekamp# netcfg create ncp office

Now we have to fill both with the configuration data. At first the one for the datacenter. I think the values are speaking for themself.

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp datacenter
netcfg:ncp:datacenter> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> static
ipv4-addr> 192.168.1.27
ipv4-default-route> 192.168.1.1
netcfg:ncp:datacenter:ncu:net0> end
Committed changes
netcfg:ncp:datacenter> exit
root@tachikoma:/home/jmoekamp#

Now we do the same for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> select ncp office
netcfg:ncp:office> create ncu phys net0
Created ncu 'net0'.  Walking properties ...
activation-mode (manual) [manual|prioritized]> manual
link-mac-addr>
link-autopush>
link-mtu>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> create ncu ip net0
Created ncu 'net0'.  Walking properties ...
ip-version (ipv4,ipv6) [ipv4|ipv6]> ipv4
ipv4-addrsrc (dhcp) [dhcp|static]> dhcp
ipv4-default-route>
netcfg:ncp:office:ncu:net0> end
Committed changes
netcfg:ncp:office> exit

Now we can just switch the networking configuration by enabling one of the both profiles:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201000843 mtu 1500 index 6
        inet 192.168.1.27 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Let's switch to the office profile:

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'
root@tachikoma:/home/jmoekamp# ifconfig net0
net0: flags=201004843 mtu 1500 index 7
        inet 192.168.1.185 netmask ffffff00 broadcast 192.168.1.255
        ether 8:0:27:b6:1:70

Of course the networking configuration is not just the interface configuration but a lot other stuff as well, link the configuration of the DNS. That options are in the location profiles. However i don't want to activate the profile as well as the location both manually. Thus i configure an automatic activation of such configuration. In my example the datacenter profile configures an IP address of 192.168.1.27, the office profile uses DHCP which results in an IP-address out of the 192.168.1.100-199 range. Thus i'm using this for my automatic location switching. I can configure an activation mode and by choosing conditional-any this location profile will be activated automatically when any of the configured conditions are true. It's pretty simple in my case: If the IP address is 192.168.1.27, i'm in the datacenter, if not i'm in the office.

So, i'm configuring the location profile for the datacenter first. I think this is speaking for itself again, when you take the information into consideration, i gave you in the last paragraph:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc datacenter
Created loc 'datacenter'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is 192.168.1.27
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> datacenter.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:datacenter>
netcfg:loc:datacenter> exit
Committed changes

Now i will configure the one for the office:

root@tachikoma:/home/jmoekamp# netcfg
netcfg> create loc office
Created loc 'office'.  Walking properties ...
activation-mode (manual) [manual|conditional-any|conditional-all]> conditional-any
conditions> ip-address is-not 192.168.1.28
nameservices (dns) [dns|files|nis|ldap]> dns
nameservices-config-file ("/etc/nsswitch.dns")>
dns-nameservice-configsrc (dhcp) [manual|dhcp]> manual
dns-nameservice-domain> office.c0t0d0s0.org
dns-nameservice-servers> 192.168.1.1
dns-nameservice-search>
dns-nameservice-sortlist>
dns-nameservice-options>
nfsv4-domain>
ipfilter-config-file>
ipfilter-v6-config-file>
ipnat-config-file>
ippool-config-file>
ike-config-file>
ipsecpolicy-config-file>
netcfg:loc:office>
netcfg:loc:office> exit
Committed changes
root@tachikoma:/home/jmoekamp# netcfg

Okay, let's try this out. At first we activate the datacenter network configuration profile.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp datacenter
Enabling ncp 'datacenter'

In theory, the resolv.conf should now change to nameserver 192.168.1.1. Let's check this.

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  datacenter.c0t0d0s0.org
nameserver      192.168.1.1

Okay, that was successful. Now we will switch to the network configuration profile office.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp office
Enabling ncp 'office'

We have a look into the resolv.conf again. The nameserver should have switched to 192.168.1.43:

root@tachikoma:/home/jmoekamp# cat /etc/resolv.conf

#
# Copyright (c) 2012, Oracle and/or its affiliates. All rights reserved.
#

#
# AUTOGENERATED_FROM_SMF_V1
#
# WARNING: THIS FILE GENERATED FROM SMF DATA.
#   DO NOT EDIT THIS FILE.  EDITS WILL BE LOST.
# See resolv.conf(4) for details.

domain  office.c0t0d0s0.org
nameserver      192.168.1.43

Nice.

Without that automatic stuff ....

Configuring all this stuff manually isn't hard as well. Perhaps you have chosen "Automatic" initially at install time in the CUI. So at first we have to get rid of the automatisms.

root@tachikoma:/home/jmoekamp# netadm enable -p ncp defaultfixed
Enabling ncp 'DefaultFixed'

Okay, now we configure an ip interface called net0 and assign an ip address

root@tachikoma:/home/jmoekamp# dladm show-phys
LINK              MEDIA                STATE      SPEED  DUPLEX    DEVICE
net0              Ethernet             unknown    0      unknown   pcn0
root@tachikoma:/home/jmoekamp# ipadm create-ip net0
root@tachikoma:/home/jmoekamp# ipadm create-addr -T static -a  192.168.1.28/24
net0/v4

Of course we need a defaultrouter. With this command we configure such a persistent default route.

root@tachikoma:/home/jmoekamp# route -p add default 192.168.1.1
add net default: gateway 192.168.1.1
add persistent net default: gateway 192.168.1.1

Configuring the dns server is a little bit more complex:

root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop "config/nameserver = net_address: ( 192.168.1.1 )"
root@tachikoma:/home/jmoekamp# svccfg  -s "dns/client" setprop 'config/domain = astring: ("c0t0d0s0.org")'
root@tachikoma:/home/jmoekamp# svccfg -s "name-service/switch" setprop 'config/host = astring: "files dns"'
root@tachikoma:/home/jmoekamp# svcadm refresh name-service/switch
root@tachikoma:/home/jmoekamp# svcadm enable dns/client

Okay, let's check the result:

root@tachikoma:/home/jmoekamp# grep "192.168.1.43" /etc/resolv.conf
nameserver      192.168.1.43
root@tachikoma:/home/jmoekamp# grep "host" /etc/nsswitch.conf
hosts:  files dns

Everything works as designed.

Robin Harris of StorageMojo pointed to an interesting research paper: "Optimizing NAND Flash-Based SSDs via Retention Relaxation". The idea of the method described in that paper is basically: The physics behind the process of writing to NAND-flash allows you to write you faster, when you shorten the retention time. The shorter the retention time, the faster the write. Retention time? That the span of time you have to be capable to read the data from a device.

Obviously I mapped that automatically into the ZFS world: For an separated ZIL device write latency is a really, really important metric.However: The normal live time of the data writen to the sZIL? Just until the transaction group has been committed to the pool. A few seconds. However the need retention time is not that short, because you have address failures. You write something like a ZIL for a reason. In the case of a system failure the data has to survive at least as long until the same or a different system is able to commit the data - that wasn't commited before the failure - that is still on the sZIL to the pool devices. However how long is this? A day … perhaps a week. But surely not the standard of 1year or 10 years. However the standard mechanism writes so it could be read for 1 or 10 years as mandated by some standard bodies. And this is exactly the margin the idea in the paper uses. The model in the paper suggest a write speedup of 2.5 in response time for a TPC-C-similar load (figure 13) when you target for a retention time of two weeks instead of a year.

When you combine sZIL and flash storage capable of such a relaxed retention you could basically use such an optimization for pretty much all data storage needs, as the data is still stored in the pool. Further more you can forget about the refreshment of the data as suggested by the paper for data with longer retention time needs, because by it's nature all data is really short-lived and other processes take care of making the data persistent for the next 10 years. And as a weird idea at the end: You could even use a retention-relaxed SSD as an sZIL device in front of a pool of SSD without relaxed retention.

My colleague Constantin wrote a nice article about using encrypted loopback file based pools for backup purposes for example. And don't forget you can do deduplication or snapshots as well for your backuped data. And in case you don't know how migrate your data into this files ... use shadow migration

I had many interesting discussions in the last few days. However some of them gave me the impression that i should explain one thing. I want to use one example for it: A customer asked me while having some coffee at the Tech Days: "Zones? Isn't that just jails like in FreeBSD?". It's that old question, i get since the introduction of Zones. Just to make it clear: It's not a text in regard of FreeBSD. It's a text about the tendency of people just to pick a single feature and to say "Isn't that feature like …". But it isn't that easy.

I don't want to discuss about that question, if a feature is really like another. However i want to introduce a different tought into this:"A feature is not only a feature when you have an overarching architecture and an overarching idea where the architecture is heading to.It's an enabler. And a feature is often just the next question".

A feature is just the next question? Yes, that right. Because every feature you introduce is just the starting point for the next feature. When we introduced Zones years ago, we had a lot open questions not much later: How do you patch those zones? Especially when you have dozens of them? How to delegate administration? How do you install zones in a fast manner? How to implement bootenvironments, for the OS as well as the Zones? How do you reduce hard-to-find problems of an architecture that shares a kernel, but has several copied userlands? Questions, that have perhaps no technical background, but resulting in technical changes because of operational requirements. Perhaps, at the beginning Jails and Zones may have been similar concepts. But when you look today into the construct, Zones is a lot more. Zones is a large interdepending web of features inside of zones and outsides of zones to enable customers to work with them as efficient as possible.

However: Some of the challenges are just solvable when you have an overarching architecture and the power to decide on the architecture. And this is what i want to say with "a feature is not only a feature". Sometimes it's an enabler for a different feature. For example you have to be capable to say "ZFS is the only filesystem for booting and keeping zone roots" then you have a foundation you can use to implement other features. You can take all the mechanisms of ZFS for example for granted to base other features on it.

An example from the automobile sector? Ever asked why the automatic parking for a VW Golf is that cheap? Well, it just reuses an electro motor that it's there for power steering. You just need some software and a computer to give directly orders to this already existing power steering motor? When you allow all engineering teams to use it's own power steering implementing automatic parking in all vehicles is much more difficult.

Or in Solaris: Fast zone cloning? You need snapshots for it. So go to the ZFS people. Exclusive IP-Stack for a zone? You need a revamped IP stack for it. Ask the Crossbow people. Many things are severely depending on each other? Boot environments like in Solaris 11? Just feasible with a filesystem capable to do snapshots. IPS was invented to a part to have a packaging format that is much more aware of a concept like zones than just post/pre install scripts where zones was just an afterthought. Bandwidth Management? The Crossbow people again. Resource Management for your zones? You could use the foundation already laid out by the people with the Solaris Resource Manager years ago.

And there i'm at the point where i'm saying it doesn't suffice to have a feature, for example like Jails. At the introduction of a feature, a new journey just begins to solve all implication of a new feature. And you have to go all the way to make it really good.

As i got the question on the Solaris 11 Techdays: As reported by the Oracle Tech Network Solaris 11 is now in evaluation under the Operating System Protection Profile using the extended packages Advanced Management, Extended Identification and Authentication, Label Security, and Virtualization at level EAL4+.

In the ZFS Storage Appliance we have little nice feature enabling you to do migrations of data in the background. It's called Shadow Migration. It's a really useful feature. Imagine you have a RAIDZ. After a time you recognize that RAIDZ wasn't a good decision for your workload and RAID10 would be much better choice. But how to transform it into a RAID10 and how to do it with minimal interruption? You can do this with the Shadow Migration feature. With the Shadow Migration feature, you can migrate the data from one local or remote filesystem to another, while you are already accessing the new one to get the data on the old ZFS filesystem. This feature is available in Solaris 11 as well.

For this demonstration we will use two zfs pools consisting out of files. So we have to create the files first:

root@test:/test/brainslug# mkfile 128m source1
root@test:/test/brainslug# mkfile 128m source2
root@test:/test/brainslug# mkfile 128m source3
root@test:/test/brainslug# mkfile 128m source4
root@test:/test/brainslug# mkfile 128m target1 
root@test:/test/brainslug# mkfile 128m target2
root@test:/test/brainslug# mkfile 128m target3
root@test:/test/brainslug# mkfile 128m target4
root@test:/test/brainslug# mkfile 128m target5
root@test:/test/brainslug# mkfile 128m target6

Now the pools are created. At first our RAIDZ pool consisting out of 4 files. It's named source

root@test:/# zpool create source raidz \
 /test/brainslug/source1 \
 /test/brainslug/source2 \
 /test/brainslug/source3 \
 /test/brainslug/source4

The second one is the future target of the shadow migration. It consists out of six "disks"

root@test:/# zpool create target
  mirror /test/brainslug/target1 /test/brainslug/target2 \
  mirror /test/brainslug/target3  /test/brainslug/target4 \
  mirror /test/brainslug/target5 /test/brainslug/target6

When you did a basic install, the tools and daemons needed for shadow-migration are not included. You have to install them and enable the shadowd afterwards:

root@test:/test/brainslug# pkg install shadow-migration
root@test:/test/brainslug# svcadm enable shadowd

Now you should see the shadowd daemon running.

root@test:/test/brainslug# ps -ef | grep "shadow"
    root  3292     1   0 14:32:33 ?           0:03 /usr/lib/fs/shadowd

Okay … to test the shadow migration we create a filesystem in the source pool:

root@test:/test/brainslug# zfs create source/somestuff

Now we have to fill this file with a some data. Let's create some play data.

root@test:/test/brainslug# dd if=/dev/urandom of=myfile bs=1024 count=300000                        
300000+0 records in
300000+0 records out
root@test:/test/brainslug# mkdir demodata
root@test:/test/brainslug# cd demodata
root@test:/test/brainslug/demodata# split -b 128k -a 5  ../myfile

This should yield a significant number of 128k files. Now we copy them to the newly created filesystem source/somestuffWe will copy the files into the zfs filesystem posing as our old filesystem:

root@test:/test/brainslug/demodata# cp * /source/somestuff/
root@test:/test/brainslug/demodata# cd /
root@test:/# zfs list source
NAME    USED  AVAIL  REFER  MOUNTPOINT
source  294M  42,1M  46,4K  /source

Just to have something to compare, you could simply count the files and calculate the md5 checksum of a file.

root@test:/# ls -l /source/somestuff | wc -l
    2345
root@test:/# md5sum /source/somestuff/xaadmd
3fb4a6be2f93c3d93998db52061244aa  /source/somestuff/xaadmd

Shadow migration will only works, when the source filesystem read-only. So we have to put the source filesystem into such a state:

root@test:/# zfs set readonly=on source/somestuff

Okay, now let's configure the shadow migration:

root@test:/# zfs create -o shadow=file:///source/somestuff \
target/newlocationforsomestuff

That's all. The command may take some moments to get back. The migration of data starts right in the moment you create the new filesystem. It runs in the background and starts to copy all data to the new filesystem. Important to know: You can do shadow migration via NFS as well and it can be an UFS filesystem as well. you just have to declare the source of the shadow migration like nfs://fileserver/directory

Okay. With shadowstat we can check the process of migration.

root@test:/# shadowstat    
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  25,5M   -       -       00:01:10

The cool think about shadow migration is: You can already use the new filesystem. Despite the fact that the migration is still running, you will already see all files and when you access one file it will be migrated in the moment you access the file on the new filesystem. You don't have to wait with the access, until the block would be migrated by the normal background migration. When you try to access data, that isn't already migrated, it's migrated in the moment you access it in the new filesystem.

root@test:/# md5sum /target/newlocationforsomestuff/xaadmd               
3fb4a6be2f93c3d93998db52061244aa  /target/newlocationforsomestuff/xaadmd
root@test:/# ls -l /target/newlocationforsomestuff | wc -l                       
    2345

Afterwards it proceeds with the further migration of all data in the pool. You can observe that with the shadowstat command.

root@test:/# shadowstat 
                                        EST             
                                BYTES   BYTES           ELAPSED
DATASET                         XFRD    LEFT    ERRORS  TIME
target/newlocationforsomestuff  97,8M   -       -       00:01:50
target/newlocationforsomestuff  128M    -       -       00:02:00
target/newlocationforsomestuff  147M    -       -       00:02:10
target/newlocationforsomestuff  165M    -       -       00:02:20
target/newlocationforsomestuff  186M    -       -       00:02:30
target/newlocationforsomestuff  202M    -       -       00:02:40
target/newlocationforsomestuff  211M    -       -       00:02:50
target/newlocationforsomestuff  224M    -       -       00:03:00
target/newlocationforsomestuff  236M    -       -       00:03:10
target/newlocationforsomestuff  243M    -       -       00:03:20
target/newlocationforsomestuff  249M    -       -       00:03:30
target/newlocationforsomestuff  256M    -       -       00:03:40
target/newlocationforsomestuff  260M    -       -       00:03:50
target/newlocationforsomestuff  266M    -       -       00:04:00
target/newlocationforsomestuff  272M    -       -       00:04:10
target/newlocationforsomestuff  278M    -       -       00:04:20
target/newlocationforsomestuff  286M    -       -       00:04:30
No migrations in progress

root@test:/#

Successfuly migrated.

Do you want to learn more?

Docs
docs.oracle.com: Migrating ZFS File Systems"
docs.oracle.com: Migrating File System Data to ZFS File Systems

Blogs
blogs.oracle.com: What is Shadow Migration
blogs.oracle.com: Shadow Migration Internals

... bleiben noch sechs Veranstaltungen (man kann sich noch anmelden). Der Event in Düsseldorf war schon mal ein guter Auftakt. Viele Leute da, viele Fragen gehabt in den Kaffeepausen. Morgen nun Stuttgart ...

Fand meinen Vortrag ein wenig holprig, zuviel Reuse, bin irgendwie nicht in den Flow gekommen. Habe meinen Vortrag aber für die nächsten Veranstaltungen noch ein wenig umgebaut. Und bevor jemand fragt: Ja ... die Slideanzahl ist noch zweistellig Hab doch nur 30 Minuten ....

PS: Mag mal jemand den Flughafenbetreibern in DUS sagen, das ein Hinweisschild ganz nett wäre, das es da einen Starbucks oben bei der Besucherterasse gibt? Ich bin stumpf dran vorbeigelaufen.

Im Februar läuft eine Reihe von Events in Deutschland und der Schweiz zum Thema Solaris 11. Die Events versprechen technisch sehr interessant werden, da die Sprecher jeweils sehr tief in der Materie sind. Über Detlef Drewanz - der bei allen Events dabei ist - muss ich seit dem Containerleitfaden genauso wie über Uli Gräf (der an einigen, aber nicht allen Orten spricht) wohl nichts mehr sagen. Christian Christian Ritzka und Elke Freymann sind ausgewiesene Experten zum Thema OpsCenter. Und ja .. ich halte auch einen Vortrag über Datamanagement in Solaris 11. Und da ich schon zweimal die Frage gesehen habe: Die Veranstaltung ist kostenfrei

08:30 - 09:00	Registrierung
09:00 - 09:15	Begrüßung
09:15 - 10:00	Was ist neu in Oracle Solaris 11 Viele Features, die im Rahmen der Solaris 10 Entwicklung in Solaris 11 Express eingeflossen sind, finden sich auch in Solaris 11 wieder. Diese Präsentation gibt einen Überblick über die neusten Features.
10:00 - 11:00	Oracle Solaris 11 Installation Die wohl herausragendste Eigenschaft von Oracle Solaris 11 ist das neue Package System IPS und der Autoinstaller, die die Installation und das Management von Oracle Solaris 11 vereinfachen. Lernen Sie die neuen Techniken kennen und lassen Sie sich zeigen, wie einfach das Patchen unter Oracle Solaris 11 ist.
11:00 - 11:30	Pause
11:30 - 12:30	Oracle Virtualisierung In Oracle Solaris 11 sind umfangreiche Virtualisierungstechniken integriert. Lernen Sie alles über die neue Netzwerk Virtualisierung in Oracle Solaris 11 und wie sie komplette multi-tier HW Infrastrukturen in einer einzelnen Maschine zusammen mit dem Oracle Virtual Machine framework und Solaris Zonen realisiert werden kann.
12:30 - 13:30	Mittagessen
13:30 - 14:15	Management von IT Infrastrukturen Virtualisierung heist nicht nur "Hypervisor". In diesem Vortrag zeigen wir, wie sich virtualisierte Oracle Solaris 11 Umgebungen zentral verwalten lassen.
14:15 - 14:45	Das Solaris Schulungsprogramm Oracle University stellt zusammen mit unseren Schulungspartnern ein umfassendes Programm zur Vertiefung von Solaris Wissen zur Verfügung. In diesem Vortrag werden die Ausbildungpfade, Kurse und Zertifizierungen für Solaris 11 beleuchtet und verfügbare Lernformen vorgestellt.
14:45 - 15:15	Pause
15:15 - 15:45	Oracle Solaris 11 Datamanagement Oracle Solaris 11 hat umfassende Datamanagement Funktionen integriert. Lernen Sie die neusten ZFS features wie Data Encryption und Deduplikation kennen und wie Sie diese Funktionen über die CIFS-Integration im Kernel auch anderen Plattformen zur Verfügung stellen können.
15:45 - 16:15	Panel, Q&A
16:15 - 16:45	Erfrischungen, Zeit zur Diskussion mit den Experten

Die genaue Agenda mit den Sprechern in den einzelnen Orten und eine Möglichkeit zur Anmeldung findet ihr auf den Eventseiten:

• Düsseldorf 6.2.2012
• Stuttgart 8.2.2012
• Hamburg 9.2.2012
• Potsdam 10.2.2012
• Frankfurt 13.2.2012
• München 14.2.2012
• Zürich 28.2.2012

Um zahlreiches Erscheinen wird gebeten!

Work in Progress - this entry will change often in the next days and weeks

A few days^H^H^H^Hweeks ago, i wrote about simulating the cloud that is most often tagged with the name "network" or "intranet" and sometimes "internet" . This would not be c0t0d0s0.org without an article to explain how you can configure this. This article will explain how you simulate a complete network on a single host with routers, switches, dynamic routing protocols and so on

Scope

At first i want to set the expectations right. I don't want to simulate a cloud in the sense of cloud computing here. I'm thinking about something more complex:



I'm talking about the simulation of this cloud, that often hides a lot of complexities and traps in architectural diagrams.

In the past i wrote quite often about a thing that i call systemic features, when features start to fit together seamlessly in order to create possibilities more than the sum of the features. One of the systemic features is the simulation of the cloud. I don't talk about that thing that most people connect in mind with the word cloud (the grid with a credit card checkout ), but the cloud-like icon in many architectural diagrams called "Network" or "Internet" that sits between the client and the application that often resembles the "a wonder happens here" box in many architectures.

It's not new: I talked about this mid November at the DOAG conference in Nuremberg. And i've playing around with this at customers an privately for a while now.

Many customers have networks as large and as complex as the internet part of a smaller country perhaps 15 years ago. The interesting question is: How can you test your application for it's resiliency against failures in this cloud shaped icon. How does your application react, when your network is doing its high availability magic.

And interestingly Solaris 11 can help you here. The thoughts behind this are pretty simple.

• A router is a computer that runs an operating environment that is tailormade to do network stuff, but at the end it's a computer with a OS (yeah, i know, hardware offloading makes this a little bit more complex, but at the end it's that way)
• A zone is a virtual operating environment.
• Each zone can have it's own set of routes.
• Each zone can have it's own set of firewall rules.
• Each zone can have it's own set of processes.
• Routing protocols are not more than processes collecting information from the network and configuring the routing table.
• You can install a vast array of dynamic routing protocols on a zone.
• I can have up to 8192 zones (given enough memory)
• In Solaris 11 i can emulate switches (etherstubs)
• I can limit bandwidths in Solaris 11 out-of-the-box with crossbow

When i'm combining all this features i can set up a vast array of zones doing nothing else taking each incoming packet on a interface, routing it on a multitude of ways between each other, and send it out on a outgoing interface. Even when the system in your environment are placed in many separate networks of your network you can still use a system with many networking cards or something called server-on-a-stick (single high-bandwidth connection to a vlan-trunking capable switch and using the switch ports as a fan-out).

So in order to emulate a complex corporate network, all i have to do is configuring a lot of etherstubs, configuring many vnics, replicate the physical bandwidths with the maxbw setting on the vnics, set up a lot of zones, perhaps translate the ACL of the routers into firewall rules for firewall functionality of Solaris, installting the routing daemons and configure it similar to the configuration of the routers (in regard of timeouts and so on).

Now i can test, how my applications react, when the network starts to converge against a new topology because of the failures of some lines. I can test, to which topology my network will converge after an line outage (which is nothing more than a deny-all firewall rule). I can test the impact, when the network converges that way, that my traffic flows over a 2 MBit/s line instead of a 155 MBit/s line. For even more complex failure modes i can even use the htbx driver to introduce additional latencies, packet drop or packet reorderung as shown in this article. In essence you can emulate your complete internal network in a single box and with Zones and Crossbow in Solaris 11 it's so low overhead (at the end it is still just one kernel) that you can really emulate the reality and not a simplyfied view, as you don't have emulate via separate hardware or many independent operating system instances in virtual machines.

At the end you could simply use a single Solaris system, put it between all your test systems and use this solaris system as a emulation device for your corporate network. It's simulating the cloud-shaped icon in your architectural diagrams.

ZFS celebrated its 10th birthday on October 31st. So whatever you plan as a filesystem to kill ZFS ... may take a while

Surely you’ve recognized that my blog was down for a few days and with it all services on the system. The problem that led to this situation was a really dumb one. Perhaps this article is more a story about not thinking about a failure mode just because it’s not a problem under your preferred operating system (Or to be exact: It was a problem before Solaris 10, but afterwards it was solved). And and most it's a story about being totally problem-blind in the first moment.

Perhaps I should explain first that c0t0d0s0.org isn’t run with Solaris, it uses this-other-unixoid-operating-system in a well-known non-commercial variant. That’s the dirty secret of c0t0d0s0.org. No technical reason for it, but webserving and mail could be done by any operating system and thus I used that operating system with ubiquitous availability at almost all providers of dedicated servers. I’m able to migrate the server from one dedicated server provider to another within 2 hours including moving the data and did this three times in the past (from 1&1 to Hetzner, and two times within Hetzner). This saved quite significant money until now and that’s the basic reason why I don’t want donations and when you do donations I would donate this money to kiva.org)

Hetzner has reasonably priced dedicated servers and I had no problems in the past, however they have one important shortfall: No serial console in the standard product. When you need a console, you have to make a support call and they connect one. As you need it really seldom, it’s okay. As I found out later: I With this serial console I would have recognized the problem within a minute, and fixed in a second. However: The console was exactly thing that I didn’t had to my disposal at this moment.

So it was a lot harder to find out what’s happened. However i wanted my server back as soon as possible (out of personal reasons I was just able to start the recovery in the evening and as I have job to do I could only do the further stuff in the evenings as well) and thus I just reimaged the server after keeping a copy of the logfiles. I have a quite extensive backup regimen with very regular rsyncs and database replication on my server at home thus I knew I would perhaps just lose an hour of minutes of data and that was okay for me, additionally I was able to mount the disks of the non-working installation and to copy the delta of mails between the last backup and the last mail in the queue to my backup.

What had happened: At 10:something my server provider had a large power outage. The UPS didn’t take over as planed and thus a lot of servers rebooted. One of them was mine. Damned … but that’s the basic reason why I’m a fan of proper enterprise architecture and not of some singular availability features, no matter what marketing tells you. Real availability is hard work and often expensive. But: When you really bet your business on IT, you need an architecture that is even capable to cover an UPS that proofs to be not so uninterruptible. The availability feature UPS may fail (and did fail my case) but a proper enterprise architecture keeps your service up and running. Even more important: With a proper enterprise architecture you don’t need the feature UPS for availability reasons at all because your service can survive the outage of some parts. Perhaps you want the UPS out of other reasons like “don’t want the hassle of bringing up all the systems again.”. But you don’t need it with such an architecture to keep your business running. By having a proper planed enterprise architecture with servers on two seperate sides with different power grids you may forget about the UPS because a UPS won’t help you with a prolonged power outage for example because of region-wide blackout. An outage that maybe will take out the connectivity as well as it’s not that unprobable that your local carrier has the same power problem However my business is being an architect, not blogging and thus I didn’t have such an architecture in place for c0t0d0s0.org But that’s a different story.

Okay: After a while my system worked again and thus I had time to find out what had happened. I knew that the system was still reacting on pings, thus I knew the kernel of this-other-unixoid-operating-system in a well-known non-commercial variant was working. Looking into the logfiles I saw complete bootup of the kernel and some of the daemons were starting up .... like acpid for example. However I couldn’t log into SSH. No signs in the logfiles of a ssh daemon startup. The apache was in a half-reacting state. Port 80 was open but it didn’t reacted to HTTP commands.

Out of this I concluded: The kernel and the boot configuration is okay. The bring up of the services is at least working partially, because otherwise it would start services at all. And as it reacted on the networking there must have been at least a working boot of services mandated by rcS.d, as otherwise there would be no networking. The problem must be in apache that is frozen halfway. And out of other reasons ssh isn’t started at all. There must have been a major fsckup in the startup of the services As I had no console as explained before I needed to conclude from the leftovers what had happened.

And now was a little bit puzzled. 5-6 years ago I would have recognized this problem in an instance (because sometimes i've produced ... well ... suboptimal startup scripts) … but now today it took a while, because I didn’t felt prey for 5-6 years to such a problem. It took me a lot of more thoughts what might had happened. When you do one operating system for a living and one for hobby, you tend to project your mindset of one to the others and you don’t do justice to this other OS.

As you all may know, Solaris ditched init.d with Solaris 10 in order to introduce SMF (not to forget the equally important features like the contract filesystem and the Fault Management Architecture). One of the nice advantages of SMF is that services that aren’t interdependent will be started in parallel without waiting for another. This has two advantages: At first the system can start up much faster, at second a service not able to start up can’t block the startup of the rest (short of services needed by all others).

The init.d concept is a different. All services are started in sequence. The sequence is numerical and then alphabetical. That is of course slower but more important … depending on the way you write your script a script or binary hanging or waiting for user interaction can block the startup. The variant of this-other-unixoid-operating-system is using init.d

And it’s quite easy to block a service. For example by integrating a new SSL key and certificate. My key had a password and apache was asking for this key in order to startup. This exacly happened. Acpi started up because it was started before Apache (guess what: ACpi is before APache in an alphabetical order, and way before Ssh). This is the basic reason why you strip of the password from your key. Guess what I did last week: I put a new key and certificate on my server and I forgot to strip the password from it.

And that exactly happened: In my version of this-other-unixoid-operating-system the ssh daemon is started after the apache daemon. When Apache waits for something you won't get SSH. Damned … it's facepalm time. Basically I felt prey to a beginners error because I’m working with an operating system that reacts totally differently on such situations. On Solaris such a situation just don’t matter at all … you get at least your ssh login and the system non-availability is just a service-non-availability you can fix within a second. However given the init.d system of this-other-unixoid-operating-system the outcome was somewhat more problematic. However: What a dumb error on my side ...

However: The reinstallation wasn't that bad ... the system could used a reinstallation because of some tests and experiments. So it was worth the work in the evenings. And on the other side: Who had the glorious idea to start apache before ssh?

That said, this-other-unixoid-operating-system in newer variants have a different startup mechanism up upstart all the services. However: My heating control is running on a beagleboard-XM at the moment using a really current variant of this-other-unixoid-operating-system just released a few days ago. It uses this new startup mechanism. And it’s justs my unimportant personal preference, that doesn’t matter: But I don’t like it. And I have a lot of reasons for it. It looks by far too much designed for desktop needs. However my dislike would require an article I won’t write in this blog nowadays. But as I wrote: That’s my personal opinion that doesn’t matter.

However it’s really important that this-other-unixoid-operating-system gets away from the old init.d mechanism to something more current. I think in 2011 every operating system deserves something more functional, something better than init.d … init.d is simple and well understood, however it creates classes of problems unnecessary today. Especially: In order to keep die-hard Solaris admins to fall prey to such a beginners error because such problems were parts of their distant past. And now i will start to cut holes for my eyes in the brown paper bag for my head.

There is a nice example of the power of boot environment. Boot environments are something like snapshots of your operating system installation made writeable. As you may already assume, they are based on ZFS snapshots and the clone functionality. This is possible due to the usage of ZFS as the root filesystem.

So: Please don't try this at home. Whey you try it, don't try it on any Solaris 11 Express installation of any value. But don't try it. I don't want to hear any story. that you've deleted your ERP system by accident because you used the wrong terminal window. Leave that to trained professional stunt admins with the right equipment (Solaris 11 Express)

Assume you have a system, configured with all your application, everything is running fine. So you think it would be nice to have something like a freezed state of this situation. No problem. This command will do the trick.

# beadm create rescuenet
# init 6

When you reboot your system you will see it as a new entry in the grub menu.



Okay, but boot into the old environment starting "Oracle Solaris ..." first by selecting it in the grub menu (it should be already selected, or you used beadm activate already. Now i will drop the atomic bomb on your installation.

# rm --no-preserve-root -rf /

Essentially we've just nuked the installation. After a moment the system should just freeze. Reset the system and boot again via grub into the boot environment starting with "Oracle Solaris ...":



Okay ... on a normal system this would send you to the tapes. With Solaris 11: Reset the system. Boot into the boot environment "rescuenet" via selecting it in grub.



Tada! Just creating a boot environment with a single command after a config change may safe your butt later .... and btw ... this even works in zones ... they know the concepts of boot environment,too.

Just a short hint: The What's new document of Solaris 10 Update 9 states, that the support for IPoIB Connected Mode has been added in the release. However you have to search a bit in order for some information how to activate it. The necessary step is documented in the manpage for the ibd driver. Let's assume you have to instances of the ibd driver running (ibd0 and ibd1). In this case you have to change one line at the end of /kernel/drv/ibd.conf file to enable_rc=1,1; and reload the ibd driver respectively reboot the system. After that you ibd devices should show an mtu size of 65520 bytes instead of 2044.

PS: The process for Solaris 11 is better, as you just use dladm for it. However connected mode is the default there anyway. In Solaris 10 unreliable datagram was kept as the default, as one of the rules in Solaris is that you have to opt-in to such changes between updates.