English

Before i forget someone: A very big "THANK YOU!" for all the nice wishes and congratulation for my birthday yesterday!

As much as there is often a lot discussion about configuration items inside the SMF repository (like the hostname), it brings an important advantage: It introduces the concept of dependencies to configuration changes. What services have be restarted when i change a configuration item. Do you remember all the services that are dependent on the hostname and need a restart after changing it? SMF solves this by putting the information about dependencies into it configuration. You define it with the manifests.

However, as much configuration you may put into SMF, most applications still insists to get it's configuration inside the traditional configuration files, like the resolv.conf for the resolver or the puppet.conf for Puppet. So you need a way to take the information in the SMF repository and generate a traditional config file with it. In the past the way to do so, was some scripting inside the start method that generated the config file before the service started.

Solaris 11.2 offers a new feature in this area. It introduces a generic method to enable you to create config files from SMF properties. It's called SMF stencils.

Pervasive monitoring is a technical attack that should be mitigated in the design of IETF protocols, where possible.

'nuff said.

This blog entry wasn't on my radar somehow, nevertheless it reports about an important change to OpenSSL on Solaris 11.2. The most important change from my point of view is the inlining of T4/T5 crypto to Solaris 11.2 openssl:

Years and years ago, I worked on the SPARC T2/T3 crypto drivers. On the SPARC T2/T3 processors, the crypto instructions are privileged; and therefore, the drivers are needed to access those instructions. Thus, to make use of T2/T3 crypto hardware, OpenSSL had to use pkcs11 engine which adds lots of cycles going through the thick PKCS#11 session/object management layer, Solaris kernel layer, hypervisor layer to the hardware, and all the way back. However, on SPARC T4/T4+ processors, crypto instructions are no longer privileged; and therefore, you can access them directly without drivers. [...]
What does that means to you? Much improved performance! No more PKCS#11 layer, no more copy-in/copy-out of the data from the userland to the kernel space, no more scheduling, no more hypervisor, NADA! [...]

Darren Moffat wrote an interesting blog entry about the security concept of Kernel zones. In "Overview of Solaris Zones Security Models". He is especially talking about a very small, but very very interesting detail:

Note that what follows is an out line of implementation details that are subject to change at any time: The kernel of a Solaris Kernel Zone is represented as a user land process in a Solaris non global zone. That non global zone is configured with less privilege than a normal non global zone would have and it is always configured as an immutable zone. So if there happened to be an exploit of the guest kernel that resulted in a VM break out you would end up in an immutable non global zone with lowered privilege.

Now you have a working Puppet testbed in your Solaris 11.2 beta installation it's time to try some Solaris specific stuff. Oracle a number of additional stuff in order to control Solaris specifics like boot environments, VNICs or SMF. You can find the respective code at java.net.

At the recent announcement we talked a lot about the Puppet integration. But how do you set it up? I want to show this in this blog entry.

However this example i'm using is even useful in practice. Due to the extremely low overhead of zones i'm frequently seeing really large numbers of zones on a single system. Changing /etc/hosts or changing an SMF service property on 3 systems is not that hard. Doing it on a system with 500 zones is ... let say it diplomatic ... a job you give to someone you want to punish.

Puppet can help in this case making of managing the configuration and to ease the distribution. You describe the changes you want to make in a file or set of file called manifest in the Puppet world and then roll them out to your servers, no matter if they are virtual or physical. A warning at first: Puppet is a really,really vast topic. This article is really basic and it doesn't goes more than just even toe's deep into the possibilities and capabilities of Puppet. It doesn't try to explain Puppet ... just how you get it up and running and do basic tests. There are many good books on Puppet. Please read one of them, and the concepts and the example will get much clearer immediately.

To show this, i have a relatively simple setup. A global zone and three non-global zones. In this example i will set up the global zone as a Puppet master and will configure three non-global zone with the Puppet agent, the component that gathers information from the Puppet master in order to do something on the server running the agent.

The Puppet master is called master, the agents are called agent1, agent2 and agent3. This tutorial assumes that you have already set up the zones and that they have working network connection.

Stefan Hinker wrote an interesting blog entry about recent performance improvements with LDOMs in his article "Improved vDisk Performance for LDoms":

In conclusion, with the new, improved virtual networking and virtual disk IO that are now available, the range of applications that can safely be run on fully virtualized IO has been expanded significantly. This is in line with the expectations I often find in customer workshops, where high end performance is naturally expected from SPARC systems under all circumstances.

Casper is - as usual - a good information about Solaris: In his blog he is writing about some changes in Solaris 11.2: For example he is writing about the removal of unlink/link for directories in "Solaris 11.2: unlink(2)/link(2) for directories: your time is up.". Another interesting article is "Solaris 11: Evolution of v_path." about the changes to the vnode-to-pathname mapping in the light of newer Solaris 11 features.

A hint for people trying out kernel zones: When you just enter the commands some blogs or websites were posting you may run into the problem that the zone won't boot and the boot fails with a message indicating lack of memory. When you want to use kernel zones please limit the ARC cache of ZFS like This can be done with a entry like set zfs:zfs_arc_max=0x40000000(for an 1 Gig ARC) in /etc/system. This is necessary due to the fact that a kernel zone is allocating a lot of memory in a short time and sometimes the ZFS arc can't get out of the way fast enough.

Sometimes small options are really useful. When you enable a service, for example the Apache HTTPD, you enter svcadm enable apache22. This command immediately return. There is no direct feedback at the command return, if the service has started. So you often see people doing something like that:

root@kusanagi:~# svcs -a| grep "apache22"
disabled       22:16:52 svc:/network/http:apache22
root@kusanagi:~# svcadm enable apache22
root@kusanagi:~# svcs -a| grep "apache22"
offline*       22:17:20 svc:/network/http:apache22
root@kusanagi:~# svcs -a| grep "apache22"
offline*       22:17:20 svc:/network/http:apache22
root@kusanagi:~# svcs -a| grep "apache22"
offline*       22:17:20 svc:/network/http:apache22
root@kusanagi:~# svcs -a| grep "apache22"
offline*       22:17:20 svc:/network/http:apache22
root@kusanagi:~# svcs -a| grep "apache22"
online         22:17:31 svc:/network/http:apache22

ust before you ask ... i included a sleep 10 at the right place in the method script of the http:apache22 service to delay the startup in order to demonstrate the command. The service goes from disabled to offline(the state where a service is enabled but the startup hasn't completed) and is flagged as a service in transition to a different state (like online) by the *.

However you can change this behaviour. When you use the -s the svcadm enable command just returns when the state transition has completed. So for example when the service got from offline to online. The svcadm command with the -s option will return as well when it determined that it can't do the desired state transition and needs admin intervention.

In our example we would use svcadm enable -s apache22. As we artificially delayed the startup of the Apache, the svcadm command should run at least 10 seconds. Let's check this:

# ptime svcadm enable -s  apache22

real       11.137908105
user        0.012195633
sys         0.018084807

This synchronous mode works with svcadm enable as well as with svcadm disable even in Solaris 10. In Solaris 11.2 the synchronous mode was introduced to all svcadm subcommands including the mechanism to define a timeout after which the svcadm command return with an error code that the state transition hasn't completed with the timeout. However it's just the waiting for the state transition that is timouted, the service itself proceeds with it's bring up.

This is a nifty small tool that i'm using quite often in scripts that stop something and do some tasks afterwards and i don't want to hassle around with the contract file system. It's not a cool feature, but it's useful and relatively less known. An example: As i wrote long ago, you should never use kill -9 because often the normal kill is intercepted by the application and it starts to do some clean up tasks first before really stopping the process. So just because kill has returned, it doesn't imply that the process is away. How do you wait for process to disappear?

Long time readers of my blog know that i'm preferring prstat over top at any time. The micro state accounting in prstat gives you a much deeper insight. Using a tool not capable to use microstate accounting is like looking a video in 240p instead of 4k ultra hd (to stay at this: dtrace is like 8k ). prstat is doing a really useful job in telling you what's happening at the moment in a processes.

However sometimes it's interesting to know, what happened in the past since the startup of the process. And there is a tool that is doing this. With ptimes -m you can lookup the information of the micro state accounting since the creation of the process:

root@kusanagi:~# ps -ef | grep "ssh"
jmoekamp  6174  6173   0 19:57:24 ?           0:00 /usr/lib/ssh/sshd
    root  6200  6188   0 19:59:06 pts/1       0:00 grep ssh
    root   539     1   0   Mai 01 ?           0:00 /usr/lib/ssh/sshd
    root  6173   539   0 19:57:24 ?           0:00 /usr/lib/ssh/sshd
root@kusanagi:~# ptime -m -p 6174

real     1:52.355845038
user        0.060116522
sys         0.049137178
trap        0.000004847
tflt        0.000000000
dflt        0.000000000
kflt        0.000000000
lock        0.000000000
slp      1:52.216832767
lat         0.029398950
stop        0.000010032

I'm using the tool quite often to get snapshots of the values for a process, writing it to a file and calculating the differences let's say over multiple hours. For this task it's much more practical than using prstat. A description of the values is in the man page, however it's pretty much the same as in prstat -m

Do you want to learn more

docs.oracle.com - manpage - ptime(1)

Yet another interesting blog article written by Casper Dik: In"No limits" he reports about removing certain limits in Solaris 11.2

In this blog entry i want to write about some small features you wouldn't read about on page 1, but which are really really helpful. Sean Wilcox wrote a nice article about SMF stencils. Stencils are a nice and build-in way to fill-in configuration information from SMF properties into legacy configuration files.

Another small but very nice feature is an extension of netstat. It now shows the pid, the user and the command of the process owning a socket. Casper Dik wrote about it in User, Pid and Commands in netstat(1m)

root@sol112:~# netstat -aun | more

UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.*                                 root        123 in.mpathd      Unbound
      *.*                                 root        123 in.mpathd      Unbound
      *.*                                 netadm      659 nwamd          Unbound
      *.*                                 netadm      659 nwamd          Unbound
      *.111                               daemon      851 rpcbind        Idle
      *.*                                 daemon      851 rpcbind        Unbound
      *.48105                             daemon      851 rpcbind        Idle
      *.111                               daemon      851 rpcbind        Idle
      *.*                                 daemon      851 rpcbind        Unbound
      *.40150                             daemon      851 rpcbind        Idle
      *.*                                 root        870 in.ndpd        Unbound
      *.631                               root        948 cupsd          Idle
      *.68                                root        985 dhcpagent      Idle
      *.546                               root        985 dhcpagent      Idle

With svcs -Lv a really helpful option found it's way to Solaris 11.2 . It's shows the logs for an SMF server. So you don't have to search for the log location anymore.

root@sol112:~# svcs -Lv svc:/network/smtp:sendmail
svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
[ Apr 30 20:48:10 Enabled. ]
[ Apr 30 20:48:10 Rereading configuration. ]
[ Apr 30 20:48:46 Executing start method ("/lib/svc/method/smtp-sendmail start"). ]
WARNING: local host name (sol112) is not qualified; see cf/README: WHO AM I?
/etc/mail/aliases: 13 aliases, longest 10 bytes, 145 bytes total
[ Apr 30 20:49:47 Method "start" exited with status 0. ]

With tcpstat and ipstat two commands from the *stat family were integrated into 11.2. Useful to find out what's happening on your network interfaces.

root@sol112:~# tcpstat -c 1
Please wait...
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global       979 TCP    sol112               22 MBP-of-c0t0d0s0.  51418   64,0
Total: bytes in:  0,0  bytes out: 64,0
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global       979 TCP    sol112               22 MBP-of-c0t0d0s0.  51418  336,0
root@sol112:~# ipstat -c 1
Please wait...
SOURCE                     DEST                       PROTO    INT        BYTES
sol112                     MBP-of-c0t0d0s0.fritz.box  TCP      net0       96,0
MBP-of-c0t0d0s0.fritz.box  sol112                     TCP      net0       32,0
Total: bytes in: 32,0  bytes out: 96,0
SOURCE                     DEST                       PROTO    INT        BYTES
sol112                     MBP-of-c0t0d0s0.fritz.box  TCP      net0      448,0
MBP-of-c0t0d0s0.fritz.box  sol112                     TCP      net0       64,0
Total: bytes in: 64,0  bytes out: 448,0

There is a nice "Getting started" guide for Openstack support in Solaris 11.2 written by David Comay: "Getting Started with OpenStack on Oracle Solaris 11.2". It's part of a large heap of information available about this topic at OTN.