English

In this blog entry i want to write about some small features you wouldn't read about on page 1, but which are really really helpful. Sean Wilcox wrote a nice article about SMF stencils. Stencils are a nice and build-in way to fill-in configuration information from SMF properties into legacy configuration files.

Another small but very nice feature is an extension of netstat. It now shows the pid, the user and the command of the process owning a socket. Casper Dik wrote about it in User, Pid and Commands in netstat(1m)

root@sol112:~# netstat -aun | more

UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.*                                 root        123 in.mpathd      Unbound
      *.*                                 root        123 in.mpathd      Unbound
      *.*                                 netadm      659 nwamd          Unbound
      *.*                                 netadm      659 nwamd          Unbound
      *.111                               daemon      851 rpcbind        Idle
      *.*                                 daemon      851 rpcbind        Unbound
      *.48105                             daemon      851 rpcbind        Idle
      *.111                               daemon      851 rpcbind        Idle
      *.*                                 daemon      851 rpcbind        Unbound
      *.40150                             daemon      851 rpcbind        Idle
      *.*                                 root        870 in.ndpd        Unbound
      *.631                               root        948 cupsd          Idle
      *.68                                root        985 dhcpagent      Idle
      *.546                               root        985 dhcpagent      Idle

With svcs -Lv a really helpful option found it's way to Solaris 11.2 . It's shows the logs for an SMF server. So you don't have to search for the log location anymore.

root@sol112:~# svcs -Lv svc:/network/smtp:sendmail
svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
[ Apr 30 20:48:10 Enabled. ]
[ Apr 30 20:48:10 Rereading configuration. ]
[ Apr 30 20:48:46 Executing start method ("/lib/svc/method/smtp-sendmail start"). ]
WARNING: local host name (sol112) is not qualified; see cf/README: WHO AM I?
/etc/mail/aliases: 13 aliases, longest 10 bytes, 145 bytes total
[ Apr 30 20:49:47 Method "start" exited with status 0. ]

With tcpstat and ipstat two commands from the *stat family were integrated into 11.2. Useful to find out what's happening on your network interfaces.

root@sol112:~# tcpstat -c 1
Please wait...
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global       979 TCP    sol112               22 MBP-of-c0t0d0s0.  51418   64,0
Total: bytes in:  0,0  bytes out: 64,0
ZONE         PID PROTO  SADDR             SPORT DADDR             DPORT   BYTES
global       979 TCP    sol112               22 MBP-of-c0t0d0s0.  51418  336,0
root@sol112:~# ipstat -c 1
Please wait...
SOURCE                     DEST                       PROTO    INT        BYTES
sol112                     MBP-of-c0t0d0s0.fritz.box  TCP      net0       96,0
MBP-of-c0t0d0s0.fritz.box  sol112                     TCP      net0       32,0
Total: bytes in: 32,0  bytes out: 96,0
SOURCE                     DEST                       PROTO    INT        BYTES
sol112                     MBP-of-c0t0d0s0.fritz.box  TCP      net0      448,0
MBP-of-c0t0d0s0.fritz.box  sol112                     TCP      net0       64,0
Total: bytes in: 64,0  bytes out: 448,0

There is a nice "Getting started" guide for Openstack support in Solaris 11.2 written by David Comay: "Getting Started with OpenStack on Oracle Solaris 11.2". It's part of a large heap of information available about this topic at OTN.

Glenn Faden is writing about Authenticated Rights Profiles. Now you can assign administrative rights to a user, so she or he doesn't need to assume a role with the role password but still require him to authenticate.

A second really interesting new feature of Solaris 11.2 Glenn is writing about is the "Qualified User Attributes" feature, this is answering the challenge to limit for example an expanded set of privileges to a host or a group of hosts when you are putting the user information into LDAP. Qualified User Attributes solves this nicely.

By the way, such nice feature as limiting access or privileges to a certain timespan is available as well is in Solaris 11.2 as well. So you can just allow your junior administrator to do certain tasks for example just on weekdays between 09:00 and 17:00 .

Mike Gerdts points in this blog entry to the various pieces of information regarding Kernel Zones and shows how to install a Kernel zones in three easy steps. Just a tip: Please keep in mind, that you can't do Kernel Zones inside a Virtualbox environment out of obvious reasons. Another blog entry is about the new dynamic reconfiguration of zones. In "Need another disk in your zone? No problem!" he is talking how you can add for example a disk to a zone without rebooting the zone.

You may ask why Oracle made such a large fuzz around Solaris 11.2 ... however a large load of new features found their way into this release. I just want to point you to some sources of information about all this new things.

Had a lot of used days in the last few days:

Day n-8: Eastern ... a few days off ... bank holiday. What a nice opportunity to get a sinusitis. Okay ... bad mood and headache galore. Will be a nice weekend. Yay!

Day n-6: Oh, friendly people at the counter in the airline lounge in HAM. I just want to scan my boarding card to get into the lounge and sit down. Get a coffee and forget the guy who drove with 130 on the fast lane almost from Lüneburg to Hamburg. Waited for a moment. Normaly it's not a problem just to hold your boarding card on the reader even while people are talking with other customers. Otherwise the usual high-traffic hours in that lounge would lead to a queue from A40 to the security in HAM. I definitely don't want to wait until you have stop talking with another guest. And i definitely don't want to wait after such a unfriendly comment telling me that i should wait. Sorry.

Day n-6: A long queue at rental car company. What a nice time to to a lunch break. At least in the opinion of the people there at Frankfurt airport station of the rental car company. And the "manager on duty" just looked and did nothing, nor stopping the employee to go into lunch break nor doing some work on her own. And then the car was at the end of nowhere. Seemingly the last possible parking bay in the last possible rental car parking lot. Is this the way of the rental car people telling you "F*ck you" because you was just minimally friendly after waiting between 10 and 15 minutes to get to the counter.

Day n-6: Oh nice ... a hotel room at the top of the hotel. How nice. Just a little bit to warm. A little bit. Or did they gave me the key card for the sauna. As no naked people opened the door, it must be really my room ... but better lock the door from my side.

Day n-5: Waking up. I'm sure the official anthem of people with sinusitis is "Pretty Paracetamol" by Fisher-Z. This song is 35 years old now .... yet another reason to get bad temper.

Day n-5: Where is the key of my car? Not for the rental car. Found that at exactly the same location i dropped it. For the car waiting in the airport parking lot in Hamburg. Checking travel time with my navigation device ... enough time. Doing a full search in the hotel room. Dang. It's not there. Looking into my backpack . Ouch ... cutting myself with a razor inside my backpack. Who was the idiot putting it into my backpack instead of the bag with the foam, soap and blades. Oh that was me.Dang. And no band-aid in reach. A piece of toilet paper must be sufficient. Checking travel time with my navigation device again ... Safety margin down to 5 minutes.Doing another search. Still not there. Safety margin almost eaten away. I have to leave the hotel. Looking around in the room a last time to check for forgotten things.

Looking onto the desk in the room. Who has slaughtered a pig? Oh, it was me, the pig and the pig slaughtering the pig ... back into the bathroom. Cleaning up the desk. Safety margin away. Checkout was faster than expected. Looking around in the rental car. Thinking how to get to my second car key. Driving to the Oracle office nearby. Want to get my badge out the backpack. The badge is not at the usual location. But my car key. WTF. At least i've reached the customer perfectly at the planned time.

Day n-5: Left the customer at the latest possible moment with no extra time or safety margin. Glad i refueled the car before driving to the customer. I don't know how how often I drove a car to the rental car return of Frankfurt Airport. I never took the wrong turn there because i know it gets difficult from there. However today i took the wrong turn to Terminal 2 instead of Terminal 1. I didn't knew that i can run that long with a notebook in my backpack and in a suit ... i'm really glad nobody shouted "Jumanji", going through security in record time and being at the gate at boarding time sharp ... only to learn that the plane is boarding 10 minutes later than expected ... and we waited even longer for pushback.

Day n-4: Just learned that i left my bag with all cables except the power supply of my notebook in Frankfurt. Damned. Really: ALL cables ranging from Ethernet Adapters to Charger for my mobile phone and for my iPad. Glad a friendly colleague took the bag with him and send it to me.

Day n-3: I almost hit a woman on a bicycle almost directly in front of the Oracle office. That was really a close call. Just didn't saw her, because i've concentrated on the "a**hole" bicycle driver driving that passed the road shortly before. I stopped immediately right on the street. She said, nothing happened except the "shock" of the close call. I have explained what happened. As i stopped my car directly on the road, I had to drive my car into the garage, shouted that i will back in a few minutes, however when I got back to the location the woman wasn't there any longer. I can just hope that really nothing happened.

Day n-0: I was attacked today. By a bottle of Ketchup. I just wanted some on my fries for lunch. However i opened the bottle and instead of my fries i got the ketchup direct into my face. Think of a really naughty thing and you have the picture.

Solaris 11.2 is a major step for Solaris, so you can expect to hear a lot of interesting news in the upcoming announcement webcast. You shouldn't miss it. The webcast takes place at April 29, 2014, starts at 01:00 PM and ends at 02:30 PM (both times are Eastern Time zone)

This isn't something new however given that i saw this problem at a customer just last week, i would like to point to something you have to keep in mind when using Oracle DB with the Veritas Filesystem.

When you have strange performance problem when using VXFS please ensure that the database isn't sitting around in the POSIX inode r/w-lock with it's database writes. Check if Quick-I/O (solves the issue) or ODM (solves the issue as well) is really activated. QIO is not activated by just using the FILESYSTEMIO_OPTION=setall or by mounting the filesystem with the mount option qio. You have to do more and it looks like this is sometimes forgotten by people setting up or migrating a system.

So far this test was the fastest for me because it checks if something is present that shouldn't present when QIO is in action:

echo '::threadlist -v' | mdb –k | sed 's:^$:§:' | tr -d '\n' | tr '§' '\n' | grep 'vx_rwsleep_rec_lock' | tr -s ' ' | cut -d ' ' -f 10 | sort | uniq -c

When this command line returns with oracle processes containing "dbw" you should really check if you have properly configured QIO or ODM. Please refer to the VXFS documentation in regard of the correct installation and configuration steps.

You will find the explanation for this at this rather old blog entry: "Hunting red herrings"

The information about Heartbleed in regard of Oracle Products is now available in OTN as well: OpenSSL Security Bug - Heartbleed / CVE-2014-0160.

I got some questions regarding Heartbleed and Oracle products from blog readers. In this regard i just want to link to the entry in the Oracle Security Blog: "‘Heartbleed’ (CVE-2014-0160) Vulnerability in OpenSSL ". The author states:

Oracle recommends that customers refer to the My Oracle Support Note Doc ID 1645479.1 for information about affected products, availability of fixes and other mitigation instructions.

In a some customer situation i'm using a number of Oracle Sun Flash Accelerator F40 PCIe Cards or F80 PCIe cards to create flash storage areas inside a server. For example i had 8 F40 cards in a server by using a SPARC M10 and a PCIe Expansion Box which enables you to connect up to 11 F40/F40 cards per expansion box.

The configuration with 8 F80 cards for example is a configuration i'm using on very special occasions and for special purposes, in this case it was a self-written application of a customer needing a lot flash storage inside the server. I won't disclose more. On the other side: I'm sizing quite frequently systems with two F80 cards for "separated ZIL" purposes . Either if you use the SSD as data storage or as separated ZIL: When you do mirroring you have to ensure that mirrors are not using mirror halves on the same card.

From the systems perspective you see four disk devices per F40/F80 card with 100 respective 200 GB capacity per disk and thus you can just add them to your zpool configuration. However configuring the system was a little bit unpractical. The problem: It's not that easy to create a configuration that ensures that no mirror has it's two vdevs on a single F40/F80 card. Perhaps there is an easier way, however I didn't found it so far.

It's a little bit hard to find acceptable disk pairs when you are looking on PCI trees like /devices/pci@8000/pci@4/pci@0/pci@8/pci@0/pci@0/pci@0/pci@1/pci@0/pci@1/pciex1000,7e@0/iport@80:scsi. Well, at two cards it's not that hard, but still not a nice job. After doing this manually a few times, i thought that at 8 or 22 cards doing this manually is a job for people who killed baby seals, listen to Justin Bieber or equivalent horrible things.

But i didn't committed to such crimes and this problem is nothing that a little bit of shell-fu can't solve. You can do it in a single line of shell. Well ... a kind of a single line of shell.

Ich habe ja schon vor einigen Tagen geschrieben, das Anfang Mai ein Oracle Business Breakfast zum Thema Oracle DB Tuning auf Solaris 11 in München stattfindet. Es ist nunmehr die Einladung da und der Anmeldelink aktiv. Ihr könnt euch hier anmelden.

Zum Inhalt:

Zwar sind die Defaults von Solaris und der Oracle Datenbank für den durchschnittlichen Anwendungsfall gut gewählt, doch manchmal möchte man seine Datenbank doch noch weiter tunen und mehr herausholen oder auf die Spezialitäten der eigenen Last anpassen. Dieser Vortrag wird eine Einführung in die Stellschrauben der Oracle Datenbank und von Solaris geben und erklären, wie sie funktionieren, warum sie funktionieren, wie sie interagieren und Hinweise für den Einsatz in der Praxis geben. Weiterhin soll der Vortrag erklären, wie man aus dem AWR-Report auf System- oder Hardwareprobleme schließen kann.
Abgerundet wird die Veranstaltung durch die Vorstellung der neuen Version des Oracle Enterprise Manager Ops Centers zur vereinfachten Administration - insbesondere für die Oracle VM auf SPARC (aka LDOMs).

Despite popular belief ethernet networks aren't lossless, on the long way between the TCP/IP stack from one side to the TCP/IP-stack of the other side there is a lot that can happen to the data. Datacenter Bridging is a mechanism to put some kind of losslessness on the network needed for shoehorning storage protocols on it (FCoE). That said most of the time ethernet networks appear as lossless however it's not guaranteed.

But that are a lot of different stories i won't write about today. That said you can make some errors in configuration that make ethernet networks more lossy than necessary and those can even haunt you in relatively simple configuration like a link aggregation.

There is a new roadmap available about the future development of SPARC at oracle.com covering the time until 2019. Especially interesting because of the list of software in silicon features planed for the next-gen iteration of SPARC. At while i'm at it: Solaris 12 is on the roadmap as well.

Given that nothing important changes (world stops turning, leaving company, illness, forget to book a conference room, outbreak of the Minbari space plague) i will doing the Oracle Oracle DB Tuning presentation i have already held in DUS and will hold in HAM on 10.4. a third time. Current state of planing is the 9.5 in Munich. The registration link will follow as soon as it's available.

You are virtuously using mdb -w to tune a system and type all the commands from memory ... however you can still fsck up the whole system with a single mistake while using it. The mistake in my case was thinking i'm in the left window whereas i was in the right window. In the left window i wanted to exit out of a shell. In the right window was a waiting mdb -kw(yes, the basic mistake to have it left open). At the end the system was doing a panic right after leaving mdb. An explanation for this behaviour can be found here. When using mdb -kw do it like Swigert in the Apollo 13 film ... put a Post-it on your screen saying "Don't exit". Control+D is your way to leave the mdb.