English

Surprisingly positive article about the SPARC M6 and Bixby presentations at Hotchips25: Oracle revs up Sparc M6 chip for seriously big iron

I think we will see some news in the usual news outlets about upcoming SPARC silicon soon. The agenda of the currently running Hotchips 25 conference has three presentations related to SPARC:

• "SPARC64 X+ : Fujitsu’s next generation processor for the UNIX servers" by Toshio Yoshida of Fujitsu
• "SPARC M6: Oracle’s Next Generation Processor for Massively Scalable Symmetric Multiprocessor (SMP) Data Center Servers with Enterprise Class RAS" by Ali Vahidsafa and Sutikshan Bhutani of Oracle
• and "Bixby: the Scalability and Coherency Directory ASIC in Oracle’s M5/M6-32 Systems" by Thomas Wicki and Juergen Schulz of Oracle

For a long time the maximum number of groups a user could belong to was 16, albeit there was a way to get 32. In Solaris 11 and recent versions of Solaris 10, the maximum number of groups a user could belong to is 1024 (which is the same limit Windows sets in this regard). It's easy to set the new limit.

set ngroups_max=1024

After a reboot, this change will be active. But why isn't this the default? There are good reasons for it. I will show you one of them in this entry. Like thinking that two digits for the year or using a signed 32-bit integer for storing the system time, the issue has it's root cause in a decision made a long time ago … in this example the moment in the past is at least 25 years ago. And often just changing something, breaks stuff that is really old, but still in use.

Experienced Solaris users, who tuned their Solaris System for up to 32 groups per user, already know the component that will be broken by having more than 16 users, because a message at the next boot of the system after the change in /etc/system that next startup will deliver a warning. It's about the problem that many people encounter when using NFS with one user in more than 16 groups (it's not an NFS problem, but i will explain that).

However, as i already said, there is a a solution for this problem since Solaris 11.1. This blog entry will show the workaround in action.

Interesting (recently updated) whitepaper about using power management on T-class servers:"How to Save Power on SPARC T5 and SPARC M5 Servers".

Oracle VM for SPARC 3.1 has been announced. You will find the documentation here. There is a multitude of new features in this release. Honglin Su writes about the news in the blog article "Announcing Oracle VM Server for SPARC 3.1 Release" and Jeff Savit writes about it as well.

In my personal opinion the extensions to IOV and the massive speed increase when using fully virtualised networking are the most interesting changes in this release.

With the speed increases with virtualised networking you can bandwidths between two logical domains in the range of 16 Gbps as explained by Raghuram Kothakota. On the path domain to external networks the throughput approaches wire-speed at 9 Gbps.

However the real nice trick is described in a blog article Raghuram wrote later on: "Live migration of a Guest with an SR-IOV VF using dynamic SR-IOV feature!". This is a really clever way to live migrate a domain and using SRIOV. This method is leveraging IPMP. Nothing hinders you to have one SRIOV-virtualized interface and a fully virtualised interface in one IPMP group. When you set the the fully virtualised interface in standby mode, the SRIOV virtualized one is the interface that carries your load. When you want to live migrate, you just remove the SRIOV interface, IPMP now switches to the fully virtualised interface, you migrate the domain, and add the SRIOV interface again to the new system and it's put into the IPMP group again.

Solaris 11.1 has now the rsyslog on board, an extended variant of the normal syslog. However the ORACLE SOLARIS 11.1—WHAT’S NEW is mistakable in this regard. Indeed the normal syslog is activated by default and it's still activated when you have rsyslog on your machine. However rsyslog is not installed by default. When you try do disable the normal syslog and enable the rsyslog, you will get a message that there is no rsyslog.

root@ca:~# svcadm enable svc:/system/system-log:rsyslog
svcadm: Pattern 'svc:/system/system-log:rsyslog' doesn't match any instances

But it's pretty easy to install it.

root@ca:~# pkg install rsyslog
           Packages to install:  5
       Create boot environment: No
Create backup boot environment: No
            Services to change:  1

DOWNLOAD                                PKGS         FILES    XFER (MB)   SPEED
Completed                                5/5       117/117      2.5/2.5  227k/s

PHASE                                          ITEMS
Installing new actions                       262/262
Updating package state database                 Done
Updating image state                            Done
Creating fast lookup database                   Done

Okay, now there is an instance of rsyslog in the SMF:

root@ca:~# svcs -a | grep "system-log"
disabled        6:13:13 svc:/system/system-log:rsyslog
online          6:47:53 svc:/system/system-log:default

Now you switch to rsyslog:

root@ca:~# svcadm disable svc:/system/system-log:default
root@ca:~# svcadm enable svc:/system/system-log:rsyslog
root@ca:~# svcs -xv

And now you can start to configure the syslog in /etc/rsyslog.conf

I wrote about it a while ago, but in the light of recent questions from customer i want to reiterate this, because with the massive compute power of a T5 or T4 i'm seeing higher an higher process counts on customer systems in the recent time.

When you are runing quite a number of zones and running applications with a lot of threads and your system sends you messages like "cannot fork: Resource temporarily unavailable" in all your zones in parallel and you are not running Solaris 11.1, you should do the following checks. The following checks are for a system without any changes in this regard of segkp in the /etc/system. The numbers used in this example are obfuscated by rounding from a real-life example.

Check at first for the number of threads:

# prstat -cmL 1 1| grep lwps
Total: 2539 processes, 56489 lwps, load averages: 13.11, 14.71, 13.33

If the number of threads is close to 64000, check the usage of the segkp memory.

$ kstat -p | grep "segkp"  | grep ":mem_"
vmem:170:segkp:mem_import 0
vmem:170:segkp:mem_inuse 2100000000
vmem:170:segkp:mem_total 2140000000

If mem_inuse is close to mem_total, check if segkp-allocations have failed (Shortcut: Just check for the alloc_fail).

unix:0:segkp_32768:alloc_fail 2492500
vmem:170:segkp:fail 2492500

If this number is significantly larger than 0 and the number is increasing when measuring two times in let's say a minute, you have to increase the segkp area. You can do this by putting the following line into the /etc/system and reboot the system.

set segkpsize=0x80000

The explanation for this issue and why it isn't a problem in Solaris 11.1 is in "Oh my god, it's full of threads ... and out of memory".

Before you ask: The default in Solaris 10 and 11 was choosen, because the segkp is allocated at startup in the configured size. When you have somewhat more normal numbers of threads, you don't need that large segkp and you could use it better for other stuff. That said, Solaris 11.1 has changed that. It's now calculated at system start on the basis of a number of parameters. I've describted in the linked older article.

There is an easy way to know for sure that you are now an old geezer: You remember sitting in front of a video showing a demo developed by Future Crew back in 1993 you've tried as a burn-test for your new system in 1995. This demo is now almost 20 years old. When you look at it, it doesn't look that much, but you have to keep in mind the time when this demo was published. A dumbphone of 2013 has more compute power and more graphics support than the system used to display such a demo in 1993. h/t to Shawn Walker to point me to this video on his Facebook page, just want to post on my blog for all the other old geezers reading my blog:


(click here to watch)

You have allowed junior to edit the httpd.conf and you are capable to monitor the changes with pfedit. However there is a little problem. She or he can't restart the apache demon to make the new config active. When junior tries to restart , he or she just gets:

junior@template:~$ svcadm refresh apache22
svcadm: svc:/network/http:apache22: Permission denied.

Of course you don't want to restart the service every time junior changes the the config yourself. On the other side you don't want to give junior the root privileges. So what can you do?

You have allowed junior to edit the httpd.conf and and some nice evening, you are sitting at home. Then: You get alerts on your mobile: Webserver down. You log into the server. You check the httpd.conf. You see an error. You correct it. You look into the change log. Nothing. You ask your colleagues, who made this change. Nobody. Dang. As always. Classic "Whodunit".

Okay, in order to prevent this for future changes, you want to record this kind of information. And working with pfedit is really useful in order to do so.

This tutorial is a follow-on to the basic pfedit tutorial. So when you want to work through this one, you have to go through the basic one first.

It's a really nifty feature: Let's assume, you have a config file in your system and you want to allow your junior fellow admin to edit it from time to time, but don't want him to pass any further rights to him, because this machine is too important.

Solaris 11.1 has an interesting feature to delegate the privilege to edit just a file. The tool enabling this is called pfedit.

A whitepaper written by Ruud van der Pas, Jeff Savit, Jeff Victor, Darryl Gove, and Harry Foxwell: READ_ME_FIRST: What Do I Do with All of Those SPARC Threads?. Really worth a read!

A disclaimer first: I found an undocumented option of a Solaris 11.1 command. Again: This is an undocumented option. So: It can break everything. It will probably break everything. Because it's undocumented, it's officially not there. And it's still undocumented, as my blog isn't the documentation of any product of Oracle. It's just a report about an discovery i made. The option, that's officially not there, can disappear at any time and without further notice. The world may be going down when using it. And a laser could rasterize you and suck you into the computer. At least when you use it wrong, you can create a lot of havoc with it.

Sometimes you find out interesting stuff just by Google and trying something out. Yesterday i've searched for some information for ZFS on FreeBSD (one the OSes i keep in my OS zoo) and in the course of it, i found a mail on a FreeBSD list talking about the command zpool import -T . -T? There is no -T option on zpool import? Or is it just in FreeBSD?

However i couldn't believe that there is a feature that interesting in ZFS on FreeBSD but not available on Solaris. So i booted up my Solaris 11.1 in my virtual box. At the end i found out how to import a zpool with a state back in the past based on the transaction group without using snapshots.

I'm playing around with Airmail for reading mails. While configuring this application, i saw the nuclear option to bring down any mail traffic in the company for days with "remove me from this discussion" mails or private comments/insults/gossip that never should have been send to all people:



I hope the developers of Airmail know that they have created a monster

Just saw that i totally forgot the 9th birthday of c0t0d0s0.org on July 15th. However i spend most time that i would invest in my blog into the Solaris 11 release of LKSF and had to pull some rabbits out of the hat for colleagues . Hope that i have more time for the blog in the future.