c0t0d0s0.org

root was sitting in SuperUser castle and everything was fine in the kingdom. But then a loud squeaking and creaking noise found root's attention. The demons of hypertext wrote into the Scrolls of Log, that they couldn't fulfill their work any longer, as the sole bridge into the vast kingdom of root lowered at this moment was broken.

This article isn't really about a feature, it's about a directory and its misuse. Furthermore it's an article about different default configurations, that lead to misunderstandings. This is a pretty old hat for experienced Solaris admins (many of them learned it the hard tour), but it seems to be totally unknown to many admins new to the business or for people switching from Linux to Solaris, as many distribution are configured in a different way per default. A reader of my blog just found a 2GB .iso in /tmp on a Solaris system and that's not really a good idea. A few days ago, a user in twitter had vast problems with memory usage on a system which boiled down to a crowed /tmp.

One of the basic objects in the new Crossbow stack is the flow. Any network traffic is separated into such flows. And with this flows you can do several interesting things. In this article i want to present two usages of them: Bandwidth Limiting and Flow Accounting

After a long time i found time to integrate all the typo and error corrections provided by Ceri Davis into the main LKSF document. Thus i´ve updated the document today. You can download it at the usual location at c0t0d0s0.org. I´ve added the kssl tutorial, too.

There is an interesting feature in Solaris. It is called kssl. One component of this feature is obvious: SSL. So it has something to do with SSL encryption. As you may have already guessed, the k at the beginning stands for kernel. And kssl is exactly this: A proxy to do all the encryption stuff in a kernel module.

This feature is already four years old. I´ve reported about kssl back in December 2005 for the first time. After talking with a customer two days ago and looking at some new material about it, i thought it could not harm to write a new tutorial about this topic. So much appearances in such a small timeframe are a hint .

With gcore you can create core dumps of processes without the need to stop them. You can do the same with the crash dumps as well. You can trigger a crash dump without rebooting the system. The command for this task is the savecore command. The normal task of this program is to take the content of the crash dump device and to create files in the regular filesystem to make them persistent for further studies after the reboot of a system.

You are logged into your system and doing a little bit of house keeping (archiving of old logfiles, deleting the rubbish on you system like backup files) and suddenly you see a core file. Heck ... you wasn´t aware of the fact, that an application wrote such a file. How can you get some basic information about it? With Solaris you can use the mdb for this task.

Let´s assume you find a core file at / on the system master:

# uname -a
SunOS master 5.11 snv_97 i86pc i386 i86pc
# ls -l core
-rw-r--r--   1 root     root     5073385 Oct 10 18:26 core

Now we can start the Modular Debugger mdb with the core file.

# mdb core
Loading modules: [ libc.so.1 ld.so.1 ]
>

After a short moment the mdb command will give you a prompt. Now just type in the ::status command

> ::status
debugging core file of sshd (32-bit) from master
file: /usr/lib/ssh/sshd
initial argv: /usr/lib/ssh/sshd
threading model: native threads
status: process core file generated with gcore(1)

Now you know, that you can savely ignore the file, as the system created it on your order while you was playing around with gcore.

With a huge amount of help from Marina Sum the first article from the LKSF book made it to the Sun Developer Network: Introducing pfexec, a Convenient Utility in the OpenSolaris OS.

One of my first tutorials was the tutorial about RBAC. In this new tutorial i want to come back to this topic. In the RBAC tutorial i used su to assume a different role. But Solaris offers an additional way to work with the privileges of a different role.

I´ve decided to stop the writing of new parts for the LKSF for a while and want to make a version 1.0 of of this document. There is one major showstopper on the way. I had just an mediocre B in English as i found physics, chemistry and biology and micro/macro economics much interesting than languages. I know there are several typos in the document and i assume many errors regarding the grammar. Thus i ask for your helping hands for proof reading the document. To give you something back for your help, i´ve introduced the "Credits/Kudos" section at the end of the book. Well, when you want to help me at this task, just drop me a mail at joerg@c0t0d0s0.org !

PS: The missing apostrophe in "aren´t" or "don´t" are not typos ... it´s an error in the .tex-source i´ve made consistently in the document. I´m writing a perl script for a mass subsitition at the moment to fix this.

I´ve uploaded an restructured version of the "Less known Solaris features" pdf book today. Well, the order of the tutorials is still the same, but i promoted each structural unit by one step. So a chapter is now a part, a section is now a chapter. This was nescessary as the new Kerberos tutorial (one of two projects in progress) was in need of a different organisation in this section. As a subsubsubsection wasn´t an option, i took the other way.

BTW: The tutorials about killing zombies and getting command line/environment are in this version, too.

There is a really interesting tool to gather informations from a running process. Let´s assume a fellow admin is in her or his well-deserved vacation and now you have to reboot the new machine, because you have to service the memory. Of course you look into the documentation and you find a process that isn´t covered by the documentation and there doesn´t seem to be an startup script or SMF manifest that starts this application. D'oh ... yet another last-minute pre-vacation change ... shouldn't happen, but it does happen.

Let´s assume you found an process called application in the process list:

bash-3.2# ps -ef | grep "application" | grep -v "grep"
    root 27943 27888   0 23:08:23 pts/1       0:00 /bin/perl ./application --start --debug --foobar=e3421
jmoekamp 27945 27824   0 23:08:28 pts/2       0:00 grep application

You look into the application source code (it´s a perl script, so this isn´t a problem and you see, that the application uses environment variables for config information. Now you have a problem. What are the correct settings of the environment variables? How can you gather the correct information to startup the application again. Solaris can help here with a programm called pargs.

At first let´s check the environment of the application first. It´s important to know, that you just can lookup this informations for processes you own. You need to get root privileges to see this information for other processes. Okay, the pargs command offers the -e switch for this task.

# pargs -e 27943
27943:  /bin/perl ./application --start --debug --foobar=e3421
[...]
envp[3]: DATABASEHOST=foobar.server.de
[...]
envp[6]: USERNAME=snafu
[...]
envp[13]: DATABASEDB=importantdatabase
[...]

There is another interesting switch. You can use pargs with the -l switch to get the full command line that was used to start the script, albeit this isn't that interesting, as you could get similar information with /usr/ucb/ps -auxwwww. But with a known process id pargs is much more comfortable.

# pargs -l 27943
/usr/perl5/5.8.4/bin/perl ./application --start --debug '--foobar=e3421' 

Now you should have all informations to restart the script and -by the way- to update the documentation for the fellow admin.

Once in a while you will see some strange processes with <defunct> instead of a process name. This happens when a child process terminates, but the parent process isn´t interested in the outcome because it didn´t waited for the childs response. Almost all resources of the child process are freed up at the moment with the exception of the entry in the process table. The parent process need it to get the exit code from it´s child, thus you can´t simply delete it on the termination of the child. The remaining process table entry will be delete, when the parent proccess reaps the child process by gathering the exit code. But when the parent forgets to reap the child, it´s undead, it´s defunct. Or to stay in the terminology: You´ve produced a Zombie process.

Let´s create such a process. It´s really easy, we just have to create a long running process forking away a child but we don´t use the wait() system call to gather it´s response at the exit.

bash-3.2$ nohup perl -e "if (fork()>0) {while (1) {sleep 100*100;};};"&

Okay, let´s check for our processes. In the output of ps -ecl the zombie processes are marked with a Z:

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD
 0 Z    100 27841 27840    -   0        -      0        - ?           0:00 <defunct>
bash-3.2$

A kill -9 to this process is without effect. Obviously, a zombie will go away when you terminate the parent process, but that isn´t alway an option. How can you get rid of this Zombies? Okay, with Solaris you can reap such processes manually. The preap forces the parent to reap the child by calling wait() system call on the child.

bash-3.2$ preap 27841
27841: exited with status 0

And when you look in the process table again you will see, that the zombie founds it´s peace...

bash-3.2$ ps -ecl |grep "Z"
 F S    UID   PID  PPID  CLS PRI     ADDR     SZ    WCHAN TTY         TIME CMD

Obviously, you should ask yourself, why an application leaves such zombie processes, when the task of reaping them away manually gets a frequent task. Often it´s because of bad programming style.

I´ve uploaded another early pdf of the tutorial. I´ve added some text and the first to figures. Kerberos is just one of the topics where you should explain only the configuration or you have to explain it almost completly. You can´t explain just a few things and still hope your explanation makes sense.

I started this afternoon to write the Kerberos tutorial for the LKSF series. I´ve just put the first fragments in the public and will update them regulary. You can find them here. I will integrate them to the LKSF book as soon i´ve finalized it, but this will take really long time. I didn´t even started decribing the Solaris implementation so far. Kerberos is such a vast topic ...