c0t0d0s0.org

Recently I was doing some work in regard of tuning systems. There is something i really hate about this topic of computing: Tuning scripts. You find them on google easily and i find them on systems at customers quite often.

Simply said: I hate them. The reasons for it are simple. For example recently I found a system with a networking tuning script dating back into 2003 or so. The problem: It was meant to increase some of the settings. However many of them were already higher in the default config of current Solaris 10 versions, thus the tuning script essentially reduced the parameters and thus reduced the performance.

Futhermore: Tuning is a lot about understanding things. Understanding how things work together. On a systemic, on an architectural level. How an application loads all the rest of components. Just dropping a script downloaded from a website found by Google - into /etc/init.d is not about understanding things.

You have to carefully consider each change from the default about the impacts. You have to check each setting, if the setting hasn’t already overtaken by the years. You have to recheck it it with every major update of your environment. You have to recheck it with each new technology you are using in your system. Network tuning scripts dating back to a time when 100 MB/s were normal and 1 GB/s are fast aren’t necessarily up to the task in a time when 10 GBit/s are fast and Infiniband IPoIB networks deliver even more. You had to turn different knobs in a time, when cpu time was precious. You’ve tuned for minium cpu utilization. CPU isn’t a large factor today, you tune for minimize latency or maximize throughput. You have to know what you want to aim for, because minimum latency and maximum throughput are often mutually exclusive. Do you want an extreme or a target in between.

Just using a script to tune something doesn’t lead you through all the thought to make really good tuning decisions.

There are some basic rules from my point of view:

• For Solaris there is for each major update a document updated called “Oracle Solaris Tunable Parameters Reference Manual”. It’s available at here.Check your tuning script at each update against this document in order to ensure that you settings make sense. The 817-0404 is a great read anyway, as it gives you an condensed overview over the possible official knobs to turn.
• Each tuning script has to be carefully designed for each system for each workload. An /etc/system for a database make perhaps no sense for a webserver or may even reduce the performance and vice versa. That’s the reason why I don’t like tuning scripts as part of jumpstart configurations. An NFS server delivering many small files to many clients may need other tunings than a server delivering few large files to a few clients.
• Understand! Understand how components of your architecture interact. Understand the load. Monitor the load for a while. Tuning is at foremost doing a lot of research before and just at last about putting some lines into some scripts. That’s one of the reason why I like tools like dtrace in general, and the scripts from the dtracetoolkit especially. Those tools are like the professionals tools of a mechanic. They can help you to understand the load. The equivalent of the basic set of screwdrivers: All the *stat tools are helpful, too. You need them to know where you can start with the more advanced tools, that are often too focused.
• Have a benchmark load. Have a mechanism to reproduce the load on the system. The more realistic and the less synthetic the better. So you can quantify the impact of each tuning.
• Just change one knob at a time. Otherwise you will never know what setting really helped you or which setting hurted the performance
• Of course: Experience helps a lot in shorten the research phase and sometimes there are some low-brainer settings. But even when you think “Nah … this is just a webserver. Let’s use script number 7b”, you may be surprised by interesting side effects. So there is such think like a no-brainer in tuning
• Large standard applications have often a set of certain tuning parameters that the vendor suggest as a good baseline: At first - look them up in the tuning guide in order if the ISV tuning guide wasn't overtaken by the defaults in the OS. At second: Check them with the vendors manuals with each release ... had once a performance problem with customer that used the tuning parameters suggested for - let's say - version 2, however there weren't that reasonable with version 5.
• It's a good practice to create two baselines: At first check the service performance with a totally untuned system ... just modify settings you need to get the software up-and-running.Run your benchmark. Configure the system with the vendor suggested parameters. Rerun the benchmark. Now do your own tunings. Rerun the benchmark with each knob you turn. And ask yourself: Is the performance impact worth the deviation from the standard parameters from a plain vanilla installation? With such baselines you can assess the impact and thus the value.

A commentator at hackernews asked how i think about -9. In my opinion: It's widespread use is a similar plaque like the –f switch. And this is pretty easy to explain (I'm simplifying things a bit).

-9 is a shorthand for SIGKILL. When you send a SIGKILL to a process, the process is terminated immediately. You can’t catch this signal, you can’t ignore it. A kill with -9 sends this SIGKILL to a process.

A kill without -9 sends a SIGTERM to process. It terminates the process like SIGKILL. However a process is allowed to catch it in order to execute a signal handler … or just ignores to ignore it. A signal handler is nothing more than a code path that is executed when the process receives a signal.

So when you kill a process with a normal kill you give the process the chance to clean up behind itself, to make files consistent, to roll back changes in the case the process isn’t using some transactional mechanisms when changing data, to delete temporary files … and so on ...

It's a good style to write such signal handlers and in many programming languages it's pretty easy. For example in perl:

$SIG{TERM}=\&signalhandler_TERM; 

sub signalhandler_TERM{ 
     $SIG{TERM} = \&signalhandler_TERM; 
     print "leaving process\n";
     #some code to clean up behind you 
     exit(1); 
}

When you send a -9 to a process you take away this chance from the process. It’s killed instantly … even if it just started to modify your files, fscking up your data in order to put it in a new form, even when you have created dozens of temporary files filling up /tmp. Things like that …

Killing a process with -9 is the last possibility. However I see people using it too often too early. A second after the normal kill is send a pgrep on the process follows. Still there and the sword of -9 is falling down.

When a process doesn’t disappear immediately after sending the SIGTERM, it may be just busy to follow your order of terminating itself and is cleaning up things. When your application is dependent to precious resources at cleaning up (for example IOPS on your rotating rust) the process of cleaning up may take a while.

The implicit question in any process, that doesn't react to a normal kill via SIGTERM is the question why it doesn't react to the signal. Just sending a -9 when a normal kill didn't worked is like "Do not care". Monitoring the process with truss or strace what the heck the process is doing after getting the SIGTERM is a good first step. Perhaps you see some cleanup work and know that you just have to wait a little bit longer. Writing a core dump of the process with gcore is often a good second step to save evidence for future research why the process didn't reacted.

And then … and only then … a kill -9 may be feasible.

In short:

Always try a normal kill first. There is just one execption.
• The exception: You know that the integrity of data is depending on the fact, that the process has no chance to write anything for example by executing a signal handler. Consider it as something like a forced panic on process level. The panic is the admins friend as it ensures that nothing is written to disk as soon as the OS is detecting something worth to trigger a panic. A panic isn't a "insult the admin"-thing … it's about protecting data.
• kill -9 is your emergency stop switch and shouldn't be part of your normal administrative procedures in any way
• Just sending a -9 after a normal kill without thinking about reasons is bad administrative style.

I'm following a discussion at the moment, where someone has done some havoc to his data. This discussion inspired me to write this: -f. The force switch. Personally i believe -f should be protected by key that you just get when you can explain the whole subsystem that has such a switch and the reason why you need -f.

• -f is not about forcing round pegs into square holes.
• -f is about forcing pegs known to be square into holes known to be squares that doesn't fit, because some idiot dented the edges of the hole.
• -f is not about "I know it better than the machine. This command is correct". Believe me ... in almost all cases the system has a point in preventing you from doing something.
• -f is about telling the system "I'm fully aware of what i'm doing at the moment"
• -f is about the system telling you "Everything from here is even more your fault than usual"

There are situations when a -f is feasible. However just do it, when you know the 7 following things:

• You know, that a command should work.
• You know, what a command is normally doing.
• You know, why a command that should work doesn't work.
• You know, that you can't repair the issue that led to the "command doesn't work" by other means than -f.
• You know, that the chance of doing greater harm to the data is low enough to risk the data.
• You know, that your backup is working, when you can harm persistent data by using -f
• You know, that your restore is working, when you can harm persistent data by using -f

Unsure about just a single point? Then don't use -f until you are sure.