Less known Solaris features: About crashes and cores - Part 3: Controlling the behaviour of the dump facilities

Solaris has mechanisms to control the behaviour of dump mechanisms. These mechanisms are different for crash and core dumps.

Crash dumps

You can configure the content of crashdumps, where they are located and what you do with them after the boot. You control this behaviour with the dumpadm command. When you use this command without any further option, it prints out the actual state.

# dumpadm<br />
      Dump content: kernel pages<br />
       Dump device: /dev/dsk/c0d0s1 (swap)<br />
Savecore directory: /var/crash/incubator<br />
  Savecore enabled: yes

This is the default setting: A crash dump contains only the memory pages of the kernel and uses /dev/dsk/c0d0s1 (the swap device) to store the crash dump in the case of a kernel panic. savecore is a special process, that runs at the next boot of the system. In the case of an crash dump at the dump device, it copies the dump to the configured directory to keep it for analysis before it´s used for swapping again.
Let´s change the behaviour. At first we want to configure, that the complete memory is saved to the crash dump in case of a panic. This is easy:

# dumpadm -c all<br />
      Dump content: all pages<br />
       Dump device: /dev/dsk/c0d0s1 (swap)<br />
Savecore directory: /var/crash/incubator<br />
  Savecore enabled: yes<br />

Okay, now let´s change the location for the crash dump. The actual name is an artefact of my original VM image called incubator. To get a new test machine i clone this image. I want to use the directory /var/crash/theoden for this purpose.

# mkdir /var/crash/theoden<br />
# chmod 700 /var/crash/theoden<br />
# dumpadm -s /var/crash/theoden<br />
      Dump content: all pages<br />
       Dump device: /dev/dsk/c0d0s1 (swap)<br />
Savecore directory: /var/crash/theoden<br />
  Savecore enabled: yes

Now the system will use the new directory to store the crash dumps. Setting the rights of the directory to 700 is important. The crash dump may contain sensitive information, thus it could be dangerous to make them readable by anyone else than root.

Core dumps

A similar facility exists for the core dumps. You can control the behaviour of the core dumps with the coreadm command. Like with dumpadm you can get the actual configuration by using coreadm without any option.

# coreadm<br />
     global core file pattern:<br />
     global core file content: default<br />
       init core file pattern: core<br />
       init core file content: default<br />
            global core dumps: disabled<br />
       per-process core dumps: enabled<br />
      global setid core dumps: disabled<br />
 per-process setid core dumps: disabled<br />
     global core dump logging: disabled

This programm has more options than dumpadm. I won´t go through all options, but some important ones. From my view the file patterns are the most interesting ones. You can control, where core dumps are stored. The default is to store the core dumps in the working directory of a process. But this may lead to core dumps dispersed over the filesystem. With core adm you can configure a central location for all your coredumps.

# coreadm -i /var/core/core.%n.%f.%u.%p<br />
# coreadm -u

With -i you tell coreadm to set the location for the per-process core dumps. The parameter for this option is the filename for new core dumps. You can use variables in this filename. For example %n will be translated to the machine name, %f to name of the file, %u to the effective user id of the process and %p will be substituted with the process id. The coreadm -u forces the instant reload the configuration. Otherwise, this setting would get active at the next boot or the next refresh of the coreadm service. Okay, let´s try our configuration.

# ps -ef | grep "bash" | grep "jmoekamp"<br />
jmoekamp   681   675   0 20:59:39 pts/1       0:00 bash<br />

Now we trigger a core dump for a running process.

<br />
# gcore -p 681<br />
gcore: /var/core/core.theoden.bash.100.681 dumped<br />

As you see, the core dump isn´t written at the current working directory of the process, it´s written at the configured position.

Core dump configuration for the normal user

The both configuration described so far are global ones, so you can do this configuration only with root privileges. But a normal user can manipulate the core dump configuration as well, albeit only for processes owned by her or him. Let´s login as a normal user. Now we check one of our processes for it´s coreadm configuration:

$ ps -ef | grep "jmoekamp"<br />
jmoekamp   712   670   0 01:27:38 pts/1       0:00 -sh<br />
jmoekamp   669   666   0 22:29:15 ?           0:00 /usr/lib/ssh/sshd<br />
jmoekamp   670   669   0 22:29:16 pts/1       0:00 -sh<br />
jmoekamp   713   712   0 01:27:38 pts/1       0:00 ps -ef<br />
$ coreadm 669<br />
669:    /var/core/core.%n.%f.%u.%p      default

Now let´s check a process owned by root.

<br />
$ ps -ef | grep "cron"<br />
jmoekamp   716   670   0 01:28:13 pts/1       0:00 grep cron<br />
    root   322     1   0 22:25:24 ?           0:00 /usr/sbin/cron<br />
$ coreadm 322<br />
322: Not owner

The system denies the access to this information. Now we change the setting for the process 669 from the first example. It´s quite simple:

$ coreadm -p /export/home/jmoekamp/cores/core.%n.%f.%u.%p 669<br />
$ coreadm 669<br />
669:    /export/home/jmoekamp/cores/core.%n.%f.%u.%p    default<br />

The per-process core file name pattern is inherited by future child processes of the affected processes. Why should you set an own path and filename for an application or an user? There are several reasons. For example to ensure that you have the correct rights to an directory for the cores, when the process starts to dump the core or to seperate the cores from certain applications a different locations.