The curious case of /tmp in Solaris

LKSF › Solaris Administration › The curious case of /tmp in Solaris

This article isn’t really about a feature, it’s about a directory and its misuse. Furthermore it’s an article about different default configurations that lead to misunderstandings. This is a pretty old hat for experienced Solaris admins (many of them learned it the hard way), but it seems to be totally unknown to many admins new to the business or for people switching from Linux to Solaris, as many distributions are configured in a different way by default. A reader of my blog just found a 2GB .iso in /tmp on a Solaris system and that’s not really a good idea. A few days ago, a user on twitter had vast problems with memory usage on a system which boiled down to a crowded /tmp.

tmpfs and its usage

In Linux the /tmp directory is a normal directory. Most often this is just a part of the partition containing root. Many admins consider /tmp as an additional home directory and put data on it they want to use later on. When you really think about it, /tmp is meant as a scratch space, where an application can put some data to process it later. Many applications use this /tmp space to write temporary data to.

Solaris introduced something called tmpfs many years ago. It’s a memory based file system. Linux has something with similar functionality and the same name. It’s just used differently in the default configuration of many distributions and there are a vast amount of articles on the web that suggest configuring /tmp in Linux to use the tmpfs filesystem as this can give you advantages, for example with MySQL (when there is a reasonable amount of SELECTs opting for filesort).

The tmpfs is not a ramdisk. A ram disk just resides in the RAM, a tmpfs resides in the virtual memory, thus it uses the swap when the memory is needed for something else. Furthermore it doesn’t have a predefined size, albeit you can define a maximum size for the tmpfs to ensure that someone who writes into the /tmp can’t eat away all your virtual memory.

It’s called tmpfs because everything you write into it is temporary, the next boot or unmount will kill all the files on it. When you look at the mount table of a Solaris System you will recognize that the usual locations for such temporary files are mounted tmpfs:

jmoekamp@a380:/var$ mount | grep "swap"
/etc/svc/volatile on swap read/write/setuid/devices/xattr/dev=4f00001 on Fri Jul 31 06:33:33 2009
/tmp on swap read/write/setuid/devices/xattr/dev=4f00002 on Fri Jul 31 06:34:14 2009
/var/run on swap read/write/setuid/devices/xattr/dev=4f00003 on Fri Jul 31 06:34:14 2009

Keeping these file systems in virtual memory is a reasonable choice. The stuff in this directory is normally stale after a reboot, most of the time the files are many, but rather small and putting them on disk would just eat away your IOPS budget on your boot disks. As the file system resides in memory, it’s much faster and that really helps on jobs with many small files. A good example is compiling software when you use /tmp as the TMPDIR.

All these advantages come with a big disadvantage, when you are not aware of the nature of the /tmp directory. I assume you already know why using /tmp for storing ISOs is a bad idea. It eats away your memory and later on your swap. And for all the experienced admins: When someone has memory problems, ask at first about the /tmp directory, we tend to forget about this, as we’ve learned this lesson a long time ago and thus don’t think about this problem. When you really need a temporary place to store data in it for a while you should use /var/tmp. This is a directory on a normal disk based filesystem and thus its content is boot-persistent.

Configuring the maximum size of the tmpfs

When you want to enforce a limit on the size of a tmpfs, this is a pretty easy task. You can do this with a mount option. Let’s assume you want to provide the /var/applicationscratchpad directory for an application, but you want to be sure that this application can’t ruin your day by eating away all your memory:

jmoekamp@a380:/var# mkdir /var/applicationscratchpad
jmoekamp@a380:/var# mount -F tmpfs -o size=128m swap /var/applicationscratchpad/
jmoekamp@a380:/var# mount | grep "application"
/var/applicationscratchpad on swap read/write/setuid/devices/xattr/size=128m/dev=4f00004 on Fri Jul 31 09:32:26 2009

When you want to make a boot-persistent change to the maximum size of the /tmp directory, you have to configure this in the /etc/vfstab:

swap    -       /tmp    tmpfs   -       yes     size=512m

Conclusion

I hope it’s now clear why you shouldn’t use /tmp as a place for storing big files, at least move them directly somewhere else when you use /tmp as the target directory to ssh a file to a system. On the other hand the /tmp in virtual memory gives you some interesting capabilities to speed up your applications. As usual, everything has two sides: Making it default in Solaris gives you speedups by default. But when you are unaware of this default situation your virtual memory may be used for a collection of videos ;) Obviously the same is valid when you configure your Linux system with a tmpfs based /tmp and don’t tell it to your fellow admins.