Written by {"display_name"=>"Joerg Moellenkamp", "login"=>"Joerg Moellenkamp", "email"=>"weblog@joerg.moellenkamp.org"}
on 

Meet the Stats - today: fsstat

There are a lot of *stat in Solaris … prstat, vmstat, iostat, vmstat. In the next weeks i want to write intermittently about all the stats. One of the lesser known members of the stat family will make the start: It’s fsstat. This small tool gives you an deep insight into the inner workings of your filesystem. It can show you what operations are really done in what frequency. So don’t have to rely on assumptions about it.
fsstat is quite new, it’s in Solaris 10 since 10/06 and in Opensolaris since Build 36. It uses some new kstat counters to deliver the numbers.

Using fsstat

When you invoke fsstat you get some general overview about the filesystem activity. Some operations are grouped into more general classes to summarize the information:

jmoekamp@kserverosol:~$ fsstat /home/jmoekamp 1
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
  250    64    67 2.91K    80  8.82K   257   724 12.4M 22.1K 21.3M /home/jmoekamp
 [..] 
    0     0     0     8     0      9     0    14 23.2K    50 39.5K /home/jmoekamp
    2     0     1     6     0     35     0     2   528     1 1.36K /home/jmoekamp
    2     0     0    15     1     44     0    46  125K    46 56.7K /home/jmoekamp
    0     0     0     1     0      3     0     2 1.18K     2   979 /home/jmoekamp
    0     0     0     0     0      4     0     1   512     5 3.09K /home/jmoekamp
    0     0     0     0     0      0     0     0     0     0     0 /home/jmoekamp
    0     0     0     1     0      7     0     5 4.66K     0     0 /home/jmoekamp
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
    0     0     0     0     0      0     0     4 13.3K     6 4.79K /home/jmoekamp

As you may have already noticed, you can specify the frequency and the number of printing those information similar to other *stat tools. fsstat /export/home/jmoekamp/ 10 10 would print this information 10 times every 10 seconds. Leaving out the third parameter will give you this information every 10 seconds until you interrupt this tool. Additionally it’s important to know that the first line of output of thi tool gives you the aggregated information since the mounting of this filesystem. It’s perfectly possible to to dig down into this groups. -a gives you detailed information about the frequency of operations relating to filesystem operations:

jmoekamp@kserverosol:~$ fsstat -a /export/home/jmoekamp
getattr setattr getsec  setsec
 1.21K       1      48       0 /export/home/jmoekamp

-v helps you to learn more about the virtual memory operations on a file system:

jmoekamp@kserverosol:~$ fsstat -v /export/home/jmoekamp
  map addmap delmap getpag putpag pagio
   19     19     11    113      4     0 /export/home/jmoekamp

The -i is of more general interest, as it gives you some insight into the read/write load on your filesystem:

jmoekamp@kserverosol:~$ fsstat -i /export/home/jmoekamp
 read read  write write rddir rddir rwlock rwulock
  ops bytes   ops bytes   ops bytes    ops     ops
  193  555K 12.2K 1.20M   173 22.0K  12.5K   12.5K /export/home/jmoekamp

Using fsstat with the -n option prints detailed information about all operations that relate to the names:

jmoekamp@kserverosol:~$ fsstat -n /export/home/jmoekamp
lookup creat remov  link renam mkdir rmdir rddir symlnk rdlnk
4.95K     44    14     1    26     0     0   173      2     1 /export/home/jmoekamp

Of course there is an option giving you all information at once.

jmoekamp@kserverosol:~$ fsstat -f /export/home/jmoekamp
/export/home/jmoekamp
 operation  #ops  bytes
      open   276 
     close   625 
      read   193   555K
     write 12.2K  1.20M
[...]
   vnevent   169 
  reqzcbuf     0 
  retzcbuf     0

Additionally you are able to get those data per type of filesystem:

jmoekamp@kserverosol:~$ fsstat -F
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
    0     0     0     0     0      0     0     0     0     0     0 ufs
1.11K   159    69  308K   788  1.08M 5.38K  221K  198M 85.2K 17.1M zfs
   47    15    59 5.44K    29  5.09K   177   199  555K 12.2K 1.28M lofs
3.11K 2.86K   126 24.0K    94  6.90K    68 45.1K 45.0M 43.0K 37.5M tmpfs
    0     0     0   643     0      0     0    42 7.29K     0     0 mntfs
    0     0     0 3.09K     0  3.05K     0     0     0     0     0 autofs

Possible usecases

It’s just a very simple usecase: I’ve used fsstat once to find a user loading the disk subsystem. All users had their own filesystem due to ZFS. So the easiest way to find the problematic user was just to print out the statistics for all user home directories.

jmoekamp@kserverosol:/export/home$ fsstat *
 new  name   name  attr  attr lookup rddir  read read  write write
 file remov  chng   get   set    ops   ops   ops bytes   ops bytes
    0     0     0     0     0      0     0     0     0     0     0 /export/home/jm0001
    0     0     0     0     0      0     0     0     0     0     0 /export/home/jm0002
    0     0     0     0     0      0     0     0     0     0     0 /export/home/jm0003
    0     0     0     0     0      0     0     0     0     0     0 /export/home/jm0004
    0     0     0     0     0      0     0     0     0     0     0 /export/home/jm0005
   50    16    33 2.01K    30  5.34K   187   209  558K 12.2K 1.37M /export/home/jmoekamp
   11     0     0   107    23    171    24     0     0     5 3.75K /export/home

When used with the * it prints out stats for all the filesystems that have their mountpoint in this directory. So it was easy to find the filesystem and thus the user with the highest load.

Do you want to learn more?

manpages
fsstat(1M) - report file system statistics