c0t0d0s0.org

With the integration of PSARC 2007/347 NFS/RDMA Solaris got a new implementation for NFS over RDMA:

This integration brings the NFS/RDMA Solaris Implementation up to datewith the latest IETF Drafts. Specifically, it removes support for the initial NFS/RDMA implementation (version 0) and adds support for rpcrdma [1] and nfsdirect [2] (version 1).

Yet another tutorial finalized - this time about one of the really hidden features of Solaris - CacheFS. CacheFS is something similar to a caching proxy. But this proxy don´t cache web page, it caches files from another filesystem. I divided the tutorial in 8 parts.

• Part 1: Introduction
• Part 2: Theory
• Part 3: A basic example
• Part 4: The Cache
• Part 5: On demand consistency checking
• Part 6: A usecase for CacheFS
• Part 7: The future of CacheFS
• Part 8: Conclusion

I hope, this tutorial gives you some new insights into Solaris. Have fun while trying out the stuff in the tutorial

CacheFS is one of this features even some experienced admins aren´t aware of. But as soon as they try it, most of them can´t live without it. You should give it a try.

CacheFS is one of this hidden gems in Solaris, it´s even that hidden, that it´s in sustaining mode since 1999. That means, that only bugfixes are made for this features but no new features found their way into this component since that day. In the recent days there was some discussion about the declaration of the End-of-Feature status for CacheFS which will lead to the announcement of the removal of CacheFS. After a few days of discussion the ARC decided in favour of the removal.

Let´s take an example: Let´s assume you have 50 webservers that serve static content or that serves and executes .php files. You may have used scp or rsync to syncronize all this servers.

With CacheFS the workflow is a little bit different: You simply put all you files on a fileserver. But you have noticed, that there was a large load on this system. To reduce load to the fileserver you use your new knowledge to create a cache on every server. After this you saw a lot of requests for directory and file metadata for the documents directory. You know that this directory is provisioned with the changes only at 3 o´clock in the morning. Thus you write a little script, that checks for the file please_check_me and starts cfsadmin -s all on the client in the case of it´s existence and it will check at the next access to the files if there´s a newer version.

Let´s assume you share a filesystem with static content (for example the copy of a cdrom) or a filesystem that changes on a regular schedule (for example at midnight every day). So it would pose unnessesary load to network to check the consistency everytime again.

CacheFS knows a special mode of operation for such an situation. It´s called on demand consistency checking. It does exactly what the name says: It checks only the consistency of files in the cache, when you tell the system to do it.

Okay, we have a working CacheFS mount but where and how is the stuff cached by the system. Let´s have a look at the cache.

Okay ... using CacheFS is really easy. Let´s assume, you have an fileserver called theoden We use the directory /export/files as the directory shared by NFS. The client in our example is gandalf.

The history of CacheFS

Sun didn´t introduced this feature for webservers. Long ago, admins didn´t want to manage dozens of operating system installations. Instead of this they wanted to store all this data on a central fileserver (you know ... the network is the computer). Thus netbooting Solaris and SunOS was invented. But there was a problem: Swap via Network was a really bad idea that days (it was a bad idea in 10 MBit/s times and it´s still a bad idea in 10 GBit/s times). Thus the diskless systems got a disk for a local swap. But there was another problem. All the users started to work at 9 o´clock ... they switched on their workstations ... and the load on the fileserver and the network got higher and higher. They had a local disk ... local installation again? No ... the central installation had it´s advantages. Thus the idea of CacheFS was born.

CacheFS is a really old feature of Solaris/SunOS. It´s first implementation dates back into the year 1991. I really think you can call this feature matured

CacheFS in theory

The mechanism of CacheFS is pretty simple. As i told you before, CacheFS is somewhat similar to a caching web proxy. The CacheFS is a proxy to the original filesystem and caches files their way through CacheFS. The basic idea is to cache remote files locally on a harddisk, so you can deliver them without using the network when you access them the second time.

Of course the CacheFS has to handle changes to the original files. So CacheFS checks the metadata of the file before delivering the copy. If the metadata has changed, the CacheFS loads the original file from the server. When the metadata hasn´t changed it delivers the copy from the cache.

The CacheFS isn´t just usable for NFS, you could use it as well for caching optical media like CD or DVD.

There is a hidden gem in the Solaris Operating Environment called CacheFS, solving a task many admins solve with scripts.

Imagine the following situation. You have a central fileserver, and let´s say a 40 webservers. All this webservers deliver static content, this content is stored on the harddisk of the fileserver. Later you recognize, that your fileserver is really loaded by the webservers. Harddisks are cheap, thus most admins will start to use an recursive rcp or an rsync to put a copy of the data to the webserver disks.

Well ... Solaris gives you a tool to solve this problem without scripting, without cron based jobs, just by using NFSv3 and this hidden gem: CacheFS. CacheFS is a really nifty tool. It does exactly what the name says. It´s an filesystem, that caches data of an other filesystem. You have to think about it like a layered cake. You mount a CacheFS filesystem with an parameter that tells CacheFS to mount another one in the background.

History
The ever reoccuring question to me at customer sites relatively new to Solaris is: "Okay, on Linux i had my homedirectories at /home. Why are they at /export/home at Solaris?" This is an old hat for seasoned admins, but i get this question quite often. Well, the answer is relativly simple and it comes from the time when started to use NIS and NFS and it had something to to with our slogan "The network is the computer", because it has to do with directories distributed in the network. Okay, we have to go 20 years in the past.

Yesterday was an important day for Opensolaris. Something found it´s way in the kernel, i already knew but wasn´t allowed to talk about (at least not without a NDA saying "When you talk about this, we send a support technican spelling a dozen mighty curses in you datacenter. And, yeah, we know where you park your car ..." ).

Alan Wright announced the putback of the code changes resulting out of PSARC 2006/715, or for mortals like me: The in-kernel CIFS service. This doesn´t mean just the integration to mount CIFS shares. That´s a different project. With this changes CIFS get a first class citizen besides NFS. Opensolaris Build 77 will give you a kernel-integrated CIFS server.

And as a CIFS server alone is only half the answer to the question, there will be the integration of NDMP for backups or ICAP for integrating virus scanning software. So stay tuned, albeit the page isn´t activated you will find more informationsat the CIFS community at opensolaris.org soon. Meanwhile you should look at Bob Porass´"Fish -n- CIFS" for more informations.

PS: I can only assume, that NTAP saw something like this coming when they filed their "all in"-lawsuit...

Nun, dritter Tag ... die letzten Breakout-Sessions ... beginnen wir mal den Tag mit "Building multipetabyte architectures with Thumper". Nicht wirklich viel los, die meisten Leute sind wohl irgendwie noch in einem Zustand der frühmorgendlichen Pseudostasis.

Der Vortrag begann damit, zu erklaeren, warum man ueberhaupt petabyteweise Daten speichen will, sollte und muss. Insbesondere Forschungsinstitute muessen heute grosse Datenmengen speichern. Wenn man bedenkt, das hoechstaufgeloeste Gehirnscans mittlerweile 1.5 Terabyte Speicher pro Bild verbrauchen, kommen diese Terabytes ziemlich schnell zusammen.

Eine Lösung um soetwas mit Thumpern darzustellem, ist die Verwendung von Lustre. Lustre ist ein Cluster Filesystem, welches darauf ausgelegt worden ist, mir sehr grossen Filemengen und sehr grossen Bandbreiten umzugehen. Das bestechende an Lustre ist der mögliche Verzicht auf ein SAN. Ich kann trotzdem auf gemeinsam von einem Client auf den Speicher zugreifen. Daszu wird auf dem Server ein Object Storage Server installiert. Die Kommunikation zwischen diesen Server und dem Client geht dann Ethernet, Myrinet oder Infiniband. Damit kann man dann auch Storage horizontal skalieren. Leider gibt es das nur fuer Linux momentan. Da muss der Hersteller von Lustre unbedingt noch nachlegen.

Einige interessante Anmerkungen:
- man kann mit 8 Gigabit-Ethernetschnittstellen das I/O-System der X4500 nicht saturieren
- es wurde auf mein Blog verwiesen