c0t0d0s0.org

I had an interesting discussion with a blog reader and customer via icq (oh wonderful new world of business). It started with a question about the J4500 and it ended in the the discussion, if MAID is really a viable way for storing data of if it´s just one of this hypes working their way through the storage business every year.

MAID? What is MAID? MAIDs are Massive Arrays of Idle Disk. The idea is to save power by switching of disks in the case they aren´t in use. You can buy such components from companies like Copan.

An interesting new feature had it´s flag day. With PSARC 2008/393 a feature called "zfs primarycache and secondarycache prop" found it´s way into Solaris for build 96. With this properties you can control, what ZFS caches in the ARC and the L2ARC.

The introduction of the 'primarycache' property will allow administrators the ability to control what is cached in the ARC. The possible choices are: all, none, and metadata. If 'all' is chosen, then both user data and metadata are cached. If 'none' is chosen, then neither user data nor metadata is cached. If 'metadata' is chosen, then only metadata is cached. The default is 'all'.

The introduction of the 'secondarycache' property will allow administrators the ability to control what is cached in the l2ARC. The possible choices are the exact same as 'primarycache'. The default is also 'all'.

This is interesting for situations, when you know that your application is caching data as well (thus it makes no sense to cache it twice) or you aware of the fact that caching is just waste of time because of the access pattern.

You can find details about this feature in the PSARC discussion for 2008/393 and in the specs.

Darren Moffat shows a nice trick in his blog. He explains how you can make a file in a ZFS filesystem even immutable for the root user. Hmm, i should integrate this to the LKSF book.

Brendan Gregg wrote a good piece about the performance of L2ARC in ZFS L2ARC:

The pool_0 disks are still serving some requests (in this output 30 ops/sec) but the bulk of the reads are being serviced by the L2ARC cache devices - each providing around 2.6K ops/sec. The total delivered by this ZFS pool is 15.8K ops/sec (pool disks + L2ARC devices), about 8.4x faster than with disks alone.

and

Our average service time is between 0.4 and 0.6 ms (wsvt_t + asvc_t columns), which is about 20x faster than what the disks were delivering.

Adam Leventhal wrote an excellent article about the advantages of flash based storage and how ZFS circumvents some of the disadvantages (like write latency). Really a must read: Communications of the ACM: Flash Storage Memory. Additionally it explains some of the inner working of the L2 ARC.

(via: Robert Milkowski: RAM->SSD->DISK + ZFS)

We´ve announced yesterday, that we will deliver solid state disks in our servers in the next fiscal year. Many journalists wrote about the complete substituion of rotating rust disks by solid state disks or drives (strange wording for a technology without disks or any moving parts), but they forgot the most obvious usage of this. One example is: Sun stretches for Flash dance

Rotating rust as it´s advantage and solid state has other advantage. So why don´t use both. Solaris and ZFS can do exactly this: Separated ZIL and L2ARC were designed to use the advantages of SSD where rotating disks can use some help. You don´t have to move all your data to solid state just to get the speed advantages of solid state. Yet another point, where you have to see the the complete package to understand a move made by Sun.

This is another artitcle of the kind i do not really understand: FAA: Sun box disk failure caused NOTAM database crash. What do you think, when you read this article at first: Sun can´t develop storage? Bad, bad Sun ? Well, the cursory observer might think that.

But at the end it´s a story about the stuff, why Sun develops stuff like ZFS. It´s just doesn´t told this way. It´s a story of undetected silent data corruption, transmission of the data corruption to the backup system, the problem of damages/power outages while updating the data on the harddisks:

"What happened was the drive in an end-of-life Sun box failed in the middle of updating the information on the hard drive, so it screwed up the database," Davis said.

and

As the technicians were working to fix the database, they decided to go to the backup system. As they did that, they soon realized they had written the error over to the backup system and had corrupted that system as well, Davis said.

It sound like a perfect example why the industry needs a filesystem like ZFS. Harddisks fails, that is inevitable, and mostly they do not fail without a last scream. And that is independend of the brand of harddisks or the storage vendor.

So ... dear Computerworld, would you mind to use a less lurid headline next time?

Roch reports in The new ZFS write throttle about a new algorithm in ZFS for throttling writes appearing in upcoming versions of Opensolaris. Every five seconds ZFS commits changes to it´s pools to the disk, even when you are not forcing this syncing. Whenever your harddrives doesn´t keep up with the amount of writes from the transaction group, it was possible, that ZFS trottled writing application to ensure that the you can write all the outstanding data in the transaction groups. Depending of the amount of the data and the speed of your harddisks this state could last a few seconds.

The new behaviour is much better from my point of view. Roch writes in his article:

So the new steady state behavior of write intensive workloads is that, starting with an empty TXG, all threads will be allowed to dirty memory at full speed until a first threshold of bytes in the TXG is reached. At that time, every write system call will be delayed by 1 tick thus significantly slowing down the pace of writes. If the previous TXG completes it's I/Os, then the current TXG will then be allowed to resume at full speed. But in the unlikely event that a workload, despite the per write 1-tick delay, manages to fill up the TXG up to the full threshold we will be forced to throttle all writes in order to allow the storage to catch up.

This new mechanism should keep away this complete write throttling pauses really effectively.

One of the limitations of Opensolaris 2008.05 will be the missing LiveUpgrade. But ... well ... you have something better. The whole concept of LifeUpgrade was transformed into the future by using the capabilities of ZFS.

In "Democratizing Storage" Jason writes about his opintion of ZFS and the usage of ZFS at his employer digitar. It´s really positive. You can summarize the article at best with the following quote:

When it comes to storing data, you’ll pry OpenSolaris (and ZFS) out of our cold dead hands. We won’t deploy databases on anything else.

... but two hard disks had a really bad day. A demonstration of the capabilities of ZFS:

The support for booting from ZFS had it´s flag day yesterday. It looks like this will appear in build 88 for x86 and SPARC. I hope this find it´s way in Update 6 of Solaris, stopping this ever reoccuring question at presentation about this feature.

Albeit it´s very easy to use iSCSI in conjunction with ZFS emulated volumes it doesn´t have to be this way. You can use different kinds of backing store for iSCSI

I had an idea to vastly reduce the network load for iSCSI harddisks while waiting for the water to boil for my first tea. There are two new features in development for ZFS: L2ARC and Separated ZIL. L2ARC enables ZFS to use a disc as a second level cache (thus speeding up reads). The Separated ZIL enables ZFS to use a disc as to use a dedicated disc for the ZFS Intent Log (thus speeding up writes). Both features were designed with solid state disks in mind to augment "rotating rust" hard disks. But this features are not limited to this high end task.

Think about the following usecase: A server has in most cases some local boot disks even when you use iSCSI or FC. From a 146 GB disc you need only roundabout 20 GB (2x 7.5 for OS incl. Live Upgrade Partition and 5 GB for swap) or so. Thus you have 126 GB left. Okay, let´s assume, that you have two X4500 exporting zvols with iSCSI.

You have another machine in your network. You create an mirrored ZFS pool out of the remote iSCSI volume. But now the both new features come into the game. We augment this iSCSI volume with a local separated ZIL and a local L2ARC. With the first one, you keep the log writes from the network and with the local L2ARC you get an really huge local cache without sacrifying precious