c0t0d0s0.org

Apropos awesomeness.

MOVE from Rick Mereki on Vimeo.

Some of the shoots are just breathtaking ...

Timelapse - The City Limits from Dominic on Vimeo.

Since the closure of the blog i got many many requests for giving content to people in form of an excerpt of the database. Even more. I got a significant numbers of angry mails. It seems that i was the best source for some things and thus the people were really angry about removing their source of information. I was somewhat astonished about the anger, however in absolute numbers it was significant but in relative numbers it was rather small. Nevertheless, i didn't expected angry people.

After thinking about the whole weekend i decided to put selected content back online instead of creating a pdf of the blog page and sending it to the people asking for content. As a side effect, it repairs a lot of links on Solaris related pages.

However the reasons for my blog closure are still valid, the blog stays in this freezed state. Thus i decided to do this as a selective republication. Certain classes of documents will stay offline. In that regard i made the following decision, which content gets back into blog: Personal content on one side, work related technical content as long it's related to Solaris on the other side.

My colleague Christophe Pauliat - Principal Sales Consultant at Oracle - came up with a really nifty way to migrate his Solaris based notebook from a smaller disk to a larger one. I will copy his mail in verbatim here, because i think it's extremely useful. It somewhat resembles the "workaround" for ZFS resizing, however Christophe does takes this significantly forward and does this for boot disks.

OS: Solaris 11 express 2010.11 + SRU8 

Steps:

1) Copy data + OS on the new HDD
    a) connection of the new 500 GB HDD as an external USB HDD (using a USB external HDD box)
    b) creation of a Solaris 2 partition with fdisk and make it active (bootable)
         # fdisk /dev/rdsk/c4t0d0p0
    c) with the format command, create a partition s0 with all cylinders except cylinder 0
    d) Mirroring the existing ZFS pool (rpool) to the new HDD
         # zpool attach -f rpool c1t0d0s0 c4t0d0s0
        notes:
        - c1t0d0 is the 80 GB HDD (old HDD)
        - c4t0d0 is the 500 GB HDD (new HDD)
        - the option -f is necessary to bypass the warning "partition 0 overlaps partition 2"
    e) wait for the sync to be finished (with zpool status)
    f) Install Grub on the new HDD
        #  installgrub -m /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c4t0d0s0
    g) Split the pool rpool by detaching to new HDD to create a new pool
        # zpool split rpool rpool2 c4t0d0s0
        note: I chose not to detach the old HDD because I wanted it to be usable in case of problem

2) Shutdown OS and laptop, disconnect the USB external HDD and replace the internal 80 GB HDD by the new one

3) Rename the new pool rpool2 to rpool
     - Boot on a Solaris 11 Express LiveCD or the network using AI
    note: In my case, I used an AI server I had installed before (Solaris 11 express 2010.11 with no SRU)
    - zpool import rpool2 rpool        to rename the pool
    - zpool export rpool                    to export it so that there is no warning in step 4

4) Boot on the new HDD
    - It works just fine, but the pool size is still the size of the old HDD (80 GB)
      altough it uses a 500 GB partition (c4t0d0s0)

5) Increase the pool size to use the whole partition
     # zpool set autoexpand=on rpool 

The autoexpand really does an large amount of the trick. The size of a mirrored pool is always the size of the smallest disk. When you have an 80 GB and a 500 GB disk, the size of the pool is 80 GB. Remove the 80 GB disk. The smallest disk is now 500 GB and the size of the pool is 500 GB now as well, as long as you've activated autoexpand.

As i wrote before, you can reach my google+ profile by using the shortcut http://c0t0d0s0.org/+ (http:/moellenkamp.org/+ would work as well). This is done by an additional line in my apache configuration.

The relevant lines are:

        RewriteEngine On
        RewriteRule ^/\+$ https://plus.google.com/110984680346995237069 [R,L]

Of course you have to activate the mod_rewrite before and you have to use the url for your own profile. The easiest way is to copy the link location with your browser while hovering over the photo on the home page of your google+ account.

Really simple.

An overwhelming number of ZFS installations work with just a bunch of disks, perhaps in a JBOD or in the server itself. However there are installations, that use disk arrays with RAID-controllers. Some of those installations are even using a single LUN. I don’t think that this is a good idea (for e.g. because ZFS can just detect corruptions without redundancies, but not repair them) but that’s a different story I don’t want to discuss here.

There is a slight change in the default parameters of ZFS in Update 9. It’s related to the parameter zfs:zfs_vdev_max_pending . This parameter controls, how many I/O requests can be pending per vdev. For example when you have 100 disks visible from your OS with a zfs:zfs_vdev_max_pending of 2, you have 200 request outstanding at maximum. When you have 100 disks hidden behind your storage controller just showing a single LUN, you will have – you will know it – 2 pending requests at maximum.

You may think, that you could increase the queue depth without end, but as usual this is a tradeoff game and not that easy, longer queue depths may increase latency of the commands. Experience showed that certain queue depth delivered the best performance on most installations.

However the installed landscape changes and sometimes you have to adjust things. Exactly this happened a while ago in Opensolaris. And it seems that this change moved into Solaris. The default for zfs:zfs_vdev_max_pending is 10 at the moment. You can check this:

# echo zfs_vdev_max_pending::print | mdb –kw
0xa
#

0xa in decimal is 10.

And this is a wise choice for most implementations out there. But it was different on older versions. I checked it on U7, i asked my twitter/facebook contacts to make quick check on U8 as i was to lazy to install it:

# echo zfs_vdev_max_pending::print | mdb –kw
0x23
#

0x23 in decimal is 35 and 35 was the default up to Update 8 of Solaris 10.

So essentially the queues are less deep than before. For JBODs this is most often a good thing, as each vdev and thus each LUN has its own queue of 10 pending I/Os. For a single LUN hiding many disks sometimes not. So how do you change it back to the old value?
You can change it dynamically:

# echo zfs_vdev_max_pending/W0t35 | mdb –kw

To make this change boot-persistent you have to add a line to /etc/system:

set zfs:zfs_vdev_max_pending = 35 

Sometimes even an higher value may be indicated with very large numbers of disks behind your controller forming a single LUN.
How do you know if this decreased queue depth is a problem for you at all? The command iostat will help you:

jmoekamp@hivemind:~$ iostat -xdn
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    6,3    1,9  525,9   31,2  0,1  0,0   16,4    6,0   2   3 c3d0
   17,1    1,0 1676,0    8,0  0,2  0,1   11,4    4,8   4   4 c3d1
    6,4    1,9  525,8   31,2  0,1  0,0   14,1    4,8   2   2 c4d0
   17,1    1,0 1675,9    8,0  0,2  0,1   12,9    4,7   4   4 c4d1
    0,0    0,0    0,0    0,0  0,0  0,0    0,0    0,0   0   0 gsdbc
jmoekamp@hivemind:~$

If you see the column actv at or near the number of zfs:zfs_vdev_max_pending, it’s worth a try. Otherwise not.

I was asked in a comment, if Solaris supports power management with the processor in the HP N36L microserver. The answer is yes.

Best way to check this is via kstat. If kstat shows multiple frequencies as supported frequencies, it supports Power Management for the processor:

jmoekamp@hivemind:~$ kstat -m cpu_info | grep "supported_frequencies_Hz"
        supported_frequencies_Hz        800000000:1100000000:1300000000
        supported_frequencies_Hz        800000000:1100000000:1300000000

However before Solaris really uses it, you have to configure powermanagement. At first add or change the following lines in /etc/power.conf:

cpupm            enable
cpu_deep_idle enable
cpu-threshold    10s

Afterwards run the command pmconfig once. Now keep the system idling for 10 seconds and check the frequencies the system runs at:

jmoekamp@hivemind:~# kstat -m cpu_info | grep "current_clock_Hz"  
        current_clock_Hz                800000000
        current_clock_Hz                800000000

Robin Harris of Storagemojo pointed to an interesting article about about deduplication and it's impact to the resiliency of your data against data corruption on ACM Queue.

The problem in short: A considerable number of filesystems store important metadata at multiple locations. For example the ZFS rootblock is copied to three locations. Other filesystems have similar provisions to protect their metadata. However you can easily proof, that the rootblock pointer in the uberblock of ZFS for example is pointing to blocks with absolutely equal content in all three locatition (with zdb -uu and zdb -r). It has to be that way, because they are protected by the same checksum. A number of devices offer block level dedup, either as an option or as part of their inner workings. However when you store three identical blocks on them and the devices does block level dedup internally, the device may just deduplicated your redundant metadata to a block stored just once that is stored on the non-voilatile storage. When this block is corrupted, you have essentially three corrupted copies. Three hit with one bullet.

This is indeed an interesting problem: A device doing deduplication doesn't know if a block is important or just a datablock. This is the reason why I like deduplication like it's done in ZFS. It's an integrated part and so important parts don't get deduplicated away. A disk accessed by a block level interface doesn't know anything about the importance of a block. A metadata block is nothing different to it's inner mechanism than a normal data block because there is no way to tell that this is important and that those redundancies aren't allowed to fall prey to some clever deduplication mechanism.

Robin talks about this in regard of the Sandforce disk controllers who use a kind of dedup to reduce some of the nasty effects of writing data to flash, but the problem is much broader. However this is relevant whenever you are using a device with block level deduplication. It's just the point that you have to activate it for most implementation by command, whereas certain devices do this by default or by design and you don't know about it. However I'm not perfectly sure about that … given that storage administration and server administration are often different groups with different business objectives I would ask your storage guys if they have activated dedup without telling somebody elase on their boxes in order to speak less often with the storage sales rep.

The problem is even more interesting with ZFS. You may use ditto blocks to protect important data to store multiple copies of data in the pool to increase redundancy, even when your pool just consists out of one disk or just a striped set of disk. However when your device is doing dedup internally it may remove your redundancy before it hits the nonvolatile storage. You've won nothing. Just spend your disk quota on the the LUNs in the SAN and you make your disk admin happy because of the good dedup ratio However you can just fall in this specific "deduped ditto block"trap when your pool just consists out of a single device, because ZFS writes ditto blocks on different disks, when there is more than just one disk. Yet another reason why you should spend some extra-thought when putting your zpool on a single LUN, especially when the LUN is sliced and dices out of a large heap of storage devices by a storage controller.

However I have one problem with the articles and their specific mention of ZFS: You can just hit by this problem when you are using the deduplicating device for the pool. However in the specifically mentioned case of SSD this isn't the usecase. Most implementations of SSD in conjunction with ZFS are hybrid storage pools and so rotating rust disk is used as pool and SSD are used as L2ARC/sZIL. And there it simply doesn't matter: When you really have to resort to the sZIL (your system went down, it doesn't matter of one block or several blocks are corrupt, you have to fail back to the last known good transaction group the device. On the other side, when a block in L2ARC is corrupt, you simply read it from the pool and in HSP implementations this is the already mentioned rust.

In conjunction with ZFS this is more interesting when using a storage array, that is capable to do dedup and where you use LUNs for your pool. However as mentioned before, on those devices it's a user made decision to do so, and so it's less probable that you deduplicating your redundancies.

Other filesystems lacking acapability similar to hybrid storage pools are more "haunted" by this problem of SSD using dedup-like mechanisms internally, because those filesystem really store the data on the the SSD instead of using it just as accelerating devices.

However at the end Robin is correct: It's jet another point why protecting your data by creating redundancies by dispersing it several disks (by mirror or parity RAIDs) is really important. No dedup mechanism inside a device can dedup away your redundancy when you write it to a totally different and indepenent device.

At approx. 16:20 the poll had the following outcome:

Gestern habe ich die Mitteilung bekommen, das mein Vortrag "Deduplication" auf der Froscon 2011 angenommen worden ist. Die Froscon findet dieses Jahr am 20./21. August 2011 an Hochschule Bonn-Rhein-Sieg statt. Ich werde dort eine vollständig überarbeitete Version meines Deduplication-Vortrags von der GUUG halten. An vielen stellen allgemeiner und nicht so ZFS-zentrisch, an anderen Stellen allerdings sehr viel ZFS-zentrischer, da ich anhand des Sourcecodes von ZFS einige Konzepte erklaeren will.

The Solaris 10/11 Express Hardware compatibility list has a new home: It's now part of the OTN and available at http://www.oracle.com/webfolder/technetwork/hcl/index.html

If it weren't about the AF447 disaster, some comments of readers in some of the forums were actually funny. Especially those ones writing that Air France just had to install GPS in addition to pitots to indicate the speed of the aircraft when the pitot tubes got unreliable due to icing. After all the el-cheapo navigation system for the car shows the speed quite well. Well, while the speed on the GPS display is pretty precise, it has almost no value for keeping the aircraft in the air.

As you you may from school aircrafts are keep in the air by a force called lift. And aircraft has to create at least enough lift force to neutralize the force of gravity. The lift is dependent on the speed you move the airfoil through the air. Okay, that's vastly simplified but sufficient for my explanation. Dependent on the aircraft design the needed speed to create such an amount of lift is different: For example the Fieseler Fi156 has an incredible low minimum speed: 27 knots. Out of this reason this aircraft is capable to fly backwards in really strong winds. But this backwards flying (negative speed if you want to say so) is just in relation to ground … the airspeed is still positive. The difference between the speed of the wind and the speed of the aircraft have just to be larger than the minimal speed of the aircraft and the wind speed must be larger than the aircraft speed. Don't believe me … just go into the next harbor and look at the gulls. Of course you won't see a Boeing or an Airbus backwards … at least not with usual winds.

Another nice example are aircraft carriers: Those have really large engines to accelerate to and to keep top speed. Out of a reason: When an aircraft carrier moves with 30 knots, the air on the flight deck on the aircraft carrier already moves along the airfoil with 30 knots even on calm day. Let's assume a fresh breeze (6 beaufort) or 30kn. When the carrier steers in the direction with the wind, the air would move along the airfoil … well … not at all. When the carrier goes in the direction against the wind, the air would flow along the airfoil with 60 kn.

That's the reason why an aircraft carrier turns into the direction against the wind to start aircrafts. It gives the aircraft a headstart in order to generate lift. And to get an aircraft to 30kn you need really big engines.

To get back to the difference between airspeed and speed over ground? What's the speed over ground? 30 knots. In all cases. But you have three different air speeds. The GPS only measures the speed over ground. However as mentioned above the lift - the force keeping the aircraft in the air - is dependent of the air speed. At 0 airspeed no lift. At 60 kn a lot of lift. At 30 kn airspeed, something in between. And then you have the jet engines and the catapult.

Speed over ground is something different than air speed. And a GPS can't measure air speed. The air speed can only be measured by measuring the speed of the air flowing along the aircraft. To do so, modern aircrafts use so called pitot tubes in conjunction with measuring equipment to measure the static pressure (the ambient air pressure). The pitot tube measures the ram air pressure into the pitot tube. If there is something blocking the air from pushing into the probe, it can't measure the speed.

So … a GPS receiver would have helped the pilots of AF447 not a bit. There are other means like the Backup Speed Scale but that is working completely different.