General

With a link to apple.com, too. Directly on the front page ...

The "Steve Jobs" links to:

BTW ... the hungarian folk dancers explain other sort algorithms as well: merge-sort, insert-sort, bubble-sort, shell-sort and select-sort

Recently I was doing some work in regard of tuning systems. There is something i really hate about this topic of computing: Tuning scripts. You find them on google easily and i find them on systems at customers quite often.

Simply said: I hate them. The reasons for it are simple. For example recently I found a system with a networking tuning script dating back into 2003 or so. The problem: It was meant to increase some of the settings. However many of them were already higher in the default config of current Solaris 10 versions, thus the tuning script essentially reduced the parameters and thus reduced the performance.

Futhermore: Tuning is a lot about understanding things. Understanding how things work together. On a systemic, on an architectural level. How an application loads all the rest of components. Just dropping a script downloaded from a website found by Google - into /etc/init.d is not about understanding things.

You have to carefully consider each change from the default about the impacts. You have to check each setting, if the setting hasn’t already overtaken by the years. You have to recheck it it with every major update of your environment. You have to recheck it with each new technology you are using in your system. Network tuning scripts dating back to a time when 100 MB/s were normal and 1 GB/s are fast aren’t necessarily up to the task in a time when 10 GBit/s are fast and Infiniband IPoIB networks deliver even more. You had to turn different knobs in a time, when cpu time was precious. You’ve tuned for minium cpu utilization. CPU isn’t a large factor today, you tune for minimize latency or maximize throughput. You have to know what you want to aim for, because minimum latency and maximum throughput are often mutually exclusive. Do you want an extreme or a target in between.

Just using a script to tune something doesn’t lead you through all the thought to make really good tuning decisions.

There are some basic rules from my point of view:

• For Solaris there is for each major update a document updated called “Oracle Solaris Tunable Parameters Reference Manual”. It’s available at here.Check your tuning script at each update against this document in order to ensure that you settings make sense. The 817-0404 is a great read anyway, as it gives you an condensed overview over the possible official knobs to turn.
• Each tuning script has to be carefully designed for each system for each workload. An /etc/system for a database make perhaps no sense for a webserver or may even reduce the performance and vice versa. That’s the reason why I don’t like tuning scripts as part of jumpstart configurations. An NFS server delivering many small files to many clients may need other tunings than a server delivering few large files to a few clients.
• Understand! Understand how components of your architecture interact. Understand the load. Monitor the load for a while. Tuning is at foremost doing a lot of research before and just at last about putting some lines into some scripts. That’s one of the reason why I like tools like dtrace in general, and the scripts from the dtracetoolkit especially. Those tools are like the professionals tools of a mechanic. They can help you to understand the load. The equivalent of the basic set of screwdrivers: All the *stat tools are helpful, too. You need them to know where you can start with the more advanced tools, that are often too focused.
• Have a benchmark load. Have a mechanism to reproduce the load on the system. The more realistic and the less synthetic the better. So you can quantify the impact of each tuning.
• Just change one knob at a time. Otherwise you will never know what setting really helped you or which setting hurted the performance
• Of course: Experience helps a lot in shorten the research phase and sometimes there are some low-brainer settings. But even when you think “Nah … this is just a webserver. Let’s use script number 7b”, you may be surprised by interesting side effects. So there is such think like a no-brainer in tuning
• Large standard applications have often a set of certain tuning parameters that the vendor suggest as a good baseline: At first - look them up in the tuning guide in order if the ISV tuning guide wasn't overtaken by the defaults in the OS. At second: Check them with the vendors manuals with each release ... had once a performance problem with customer that used the tuning parameters suggested for - let's say - version 2, however there weren't that reasonable with version 5.
• It's a good practice to create two baselines: At first check the service performance with a totally untuned system ... just modify settings you need to get the software up-and-running.Run your benchmark. Configure the system with the vendor suggested parameters. Rerun the benchmark. Now do your own tunings. Rerun the benchmark with each knob you turn. And ask yourself: Is the performance impact worth the deviation from the standard parameters from a plain vanilla installation? With such baselines you can assess the impact and thus the value.

A commentator at hackernews asked how i think about -9. In my opinion: It's widespread use is a similar plaque like the –f switch. And this is pretty easy to explain (I'm simplifying things a bit).

-9 is a shorthand for SIGKILL. When you send a SIGKILL to a process, the process is terminated immediately. You can’t catch this signal, you can’t ignore it. A kill with -9 sends this SIGKILL to a process.

A kill without -9 sends a SIGTERM to process. It terminates the process like SIGKILL. However a process is allowed to catch it in order to execute a signal handler … or just ignores to ignore it. A signal handler is nothing more than a code path that is executed when the process receives a signal.

So when you kill a process with a normal kill you give the process the chance to clean up behind itself, to make files consistent, to roll back changes in the case the process isn’t using some transactional mechanisms when changing data, to delete temporary files … and so on ...

It's a good style to write such signal handlers and in many programming languages it's pretty easy. For example in perl:

$SIG{TERM}=\&signalhandler_TERM; 

sub signalhandler_TERM{ 
     $SIG{TERM} = \&signalhandler_TERM; 
     print "leaving process\n";
     #some code to clean up behind you 
     exit(1); 
}

When you send a -9 to a process you take away this chance from the process. It’s killed instantly … even if it just started to modify your files, fscking up your data in order to put it in a new form, even when you have created dozens of temporary files filling up /tmp. Things like that …

Killing a process with -9 is the last possibility. However I see people using it too often too early. A second after the normal kill is send a pgrep on the process follows. Still there and the sword of -9 is falling down.

When a process doesn’t disappear immediately after sending the SIGTERM, it may be just busy to follow your order of terminating itself and is cleaning up things. When your application is dependent to precious resources at cleaning up (for example IOPS on your rotating rust) the process of cleaning up may take a while.

The implicit question in any process, that doesn't react to a normal kill via SIGTERM is the question why it doesn't react to the signal. Just sending a -9 when a normal kill didn't worked is like "Do not care". Monitoring the process with truss or strace what the heck the process is doing after getting the SIGTERM is a good first step. Perhaps you see some cleanup work and know that you just have to wait a little bit longer. Writing a core dump of the process with gcore is often a good second step to save evidence for future research why the process didn't reacted.

And then … and only then … a kill -9 may be feasible.

In short:

Always try a normal kill first. There is just one execption.
• The exception: You know that the integrity of data is depending on the fact, that the process has no chance to write anything for example by executing a signal handler. Consider it as something like a forced panic on process level. The panic is the admins friend as it ensures that nothing is written to disk as soon as the OS is detecting something worth to trigger a panic. A panic isn't a "insult the admin"-thing … it's about protecting data.
• kill -9 is your emergency stop switch and shouldn't be part of your normal administrative procedures in any way
• Just sending a -9 after a normal kill without thinking about reasons is bad administrative style.

Yesterday i held my deduplication talk at the Froscon 2011. I think it was acceptable and the lecture room 3 was really filled. To be honest: I don't expected such an audience at that time. So a big thank you for all who attended the talk.

My talk started at 10:00 o'clock and thanks to apron parking, a rental car pickup in the feeled middle of nowhere in Düsseldorf and a larger traffic jam near Cologne led to an arrival at 09:58 … with a presentation at 10:00 and the urgent need to visit the restroom before a disaster happens.

I had just a short time at Froscon … i had a date with my house, that still needs work and don't accept when i'm saying "No … not this weekend". And thus i was back in the rental car at 11:45 and was back in Lueneburg shortly after 15:00, ten hours after leaving home. 3.30h for 430km … not that bad for an Opel Corsa.

However there is something i've recognized: I'm really missing standing in front of customers and technically interested people and trying to transfer my enthusiasm for technology. I'm really missing being on the road.

Ryanair? I will never ever put a foot in one of their aircrafts. Even when I have to book an private jet instead … I will never ever put a feet into one of their aircrafts. Period.
The flight back to Bremen was my 250th flight since loosing my fear of flying five - maybe six - years ago. I experienced a lot of things on my journeys … an aircraft from FRA to TXL that turned back to FRA just short of TXL because of inoperational slats. Unfriendly stewardesses. A go-around once. Approaches giving a roller coaster a run for their money. But yesterday was something completely new to me.

I have to explain something different first: For all the work and the hassles with helping me with the construction of my new home, I invited my mother, two of my sisters and my brother and his girlfriend to London for a one-day trip (my father can't use an aircraft out of health reasons). I found a reasonable cheap ticket at Ryanair and with the flight pricing at other carriers such day would much more expensive.

There is something important to know to undestand the whole situation: My brothers girlfriend is from Chile, she is living in Hamburg for a long time now. So she is not an EU citizen We knew that you have to go to the check-in to get a stamp on the self-printed ticket when you are not a resident of the European Union. However when going through the boarding in Bremen this was an integral process of boarding along with the weighing of the hand luggage. They checked the documents, put the stamp on it and we were able to enter the plane without problems.

We had a nice day in London and at the end we went back to Stansted. It was a really great day. We were a little bit low on time however more than sufficient to do some last minute souvenir shopping, going through security, getting to the gate, albeit we were at the rear end of the row, boarding hasn't started at that time.

So my sisters and my mother went through the gate no much later, I went to through the gate and walked down in order to help my family (they don't fly that often), however when walking on apron I realized that my brother and his girlfriend didn't went with us. However I entered the aircraft to look after the rest of my family. But after an additional minute my brother and his girlfriend were still not on board. So what had happened? I had already an idea because as a experience troubleshooter, you immediately go through the possible failure modes of an situation. The stamp. Dang. But at that moment I thought "Well, they will check that and put the stamp on it when all people have boarded in order to speed up boarding". However when they closed the doors, I quickly called my brother (call ended before push-back) and I was right with my foreboding.

The girlfriend of my brother wasn't allowed to enter the aircraft because this stamp was missing. Just to explain it again: It's not an official stamp, it's a stamp the Ryanair visa counter put onto the ticket just to show, that you have showed them your passport that you have the valid visa. She had all the needed documents. Nothing was missing, nothing was unclear. (The proof : She flew with Easyjet to Hamburg today - with no hassles .. with no problems).The stamp was missing, they denied boarding, they closed the gate, and I flew with my mother and my sisters to Bremen. Of course my brother stayed in London as he couldn't leave his girlfriend behind.

Yes, we made the error not to go to the Visa counter as we thought in stress that the persons in front of the security have the same role as in Bremen. And so I have no hope to write a complaint that will yield more than a generic mail.

However the behaviour of "Don't give a shit, if you have to enter the flight as you need to get home and your group is already on the plane" attitude, the denied boarding because of a missing stamp Ryanair put themself on the printed ticked as part of their processes, the merciless closing of the gate, the inability of checking the visa at the gate (in my experience the people at the check-in and at the gate are often recruited from the same group of people, so it's not uncommon to see the person from the check-in at the gate again). This is an unacceptable behaviour to me. It's just unfriendly and it just show that passengers at Ryanair are just sentient lifestock that they have to fly from A to B.

I think at any other airline we could have used, they would have checked it at the gate, perhaps after calling the supervisor, and allowed her to board. But not at Ryanair.
Yes I know it's my fault. I booked at Ryanair. I would never do that for when just flying alone: I'm prefering Lufthansa and Germanwings, but Easyjet or Air Berlin are good as well e.g. . However I was able to purchase six return tickets at 144 Euro and that looked as a good deal then (half of it was credit card costs as I don't own the only obscure card allowing you to pay without extra costs). I knew that I have to expect nothing that just gives you hint that you are considered as valued customer. You know that when you book Ryanair.

How did this story found it's happy end: We were limited to help from remote: My father and I were able to help. They drove at night by bus to London Victoria and at the end they were able to sleep in a bed instead of a airport chair (70 GBP), I was able to book an Easyjet flight at 1:30 AM for (160 Euros for 2 tickets). 1:30 because I drove my sis from Oldenburg home and then back to Lueneburg. To add insult to injury: As my brother had my car keys, I had to rent a car (61 Euros) in Bremen … however I was glad that there was still a rental car at 21:45 at this rather small airport.

This morning my brother and his girl friend used the train from St. Pancras to go Luton (don't know what this will add to the bill). Had to find out a lot of things this morning how to get them to the airport in Luton without the need to print something out. E.g. the EasyJet bus service to Luton don't have a "buy ticket at curbside" option according to the website) … dang. My brother and his girlfriend were rather jazzed by situation and fscking tired, so I helped them by phone to get though London.

BTW: A big thankyou to my manager Kristan that he gave me an additional day of vacation without asking questions … so I had time to work things out

Just rebooking the flight wasn't an option. Rebooking 100 Euros … per person. As they would have to go to the checkin to get the boarding passes (as they hadn't an portable printer): 40 Euros … per person. Nice extra 280 Euros for Ryanair due to denied boarding. You could even speculate, if this is the reason for such a draconian enforcement of rules. Don't know what they want to achieve with such an behaviour. However I can tell you, what they achieved: Ryanair will not get a single Euro from me …

My brother arrived in Hamburg he is driving to Bremen in order to fetch my car from the car parking. Hell knows what this will add to the bill as you don't know if you just rented the parking lot for 24 hours or if you actually purchased it for life …

All this because of a missing stamp. And perhaps you know understand why I would rather use a bathtub and a paddle to get from UK to Europe than to board a Ryanair aircraft. Ryanair lost 9 customers yesterday and I suspect some additional people in the family of my brothers girlfriend. And when just one reader is as disgusted from this behavior like me the time invested in this blog entry is a better investment than the time I would need to write complaint that would be finally futile at the end.

PS: As fate isn't without a sense of irony, the aircraft hat to go around at Hamburg and they flew circles for an additional 15 minutes because of problems with the runway in HAM. When it rains it pours.

A few days i linked to a paper, that explains why it's hard to really delete all data from an SSD. You will find the link to the paper in that article. In those paper, the authors argued, that the SSD need better mechanisms to securely delete data. A paper from 2010 goes in the opposite direction, argueing that SSD make forensics more difficult due to some tuning tricks used in SSD.

When you look at the paper "Solid State Drives: The Beginning of the End for Current Practice in Digital Forensic Recovery?", the situation gets a little bit more complex. The document looks from the perspective of people, who try to gather data from disks for example for law enforcement.

The basic problem is: To keep performance at a high level, modern SSDs are using garbage collection. This is a good thing, as writing into an already used block needs twice as long as writing in an empty block. So the internal controller of the SSD runs a garbage collection that erases areas before the free areas is needed. This is done, when the SSD is idling. On page 8 it's

The X axis shows time and the Y axis shows the approximate percentage of the drive that has been zeroed out. In all three SSD runs, around 160 seconds from the log-in time (i.e. around 200 seconds from power-on), the SSD begins to wipe the drive. After approximately 300 seconds from log-in, the SSD consistently appears to pause briefly before continuing. 350 seconds after log-in, the SSD’s pre-existing evidence data has been almost entirely wiped.

To comparision: On a quick formated HDD, they were able to recover almost all data.

The problem for the digital forensics is: The disk is doing this after a short time you've powered it up. It just wipe out those blocks it knows they are unused by some clever algorithms. And it gets even worse: When your OS uses the TRIM command, it doesn't have to find out by clever tricks, the OS tells the blocks that can be wiped out and it directly wipe them after the TRIM command when there is some spare on the SSD, as the SSD is doing this on it's own controller. And this is the next problem: You can't do anything top stop besides of powering the device down. Perhaps the only way to get to the data is to use a device like the flash chip reader from the article about secure deletion to circumvent the flash firmware in total. But i don't think a police team can seize the computer and power it down in 300 seconds. However it's still not a secure delete, as they were still able to get fragments from the disk.

The conclusion is really interesting:

It seems possible that the golden age for forensic recovery and analysis of deleted data and deleted metadata may now be ending.

No wonder law enforcement agency are so keen on having trojans in their repository. Analysing SSD looks like a really futile endeavour.

An interesting ARC case found it's way into the case log: Stephen Lawrence proposed the zonestat to the architecture review comitee.

This feature consists out of two components. The zonestatd and the zonestat command. Stephen writes about the zonestatd:

A new smf service svc:/system/zones-monitoring:default will execute a zones monitoring daemon. This daemon will implement monitoring the system configuration, managing extended accounting, and computing memory utilization on behalf of clients.

This zonestatd is queried by the zonestat command to deliver data about the resource utilization of zones.

# zonestat 5 1
    SUMMARY
                        -----CPU------------- ----PHYSICAL--- ----VIRTUAL----
                   ZONE USED %PART %CAP %SHRU USED  PCT  %CAP  USED  PCT %CAP
                [total] 9.74   30%    -     - 7576M  23%    - 11.6G  24%    -
               [system] 0.28  0.8%    -     - 6535M  19%    - 10.4G  21%    -
                 global 9.10   28%    -     -  272M 0.8%    -  366M 0.7%    -
              kodiak-ab 0.32  1.0%    -     -  256M 0.7%    -  265M 0.5%    -
             kodiak-rie 0.00  0.0%    -     - 22.7M 0.0%    - 49.4M 0.1%    -

Due to the implementation with a daemonized part, you can even tell the daemon to compute a report about the utilization in certain intervals:

# zonestat -p -r memory -q -R average,high 30s 24h 1h

Please read Stephens case document for for more information.

In unserem Staatssystem hat ja der Bundespräsident in der Hauptsache ornamentale Funktion, seit Richard von Weizsäcker aber auch die Rolle eines über den Dingen stehenden Mahners, des über den Dingen stehenden Integrierers. Jetzt wurde nach dem Köhler-Rücktritt Frau von der Leyen ins Gespräch gebracht. Vielleicht vermag Frau von der Leyen die ornamentale Funktion auszufüllen, ich sehe aber nicht wie diese Dame die Funktion des Mahners ausfüllen soll ... alleine schon durch ihr recht geringes Alter. Herr Weizsäcker, Herr Herzog und Herr Rau, die ich alle als Bundespräsidenten betrachte, die dem Anspruch ihres Amtes gerecht wurden, waren bei Amtsübernahme 60 und mehr Jahre alt. Frau Von der Leyen ist gerade einmal etwas über 50. Und was den Integrierer angeht ... nunja .. was darf man von einer Frau halten, die trotz besseren Wissens des Expertentums auf ihren Netzsperren bestand und nicht die geringsten Anstalten machte, diese in die Diskussion zu integrieren. Also von meiner Seite gibt es da ein klares "Noe ... danke". Wenn man die Dame abschieben möchte, dann doch bitte nicht ins Präsidentenamt ...

Von all den gehandelten Kandidaten erscheinen mir Joschka Fischer, Herr Gauck und Herr Töpfer eigentlich am geeignesten ...

Schüler-VZ laesst sich Daten von 1.6 Millionen User abspidern, die Privacystrategie von Facebook ist nebuloes. Loyality-Karten werden noch und nöcher benutzt. Kreditkarten sowieso. Millionen Deutsche tragen ein Trackingdevice mit sich rum, das es dem Staat leicht macht, sie zu orten (Es gibt Leute, die nennen dieses Teil Mobiltelephon).

Aber trotzdem regen sich die Medien darüber auf, das Google Daten versehentlich (und das glaube ich ihnen "Cool, hier hat schon jemand code geschrieben, der Beacons rauswirft. Muss ich das nicht tun ....") mitgeschnitten hat, die zu nichts nuetzen (zu wenig Daten um die Verschluesselung zu cracken, zu gering die Wahrscheinlichkeit beim Durchfahren zufällig irgendwas sinnvolles zu lesen, und wer in einem WLAN ein unverschlüsseltes Protokoll benutzt, gehoert eh ausgepeitscht), meist verschluesselt sind, ohnehin öffentlich sind und meines wissens keine Reidentifizierbarkeit zulässt (Wer sein Netzwerk schön selbstbeschreibend beispielsweise "JoergMoellenkampsWLAN" nenn, ist selber schuld, ansonsten wäre mir keine Liste bekannt in der WLANs mit MAC-Addressen auf Personen gemapt werden).

Ich sehe da nicht mal einen Fehler von Google. Die Rohdaten sind nicht öffentlich, sowie vermutlich gefiltert und gereinigt (naemlich reduziert auf Koordinaten und MAC-Adresse, alles andere wäre wohl Platzverschwendung) und nur über eine API verfügbar. Es sind Daten, die jeder x-beliebige Computernutzer auch hätte abgreifen können.

Man muss hier Google zugute halten, das sie einen Fehler zugegeben haben, der keinerlei praktische Bedeutung hat. Die Kristik ist eher der unreflektierte, aber übliche Google-Profilierungsbeissreflex, der da mittlerweile andauernd an den Tag tritt.

Wir haben andere Privacy-Probleme ... alle wichtiger als die Traffic-Fragmente, die ihren Weg auf die Festplatten der Streetview-Autos gefunden haben.

Eine Frage am Rande übrigens: Was glauben die Leute eigentlich, wie die Geschwindigkeitssymbole und Strassenschilder in die Navigationskarten ihres bevorzugten Navigationssystems kommen?

IPMP vs. Link aggregation

Link aggregation is available on many switches for quite a while now. With link aggregation it is possible to bundle a number of interfaces into a single logical interface.

"Nichts ist so beständig wie die ständige Veränderung", "Planung heisst den Zufall durch den Irrtum zu ersetzen" ... es gibt eine Vielzahl ähnlicher Grundsätze im Leben. Und genau das ist jetzt auch bei mir eingetreten: Mein Umzug nach Bremen findet nicht statt. Die Zeit zwischen den Jahren hat Gründe für Bremen/Oldenburg verschwinden lassen und somit deutet wieder alles auf Hamburg. Also: Ich bleibe Hamburger.