c0t0d0s0.org

While sitting in a workshop meeting, my thoughts went away from the meeting and circled around the Turbo Boost feature of the Nehalem processor. Too make it clear at the beginning: Turbo Boost is a good feature. It helps to use the existent potential of the CPU instead of the expected potential. Overclockers have shown in the past, das most CPU have more performance potential as the print on the heat spreader says. But while I think this is a good idea, this will open some interesting benchmarketing tricks.

There must be a standard course at the IBM marketing school. The tricks of the benchmarkting are the same again and again. At least it doesn´t get boring to read at the IBM website. Now i found another benchmarketing stunt at their website. It´s called "Move Up to IBM Power Systems".

It was really fun to read it. IBM Benchmarketing - now with 100% more nonsense. One important disclaimer: The following article just contains the stuff that was obvious at my first read. I have a daytime job that isn´t dissecting IBM marketing stunts

I already wrote about the latest benchmarketing trick of IBM in the last blog article, but this article was in german, so i repeat this in english. There was an bold statement in a recent press anouncement of IBM - "IBM Builds on Industry-Leading UNIX Portfolio With New Servers, Software":

The Power 560 can save up companies up to $840,000 and 80-percent in energy by consolidating 13 Sun Fire V490 servers on a single Power 560 server with PowerVM, as compared to consolidating the same number on four Sun SPARC Enterprise M5000 servers with dynamic system domains."

I´ve asked myself, how they get to such numbers. This number of servers couldn´t based on performance. We don´t need 4 M5000 just to substitute 13 V490. But after thinking about after reading the article in the Computerwoche i found out what´s the trick of this comparision. The trick is a cheap one ... even for IBM marketing.

You can partition an M5000 in up to 4 domains. When you just want to consolidate 13 servers, you obviously need 4 systems. This comparision doesn´t compare the compute power of the M5000 with the compute power of the p560. It compares two different virtualisation technologies. So the even the choice of 13 V490 is a really perfidious one. Twelve systems to consolidate would lead to 3 M5000, 13 systems lead to 4 because you have one domain too few. But that´s not the point: You won´t consolidate 13 V490 by using domains. You would use Solaris Containers (perhaps in conjunction with Solaris 9 Containers) for this tasks. By using this Containers you would need only one system, too. And you would need less processing power for it, as Container are a more efficient virtualisation technology in comparison to *PARS.

By the way: The answer "one system" is false for both systems. Independently from the system architecture, virtualisation technology you want at least two systems and a cluster when you consolidate 13 systems on one. Without an additional standby system you are toast in the case of a system failure or maintainance. But that´s a persistent error in every benchmarketing comparision of IBM.

When you look at the recent SAP SD certifications about the HP and the Sun systems based on Intels new hexacore proc, a cursory observer may have the impression, that either Solaris10+MaxDB is much slower than Windows+SQLServer or that Sun has somehow has fscked up the System. But as usual the explanation is in the details: The HP/Microsoft result is a non-unicode result, the Sun result is a unicode result. Why is this important? There is a good presentation made by SAP about the expected performance impact: between 15% more CPU load on new procs and 30% more CPU load on older procs and 50% more memory by using Unicode. So it´s important to know for reading benchmarketing: You can´t compare non-unicode and unicode benchmark results for SAP.

I really think this is benchmarketing at full throttle. IBM published an new TPC-C record for non-clustered systems. Okay, TPC-C is meaningless, but it´s fun to read the disclosures. Before i start: I hope you remeber the saying "TPC-C is a function about the number of disks, not a benchmark of the complete system".

A configuration with IBM Power 595 Server Model 9119-FHA yielded a TPC-C throughput of 6,085,166 transactions. That´s impressive. Even more impressive: This benchmarketing configuration used the small number of 10.992 disks with 73.4 GB capacity (The second on the list, an HP Superdome, reached 4,092,799 with just 6000 disks ). Ten thousand disks ... sounds like a real-world configuration

PS: It´s a disclosure with an availability 6 month in the future again, the maximum amount of time allowed under the rules of the TPC.

Many intelligent people work at IBM. I know some of them. So i have some difficulties to understand such blog postings like this one. Mr. Pearson touts in "Yes, Jon, there is a mainframe that can help replace 1500 x86 servers" the myth again, that you can consolidate 1500 x86 servers on one big mainframe. In a strange way, he is correct. You can consolidate this number of servers. But only in a certain corner case: Almost all of this servers has to run almost without any load. Mr Pearson writes:

125 Production machines running 70 percent busy
125 Backup machines running idle ready for active failover in case a production machine fails
1250 machines for test, development and quality assurance, running at 5 percent average utilization

What does this mean? Well ... To get to this 10% myth you add 125 idle machines and 1250 almost idle myth. I think it's save to assume that consolidating servers without load is like fishing in a barrel.

The next joke of this blog entry is how they compare to a solution with x86 servers. At first they do the utilisation game again. This time they assume, that every x86 system is only loaded at 10%. (I assume this is the reason for assuming this large amount of test systems). By the way: Obviously they use Sun equipment for such tasks, not Dell, HP or something like that ... don't know why they have this preference

Besides all this nonsense in this blog entry: Nobody would buy a multi-million mainframe for such a task. You would buy some large x86 systems like the X4450, X4440 or X4600 with up to 32 x86 cores. To consolidate your test servers you would use VMware, Xen, xVM, Container or something like that. Let's play IBM's game for a second. Let's assume we can rise the load to 90% by using xVM Server. So we can consolidate 9 systems on a two core two sockets machine. We use a system with twice the sockets and twice the cores. So we would be able to consolidate 36 system on a 2 rack unit system.

You would need 41 of this systems for the idle systems. Now let's assume you need the power of 105 (150 systems at 70%) systems equivalent to the X2200M2 for the production systems. Divide this by 4 as we use more powerful systems. Makes 25 systems. Add 5 spare systems to move services in case of the failure of a single system. You need a total of 71 X4440 systems to consolidate the load. List price for a quadcore quad socket system is $ 13,445.00 at the moment. This leads to a price of roundabout 954595$. For this price you get 80 TB of local boot storage, 4.4 TB main memory with 1136 cores of computing power. I'm not sure about the exact pricing of a 64-way z10 mainframe but i'm sure that you can't buy it for a million list price.

I want to quote an article from the ITjungle:

IBM did not, as you might imagine, announce list prices for these new z10 EC boxes, but Karl Freund, vice president of marketing for the System z line (who used to have the same job in the System p line) said that the entry configuration of the z10 EC starts at around $1 million, not including software; he didn't tell me I was crazy when I suggested that a fully loaded z10 EC with 64 cores activated for processing, lots of main memory, and other required hardware would cost tens of millions of dollars. Even at $1,000 per MIPS, it works out to $30 million, and at $1,500 per MIPS, that's $45 million.

$30 million hardware investment to substitute $1 million worth of x86 servers. What a bargain ....

Well, however you turn it ... consolidating Unix workloads on mainframes stays a bad decision. And i didn't even talked about ISV support, software availability, know-how (you know, there are not that much of non-retired mainframe specialists)

By the way: There is another interesting gem in the blog entry:

While a 1-way z10 EC can handle 920 MIPS, the 64-way can only handle 30,657 MIPS. The 29,477 MIPS needed for the Sun x2100 workload can be handled by a 61-way, giving you three extra processors to handle unexpected peaks in workload.

Somewhere on the road from 1 to 64 ways the system looses half of it's performance amount of performance. When you assume perfect scaling you would expect 58880 MIPS. But you get only 52% of it and that's far from a reasonable scaling. For each way you spend you get a half way back. You buy 64 ways and get only 32 ways back. Just for comparision: Using 64 cores in a M8000 gives you the performance of 55 cores (based on a conservative internal benchmark with real applications, not SPECint_rate or something easy like that)

IBM understands the art of nonsensical comparision to perfection. IBM Germany tries to convince the customers, that Sun is extremly expensive. Okay, at first: They use the oldest trick in their portfolio again. Comparing a new system with an old system. The p570 is a system introduced last year. The UltraSPARC IV+ with 1.8 GHz was introduced in August 2006. We´ve introduced the UltraSPARC T2 and the SPARC64-based system since.

Okay, but let´s play the game of IBM. Just for fun ...

Wow, it´s benchmarketing time at IBM again. IBM Announces DB2 and POWER6 Combo Trounces HP and Sun by 2x and 3x Respectively in New Business Intelligence Benchmark. IBM you are soooo great .... you was able to beat our TPC-H 10 TB by a factor of 3. It´s really an technological archievement in 2007 to beat a TPC-H benchmark result generated on on 29th November 2005 ... exactly 2 years ago today. Come on ... in IT 2 years are a fscking long time ....

Everytime i read a benchmark from IBM, i really looking forward to read their disclosure to see, what you can do in benchmarking, when you have nearly unlimited resources.

The configuration of IBM consists out of: 32 p6 p570 servers, each server had 32 GB of memory, resulting in 1 TB memory. Each of the systems had 96 36 Gigabyte 15k harddisks with 4 GB/s FC. A whooping amount of 3072 harddisks. A fully 10 GB enabled network.

The SF25K result was done on 1296 disk, with 10k rounds with 2 GB/s. The memory had a size of 288 GB. On a single system with a single operating system image The system used the old UltraSPARC with 1.5 GHz

Not that you can´t buy an system from Sun equivalent system from IBM. The venerable SF25k now has up to 1.15 TB per domain. And the M9000 class servers are a completly different story. But,dear IBM,when you want to make a statement of relative performance, the system should be at least somewhat similar.

So dear IBM, next time you want to showcase you system, don´t do it against a benchmark 2 years old. What´s the statement of such a press announcement: We are 3 times faster with our top of the line system with a benchmarking configuration than a 2 years old configuration. Big deal, IBM, really big deal ...

When you read disclosures about somewhat strange benchmark configurations, it would be nice to mandate explanations of benchmark configuration. Why did you opt for this configuration? For example: "As part x of benchmark y is disk-bound, the most feasible choice was doubling the harddisks" or "As TPC-H scales better on our smaller nodes, we choosed them in favor of larger nodes"). The customer can learn a thing or two about hardware sizing and the tradeoff game in finding the right hardware and the manufacture can hide less issues in their product portfolio. Sometimes it´s more interesting to know why a system wasn´t used instead of knowing the outcome of the benchmark for the choosen system.

Nun, das Heise sagen wir mal sehr IBM-minded ist (ich verkneife mir mal stärkere, möglicherweise justitiable Ausdrücke) ist, kann man wieder mal sehr schön am Heise Artikel über das UltraSPARC IV+ Upgrade:

Sun steigert die Taktfrequenz der UltraSparc-IV+-Prozessoren von 1,8 auf 1,95 und 2,1 GHz. Die CPUs kommen in den Sun-Fire-Servern V490, V890, E4900, E6900, E20K und E25K zum Einsatz. Der nächste größere Schritt soll für Sun der Niagara 2 sein, der aber noch auf sich warten lässt.

Das war der komplette Text. Keine Hinweise auf Benchmarks (Wenn man keine Lust hat zu suchen, kann man sich als Redakteur uebrigens gut auf diesen Artikel bei BM Seer berufen. Dort ist das schon schön vorgekaut)

Danach vergleiche man, zu welchen Wortmengen sich ein Heise-Redakteur bringen laesst, wenn es um Schnellere Power5+-Prozessoren und eine PowerPC-Workstation geht. Wohlgemerkt, es geht da auch nur um einen Speedbump von 300 MHz. Verehrte Redakteure bei Heise .... das ist ekelhaft ...

It´s a rule I tend to preach everytime I have to fight against misinterpreted benchmarks: Capacity of the system tells you nothing. It´s important how long every user has to wait. A system capable to handle 1000 user with reaction times around one second and a system capable to handle 1000 users with reaction times around 3 seconds are not the same. But, most benchmarks define a maximum reaction time, for example: The maximum capacity is reached, when the answer of the system takes 4 seconds. When you want to make some benchmarking statements, you load the system until you reach the maximum reaction time. Fits the rule of the benchmark? Yes, definitly. Is it a realistic approach for a successful project: No!

Paul Murphy gives an interesting example in What users want:

Except that when you think about user perception, the Sun's 0.7 second response time would make it seem fast to users while IBM's 3.056 seconds would make them wait long enough to become conscious that they were waiting - and that perceptional difference doesn't count for the Sun so much as it counts against the IBM, because its apparent slowness will frustrate users, thereby dooming your project.

So: Even when the IBM is in striking distance to the Sun system benchmarkwise, the systems play in different classes: Fast or dog slow. 2.3 seconds per transaction, 8 hours a day, 5 days a week, roundabout 48 weeks a year are the difference between satisfied users and a lynch mob in front of the IT department.

In the case you don´t believe me: I´ve worked for a unified messaging provider. For every additional second, our systems needed to react our help desk got more work to answer the complaints.

I knew it ... i knew it right from the start. Either you have performance or you have power efficiency. I predict something additional for 2007: Massive Benchmarketing from IBM. We will see fast Power6 processors and we will see powerefficent Power6 processors. But not both. Source of my prediction? This blog entry of Steven Shankland:

For servers, IBM has said its Power6 processor, due to ship in servers in 2007, will run between 4GHz and 5GHz. But in the ISSCC program, Big Blue said the chip's clock will tick at a rate "over 5GHz in high-performance applications." In addition, the chip "consumes under 100 watts in power-sensitive applications," a power range comparable to mainstream 95-watt AMD Opteron chips and 80-watt Intel Xeon chips.

I assume, they didn´t solved the heat problem of processores. They simply ignore it. You can call it supported overclocking. Well, a have an inkling, that we will see the benchmarks with the highest clocking, hand selected Power6 with absurd prices or absurd leadtime to prevent customers from buying them. The normal customers will only see the normal frequency parts. Hey, at the end it´s the same game like in x86. Or do you really know someone who uses the latest CPUs money can buy. These processors are made for press releases.

I knew it. I knew it right from the start. The first samples of AMD 4x4 are available and the benchmarking starts. And it seems to be inevitable: Benchmarking an UMA-architecture against a NUMA-architecture with a non-NUMA-aware operating system. Well, when will this fanboiz-sites get a little more sophisiticated in their benchmark-spindoctoring ... at least the quality sites points to the fact that 4x4 needs Vista or an real operating system to take off ...

PS: For a real 4x4 versus Kentsfield benchmark i suggest: Vista , one core for operating system, occassional mail and browsing while photoshops works on something real (filter set on a 36M Hasselblad raw) , two cores for photoshop itself , one cores for rendering a DVD.

Okay, very impressive number: 1.000.000 operation per second. But look at the bill of material: 24 Filerheads, not the small one, the NetApp 6040 is one of the bigbadass machines, 768 Gigabytes of Cache, 48 GB NVRAM, 1152 Harddisks. Does anybody uses really a environment like this? It´s seems to me like using an SF25K for fileserving.
This is the nature of benchmarketing. It´s not a matter of being realistic of realistic workloads, it´s a expression of the fine art of understanding and outsmart a benchmark to make a marketing statement.

Hamburg, 6:21. Der ICE nach Hannover stand so einladend da. Grund genug, um einfach mal von den bekannten Schemata abzuweichen, um dort einzusteigen. Noch 27 Stunden und ich kann den Punkt "ganze CeBIT mitmachen" auf die Liste "been there, done that" setzen. Was für eine Bedeutung das auch immer hat. Wenn es eine Bedeutung hat.