c0t0d0s0.org

Paul Murphy about his findings about the software lists of Sun and IBM:

Or, if you prefer to see the glass as half empty: the chances that the lack of a critical application forces you into a risky and expensive development or porting project are at least ten times greater with AIX/Power than with Solaris/SPARC

The basic point behind Murphys article it the problem, that the numbers can´t be compares. IBM counts different than Sun, Sun different than HP, HP different than Redhat, and so on. I would like to see a definitive set of rules to measure ISV support, somewhat like a SPECisvsupport.

I would suggest the following rules: To count as an application for such a number, the application has to be available as a binary, it has to be fully supported on the os version by an ISV or by the vendor of the OS. In case of the emulation of an older os version, the application counts only when the software is supported in the emulation. The application version must be available for new deployments (after EOL, but before EOSL doesn´t count). As a second number, i would suggest a "known to work" benchmark, which removes the "has to be supported" condidtions from the first ruleset.

Without such an ruleset, the ISV numbers are "indicative, but not definitive" as Paul stated in a comment to his article.

It´s a rule I tend to preach everytime I have to fight against misinterpreted benchmarks: Capacity of the system tells you nothing. It´s important how long every user has to wait. A system capable to handle 1000 user with reaction times around one second and a system capable to handle 1000 users with reaction times around 3 seconds are not the same. But, most benchmarks define a maximum reaction time, for example: The maximum capacity is reached, when the answer of the system takes 4 seconds. When you want to make some benchmarking statements, you load the system until you reach the maximum reaction time. Fits the rule of the benchmark? Yes, definitly. Is it a realistic approach for a successful project: No!

Paul Murphy gives an interesting example in What users want:

Except that when you think about user perception, the Sun's 0.7 second response time would make it seem fast to users while IBM's 3.056 seconds would make them wait long enough to become conscious that they were waiting - and that perceptional difference doesn't count for the Sun so much as it counts against the IBM, because its apparent slowness will frustrate users, thereby dooming your project.

So: Even when the IBM is in striking distance to the Sun system benchmarkwise, the systems play in different classes: Fast or dog slow. 2.3 seconds per transaction, 8 hours a day, 5 days a week, roundabout 48 weeks a year are the difference between satisfied users and a lynch mob in front of the IT department.

In the case you don´t believe me: I´ve worked for a unified messaging provider. For every additional second, our systems needed to react our help desk got more work to answer the complaints.

When you want to know more about the world record benchmark for BEA have a look at this blog entry.

In an earlier posting, I talked about this result Six Sun Fire T2000 application server running BEA Weblogic 9.0 and a Sun Fire E6900 with 16 UltraSPARC IV+ running Oracle 10g Database World Record result of 3,328.80 JOPS@Standard. for the SPECjAppServer2004 benchmark. That's 13% better performance than EIGHT IBM p5 550 (4-way 4 core POWER5+ 1.9 GHz) and a IBM p570 (16-way 16 core POWER5 1.9 GHz). I won't go into the HP result, when will they post somthing above 3000? HP's current best is in the footnote below.

This matches an experience i´ve made in some projects. The POWER-Processor is a beast when you want to compete at syntetic benchmark level. In reality we were able to beat Power-5 even with UltraSPARC-IV in real workload/real environment benchmarking. There is more in an enterprise environment than just an processor.