Entries by Joerg Moellenkamp

There is a new Oracle Solaris Podcast available:

By combining the great products and technologies from Oracle and Sun, we are able to deliver a complete solution to our customers - from applications to disk. In this episode, the hosts dive into some of the things we've been doing over the last six months to integrate testing, support, and applications. They also cover early plans for deploying Sun systems in Oracle's Global IT infrastructure and how this will benefit our customers.

$2.07 billion for 3PAR - a company just working in the storage area - ... now i'm sure that $7.4 billion for Sun was a steal.

I sat a on my fingers for all the weekend but now i can't take all this comments in the blogs and the mailing lists any longer. Ten years ago, i would have gone ballistic long ago... but the fuse got longer of the past few years. However i'm still able to detonate, however despite the undirected explosion 10 years ago, it's more directed today.

And this article is such a directed explosion. I worked a while on this article and this was one of the reasons why it was relatively silent on this blog the last few days. At first i didn't wanted to publish it, but some events of today led me to think otherwise. I don't know if this is a wise move, because the torch of this rant will burn down some beards. On the other side, i think the article is worth the publication, but you have to judge about that.

However i try to prevent me from exploding thus i used this weekend to play with my new toy (i purchased an iPad last week and it's really fscking cool, i just thought "Like Enterprise ... i just wouldn't gave them away like they do in TNG" ) and worked a lot on the planing on the modifications on my newly purchased building.

However the mail by Garrett d'Amore asking about the removal of SVM was a drop. It was one of those drops, that led to a severe spill-over. So this blog article will be a long rant. If you don't like rants, just skip this.

And keep in mind, it's a rant ... it's not meant to be fair or objective ... think of it as a way to write down my frustration about a lot of events in the last few weeks and especially in the last few days.

Just in case you wonder about the relative silence in this blog ... i wrote a longer article in the last few days circling about Illumos, OpenSolaris, the Java lawsuit, some people in the Solaris community, an article about the live, the universe and all the rest ... it was a large big rant. I've decided not to put it on the blog ... a lot of work in the evenings for the bucket ... but believe me ... it's better that way

Last week i've reported about the point that the X-Force numbers regarding unpatched disclosures could be sorted in a different way to yield a completely different view on the data. More interesting is a recent development: After reassessing the data, many of the vulnerabilities had to be sorted into different categories. So the numbers were fundamentally incorrect as well.

The list changed a lot due to this changes: Sun went from 9% high+critical to 0%. IBM leads the pack with 29% unpatched high+critical vulnerabilities without patches. However 22% for Oracle doesn't look that good as well. You will find the updated list in the blog entry " Mid-Year 2010 X-Force Trend and Risk Report - Update to Unpatched Vulnerabilities Chart".

Nach der Sommerpause findet nun wieder ein HHOSUG Meeting statt. Am 23. September (nicht zufällig einen Tag nach der Oracle Open World) findet um 19:00 Uhr das 13. HHOSUG Meeting in der Hamburger Oracle Geschäftsstelle am Nagelsweg 55 statt.

Agenda bisher:
1. Wie gehts mit der HHOSUG weiter? Weiterführung der Diskussion vom letzten Mal.
2. Was zwischenzeitlich passierte? (Illumos, die merkwürdige Mail, Solaris 11 Announcement)
3. Was gabs auf der Oracle Open World für Solaris an relevanten Announcements?
4. ...

Weitere Vorschläge und Vorträge sind erbeten

Bitte wegen der Pizzabestellung entweder via Xing anmelden oder eine Mail an hhosug13@joerg.moellenkamp.org schicken.

At the moment you read a lot about this X-Force report and Sun is said to keep more vulnerabilities unpatched. But More interesting than the number of unpatched number of patches is the number of "Percentage of Critical and High 2010 H1 Disclosures with no patch" on page 20 in this report.

1. Google: 33%
2. IBM: 29% (the owner of X-Force)
3. Oracle: 22%
4. Linux: 20%
5. Microsoft: 11%
6. Novell: 10%
7. Sun: 8%

Chris Wong writes in "Why Java needs Oracle":

Java itself succeeded because of Sun's corporate backing. Today, Java still needs a sponsor, and that appears to be Oracle. It was either Oracle or IBM: two Old Ones who are very much invested in Java's success. Both vied to acquire Sun, and the decision was made for us. For better or for worse, Oracle is now Java's champion and protector. The legal landscape is too dangerous out there for a major platform to be without one.

Some code is so old that it predates the "invention" ( ) of free software by the FSF. Such an example is the RPC code provided by Sun to the world under a permissive license in 1984 and it was use in many implementations like the one used in Linux to provide NFS services. This was possible due to the licensing Sun choose at that time:

Sun RPC is a product of Sun Microsystems, Inc. and is provided for
unrestricted use provided that this legend is included on all tape
media and as a part of the software program in whole or part. Users
may copy or modify Sun RPC without charge, but are not authorized
to license or distribute it to anyone else except as part of a product or
program developed by the user.

However it isn't technically free software, as it was freeded before free software was formally defined ... and out of some strange reasons, people found the license now free enough. In some Linux distributions this situation was considered as a serious bug (I would file a bug against something different in regard of this, but that's a different story ) However due to whatever reasons, this particular issue wasn't resolved for years ...

As Tom Callaway wrote in a recent blog entry, this situation has been resolved now:

So, we restarted the effort with Oracle, and on August 18, 2010, Wim Coekaerts, on behalf of Oracle America, gave permission for the remaining files that we knew about under the Sun RPC license (netkit-rusers, krb5, and glibc) to be relicensed under the 3 clause BSD license.

Die Videoaufzeichnung von meinem Vortrag auf der Froscon ist nunmehr online: "Was treibt eigentlich mein Unix?". Man verzeihe die Ähs und den dicken Typen da vorne. Die Tonspur ist leider nur auf deutsch erhältlich.

Update: Der Link ist nunmehr repariert ...

The OGB has pulled the trigger today. The members resigned today as reported by the last meeting minutes ever of the OGB . After the the developments of the last few weeks this was just a formal, but inevitable step. Will it change something? Don't think so ...

Sometimes charts look to perfect to be measured. I had this feeling when i saw the rperf numbers of the 795 and put them into a chart. I'm a very visual person, so i put everything in a chart just to get a feeling for numbers.

At first i thought i was paranoid, but then my colleague Jan Brosowski mailed to me that he had thought the same, albeit he approached the problem from the mathematical point of view. Okay ... that left me with a lot of questions and so i did some quick bullshit-testing math on the datapoints.

Some math

After reading his mail i wanted to do some tests on my own. So i did a short test with the numbers. At first i've put the data of the 3,7 Ghz P7 into my favorite statistical programm R.

> procs <- c(24,48,72,96,120,144,168,192)
> rperf <- c(273.51,547.02,820.53,1094.04,1367.55,1641.06,1914.57,2188.08)
> fm <- lm (rperf ~ procs)
> fitted.values(fm)
      1       2       3       4       5       6       7       8 
 273.51  547.02  820.53 1094.04 1367.55 1641.06 1914.57 2188.08 
> residuals(fm)
            1             2             3             4             5 
-7.886829e-14 -1.045378e-14  6.540129e-14  6.414338e-14  3.446377e-14 
            6             7             8 
-2.363755e-14 -2.489546e-14 -2.615336e-14 
> coefficients(fm)
 (Intercept)        procs 
1.607775e-13 1.139625e+01
> summary(fm)

Call:
lm(formula = rperf ~ procs)

Residuals:
       Min         1Q     Median         3Q        Max 
-7.887e-14 -2.521e-14 -1.705e-14  4.188e-14  6.540e-14 

Coefficients:
             Estimate Std. Error   t value Pr(>|t|)    
(Intercept) 1.608e-13  4.241e-14 3.791e+00  0.00906 ** 
procs       1.140e+01  3.499e-16 3.257e+16  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 5.442e-14 on 6 degrees of freedom
Multiple R-squared:     1,	Adjusted R-squared:     1 
F-statistic: 1.061e+33 on 1 and 6 DF,  p-value: < 2.2e-16

Strange ... the linear model leads to coeffcients able to predict the rperf value per core with minimal residuals. And i learned not to trust data with a R-squared of 1. Okay ... let's check for the 4.0 GHz P7 perf numbers:

> procs <- c(32,64,96,128,160,192,224,256)
> rperf <- c(372.27,744.54,1116.81,1489.08,1861.35,2233.62,2605.89,2978.16)
> fm <- lm (rperf ~ procs)
> residuals(fm)
            1             2             3             4             5 
 1.627514e-13 -1.454981e-13  4.364426e-16 -4.677910e-15 -3.821397e-14 
            6             7             8 
-1.490662e-14 -1.052861e-13  1.453949e-13 
> fitted.values(fm)
      1       2       3       4       5       6       7       8 
 372.27  744.54 1116.81 1489.08 1861.35 2233.62 2605.89 2978.16 
> coefficients(fm)
  (Intercept)         procs 
-3.215549e-13  1.163344e+01 

Again ... minimal residuals. Okay ... last check ... for the 4.25 GHz procs.

> procs <- c(24,32,48,64,80,96,112,128)
> rperf <- c(347.36,463.14,694.71,926.28,1157.85,1389.42,1620.99,1852.56)
> fm <- lm (rperf ~ procs)
> residuals(fm)
            1             2             3             4             5 
 3.163842e-03 -1.638418e-03 -1.242938e-03 -8.474576e-04 -4.519774e-04 
            6             7             8 
-5.649718e-05  3.389831e-04  7.344633e-04 
> fitted.values(fm)
        1         2         3         4         5         6         7 
 347.3568  463.1416  694.7112  926.2808 1157.8505 1389.4201 1620.9897 
        8 
1852.5593
> coefficients(fm)
 (Intercept)        procs 
 0.002429379 14.473100282

Sorry ... that's is looking to perfect to me.

When you put the 64-cores LPARs data into the same system, you will see for 4.0 GHz:

> procs <- c(128,256)
> rperf <- c(1406.36,2812.72)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
  (Intercept)         procs 
-3.215549e-13  1.098719e+01

And now for the 4.25 GHz P7:

> procs <- c(64,128)
> rperf <- c(777.09,1554.18)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
  (Intercept)         procs 
-1.607775e-13  1.214203e+01

Both times the intercept is 0 (i assume the small intercept is owed to rounding in data point by IBM or by the challenges of floating point arithmetic on computers.

That's totally unreasonable for measured data. When you just assume 99% of the performance for the 256 cores datapoint (thus an practically impossible scaling factor) you would have an intercept in the range of 28.13.

> procs <- c(128,256)
> rperf <- c(1406.36,2812.72*0.99)
> fm <- lm (rperf ~ procs)
> coefficients(fm)
(Intercept)       procs 
   28.12720    10.76744

Conclusion

At the moment i don't believe that IBM has really measured all the data it provides in the rperf list. The data fits to perfect in a linear model. The interesting question is: "Which data points were really measured?" All the data provided for the configurations look computed/guessed or something like that and not measured. Even when you want to assume that IBM found a way to the holy grail of linear scaleability, an R-squared of 1 and residuals at 0 are just ridiclious. I would really like to know what data points were really measured.

The new "IBM Power Systems Performance Report POWER7, POWER6 and POWER5 results" holds an interesting piece of information. A reoccuring question be colleagues and befriended admins is the impact of LPARs to the performance. It looks like IBM needs the LPARS to get some speed out of their larger systems.

Just a few examples: When using a 795 with 4.25 GHz and 64 cores a configuration with 4 LPARS a 16 cores yield a relative performance of 926.28. The same system with just 1 LPAR with 64 processors yield a relative performance of 777.09. So leaving the scaling to the OS instead of dividing it into 4 small systems gives you just 83.89% of the performance. When using a 795 with 4.25 GHz and 64 cores a configuration with 8 LPARs with 16 cores each yields a rperf value of 1852,56. With 2 LPAR with 64 cores each you get 1554,18. Interestingly is 83,89% again.

At first i thought "16 cores are easily fitting on a processor book (with 4 procs each). A 64 cores LPAR has to use two processor books. So when you use a configuration larger than a processor book you will leave 16,11% on the way". But doing the same calculation with some other data showed otherwise. But the move from 32 to 64 core lpars just reduce the performance by 5,7 percent respectively 5,4 %. 32 cores fit on a processor book, too. Thus the difference should be similar to the 64 to 16 cores situation.
So my interpretation is a little bit different: The scalability of AIX seems to have sweet spot between 16 and 32 cores. I thought a moment about an intra-book bottleneck, but the CPUs on the book are fully meshed (1 hop from each CPU to every other), so i don't think it's a problem.



When you look into this chart (please click into the image for a larger version), you may find some interesting points. The light blue line is a hypothetical perfectly scaling 4.0 GHz Power7 in a 795. The data is based on rPerf number of 103.41 for an 8 core system(source: Page 20 of the document). Please look at the right side of the chart at 256 cores. You end up at 81,9% of the hypothetical performance when you use 64 core LPARs and at 86% of the hypothetical performance when using 32 core LPARS (the difference is interestingly pretty much the same as computed before for 64 cores instead). At 64 cores your load is distributed at 8 processors, thus just 2 processor board. Still there is an serious impact of almost 20%. Will be interesting to further dig into this topic.

However it's important to know, that the operating system is limited to 64 core SMP no matter how many cores are in the system by the LPARs configuration. So this numbers doesn't factor in scalability challenges of AIX above 64 cores as the os has not to scale above this point while generating this rperf numbers. The numbers for the large core number configurations are not single OS image numbers. Further penalties for the OS scaling comes on top. Furthermore this benchmark is a pure CPU/memory benchmark. As IBM explained in their own description of the benchmark, there is no I/O and no networking involved.

That said, a number of really interesting data points are missing in the pdf:

• The rperf number of a fully blown unpartitioned system
• The rperf number on a fully blown system with just one LPAR in the size of the complete system
• Somehow i have the impression, IBM is hiding something. When you look into the mentioned pdf there is SPEC number for an unpartitioned system, but no rperf for it. The existence of the SPEC numbers hints to the point, that they had indeed an OS that was able to scale to 256 cores. On the other side, there is no SPEC number for the partitioned systems, but the rperf numbers. Just call it a presentiment ...