c0t0d0s0.org

I had recently a discussion about that. Perhaps it was teached in my generation and the generation of IT people before me. However it seems this has been omitted in the newest generation of IT people, because i just had to do this discussions with people just a few years out of education. Perhaps it's just a thing of experience, however it's so basic that it should be part of experience, but part of learning. As i tend to speak with clever people, most of them get it immediately, however i find it quite astonishingly that this simple thing gets out of focus.

Many people believe that computers are exact devices. That data stored on disk will get back to you unharmed. That data you send over the network can be assumed as correct. That computers are deterministic devices. But they aren't. Not at all. I'm talking about this quite often when talking about ZFS.

Alec Muffet pointed on Facebook to a really interesting presentation that is showing errors due to corruption of data in a different context. It's about a DNS attack without attacking something. It's just waiting for errors to happen. It's something like waiting for typos in DNS names made by users. Just this time the error is introduces by computers. By reserving domains that are just one bit different from the original domain. Of course the probability is really small that such an error will happen, but given the enormous amount of requests the DNS system handles every second of the day, the number of events isn't that small as you would assume at first: "Bitsquatting: DNS Hijacking without exploitation". Really worth a read.

Brendan Gregg has written an interesting piece about finding performance problems: "The USE method addresses shortcomings in other commonly used methodologies".

It's a good paper, however ... well let's say, I don't understand why so many people find it especially cool or especially good, because at the end it isn't something really new. Don't understand me wrong: It's good. But not extraordinarily good. Like many methodologies it's basically just codified common sense with a personal spin. So I would prefer to say "My-personal-way-of-doing-stuff" instead of calling it methodology. There is nothing new in it. Just a lot of common sense.

I really think that performance analysis is not so much about a "methodology" you can simply follow that will lead you magically to a result. It's about a mindset how to tackle problems, it's about being structured in the approach, it's about "being prepared", it's a lot about knowing stuff.

As I do performance analysis quite frequently, I have created my own "methodology", or to be more correct ... my own mindset of doing such stuff. I don't call it method or methodologies. Perhaps it's useful for some or the other ... so i write it down here.

Just to get back to my article a few days ago: I really think that we won't tapes disappearing whatever some media pundits write all the time, especially not the areas where tape drives like the T10000 are used for. In fact tapes will get more important, because of a wall that is imminent. Yeah ... you may think "Sure, those limits were always broken in the past". But here we speak of a limit that is imposed by the older brother of computer science, physics. However this article isn't about tape and why i think we need them more than ever. I just want to show with this blog entry that the future in magnetic storage isn't that easy as you might think due to several years of steady increases in disks.

Just a few words about this blog entry: Since i'm an avid supporter of ZFS, i'm reading a lot about data storage and try to know as much as possible. It's a really interesting topic. When you think that a hard drive is a relatively simple device, just forget this assumption.

For this article I simplified stuff for this articles, at first to keep this article relatively short. Additional I used the same simplification that helped me to understand this topic. However: Perhaps i forgot something or i'm just wrong with what i've learned. It's just my summarization that was inspired by a internal presentation i recently read. On the other side I will reread the text in the next days and may make some change when i'm detecting that i wrote something incorrectly or unclear.

Whenever i hear "Oh ... tape is dead ... everything will be saved in the cloud", i wan't to answer "Yeah, sure, you don't need power plants as well because at at your home the electrical power comes from the wall sockets". Don't know what those people thinking how those providers are doing backup,restore and recovery.

For this discussions there is a nice data point: Google published a nice video and 65 seconds in video the device looks like a large tape library to me and they are saying: "We store another copy of the most important data on digital tapes". There they are again: Tapes.

On a related note, the Register writes about a rumor "Insider 'fesses up: Amazon's Glacier cloud is made of ... TAPE". By the way, when Glacier was announced, a lot of news outlets said "Nail into the coffin of tape". Well ... just to start : Would only believe it when you show me an internet connection north of several hundreds of megabytes per second sustained. There are several reasons why such a service is no competition for tapes as a backup,restore, recovery mechanism as well as for archiving for people (However i see really a market for such a service: Rescuing your life when you have to fulfill on the hardest SLA: When you have recover photos important to your significant other, photos you have just deleted. Waiting a little bit is nothing then, just getting the photos back is important )

PS: This "additional copy on tape" thing sounds familiar to me. Back at my former employee Canbox in 2000 i've planed implemented together with Sun a hierarchical storage management for the mail store. I used a slightly modified IMAP daemon to prevent a staging storm (metadata was backuped differently). One copy on disk (of course mirrored ... on Photon arrays), two copies on tapes in a STK 9310 aka Powderhorn I coined the phrase at that time "You can do anything with a user, but you are not allowed to loose a single mail of her or him" and to live up to this the mail system was planed that way. Oh god, i'm already that old that i'm telling old stories from the trenches

This must be something like their inner October Revolution Parade or 4th of July fireworks for our beloved competitor. Their FUD seems to be effective. Reuters writes in "Sun Micro losing $100 million a month, Ellison says":

Oracle Corp (ORCL.O) Chief Executive Larry Ellison said Sun Microsystems Inc (JAVA.O) is losing about $100 million a month as European regulators delay approving his company's $7 billion purchase of the struggling hardware maker.

I'm still asking myself, who told the EC this Mysql bullshit. Sorry colleagues at Mysql, but Mysql will be never in the same market like the RDBMS market. And it shouldn't be in that market. For some people it's already too far in this market, today. That's one of the reasons for Drizzle. I think the markets are pretty much complementary.

Additionally: The code is open source. Whenever Oracle would decide to stop developing Mysql, there would be other companies starting development instead of Oracle. When Oracle Support of Mysql is too expensive, there will be other companies offering cheaper support. That's is the only advantage of open-source in my opinion for 95% of all users, because most people never see the source even at Makefile distance, as they just install a package.

An interesting article about Sun: Sun Microsystems Autopsy: Death by Reverse Darwinism. His statement at the end is a really good one:

Think of who the winners are in this acquisition: certainly Oracle and Sun, and the soon to be combined customer base. Now the losers: IBM and HP. Now ask yourself who is really behind the EU delay?

So ... 16.000 pageviews later, i've read through several reactions to my article "Some perspective to this DIY storage server mentioned at Storagemojo". I wrote that article after reading Robin Harris' article about a DIY high density storage server. Thanks to Google Translate even those in russian language (i've realized that my french is still good enough to read through texts, albeit i have to admit that reading a text in french written by me would be an even more horrible experience than a text in english). Just a warning at the beginning. This text is 32 hard disk marketing KBytes long. 31 of the real ones

Introduction

Okay, the following article isn't really about this Backblaze device, it's more about some misunderstandings regarding storage. For every suggestion i will make in this article, i see a reason why Backblaze didn't go this way. But everyone who just looks at the price and think of Sun Hardware as gold-plated, should rethink. Because for everything Sun (or other storage vendors) did at their devices, there are several good reasons as well.

The only think i've really hated about this article at Backblaze was this figure about the costs. Many people just reposted this figure but not the statement that clearly said, that this isn't a general purpose device. Without their software this storage pod would be just an horrible example to explain how you shouldn't design storage. With their software it's may be an adequate solution for their special needs and workload, and just for their special needs and their workload. No wonder, as the just developed this software for this use case.

I've received yesterday some mails/tweets with hints to a "Thumper for poor" DIY chassis. Those mails asked me for an opinion towards this piece of hardware and if it's a competition to our X4500/X4540. Those questions arised after Robin Harris wrote his article "Cloud storage for $100 a terabyte", which referred to the company Backblaze, which constructed a storage server on its own and described it on their blog in the article "Petabytes on a budget: How to build cheap cloud storage". Sorry, that this article took so long and there may be a higher rate of typos, as my sinusitis came back with a vengeance ... right in the second week of my vacation. But now this rather long article is ready

I just found this piece of FUD in a tweet. IBM tries to move Sun customers to their pSeries. I don't want to talk about the point, that they have to give massive discounts to do so, so the story of “an uncertain future” for Sun in the light of the proposed Oracle merger doesn't seem to generate the intended attraction. But this is a different story. At the end it's just the current fashion in FUD. But it was pretty obvious that they would roll out their FUD thrower out the the garage.

There was an interesting article at Reuters about the upcoming Power7 CPU. I don´t want to talk about the point, that his announcement was well times to our announcement about the 1.6 GHz. I don´t want to talk about the announcement, that systems will be available in 2010 when you take into consideration, that the last rollout of Power6 took almost a year.

No, i want to hint you on an interesting fact in this article. But at first you should take into consideration that IBM tries to tell the world, that per core performance is the most important factor at all. Now look at the following paragraph the Reuters article:

It also said POWER7 will be more efficient than the POWER6, which was launched in May 2007, capable of two to three times performance while using the same amount of energy.

Two to three times the performance ... sounds nice. Well ... it doesn´t sound that nice, when you do some math on the numbers. Let´s assume the Power6 is a 2-core system. Let´s further assume, that Power7 is a 4 core architecture. Then Power7 delivers the same performance per core than Power6 ... just more cores. Nice, but nothing earthshattering. But it gets even more interesting. Back in July 2008 Ashlee Vance wrote in the Register about Power7:

IBM looks set to join the seriously multi-core set with the Power7 chip. Internal documents seen by The Register show Power7 with eight cores per processor and also some very, very large IBM boxes based on the chip.

Okay, let´s do the same math again: Let´s assume the Power6 is 2-core, let´s further assume that Power7 is a 8-core design. Obviously a single core of the Power7 has just half the performance of the Power6 core. Hey, that will be interesting talks for the IBM sales people explaining that they were incorrect with the assumption that speed per core is the only interesting metric.

Scotty was asked by the new administration to help on a strategy to use open source for government purposes. The BBC reports in Calls for open source government:

The secret to a more secure and cost effective government is through open source technologies and products. The claim comes from one of Silicon Valley's most respected business leaders Scott McNealy, a co-founder of Sun Microsystems. He revealed he has been asked to prepare a paper on the subject for the new administration.

While sitting in a workshop meeting, my thoughts went away from the meeting and circled around the Turbo Boost feature of the Nehalem processor. Too make it clear at the beginning: Turbo Boost is a good feature. It helps to use the existent potential of the CPU instead of the expected potential. Overclockers have shown in the past, das most CPU have more performance potential as the print on the heat spreader says. But while I think this is a good idea, this will open some interesting benchmarketing tricks.

Cisco seems to look for new markets - Cisco planning significant data center assault:

Perhaps the most important example of that will be a new Cisco blade server system expected next year. This will take the company into the data center compute space, right up against longtime stalwarts — and up to now, Cisco partners — IBM and HP.

This matches to rumours and hints i´ve got in in the last few weeks. But i don´t believe that Cisco will play a large role in this business. There are already several players in this game ... IBM,HP,Dell,FSC and Sun. And more important: People think of Cisco as the supplier of their networking equipment ... not of their servers ... and Cisco hasn´t a long track record as a supplier of servers. Cisco needs a real differentiator ... i´m curious what this differentiator might be ... but at the moment i´m not convinced about the success.

And there is another problem: Network admin departments, storage admin departments and server admin departments have seperated supplier spheres in many companies. And Cisco is a supplier for the network admin departments and they rarely connection to the server departments. And: Did you know Sun sold an hardware load balancer for a while? No? Cisco will have a similar problem ...

Funny discussion about boot-up times of pSeries systems:

Hey, I manage 7 p595 squadrons and they are all different. For the most part, the startup time depends on the amount of I/O you have installed. Two of mine are 3 frame p595's with 64 hba, 64 nic, about 20 internal HDD, dual power systems and I think 10 d20 I/O drawers. These servers take about 90 minutes just to initialise the hardware, not including starting any LPAR's. So tell your friend that this is normal and take a good book and a really large coffee with him each time he needs a restart