c0t0d0s0.org

I thought a little bit about the idea of transforming server into solid state disks. The idea in the mail of Chris Greer on zfs-discuss was to use mirrored iSCSI shared ramdisks as a storage for the seperated ZILs. But i think you could use the concept as well for L2ARC as well - e.g. for large databases. One of the sizing rules of databases: More main memory never hurts. Nothing helps the performance of a database more than even more memory. The rule of "main memory never hurts" is based on the fact, that a hard disk has only a few IOPS compared with the main memory and hard drive access massively hurts the performance of your database.

But obviously the size of memory is limited, albeit the this limit is quite high with systems with memory sizes in the range of 512 GB on 4 rack units. But how can you get more memory into your database system, when all DIMM slots are filled with the biggest available DIMMS.

I had an idea while cooking tea this evening while i thought about a discussion with a colleague: Let´s assume an architecture based on a X4600 as a head in front of four X4600 each fully maxed to 512GB. All the nodes are connected with Infiniband. The first X4600 is your normal database server (for example mysql or LarryBase). You put your data into an ZFS storage pool. This storage pool is augumented with L2ARC devices. But now comes the plot twist. Let´s use the 512GB X4600 as huge ramdisks (yes, i know, every engineers heart will crying now) speaking via iSER (no TCP/IP, just RDMA) at 20 GBit/s to the central database node. This would give you a cache in the size of almost 2 TB plus the cache on the database server itself.. By using L2ARC you could use the memory as database caches of other systems without using a database doing a combination of the memory resources by other means, for example the CacheFusion stuff of Oracle. You don´t have to fuse the caches of other databases servers. The other servers are caches. You don´t have to partition the databases.

It would be interesting how such an system would perform in comparision to a Oracle RAC or other memory implementations. Anybody out there willing to test this ... my Infiniband switches are in the laundry at the moment

An important update for the Infiniband Stack in Solaris was released yesterday. Josh Simons reports in his blog:

Yesterday, the Sun InfiniBand engineering team released Solaris 10 driver support for ConnectX (a.k.a. Hermon), the latest generation of InfiniBand silicon from Mellanox.[...]In addition to the driver, the update also includes a new flash updating tool for ConnectX, a uDAPL update, and several additional components, all of which is described in the documentation.

You can get Solaris Infiniband Update 2 at the usual location ...

Mellanox made the silicon available for 40 GBit/s Infinimand: 40gig InfiniBand arrives. This is an interesting development as this gives Infiniband a nice performance advantage over Ethernet. In the race for the dominant interconnect for single fabric datacenter this really important.

Interesting editorial at HPCwire about the competition between Infiniband and 10 GbE. I´m not sure about the final results of this compeition. 10 GBe will win the WAN and Campus Network part for sure, but i see a vast market for Infiniband in the datacenter, especially for in-data-center fabrics. 10GBASE-T is signal processing at the border of the economical unreasonable effort. You need digital signal processors to do the job. I don´t believe we will see prices of 10GB-T NICs in the range of Infiniband anytime soon, and NIC costs are quite a factor in large fabrics. BTW: Out of this reason, 10GBASE-CX4 looks like a reasonable solution to me, as it uses similar cabling like Infiniband and they made the same design decision as IB, they´ve sacrified range for speed to keep the transceivers simpler. At the end: The race of datacenter fabrics isn´t decided at all ...

EETimes wrote an interesting article about some statements of our top management regarding the future architecture of datecenter. In "Sun preps Infiniband for broad net role" EETimes writes:

"In most data centers there are three installations often run by three different teams and using systems from three different vendors," said John Fowler, executive vice president of the systems division at Sun. "We think there are technology and economic advantages for bringing these three together," he said at a press event Wednesday (Feb. 13).
Fowler said Sun will roll out in 2008 products to run all data center traffic on Infiniband.

Nice to see Andy and John sharing my opinion;) Joke beside. I talked about something similar with colleagues some month ago while driving to a Sales Kick Off somewhere in the middle of Nowhere and thought and talked about such an solution while designing the network infrastructure for a really big RfP. It´s really obvious, that Infiniband will play a bigger role in future. Andy and John have forgotten more about computer systems than i know about computer systems, so it seems my opinion isn´t completly weird.

3456 ports in a single switch: the IB switch component of the the Sun Constallation System.