c0t0d0s0.org

Roch Bourbonnais published an article about the new iSCSI framework in the S7000 series. Earlier versions of the S7000 firmware used the userland iSCSI daemon that was used in Solaris 10. Now the iSCSI functionality of the appliance is based on COMSTAR. This change is quite significant: Roch observed in a test an speedup by factor 2.7 as he states in "iSCSI unleashed". Really worth a read.

Some new systems were announced a few days ago. All are Nehalem based. And all of them have some special features seperating them from the competition. But this isn´t the point of this article.

I find one system the most interesting: Really long ago i´ve speculated about the usage of internal storage for other systems to provide a SAN without conventional storage , at this time we´ve just talked about iSCSI and about system with 2.5" disks. Now look at the X4275. The system with 12 3,5" disks on 2 rack units.

Yet another interesting putback to Opensolaris: PSARC 2008/395 iSER: iSCSI Extensions for RDMA. With this extension it´s for example possible to do iSCSI over Infiniband circumventing the complete TCP/IP stack and just copy the data directly from memory to memory via Remote DMA without involving OS infracstructure for this task. The ARC case is in redacted-mode at the moment, so the arclog for this case isn´t really useful at the moment.

Frequent readers of my blog may remember, that i´m really a little bit hesitant to believe that there is really a technical reason for server virtualisation - at least in the Unix space. As Alec Muffet once coined: Virtualisation is a security feature, not a consolidation feature. But there is one area where the need for virtualisation is more obvious: Desktop Virtualisation. This area provides the right combination: An operating system (Win XP predominantly) not really able to consolidate workloads and the ratio between needed compute power (Office, Web) and provided compute power per Desktop (Quadcore 3,0 Ghz) is just absurd. Thus virtualisation in conjunction with a really thin client (and not these stripped down Linux Desktops) is one of the few really valid business cases for virtualisation in my opinion.

Well, Sun has an interesting solution for desktop virtualisation with SunRays and the Virtual Desktop Infrastructure. This VDI is short before it´s third major release and it contains many interesting feature like the emancipation of the solution from VMware by the support of Virtualbox on a server or the direct support for iSCSI for the deployment of desktops. It´s a really interesting solution with many facets. To describe them i just want to link to the blog of an expert. In the blog of Dirk Grobler you will find many informations about VDI3.

I´ve reported back in August that the case for iSCSI boot was approved by the ARC. The PSARC 2008/427 (iSCSI Boot) and PSARC 2008/640 (iSCSI With DHCP) cases had their flag day yesterday for the build 104 of Solaris.

I thought a little bit about the idea of transforming server into solid state disks. The idea in the mail of Chris Greer on zfs-discuss was to use mirrored iSCSI shared ramdisks as a storage for the seperated ZILs. But i think you could use the concept as well for L2ARC as well - e.g. for large databases. One of the sizing rules of databases: More main memory never hurts. Nothing helps the performance of a database more than even more memory. The rule of "main memory never hurts" is based on the fact, that a hard disk has only a few IOPS compared with the main memory and hard drive access massively hurts the performance of your database.

But obviously the size of memory is limited, albeit the this limit is quite high with systems with memory sizes in the range of 512 GB on 4 rack units. But how can you get more memory into your database system, when all DIMM slots are filled with the biggest available DIMMS.

I had an idea while cooking tea this evening while i thought about a discussion with a colleague: Let´s assume an architecture based on a X4600 as a head in front of four X4600 each fully maxed to 512GB. All the nodes are connected with Infiniband. The first X4600 is your normal database server (for example mysql or LarryBase). You put your data into an ZFS storage pool. This storage pool is augumented with L2ARC devices. But now comes the plot twist. Let´s use the 512GB X4600 as huge ramdisks (yes, i know, every engineers heart will crying now) speaking via iSER (no TCP/IP, just RDMA) at 20 GBit/s to the central database node. This would give you a cache in the size of almost 2 TB plus the cache on the database server itself.. By using L2ARC you could use the memory as database caches of other systems without using a database doing a combination of the memory resources by other means, for example the CacheFusion stuff of Oracle. You don´t have to fuse the caches of other databases servers. The other servers are caches. You don´t have to partition the databases.

It would be interesting how such an system would perform in comparision to a Oracle RAC or other memory implementations. Anybody out there willing to test this ... my Infiniband switches are in the laundry at the moment

I talked about the advantages of the separated ZIL and the L2ARC even on rotating rust in my article "iSCSI for I/O intensive tasks" in March 2008. I didn´t found the time to write down the findings of my tests as i started LKSF not much later. But Jignesh Shah did some tests with postgres and wrote down his results. He even took the concept a step farther:

Unfortunately the storage that's available is in Colorado while my PostgreSQL server is located in Massachusetts.

He tested an database with local sZIL and L2ARC and a remote iSCSI pool and got excellent results:

Not bad. Cutting the latency of writes of something that would have taken in excess of 8-10 minutes is atleast recorded within 4 seconds on nonvolatile cache/log combination and allowing ZFS to sync it up to the actual storage.

The whole concept of an ZFS augmented iSCSI shows promise even for latency-dependend applications.

The ARC has approved the PSARC/2008/427 iSCSI Boot case.

This project is to enable Solaris to boot off iSCSI luns via regular network adapters. Different approaches, iBFT/OBP, are adopted to implement this feature on x86/sparc platforms.

For the readers unaware of iBFT: This is a part of the ACPI 3.0 specification to enable a kernel to gather additional data needed to boot via iSCSI.

There is an interesting walkthrough for testing Solaris Cluster with iSCSI and Virtualbox at Building a Solaris Cluster Express cluster in a VirtualBox on OpenSolaris. Whoever wants to play with a mature cluster framework should give this a try.

The development team for iSER (iSCSI over RDMA over Infiniband) announced, that their prototype is up and running : [iser-dev] iSER prototype implementation is working!!!

The next release of Solaris 10 is imminent. As the Sun website states: Solaris 10 5/08 will be available for download on 4/16/08. So it may be interesting to you, what new features found their way in 5/08. Well, you can already look into the What's New in the Solaris 10 5/08 Release document.

• System Resource Enhancements
• CPU Caps
• projmod(1M) Option


• Device Management Enhancements
• Tape Self-Identification
• x86: Enhanced Speedstep CPU Power Management
• iSNS Support in the Solaris iSCSI Target


• Security Enhancements
• Solaris Trusted Extensions Supports Mounting Labeled File Systems With the NFSv3 Protocol
• SPARC: Hardware -Accelerated Elliptical Curve Cryptography (ECC) Support


• Networking Enhancements
• Sockets Direct Protocol
• Tunable inetd Backlog Queue Size


• X11 Windowing Enhancements
• Xvnc Server and Vncviewer Client


• System Administration Enhancements
• Solaris Trusted Extensions Administrator Procedures
• Flash Update Tool
• PPD File Management Utility
• Internet Printing Protocol Client-Side Support
• Selectable Use of localhost for Solaris Print Server Database Hostname
• Fault Management for T5140/T5240 Platforms
• SunVTS 7.0


• Desktop Tools Enhancements
• new version StarOffice 8
• Flash Player 9
• Pidgin 2.0
• PAPI Print Commands


• System Performance Enhancements
• 64-bit SPARC: Memory Placement Optimization Support for sun4v Platforms
• SPARC: Shared Contexts Support
• x86: CPUID-Based Cache Hierarchy Awareness


• Language Support Enhancements
• Locale Creator
• libchewing 0.3.0
• File Encoding Examiner


• Kernel Functions Enhancements
• x86: MONITOR and MWAIT CPU Idle Loop


• Driver Enhancements
• x86: Support Sun Fire X4540 Disk Status Indicators
• MPxIO Extension for Serial Attached SCSI Devices on mpt(7D)
• x86: SATA ATAPI Support in AHCI Driver
• x86: GLDv3 Version bnx II Driver
• x86: AMD–8111
• SATA NCQ Support in AHCI Driver
• x86: bnx II Ethernet Driver
• USB-to-Serial Driver for Keyspan Adapters


• Freeware Enhancements
• 32-bit: pgAdmin III
• p7zip

Pheewww ... i´ve finalized my SamFS tutorial, a long text. SamFS is the hierarchical Storage Management Filesystem from sun. Several new concepts, so you have to explain some theory before getting to the shell:

• Part 1: Introduction.html
• Part 2: The theory of Hierarchical Storage Management
• Part 3: The Jargon of SamFS
• Part 4: Installation of the packages
• Part 5: Configuring a SamFS filesystem
• Part 6: Configuring disk archiving
• Part 7: Working with SamFS
• Part 8: Usecases and future directions
• Part 9: Conclusion

As usual: Have fun!