The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.
Sunday, November 22. 2009
A while ago, i wrote a tutorial about using iSCSI in Opensolaris and Solaris 10. While the tutorial is still valid for Solaris 10, Opensolaris got a new, more efficient iSCSI Target. iSCSI in Opensolaris is a part of a more generic in-kernel framework to provide SCSI Target services. This framework isn't just capable to deliver iSCSI, you can have FCoE, SRP, FC targets as well. This generic approach led to a different administrative model. Thus when you want to configure iSCSI on a OpenSolaris system and you want the new framework (the old userland based version is still available) you have to configure the target side differently. This tutorial will do pretty much the same than the old iSCSI tutorial, just with the new framework to give you an overview about the changes.
Why does COMSTAR need a different administrative model?The reason for the this change is somewhat based in the point for it's name: It's a COmmon Multiprotocol Scsi TARget framework. iSCSI, SRP, iSER, SAS, FCoE, FC have something in common. They use SCSI commands. SCSI isn't just a physical interface. It's a way to transmit data over an interface. It's a set of commands. And independently from the physical interface technology you use, the commands are pretty much the same. A good sign for this commonality is the same jargon in all this technologies: The entity executing SCSI command is called target in Fibrechannel jargon as well in iSCSI as well in SCSI over RDMA over Infiniband. The same is valid for the entity issuing commands to a target. It's called initiator in all those protocols as well.
The differences between all the proctols mentioned above is the way, how the SCSI commands are transmitted over the wire. This led to an obvious idea: Separating the layer providing the SCSI target from the layer providing the protcol to access the SCSI target. The component providing this protocols are called "Port Provider" in COMSTAR.
On the other side, COMSTAR isn't just meant to provide access to SCSI disks. Providing tape devices is in the focus as well. The component providing disk or tape devices is called "logical unit provider". The logical unit is the device within a target responsible for executing SCSI I/O commands. Perhaps you've heard about the acronym LUN in the past. LUN is the Logical Unit Number, a number designating a certain logical unit.
Between the both components is the
What's the advantage: You just have to develop the SCSI target once and for new protocols you just have to develop a relatively simple port provider. This is one of the reasons why we see new port providers quite frequently. In essence this is the reason why the administrative model has changed with the introduction of COMSTAR: The logical unit, the port and the logic to combine both are separate entities thus you configure them separately.
PrerequisitesOkay, let's look how you configure this stuff. I will use two systems in this example:
Preparing the target systemThe COMSTAR framework isn't installed per default on OpenSolaris, thus you have to install it on your own. But there is an easy way to install all the components to make a storage server out of your installation. Doing so installs much more than you need for iSCSI, but on the other side it installs all other components, when you want to make a CIFS/iSCSI/NFS/MDNP/etc-capable storage server on your system.
Please reboot the system after this step. This framework has many connections to the rest of the system and it's just easier to reboot after the initial install.
Okay, right after the boot the service
We need this service, so we enable it.
Now we can check the successful startup of the
Obviously we need a backing store for our iSCSI target. In this case i will use a spare provisioned 10 Terabyte emulated ZFS volume.
There are other options besides an emulated ZFS volume like a pregenerated file (
Now we have to configure a logical unit provider in the COMSTAR framework to use our backing store. We want a disk, so we use the disk logical unit provider. We have to use the
Let's do a short check, if everything went well.
But at the moment your brand new LU isn't visible. You have to add it to an entity called view. In a view you can define which targets a certain initiator can see. For example you can configure with this views, that all your SAP systems just see the SAP logical units and all MS Exchange Servers just see the Exchange logical units on your storage service. Other logical units aren't visible to the system and they are totally unaware of their existence. For people with a storage background: This is somewhat similar to the LUN masking part
But for this tutorial i will use a simple configuration. I will not impose any access control to this logical unit :
Let's check the configuration:
However, the logical unit is still not visible to the outside, as we haven't configured a port provider. The port provider is the component that provides access to the logical unit via iSCSI or FC for example.
Configuring an iSCSI targetSo let's start with the configuration of the iSCSI target. This is the first step where the commands have something to do with iSCSI:
Okay, now we configure an iSCSI target entity on this portal group:
This IQN uniquely identifies the target in a network. We will use it from now on when we address this iSCSI target.
That's all. For unauthenticated iSCSI this is all you have to do on the target side: We've configured the logical unit provider and enabled a port provider to give other system access to the target with iSCSI.
Configuring the initator without authenticationOkay ... now we log into the initiator. Just a quick check ...
Just our boot disk. At first we have to install the iSCSI initator. This can be done by
Okay, the iSCSI service runs on the host
At first we told the iSCSI initiator to discover logical units at the target portal on
Now let's look, what targets were discovered by the iSCSI initiator:
The iSCSI target IQN you saw at the time when you configured the target on the host
When we start format again, we will see that our host
A brand new 10 TB iSCSI volume. Of course we can use it for zfs again:
Works as designed. A nice 10 TB ZFS pool on
Configuring the initator with authenticationOkay, in the non-COMSTAR iSCSI tutorial i explained how to use authentication. With authentication the iSCSI target and the iSCSI initiator can check if the host on the other side of the connection is really the expected node. This works with the CHAP mechanism. Perhaps you know it from PPP for your internet access. It works on the foundation of shared secrets. Only when both partners of a communication know the same secret, the communication partner is authentic. It similar to a secret handshake between two peoples.
Okay, let's configure a bidirectional secret handshake. Just as a good admin we export the pool on it, as the following configuration will terminate the iSCSI connection for a moment.
Okay, to configure the authentication we need to pieces of information at first. Log into the system
Now log into the system
At first we configure the iSCSI target in a way, that it uses chap authentication. We need the IQN of the target here.
Now we have to configure the secrets. The CHAP secrets have be at last 12 characters. At first we set configure the secret, that the initiator will use to authenticate at the target.
Now we configure the secret that the target uses to authenticate itself at the initiator.
Okay, we have to something similar on the system
At first we have to configure the CHAP secret that the initiator uses to authenticate itself
Now we tell the system that this initiator should use CHAP authentication to authenticate a target.
The next steps configure the authentication relation between our initiator and a defined target. At we activate an bi-directional authentication. So the initiator has to authenticate at the target as well as the target has to authenticate itself at the initiator.
Now we tell the iSCSI initator that the iSCSI target uses CHAP to authenticate the ISCSI initiator.
In a last step we configure the shared secret, that the target uses to authenticate the initiator.
Perhaps this figure explain the relation between the secrets better than just command lines can do:
Now we are done. We can do a quick check if our configuration found it's way to the system. At first a quick look at the iSCSI Target on
Now we do the same check on
Okay, everything is okay ... now let's just reimport the testpool
We are back in business.
ConclusionThe configuration of an iSCSI Target isn't more difficult than before. It's just different. I hope i gave you some good insight into this topic.
Do you want to learn more?
docs.sun.com: sbdadm(1M) - SCSI Block Disk command line interface
docs.sun.com: itadm(1M) - administer iSCSI targets
docs.sun.com: iscsiadm(1M) - enable management of iSCSI initiators
c0t0d0s0.org : Less known Solaris Features: iSCSI - tutorial for the userland iSCSI implementation in Solaris.
Display comments as (Linear | Threaded)
Please take note : migration from existing userspace iscsi with zfs backing store will erase your existing zvols
when you swtich to comstar
@Joerg - keep up the good work!
Regarding COMSTAR with iSCSI: we have found that performance over ordinary 1 GBit lines changes heavily between OpenSolaris releases. We have had better success using svn_124 than with releases in the range of svn_118-121 (or 122?). Using vanilla x4100/x4200 stuff, no strange hardware involved, btw.
As a suggestion, some performance tips would nicely complement this guide. Information on this is relatively scattered so far, and we had quite some fun until we got COMSTAR to saturate a 1Gbit port connected to an OpenISCSI Linux client. Jumbo frames alone were around 15-20% improvement.
One additional remark regarding the FC targets: in practice, this capability is limited by the available FC host adapter support. The COMSTAR FAQ as of 2009/10/26 still lists only Qlogic 4GB/8GB HBAs as working. No Qlogic 2GB, and the entire Emulex support is still listed as "under way". Pretty unfortunate if you want to evaluate and all you have is Emulex (or Qlogic 2GB).
#2 Arnd Bei▀ner on 2009-11-23 12:46
One question I encountered while trying this out:
In the target creation stage (itadm create-target): how or where is it determined which LU (or view) is bound to the new target? Assume I had created two LUs (and two views) before, which one would it use?
The author does not allow comments to this entry
The LKSF book
The book with the consolidated Less known Solaris Tutorials is available for download here
Martin about End of c0t0d0s0.org
Mon, 01.05.2017 11:21
Thank you for many interesting blog posts. Good luck with al l new endeavours!
Hosam about End of c0t0d0s0.org
Mon, 01.05.2017 08:58
Joerg Moellenkamp about tar -x and NFS - or: The devil in the details
Fri, 28.04.2017 13:47
At least with ZFS this isn't c orrect. A rmdir for example do esn't trigger a zil_commit, as long as you don't speci [...]
Thu, 27.04.2017 22:31
You say: "The following dat a modifying procedures are syn chronous: WRITE (with stable f lag set to FILE_SYNC), C [...]