Less known Solaris Features...

Why does COMSTAR need a different administrative model?

The reason for the this change is somewhat based in the point for it's name: It's a COmmon Multiprotocol Scsi TARget framework. iSCSI, SRP, iSER, SAS, FCoE, FC have something in common. They use SCSI commands. SCSI isn't just a physical interface. It's a way to transmit data over an interface. It's a set of commands. And independently from the physical interface technology you use, the commands are pretty much the same. A good sign for this commonality is the same jargon in all this technologies: The entity executing SCSI command is called target in Fibrechannel jargon as well in iSCSI as well in SCSI over RDMA over Infiniband. The same is valid for the entity issuing commands to a target. It's called initiator in all those protocols as well.

The differences between all the proctols mentioned above is the way, how the SCSI commands are transmitted over the wire. This led to an obvious idea: Separating the layer providing the SCSI target from the layer providing the protcol to access the SCSI target. The component providing this protocols are called "Port Provider" in COMSTAR.

On the other side, COMSTAR isn't just meant to provide access to SCSI disks. Providing tape devices is in the focus as well. The component providing disk or tape devices is called "logical unit provider". The logical unit is the device within a target responsible for executing SCSI I/O commands. Perhaps you've heard about the acronym LUN in the past. LUN is the Logical Unit Number, a number designating a certain logical unit.

Between the both components is the stmf,the SCSI target mode framework. It connects the port provider and the logical unit provider.



What's the advantage: You just have to develop the SCSI target once and for new protocols you just have to develop a relatively simple port provider. This is one of the reasons why we see new port providers quite frequently. In essence this is the reason why the administrative model has changed with the introduction of COMSTAR: The logical unit, the port and the logic to combine both are separate entities thus you configure them separately.

Prerequisites

Okay, let's look how you configure this stuff. I will use two systems in this example:

192.168.56.101 initiator
192.168.56.102 target

The system target has to use a recent of OpenSolaris. You can use OpenSolaris as well as Opensolaris Community Edition. My testbed used OpenSolaris Developer Build 127, but it should work with any reasonable recent version. On the system initiator any Solaris system will do it, as COMSTAR is a change on the target side, it doesn't change anything on the initiator part.

Preparing the target system

The COMSTAR framework isn't installed per default on OpenSolaris, thus you have to install it on your own. But there is an easy way to install all the components to make a storage server out of your installation. Doing so installs much more than you need for iSCSI, but on the other side it installs all other components, when you want to make a CIFS/iSCSI/NFS/MDNP/etc-capable storage server on your system.

jmoekamp@target:~# pkg install storage-server
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                24/24   1026/1026    34.3/34.3 

PHASE                                        ACTIONS
Install Phase                              2205/2205

Please reboot the system after this step. This framework has many connections to the rest of the system and it's just easier to reboot after the initial install.

Okay, right after the boot the service stmf is disabled.

jmoekamp@target:~# svcs stmf
STATE          STIME    FMRI
disabled       13:44:42 svc:/system/stmf:default

We need this service, so we enable it.

jmoekamp@target:~# svcadm enable stmf
jmoekamp@target:~# svcs stmf
STATE          STIME    FMRI
online         13:48:07 svc:/system/stmf:default

Now we can check the successful startup of the stmf service:

jmoekamp@target:~# stmfadm list-state
Operational Status: online
Config Status     : initialized
ALUA Status       : disabled
ALUA Node         : 0

Obviously we need a backing store for our iSCSI target. In this case i will use a spare provisioned 10 Terabyte emulated ZFS volume.

jmoekamp@target:~# zfs  create -V 10T -s rpool/iscsivol

There are other options besides an emulated ZFS volume like a pregenerated file (mkfile 10T backingstore), a file that grows from zero to a certain preconfigured file size (touch backingstore) and finally just mapping an physical device through the system.

Now we have to configure a logical unit provider in the COMSTAR framework to use our backing store. We want a disk, so we use the disk logical unit provider. We have to use the sbdadm command for this task.

jmoekamp@target:~# sbdadm create-lu /dev/zvol/rdsk/rpool/iscsivol 
Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0b152cc0000004b080f230004  10995116277760       /dev/zvol/rdsk/rpool/iscsivol

Let's do a short check, if everything went well.

jmoekamp@target:~# sbdadm list-lu

Found 1 LU(s)

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0b152cc0000004b080f230004  10995116277760       /dev/zvol/rdsk/rpool/iscsivol

With the stmfadm command you can get some additional insight to your newly created logical unit.

jmoekamp@target:~# stmfadm list-lu -v
LU Name: 600144F0B152CC0000004B080F230004
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/rpool/iscsivol
    View Entry Count  : 1
    Data File         : /dev/zvol/rdsk/rpool/iscsivol
    Meta File         : not set
    Size              : 10995116277760
    Block Size        : 512
    Management URL    : not set
    Vendor ID         : SUN     
    Product ID        : COMSTAR         
    Serial Num        : not set
    Write Protect     : Disabled
    Writeback Cache   : Enabled
    Access State      : Active

But at the moment your brand new LU isn't visible. You have to add it to an entity called view. In a view you can define which targets a certain initiator can see. For example you can configure with this views, that all your SAP systems just see the SAP logical units and all MS Exchange Servers just see the Exchange logical units on your storage service. Other logical units aren't visible to the system and they are totally unaware of their existence. For people with a storage background: This is somewhat similar to the LUN masking part

But for this tutorial i will use a simple configuration. I will not impose any access control to this logical unit :

jmoekamp@target:~# stmfadm add-view 600144F0B152CC0000004B080F230004

Let's check the configuration:

jmoekamp@target:~# stmfadm list-view -l 600144F0B152CC0000004B080F230004
View Entry: 0
    Host group   : All
    Target group : All
    LUN          : 1

However, the logical unit is still not visible to the outside, as we haven't configured a port provider. The port provider is the component that provides access to the logical unit via iSCSI or FC for example.

Configuring an iSCSI target

So let's start with the configuration of the iSCSI target. This is the first step where the commands have something to do with iSCSI: itadm is the tool to configure an iSCSI target. At first we configure a target portal group. A target portal is an IP and a portnumber you use to access an iSCSI target. Target portal groups are used to bundle such portals to ease the configuration of access controls.

# itadm create-tpg e1000g0 192.168.56.102

Okay, now we configure an iSCSI target entity on this portal group:

# itadm create-target -t e1000g0
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully created.

This IQN uniquely identifies the target in a network. We will use it from now on when we address this iSCSI target.

That's all. For unauthenticated iSCSI this is all you have to do on the target side: We've configured the logical unit provider and enabled a port provider to give other system access to the target with iSCSI.

Configuring the initator without authentication

Okay ... now we log into the initiator. Just a quick check ...

jmoekamp@inititator:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c7d0 
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number): ^C

Just our boot disk. At first we have to install the iSCSI initator. This can be done by pkg install iscsi.After this step you have to reboot the system.

jmoekamp@inititator:~# svcs -a | grep "iscsi"
online         14:08:34 svc:/network/iscsi/initiator:default

Okay, the iSCSI service runs on the host initiator. Now we have to configure the initiator to discover possible logical units on our target:

jmoekamp@inititator:~# iscsiadm add discovery-address 192.168.56.102:3260
jmoekamp@inititator:~# iscsiadm modify discovery --sendtargets enable

At first we told the iSCSI initiator to discover logical units at the target portal on 192.168.56.102:3260. After this step, we configured the iSCSI initiator to use the SendTargets method to discover logical units on the other side. The SendTarget command is a command the initiator sends to the configured hosts to gather all targets available to the initiator issuing the SendTarget command. Every iSCSI target implementation has to support the SendTarget command. This is specified by Appendix D of the RFC 3720 This makes the configuration of the iSCSI easier, as you don't have to configure all the targets manually. Okay, let's check if the configuration of the discovery was successful:

jmoekamp@inititator:~# iscsiadm list discovery
Discovery:
        Static: disabled
        Send Targets: enabled
        iSNS: disabled

Now let's look, what targets were discovered by the iSCSI initiator:

jmoekamp@inititator:~# iscsiadm list target
Target: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
        Alias: -
        TPGT: 2
        ISID: 4000002a0000
        Connections: 1

The iSCSI target IQN you saw at the time when you configured the target on the host target reappears here. Okay, now we have to make all discovered logical units for the use as block devices. In Solaris you use the devfsadm command for this task. This will populate the /dev tree:

jmoekamp@inititator:~# devfsadm -C -i iscsi
jmoekamp@inititator:~# 

When we start format again, we will see that our host initiator has a new block device:

jmoekamp@inititator:~# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0t600144F0B152CC0000004B080F230004d0 
          /scsi_vhci/disk@g600144f0b152cc0000004b080f230004
       1. c7d0 
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
Specify disk (enter its number): ^C
jmoekamp@inititator:~# 

A brand new 10 TB iSCSI volume. Of course we can use it for zfs again:

jmoekamp@inititator:~# zpool create testpool c0t600144F0B152CC0000004B080F230004d0
jmoekamp@inititator:~# zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
rpool                       10,1G  5,52G    81K  /rpool
[...]
testpool                      72K  9,78T    21K  /testpool

Works as designed. A nice 10 TB ZFS pool on initiator.

Configuring the initator with authentication

Okay, in the non-COMSTAR iSCSI tutorial i explained how to use authentication. With authentication the iSCSI target and the iSCSI initiator can check if the host on the other side of the connection is really the expected node. This works with the CHAP mechanism. Perhaps you know it from PPP for your internet access. It works on the foundation of shared secrets. Only when both partners of a communication know the same secret, the communication partner is authentic. It similar to a secret handshake between two peoples.

Okay, let's configure a bidirectional secret handshake. Just as a good admin we export the pool on it, as the following configuration will terminate the iSCSI connection for a moment.

jmoekamp@inititator:~# zpool export testpool
jmoekamp@inititator:~# 

Okay, to configure the authentication we need to pieces of information at first. Log into the system target to gather the iqn of the target.

jmoekamp@target:~# itadm  list-target
TARGET NAME                                                  STATE    SESSIONS 
iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422  online   1

Now log into the system initiator to gather the IQN of the initiator.

jmoekamp@inititator:~# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:809526a500ff.4b07e652
[...]
        Configured Sessions: 1

At first we configure the iSCSI target in a way, that it uses chap authentication. We need the IQN of the target here.

jmoekamp@target:~# itadm modify-target -a chap iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully modified

Now we have to configure the secrets. The CHAP secrets have be at last 12 characters. At first we set configure the secret, that the initiator will use to authenticate at the target.

jmoekamp@target:~# itadm create-initiator -s iqn.1986-03.com.sun:01:809526a500ff.4b07e652
Enter CHAP secret: foobarfoobarfoobar
Re-enter secret: foobarfoobarfoobar 

Now we configure the secret that the target uses to authenticate itself at the initiator.

jmoekamp@target:~# itadm modify-target -s iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Enter CHAP secret: snafusnafusnafu
Re-enter secret: snafusnafusnafu
Target iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422 successfully modified

Okay, we have to something similar on the system initiator
At first we have to configure the CHAP secret that the initiator uses to authenticate itself

jmoekamp@inititator:~# iscsiadm modify initiator-node --CHAP-secret
Enter secret: foobarfoobarfoobar
Re-enter secret: foobarfoobarfoobar

Now we tell the system that this initiator should use CHAP authentication to authenticate a target.

jmoekamp@inititator:~# iscsiadm modify initiator-node --authentication CHAP

The next steps configure the authentication relation between our initiator and a defined target. At we activate an bi-directional authentication. So the initiator has to authenticate at the target as well as the target has to authenticate itself at the initiator.

jmoekamp@inititator:~# iscsiadm modify target-param \
 --bi-directional-authentication enable iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422

Now we tell the iSCSI initator that the iSCSI target uses CHAP to authenticate the ISCSI initiator.

jmoekamp@inititator:~# iscsiadm modify target-param \ 
 --authentication CHAP iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422

In a last step we configure the shared secret, that the target uses to authenticate the initiator.

jmoekamp@inititator:~# iscsiadm modify target-param \
 --CHAP-secret iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
Enter secret: snafusnafusnafu
Re-enter secret: snafusnafusnafu

Perhaps this figure explain the relation between the secrets better than just command lines can do:



Now we are done. We can do a quick check if our configuration found it's way to the system. At first a quick look at the iSCSI Target on target:

jmoekamp@target:~# itadm  list-initiator
INITIATOR NAME                                               CHAPUSER  SECRET 
iqn.1986-03.com.sun:01:809526a500ff.4b07e652                          set      
jmoekamp@target:~# itadm list-target -v
TARGET NAME                                                  STATE    SESSIONS 
iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422  online   1        
        alias:                  -
        auth:                   chap 
        targetchapuser:         -
        targetchapsecret:       set
        tpg-tags:               e1000g0 = 2

Now we do the same check on initiator:

jmoekamp@inititator:~# iscsiadm list initiator-node
Initiator node name: iqn.1986-03.com.sun:01:809526a500ff.4b07e652
Initiator node alias: inititator
[...]
        Authentication Type: CHAP
                CHAP Name: iqn.1986-03.com.sun:01:809526a500ff.4b07e652
[...]
        Configured Sessions: 1

and

jmoekamp@inititator:~# iscsiadm list target-param -v
Target: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
        Alias: -
        Bi-directional Authentication: enabled
        Authentication Type: CHAP
                CHAP Name: iqn.1986-03.com.sun:02:fd9d53cd-6fe5-ecd7-de40-cb3bbb03b422
[...]
        Configured Sessions: 1

jmoekamp@inititator:~#

Okay, everything is okay ... now let's just reimport the testpool

jmoekamp@inititator:~# zpool import testpool
jmoekamp@inititator:~#

We are back in business.

Conclusion

The configuration of an iSCSI Target isn't more difficult than before. It's just different. I hope i gave you some good insight into this topic.

Do you want to learn more?

man pages:
docs.sun.com: sbdadm(1M) - SCSI Block Disk command line interface
docs.sun.com: stmfadm(1M)
docs.sun.com: itadm(1M) - administer iSCSI targets
docs.sun.com: iscsiadm(1M) - enable management of iSCSI initiators

Tutorials
c0t0d0s0.org : Less known Solaris Features: iSCSI - tutorial for the userland iSCSI implementation in Solaris.