on
Less known Solaris features - IP Multipathing (Part 8): Classic IPMP
IPMP itself is a really old feature. It’s in Solaris for several versions now. Just the implementation I’ve described before is a new one. But in Solaris 10 you don’t have this new IPMP implementation. Solaris 10 still uses the old implementation. I will call the old implementation classic IPMP. The basic mechanism of new and classic IPMP is pretty much the same: Providing failure detection mechanisms and switch something to do a failover thus the data address stays available. But internally it’s a completely different implementation. While the new mechanism is certainly the future of IPMP, I’ m pretty sure you will old mechanism more often in the wild.
Prerequisites
This example works with the same configuration, but you need a system with Solaris 10 or an Opensolaris System with a build earlier than 107. I just use Opensolaris on my lab machines, thus i used an virtualized Solaris 10 to explain the configuration of classic IPMP. I will use the following addresses:
192.168.178.200 vhivemind-prod
192.168.178.201 vhivemind-e1000g0
192.168.178.202 vhivemind-e1000g1I will demonstrate this on a recent release of Solaris 10:
bash-3.00# cat /etc/release
Solaris 10 5/09 s10x_u7wos_08 X86
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 30 March 2009Link based classic IPMP
An important difference to the new IPMP implementation is the point that you doesn’t create a distinct IPMP interface because the concept of such a thing doesn’t exist in classic IPMP. With link based classic IPMP you just put the interfaces in a group
bash-3.00# ifconfig e1000g0 plumb
bash-3.00# ifconfig e1000g1 plumb
bash-3.00# ifconfig e1000g0 vhivemind-prod netmask + broadcast + group production0 up
bash-3.00# ifconfig e1000g1 group production0 up
bash-3.00# ifconfig -aLet’s have a short look onto the network configuration.
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 15
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:11:34:43
e1000g1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 16
inet 0.0.0.0 netmask ff000000
groupname production0
ether 8:0:27:6d:9:beThe data address is directly bound to one of the interfaces. It’s important to know,that even when the ifconfig output suggest something different, outbound data flows to the network on both interfaces, not just the one which holds the data address.
Now unplug the cable connecting to e1000g0:
bash-3.00# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=219000802<BROADCAST,MULTICAST,IPv4,NOFAILOVER,FAILED,CoS> mtu 0 index 15
inet 0.0.0.0 netmask 0
groupname production0
ether 8:0:27:11:34:43
e1000g1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 16
inet 0.0.0.0 netmask ff000000
groupname production0
ether 8:0:27:6d:9:be
e1000g1:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 16
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255The data address was moved away from e1000g1 and a logical interface was created to hold it instead.
Probe based classic IPMP
The failure detection by IPMP probes is available in classic IPMP as well. Again all the configuration is done via ifconfig
bash-3.00# ifconfig e1000g0 plumb
bash-3.00# ifconfig e1000g1 plumb
bash-3.00# ifconfig e1000g0 vhivemind-e1000g0 deprecated -failover netmask + broadcast + group production0 up
bash-3.00# ifconfig e1000g0 addif vhivemind-prod netmask + broadcast + up
Created new logical interface e1000g0:1
bash-3.00# ifconfig e1000g1 vhivemind-e1000g1 deprecated -failover netmask + broadcast + group production0 up
bash-3.00# ifconfig -aPlease note that you have to use the deprecated option to set the DEPRECATED flag on your own. New IPMP do this automagically. Forgetting this option leads to interesting, but not always obvious malfunctions. Let's check the network configuration.
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=209040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,CoS> mtu 1500 index 11
inet 192.168.56.201 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:11:34:43
e1000g0:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 11
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255
e1000g1: flags=209040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,CoS> mtu 1500 index 12
inet 192.168.56.202 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:6d:9:beBoth interfaces have their test addresses. The data address is configured to an additional logical interface. As it’s the only interface without the -failover statement, this interface is automatically managed by IPMP. Now remove the cable from the e1000g0 networking card.
bash-3.00# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=219040803<UP,BROADCAST,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FAILED,CoS> mtu 1500 index 11
inet 192.168.56.201 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:11:34:43
e1000g1: flags=209040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,CoS> mtu 1500 index 12
inet 192.168.56.202 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:6d:9:be
e1000g1:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 12
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255The virtual interface with the data address has moved from e1000g0 to e1000g1
Making the configuration boot persistent
Making the configuration boot-persistent works works pretty much the same in both implementation. As we used ifconfig commands again, we can use the hostname.* files. We just have to translate the command lines accordingly:
Link-based IPMP
At first we configure the e1000g0 interface by creating the file /etc/hostname.e1000g0 containing a single line.
vhivemind-prod netmask + broadcast + group production0 upAfterwards we do the same for e1000g1. We create a file named /etc/hostname.e1000g1 and put the following line (and just this line) in it:
group production0 upNow reboot the system. After a few moments you can get a shell and check your configuration.
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:11:34:43
e1000g1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
groupname production0
ether 8:0:27:6d:9:beEverything configured as we’ve planned it.
Probe-based IPMP
Okay, let’s do the same for the probe-based IPMP. This is the /etc/hostname.e1000g0 file configuring the test address on the physical interface and the data address:
vhivemind-e1000g0 deprecated -failover netmask + broadcast + group production0 up \
addif vhivemind-prod netmask + broadcast + upThe file /etc/hostname.e1000g1 with the following line will configures the e1000g1 interface of our system at boot:
vhivemind-e1000g1 deprecated -failover netmask + broadcast + group production0 upOkay, reboot your system and you should yield an ifconfig output like this one afterwards.
# ifconfig -a
lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
inet 127.0.0.1 netmask ff000000
e1000g0: flags=209040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,CoS> mtu 1500 index 2
inet 192.168.56.201 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:11:34:43
e1000g0:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
inet 192.168.56.200 netmask ffffff00 broadcast 192.168.56.255
e1000g1: flags=209040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,CoS> mtu 1500 index 3
inet 192.168.56.202 netmask ffffff00 broadcast 192.168.56.255
groupname production0
ether 8:0:27:6d:9:beEverything is fine.