Less known Solaris features - IP Multipathing (Appendix A): if_mpadm

Quicksearch

Disclaimer

The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.

Navigation

< Less known Solaris features - Persistant routes | Perceived Risk >

Less known Solaris features - IP Multipathing (Appendix A): if_mpadm

Wednesday, February 3. 2010

Of course it's a little bit tedious to zip cables from your switch to test IPMP or to deconfigure an interface to force traffic on different interfaces when you want to do some maintainance work in your network. There is an command to on- and offline physical connections in an IPMP group. It's called if_mpadm.

Using if_mpadm

Let's check, if we have an working IPMP configuration:

jmoekamp@hivemind:~# ipmpstat -g
GROUP       GROUPNAME   STATE     FDT       INTERFACES
production0 production0 ok        10,00s    rge0 aggregate0
jmoekamp@hivemind:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
rge0        yes     production0 -------   up        ok        ok
aggregate0  yes     production0 --mb---   up        ok        ok

Looks good. Now let's offline one of the interfaces. We will now offline the interface aggregate0 with the if_mpadm:

jmoekamp@hivemind:~# if_mpadm -d aggregate0

Okay, now let's check the IPMP system again:

jmoekamp@hivemind:~# ipmpstat -i
INTERFACE   ACTIVE  GROUP       FLAGS     LINK      PROBE     STATE
rge0        yes     production0 --mb---   up        ok        ok
aggregate0  no      production0 -----d-   up        disabled  offline

You will see some important differences. At first rge0 is now responsible for multicasts and broadcasts. It switched over from aggregate0. Furthermore you will see, that aggregate0 is not longer active, the probing is disabled and the state has switched to "offline".

jmoekamp@hivemind:~# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
hivemind-prod             up     production0 rge0        rge0

When you look into the output of ipmpstat -a, you will recognize that aggregate0 isn't used for inbound as well as for outbound traffic of the interface hivemind-prod. So it worked as designed.

Okay, lets' play a little bit with this command:

jmoekamp@hivemind:~# if_mpadm -d rge0
if_mpadm: fatal: cannot offline rge0: no other functioning interfaces are in its IPMP group

The if_mpadm tool prevents you from shooting into your own foot by denying the offlining of the last functional interface in the ipmp group. Now let's put the device online again.

jmoekamp@hivemind:~# if_mpadm -r aggregate0
jmoekamp@hivemind:~# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
hivemind-prod             up     production0 rge0        rge0 aggregate0

The aggreagte0 interface reappeared as an active device. Now we are able to disable the rge0 interface.

jmoekamp@hivemind:~# if_mpadm -d rge0

Let's check the status of the IPMP subsystem:

jmoekamp@hivemind:~# ipmpstat -a
ADDRESS                   STATE  GROUP       INBOUND     OUTBOUND
hivemind-prod             up     production0 aggregate0  aggregate0

When you look at the ifconfig -a output you will recognize some additional flags.

jmoekamp@hivemind:~# ifconfig -a
[..]
rge0: flags=89040842
         <BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,OFFLINE>
         mtu 1500 index 4
	inet 192.168.2.210 netmask ffffff00 broadcast 192.168.2.255
	groupname production0
	ether 0:24:1d:7d:f4:b3 
[..]

When an interface has been offlined, it doesn't carry the UP flag, but it got the OFFLINE flag instead. By the way: As the interface is still perfectly operational (just offlined by the admin) it keeps the RUNNING flag. Now we reactivate the rge0 interface.

jmoekamp@hivemind:~# if_mpadm -r rge0

Let's look at the ifconfig -a output again:

jmoekamp@hivemind:~# ifconfig -a
[..]
rge0: flags=9040843
         <UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
         mtu 1500 index 4
	inet 192.168.2.210 netmask ffffff00 broadcast 192.168.2.255
	groupname production0
	ether 0:24:1d:7d:f4:b3 
[..]
jmoekamp@hivemind:~#

The OFFLINE flag disappeared and the UP status reappeared.

Conclusion

if_mpadm is a somewhat less known and small tool, albeit every admin working with IP Multipathing should know it, as it's really useful for testing and maintainance purposes.