Of course it's a little bit tedious to zip cables from your switch to test IPMP or to deconfigure an interface to force traffic on different interfaces when you want to do some maintainance work in your network. There is an command to on- and offline physical connections in an IPMP group. It's called if_mpadm
.
Using if_mpadm
Let's check, if we have an working IPMP configuration:
jmoekamp@hivemind:~# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
production0 production0 ok 10,00s rge0 aggregate0
jmoekamp@hivemind:~# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
rge0 yes production0 ------- up ok ok
aggregate0 yes production0 --mb--- up ok ok
Looks good. Now let's offline one of the interfaces. We will now offline the interface
aggregate0
with the
if_mpadm
:
jmoekamp@hivemind:~# if_mpadm -d aggregate0
Okay, now let's check the IPMP system again:
jmoekamp@hivemind:~# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
rge0 yes production0 --mb--- up ok ok
aggregate0 no production0 -----d- up disabled offline
You will see some important differences. At first
rge0
is now responsible for multicasts and broadcasts. It switched over from
aggregate0
. Furthermore you will see, that
aggregate0
is not longer active, the probing is disabled and the state has switched to "offline".
jmoekamp@hivemind:~# ipmpstat -a
ADDRESS STATE GROUP INBOUND OUTBOUND
hivemind-prod up production0 rge0 rge0
When you look into the output of
ipmpstat -a
, you will recognize that
aggregate0
isn't used for inbound as well as for outbound traffic of the interface
hivemind-prod
. So it worked as designed.
Okay, lets' play a little bit with this command:
jmoekamp@hivemind:~# if_mpadm -d rge0
if_mpadm: fatal: cannot offline rge0: no other functioning interfaces are in its IPMP group
The
if_mpadm
tool prevents you from shooting into your own foot by denying the offlining of the last functional interface in the ipmp group. Now let's put the device online again.
jmoekamp@hivemind:~# if_mpadm -r aggregate0
jmoekamp@hivemind:~# ipmpstat -a
ADDRESS STATE GROUP INBOUND OUTBOUND
hivemind-prod up production0 rge0 rge0 aggregate0
The
aggreagte0
interface reappeared as an active device. Now we are able to disable the
rge0
interface.
jmoekamp@hivemind:~# if_mpadm -d rge0
Let's check the status of the IPMP subsystem:
jmoekamp@hivemind:~# ipmpstat -a
ADDRESS STATE GROUP INBOUND OUTBOUND
hivemind-prod up production0 aggregate0 aggregate0
When you look at the
ifconfig -a
output you will recognize some additional flags.
jmoekamp@hivemind:~# ifconfig -a
[..]
rge0: flags=89040842
<BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,OFFLINE>
mtu 1500 index 4
inet 192.168.2.210 netmask ffffff00 broadcast 192.168.2.255
groupname production0
ether 0:24:1d:7d:f4:b3
[..]
When an interface has been offlined, it doesn't carry the
UP
flag, but it got the
OFFLINE
flag instead. By the way: As the interface is still perfectly operational (just offlined by the admin) it keeps the
RUNNING
flag. Now we reactivate the
rge0
interface.
jmoekamp@hivemind:~# if_mpadm -r rge0
Let's look at the
ifconfig -a
output again:
jmoekamp@hivemind:~# ifconfig -a
[..]
rge0: flags=9040843
<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER>
mtu 1500 index 4
inet 192.168.2.210 netmask ffffff00 broadcast 192.168.2.255
groupname production0
ether 0:24:1d:7d:f4:b3
[..]
jmoekamp@hivemind:~#
The
OFFLINE
flag disappeared and the
UP
status reappeared.
Conclusion
if_mpadm
is a somewhat less known and small tool, albeit every admin working with IP Multipathing should know it, as it's really useful for testing and maintainance purposes.
You you want to learn more ?
man pages:
docs.sun.com:
if_mpadm (Opensolaris)
docs.sun.com:
if_mpadm (Solaris 10)