Less known Solaris features - IP Multipathing (Part 6): New IPMP
When you want to try new IPMP, you need a fairly recent build of OpenSolaris. New IPMP was integrated into Build 107 for the first time. At first: If you have already a working IPMP configuration, you can simply reuse this config. However it yields a different looking, but functionally equivalent result compared to your system with Classic IPMP. This is made possible by some automagic functions in New IPMP. One example is the implicit creation of the IPMP interface with the name of the IPMP group when there isn’t already an IPMP interface in the group. However explicit creation should be prefered as you can choose a better name for your IPMP interface and the dependencies are much more obvious. As I wrote before, the new IPMP doesn’t use a logical interface switching from one interface to the other. You have a special kind of interface for it. It’s a virtual interface. It’s looking like a real interface but there is no hardware behind this interface. So … at first we have to configure this interface:
With this command you’ve configured the IPMP interface. You can use any name for it you want, it just has to begin with a letter and has to end on a number. I have chosen the name
production0 for this tutorial
Now let’s look at the interface:
As you see, it’s pretty much looking like a normal network interface with some specialities: At first it’s in the mode
FAILED at the moment. There are no network interfaces configured to the group, thus you can’t connect anywhere over this interface.
The interface is already configured with the data address.(Additional data addresses are configured as logical interfaces onto this virtual interface. You won’t configure additional virtual IPMP interfaces). The data address will never move away from there. At the end you see the name of the IPMP group. The default behavior sets the name of the IPMP group and the name of the IPMP interface to the same value.
Okay, now we have to assign some physical interfaces to it. This is the moment where we have to make a decision. Do we want to use IPMP with probes or without probes? As I’ve explained before it’s important to know at this point, what failure scenarios you want to cover with your configuration. You need to know it now, as the configuration is slightly different.
Link based failure detection
I want to explain the configuration of the link based failure detection first not only because it’s easier, but to show you the problems of link based failure detection, too.
As explained before, the link based failure detection just snoops on certain events of the networking card like a lost link. Thus we just have to configure the interface into the IPMP group that you want to protect against a link failure, but you don’t have to configure any IP addresses on the member interfaces of the IPMP group. Okay, at first we plumb the interfaces we want to use in our IPMP group:
Okay, now we add the three member interfaces into the IPMP group:
As you may have noticed, we really didn’t specify an IP address or a hostname. With link-based failure detection you don’t need it. The IP address of the group is located on the IPMP interface we’ve defined a few moments ago.
But let’s have a look at the
ifconfig statements. There are two parameters you may not know:
-failover: This parameter marks an interface as a non-failover one. In case of a failure, this interface configuration doesn't move. While a little bit strange in the context of a physical interface (Moving hardware is a little bit problematic just by software....), but the rationale gets clearer with probe-based IPMP.
group production0: the parameter
groupdesignates the IPMP group membership of an interface.
Let’s look at one of the interfaces:
We find the consequences of both
ifconfig parameters: The
NOFAILOVER is obviously the result of the
groupname production0 is the result of the
group production0 statement. But there is another flag that is important in the realm of IPMP. It’s the
DEPRECATED flag has a very simple meaning: Don’t use this IP interface. When an interface has this flag, the IP address won’t be used to send out data (Of course there are exceptions like an application specifically binding to the interface. Please look into the man page for further information). As those IP addresses are just for test purposes, you don’t want them to appear in packets to the outside world.
Now we need to interact with the hardware, as we will fail network connections manually. Or to say it differently: We will pull some cables.
But before we are doing this, we look at the initial status of our IPMP configuration. The new IPMP model improved the monitoring capabilities of it’s state by introducing a command for this task. It’s called
Just to give you a brief tour through the output of the command. The first column reports the name of the interface, the next one reports the state of the interface from the perspective of IPMP. The third column tells you which IPMP group was assigned to this interface.
The next columns gives us some more in-depth information about the interface. The fourth column is a multipurpose column to report a number of states. In the last output, the
--mb-- tells us, that the interface e1000g0 was chosen for sending and receiving multicast and broadcast data. Other interfaces doesn’t have a special state, so there are just dashes in the respective
=FLAGS= field of these interfaces. The fifth column reveals, that we’ve disabled probes (Or to be more exact, that there is no probing as we didn’t configured it so far). The last column details on the state of the interface. In this example it is
OK and so it’s used in the IPMP group.
Okay, now pull the cable from the
e1000g0 interface. It’s Cable 1 in the last figure. The system automatically switches to e1000g1 as the active interface.
As you can see, the failure has been detected on the
e1000g0 interface. The link is down, thus it is no longer active. Okay, let’s repair it.
Put the cable back to the port of the
e1000g0 interface. After a moments, the link is up. The
in.mpathd gets aware of the
RUNNING flag on the interface.
in.mpathd assumes that the network connection got repaired, so the state of the interface is set to
ok and thus the interface is reactivated.
The problem with link-based failure detection
Just in case you’ve played with the ethernet cables, ensure that IPMP chooses an interface connecting via Switch A as the active interface by zipping Cable 3 from the switch B for a moment. When you check with
ipmpstat -i the
mb has to be assigned to the interface
As i wrote before there are failure modes link-based failure detection can’t detect. Now let’s introduce such a fault. To do so, just remove Cable 4 between switch A and B.
As there is still a link on the Cables 1 and 2 everything is fine from the perspective of IPMP. It doesn’t switch to the connection via
rge0 which presents the only working connection to the outside world. IPMP is simply not aware of the fact that Switch A was seperated from the IP link
192.168.178.0/24 due to the removal of cable 4.
Probe based failure detection
The probe based detection has some additional capabilities. At first it has all the capabilities of the link-based detection. It switches over to the other network card as soon as the card loses the link. But additionally it checks the availability of the connection by pinging other IP addresses called target systems. When the system doesn’t get a reply on the ICMP messages, the interface is assumed to be in failure state and it isn’t used anymore.
in.mpathd switches the data addresses to other interfaces. So how do you configure probe based IPMP?
Okay, at first we revert back to the original state of the system. This is easy, we just have to unplumb the interfaces. In my example I’m unplumbing all interfaces. You could reuse the
production0 interface, but I’m including it here just in case you’ve started reading this tutorial at the beginning of this paragraph (In this case, the first three commands will fail, but you have the explicitly defined IPMP interface). It’s important that you unplumb the member interfaces of the group before you unplumb the IPMP interface, otherwise you get an error message:
Okay, now all the interfaces are away. Now we recreate the IPMP group.
We can check the successful creation of the IPMP interface by using the
At start there isn’t an interface configured into the IPMP group. So let’s start to fill the group with some life.
There is an important difference. This
ifconfig statement contains an IP address, that is assigned to the physical interface. This automatically configures IPMP to use the probe based failure detection.
The idea behind the
-failover= setting gets clearer now. Obviously the test addresses of an interface should be failovered by IPMP. They should stay on the logical interface. As the interface has the
FAILOVER flag, the complete interface including it's IP address is exempted from any failover.
Let's check the ipmp group again:
There is now an interface in the group. Of course an IPMP group with just one interface doesn’t really make sense. So configure we will configure a second interface into the group. You may have recognized the
FTD stands for “Failure Detection Time”. Why is there an own column for this number? Due to the dynamic nature of the Failure Detection time, the
FDT may be different for every group. With this column you can check the the current
Let’s check again.
Now we add the third interface that is connected to the default gateway just via Switch B.
Let’s check again.
All three interfaces are in the IPMP group now. And that’s all … we’ve just activated failure detection and failover by this four commands. Really simple, isn’t it?
I hope, you have still the hardware configuration in place, I used to show the problems of link based failure detection. In the case you haven’t please create the configuration we’ve used there. At first we do a simple test: We simply unplug a cable from the system. In my case I removed the cable 1:
The system reacts immediately, as the link-based failure detection is still active, even when you use the probe-based mechanism. You can observe this in the
ipmpstat output by monitoring the state of the link column. It’s
down at the moment and obviously probes can’t reach their targets. The state is assumed as
failed. Now plug the cable back to the system:
The link is back, but the interface is still failed. IPMP works as designed here. The probing of the interface with ICMP messages still considers this interface as down. As we have now two mechanism to check the availability of the interface, both have to confirm the repair. IPMP doesn’t consider an interface as repaired when just one ICMP probe gets through, it waits until 20 ICMP probes were correctly replied by the target system. Due to this probing at repair time instead of just relying on the link, you can prevent that an interface is considered as
OK when an unconfigured switch brings the link back online, but the configuration of the switch doesn’t allow to the server to connect anywhere (because of a missing VLAN configuration for example).
As soon as the probing of the interface is successful, it brings the interface back to the
OK state and everything is fine.
Now we get to a more interesting use case of probe-based failure detection. Let’s assume we’ve repaired everything and all is fine. You should see a situation similar to this one in your
Now unplug cable 4, the cable between the switch A and B. At first nothing happens, but a few seconds later IPMP switches the IP addresses to
rge0 and set the state of the other interfaces to
When you look at the output of
ipmpstat you will notice that the link is still up, but the probe has failed, thus the interfaces were set into the state failed.
When you plug the cable 3 back to the switches nothing will happen at first. You have to wait until the probing mechanism reports that the IPMP messages were correctly returned by the target systems.
After a few seconds it should deliver an
ipmpstat output reporting everything is well again.
Making the configuration boot persistent
As you have recognized for sure, all this configuration took place with the
ifconfig statement. This configuration is lost when you reboot the system. But there is already an entity that configures the interfaces at system start. It’s using the
hostname.* files. Thus we could use these files for IPMP as well.
Boot persistent link-based configuration
Okay, to recreate our link-based IPMP configuration in a boot persistent, we need to fill the
hostname.* files with the following statements:
We reboot the system now to ensure that we did everything correctly. When the system has booted up, we will check if we made an error.
Looks good. Now let’s look into the list of interfaces.
IPMP configured and boot-persistent? Check.
Boot persistent probe-based configuration
We can do the same for the probe-based IPMP:
Reboot the system and login afterwards to check the list of interfaces.
Let’s check the configuration via
Everything is fine.