Less known Solaris features: hxbt - or: WAN emulation

Quicksearch

Disclaimer

The individual owning this blog works for Oracle in Germany. The opinions expressed here are his own, are not necessarily reviewed in advance by anyone but the individual author, and neither Oracle nor any other party necessarily agrees with them.

Navigation

< Slowing down resilvering of ZFS | Practical DTracing >

Less known Solaris features: hxbt - or: WAN emulation

Friday, June 11. 2010

Some features are so "less known" that even people with the reputation of knowing almost everything about Solaris aren't aware of them. One of this features is the hxbt driver. In this tutorial i want to explain how you get this drive and how you install and use it.

What is hxbt?

Whenever you are testing an application or feature there is a single infrastructure component that's somewhat hard to test. It's the network between a client and a server ... or between two servers replicating data between them. At LAN speed everything is fine, but what's with WAN speed. 1 GBps leased lines aren't that common and latencies from your notebook to the server under your desktop aren't really representative.

So it would be nice too simulate a WAN connection, with longer delays, with corrupted or missing packets to test this functionality.

With hxbt there is a STREAMS driver for Solaris that implements such a WAN simulation that's able to simulate certain characteristics of a wide area connection.

How do i get it and how do i use it?

hxbt is not part of the Solaris. So you have to download it first. You will find it at the network community page of opensolaris.org

jmoekamp@server:~$ wget http://hub.opensolaris.org/bin/download/Community+Group+networking/WebHome/hxbt.tar.gz --2010-06-11 15:48:26-- http://hub.opensolaris.org/bin/download/Community+Group+networking/WebHome/hxbt.tar.gz Resolving hub.opensolaris.org (hub.opensolaris.org)... 192.9.164.72 Connecting to hub.opensolaris.org (hub.opensolaris.org)|192.9.164.72|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 485471 (474K) [application/x-gzip] Saving to: `hxbt.tar.gz' 100%[======================================>] 485,471 255K/s in 1.9s 2010-06-11 15:48:29 (255 KB/s) - `hxbt.tar.gz' saved [485471/485471]

Now you can untar this file:

jmoekamp@server:~$ tar xzfv hxbt.tar.gz hxbt/README.hxbt hxbt/onnv/usr/src/uts/intel/hxbt/ [...] hxbt/onnv/usr/src/uts/sparc/Makefile.sparc $

You will find the hxbt driver for SPARC as well as for x86 in the tarball.

When you are using an x86 system, you have to copy it from here:

jmoekamp@server:~# cp ./hxbt/onnv/usr/src/uts/intel/hxbt/obj32/hxbt /kernel/drv jmoekamp@server:~# cp ./hxbt/onnv/usr/src/uts/intel/hxbt/obj64/hxbt /kernel/drv/amd64

When you are using a SPARC system, then you have to use a different driver:

cp ./hxbt/onnv/usr/src/uts/sparc/hxbt/obj64/hxbt /kernel/drv/sparcv9

Independent from the architecture you have to copy hxbt.conf into the correct location.

jmoekamp@server:~$ cp ./hxbt/onnv/usr/src/uts/common/inet/hxbt/hxbt.conf /kernel/drv /kernel/drv

Obviously you need a tool to configure the WAN emulation. This tool is ifhit. However it gets a little bit difficult here. The tarball at opensolaris.org just contains the ifhit tool for SPARC. So when you use a SPARC system just copy the tool from the untarred distribution:

jmoekamp@server:~# cp ./hxbt/onnv/usr/src/cmd/cmd-inet/usr.sbin/ifhit/ifhit /usr/sbin

In the case you want to use hxbt you have to compile the tool on your own. To make it a little bit easier, i've compiled a version for you ... you will find it at my website.

jmoekamp@server:~$ wget http://uploads.rootpool.org/ifhit jmoekamp@server:~$ cp ifhit /usr/sbin jmoekamp@server:~$ chown root:root /usr/sbin/ifhit jmoekamp@server:~$ chmod u+x /usr/sbin/ifhit

Okay, now we have to add the driver and create the /dev/hxbt device node.

jmoekamp@server:~# add_drv -m"* 0666 root root" hxbt

The hxbt is an STREAMS driver that sit's between higher protocol layers and the network hardware driver. We have to insert it here, but at first we have to find the exact location in the stack.

jmoekamp@server:~# ifconfig e1000g0 modlist 0 ip 1 e1000g

Okay, the network hardware driver is at position 1, we want to insert the hxbt there, so so we put that driver at that location

jmoekamp@server:~# ifconfig e1000g0 modinsert hxbt@1

Now we check if everything went well ...

jmoekamp@server:~# ifconfig e1000g0 modlist 0 ip 1 hxbt 2 e1000g

Perfect ... not let's make a last check if the driver is really loaded .. just to be sure ...

jmoekamp@server:~# modinfo Id Loadaddr Size Info Rev Module Name 0 fffffffffb800000 1cb9fa - 0 unix () [...] 260 fffffffff7aa5000 22c8 - 1 hxbt (hxbt stream module v1.1) 260 fffffffff7aa5000 22c8 292 1 hxbt (hxbt stream driver v1.1)

Using `hxbt`

Let's play around with it ... let's assume we want to introduce an additional latency of 100 ms to our communication to the IP address 192.168.2.200:

jmoekamp@server:~# ifhit 192.168.2.200 -l 100 (::ffff:192.168.2.200) Delay = 100 ms

Let's check it ...

jmoekamp@server:~# ping -s 192.168.2.200 PING 192.168.2.200: 56 data bytes 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=93.191 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=104.319 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=2. time=102.006 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=3. time=103.962 ms ^C ----192.168.2.200 PING Statistics---- 4 packets transmitted, 4 packets received, 0% packet loss round-trip (ms) min/avg/max/stddev = 93.191/100.870/104.319/5.219

As you surely have noticed, the ping time is got longer. Okay ... let's remove this artifical latency. You use the -z option for this task.

jmoekamp@server:~# ifhit 192.168.2.200 -z (::ffff:192.168.2.200) is turned off

Let's check the ping times ...

jmoekamp@server:~# ping -s 192.168.2.200 PING 192.168.2.200: 56 data bytes 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=6.024 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=4.099 ms ^C ----192.168.2.200 PING Statistics---- 2 packets transmitted, 2 packets received, 0% packet loss round-trip (ms) min/avg/max/stddev = 4.099/5.061/6.024/1.361 jmoekamp@server:~#

They look vastly more normal than the ones with the artificial delay.Let's use a more complex pattern of problems.

jmoekamp@server:~# ifhit 192.168.2.200 -i 5 -n 2 -o 1 (::ffff:192.168.2.200) Drop interval = 5 pkts, drop length = 2 pkts, drop gap = 1 pkts

This command introduces a packet drop every 5 packets, two packets will be dropped with an undropped packet between them.

jmoekamp@server:~# ping -s 192.168.2.200 PING 192.168.2.200: 56 data bytes 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=0. time=4.060 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=1. time=3.953 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=2. time=4.776 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=3. time=4.264 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=4. time=7.889 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=6. time=2.406 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=8. time=4.824 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=9. time=3.739 ms 64 bytes from Server.home.moellenkamp.org (192.168.2.200): icmp_seq=10. time=3.593 ms ^C ----192.168.2.200 PING Statistics---- 11 packets transmitted, 9 packets received, 18% packet loss round-trip (ms) min/avg/max/stddev = 2.406/4.389/7.889/1.495

As you surely have noticed, you see that the fifth and the seventh package has been dopped by htbx.

There are a lot of other options to introduce limits, latencies or problems to a network connection. As there is no man page for the tool, i will just post the printout:

jmoekamp@server:~# ifhit  -h
Usage: ifhit [<host> | off]
  [ -z ]
  [ -d  | [ -i  -n  [ -o ]]]
  [ -l  ]
  [ -b  ]
  [ -r  -C  ]

  <host>: name of host to hit
  off: turn off hxbt for all targets
  -?: usage
  -z: turn off hxbt for this target
  -d : drop rate in %
  -i : interval between drops
  -n : how many packets to drop
  -o : (with -n), packets between drops
    NOTE: -d and -i -n are mutually exclusive
  -l : delay in ms
  -b : emulated link bandwidth in Kbps, default: interface bandwidth
  -r : re-ordering threshold
  -R : interval between reordering
  -c : packet corruption rate in %
  -C : number of bytes to corrupt per packet
jmoekamp@server:~#

Conclusion

htbx is really useful, especially for test environments. In the past i've used it to demonstrate the impact of network latencies to synchronous storage replication. It's a pity that it's isn't a part of any Solaris distribution.

Posted by Joerg Moellenkamp in English, Oracle, Solaris at 19:56 | Comments (5) | Trackbacks (0)

View as PDF: This entry | This month | Full blog

Trackbacks

Trackback specific URI for this entry

No Trackbacks

Comments

Display comments as (Linear | Threaded)

Nice article Joerg,

Do you have an recommendations on how to make these changes permanent to allow a zone/server reboot?

Ta.

#1 Keith on 2010-06-11 21:35 (Reply)

In the past i've used a small legacy init.d script for this tasks ....

#2 Joerg M. (Homepage) on 2010-06-12 08:17 (Reply)

Interesting ! htbx might be useful as well for streaming powered services: Flash, Darwin. I need to check this out.

#3 Stefan Parvu (Homepage) on 2010-06-13 13:19 (Reply)

Very nice. So this would be a nice way to test the limits of ZFS replication over a WAN?

#4 Scott on 2010-06-13 20:57 (Reply)

Cool - danke. Kannt ich echt noch nicht.

S.

#5 stepheaux on 2010-06-17 10:08 (Reply)

Add Comment

Name
Email
Homepage
In reply to
Comment	E-Mail addresses will not be displayed and will only be used for E-Mail notifications. To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly. Enter the string from the spam-prevention image above: Standard emoticons like :-) and ;-) are converted to images. Enclosing asterisks marks text as bold (word), underscore are made via _word_.
	Remember Information? Subscribe to this entry