Written by J. Moellenkamp
on 
Reading time: 6 minutes
Solaris   English

Simulating the cloud - a practical example.

Work in Progress - this entry will change often in the next days and weeks A few days\^H\^H\^H\^Hweeks ago, i wrote about simulating the cloud that is most often tagged with the name “network” or “intranet” and sometimes “internet” . This would not be c0t0d0s0.org without an article to explain how you can configure this. This article will explain how you simulate a complete network on a single host with routers, switches, dynamic routing protocols and so on

Scope

At first i want to set the expectations right. I don’t want to simulate a cloud in the sense of cloud computing here. I’m thinking about something more complex:

I’m talking about the simulation of this cloud, that often hides a lot of complexities and traps in architectural diagrams.

A word of caution first

This article uses a invisible feature. You don’t see that it’s there because it isn’t in the man page, it isn’t in the help output of the dladm command. But it’s there. It’s the commands dladm create/modify-simnet. As it’s undocumented i assume it just can disappear without any notice, because it’s not there. Don’t complain here when it disappears, don’t complain at Oracle. Of course no support. You know the game. Consider it as an artifact. As a diagnosis socket labeled “Only for factory use”. Consider it as the testing wiring existing in every technical product that’s just use for the testing when the product leaves the factory. Never ever use it in production. Why i’m writing about this “feature” here? Because it’s useful. Because there are a multitude of hints that this function exists. All of them are public. The zonestat documentation mentions a “simnet” type at docs.oracle.com and from there you are just a google away from the PSARC case 2009/200. And the source code at src.opensolaris.org shows it as well. From there it’s just curiosity to find everything else out that is used in this text.

About this article

I stumbled the first time over this command when i searched for something in the dladm source at src.opensolaris.org. A month ago my former colleague Brian Utterback remembered me of this and i though “let’s check if this is still working”. And to my astonishment it still worked. Writing this article takes virtually forever. Because of my broken ankle i took painkillers and that made me somewhat drownsy. And this drownsyness slowed down everything. Thus i decided to create this article under your observation to get it finally out of the door. Thus it’s work in progress.

simnet

I just write about simnet. What are simnets? I just want to point you to the PSARC case for indepth information. It’s available on opensolaris.org in the caselog. But in short: Simnets are simulated networks. It’s a mechanism to test networking protocols. And in this example we will use it exactly for this purpose. Testing networking. <h3Simulating the cloud</h3> Okay, let’s assume you are admin of FUBAR Inc. You want to recreate your network in a box. You have offices in Hamburg, London, Singapore, New York and San Francisco. In each office you have a multi-legged router, connecting to a switch for the internal network with servers an clients, the other interfaces of the switch are connecting to the other routers. As an image says more than 1000 words i will just summarize the network with this figure.

Configuring it

Of course the and the servers will be zones. However we have to recreate the network topology as well. And that’s the point where we use the the simnet non-feature. We need a the switches in our offices first. Those are really easy to configure

root@cloudinabox:/opt/cloudsimulation/zones# dladm create-bridge london
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-bridge hamburg
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-bridge singapore
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-bridge newyork
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-bridge sanfrancisco

Now i need some switchports. At first i create some switch ports in order to connect the switch to the router.

root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonsw1_255
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sanfranciscosw1_255
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet newyorksw1_255
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgsw1_255
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet singaporesw1_255

Now i create some additional switchports to connect servers.

root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonsw1_1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonsw1_2
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgsw1_1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet singaporesw1_1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet newyorksw1_1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sanfranciscosw1_1

Ports meant for the bridge are nice, however they should be connected with the bridge.

root@cloudinabox:/opt/cloudsimulation/zones# dladm add-bridge -l londonsw1_1 -l londonsw1_2 -l londonsw1_255 london
root@cloudinabox:/opt/cloudsimulation/zones# dladm add-bridge -l hamburgsw1_1 -l hamburgsw1_255 hamburg
root@cloudinabox:/opt/cloudsimulation/zones# dladm add-bridge -l singaporesw1_1 -l singaporesw1_255 singapore
root@cloudinabox:/opt/cloudsimulation/zones# dladm add-bridge -l sanfranciscosw1_1 -l sanfranciscosw1_255 sanfrancisco
root@cloudinabox:/opt/cloudsimulation/zones# dladm add-bridge -l newyorksw1_1 -l newyorksw1_255 newyork

Let’s now create all the interfaces we need for the routers.

root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonrouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgrouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgrouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgrouter3
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet newyorkrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet newyorkrouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sinrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sinrouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sinrouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sforouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sforouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sforouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sforouter3

And of course we need interfaces for all the servers

root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonsrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet londonsrv2
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet hamburgsrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet singaporesrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet sanfranciscosrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm create-simnet newyorksrv1

Now we have to create logical cables … lots of them. At first the routers with their switches.

root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p londonsw1_255 londonrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p hamburgsw1_255 hamburgrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p singaporesw1_255 sinrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p newyorksw1_255 newyorkrouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p sanfranciscosw1_255 sforouter0
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p hamburgrouter1 londonrouter1
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p sforouter1 hamburgrouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p sinrouter1 hamburgrouter3
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p sforouter3 sinrouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p newyorkrouter1 sforouter2
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p hamburgsw1_1 hamburgsrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p singaporesw1_1 singaporesrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p newyorksw1_1 newyorksrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p londonsw1_1 londonsrv1
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p londonsw1_2 londonsrv2
root@cloudinabox:/opt/cloudsimulation/zones# dladm modify-simnet -p sanfranciscosw1_1 sanfranciscosrv1

Uff … on the networking side this is all. The active configuration should look something like that …

root@cloudinabox:/home/jmoekamp# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net1                phys      1500   unknown  --
net2                phys      1500   up       --
net0                phys      1500   unknown  --
london0             bridge    1500   up       londonsw1_1 londonsw1_2 londonsw1_255
hamburg0            bridge    1500   up       hamburgsw1_1 hamburgsw1_255
singapore0          bridge    1500   up       singaporesw1_1 singaporesw1_255
newyork0            bridge    1500   up       newyorksw1_1 newyorksw1_255
sanfrancisco0       bridge    1500   up       sanfranciscosw1_1 sanfranciscosw1_255
londonsw1_255       simnet    1500   up       londonrouter0
sanfranciscosw1_255 simnet    1500   up       sforouter0
newyorksw1_255      simnet    1500   up       newyorkrouter0
hamburgsw1_255      simnet    1500   up       hamburgrouter0
singaporesw1_255    simnet    1500   up       sinrouter0
londonsw1_1         simnet    1500   up       londonsrv1
londonsw1_2         simnet    1500   up       londonsrv2
hamburgsw1_1        simnet    1500   up       hamburgsrv1
singaporesw1_1      simnet    1500   up       singaporesrv1
newyorksw1_1        simnet    1500   up       newyorksrv1
sanfranciscosw1_1   simnet    1500   up       sanfranciscosrv1
londonrouter0       simnet    1500   up       londonsw1_255
londonrouter1       simnet    1500   up       hamburgrouter1
hamburgrouter0      simnet    1500   up       hamburgsw1_255
hamburgrouter1      simnet    1500   up       londonrouter1
hamburgrouter2      simnet    1500   up       sforouter1
hamburgrouter3      simnet    1500   up       sinrouter1
newyorkrouter0      simnet    1500   up       newyorksw1_255
newyorkrouter1      simnet    1500   up       sforouter2
sinrouter0          simnet    1500   up       singaporesw1_255
sinrouter1          simnet    1500   up       hamburgrouter3
sinrouter2          simnet    1500   up       sforouter3
sforouter0          simnet    1500   up       sanfranciscosw1_255
sforouter1          simnet    1500   up       hamburgrouter2
sforouter2          simnet    1500   up       newyorkrouter1
sforouter3          simnet    1500   up       sinrouter2
londonsrv1          simnet    1500   up       londonsw1_1
londonsrv2          simnet    1500   up       londonsw1_2
hamburgsrv1         simnet    1500   up       hamburgsw1_1
singaporesrv1       simnet    1500   up       singaporesw1_1
sanfranciscosrv1    simnet    1500   up       sanfranciscosw1_1
newyorksrv1         simnet    1500   up       newyorksw1_1

Zone Creation

Okay, now we have to create the zones.

mkdir -p /opt/cloudsimulation/zones
zfs create rpool/zones
zfs set mountpoint=/zones rpool/zones

We create a lot of controlfiles first. With this controlfiles we will feed zonecfg later on. I created the /opt/cloudsimulation/zones directory to hold them. Of course it’s useful to have an own ZFS filesystem in order to enable the zone creation process to simply copy the data needed by a zone by creating a clone of a filesystem.

Whois is wondering about the sfo and sin IATA shorthands that i’ve used instead of the long names in other “cities”. Quagga doesn’t seem to like interface names longer than 16 characters. Okay. Now we have to create all the zones. That’s easy. As i said, i will just feed the control files into zonecfg with the -f option.

root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z templateserver -f	/opt/cloudsimulation/zones/templateserver 
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z templaterouter -f /opt/cloudsimulation/zones/templaterouter
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z londonrouter	 -f /opt/cloudsimulation/zones/londonrouter 
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z singaporerouter -f /opt/cloudsimulation/zones/singaporerouter 
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z hamburgrouter	 -f /opt/cloudsimulation/zones/hamburgrouter 
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z sanfranciscorouter -f /opt/cloudsimulation/zones/sanfranciscorouter
root@cloudinabox:/opt/cloudsimulation/zones# zonecfg -z newyorkrouter -f /opt/cloudsimulation/zones/newyorkrouter

Okay, at first we install the template zone. We do a full install here. and that’s pretty much the only purpose … to have one installed baseline zone as providing the starting point for all other zones. This may take a while. Depending on your system you may opt for a coffee or two.

root@cloudinabox:/opt/cloudsimulation/zones# zoneadm -z templateserver install
A ZFS file system has been created for this zone.
Progress being logged to /var/log/zones/zoneadm.20111217T184237Z.templateserver.install
       Image: Preparing at /zones/templateserver/root

 Install Log: /system/volatile/install.4469/install_log
 AI Manifest: /tmp/manifest.xml.oBayTi
  SC Profile: /usr/share/auto_install/sc_profiles/enable_sci.xml
    Zonename: templateserver
Installation: Starting ...

              Creating IPS image
              Installing packages from:
                  solaris
                      origin:  http://pkg.oracle.com/solaris/release/
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                              167/167 32062/32062  175.8/175.8

PHASE                                        ACTIONS
Install Phase                            44313/44313 

PHASE                                          ITEMS
Package State Update Phase                   167/167 
Image State Update Phase                         2/2 
Installation: Succeeded

        Note: Man pages can be obtained by installing pkg:/system/manual

 done.

        Done: Installation completed in 1423,641 seconds.


  Next Steps: Boot the zone, then log into the zone console (zlogin -C)

              to complete the configuration process.

Log saved in non-global zone as /zones/templateserver/root/var/log/zones/zoneadm.20111217T184237Z.templateserver.install

We never boot this one, it’s just to ease the next steps. Okay, now we prepare the real zones. You don’t have to to the next steps, however they relief you from login into each zones and going to the same dialog windows. We will use a simple trick to circumvent the need to go through each sysconfig dialog in each router we will use a simple trick. You can create a xml file containing the necessary data and pass it to the cloning of the zone. Important: I want to make the resulting xml file as generic as possible, thus i won’t configure networking via this process, albeit this is possible. As it’s a CUI, i will guide you through this dialog with some pictures.

root@cloudinabox:/opt/cloudsimulation/zones# sysconfig create-profile -o template.xml

After leaving the last screen, you should yield a file with content similar to this:

root@cloudinabox:/opt/cloudsimulation/zones# cat template.xml 
&lt;!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1"&gt;
&lt;service_bundle type="profile" name="sysconfig"&gt;
  &lt;service version="1" type="service" name="system/config-user"&gt;
    &lt;instance enabled="true" name="default"&gt;
      &lt;property_group type="application" name="root_account"&gt;
        &lt;propval type="astring" name="login" value="root"/&gt;
        &lt;propval type="astring" name="password" value="$5$35worB11$/EeCnO5t2zOHhasRQeWeVyWuGLFFUFLQGmOhKPX82m2"/&gt;
        &lt;propval type="astring" name="type" value="role"/&gt;
      &lt;/property_group&gt;
      &lt;property_group type="application" name="user_account"&gt;
        &lt;propval type="astring" name="login" value="radmin"/&gt;
        &lt;propval type="astring" name="password" value="$5$XztZ799F$GVL48echivvJcPl.BRcVvnn3/M8Z7L6LhmyVPP04J/2"/&gt;
        &lt;propval type="astring" name="type" value="normal"/&gt;
        &lt;propval type="astring" name="description" value="routeradm"/&gt;
        &lt;propval type="count" name="gid" value="10"/&gt;
        &lt;propval type="astring" name="shell" value="/usr/bin/bash"/&gt;
        &lt;propval type="astring" name="roles" value="root"/&gt;
        &lt;propval type="astring" name="profiles" value="System Administrator"/&gt;
        &lt;propval type="astring" name="sudoers" value="ALL=(ALL) ALL"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="system/timezone"&gt;
    &lt;instance enabled="true" name="default"&gt;
      &lt;property_group type="application" name="timezone"&gt;
        &lt;propval type="astring" name="localtime" value="UTC"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="system/environment"&gt;
    &lt;instance enabled="true" name="init"&gt;
      &lt;property_group type="application" name="environment"&gt;
        &lt;propval type="astring" name="LANG" value="C"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="system/identity"&gt;
    &lt;instance enabled="true" name="node"&gt;
      &lt;property_group type="application" name="config"&gt;
        &lt;propval type="astring" name="nodename" value="jamphfhn"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="system/keymap"&gt;
    &lt;instance enabled="true" name="default"&gt;
      &lt;property_group type="system" name="keymap"&gt;
        &lt;propval type="astring" name="layout" value="German"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="system/console-login"&gt;
    &lt;instance enabled="true" name="default"&gt;
      &lt;property_group type="application" name="ttymon"&gt;
        &lt;propval type="astring" name="terminal_type" value="sun-color"/&gt;
      &lt;/property_group&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
  &lt;service version="1" type="service" name="network/physical"&gt;
    &lt;instance enabled="true" name="default"&gt;
      &lt;property_group type="application" name="netcfg"/&gt;
    &lt;/instance&gt;
  &lt;/service&gt;
&lt;/service_bundle&gt;

Before you ask, the password for radmin and root is n0mn0mn0m. And the jamphfhn just stands for “just a meaningless placeholder for hostname”. Okay, i will create another template zone. This is because a routing zone will have some special properties that a zone acting as a server doesn’t need and i don’t want such properties in the server zones. At first i just take the template.xml script and substitute the hostname. I could simply do it via vi, but for a tutorial a simple shell line is more efficient.

root@cloudinabox:/opt/cloudsimulation/zones# cat template.xml | sed s/jamphfhn/templaterouter/ > templaterouter.xml

I use the newly created file as an input for the zone clone command.

root@cloudinabox:/opt/cloudsimulation/zones# zoneadm -z templaterouter clone -c /opt/cloudsimulation/zones/templaterouter.xml templateserver
A ZFS file system has been created for this zone.
Progress being logged to /var/log/zones/zoneadm.20111217T193101Z.templaterouter.clone
Log saved in non-global zone as /zones/templaterouter/root/var/log/zones/zoneadm.20111217T193101Z.templaterouter.clone

As the system just creates a zfs clone the command should return after a small period of time. Now we can log into the console of the zone with zlogin.

root@cloudinabox:/opt/cloudsimulation/zones# zoneadm -z templaterouter boot
root@cloudinabox:/opt/cloudsimulation/zones# zlogin -C templaterouter                
[Connected to zone 'templaterouter' console]

Hostname: unknown
Hostname: templaterouter

templaterouter console login: radmin
Password: 
Oracle Corporation      SunOS 5.11      11.0    November 2011
radmin@templaterouter:~$

radmin@templaterouter:~$ sudo bash
Password: 
Dec 17 19:36:33 templaterouter sudo:   radmin : TTY=console ; PWD=/home/radmin ; USER=root ; COMMAND=/usr/bin/bash
root@templaterouter:/home/radmin#

I wrote earlier, that the template for the router contains some additional stuff. At first i need a telnet client. It will get obvious why i need it later on:

root@templaterouter:/home/radmin# pkg install pkg://solaris/network/telnet
           Packages to install:  1
       Create boot environment: No
Create backup boot environment: No

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         8/8      0.1/0.1

PHASE                                        ACTIONS
Install Phase                                  22/22 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2

Okay, now let’s install quagga. Quagga is a suite of daemons to implement dynamic routing protocols:

root@templaterouter:/home/radmin# pkg install quagga                      
           Packages to install:  1
       Create boot environment: No
Create backup boot environment: No
            Services to change:  3

DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1       89/89      2.7/2.7

PHASE                                        ACTIONS
Install Phase                                132/132 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2 
Loading smf(5) service descriptions: 2/2

Okay, now we have to configure some basics that are equal to all the router in the network.
At first we activate forwarding. With this activation, you enable the operating system to accept packets on one interface

root@templaterouter:/home/radmin# routeadm -e ipv4-forwarding

ipv4-routing tells the system to startup routing protocol daemons. When you have a default router configured it’s disabled, when there isn’t one this setting is enabled per default.

root@templaterouter:/home/radmin# routeadm -e ipv4-routing

Okay, now we have to do some quagga configurations. I want to use quagga with OSPF, so there are two important services for me. Zebra and ospf. Zebra is the layer, that the quagga suite used to interact with the system. Why is it called Zebra? I assume it’s history, the old GNU routing protocol daemon suite was called zebra, quagga is the follow-on project as zebra is now a defunct software development project. What do we configure here. Both daemons offer a command line for interfaction with the daemon. We configure both just to react from 127.0.0.1 (aka localhost). The zebra daemon has it’s console on port 2602, the ospf daemon listens on port 2601. And this both ports are the reason we need telnet on our routers. You access the consoles via telnet.

root@templaterouter:/home/radmin# routeadm -m zebra:quagga vty_port="2602"
root@templaterouter:/home/radmin# routeadm -m ospf:quagga vty_port="2601"
root@templaterouter:/home/radmin# routeadm -m zebra:quagga vty_address="127.0.0.1"
root@templaterouter:/home/radmin# routeadm -m ospf:quagga vty_address="127.0.0.1"

With this command we tell Solaris to use ospf as the routing protocol for ipv4 purposes.

root@templaterouter:/home/radmin# routeadm -s routing-svcs=ospf:quagga -e ipv4-routing

Now we have to activate the new setting

root@templaterouter:/home/radmin# routeadm -u

You should now get some weired SMF error messages that some services couldn’t start up. that’s normal because there are no configuration files available for the quagga suite. Don’t think about it, just shut the zone down now.

root@cloudinabox:/home/jmoekamp# zoneadm -z templaterouter halt

Okay, now we have derived our template for the router zones from the generic template for zones. We use this template for installing all the router zones now. Okay, i just wrote about quagga config files. I want to prepare them now in order to be able just to copy them into the zones before starting them up and thus to circumvent the error messages. We need a lot of them.