Basic Puppet installation with Solaris 11.2 beta

At the recent announcement we talked a lot about the Puppet integration. But how do you set it up? I want to show this in this blog entry. However this example i’m using is even useful in practice. Due to the extremely low overhead of zones i’m frequently seeing really large numbers of zones on a single system. Changing /etc/hosts or changing an SMF service property on 3 systems is not that hard. Doing it on a system with 500 zones is … let say it diplomatic … a job you give to someone you want to punish. Puppet can help in this case making of managing the configuration and to ease the distribution. You describe the changes you want to make in a file or set of file called manifest in the Puppet world and then roll them out to your servers, no matter if they are virtual or physical. A warning at first: Puppet is a really,really vast topic. This article is really basic and it doesn’t goes more than just even toe’s deep into the possibilities and capabilities of Puppet. It doesn’t try to explain Puppet … just how you get it up and running and do basic tests. There are many good books on Puppet. Please read one of them, and the concepts and the example will get much clearer immediately. To show this, i have a relatively simple setup. A global zone and three non-global zones. In this example i will set up the global zone as a Puppet master and will configure three non-global zone with the Puppet agent, the component that gathers information from the Puppet master in order to do something on the server running the agent. The Puppet master is called master, the agents are called agent1, agent2 and agent3. This tutorial assumes that you have already set up the zones and that they have working network connection.

Preparation

Okay, at first it’s really important that you have a working resolution of the hostnames. The resolution is key to get Puppet working. This is what i did in the global zone and in the non-global zones. The /etc/hosts of all zones (either global or non-global) is now like this.

root@agent1:~# cat /etc/hosts
#
# Copyright 2009 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# Internet host table
#
::1             localhost
127.0.0.1       localhost loghost
192.168.1.16    master.puppet.c0t0d0s0.org master kusanagi
192.168.1.230   agent1.puppet.c0t0d0s0.org agent1
192.168.1.231   agent2.puppet.c0t0d0s0.org agent2
192.168.1.232   agent3.puppet.c0t0d0s0.org agent3

Further i’ve set the hostname on all servers.

root@master:~# hostname master.puppet.c0t0d0s0.org
root@agent1:~# hostname agent1.puppet.c0t0d0s0.org
root@agent2:~# hostname agent2.puppet.c0t0d0s0.org
root@agent3:~# hostname agent3.puppet.c0t0d0s0.org

Installing and configuring Puppet

Okay, now install Puppet in all zones. This is quite easy as since 11.2 beta , puppet is in the standard repository . You just have to execute pkg install puppet Ensure that the package has been installed on all zones you want to work with. When you have installed the puppet package on all zones, login as root on the master At first bring up the master.

root@master:/etc/puppet/ssl# svccfg -s puppet:master setprop config/server = master.puppet.c0t0d0s0.org
root@master:/etc/puppet/ssl#  svccfg -s puppet:master refresh
root@master:/etc/puppet/ssl# svcadm enable puppet:master

Afterwards you should see the puppet master service running on your system.

root@master:/etc/puppet/manifests# svcs puppet
STATE          STIME    FMRI
disabled       Mai_19   svc:/application/puppet:agent
online         Mai_19   svc:/application/puppet:master

Now we have prepare the agent systems. At first log into agent1 system

root@agent1:# puppet agent --test --server master.puppet.c0t0d0s0.org
Info: Creating a new SSL key for agent1.puppet.c0t0d0s0.org
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for agent1.puppet.c0t0d0s0.org
Info: Certificate Request fingerprint (SHA256): 14:20:1E:C8:D8:78:1D:DF:9C:92:75:F2:72:C6:61:61:AC:56:82:06:FC:A4:6D:5E:DA:5F:7E:12:80:5B:90:A9
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled
root@agent1:~#

The connection between the agent and the master is protected by SSL. At the first execution the agent creates a key and and certificate request.
You have to repeat the steps in the zones agent2 and on agent3 as well. At first on agent2

root@agent2:~# puppet agent --test --server master.puppet.c0t0d0s0.org
Info: Creating a new SSL key for agent2.puppet.c0t0d0s0.org
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for agent2.puppet.c0t0d0s0.org
Info: Certificate Request fingerprint (SHA256): 76:A7:23:A7:4D:41:66:DD:71:B0:4E:AA:62:EC:0B:DB:61:59:BE:56:43:15:E7:BA:C3:CB:AF:D3:98:D3:30:18
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled
root@agent2:~#

Now on agent3:

root@agent3:~# puppet agent --test --server master.puppet.c0t0d0s0.org
Info: Creating a new SSL key for agent3.puppet.c0t0d0s0.org
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for agent3.puppet.c0t0d0s0.org
Info: Certificate Request fingerprint (SHA256): C4:96:A1:B1:D5:F3:EB:AA:2C:B8:9B:9E:83:C5:F8:64:2E:F7:D8:50:49:AA:0A:E8:73:BE:E0:53:D8:12:95:F7
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled
root@agent3:~#

Now we have to sign this certificates on the master. Please go back to your shell of master

root@master:/etc/puppet/ssl# puppet cert list
  "agent1.puppet.c0t0d0s0.org" (SHA256) 14:20:1E:C8:D8:78:1D:DF:9C:92:75:F2:72:C6:61:61:AC:56:82:06:FC:A4:6D:5E:DA:5F:7E:12:80:5B:90:A9
  "agent2.puppet.c0t0d0s0.org" (SHA256) 76:A7:23:A7:4D:41:66:DD:71:B0:4E:AA:62:EC:0B:DB:61:59:BE:56:43:15:E7:BA:C3:CB:AF:D3:98:D3:30:18
  "agent3.puppet.c0t0d0s0.org" (SHA256) C4:96:A1:B1:D5:F3:EB:AA:2C:B8:9B:9E:83:C5:F8:64:2E:F7:D8:50:49:AA:0A:E8:73:BE:E0:53:D8:12:95:F7

So we have three requests to sign. Let’s sign them.

root@master:/etc/puppet/ssl# puppet cert sign agent1.puppet.c0t0d0s0.org
Notice: Signed certificate request for agent1.puppet.c0t0d0s0.org
Notice: Removing file Puppet::SSL::CertificateRequest agent1.puppet.c0t0d0s0.org at '/etc/puppet/ssl/ca/requests/
root@master:/etc/puppet/ssl# puppet cert sign agent2.puppet.c0t0d0s0.org
Notice: Signed certificate request for agent2.puppet.c0t0d0s0.org
Notice: Removing file Puppet::SSL::CertificateRequest agent2.puppet.c0t0d0s0.org at '/etc/puppet/ssl/ca/requests/agent2.puppet.c0t0d0s0.org.pem'
root@master:/etc/puppet/ssl# puppet cert sign agent3.puppet.c0t0d0s0.org
Notice: Signed certificate request for agent3.puppet.c0t0d0s0.org
Notice: Removing file Puppet::SSL::CertificateRequest agent3.puppet.c0t0d0s0.org at '/etc/puppet/ssl/ca/requests/agent3.puppet.c0t0d0s0.org.pem'

Okay, let’s check if there is something left:

root@master:/etc/puppet/ssl# puppet cert list

Now you can check on each agent system the connectivity. You will see quite a long output

root@agent3:~# puppet agent --test --server=master.puppet.c0t0d0s0.org
Info: Caching certificate for agent3.puppet.c0t0d0s0.org
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for agent3.puppet.c0t0d0s0.org
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
[...]
Info: Caching catalog for agent3.puppet.c0t0d0s0.org
Info: Applying configuration version '1400520716'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.09 seconds
root@agent3:~# svccfg -s puppet:agent setprop config/server = master.puppet.c0t0d0s0.org
root@agent3:~#  svccfg -s puppet:agent refresh
root@agent3:~# svcadm enable puppet:agent

Do the same in the other zones. At first agent2

root@agent2:~# puppet agent --test --server master.puppet.c0t0d0s0.org
Info: Caching certificate for agent2.puppet.c0t0d0s0.org
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for agent2.puppet.c0t0d0s0.org
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
[...]
Info: Caching catalog for agent2.puppet.c0t0d0s0.org
Info: Applying configuration version '1400520716'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.09 seconds
root@agent2:~# svccfg -s puppet:agent setprop config/server = master.puppet.c0t0d0s0.org
root@agent2:~# svccfg -s puppet:agent refresh
root@agent2:~# svcadm enable puppet:agent

Afterwards on agent1:

root@agent1:~# puppet agent --test --server master.puppet.c0t0d0s0.org
Info: Caching certificate for agent1.puppet.c0t0d0s0.org
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for agent1.puppet.c0t0d0s0.org
Info: Retrieving plugin
Notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
[...]
Info: Caching catalog for agent1.puppet.c0t0d0s0.org
Info: Applying configuration version '1400520716'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.09 seconds
root@agent1:~# svccfg -s puppet:agent setprop config/server = master.puppet.c0t0d0s0.org
root@agent1:~# svccfg -s puppet:agent refresh
root@agent1:~# svcadm enable puppet:agent

Do something slightly useful with the installation

Okay, now we are ready to try this our puppet installation. I won’t start with the basic help world feature, but something slightly more useful. However we have to some configuration first. I won’t explain much, as when i start to explain everything about it, this article will be veeeeery long. Basically the following stuff is creating a Puppet module called etchosts which delivers an /etc/hosts file (thus the name) to the system. The files is at /etc/puppet/modules/etchosts/files, which can be accessed by Puppet under the URL puppet:///modules/etchosts/hosts. The module code is located in a file init.pp (a convention in Puppet) inside the modules manifest directory at /etc/puppet/modules/etchosts/manifests directory. Afterwards i’m creating nodes.pp file which essentially says “By default - when there is no node definition that matches - execute the function etchosts imported from the module etchosts and so the /etc/puppet/modules/etchosts/files/hosts is moved to </code>/etc/hosts</code> if necessary (something has changed).” This file is invoked by the site.pp file.
At first we need some additional structure in some filesystems

root@master:~# mkdir /etc/puppet/modules/etchosts
root@master:~# mkdir /etc/puppet/modules/etchosts/files
root@master:~# mkdir /etc/puppet/modules/etchosts/manifests
root@master:~# cp /etc/hosts /etc/puppet/modules/etchosts/files/hosts

Now we will give some life to the module /etc/hosts

root@master:~# cat << EOT > /etc/puppet/modules/etchosts/manifests/init.pp
> class etchosts {
>         file { "/etc/hosts":
>                 source => 'puppet:///modules/etchosts/hosts',
>         }
> }
> EOT

Afterwards we creating a file just including the nodes definition file:

root@master:~# cat << EOT > /etc/puppet/manifests/sites.pp
> import 'nodes.pp'
> EOT

At last we define the behaviour for the “default” node:

root@master:~# cat << EOT > /etc/puppet/manifests/nodes.pp
import 'etchosts'

node 'default' {
 include etchosts
}

In order to have something to play i just add a line to the /etc/puppet/modules/etchosts/files/hosts file.

echo "# Do not edit - modified by puppet" >> /etc/puppet/modules/etchosts/files/hosts

Okay, now log into one of the zones with an agent and check the current /etc/host

jmoekamp@agent2:~$ cat /etc/hosts
[...]
192.168.1.232   agent3.puppet.c0t0d0s0.org agent3
jmoekamp@agent2:~$

At the moment there there isn’t the additional line in the file. You can now wait for 1800 seconds at maximum (because the agent checks every 1800 seconds if there is something to do by default) or you can force the check. Let’s force it.

root@agent2:~# puppet agent --test
Info: Retrieving plugin
Info: Caching catalog for agent2.puppet.c0t0d0s0.org
Info: Applying configuration version '1400529373'
Notice: /Stage[main]/Etchosts/File[/etc/hosts]/content:
--- /etc/hosts  Tue May 20 04:33:32 2014
+++ /tmp/puppet-file20140520-8490-tfgwja        Tue May 20 19:52:02 2014
@@ -11,2 +11,3 @@
 192.168.1.231  agent2.puppet.c0t0d0s0.org agent2
 192.168.1.232  agent3.puppet.c0t0d0s0.org agent3
+# Do not edit - modified by puppet

Info: /Stage[main]/Etchosts/File[/etc/hosts]: Filebucketed /etc/hosts to puppet with sum 38f6c964aab77edb2ff938094f13e2d0
Notice: /Stage[main]/Etchosts/File[/etc/hosts]/content: content changed '{md5}38f6c964aab77edb2ff938094f13e2d0' to '{md5}49b07e8c62ed409a01216bf9a35ae7ae'
Notice: Finished catalog run in 0.60 seconds

Now let’s check the file again … et voila … you find the line you have added into the /etc/puppet/modules/etchosts/files/hosts in the /etc/hosts of you file.

root@agent2:~# cat /etc/hosts
[...]
192.168.1.230   agent1.puppet.c0t0d0s0.org agent1
192.168.1.231   agent2.puppet.c0t0d0s0.org agent2
192.168.1.232   agent3.puppet.c0t0d0s0.org agent3
# Do not edit - modified by puppet

When go for a coffee and wait the mentioned 1800 seconds you will find the change in all your zones.

Conclusion

One of the points of the puppet integration into Solaris 11.2 was the development of a lot of solaris specific modules for managing boot environments or configuring VNICs. This example doesn’t touch those topics at all. But now you have a foundation for follow on articles going into details about Solaris specific modules.