New Solaris features: How t...

Some theory

The capability to send notifications in case of service transitions is just a part of the capabilities of this feature. At first the feature can't only send mails, you can send SNMP traps as well. Additionally the feature isn't just able to send mails or SNMP traps in case of service state transitions, you can also configure the system to send mail in case of problems detected by the Fault Management Architecture at the hardware. But both is stuff for different articles.

In it's lifetime a service goes trough a number of states. From being disabled up to up-and-running. You will find a description of the states in my tutorial about SMF. The description of the states is located almost at at the end of part 2.

The notification frameworks adds mechanisms to enable SMF to tell the outside world, when a services transitions away from one state or into a state. So number of such transitions are monitored by the notification framework:

to-uninitialized from-uninitialized to-maintenance from-maintenance to-offline from-offline

to-disabled from-disabled to-online from-online to-degraded from-degraded

Installation

The demon needed to send this mail isn't in the default installation. So you have to install the package pkg install service/fault-management/smtp-notify in addition to your basic system:

root@solaris:/# pkg install service/fault-management/smtp-notify
               Packages to install:     1
           Create boot environment:    No
               Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                  1/1         4/4      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                  24/24 

PHASE                                          ITEMS
Package State Update Phase                       1/1 
Image State Update Phase                         2/2

Now you can enable the daemon by enabling the service controlling it.

root@solaris:~# svcadm enable svc:/system/fm/smtp-notify:default
root@solaris:/#

Afterwards you should see an the smtp-notify via ps

root@solaris:/var/adm# ps -ef | grep "smtp-notify"
noaccess 1668 1 0 05:10:38 ? 0:00 /usr/lib/fm/notify/smtp-notify

Now let's configure the notifications:

root@solaris:/# svccfg -s svc:/system/svc/global:default setnotify -g from-online  mailto:root@localhost

I will use the Apache service to demonstrate the functionality, because it's extremely easy to send the service into maintenance mode.

root@solaris:/# pkg install apache-22
root@solaris:/# svcadm enable apache22

To force it into maintainance mode i just added a SNAFU at the beginning of the file /etc/apache2/2.2/httpd.conf and restarted the service with

svcadm restart apache-22

Let's check the situation. The apache-22 service has transitioned into the maintainance mode:

root@solaris:/# svcs -a  | grep "apache"
maintenance     6:12:58 svc:/network/http:apache22

When you check your mail you will find a mail:

root@solaris:/# mail
From noaccess@solaris.local Thu Nov 18 06:12:59 2010
Date: Thu, 18 Nov 2010 06:12:59 +0100 (CET)
From: No Access User
Message-Id: <201011180512.oAI5CxVV001278@solaris.local>
Subject: Fault Management Event: solaris:SMF-8000-YX
To: root@solaris.local
Content-Length: 668

SUNW-MSG-ID: SMF-8000-YX, TYPE: defect, VER: 1, SEVERITY: major
EVENT-TIME: Thu Nov 18 06:12:58 CET 2010
PLATFORM: VirtualBox, CSN: 0, HOSTNAME: solaris
SOURCE: software-diagnosis, REV: 0.1
EVENT-ID: 345ed233-8d58-ef72-d050-c552d0c78670
DESC: A service failed - a start, stop or refresh method failed.
Refer to http://sun.com/msg/SMF-8000-YX for more information.
AUTO-RESPONSE: The service has been placed into the maintenance state.
IMPACT: svc:/network/http:apache22 is unavailable.
REC-ACTION: Run 'svcs -xv svc:/network/http:apache22' to determine the generic reason why the service failed, the location of any logfiles, and a list of other services impacted.


? q

Managing notifications

When you want to know what notification are already configured for a service, you can use the listnotify subcommand of svccfg.

root@solaris:/# svccfg -s svc:/system/svc/global:default listnotify

    Event: from-online (source: svc:/system/svc/global:default)
        Notification Type: smtp
            Active: true
            to: root@localhost

When you want to stop notifications, you can use the delnotify subcommand.

root@solaris:/home/jmoekamp# svccfg -s svc:/system/svc/global:default delnotify -g all 
root@solaris:/# svccfg -s svc:/system/svc/global:default listnotify
root@solaris:/# 

Notifications for a single service

When you don't want to activate notifications for all services, you can just activate it service by service:

root@solaris:/# svccfg -s svc:/network/http:apache22 setnotify  from-online mailto:root@localhost
root@solaris:/#

Please note, that the -g option is missing. When you use the -g the configuration done in svc:/system/svc/global:default no matter what service you've declare later on in the command line. Let's check the configuration:

root@solaris:/home/jmoekamp# svccfg -s svc:/network/http:apache22  listnotify
    Event: from-online (source: svc:/network/http:apache22)
        Notification Type: smtp
            Active: true
            to: root@localhost

root@solaris:/home/jmoekamp#

It's important to know, that a service has as well the global as the locally set notifications. Let's assume you've set from-maintenance at svc:/system/svc/global and you've set from-online at svc:/network/http:apache22. Then the service svc:/network/http:apache22 will have both notifications:

root@solaris:/# svccfg -s svc:/network/http:apache22  listnotify
    Event: from-maintenance (source: svc:/system/svc/global:default)
        Notification Type: smtp
            Active: true
            to: root@localhost

    Event: from-online (source: svc:/network/http:apache22)
        Notification Type: smtp
            Active: true
            to: root@localhost

root@solaris:/home/jmoekamp#

When you look at the source for each event, you can see, what's the source of this notification setting.

Conclusion

Due to the smtp-notify capability of the Service Management facility it's really easy to give all services managed by SMF the capability to send some kind of notification to the admin.

Want to learn more?

man pages
docs.sun.com - manpage smtp-notify - the daemon that sends the mails
docs.sun.com - manpage svccfg - for the setnotify,delnotify,listnotify subcommands
Misc.
PSARC 2009/619 - PSARC case for email notifications in SMF