Indication Policy

The Indication Policy is used to define two major types of Policies:

Indication Policy:	An Indication policy is used for processing incoming messages. This policy doesn't use the "Nagin Trigger/LogFileMonitor Details" field. Examples: SNMP Trap policy, boomindi
Hybrid Policy:	A Hybrid Policy is used for processing text output from executables or scripts. A Hybrid Policy can trigger java based tasks or Nagios® like plugins (NAGINs) for every specified interval. Actually any executable or script that produces an output can be used as trigger. The output will be processed by conditions specified in the Policy.

The main difference of these two types is that Hybrid policies are excluded from the global message filtering and only used for processing the output of the defined trigger.

An Indication Policy becomes a Hybrid Policy when the "Type" field in the "Nagin trigger" section is not empty.

For more information about Hybrid Policies see Chapter Hybrid Indication Policy.

Field Description

Policy Version:	This shows the current version of the Policy. The field will be automatically updated when changes are saved on the boom Server. The drop-down menu at the right side displays all previous versions of this Policy. The Compare button allows you to review the history of changes that have been made to the the Policy.
Compare:	With the compare function you can compare the current Policy with an older version. More details about the compare function see the chapter Policy Management
Plugin Name:	The Plugin Name is a non-editable field. The name comes from 3d party plug-ins.
Deactivation Schedule:
Indication Name:	This is a unique name of the Policy (unique across all kind of policies).
Set Application:	Default application that will be assigned to the generated indication if no condition specific application ist set.
Set Group:	Default indication group that will be assigned to the generated indication if no condition specific indication group ist set.
Description:	A description of the Policy
Global Variable:	The policy is only activated if the specified Global Variable matches the specified value. For more information on setting Global variables see chapter "Supported Variables and Functions".
Nagin Trigger:	Mandatory section for the Hybrid Policies, where the following parameters have to be specified: trigger type, interval and trigger parameters. For more information see Chapter Logfile Monitors.

The Condition section contains a list of filtering rules for processing the incoming text messages

Field Description

Condition List:
Description:	A condition description.
ID:	An auto generated ID of the condition.
Advice:	An advice message that will be shown in the Indication Browser
Instruction URL:	Define the URL of an external knowledge base e.g. http://myInstruction.server.net. Agent and indication variables can be used in the Instruction URL.
Availability Metric:	Defines if the Indication that matched with this condition has an impact on the availability of the monitored system.
KPI Metric:	This flag defines if the Indication must be treated as a Key Performance Indicator.
Detect Duplicates:	This flag notifies the boom Server to recognize duplicates. By receiving duplicate Indications the boom Server increases the "duplicate count" and updates the "last duplicate time" of the first received Indication. No new entry will be created in the database. The following message attributes are used for recognizing duplicates: AGENT_HOST,HOST,APPLICATION,GROUP,OBJECT,SEVERITY,INDICATION KEY, MESSAGE TEXT
De-Duplicate KeyOnly:	This flag notifies the boom Server to recognize duplicates based on the defined Indication Key. By receiving duplicate Indications the boom Server increases the "duplicate count" and updates the "last duplicate time" of the first received Indication. No new entry will be created in the database. The Indication Key attributes are used for recognizing duplicates hence an Indication Key has to be defined.
Add As closed:	This will add the message already as a 'closed message' to the Indication Browser.
Custom Attributes:	User defined attributes which can be either just a text or a variable value. CA's are forwarded to the boom server as specified, they aren't subject to match on the agent. Up to 15 Custom Attributes can be defined in the Custom Attributes Dialog. Names of the Custom Attribute should not contain '=' sign. Values can use supported variables.
Indication Key:	A key template that is used for correlation. The Indication Key contains a list of supported variables which will be resolved during the processing of an incoming value. The Result string is used for correlation with following Indication for auto acknowledgment and duplicate suppression.
Close Mask:	Pattern that is used to search and close previously submitted Indications.
Filter Section:	Defines filters for Application, Indication Group, Object, Host and Severity. Important: Pre-Filtering for SNMP Traps: Application has to be set to SNMPTrapd Indication Group has to be set to SNMP Please note: These fields support only limited subset of the pattern notation. <*> matches all abc matches "abc" string abc | dce | aaa     matches one of the {"abc","dce", "aaa"}
Overwrite Attributes:	This section allows to overwrite incoming values with user specified values.
Set Host:	Overwrites the hostname of the submitted Monitor value. Default value is an Agent hostname. All supported and optional variables can be used in this field as well. In case the monitor submits the hostname as optional variable you can use this variable in the "Set Host" field.
Set Application:	Overwrites the application name of the incoming message.
Set Object:	Overwrites the object of the incoming message.
Set Group:	Overwrites the group of the incoming message.
Search Text:	Main text filter and parser template.
Match Variables:	Variable: <$n> incoming variable (n = number of variable) Simplified Pattern: BOOM simplified pattern definition for matching the incoming variable (see also chapter "patterns") These variables are subject to match on the agent, they aren't forwarded to the server. They can be used for checking SNMP Trap variables or any other variables e.g. boomindi optional variables.
Set Text:	Define the message text that should appear in the browser.
Silence Time:	Suppression time interval since the first match.
Silence Count:	Suppress number of messages since the first Indication has been sent.
Negation:	A result of matching will be inverted at the end. This flag gives a possibility to create exclude condition. NEGATION=FALSE (default) the matching result won't be inverted. NEGATION=TRUE the matching result will be inverted.
Condition Type:	STOP | SEND STOP – means that a incoming message that matched with the current condition must be ignored and all following conditions must not be checked. This helps to increase the performance of the boom Agent by using a STOP condition at the beginning of the Policy. As result the Correlation engine doesn't need to check all following condition and drops all text messages that have no interest. SEND value indicates that the matched text must be send as an Indication. The rest of the conditions inside the current Policy will be automatically ignored.
Auto Action:	Specifies automatic action that must be executed on the incoming Indication. An automatic action will be triggered from the boom Server. The result of this action will be stored in the Annotation Field of the Indication. All indication variables are supported and will be resolved.
AA Host:	The target Agent hostname for the remote automatic action. If nothing is specified, the action will be triggered on the boom Agent where the Indication was created. All indication variables are supported and will be resolved.
AA Timeout:	Timeout in seconds for the remote automatic action.
Operator Action:	Recommended operator action.

Mandatory Nagin section for the Hybrid Policies, where you have to specify an interval, some trigger parameters and one of the trigger type: JAVA, NAGIN, Logfile Monitor, Logfile Transaction Monitor.

Note: This section is NOT used for SNMP Trap policies.

For more information see Chapter Logfile Monitors.

JAVA:
NAGIN:
Logfile Monitor:
Logfile Transaction Monitor:

Supported Variables

For setting up an Indication Policy for SNMP traps you have to consider the following points:

The SNMP Trap assignment group "SNMP-TrapReceiver" or at least the "SNMP package" and the "SNMPTrapd_Trigger" policy must be assigned to the specified agent to enable trap processing.

Do not use a trigger for setting up an indication policy for SNMP traps. That means that the section "Nagin Trigger/LogFileMonitor Details" is not used for a trap indication policy.

All conditions must have a filter for:
Application=SNMPTrapd
Indication Group=SNMP

The object field is used to filter the trap OID: OBJECT=<Trap OID>

<$1>...<$n> are the trap variables.

The "Search Text" field can be used for plain pattern matches on concatenated TrapOID and variables.

The "Match Variables" field can be used to construct text from variables and define match conditions.

Hint: Use a "stop condition" for non matching traps (e.g. different OID) as first condition. This stops further processing of the incoming trap if the trap doesn't match this condition. In this case the NEGATION flag has to be set to TRUE and Condition Type to STOP.

An indication policy with trigger is used for fetching strings with SNMP get.

The Monitor Type is "JAVA", the Monitor Name is "com.blixx.boom.snmp.SNMPWalkPrinter".

SNMPWalkMonitor supports following parameters:
-h <hostIP>
-o <oid of target field>
-c <community>
-v <SNMP version[1,2]>
-p <port>
-t <timeout in seconds>
-r <retries>

Object=OID
Text=SNMP value as string

The Logfile Monitor has flexible possibilities to specify the name of the monitored log file. It supports file masks for finding rotating log files.

For setting up an Indication Policy for logfile monitoring (Monitor Type=Logfile Monitor) you have to consider the following points:

Set the "Monitor Type" in the "Nagin Trigger/LogFile Monitor details" section to "Logfile Monitor".

Interval: The interval specifies how often logfile is checked.

Logfile Path: The logfile path contains the path of the monitored file or it can contain a mask for the filename. The latest file matching the mask will be monitored.

e.g. /var/webapps/app1/error_*.log
'*' (star) symbol matches any char sequence within the name. Among all matching files, the file with the latest modification time will be taken.

Split Records: Split Records allows multi-line processing, because many of the log files contain multi-line entries. It is necessary to have here a Java Pattern that is matching the start of the message. Usually it can be a fixed prefix containing the date, severity, log level, etc. If a logfile has no multi-line entries a simple ".*" (match all) pattern can be used.

General Pattern: A Java Pattern as a pre-process filter. This filter allows to reduce the number of messages that must be processed by an Agent. Pattern ".*" matches all lines - that means all entries will be processed.

FROM_START: Indicates that the logfile has to be monitored from the begin on every scheduled interval

FROM_LAST: The logfile will be scanned from the last processed point.

Executable Call: Allows you to specify a trigger that needs to be executed before parsing the logfile. This can be used together with FROM_START parameter to process logfiles re-generated by trigger on every polling interval. This parameter is optional.

The following optional parameters are coming with every indication:

<$LOGPATH> = current log file full path
<$LOGFILE> = current log file name

The Multipath Logfile Monitors are extensions of the Logfile Monitors. The main difference is that the file/path mask is extended to support not only file masks but also path masks. It monitors log file contents for entries of interest. It allows flexible configuration of file rotation policy, specifications of the logfile format and supports pre-processing filter.

The monitor can be configured to monitor a single particular file or a list of logfiles of similar format, or even load a batch list of files to process, provided by an external command.

For setting up an Indication Policy for multipath logfile monitoring (Monitor Type=(MP) Logfile Monitor) you have to consider the following points:

Set the "Monitor Type" in the "Nagin Trigger/LogFile Monitor details" section to "(MP) Logfile Monitor".

Logfile Path: The logfile path contains the path of the monitored file.

The path mask syntax is specialized for finding rotating log files. It supports file and path masks and consists of the following rules:

• '*' (star) symbol matches any char sequence within path element name. Among all matching files, the file with latest modification time should be taken. This is useful to monitor rotating logfiles, so if logging has switched to another file, the monitor will also pick the newest file (after finishing the previous one).

• Several '*' (star) symbols in different path elements (i.e. on different file tree levels) still comprise the same single group for selecting the newest file.

• '**' (double star) symbol matches any char sequence within path element name. All matching files should be taken, regardless of modification time.

• The path element consisting of exactly double star (like '/**/') specifies recursive scan and matches arbitrary path depth, including zero.

• double star overrides single star in the same path element, that is all matching files will be taken.

• path starting from wildcard is considered relative to current dir, unless followed by colon on MS Windows. This is reserved as a special notation for selecting all disk drives on MS Windows: *: (is the same as **:).

• All slashes and backslashes are treated as file (path element) separators, interim double slashes are reduced to single slash. On MS Windows, network paths starting with double (back-)slash are also recognized.

• Several independent patterns can be specified, separated by the ‘|’ (pipe) symbol.

Please see the above section "Logfile Monitors" for an explanation of all other parameters.

The following optional parameters are coming with every indication:

<$PATTERN> = active pattern
<$LOGPATH> = current log file full path
<$LOGFILE> = current log file name

Examples:

	1) We have the following directory Structure:
		/web/logs/error1.log
		/web/logs/error2.log
		/web/logs/error3.log
		/web/logs/error4.log

	   Path/File Mask:
		/web/logs/error*.log

	   Results:
		Assuming that the most recent file is "/web/logs/error4.log" this file will be
		monitored.

	   Explanation:
		picks a newest (most lately modified) file whose name starts with ‘error’ and 
		ends with ‘.log’ in the directory /web/logs. This is a typical configuration to 
		monitor a rotated log file, e.g. Apache’s logs.


	2) We have the following directory Structure:
		/app/services/errors.log
		/app/services/warnings.log
		/app/services/information.log
		/app/services/debug.log
		/app/services/exportedfile.txt

	   Path/File Mask:
		/app/services/**.log

	   Results:
		/app/services/errors.log
		/app/services/warnings.log
		/app/services/information.log
		/app/services/debug.log

	   Explanation:
		The Path/File mask selects all files with extension .log in directory /app/services.
		Unlike the previous example, this pattern assumes no log rotation, and simply selects 
		a bunch of existing files for batch processing.


	3) We have the following directory Structure:
		/var/log/apache/instance1/access_log1
		/var/log/apache/instance1/access_log2
		/var/log/apache/instance1/access_log3
		/var/log/apache/instance1/access_log4
		/var/log/apache/instance2/access_log1
		/var/log/apache/instance2/access_log2
		/var/log/apache/instance2/access_log3
		/var/log/apache/instance2/access_log4
		/var/log/apache/instance3/access_log1
		/var/log/apache/instance3/access_log2
		/var/log/apache/instance3/access_log3
		/var/log/apache/instance3/access_log4

	  Path/File Mask:
		/var/log/apache/instance**/access_log*

	  Results:
		Assuming that the last logfile of every directory is the newest one the following 
		files would be selected:
		/var/log/apache/instance1/access_log4
		/var/log/apache/instance2/access_log4
		/var/log/apache/instance3/access_log4

	  Explanation:
		The Path/File mask separates every directory in a virtual group, so 3 groups will 
		be created:
		Group1:
		/var/log/apache/instance1/access_log1
		/var/log/apache/instance1/access_log2
		/var/log/apache/instance1/access_log3
		/var/log/apache/instance1/access_log4

		Group2:
		/var/log/apache/instance2/access_log1
		/var/log/apache/instance2/access_log2
		/var/log/apache/instance2/access_log3
		/var/log/apache/instance2/access_log4

		Group3:
		/var/log/apache/instance3/access_log1
		/var/log/apache/instance3/access_log2
		/var/log/apache/instance3/access_log3
		/var/log/apache/instance3/access_log4

		Now from every group the most recent file will be selected leaving us with the 
		files specified in "results".

		So the mask /var/log/apache/instance**/access_log* in our directory structure 
		would be the same as:
		/var/log/apache/instance1/access_log*
		/var/log/apache/instance2/access_log*
		/var/log/apache/instance3/access_log*

		But the advantage of using "**" is that it is flexible in the number of instances 
		(directories) since they don’t have to be entered manually.

	4) We have the following directory Structure:
		/archive/2011-09-15/outage1.log
		/archive/2011-09-15/outage2.log
		/archive/2011-09-15/outage3.log
		/archive/2011-09-15/error.log
		/archive/2011-09-16/outage1.log
		/archive/2011-09-16/outage2.log
		/archive/2011-09-16/outage3.log
		/archive/2011-09-16/error.log
		/archive/2011-09-17/outage1.log
		/archive/2011-09-17/outage2.log
		/archive/2011-09-17/outage3.log
		/archive/2011-09-17/error.log

	  Path/File Mask:
		/archive/*/outage*.log

	  Results:
		Assuming that the last logfile of every directory is the newest one the following 
		files would be selected:
		/archive/2011-09-17/outage3.log

	  Explanation:
		The Path/File mask selects a newest file named like outageXXX.log among all files 
		in direct subdirectories of directory /archive. This is a more sophisticated 
		example of log rotation policy, e.g. when logs are grouped per sub-directories by 
		date: 
		/archive/--/outage-.log


	5) We have the following directory Structure:
		/var/log/archive/2011-09-15/outage1.log
		/var/log/archive/2011-09-15/outage2.log
		/var/log/archive/2011-09-15/error.log
		/var/log/archive/2011-09-16/outage1.log
		/var/log/archive/2011-09-16/outage2.log
		/var/log/archive/2011-09-16/error.log
		/var/log/archive/2011-09-17/outage1.log
		/var/log/archive/2011-09-17/outage2.log
		/var/log/archive/2011-09-17/error.log
		/var/log/apache/instance1/access_log1
		/var/log/apache/instance1/access_log2
		/var/log/apache/instance2/access_log1
		/var/log/apache/instance2/access_log2
		/var/log/apache/instance3/access_log1
		/var/log/apache/instance3/access_log2
		/var/log/app/services/errors.log
		/var/log/app/services/warnings.log
		/var/log/app/services/information.log
		/var/log/app/services/debug.log
		/var/log/app/services/exportedfile.txt
		/var/log/web/error1.log
		/var/log/web/error2.log
		/var/log/web/error3.log
		/var/log/web/error4.log
		/var/log/test.log
		/var/log/error.log
		/var/log/messages

	  Path/File Mask:
		/var/log/**/**.log

	  Results:
		/var/log/archive/2011-09-15/outage1.log
		/var/log/archive/2011-09-15/outage2.log
		/var/log/archive/2011-09-15/error.log
		/var/log/archive/2011-09-16/outage1.log
		/var/log/archive/2011-09-16/outage2.log
		/var/log/archive/2011-09-16/error.log
		/var/log/archive/2011-09-17/outage1.log
		/var/log/archive/2011-09-17/outage2.log
		/var/log/archive/2011-09-17/error.log
		/var/log/app/services/errors.log
		/var/log/app/services/warnings.log
		/var/log/app/services/information.log
		/var/log/app/services/debug.log
		/var/log/web/error1.log
		/var/log/web/error2.log
		/var/log/web/error3.log
		/var/log/web/error4.log
		/var/log/test.log
		/var/log/error.log

	  Explanation:
		The double star "**" will recursively scan every number of directories and 
		subdirectories, including zero (so files directly in "/var/log" will be 
		matched two; in this example /var/log/test.log and /var/log/error.log). The 
		double star "**" of the files will select every file which ends in ".log".


	6) We have the following directory Structure:
		/var/log/archive/2011-09-15/outage1.log
		/var/log/archive/2011-09-15/outage2.log
		/var/log/archive/2011-09-15/error.log
		/var/log/archive/2011-09-16/outage1.log
		/var/log/archive/2011-09-16/outage2.log
		/var/log/archive/2011-09-16/error.log
		/var/log/archive/2011-09-17/outage1.log
		/var/log/archive/2011-09-17/outage2.log
		/var/log/archive/2011-09-17/error.log
		/var/log/apache/instance1/access_log1
		/var/log/apache/instance1/access_log2
		/var/log/apache/instance2/access_log1
		/var/log/apache/instance2/access_log2
		/var/log/apache/instance3/access_log1
		/var/log/apache/instance3/access_log2
		/var/log/app/services/errors.log
		/var/log/app/services/warnings.log
		/var/log/app/services/information.log
		/var/log/app/services/debug.log
		/var/log/app/services/exportedfile.txt
		/var/log/web/error1.log
		/var/log/web/error2.log
		/var/log/web/error3.log
		/var/log/web/error4.log
		/var/log/test.log
		/var/log/error.log
		/var/log/messages

	  Path/File Mask:
		/var/log/**

	  Results:
		Every file in /var/log and subdirectories will be matched

	  Explanation:
		The double star "**" will recursively scan every number of directories and 
		subdirectories and select every file.
	

The policy specific logfile shows information about incoming indications and results of matching/non-matching conditions.
Note: Monitor policies will not be taken into account, since the correlation is straight forward.

Enable/disable policy specific logging

The Indication policy specific logging will be activated on an Agent for a specified set of policies. The server and the policies on the server are not affected by these settings. This means the policy logging is no global setting for the policies, but an agent specific setting.

The logging is switched off and on via a BOOM_AGENTS Action:

	   SET_PLOG <ON|OFF> 
	       <LOGCOUNT>
               <LOGSIZE>
               <polName1> [<polName2> ...]
	  

LOGCOUNT: maximum number of logfiles
LOGSIZE: maximum size of each logfile in MB
polName: policy names

The activation flags will be stored in the policy xml file only on the Agent side as:

	   <PLOG_ENABLED>true</PLOG_ENABLED>
	   or
	   <PLOG_ENABLED>false</PLOG_ENABLED>
	  
	   Number of Logfiles allowed:
	  
	   <PLOG_LOGCOUNT>3</PLOG_LOGCOUNT>
	   Size of each logfile in MB:
           <PLOG_LOGSIZE>1</PLOG_LOGSIZE>
	   Default: 3x1MB = 3MB
	  

Logfiles and content

The logfiles will be written to the following output directory:

	   <boom_agent>/plogs/
	  

	   <policyName>_<date>_<count>.log
	  

Logfile content (separated by space):

	   <date> <result> <conditionUUID> <indi_UUID> <DETAILS>
	  
	   Where <result> is 3 characters code and can be one of the following:
	       NMA - not matched with any condition
	       MSE - matched and sent
	       SUT - suppressed by time
	       SUC - suppressed by count
	       MST - matched and dropped because the condition’s type is STOP
	   If result is NMA, the condition UUID will be "null"
	  

The Logfiles will be limited by size and count of Logfiles.

DETAILS: fields that are coming with indication (incoming values, not transformed by policy)

host@@group@@appl@@object@@sev@@text_as_one_line_all_NEWLINE_chars_converted_to_TAB

In case of match (MSE): DETAILS will be changed to result values! So Application, Group, Object, Severity and Text can be changed. Indication UUID remains unchanged.

In many cases the input parameters for the NAGIN monitors are different from system to system. It is possible to include in the parameter block following special triggers:

ONINIT  is supported by JavaMonitors as well, but not ONSTART!