Watchers

A Siren Alert watcher is created by using the following structure:

➔ Trigger Schedule

When and How to run the watcher

➔ Input Query

What Query or Join Query to Execute

➔ Condition

How to conditionally Analyze Response

➔ Transform

How to Adapt or Post-Process data

➔ Actions

How to Notify users about this event

Trigger schedule

The schedule defines a set of constraints that must be met to execute a saved watcher. Any number of constraints can be added to a single schedule, and multiple rules can be combined to achieve complex intervals, programmed using simple text expressions using the Node.JS Later module.

watcher anatomy

Interval exceptions can also be defined as follows:

every 2 hours except after 20th hour

Input query

The input parameter is the key element of a watcher, and defines a dynamic range index query feeding the circuit. The input field accepts any standard Elasticsearch query including server side scripts in supported languages and fully supports the Siren Join capabilities out of the box.

"input": {
  "search": {
    "request": {
      "index": [
        "<mos-{now/d}>",
        "<mos-{now/d-1d}>"
      ],
      "body": {}
    }
  }
}

Condition

The condition block is the "entry gate" into the processing pipeline of a watcher and determines its triggered status.

  • On true condition, the pipeline will proceed further.

  • On false condition, the pipeline will stop (no action will be executed) until its next invocation.

Never condition

Use the never condition to set the condition to false. This means the watch actions are never executed when the watch is triggered. Nevertheless, the watch input is executed. This condition is used for testing. There are no attributes to specify for the never condition.

condition: {
  "never" : {}
}

Compare condition

Use the compare condition to perform a simple comparison against a value in the watch payload.

condition: {
  "compare" : {
    "payload.hits.total" : {
      "gte" : 5
    }
}

Comparison operators (apply to numeric, string and date).

eq Returns true when the resolved value equals the given one

not_eq

Returns true when the resolved value does not equal the given one

lt

Returns true when the resolved value is less than the given one

lte

Returns true when the resolved value is less/equal than/to the given one

gt

Returns true when the resolved value is greater than the given one

gte

Returns true when the resolved value is greater/equal than/to the given one

Array compare condition

Use array_compare to compare an array of values. For example, the following array_compare condition returns true if there is at least one bucket in the aggregation that has a doc_count greater than or equal to 25:

"condition": {
  "array_compare": {
    "payload.aggregations.top_amounts.buckets" : {
      "path": "doc_count" ,
      "gte": {
        "value": 25,
      }
    }
  }
}

Options

array.path

The path to the array in the execution context, specified in dot notation.

array.path.path

The path to the field in each array element that you want to evaluate.

array.path.operator.quantifier

How many matches are required for the comparison to evaluate to true: some or all. Defaults to some, there must be at least one match. If the array is empty, the comparison evaluates to false.

array.path.operator.value

The value to compare against.

Script condition

A condition that evaluates a script. The scripting language is JavaScript. It can be as simple as a function expecting a Boolean condition or counter.

condition: {
  "script": {
    "script": "payload.hits.total > 100"
  }
}

Or it can be as complex as an aggregation parser to filter buckets.

condition: {
  "script": {
    "script": "payload.newlist=[];var match=false;var threshold=10;var start_level=2;var finish_level=3;var first=payload.aggregations[start_level.toString()];function loop_on_buckets(element,start,finish,upper_key){element.filter(function(obj){return obj.key;}).forEach( function ( bucket ) { if (start == finish - 1) { if (bucket.doc_count >= threshold) { match=true;payload.newlist.push({line: upper_key + bucket.key + ' ' + bucket.doc_count}); } } else { loop_on_buckets(bucket[start + 1].buckets, start + 1, finish, upper_key + ' ' + bucket.key); } }); } var upper_key = ''; loop_on_buckets(first.buckets, start_level, finish_level, upper_key);match;"
  }
}

Anomaly detection

Simple anomaly finder based on the three-sigma rule.

  1. Dynamic detection of outliers/peaks/drops:

    {
  "script": {
    "script": "payload.hits.total > 0"
  },
  "anomaly": {
    "field_to_check": "fieldName"
  }
}

  2. Static detection for known ranges/interrupts:

    {
  "script": {
    "script": "payload.hits.total > 0"
  },
  "anomaly": {
    "field_to_check": "fieldName",
    "normal_values": [
      5,
      10,
      15,
      20,
      25,
      30
    ]
  }
}

Range filtering

Use for getting documents which have a value in between some values. For example, get only the documents which have values from 45 to 155 inside Amount field.

{
  "script": {
    "script": "payload.hits.total > 0"
  },
  "range": {
    "field_to_check": "Amount",
    "min": 50,
    "max": 150,
    "tolerance": 5
  }
}

Transform

A transform processes and changes the payload in the watch execution context to prepare it for the watch actions. No actions are executed in the case that the payload is empty after transform processing.

Search transform

A transform that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search response.

"transform": {
  "search": {
    "request": {
      "index": [
        "credit_card"
      ],
      "body": {
        "size": 300,
        "query": {
          "bool": {
            "must": [
              {
                "match": {
                  "Class": 1
                }
              }
            ]
          }
        }
      }
    }
  }
}

Script transform

A transform that executes a script (JavaScript) on the current payload and replaces it with a newly generated one. Its uses include:

  • Converting format types.

  • Generating new payload keys.

  • Interpolating data

Create new payload property:

"transform": {
  "script": {
    "script": "payload.outliers = payload.aggregations.response_time_outlier.values['95.0']"
  }
}

Filter aggregation buckets:

"transform": {
  "script": {
    "script": "payload.newlist=[]; payload.payload.aggregations['2'].buckets.filter(function( obj ) { return obj.key; }).forEach(function(bucket){ console.log(bucket.key); if (doc_count.length > 1){ payload.newlist.push({name: bucket.key }); }});"
  }
}

Chain transform

A transform that executes an ordered list of configured transforms in a chain, where the output of one transform serves as the input of the next transform in the chain.

"transform": {
  "chain": [
    {
      "search": {
        "request": {
          "index": [
            "credit_card"
          ],
          "body": {
            "size": 300,
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "Class": 1
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    {
      script: {
        script: "payload.hits.total > 100"
      }
    }
  ]
}

Actions

Actions are used to deliver any results obtained by a watcher to users, APIs or new documents in the cluster. Multiple Actions and Groups can be defined for each.

Actions use the {{ mustache }} logic-less template syntax, and work by iterating arrays and expanding tags in a template using values provided in the watcher body and the response payload. Payload’s sample available in actions contains data from selected index in following format:

{
  hits: {
    total: number,
    hits: hit[]
  }
}

For more details, see Supported actions.

Full watcher example

{
  "_index": "watcher",
  "_id": "new",
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<mos-{now/d}>",
            "<mos-{now/d-1d}>"
          ],
          "body": {}
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 100"
      }
    },
    "transform": {
      "script": {
        "script": "payload.hits.total += 100"
      }
    },
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "alarm@localhost",
          "subject": "Siren Alert Alarm",
          "priority": "high",
          "body": "Found {{payload.hits.total}} Events"
        }
      },
      "slack_admin": {
        "throttle_period": "15m",
        "slack": {
          "channel": "#kibi",
          "message": "Siren Alert Alert! Found {{payload.hits.total}} Events"
        }
      }
    }
  }
}

Supported actions

The following actions are currently supported by Siren Alert watchers.

Actions must be enabled by setting the active property to true in the Investigate configuration file. For example:

sentinl:
  settings:
    email:
      active: true

Email

Send Query results and message using Email/SMTP.

Requires action settings in Siren Investigate configuration.

"email" : {
  "active": true,
  "to" : "root@localhost",
  "from" : "sirenalert@localhost",
  "subject" : "Alarm Title",
  "priority" : "high",
  "body" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "stateless" : false
}

Email HTML

Send Query results and message using Email/SMTP using HTML body.

Requires action settings in Siren Investigate configuration.

"email_html" : {
  "to" : "root@localhost",
  "from" : "sirenalert@localhost",
  "subject" : "Alarm Title",
  "priority" : "high",
  "body" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "html" : "<p>Series Alarm {{ payload._id}}: {{payload.hits.total}}</p>",
  "stateless" : false
}

webHook

Deliver a POST request to a remote web API

"webhook" : {
  "active": true,
  "method" : "POST",
  "host" : "remote.server",
  "port" : 9200,
  "path": ":/{{payload.watcher_id}}",
  "body" : "{{payload.watcher_id}}:{{payload.hits.total}}",
  "create_alert" : true
  "cacert" : "ca-certificate.pem",
  "cert" : "certificate.pem",
  "key" : "key.pem"
}

Deliver a GET request to a remote web API

"webhook" : {
  "active": true,
  "method" : "GET",
  "host" : "remote.server",
  "port" : 9200,
  "path" : "/trigger",
  "params" : {
    "watcher": "{{watcher.title}}",
    "query_count": "{{payload.hits.total}}"
  },
  "cacert" : "ca-certificate.pem",
  "cert" : "certificate.pem",
  "key" : "key.pem"
}

webHook using Proxy

Deliver message to remote API using Proxy - Telegram example:

"webhook": {
  "active": true,
  "method": "POST",
  "host": "remote.proxy",
  "port": "3128",
  "path": "https://api.telegram.org/bot{botId}/sendMessage",
  "body": "chat_id={chatId}&text=Count+total+hits:%20{{payload.hits.total}}",
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  },
  "create_alert" : true
}

Slack

Delivery Message to #Slack channel.

Requires action settings in Siren Investigate configuration.

"slack" : {
  "active": true,
  "channel": "#channel",
  "message" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "stateless" : false
}

Report

Take a website snapshot using PhantomJS and send it using Email/SMTP.

  • Requires action settings in Siren Investigate configuration.

  • Requires Pageres/PhantomJS: npm install -g pageres.

  • Requires Chromium browser to be installed. If not, then Siren downloads chromium provided the user has the permission to do so.

"report" : {
  "active": true,
  "to" : "root@localhost",
  "from" : "kaae@localhost",
  "subject" : "Report Title",
  "priority" : "high",
  "body" : "Series Report {{ payload._id}}: {{payload.hits.total}}",
  "snapshot" : {
    "res" : "1280,900",
    "url" : "http://127.0.0.1/app/kibana#/dashboard/Alerts",
    "path" : "/tmp/",
    "params" : {
      "username" : "username",
      "password" : "password",
      "delay" : 5000,
      "crop" : false
    }
  },
  "stateless" : false
}

Console

Output Query results and message to Console.

"console" : {
  "priority" : "DEBUG",
  "message" : "Average {{payload.aggregations.avg.value}}"
}

Watcher controllers

The following controls are presented when listing existing watchers:

Watcher controllers

  1. Expand or edit a watcher.

  2. Manually execute a watcher.

  3. Remove a watcher.

  4. Toggle a watcher on or off.

Security for watchers and alerts

When security is enabled, you can define the visibility of a watcher, alarm or report for different groups of users.

Watcher visibility

If you want each user to see only their watchers, complete the following steps in the ACL Roles tab

Step one: Open the everyone ACL role and add a saved object rule to allow the Create permission for a watcher as follows:

Everyone watcher permission

Step two: For all the other ACL roles, add a saved object rule to deny the View permission for a watcher as follows:

Other roles watcher permissions

If there is a need to share watchers with some specific roles, this can be configured in the saved objects configuration page (see here) because watchers are saved objects.

Alarms/Reports visibility

You can restrict the visibility of alarms and reports in the Watcher editor, by using the privacy section. The restrictions can be set as follows:

Alerts privacy settings

To restrict the permissions of alarms and reports, you must use document level security (DLS) by adding the following settings to the sg_roles.yml file of your searchguard security and running the sgadmin.sh script:

//ES6
indices:
  'watcher_alarms-*':
    '*':
      - READ
      - VIEW_INDEX_METADATA
    _dls_: '{
              "bool": {
                "should": [
                  {
                    "term": {
                      "is_private": false
                    }
                  },
                  {
                    "bool": {
                      "must_not": [
                        {
                          "exists": {
                            "field": "is_private"
                          }
                        }
                      ]
                    }
                  },
                  {
                    "terms": {
                      "roles": [${user.roles}]
                    }
                  },
                  {
                    "term": {
                      "created_by": "${user.name}"
                    }
                  }
                ]
              }
            }'

//ES7
index_permissions:
- index_patterns:
  - 'watcher_alarms-*'
  allowed_actions:
  - READ
  - VIEW_INDEX_METADATA
  dls: '{
          "bool": {
            "should": [
              {
                "term": {
                  "is_private": false
                }
              },
              {
                "bool": {
                  "must_not": [
                    {
                      "exists": {
                        "field": "is_private"
                      }
                    }
                  ]
                }
              },
              {
                "terms": {
                  "roles": [${user.roles}]
                }
              },
              {
                "term": {
                  "created_by": "${user.name}"
                }
              }
            ]
          }
        }'

Examples

Watchers can be as simple or complex as the query and aggregations they use. Here are some examples to get started with.

Hit watcher

{
  "_index": "watcher",
  "_id": "new",
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<mos-{now/d}>",
            "<mos-{now/d-1d}>"
          ],
          "body": {}
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 100"
      }
    },
    "transform": {},
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "alarm@localhost",
          "from": "sirenalert@localhost",
          "subject": "Siren Alert Alarm",
          "priority": "high",
          "body": "Found {{payload.hits.total}} Events"
        }
      },
      "slack_admin": {
        "throttle_period": "15m",
        "slack": {
          "channel": "#kibi",
          "message": "Siren Alert Alert! Found {{payload.hits.total}} Events"
        }
      }
    }
  }
}

Transform (Elasticsearch 2.x)

{
  "_index": "watcher",
  "_id": "95th",
  "_score": 1,
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<access-{now/d}>",
            "<access-{now/d-1d}>"
          ],
          "body": {
            "size": 0,
            "query": {
              "filtered": {
                "query": {
                  "query_string": {
                    "analyze_wildcard": true,
                    "query": "*"
                  }
                },
                "filter": {
                  "range": {
                    "@timestamp": {
                      "from": "now-5m"
                    }
                  }
                }
              }
            },
            "aggs": {
              "response_time_outlier": {
                "percentiles": {
                  "field": "response_time",
                  "percents": [
                    95
                  ]
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.aggregations.response_time_outlier.values['95.0'] > 200"
      }
    },
    "transform": {
      "script": {
        "script": "payload.myvar = payload.aggregations.response_time_outlier.values['95.0']"
      }
    },
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "username@mycompany.com",
          "from": "sirenalert@mycompany.com",
          "subject": "Siren Alert ALARM {{ payload._id }}",
          "priority": "high",
          "body": "Series Alarm {{ payload._id}}: {{ payload.myvar }}"
        }
      }
    }
  }
}

Siren Alert: insert back to Elasticsearch bulk (using nginx or direct).

{
 "_index": "watcher",
 "_id": "surprise",
 "_score": 1,
 "_source": {
   "trigger": {
     "schedule": {
       "later": "every 50 seconds"
     }
   },
   "input": {
     "search": {
       "request": {
         "index": "my-requests-*",
         "body": {
           "query": {
             "filtered": {
               "query": {
                 "query_string": {
                   "query": "*",
                   "analyze_wildcard": true
                 }
               },
               "filter": {
                 "range": {
                   "@timestamp": {
                     "from": "now-5m"
                   }
                 }
               }
             }
           },
           "size": 0,
           "aggs": {
             "metrics": {
               "terms": {
                 "field": "first_url_part"
               }
             }
           }
         }
       }
     }
   },
   "condition": {
     "script": {
       "script": "payload.hits.total > 1"
     }
   },
   "transform": {},
   "actions": {
     "ES_bulk_request": {
       "throttle_period": "1m",
       "webhook": {
         "method": "POST",
         "host": "elasticsearch.foo.bar",
         "port": 80,
         "path": ":/_bulk",
         "body": "{{#payload.aggregations.metrics.buckets}}{\"index\":{\"_index\":\"aggregated_requests\"}}\n{\"url\":\"{{key}}\", \"count\":\"{{doc_count}}\", \"execution_time\":\"tbd\"}\n{{/payload.aggregations.metrics.buckets}}",
         "headers": {
           "Content-Type": "text/plain; charset=ISO-8859-1"
         },
         "create_alert": true
       }
     }
   }
 }
}

Wizard

Siren Alert provides a wizard to help create watchers using a step-by-step sequence.

Give the watcher a name and choose an execution frequency. For example, run every day.

image

Specify the input query parameters and the condition to trigger on (based on a date histogram aggregation. For example, trigger an alert when there are more than two articles in an hour during the day.

image

To send an alert, set up a variety of actions to happen when your condition is met. For example, send a HTML-formatted email injected with data from the watcher and query response using the mustache templating language.

image

Custom watchers

Together with standard watchers, Siren Alert supports an additional list of use case-specific watcher types that are created from the dashboard. These types of watchers are called custom watchers.

Dashboard button

Custom watchers created from a dashboard inherit its search criteria, including the search pattern, search query, and filters. Each custom watcher type applies specific processing and trigger conditions.

image

Custom watcher types

New results and geo fence

The 'new results' watcher alerts you when new data is added to Elasticsearch that is in a saved search.

The 'geo fence' watcher allows you to select a geographical area on your dashboard and alerts you if Elasticsearch documents with position data inside this 'fence' are added.

These custom watchers can be created from a dashboard by using the Watcher button, and they use all the filters that are applied to the dashboard.

Proximity

The proximity watcher type alerts you when two entities are closer together or further apart from each other than the distance that you specify.

The proximity watcher type must be enabled manually, because it is not applicable to all data sets. Siren provides support for this manual enablement.

Proximity watchers are particularly useful if you have an index that logs the positional updates of several entities. For example, you can retrieve periodic positional updates from a set of mobile phones.

Like the new results and geo fence watchers, this custom watcher is created from a dashboard and monitors a saved search with any added filters. When you create this type of custom watcher, you must specify the following parameters:

Proximity watcher parameters
ID field and entity IDs

Specify the two entities that you want to keep track of. The ID field that you choose must have the keyword mapping and the Entity fields represent the entities that stream positional data to Elasticsearch.

Location field and distance

These fields allow you to set the boundary distance for the proximity of the specified entities. You can choose to send an alert when the distance between the entities is greater or less than the specified distance.

This type of alerting is useful for real-world scenarios such as policing restraining orders, where two people cannot be within 1km of each other, or house arrests, when a person is not allowed to travel beyond 500 meters of their home.

Cooldown tracker

The cooldown tracker monitors the proximity of two entities by using their last positional update. However, the accuracy of the proximity calculation is dependent on the time when the last positional update was received. This custom watcher type allows you to send an alert if the positional update is older than a specified value.

For example, if you are monitoring how close two mobile phones come to each other, several positional updates might be expected per hour. But if an hour or more elapses since the last positional update of one of the phones, it might mean that a target has turned off their phone. In this situation, an alert can allow an investigator to handle the scenario quickly.

Creating custom watchers

Custom watchers are managed in the Script tab of the Management section. Creating a custom watcher requires writing a script that provides an object with the attributes described below. Siren provides support for creating these scripts.

Attribute Type Description

dashboard.order

number

Used to place watcher type in dashboard box.

dashboard.show

function(dashboard, indexMappings)

Tests whether to show the watcher type in the dashboard box.

params

object<string>

The collection of parameters and default values required by the watcher.

search

function(client, searchParams, params, watcher) ⇒ searchResponse

Executed in the backend that searches Elasticsearch using the client object. searchParams provides search criteria (index, filters, queries, time, originalTimeField). params provides direct access to collection of parameters on the watcher. watcher provides access to the watcher object. Example for the watcher object can be found here.

condition

function(searchResponse) ⇒ boolean

Evaluates the response from the search function and determines if the watcher should perform the alert actions.

template

string

Deprecated: Use attributes below. The HTML template presenting inputs for the watcher parameters. The params attribute is injected (for example ng-model="params.param1").

template.configuration

string

The HTML template presenting the configuration panel.

template.parameters

string

The HTML template presenting inputs for the watcher parameters. The params attribute is injected (for example ng-model="params.param1").

convertTime

function(start, end) ⇒ [moment | string, moment | string]

Converts the time that will be searched. Moment.js objects are passed in to facilitate relative date manipulation (e.g. (start, end) => ([start.startOf('day'), end.subtract(1, 'minute')])). Strings may also be returned for absolute times (e.g. (start, end) => (['2020-01-01T00:00:00.000', '2021-01-01T00:00:00.000'])).

alternativeTimeField.timeField

string

Changes the time field used in the search query. If this is set, dashboardTimeRange must also be set so the dashboard link applies the appropriate global time range. The link will also add a normal filter that selects the time range searched with the alternative time field.

alternativeTimeField.dashboardTimeRange

[string, string]

The global time range set on a dashboard by a watcher’s dashboard link. This is a mandatory field if alternativeTimeField.timeField is provided. This supports both absolute and relative times (e.g. ['1900-01-01T00:00:00', 'now-1y'])).

hideSettings

boolean

Default: false. Hides the settings on the default template for the configuration and actions panels.

ignoreConfigActionParams

boolean

Default: false. Ignores the actions' parameters defined on Siren Investigate configuration and use the actions' parameters defined on the watcher body.

addActionsList

array of objects

Defines the array of allowed actions for the watcher and the settings for these default actions. If any action type defined in actions is not found in addActionsList then that action will be disabled for the user. Schema is the same as schema for actions. For example, the following code allows the Elastic action to show up in the Add Actions dropdown on the watcher configuration page.

[{
  name: 'Elasticsearch alarm',
  typeLabel: 'Elastic',
  throttle_period: '1s'
}]

If this is an empty array, the "Add action" button will be hidden.

defaults

object or function

Allows passing of default values via custom watcher script. Currently only title and schedule are supported.

({
  ...,
  defaults: {
    title: 'my watcher',
    schedule: {
      frequency: 'every 5 minutes',
      startDateTime: new Date(2020, 05, 05),
      endDateTime: new Date(2021, 05, 05)
    }
  }
});

or using data provided from the dashboard

({
  ...,
  defaults: (dash) => ({
    title: dash.title,
    schedule: {
      frequency: dash.title === 'CDR' ? 'every 1 minutes' : 'every 1 hours',
      startDateTime: new Date(2020, 05, 05),
      endDateTime: new Date(2021, 05, 05)
    }
  })
});

Default values for startDateTime and endDateTime can be only instances of JavaScript Date class.

paramsValidator

function(params) => void

Enables users to create bespoke validation rules for a watcher’s parameters. Throw an error if a parameter value is invalid, for example:

({
  ...,
  params: {
    property1: '<an-invalid-value>'
  },
  paramsValidator: (params) => ({
    if (params.property1 !== 'valid value!') {
      throw new Error('property1 is invalid');
    }
  })
});

deprecated

boolean

Default: false. Prevents users from creating new watchers from the template and deactivates existing watchers that are based on it.

Specific custom watcher information

Additional watcher information is provided in the watcher variable and can be accessed on watcher actions.

Property Type Description

dashboard_link

string

Dashboard link with the 'filtered by watcher' results

custom.dashboard_id

string

ID for associated dashboard