Setting up Siren Investigate

This section includes information on how to set up Siren Investigate and get it running, including:

  • Downloading

  • Installing

  • Starting

  • Configuring

  • Upgrading

The instructions in this section are relevant for production deployments only. For a quick try of Siren, just download the "Easy Start" version https://siren.io/downloads/?product=siren-platform and follow the instructions in the readme file.

Supported platforms

Packages of Siren Investigate are provided for and tested with Linux and Windows. Because Siren Investigate runs on Node.js, the required Node.js binaries for these platforms are included. Running Siren Investigate against a separately maintained version of Node.js is not supported.

Version compatibility

Since Siren 10.3 there is a capability that allows users to upgrade to a newer Investigate version without having to upgrade their Elasticsearch and Federate backend. Upgrading the backend is a complex procedure as the Elasticsearch version has to be upgraded with every major/minor Federate release. By removing the dependence on a specific backend, users can benefit from new advanced features by upgrading to the latest Investigate version, without the complexity of a simultaneous Federate/Elasticsearch upgrade.

Each Investigate version now supports several Elasticsearch/Federate versions in the following way:

  • Minimum supported Elasticsearch/Federate version - runs the set of Siren features compatible with the Elasticsearch version used. This includes all the features available in Siren 10.1, and all the features in 10.2 except Data Reflection, Neo4j Support, Ingestion, and Export/Scroll API.

  • Feature complete Elasticsearch/Federate version - runs all current Siren features

  • Target Elasticsearch/Federate version - runs all current Siren features and, being the most up-to-date version, is recommended in most cases

Siren 10.4 compatibility

Compatibility Level Federate Version

Minimum

5.6.10 - 10.1.1

Feature Complete

6.8.0 - 10.3.0

Target

6.8.6 - 10.3.4

If the Elasticsearch/Federate backend is more recent than the Investigate version, it has to be within a minor release from Investigate Target version so as not to break APIs.

When Investigate starts up, it checks the version of Federate it is working with and compares with a compatibility matrix. It detects the features that cannot be supported and switches them off in the user interface. This is logged in the Investigate server, and the user can see which features are unsupported by checking the About section (click Management to see the About section).

Feature Description Minimum Federate Version

Datasource Reflection

Import and mapping of data to Elasticsearch from registered datasources

6.5.4-10.2.0

Neo4j Compatibility

Ability to register a Neo4j datasource, which can be used in Data Reflection

6.5.4-10.2.0

Remote ES Connector

Ability to register a remote Elasticsearch cluster as a datasource and create virtual indices on it

6.8.0-10.3.0

Earlier version compatibility

Table 1. Version compatibility matrix
Siren Investigate Siren Federate Elasticsearch

10.2.4

Siren Federate 6.8.0-10.2.4

6.8.0

Siren Federate 6.5.4-10.2.4

6.5.4

10.2.3

Siren Federate 6.8.0-10.2.3

6.8.0

Siren Federate 6.5.4-10.2.3

6.5.4

10.2.2

Siren Federate 6.5.4-10.2.2

6.5.4

10.2.1

Siren Federate 6.5.4-10.2.1

6.5.4

10.2.0

Siren Federate 6.5.4-10.2.0

6.5.4

10.1.3

Siren Federate 6.3.2-10.1.3-1

6.3.2

Siren Federate 5.6.16-10.1.3-1

5.6.16

10.1.2

Siren Federate 6.3.2-10.1.2

6.3.2

Siren Federate 5.6.14-10.1.2

5.6.14

10.1.1

Siren Federate 6.3.2-10.1.1

6.3.2

Siren Federate 5.6.10-10.1.1

5.6.10

Running different major version releases of Siren Investigate and Elasticsearch, for example Siren Investigate 10.x and Elasticsearch 2.x, is not supported, nor is running a minor version of Siren Investigate that is newer than the version of Elasticsearch, for example Siren Investigate 10.0.x and Elasticsearch 5.0.x.

Running a minor version of Elasticsearch that is higher than supported version will generally work to facilitate an upgrade process where Elasticsearch is upgraded first, for example Siren Investigate 10.0.x and Elasticsearch 5.7.x. In this configuration, a warning will be logged on Siren Investigate server startup, so it is only meant to be temporary until Siren Investigate is upgraded to the same version as Elasticsearch.

Running different patch version releases of Siren Investigate and Elasticsearch, for example Siren Investigate 10.0.0-1 and Elasticsearch 5.6, is generally supported, though we encourage users to run the supported versions of Siren Investigate and Elasticsearch down to the patch version.

Installing Siren Investigate

Siren Investigate is provided in the following package formats:

zip

The zip packages are provided for installation on Linux and Windows and are the easiest choice for getting started with Siren Investigate.

docker

Siren Investigate Docker images are available at https://hub.docker.com/u/sirensolutions/.

If your Elasticsearch installation is protected by X-Pack Security, see Using Kibana with X-Pack Security for additional setup instructions.

Running Siren Investigate on Docker

Docker images for Siren Investigate are available from the sirensolutions organization on Dockerhub.

Pulling the Image

Obtaining Siren Investigate for Docker is as simple as issuing a docker pull command.

The Docker image for the latest Siren Investigate can be retrieved with the following command:

docker pull sirensolutions/siren-platform:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:latest

Or for a specific version, for example 10.0.0:

docker pull sirensolutions/siren-platform:10.0.0
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:10.0.0

For an image pre-populated with demonstration data:

docker pull sirensolutions/siren-platform-demo-data:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform-demo-data:latest

Environment variable configuration

Under Docker, Siren Investigate can be configured using environment variables. When the container starts, a helper process checks the environment for variables that can be mapped to Siren Investigate command-line arguments.

For compatibility with container orchestration systems, these environment variables are written in all capitals, with underscores as word separators. The helper translates these names to valid Siren Investigate setting names.

Some example translations are shown here:

Table 2. Example Docker Environment Variables

Environment Variable

Siren Investigate Setting

SERVER_NAME

server.name

KIBANA_DEFAULTAPPID

kibana.defaultAppId

XPACK_MONITORING_ENABLED

xpack.monitoring.enabled

In general, any setting listed in Configuring Siren Investigate or X-Pack Settings can be configured with this technique.

These variables can be set with docker-compose like this:

services:
  investigate:
    image: docker.elastic.co/kibana/kibana:10.0.0
    environment:
      SERVER_NAME: siren.example.org
      ELASTICSEARCH_URL: http://elasticsearch.example.org

Because environment variables are translated to CLI arguments, they take precedence over settings configured in investigate.yml.

Docker defaults

The default settings when using the siren-investigate image (standalone Siren Investigate image) are:

elasticsearch.url

http://localhost:9220

server.basepath

""

kibana.index

.siren

Install Siren Investigate with ZIP package

Siren Investigate is provided for Linux and Windows as a .zip package. These packages are the easiest formats to use when trying out Siren Investigate.

The latest stable version of Siren Investigate can be found on the Siren Support Portal. Descriptions of the separate demonstration packages are also available there.

Download and install the Linux 64-bit package

After you have obtained your license, you should have access to the Siren Support Portal's download pages. The siren-investigate package can be downloaded from there, either by clicking the package link in the browser or, right click the link, then use the copied link as investigate-link as follows:

wget {investigate-link}.zip
sha1sum siren-investigate-10.0.0-linux-x86_64.zip (1)
unzip siren-investigate-10.0.0-linux-x86_64.zip -d .
cd siren-investigate-10.0.0-linux-x86_64/ (2)
1 Compare the SHA produced by sha1sum or shasum with the published sha1.txt (found on the Siren Investigate download page on the Siren Support Portal).
2 This folder is known as $INVESTIGATE_HOME.

Running Siren Investigate from the command prompt

Siren Investigate can be started from the command prompt as follows:

./bin/investigate

By default, Siren Investigate runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl+C.

Siren Investigate configuration

Siren Investigate loads its configuration from the $INVESTIGATE_HOME/config/investigate.yml file by default. The format of this configuration file is explained in Configuring Siren Investigate.

Folder layout of Linux ZIP archive

The .zip packages are entirely self-contained.

This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling is as easy as removing the folder. However, it is advisable to change the default locations of the configuration and data folders so that you do not remove important data later on.

Type Description Default Location Setting

home

Siren Investigate home folder or $INVESTIGATE_HOME

Folder created by unpacking the archive; in demonstration distributions, the folder is siren-investigate.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

$INVESTIGATE_HOME\bin

config

Configuration files including investigate.yml

$INVESTIGATE_HOME\config

data

The location of the data files written to disk by Siren Investigate and its plugins

$INVESTIGATE_HOME\data

optimize

Transpiled source code. Certain administrative actions, for example plugin install, result in the source code being retranspiled on the fly.

$INVESTIGATE_HOME\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subfolder.

$INVESTIGATE_HOME\plugins

Install Siren Investigate on Windows

Siren Investigate can be installed on Windows using the .zip package; zip packages can be downloaded from the download page. The demonstration versions contain a preconfigured Elasticsearch cluster in addition to Siren Investigate.

Running Siren Investigate from the command prompt

Siren Investigate can be started from the command prompt as follows:

.\bin\investigate.bat

By default, Siren Investigate runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl+C.

Siren Investigate configuration

Siren Investigate loads its configuration from the $INVESTIGATE_HOME/config/investigate.yml file by default. The format of this configuration file is explained in Configuring Siren Investigate.

Folder layout of Windows ZIP archive

The .zip package is entirely self-contained.

This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling Siren Investigate is as easy as removing the folder. However, it is advisable to change the default locations of the configuration and data folders so that you do not remove important data later on.

Type Description Default Location Setting

home

Siren Investigate home folder or %INVESTIGATE_HOME%

Folder created by unpacking the archive; in demonstration distributions, the folder is kibi.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

%INVESTIGATE_HOME%\bin

config

Configuration files including investigate.yml

%INVESTIGATE_HOME%\config

data

The location of the data files written to disk by Siren Investigate and its plugins

%INVESTIGATE_HOME%\data

optimize

Transpiled source code. Certain administrative actions, for example plugin install, result in the source code being retranspiled on the fly.

%INVESTIGATE_HOME%\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subfolder.

%INVESTIGATE_HOME%\plugins

Configuring Siren Investigate

The Siren Investigate server reads properties from the investigate.yml file on startup. The default settings configure Siren Investigate to run on localhost:5606. To change the host or port number, or connect to Elasticsearch running on a different machine, you must update your investigate.yml file. You can also enable SSL and set a variety of other options.

External datasource configuration is documented in the Working with JDBC datasources and Legacy REST datasources sections, while access control configuration is documented in Search Guard integration and Siren Investigate access control.

Environment Variable Placeholders.

It is possible to use environment variable placeholders in configuration settings. The syntax of placeholders is ${ENV_VARIABLE_NAME}.

For example, to set elasticsearch.url to the value of the environment variable ES_URL, edit config/investigate.yml as follows:

elasticsearch.url: ${ES_URL}
Table 3. Configuration Settings
Setting Description

server.port:

Default: 5606. Siren Investigate is served by a back end server. This setting specifies the port to use.

server.host:

Default: localhost. This setting specifies the host of the back end server.

server.basePath:

Enables you to specify a path to mount Siren Investigate as if you are running behind a proxy. This only affects the URLs generated by Siren Investigate, your proxy is expected to remove the basePath value before forwarding requests to Siren Investigate. This setting cannot end in a slash (/).

server.maxPayloadBytes:

Default: 1048576. The maximum payload size in bytes for incoming server requests.

server.name:

Default: your-hostname. A human-readable display name that identifies this Siren Investigate instance.

server.defaultRoute:

Default: /app/kibana. This setting specifies the default route when opening Siren Investigate. You can use this setting to modify the landing page when opening Siren Investigate.

elasticsearch.url:

Default: http://localhost:9220. The URL of the Elasticsearch instance to use for all your queries.

elasticsearch.preserveHost:

Default: true. When this setting’s value is true Siren Investigate uses the hostname specified in the server.host setting. When the value of this setting is false, Siren Investigate uses the hostname of the host that connects to this Siren Investigate instance.

kibana.index:

Default: .siren. Siren Investigate uses an index in Elasticsearch to store saved searches, visualizations and dashboards. Siren Investigate creates a new index if the index does not already exist.

kibana.defaultAppId:

Default: discover. The default application to load.

tilemap

The tilemap is the base layer the map visualizations. To increase max zoom level for mapping visualizations, copy the following object in the `investigate.yml`file, this will increase the zoom level to 19 from the default of 15.

tilemap:
  url: "https://\{s}.tile.openstreetmap.org/\{z}/\{x}/\{y}.png"
  options:
    attribution: '©
[OpenStreetMap("http://www.openstreetmap.org/copyright")'
    subdomains: ['a']
    minZoom: 0
    maxZoom: 19

tilemap.url:

The URL to the tile service that Siren Investigate uses to display map tiles in tilemap visualizations. By default, Siren Investigate reads this URL from an external metadata service, but users can still override this parameter to use their own Tile Map Service.

tilemap.options.minZoom:

Default: 1. The minimum zoom level.

tilemap.options.maxZoom:

Default: 15. The maximum zoom level.

tilemap.options.attribution:

Default: © [OpenStreetMap]("http://www.openstreetmap.org/copyright"). The map attribution string.

tilemap.options.subdomains:

An array of subdomains used by the tile service. Specify the position of the subdomain in the URL with the token {s}.

regionmap

Specifies additional vector layers for use in Region Map visualizations. Each layer object points to an external vector file that contains a GeoJSON FeatureCollection. The file must use the WGS84 coordinate reference system and only include polygons. If the file is hosted on a separate domain from Siren Investigate, the server needs to be CORS (Cross Origin Request Security) enabled so that Siren Investigate can download the file. The following example shows a valid regionmap configuration.

regionmap:
  layers:
       name: "Departments of France"
       url:
"http://my.cors.enabled.server.org/france_departements.geojson"
       attribution: "INRAP"
       fields:
          name: "department"
          description: "Full department name"
          name: "INSEE"
          description: "INSEE numeric identifier"

regionmap.layers.name:

Mandatory. A description of the map being provided.

regionmap.layers.url:

Mandatory. The location of the GeoJSON file as provided by a web server.

regionmap.layers.attribution:

Optional. References the originating source of the GeoJSON file.

regionmap.layers.fields:

Mandatory. Each layer can contain multiple fields to indicate what properties from the geojson features you want to expose. This example shows how to define multiple properties.

regionmap.layers.fields.name:

Mandatory. This value is used to do an inner-join between the document stored in Elasticsearch and the GeoJSON file. For example, if the field in the GeoJSON is called Location and has city names, there must be a field in Elasticsearch that holds the same values that Siren Investigate can then use to lookup for the GeoShape data.

regionmap.layers.fields.description:

Mandatory. The human readable text that is shown under the Options tab when building the Region Map visualization.

elasticsearch.username: and elasticsearch.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Siren Investigate server uses to perform maintenance on the Siren Investigate index at startup. Your Siren Investigate users still need to authenticate with Elasticsearch, which is proxied through the Siren Investigate server.

server.ssl.enabled

Default: false. Enables SSL for incoming requests from the browser to Siren Investigate. When set to true, server.ssl.certificate and server.ssl.key are required

server.ssl.certificate: and server.ssl.key:

Paths to the PEM-format SSL certificate and SSL key files, respectively.

server.ssl.keyPassphrase:

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

server.ssl.certificateAuthorities:

List of paths to PEM encoded certificate files that should be trusted.

server.ssl.supportedProtocols:

Default: TLSv1, TLSv1.1, TLSv1.2. Supported protocols with versions. Valid protocols: TLSv1, TLSv1.1, TLSv1.2

server.ssl.cipherSuites:

Default: ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384, DHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA384, ECDHE-RSA-AES256-SHA256, DHE-RSA-AES256-SHA256, HIGH,!aNULL, !eNULL, !EXPORT, !DES, !RC4, !MD5, !PSK, !SRP, !CAMELLIA.

For more information on the format, and valid options, refer to the OpenSSL cipher list format documentation (https://www.openssl.org/docs/man1.0.2/apps/ciphers.html#CIPHER-LIST-FORMAT).

elasticsearch.ssl.certificate: and elasticsearch.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files are used to verify the identity of Siren Investigate to Elasticsearch and are required when xpack.ssl.verification_mode in Elasticsearch is set to either certificate or full.

elasticsearch.ssl.keyPassphrase:

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

elasticsearch.ssl.certificateAuthorities:

Optional setting that enables you to specify a list of paths to the PEM file for the certificate authority for your Elasticsearch instance.

elasticsearch.ssl.verificationMode:

Default: full. Controls the verification of certificates presented by Elasticsearch. Valid values are none, certificate, and full. full performs hostname verification, and certificate does not.

elasticsearch.pingTimeout:

Default: the value of the elasticsearch.requestTimeout setting. Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.requestTimeout:

Default: 30000. Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.requestHeadersWhitelist:

Default: [ 'authorization' ]. List of Siren Investigate client-side headers to send to Elasticsearch. To send no client-side headers, set this value to [] (an empty list).

elasticsearch.customHeaders:

Default: {}. Header names and values to send to Elasticsearch. Any custom headers cannot be overwritten by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

elasticsearch.shardTimeout:

Default: 0. Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to switch off.

elasticsearch.startupTimeout:

Default: 5000. Time in milliseconds to wait for Elasticsearch at Siren Investigate startup before retrying.

pid.file:

Specifies the path where Siren Investigate creates the process ID file.

path.data:

Default: ./data. The path where Siren Investigate stores persistent data not saved in Elasticsearch

logging.dest:

Default: stdout. Enables you specify a file where Siren Investigate stores log output.

logging.silent:

Default: false. Set the value of this setting to true to suppress all logging output.

logging.quiet:

Default: false. Set the value of this setting to true to suppress all logging output other than error messages.

logging.verbose:

Default: false. Set the value of this setting to true to log all events, including system usage information and all requests.

ops.interval:

Default: 5000. Set the interval in milliseconds to sample system and process performance metrics. The minimum value is 100.

status.allowAnonymous:

Default: false. If authentication is enabled, setting this to true allows unauthenticated users to access the Siren Investigate server status API and status page.

cpu.cgroup.path.override:

Override for cgroup cpu path when mounted in manner that is inconsistent with /proc/self/cgroup

cpuacct.cgroup.path.override:

Override for cgroup cpuacct path when mounted in manner that is inconsistent with /proc/self/cgroup

console.enabled

Default: true. Set to false to disable Console. Toggling this will cause the server to regenerate assets on the next startup, which may cause a delay before pages start being served.

elasticsearch.tribe.url:

Optional URL of the Elasticsearch tribe instance to use for all your queries.

elasticsearch.tribe.username: and elasticsearch.tribe.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Siren Investigate server uses to perform maintenance on the Siren Investigate index at startup. Your Siren Investigate users still need to authenticate with Elasticsearch, which is proxied through the Siren Investigate server.

elasticsearch.tribe.ssl.cert: and elasticsearch.tribe.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files validate that your Elasticsearch backend uses the same key files.

elasticsearch.tribe.ssl.keyPassphrase:

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

elasticsearch.tribe.ssl.certificateAuthorities:

Optional setting that enables you to specify a path to the PEM file for the certificate authority for your tribe Elasticsearch instance.

elasticsearch.tribe.ssl.verificationMode:

Default: full. Controls the verification of certificates. Valid values are none, certificate, and full. While full performs hostname verification, certificate does not.

elasticsearch.tribe.pingTimeout:

Default: the value of the elasticsearch.tribe.requestTimeout setting. Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.tribe.requestTimeout:

Default: 30000. Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.tribe.requestHeadersWhitelist:

Default: [ 'authorization' ].  List of Siren Investigate client-side headers to send to Elasticsearch. To send no client-side headers, set this value to [] (an empty list).

elasticsearch.tribe.customHeaders:

Default: {}. Header names and values to send to Elasticsearch. Any custom headers cannot be overwritten by client-side headers, regardless of the elasticsearch.tribe.requestHeadersWhitelist configuration.

White-labelling constants and images in Siren

White-labelling is the process of changing some elements in a product to reflect the branding of a company or organization deploying that product. You can now label certain constants and modify images and icons in Siren Investigate to reflect the branding of your company or organization.

Changing company and product names

The company and product names default to ‘Siren’ and ‘Investigate’, respectively. To change them, on the server, go to Siren Investigate > config and edit the investigate.yml file. Search for the term whiteLabel and you should find the company and product name keys as shown below.

#The white labels.
whiteLabel:
  companyName: Siren
  productName: Investigate
If the keys are not in investigate.yml, you can add them, ensuring they are formatted exactly as shown.

Change the names as required and once done, restart the server and refresh Investigate to view the changes.

The copyright text is normally visible on the user interface under Management. If you want to suppress the display of this text, go to Siren Investigate > config and in the investigate.yml file change the value of the whiteLabel.hideCopyright key to true. If the key is not there, add it as shown below.

#The white labels.
whiteLabel:
  companyName: Siren
  productName: Investigate
  hideCopyright: false

Add management information

If you wish to add more information on the user interface under Management, it can be done by adding keys under whiteLabel.management.info. For example, doing

#The white labels
whiteLabel:
  companyName: Siren
  productName: Investigate
  hideCopyright: false
  management:
    info:
      App Version: 1.4.5

would add the "App Version" on the Management page as shown below

image

Multiple keys can also be added under whiteLabel.management.info.

The key(s) can also be added by passing the additional information as an environment variable to Investigate’s docker image. To do it, prepend the actual key you wish to pass with the string "WHITELABEL_MANAGEMENT_INFO_". For example, to add the "App Version" and the "Company Motto", a parameter will be passed to the docker run command as follows

docker run -e WHITELABEL_MANAGEMENT_INFO_App_Version=1.4.5 -e WHITELABEL_MANAGEMENT_INFO_Company_Motto=Search_away_anything_andEverything <imageId>
In order to add space(s) between the strings of the keys or the values, add a "underscore" between them. The underscores are post processed to be replaced by space before showing up on the user interface. Everything else (including the case of the keys and values) remain the same. For example, running the above docker command will show the key and value on the user interface as follows

image

Changing images and favicons

Custom images and favicons can be configured through imagesDir and faviconsDir respectively.

#The white labels.
whiteLabel:
  companyName: Siren
  productName: Investigate
  hideCopyright: false
  management:
    info:
      App Version: 1.4.5
  faviconsDir: /path/to/your/favicons/folder
  imagesDir: /path/to/your/images/folder

To change the images and/or favicons, perform the following steps.

  1. Update imagesDir and/or faviconsDir configurations with a non-empty and accessible directory

  2. Delete the Siren Investigate > optimize folder

  3. Restart the server

The logo images that can be customized are as follows:

Logo Name Dimensions Default (Preview)

logo.png

1026x1021

image

full_logo.png

348x102

image

loading_logo.gif

66x65

image

Favicons folder is expected to have the standard set of favicons files as follows:

Filename

android-chrome-192x192.png

android-chrome-256x256.png

android-chrome-512x512.png

apple-touch-icon.png

browserconfig.xml

favicon-16x16.png

favicon-32x32.png

favicon.ico

manifest.json

mstile-150x150.png

safari-pinned-tab.svg

Ensure that the new image has exactly the same name as the image you are replacing it with. For example, if you change the loading graphic (normally an animated GIF file), ensure that the name of the new GIF file is loading_logo.gif.

Adding icons

Siren provides the ability to import custom icons into the application. The details of how to do this are specified in Adding custom icon packs.

Accessing Siren Investigate

Siren Investigate is a web application that you access through port 5606. All you need to do is point your web browser at the machine where Siren Investigate is running and specify the port number. For example, http://localhost:5606 or http://example.com:5606http://YOURDOMAIN.com:5606.

When you access Siren Investigate, the Discover page loads by default with the default index pattern selected. The time filter is set to the last 15 minutes and the search query is set to match-all (*).

If you do not see any documents, try setting the time filter to a wider time range. If you still do not see any results, it is possible that you do not have any documents.

Checking Siren Investigate status

You can reach the Siren Investigate server’s status page by navigating to http://localhost:5606/status. The status page displays information about the server’s resource usage and lists the installed plugins.

Siren Investigate status page.

Collecting Elasticsearch diagnostics

Click Elasticsearch diagnostics to generate a single file by collecting different metrics about your Elasticsearch cluster. All collected information are saved to a local file and never transferred over a network. You can see a full list of Elasticsearch API calls by clicking the more info (i) icon.

Siren Investigate status page diagnostics help.

For JSON formatted server status details, use the API endpoint at http://localhost:5601/api/status.

Connecting Siren Investigate to backend datasources

Siren can visualize data that is accessible by the Siren backend, which is an Elasticsearch cluster enhanced by the Federate plugin.

The following diagram represents this concept. On the frontend, a user looks at data, for example data in a dashboard or on the Graph Link Analysis system. This data comes from Siren "searches", which reflect queries that are sent to indices that are on the Elasticsearch backend.

It is useful to distinguish between two types of searches:

  • Index Pattern Searches: Root definitions that describe which Elasticsearch indices will be used. They can be as simple as the name of the index itself or can be “patterns” e.g. new*_ to indicate “all the indices that begin with news_”. For more information and examples, see Index Pattern Searches.

  • Other searches: These are defined by filtering an Index Pattern Search, for example starting from a “News” Index Pattern Search, one might create a “News that contain the word Brexit” search.

Inside the main Elasticsearch cluster (the one that the NodeJS Siren application is connected to), indices can be of two types:

  • Physical indices: These are regular Elasticsearch indices, typically created via external ETL (e.g. Logstash) or via the Siren reflection process (an ETL that keeps external tables in sync). For more information, see Data Reflection.

  • Virtual indices: These will simulate Elasticsearch indices but will not copy data; instead, they will send queries to the remote backends and will translate the query results.

Both physical and virtual indices can be used to create index patterns (although in Siren 10.3, you cannot create wildcard patterns (*) including virtual indices).

image

Before you can start using Siren Investigate, you need to tell it which Elasticsearch indices you want to explore. The first time you access Siren Investigate, you are prompted to define an index pattern search that matches the name of one or more of your indices. That is it. That is all you need to configure to start using Siren Investigate. You can create index pattern searches at any time from the Management tab.

By default, Siren Investigate connects to the Elasticsearch instance running on localhost. To connect to a different Elasticsearch instance, modify the Elasticsearch URL in the investigate.yml configuration file and restart Siren Investigate. For information about using Siren Investigate with your production nodes, see Using Siren Investigate in a production environment
  1. Point your browser at port 5606 to access the Siren Investigate user interface. For example, http://localhost:5606 or \http://example.com:5606.

  2. Specify an index pattern search that matches the name of one or more of your Elasticsearch indices. You may have to access the index pattern search management in the Management tab. By default, Siren Investigate guesses that you are working with data being fed into Elasticsearch by Logstash. If that’s the case, you can use the default logstash-* as your index pattern search. The asterisk (*) matches zero or more characters in an index’s name. If your Elasticsearch indices follow some other naming convention, enter an appropriate pattern. The "pattern" can also be the name of a single index.

    image

  3. Select the index field that contains the timestamp that you want to use to perform time-based comparisons. Siren Investigate reads the index mapping to list all the fields that contain a timestamp. If your index does not have time-based data, switch off the  Index contains time-based events option.

  4. Click Create to add the index pattern search. This first pattern is automatically configured as the default. When you have more than one index pattern search, you can designate which one to use as the default by clicking on the star icon above the index pattern search title from Management > Data Model.

All done. Siren Investigate is now connected to your Elasticsearch data. Siren Investigate displays a read-only list of fields configured for the matching index.

Siren Investigate relies on dynamic mapping to use fields in visualizations and manage the .siren index. If you have switched off dynamic mapping, you need to manually provide mappings for the fields that Siren Investigate uses to create visualizations. For more information, see Siren Investigate and Elasticsearch dynamic mapping

Start exploring your data

You are ready to dive in to your data:

  • Search and browse your data interactively from the Discover page.

  • Chart and map your data from the Visualizations page.

  • Create and view custom dashboards from the Dashboard page.

For a step-by-step introduction to these core Siren Investigate concepts, see the Getting Started tutorial.

Siren Investigate and Elasticsearch dynamic mapping

By default, Elasticsearch enables dynamic mapping for fields. Siren Investigate needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .siren index where saved searches, visualizations, and dashboards are stored.

If your Elasticsearch use case requires you to switch off dynamic mapping, you need to manually provide mappings for fields that Siren Investigate uses to create visualizations. You also need to manually enable dynamic mapping for the .siren index.

The following procedure assumes that the .siren index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  1. Start Elasticsearch.

  2. Create the .siren index with dynamic mapping enabled just for that index:

    PUT .siren
    {
      "index.mapper.dynamic": true
    }
  3. Start Siren Investigate and navigate to the web UI and verify that there are no error messages related to dynamic mapping.

Using Siren Investigate with tribe nodes

While tribe nodes have been deprecated in Elasticsearch in favor of Cross-cluster search, you can still use Siren Investigate with tribe nodes until Elasticsearch version 7.0. Unlike tribe nodes, using cross-cluster search in Siren Investigate requires no server-side configurations and does not switch off functionality like Console.

Siren Investigate can be configured to connect to a tribe node for data retrieval. Because tribe nodes cannot create indices, Siren Investigate additionally requires a separate connection to a node to maintain state. When configured, searches and visualizations will retrieve data using the tribe node and administrative actions (such as saving a dashboard) will be sent to non-tribe node.

Configuring Siren Investigate for tribe nodes

Tribe nodes take all of the same configuration options used when configuring Elasticsearch in investigate.yml. Tribe options are prefixed with elasticsearch.tribe and at a minimum requires a URL:

elasticsearch.url: "<your_administration_node>"
elasticsearch.tribe.url: "<your_tribe_node>"

When configured to use a tribe node, actions that modify Siren Investigate’s state will be sent to the node at elasticsearch.url. Searches and visualizations will retrieve data from the node at elasticsearch.tribe.url. It’s acceptable to use a node for elasticsearch.url that is part of one of the clusters that a tribe node is pointing to.

The full list of configurations can be found at Configuring Kibana.

Limitations

Due to the ambiguity of which cluster is being used, certain features are switched off in Siren Investigate:

  • Console

  • Managing users and roles with the X-pack plugin

Using Siren Investigate in a production environment

How you deploy Siren Investigate largely depends on your use case. If you are the only user, you can run it on your local machine and configure it to point to whatever Elasticsearch instance you want to interact with. Conversely, if you have a large number of heavy users, you may need to load balance across multiple instances that are all connected to the same Elasticsearch cluster.

While Siren Investigate is not terribly resource intensive, we still recommend running Siren Investigate separate from your Elasticsearch data or master nodes.

Make sure to set the configuration file as readable only to the user running the Siren Investigate process as it contains encryption keys to protect configuration settings stored in Elasticsearch; if you are connecting Siren Investigate to external datasources, we also recommend to use credentials with no write privileges as they are not required by the system.

Access control

Siren Investigate is compatible with Elastic x-pack and Search Guard to enable index and document level access control.

For more information about access control features, see the Authentication and access control section.

Load balancing across multiple Elasticsearch nodes

If you have multiple nodes in your Elasticsearch cluster, the easiest way to distribute Siren Investigate requests across the nodes is to run an Elasticsearch client node on the same machine as Siren Investigate. Elasticsearch client nodes are essentially smart load balancers that are part of the cluster. They process incoming HTTP requests, redirect operations to the other nodes in the cluster as needed, and gather and return the results. For more information, see Node in the Elasticsearch reference.

To use a local client node to load balance Siren Investigate requests:

  1. Install Elasticsearch on the same machine as Siren Investigate.

  2. Configure the node as a client node. In elasticsearch.yml, set both node.data and node.master to false:

    #    You want this node to be neither master nor data node, but
    #    to act as a "search load balancer" (fetching data from nodes,
    #    aggregating results, and so on)
    #
    node.master: false
    node.data: false
    node.ingest: false
  3. Configure the client node to join your Elasticsearch cluster. In elasticsearch.yml, set the cluster.name to the name of your cluster.

    cluster.name: "my_cluster"
  4. Make sure Siren Investigate is configured to point to your local client node. In investigate.yml, the elasticsearch.url should be set to http://localhost:9220.

    # The Elasticsearch instance to use for all your queries.
    elasticsearch.url: "http://localhost:9220"

Upgrading from Investigate 10.1.x or 10.2.x

You can migrate directly from Investigate 10.1.3 and higher to 10.3.0. If upgrading from an earlier version of Investigate, it is recommended that you first upgrade to 10.1.3 (which also supports Elasticsearch 5.x). You can then upgrade from 10.1.3 to 10.3.0.

When upgrading both Investigate and Elasticsearch, you should upgrade the Investigate objects first, and only then upgrade Elasticsearch.

The recommended version of Elasticsearch is 6.8.0 (version 6.5.4 is also supported). See Search Guard Integration and Siren Investigate access control to configure Searchguard security settings correctly for Elasticsearch 6.8.0.

Before you upgrade Siren Investigate, you must take account of the security implications of the new features installed. Details of the necessary security procedures can be found in Security standardization.

An existing Siren Investigate installation can be upgraded as follows:

  1. Back up the 10.1.x .siren index.

  2. Back up the Siren Investigate configuration file (config/investigate.yml).

  3. Back up the .sirenaccess if ACL (Access Control Layer) is enabled

  4. If you are running Elasticsearch 6.3.2, you can optionally upgrade to version 6.8.0 (https://www.elastic.co/guide/en/elasticsearch/reference/6.5/setup-upgrade.html).

  5. Before restarting each Elasticsearch node, ensure you install a compatible version of the Siren Federate plugin and access control plugins (Searchguard or X-pack security) if required.

  6. Download and extract the new Siren Investigate version.

  7. Copy the previous configuration file to the config folder of the new installation.

  8. Check for breaking changes to the configuration. See Search Guard Integration and Siren Investigate access control.

  9. Install the compatible versions of third party Siren Investigate/Kibana plugins that you may need into the /plugins folder.

  10. Execute the upgrade command.

Elasticsearch requires a matching version of Search Guard. For example, Elasticsearch 6 requires Search Guard 6. For information on upgrading Search Guard, see https://docs.search-guard.com/latest/upgrading-560.

Backing up and restoring the Siren Investigate indices

Before upgrading, you should have a backup of the .siren index; the recommended way to perform regular backups of Elasticsearch indexes is through the snapshot and restore modules.

Siren Investigate ships with a command line interface for creating dumps of the .siren index and, in case the ACL is enabled, the .sirenaccess index as well. An index dump is composed of two parts: its mappings and its data.

Backup

The backup command requires a running Elasticsearch instance and the path to a folder where the dumps will be written to.

You can find out more about its options by executing the following:

$ ./bin/investigate backup --help

For example, the following line will dump in <MY_FOLDER> the .siren index and the .sirenaccess index if the option investigate_access_control.acl.enabled is true in investigate.yml:

$ ./bin/investigate backup --backup-dir <MY_FOLDER>

Restore

The restore command requires a running Elasticsearch instance and the path to a folder where the dumps were written to by the previous backup command.

You can find out more about its options by executing the following:

$ ./bin/investigate restore --help

For example, you can restore the previously saved indices by executing the command and pointing to the dump folder, with .sirenaccess as well if the option investigate_access_control.acl.enabled is true in investigate.yml:

$ ./bin/investigate restore --backup-dir <MY_FOLDER>

Upgrading the Siren index

To upgrade the objects in the .siren index (dashboards, visualizations, and so on), move to the folder in which Siren Investigate is installed and execute the following command:

bin/investigate upgrade

The command will look for out of date objects and upgrade them, for example:

$ bin/investigate upgrade
  log   [17:58:33.494] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [17:58:36.127] [info][migrations] Executing migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.141] [info][migrations] Executed migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.142] [info][migrations] Executing migration "Upgrade graph browser visualization to version 2."
  log   [17:58:36.157] [info][migrations] Executed migration "Upgrade graph browser visualization to version 2."
  log   [17:58:36.158] [info][migrations] Executing migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executed migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executing migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executed migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executing migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
  log   [17:58:36.400] [info][migrations] Executed migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
Upgraded 20 objects.

It is possible to run the command multiple times, however running the command at the same time from multiple machines is not supported.

The upgrade command runs an automatic backup of the siren indices (.siren, .sirenaccess) and restores them (after deleting the existing index) in the event of a problem in the upgrade process - ensuring the system is not left in an unusable state. In the event of a successful upgrade, the backup is removed but if there is an issue, the backed up indexes are stored in the backup folder (defaults to the /data folder).

You can specify various flags to control the backup/restore process.

  • --backup-dir <path>: Custom backup folder path to store the index backup.

  • --config <path>: Path to the configuration file.

  • --delete-backup: Deletes the backup of the indexes after the upgrade process completes.

  • --dont-backup: Runs the upgrade process without creating a backup of the indexes.

  • -y: Accepts all of the options, for example, backup the indexes and remove the indexes before restoring.

Security standardization

The purpose of security standardization is to provide standard security configuration for all possible backends, and full support for Investigate 10.3.0.

This entails standardizing both action groups and role mappings.

Procedure

To standardize your security according to new action groups:

  • Download the applicable security bundle below.

  • Add any new client-specific roles or action groups you have added (if any) based on the action group and action name changes listed below.

  • Put the modified configuration files in the ./elasticsearch/config/sgconfig/ folder and upload the configuration to the cluster.

  • Ensure that you explicitly specify your ACL admin_role and Sentinl user_role in investigate.yml as defaults may have changed.

Standard sgconfig bundles

The following sgconfig bundles are available:

Action Group changes

Removed Can be (loosely) replaced by

ALL

INDICES_ALL

CREATE_INDEX

WRITE, MANAGE

MANAGE_ALIASES

VIEW_INDEX_METADATA

MONITOR

MANAGE

INDICES_MONITOR

VIEW_INDEX_METADATA

DATA_ACCESS

WRITE, READ, VIEW_INDEX_METADATA

DELETE

WRITE

CRUD

READ, WRITE

SEARCH

READ

SUGGEST

READ

INDEX

WRITE

GET

READ

SIREN_READONLY

READ, VIEW_INDEX_METADATA

SIREN_READWRITE

READ, WRITE, VIEW_INDEX_METADATA

SIREN_CLUSTER

CLUSTER_MANAGE, CLUSTER_MONITOR, CLUSTER_COMPOSITE_OPS_RO

Elasticsearch Action Name Changes

ES 6.3.x onwards:

Index-level actions:

  • indices:data/write/bulk to → indices:data/write/bulk[s] or you may just use indices:data/write/bulk*.

  • indices:data/read/coordinate-msearch* deprecated (not used by Siren).

Federate Action Name Changes

Federate 10.2.0 onwards:

Index-level actions:

  • indices:siren/plan* to → indices:data/read/federate/planner/search*

  • indices:siren/mplan* to → indices:data/read/federate/planner/msearch*

  • indices:data/siren/connector/mappings/get* to → indices:admin/mappings/federate/connector/get*

  • indices:data/siren/connector/mappings/fields/get* to → indices:admin/mappings/federate/connector/fields/get*

Cluster-level actions:

  • indices:siren/plan* to → indices:data/read/federate/planner/search*

  • indices:siren/mplan* to → indices:data/read/federate/planner/msearch*

  • cluster:siren/internal* to → cluster:internal/federate/*

  • cluster:admin/plugin/siren/license/get to → cluster:admin/federate/license/get, or you may just use cluster:admin/federate/*

  • indices:data/siren/connector/* to → indices:data/read/federate/*, indices:admin/federate/* and cluster:admin/federate/*