Management
The Management application is where you perform your run time configuration of Siren Investigate, including both the initial setup and ongoing configuration of index patterns, advanced settings that tweak the behaviors of Siren Investigate itself, and the various "objects" that you can save throughout Siren Investigate such as entity tables and searches, visualizations, and dashboards.
Entity tables
To use Siren Investigate, you have connect to the Elasticsearch indices that you want to explore, by configuring one or more entity tables.
For more information, see Creating entity tables.
You can also:
-
Create scripted fields that are computed on the fly from your data. You can browse and visualize scripted fields, but you cannot search them.
-
Set advanced options such as the number of rows to show in a table and how many of the most popular fields to show. Note: Use caution when modifying advanced options, as it is possible to set values that are incompatible with one another.
-
Configure Siren Investigate for a production environment.
Cross-cluster search
Elasticsearch supports the ability to run search and aggregation requests across multiple clusters using a module called cross-cluster search.
Siren Federate does not currently support cross-cluster search. |
To take advantage of cross-cluster search, you must configure your Elasticsearch clusters accordingly. Refer to the corresponding Elasticsearch documentation before attempting to use cross-cluster search in Siren Investigate.
After your Elasticsearch clusters are configured for cross-cluster
search, you can create specific index patterns in Siren Investigate to
search across the clusters of your choosing. Using the same syntax that
you would use in a raw cross-cluster search request in Elasticsearch,
create your index pattern in Siren Investigate with the convention
<cluster-names>:<pattern>
.
For example, if you want to query logstash indices across two of the
Elasticsearch clusters that you set up for cross-cluster search, which
were named cluster_one
and cluster_two
, you would use
cluster_one:logstash-*,cluster_two:logstash-*
as your index pattern
in Siren Investigate.
Just like in raw search requests in Elasticsearch, you can use wildcards
in your cluster names to match any number of clusters, so if you wanted
to search logstash indices across any clusters named cluster_foo
,
cluster_bar
, and so on, you would use cluster_*:logstash-*
as
your index pattern in Siren Investigate.
If you want to query across all Elasticsearch clusters that have been
configured for cross cluster search, then use a standalone wildcard for
your cluster name in your Siren Investigate index pattern:
*:logstash-*
.
After an index pattern is configured using the cross-cluster search syntax, all searches and aggregations using that index pattern in Siren Investigate take advantage of cross-cluster search.
Advanced settings for relations
From the Relations tab of the Data model app, click Edit (the pencil icon) next to a relation and then click on Advanced settings (the gear icon) to edit the advanced settings for each relation.
Click on Save relation (the blue button) to save the relation.
The following settings are available:
-
Join type: You can choose an automatic join type or select from a hash join, a broadcast join, or an index join.
-
Relation join task timeout: You can set the maximum time spent by each join task for that relation in milliseconds. After the timeout has expired, the task passes the documents accumulated at that point on to the next task.
This is a per-task time limit and as each join contains several tasks, the overall response to the request can be a number of multiples of the joinTaskTimeout
.
As a semi-join, these documents will be filtered based on the presence of a non-empty value for the join
field in the other index pattern in the relation.
The index pattern in question is then filtered by the values returned.
Setting the limit here to -1
here sets the limit to the default siren:joinTaskTimeout
set in the Advanced Settings and setting the limit to 0
here removes the limit entirely.
Join types
Siren Federate provides four types of join algorithms. The plugin tries to pick the best algorithm for a given join automatically. However, you can force the selection by choosing one of the available options:
-
HASH_JOIN
-
BROADCAST_JOIN
-
INDEX_JOIN
-
ROUTING_JOIN
A detailed description of each algorithm can be found in the Siren Federate plugin documentation.
Minimizing expensive queries
Dashboards connected to large datasets can produce queries that are expensive to process if the operation on the data does not limit the data being queried, for example, when a filter is removed or a broad time range is set.
To help prevent this, some limits can be set on the entity tables and the associated dashboards.
To set the limits, go to the Data model app, select the entity table or search to limit, then go to the Options tab.
For example, the time range of the search might be limited to the last thirty days. An associated dashboard can only be navigated to if there are fewer than five thousand documents, but there are no limits on the number of documents that the dashboard can display.
Users can be permitted to override warnings by setting the investigate override expensive queries limits
permission in the
ACL
plugin. If permitted, the warnings will still be displayed but the user will be able to explicitly allow the operation.
Limit time range
For time based indices, the maximum time range can be defined. Setting a longer time range will be prevented by the system and a warning will be displayed.
Limit the maximum number of documents per dashboard
The maximum number of documents that can be displayed on a dashboard can be set. If the filters are changed to show a larger number of documents, the system will not allow the operation and show a warning.
Limit the maximum number of documents on the dashboards in a join operation
If you attempt to make a join between two dashboards by using the relational navigator and the number of documents on one or both of the dashboards exceeds the limit, the navigation is prevented and a warning message is displayed.
Datasources
For an overview of datasources, see Siren Investigate datasource configuration.
Managing fields
The fields for the index pattern are listed in a table. Click a column header to sort the table by that column. Click Controls in the rightmost column for a given field to edit the field’s properties. You can manually set the field’s format from the Format box. Format options vary based on the field’s type.
You can also set the field’s popularity value in the Popularity text entry box to any desired value. Click Update Field to confirm your changes or Cancel to return to the list of fields.
Siren Investigate has field formatters for the following field types:
String field formatters
String fields support the String
and URL
formatters.
The String
field formatter can apply the following transformations
to the field’s contents:
-
Convert to lowercase.
-
Convert to uppercase.
-
Convert to title case.
-
Apply the short dots transformation, which replaces the content before a
.
character with the first character of that content, as in the following example:
Original |
Becomes |
|
|
The URL
field formatter can take on the following types:
-
The Link type turn the contents of the field into a URL.
-
The Image type can be used to specify an image folder where a specified image is located.
You can customize either type of URL field formats with templates. A
URL template enables you to add specific values to a partial URL. Use
the string {{value}}
to add the contents of the field to a fixed
URL.
For example, when:
-
A field contains a user ID.
-
That field uses the
URL
field formatter. -
The URI template is
http://company.net/profiles?user_id={{value}}
.
The resulting URL replaces {{value}}
with the user ID from the
field.
The {{value}}
template string URL-encodes the contents of the field.
When a field encoded into a URL contains non-ASCII characters, these
characters are replaced with a %
character and the appropriate
hexadecimal code. For example, field contents users/admin
result in
the URL template adding users%2Fadmin
.
When the formatter type is set to Image, the {{value}}
template
string specifies the name of an image at the specified URI.
To pass unescaped values directly to the URL, use the {{rawValue}}
string.
A Label Template enables you to specify a text string that displays
instead of the raw URL. You can use the {{value}}
template string
normally in label templates. You can also use the {{url}}
template
string to display the formatted URL.
Date field formatters
Date fields support the Date
, Url
, and String
formatters.
The Date
formatter enables you to choose the display format of date
stamps using the moment.js standard format
definitions.
The String
field formatter can apply the following transformations
to the field’s contents:
-
Convert to lowercase
-
Convert to uppercase
-
Convert to title case
-
Apply the short dots transformation, which replaces the content before a
.
character with the first character of that content, as in the following example:
Original |
Becomes |
|
|
The URL
field formatter can take on the following types:
-
The Link type turn the contents of the field into a URL.
-
The Image type can be used to specify an image folder where a specified image is located.
You can customize either type of URL field formats with templates. A
URL template enables you to add specific values to a partial URL. Use
the string {{value}}
to add the contents of the field to a fixed
URL.
For example, when:
-
A field contains a user ID.
-
That field uses the
URL
field formatter. -
The URI template is
http://company.net/profiles?user_id={{value}}
.
The resulting URL replaces {{value}}
with the user ID from the
field.
The {{value}}
template string URL-encodes the contents of the field.
When a field encoded into a URL contains non-ASCII characters, these
characters are replaced with a %
character and the appropriate
hexadecimal code. For example, field contents users/admin
result in
the URL template adding users%2Fadmin
.
When the formatter type is set to Image, the {{value}}
template
string specifies the name of an image at the specified URI.
To pass unescaped values directly to the URL, use the {{rawValue}}
string.
A Label Template enables you to specify a text string that displays
instead of the raw URL. You can use the {{value}}
template string
normally in label templates. You can also use the {{url}}
template
string to display the formatted URL.
Geographic point field formatters
Geographic point fields support the String
formatter.
The String
field formatter can apply the following transformations
to the field’s contents:
-
Convert to lowercase
-
Convert to uppercase
-
Convert to title case
-
Apply the short dots transformation, which replaces the content before a
.
character with the first character of that content, as in the following example:
Original |
Becomes |
|
|
Numeric field formatters
Numeric fields support the URL
, Bytes
, Duration
, Number
,
Percentage
, String
, and Color
formatters.
The URL
field formatter can take on the following types:
-
The Link type turn the contents of the field into a URL.
-
The Image type can be used to specify an image folder where a specified image is located.
You can customize either type of URL field formats with templates. A
URL template enables you to add specific values to a partial URL. Use
the string {{value}}
to add the contents of the field to a fixed
URL.
For example, when:
-
A field contains a user ID
-
That field uses the
URL
field formatter -
The URI template is
http://company.net/profiles?user_id={{value}}
The resulting URL replaces {{value}}
with the user ID from the
field.
The {{value}}
template string URL-encodes the contents of the field.
When a field encoded into a URL contains non-ASCII characters, these
characters are replaced with a %
character and the appropriate
hexadecimal code. For example, field contents users/admin
result in
the URL template adding users%2Fadmin
.
When the formatter type is set to Image, the {{value}}
template
string specifies the name of an image at the specified URI.
To pass unescaped values directly to the URL, use the {{rawValue}}
string.
A Label Template enables you to specify a text string that displays
instead of the raw URL. You can use the {{value}}
template string
normally in label templates. You can also use the {{url}}
template
string to display the formatted URL.
The String
field formatter can apply the following transformations
to the field’s contents:
-
Convert to lowercase
-
Convert to uppercase
-
Convert to title case
-
Apply the short dots transformation, which replaces the content before a
.
character with the first character of that content, as in the following example:
Original |
Becomes |
|
|
The Duration
field formatter can display the numeric value of a
field in the following increments:
-
Picoseconds
-
Nanoseconds
-
Microseconds
-
Milliseconds
-
Seconds
-
Minutes
-
Hours
-
Days
-
Weeks
-
Months
-
Years
You can specify these increments with up to 20 decimal places for both input and output formats. The default number of decimals for the Number format is 3, i.e. 0,0.[000]. If there are values smaller than this, but larger than 1e-7, they will be rounded to 0. The fix is to change the Numeral.js format pattern to: 0,0.[0000000]
The Color
field formatter enables you to specify colors with
specific ranges of values for a numeric field.
When you select the Color
field formatter, Siren Investigate
displays the Range, Font Color, Background Color, and Example
fields.
Click Add Color to add a range of values to associate with a particular color. You can click in the Font Color and Background Color fields to display a color picker. You can also enter a specific hex code value in the field. The effect of your current color choices are displayed in the Example field.
The Bytes
, Number
, and Percentage
formatters enable you to
choose the display formats of numbers in this field using the
numeral.js standard format
definitions.
Scripted fields
Scripted fields compute data on the fly from the data in your Elasticsearch indices. Scripted field data is shown on the Discover tab as part of the document data, and you can use scripted fields in your visualizations. Scripted field values are computed at query time so they are not indexed and cannot be searched. Note that Siren Investigate cannot query scripted fields.
Computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Siren Investigate’s performance. Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you will get exceptions whenever you try to view the dynamically generated data. |
When you define a scripted field in Siren Investigate, you have a choice of scripting languages. Starting with 5.0, the default options are Lucene expressions and Painless. While you can use other scripting languages if you enable dynamic scripting for them in Elasticsearch, this is not recommended because they cannot be sufficiently sandboxed.
You can reference any single value numeric field in your expressions, for example:
doc['field_name'].value
For more background on scripted fields and additional examples, refer to Using Painless in Kibana scripted fields.
Creating a scripted field
-
Go to Settings > Indices.
-
Select the index pattern you want to add a scripted field to.
-
Go to the pattern’s Scripted Fields tab.
-
Click Add Scripted Field.
-
Enter a name for the scripted field.
-
Enter the expression that you want to use to compute a value on the fly from your index data.
-
Click Save Scripted Field.
For more information about scripted fields in Elasticsearch, see Scripting.
Modifying a scripted field
-
Go to Settings > Indices
-
Click Edit for the scripted field you want to change.
-
Make your changes and then click Save Scripted Field to update the field.
Deleting a scripted field
-
Go to Settings > Indices.
-
Click Delete for the scripted field you want to remove.
-
Confirm that you really want to remove the field.
Setting advanced options
The Advanced Settings page enables you to directly edit settings that control the behavior of the Siren Investigate application. For example, you can change the format used to display dates, specify the default index pattern, and set the precision for displayed decimal values.
-
Go to
. -
Click Edit for the option you want to modify.
-
Enter a new value for the option.
-
Click Save.
Modifying the following settings can significantly affect Siren Investigate’s performance and cause problems that are difficult to diagnose. Setting a property’s value to a blank field will revert to the default behavior, which may not be compatible with other configuration settings. Deleting a custom setting removes it from Siren Investigate permanently. |
Name | Description | Example |
---|---|---|
|
Enable experimental features in Siren Alert. |
|
|
Sets the default highlighter type for highlighted query matches. Siren Investigate applies Lucene highlighter methods to compute how the highlighting is applied to the text. The two types of highlighter are |
|
|
Options for the Lucene query string parser. |
|
|
|
|
|
The format to use for displaying formatted dates. |
|
|
The timezone that Siren Investigate uses. The
default value of |
|
|
These values define the format used to render ordered time-based data. Formatted timestamps must adapt to the interval between measurements. Keys are ISO8601 intervals. |
|
|
This property defines what day weeks should start on. |
|
|
Default is |
|
|
An array of fields outside of |
|
|
The number of rows to show in the Discover table. |
|
|
Determines how many terms will be
visualized when clicking the "visualize" button, in the field boxes, in
the discover sidebar. The default value is |
|
|
Highlight results in Discover and Saved
Searches Dashboard. Highlighting makes request slow when working on big
documents. Set this property to |
|
|
Improves highlighting by using a
separate |
|
|
Siren Investigate splits requests in the Discover page into segments to limit the size of requests sent to the Elasticsearch cluster. This setting constrains the length of the segment list. Long segment lists can significantly increase request processing time. |
|
|
Set this property to
|
|
|
This setting governs how many of the top most popular fields are shown. |
|
|
When date histograms use the |
|
|
Date histograms are not generated with more bars than the value of this property, scaling values when necessary. |
|
|
The maximum geohash precision displayed on tile maps: 7 is high, 10 is very high, 12 is the maximum. Explanation of cell dimensions. |
|
|
Default properties for the WMS map server support in the coordinate map. |
|
|
Whether the region map shows a warning when terms cannot be joined to a shape on the map. |
|
|
Maps values to specified colors within visualizations. |
|
|
Time to wait before dimming visualizations during query. |
|
|
When part of a visualization is highlighted, by moving the mouse pointer over it for example, this is the opacity applied to the other elements. A higher number means other elements will be less opaque. |
|
|
A string that serves as the separator for exported values. |
|
|
Set this property to |
|
|
In fields that have history, such as query inputs, the value of this property limits how many recent values are shown. |
|
|
Set this property to |
|
|
This property specifies the maximum height that a cell occupies in a table. A value of 0 switches off truncation. |
|
|
A map of the default format name for each field type. Field types that are not explicitly mentioned use "default". |
|
|
Default numeral format for the "number" format. |
|
|
Default http://numeraljs.com/[numeral format] numeral format for the "bytes" format. |
|
|
Default http://numeraljs.com/[numeral format] numeral format for the "percent" format. |
|
|
Default http://numeraljs.com/[numeral format] numeral format for the "currency" format. |
|
|
The number of objects shown on each page of the list of saved objects. The default value is 5. |
|
|
Number of objects to fetch for the listing pages. |
|
|
The default time filter selection. |
|
|
The time filter’s default refresh interval. |
|
|
Set this property to |
|
|
Set this property to |
|
|
Set this property to |
|
|
You can specify a custom banner to display temporary notices to all users. This field supports Markdown. |
|
|
Specifies the duration in
milliseconds for banner notification displays. The default value is
3000000. Set this field to |
|
|
Specifies the duration in
milliseconds for error notification displays. The default value is
300000. Set this field to |
|
|
Specifies the duration in
milliseconds for warning notification displays. The default value is
10000. Set this field to |
|
|
Specifies the duration in milliseconds
for information notification displays. The default value is 5000. Set
this field to |
|
|
The maximum numbers of buckets that cannot be exceeded. For example, this can arise when the user selects a short interval like (for example 1s) for a long time period (for example 1 year). |
|
|
[experimental] Siren Investigate tracks UI state in the URL, which can lead to problems when there is a lot of information there and the URL gets very long. Enabling this will store parts of the state in your browser session instead, to keep the URL shorter. |
|
|
The placeholder for the field "Index name or pattern" in the "Settings > Indices" tab. |
|
|
The number of surrounding entries to show in the context view. |
|
|
The step size to increment or decrement the context size by. |
|
|
A comma-separated list of fields to use for tie breaking between documents that have the same timestamp value. From this list the first field that is present and sortable in the current index pattern is used. |
|
|
Set this property to |
|
|
Default field containing a timestamp when
using the |
|
|
Default index when using the |
|
|
Used for calculating automatic intervals in visualizations, this is the number of buckets to try to represent. |
|
|
Used for calculating automatic intervals in visualizations, this is the maximum number of buckets to represent. |
|
|
The default number of columns to use on a Timelion sheet. |
|
|
The default number of rows to use on a Timelion sheet. |
|
Name | Description | Example |
---|---|---|
|
Set to generate time filters with certain precision; possible values are: y, M, w, d, h, m, s, ms. It is set to m (minute) by default, to make the best use of Federate cache on time-based data. However, if the data is updated live and better precision is needed, it can be set to s (second) or ms (millisecond). |
|
|
Default timeout for join task in milliseconds. Join tasks will return the results gathered at that point when the timeout expires. Set to 0 to disable the global timeout. Can be overwritten per relation in each relation’s advanced options in the relational panel. |
|
|
Set to change the default vertical panel size. |
|
|
Set to change vertical grid resolution. |
|
|
Enable counts on all relational buttons. |
|
|
The dashboard that is displayed when clicking the Dashboard tab for the first time. |
|
|
A comma separated list of indices / patterns to exclude when performing searches against wildcard patterns. |
|
|
Set how many undo/redo steps you want to maintain in memory |
|
|
Limit the number of elements to retrieve during the graph expansion. |
|
|
Limit the number of concurrent calls done by the Graph Browser. |
|
|
Strategy for fetching dashboard counts. The parameters instruct the count manager how to issue queries to Elasticsearch; this is used to improve performance when indices contain very large numbers of documents. |
|
|
Strategy for fetching counts for relational filters. Parameters are the same as for |
|
|
Enable introductory videos. |
|
|
Return stack_trace in search or msearch error responses if true. |
|
|
Milliseconds reserved for computing a single Fingerprints/Relations Wizard request. Requests will return the results gathered at that point when the timeout expires, possibly leading to suboptimal overall results. It does not apply to virtual indices. |
|
Managing entity tables, visualizations, and dashboards
You can view, edit, and remove entity tables, visualizations, and dashboards from Settings > Objects. You can also export or import sets of entity tables, visualizations, and dashboards.
Viewing a saved object displays the selected item in the Discover, Visualize, or Dashboard page. To view a saved object:
-
Go to Management > Saved Objects.
-
Select the object you want to view.
-
Click the View button.
Editing a saved object enables you to directly modify the object definition. You can change the name of the object, add a description, and modify the JSON that defines the object’s properties.
If you attempt to access an object whose index has been removed, Siren Investigate displays its Edit Object page. You can:
-
Recreate the index so you can continue using the object.
-
Remove the object and recreate it using a different index.
-
Change the index pattern to match new index by editing
indexPattern.pattern
to point to an existing index pattern. This is useful if the index you were working with has been renamed.
No validation is performed for object properties. Submitting invalid changes will render the object unusable. Generally, you should use the Discover, Visualize, or Dashboard pages to create new objects instead of directly editing existing ones. |
To edit a saved object:
-
Go to
. -
Select the object you want to edit.
-
Click the Edit button.
-
Make your changes to the object definition.
-
Click the Save Object button.
To remove a saved object:
-
Go to
. -
Select the object you want to remove.
-
Click Delete.
-
Confirm that you really want to remove the object.
To export a set of objects:
-
Go to
. -
Select the type of object you want to export. You can export a set of dashboards, entity tables, or visualizations.
-
Click the selection box for the objects you want to export, or click the Select All box.
-
To export the selected objects without their dependant saved objects, click Export. To attach the dependent saved object(s) click Export with dependencies.
-
Select a location to write the exported JSON.
If dashboards are exported without dependencies, they do not include their associated index patterns, visualizations or any other objects. Recreate these objects manually before importing saved dashboards to a Siren Investigate instance running on another Elasticsearch cluster. To include all objects, click Export with dependencies. |
To import a set of objects that doesn’t exist in your connected Elasticsearch cluster:
-
Go to
. -
Click Import to navigate to the JSON file representing the set of objects to import.
-
Click Open after selecting the JSON file.
To import a set of objects that already exist in Elasticsearch cluster:
-
Go to
. -
Click Import to navigate to the JSON file representing the set of objects to import.
-
Click Open after selecting the JSON file.
-
In the Duplicates Saved Objects Found modal, the following options can be chosen:
-
Overwrite all existing objects with imported objects - This would update all objects in the Elasticsearch with the imported objects
-
Do not overwrite existing objects - This would ignore the duplicate objects from the imported set of objects. Only the new objects will be imported.
-
Decide for each object - Here we can decide for each duplicate object from the imported set of objects whether to choose the Existing object in the Elasticsearch or the Imported Object.
-
-
Click Confirm to import the selected objects.
Adding custom icon packs
You can import custom icons into the application, see Adding custom icons.