Search APIs

Siren Federate introduces the following new search actions:

  • /siren/<INDEX>/_search replaces the /<INDEX>/_search Elasticsearch action; and

  • /siren/<INDEX>/_msearch replaces the /<INDEX>/_msearch Elasticsearch action.

Both actions are extensions of the original Elasticsearch actions and therefore support the same API.

You must use these actions with the join query clause, as the join query clause is not supported by the original Elasticsearch actions.

Permissions: To use the APIs that are listed in this section, ensure that the cluster-level wildcard action cluster:internal/federate/* is granted by the security system.

Search API

The search API allows you to execute a search query and get back search hits that match the query.

Request

curl -XGET 'http://localhost:9200/siren/<INDEX>/_search'

curl -XPOST 'http://localhost:9200/siren/<INDEX>/_search'

curl -XGET 'http://localhost:9200/siren/_search'

curl -XPOST 'http://localhost:9200/siren/_search'

Path parameter

<index>

(Optional, string) Comma-separated list or wildcard expression of index names used to limit the request.

Permissions: To use this API, ensure that the index-level wildcard action indices:data/read/federate/search* and the indices:data/read/federate/planner/search action are granted by the security system.

Multi Search API

The multi search API allows to execute several search requests within the same API.

Request

curl -XGET 'http://localhost:9200/siren/<INDEX>/_msearch'

curl -XPOST 'http://localhost:9200/siren/<INDEX>/_msearch'

curl -XGET 'http://localhost:9200/siren/_msearch'

curl -XPOST 'http://localhost:9200/siren/_msearch'

Path parameter

<index>

(Optional, string) Comma-separated list or wildcard expression of index names used to limit the request.

Permissions: To use this API, ensure that the index-level wildcard action indices:data/read/federate/search* and the indices:data/read/federate/planner/msearch action are granted by the security system.

Search Request

The syntax for the body of the search request is identical to the one supported by the Elasticsearch search API, with the additional support for the join query clause in the Query DSL.

Parameters

In addition to the parameters supported by the Elasticsearch search API, the Federate search API introduces the following additional parameters:

task_timeout

A task timeout, bounding a task to be executed within the specified time value (in milliseconds) and returns with the values accumulated up to that point when expired. Defaults to no timeout (-1).
debug

To retrieve debug information from the query planner. Defaults to false.

ignore_query_cache

To bypass the query cache. When set to true, the planner will not attempt to retrieve results from the query cache. Defaults to false.

siren-federate-features

To enable federate features during a join request. Available options are: AGG_PUSH_QUERY, to push down query clauses for joins in the aggregation context; AGG_PUSH_ANCESTORS, to push down filter clauses found in the ancestors aggregation buckets and AGG_PUSH_NONE, to do not push down any filter. Default behaviour is AGG_PUSH_QUERY and AGG_PUSH_ANCESTORS.

Taking advantage of the join query cache

The join query cache is responsible for caching the results of a join query clause at the shard level. If an index has one or more replicas, it is recommended that you specify the preference parameter of the search request.

If no preference parameter is specified, the search request is processed against a random selection of shards. In such a scenario, the join query cache on every shard may differ and the chance of having a positive cache hit decreases.

For example, it is common practice to specify a user session ID as preference, so that the same set of shards are selected across the search requests of a same user.

Search Response

The response returned by Federate’s search API is similar to the response returned by Elasticsearch’s search API. It extends the response with a planner object which includes information about the query plan execution.

Example 1. Search response example
{
    "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 5,
        "total": 5
    },
    "hits": {
        "hits": [],
        "max_score": 0.0,
        "total": 0
    },
    "planner": {
        "node": "AYex2HdPTu-cwkqwaquH1w",
        "is_truncated": true, (1)
        "took_in_millis": 558 (2)
        "timestamp": { (3)
            "start_in_millis": 1728902627309, (4)
            "stop_in_millis": 1728902627867, (5)
            "took_in_millis": 558, (2)
            "job_waiting": { (6)
                "took_in_millis": 1,
                "start_in_millis": 1728902627309,
                "stop_in_millis": 1728902627310
            },
            "planning": { (7)
                "took_in_millis": 235,
                "start_in_millis": 1728902627423,
                "stop_in_millis": 1728902627658
            },
            "execution": { (8)
                "took_in_millis": 209,
                "start_in_millis": 1728902627658,
                "stop_in_millis": 1728902627867
            }
        },
        "query_plan": {  (9)
            "children": [
                {
                    "failures": [
                        {
                            "reason": "[ROOT allocator] Unable to allocate a buffer of size 1mb due to the allocator's limit of 1023.2mb. Current allocation: 1023.2mb - Allocator(child) 0/1072955392/1072955400/1072956000 (res/actual/peak/limit)",
                            "type": "out_of_memory_exception"
                        }
                    ],
                    "type": "SearchTaskBroadcastRequest"
                }
            ],
            "type": "SearchJoinRequest"
        }
    },
    "timed_out": false, (10)
    "took": 558 (11)
}
1 The request response may have been truncated for several reasons and the flag is_truncated indicates that the search results are incomplete in the following cases:
2 Planner request overall duration (in milliseconds)
3 Planner request detailed report section
4 Planner request creation time
5 Planner request completion time
6 Planner request waiting time report (queuing time)
7 Time required to generate an execution plan
8 Time required to execute all the planned tasks
9 When debug parameter is enabled, detailed information and statistics about the query plan execution will be returned within a query_plan node. If the debug parameter was disabled and the response was truncated, then a simplified query plan is displayed with information detailing the causes of the truncation
10 True if the processing of the search request timed out
11 Planner request overall duration (in milliseconds), including sub-tasks executed during the evaluation of joins.

Multi Search Response

Differently from the (single) response, the multi search response contains an array of responses, one for each search contained in the request.

Example 2. Multi Search reponse example
{
  "responses": [
    {
      "took": 38,
      "timed_out": false,
      "status": 200, (1)
      "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 5,
        "total": 5
      },
      "hits": {
        "hits": [],
        "max_score": 0.0,
        "total": 0
      },
      "planner": {
        "node": "HvBlvfIQTqmW_B9f7kXuCw",
        "took_in_millis": 535,
        "is_truncated": false,
        "type": "adaptive",
        "timestamp": {
          "start_in_millis": 1728903224688,
          "stop_in_millis": 1728903225223,
          "took_in_millis": 535,
          "job_waiting": {
            "took_in_millis": 2,
            "start_in_millis": 1728903224688,
            "stop_in_millis": 1728903224690
          },
          "planning": {
            "took_in_millis": 281,
            "start_in_millis": 1728903224688,
            "stop_in_millis": 1728903224969
          },
          "execution": {
            "took_in_millis": 254,
            "start_in_millis": 1728903224969,
            "stop_in_millis": 1728903225223
          }
        }
      }
    },
    {
      "took": 38,
      "timed_out": false,
      "status": 200, (1)
      "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 5,
        "total": 5
      },
      "hits": {
        "hits": [],
        "max_score": 0.0,
        "total": 0
      },
      "planner": {
        "node": "HvBlvfIQTqmW_B9f7kXuCw",
        "took_in_millis": 536,
        "is_truncated": false,
        "type": "adaptive",
        "timestamp": {
          "start_in_millis": 1728903224688,
          "stop_in_millis": 1728903225224,
          "took_in_millis": 536,
          "job_waiting": {
            "took_in_millis": 2,
            "start_in_millis": 1728903224688,
            "stop_in_millis": 1728903224690
          },
          "planning": {
            "took_in_millis": 281,
            "start_in_millis": 1728903224688,
            "stop_in_millis": 1728903224969
          },
          "execution": {
            "took_in_millis": 255,
            "start_in_millis": 1728903224969,
            "stop_in_millis": 1728903225224
          }
        }
      }
    }
  ],
  "job_waiting": { (2)
    "took_in_millis": 2,
    "start_in_millis": 1728903224550,
    "stop_in_millis": 1728903224552
  },
  "job_total": {  (3)
    "took_in_millis": 782,
    "start_in_millis": 1728903224552,
    "stop_in_millis": 1728903225254
  }

}
1 the HTTP response status of each response
2 Multi-planner request waiting time report (a.k.a. queuing time)
3 Multi-planner request overall duration time report

Cancelling a request

A search or a multi search request can be cancelled explicitly by a user. In order to do so, you need to pass a X-Opaque-Id header which is used to identify the request. The endpoint for cancelling a request is /_siren/job/<ID>/_cancel. By default, the cancel request will wait for all tasks associated to the search to be cancelled. This can be disabled by passing false to the boolean parameter wait_for_completion.

Permissions: To use this API, ensure that the cluster-level action cluster:admin/federate/job/cancel is granted by the security system.

Usage

Let’s identify a search request with the name my-request:

$ curl -H "Content-Type: application/json" -H "X-Opaque-Id: my-request" 'http://localhost:9200/siren/_search'

Then to cancel it, issue a request as follows:

$ curl -XPOST -H "Content-Type: application/json" 'localhost:9200/_siren/job/my-request/_cancel'

If successful, the response will acknowledge the request and give a listing of the cancelled tasks:

{
  "acknowledged" : true,
  "tasks" : [
    {
      "node" : "5ILUA44uSee-VxsBsNbsNA",
      "id" : 947,
      "type" : "transport",
      "action" : "indices:siren/plan",
      "description" : "federate query",
      "start_time_in_millis" : 1524815599457,
      "running_time_in_nanos" : 199131478,
      "cancellable" : true,
      "headers" : {
        "X-Opaque-Id" : "my-request"
      }
    }
  ]
}