Cluster APIs
The cluster APIs enables the retrieval of cluster and node level information, such as statistics about off-heap memory allocation.
Nodes Statistics
The cluster nodes stats API allows to retrieve one or more (or all) of the cluster nodes statistics.
curl -XGET 'http://localhost:9200/_siren/nodes/stats'
curl -XGET 'http://localhost:9200/_siren/nodes/nodeId1,nodeId2/stats'
The first command retrieves stats of all the nodes in the cluster. The second command selectively retrieves nodes stats
of only nodeId1
and nodeId2
By default, all stats are returned. You can limit this by combining any of the following stats:
memory
-
Memory allocation statistics
planner
-
Statistics about the planner job and task pools.
Memory Information
The memory
flag can be set to retrieve information about the memory allocation:
curl -XGET 'http://localhost:9200/_siren/nodes/stats/memory'
The response includes memory allocation statistics for each node node as follows:
{
"se6baEC9T4K7-14yuG2qgA": {
"memory": {
"allocated_direct_memory_in_bytes": 0,
"allocated_root_memory_in_bytes": 0,
"root_allocator_dump_reservation_in_bytes": 0,
"root_allocator_dump_actual_in_bytes": 0,
"root_allocator_dump_peak_in_bytes": 0,
"root_allocator_dump_limit_in_bytes": 1073741824
}
},
"sKnVUBo9ShGzkl4GYih7BA": {
"memory": {
"allocated_direct_memory_in_bytes": 0,
"allocated_root_memory_in_bytes": 0,
"root_allocator_dump_reservation_in_bytes": 0,
"root_allocator_dump_actual_in_bytes": 0,
"root_allocator_dump_peak_in_bytes": 0,
"root_allocator_dump_limit_in_bytes": 1073741824
}
}
}
allocated_direct_memory_in_bytes
-
The actual direct memory allocated by Netty in bytes
allocated_root_memory_in_bytes
-
The actual direct memory allocated by the root allocator in bytes
root_allocator_dump_reservation_in_bytes
-
Dump of the root allocator initial reservation direct memory allocated.
root_allocator_dump_actual_in_bytes
-
Dump of the root allocator actual direct memory allocated.
root_allocator_dump_peak_in_bytes
-
Dump of the root allocator peak direct memory allocated.
root_allocator_dump_limit_in_bytes
-
Dump of the root allocator limit direct memory allocated.
Planner Information
The planner
flag can be set to retrieve information about the planner job and task pools:
curl -XGET 'http://localhost:9200/_siren/nodes/stats/planner'
The response includes statistics about the planner job and task pools for each node as follows:
{
"se6baEC9T4K7-14yuG2qgA": {
"planner": {
"thread_pool": {
"job": {
"permits": 1,
"queue": 0,
"active": 0,
"largest": 1,
"completed": 538
},
"task": {
"permits": 3,
"queue": 0,
"active": 0,
"largest": 3,
"completed": 3955
}
}
}
},
"sKnVUBo9ShGzkl4GYih7BA": {
"planner": {
"thread_pool": {
"job": {
"permits": 1,
"queue": 0,
"active": 0,
"largest": 1,
"completed": 537
},
"task": {
"permits": 3,
"queue": 0,
"active": 0,
"largest": 3,
"completed": 3863
}
}
}
}
}
Query cache information
To retrieve information about Siren Federate’s query cache, you can set the query_cache
flag, as follows:
curl -XGET 'http://localhost:9200/_siren/nodes/stats/query_cache'
The response includes statistics about the query cache on each node:
{
"_nodes": {
"total": 2,
"successful": 2,
"failed": 0
},
"cluster_name": "my_cluster",
"nodes": {
"tEwWYjpbQzSYghVJVt87QQ": {
"timestamp": 1545408407569,
"name": "node_s0",
"transport_address": "127.0.0.1:41639",
"host": "127.0.0.1",
"ip": "127.0.0.1:41639",
"roles": [
"master",
"data",
"ingest"
],
"query_cache": {
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
}
},
"Dw06QS6oRbS3fEMazn5llQ": {
"timestamp": 1545408407569,
"name": "node_s1",
"transport_address": "127.0.0.1:42841",
"host": "127.0.0.1",
"ip": "127.0.0.1:42841",
"roles": [
"master",
"data",
"ingest"
],
"query_cache": {
"memory_size_in_bytes": 0,
"total_count": 0,
"hit_count": 0,
"miss_count": 0,
"cache_size": 0,
"cache_count": 0,
"evictions": 0
}
}
}
}
memory_size_in_bytes
-
The size in bytes of the cache
total_count
-
The total number of lookups in the cache
hit_count
-
The number of successful lookups in the cache
miss_count
-
The number of lookups in the cache that failed to retrieve data
cache_size
-
The number of entries in the cache
cache_count
-
The number of entries that have been cached
evictions
-
The number of entries that have been evicted from the cache
Planner Caching Layers
The Federate planner leverage several caching layers. There are APIs detailed below that can be used to interact with them.
-
The optimizer cache allows to cache response to searches executed to compute statistics required by the query optimizer.
-
The field capabilities cache is responsible in caching responses to the _field_caps API of Elasticsearch.
The following REST endpoints have been deprecated and are scheduled to be removed in a next major release: * /_siren/cache * /_siren/<nodeId>/cache * /_siren/cache/clear * /_siren/<nodeId>/cache/clear The functionalities are now available at the following endpoints: * /_siren/cache/planner * /_siren/<nodeId>/cache/planner * /_siren/cache/planner/clear * /_siren/<nodeId>/cache/planner/clear |
curl -XGET 'http://localhost:9200/_siren/cache/planner' (1)
curl -XGET 'http://localhost:9200/_siren/nodeId1,nodeId2/cache/planner' (2)
curl -XPOST 'http://localhost:9200/_siren/cache/planner/clear' (3)
curl -XPOST 'http://localhost:9200/_siren/nodeId1,nodeId2/cache/planner/clear' (4)
1 | This retrieves usage statistics of the different planner caches for all the nodes in the cluster. |
2 | Similar to the previous endpoint, but only for the desired list of node IDs. |
3 | This invalidates the planner caches on every node. |
4 | Similar to the previous endpoint, but only for the desired list of node IDs. |
The response includes usage statistics about the planner caches use on each node:
{
"nodes": [
{
"ebznyAnkSSa4TqGHHPk0wQ": {
"name": "node_s1",
"ephemeral_id": "xIMFXcVcToaxeZM-WUfXrg",
"transport_address": "127.0.0.1:13301",
"external_id": "node_s1",
"attributes": {},
"roles": [],
"version": "8.13.4",
"min_index_version": 7000099,
"max_index_version": 8503000
},
"optimizer": {
"size": 0,
"hit_count": 0,
"miss_count": 0,
"eviction_count": 0,
"load_exception_count": 0,
"load_success_count": 0,
"total_load_time_in_millis": 0
},
"field_caps": {
"size": 0,
"hit_count": 0,
"miss_count": 0,
"eviction_count": 0,
"load_exception_count": 0,
"load_success_count": 0,
"total_load_time_in_millis": 0
}
}
]
}
size
-
The estimated number of entries in the cache.
hit_count
-
The number of cache hits.
miss_count
-
The number of cache misses.
eviction_count
-
The number of evicted entries.
load_exception_count
-
The number of times a request failed to execute as its response was to be put in the cache.
load_success_count
-
The number of times a request was executed successfully as its response was to be put in the cache.
total_load_time_in_millis
-
The time spent in milliseconds to load request responses into the cache.
Nodes Benchmark
The cluster nodes benchmark API allows to execute a benchmark on one or more (or all) of the cluster nodes.
curl -XGET 'http://localhost:9200/_siren/nodes/benchmark'
curl -XGET 'http://localhost:9200/_siren/nodes/nodeId1,nodeId2/benchmark'
The first command executes the benchmark of all the nodes in the cluster. The second command selectively executes the
benchmark on nodeId1
and nodeId2
only.
This API provides several benchmarks which are meant to evaluate different parts of a search request:
-
UPLOAD_PACKET: A benchmark task that measures how fast we can push packets through the transport layer.
-
DATA_UPLOAD_QUEUE: A benchmark task that measures how fast we can push packets through the data upload API.
-
HASH_PARTITION_SINK: A benchmark task that measures how fast we can push packets through the hash-partitioning sink.
-
HASH_PARTITION_PIPELINE: A benchmark task that measures how fast we can push batches through the hash partitioning pipeline.
The cluster nodes benchmark API accepts the following request parameters:
type
-
(Optional, string) The type of benchmark to execute:
UPLOAD_PACKET
,DATA_UPLOAD_QUEUE
,HASH_PARTITION_SINK
,HASH_PARTITION_PIPELINE
. Defaults toUPLOAD_PACKET
. packet_count
-
(Optional, integer) The number of packets to upload during the duration of the test. Default to 50.
packet_size
-
(Optional, integer) The size in bytes of the data packet. Default to 1MB.
threads
-
(Optional, integer) The number of upload threads. Default to 1.
target_nodes
-
(Optional, string) The list of node IDs over which the data packets will be sent to. Default to all nodes.
number_of_partitions_per_node
-
(Optional, integer) The number of partitions per node. Default to 32.