Troubleshooting guide

Installation error when extracting the plugin ZIP file

When you install the Siren Federate plugin , you might get the following error:

On Windows

elasticsearch-plugin install PATH-TO-SIREN-FEDERATE-PLUGIN\siren-federate-9.4.5-40.0.zip
> Downloading PATH-TO-SIREN-FEDERATE-PLUGIN\siren-federate-9.4.5-40.0.zip
[=================================================] 100%
Exception in thread "main" java.nio.file.NoSuchFileException: PATH-TO-ELASTICSEARCH\plugins.installing-3603227438462114792\plugin-descriptor.properties
at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:162)
at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:724)
at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:803)
at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:786)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:232)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:217)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)

On Linux

Exception in thread "main" java.nio.file.NoSuchFileException: PATH-TO-ELASTICSEARCH/plugins/.installing-2425832270248497263/plugin-descriptor.properties
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:162)
at org.elasticsearch.plugins.InstallPluginCommand.loadPluginInfo(InstallPluginCommand.java:724)
at org.elasticsearch.plugins.InstallPluginCommand.installPlugin(InstallPluginCommand.java:803)
at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:786)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:232)
at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:217)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)

This error occurs because the command is being run on the distribution ZIP file, rather than on the plugin ZIP file inside it.

To resolve this error, complete the following steps:

  1. Extract the Siren Federate distribution ZIP file into a local directory.

  2. In the extracted directory, locate the plugin ZIP file named siren-federate-9.4.5-40.0-proguard-plugin.zip. The path to this plugin ZIP file is represented by PATH-TO-SIREN-FEDERATE-PLUGIN in the command that follows.

  3. Run the installation command: $ ./bin/elasticsearch-plugin install file:///PATH-TO-SIREN-FEDERATE-PLUGIN/siren-federate-9.4.5-40.0-proguard-plugin.zip

Cannot start the buffer allocator service

The following error message is displayed:

elasticsearch.bootstrap.StartupException: BufferAllocatorException[Cannot start the buffer allocator service];

This error occurs when the memory root limit is higher than the direct memory size (-XX:MaxDirectMemorySize) of the JVM. The default root memory limit is 1GB, or the maximum direct memory size of the JVM if lower than 1GB.

You can resolve this in one of the following ways:

  • Open the config/elasticsearch.yml file and reduce the siren.memory.root.limit parameter to a value in bytes that is lower than the direct memory limit.

  • Open the jvm.options file and increase the direct memory limit. For more information, see Setting off-heap memory.

Rejected execution exception

The following error message is displayed:

org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of io.siren.federate.core.planner.c.a.h$1$$Lambda$6065/0x0000000801b77550@39949eae on EsThreadPoolExecutor[name = es-master-0/federate.planner, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@e70a7[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 357087]]
	at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:827)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1357)
	at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:84)
	at com.google.common.util.concurrent.ForwardingExecutorService.execute(ForwardingExecutorService.java:101)
	at io.siren.federate.core.planner.c.a.h$1.execute(JobRunnable.java:113)
	at java.base/java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:568)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:638)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2137)
	at io.siren.federate.core.planner.c.a.b$3.a(ElasticsearchIndexRegistry.java:193)
	at io.siren.federate.core.planner.c.a.b$3.onResponse(ElasticsearchIndexRegistry.java:190)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:83)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.finishHim(TransportBroadcastAction.java:249)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction.onOperation(TransportBroadcastAction.java:213)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:193)
	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction$AsyncBroadcastAction$1.handleResponse(TransportBroadcastAction.java:180)
	at org.elasticsearch.transport.TransportService$6.handleResponse(TransportService.java:634)

Exception meaning

es_rejected_execution_exception means a thread pool cannot accept more work at that moment. In practice, this usually happens when all worker threads are busy and the queue has reached its configured limit.

  1. From the stack trace, identify these fields:

    • Thread pool name, for example, federate.planner or federate.task.worker.

    • pool size: the configured worker threads in that pool.

    • active threads: the threads currently executing work.

    • queue capacity and queued tasks: the configured queue limit and current queue usage.

    • completed tasks: helps estimate whether the pool is progressing or stalling.

  2. Verify real-time thread pool metrics with the Elasticsearch thread pool APIs.

federate.planner thread pool

This pool runs federate request planning and coordination. Rejections here indicate planning work arrives faster than it can be scheduled and executed.

Common causes:

  • Too many concurrent requests.

  • Planner pool size or queue size is too small for the incoming workload.

  • Individual joins with expensive planning (many index patterns, shards, or complex joins).

Check the following:

  • The planner thread pool saturation (active, queue, rejected, completed) using the cat thread pool API.

  • Running and queued tasks with the cat tasks API to identify long-running planner-related operations.

  • Node load (CPU, heap pressure, and node-level stress) with the cat nodes API.

How to address it:

  • Reduce client-side concurrency for join-heavy queries.

  • Increase cluster capacity when sustained planner queue pressure is observed.

  • Review query patterns and simplify high-cost joins where possible.

  • Consider increasing the federate.planner thread pool size and queue capacity if the workload justifies it and cluster resources allow.

  • If you encounter this under normal load, it may indicate a performance regression or an underlying issue. Open a support ticket and include the relevant logs and metrics to assist with the investigation.

  • If you identify a specific join that takes an excessive amount of time, share it with support to help determine the root cause.

federate.task.worker thread pool

This pool executes task-worker runnables generated during the planning. Rejections here indicate data-node execution pressure.

Common causes:

  • High fan-out of work per join.

  • Many indices/shards participating in the join.

  • Too many Lucene segments, Federate typically schedules one worker runnable per segment.

Check the following:

How to address it:

  • Reduce shard and segment fan-out for join workloads.

  • Optimize index lifecycle/merge behavior to reduce excessive segment counts.

  • Scale out data nodes when worker queue pressure remains high under normal traffic.

Other tools for investigation

Use these APIs for deeper diagnosis:

Out of memory exception

The following error message is displayed:

out_of_memory_exception: [ROOT allocator] Unable to allocate a buffer of size 1mb due to the allocator's limit of 1023.2mb. Current allocation: 1023.2mb - Allocator(child) 0/1072955392/1072955400/1072956000 (res/actual/peak/limit)

This message indicates that off-heap memory allocation is used up. A join might require more memory than that which is available.

Open the config/elasticsearch.yml file and increase the siren.memory.root.limit parameter to a value in bytes for the root allocator.

Changing the thread pool queue size

You can modify the thread pools in Siren Federate.

To change the thread pool queue size, follow the instructions in the Elasticsearch documentation.

To verify how much thread pool queue is in use, follow the instructions in the Elasticsearch documentation.

Supported data types in a join

Siren Federate supports all primitive data types, however you must ensure that the data type of the joined fields across index patterns is the same.

For example, if you try to join a field from the pattern index*, but the field is an integer in index1 while it is a keyword in index2, an error will result.

For more information, see Configuring joins by type.

Support for joining on the document ID

When you add the on parameter to the join query, Siren Federate does support joining on the _id metadata field.

However, for system performance reasons - and because the _id metadata field does not have doc_values enabled - we recommend that you use a dedicated field with doc_values enabled, such as a keyword field.

One way to get around this issue is to index the _id metadata field in a secondary field that has doc_values enabled, and use that field in the join.

You can duplicate the content of the _id field into another field that has doc_values enabled on the client side or use a set ingest processor.

For more information about how doc_values are scanned during a join, see Configuring joins by type.

Minimum memory requirements

If you are uncertain about the minimum memory that is required to use Siren Federate, it depends on the size of your data set and the size of the join.

For more information, see Configuring off-heap memory.

System performance

If the response time of search requests that involve joins is too long, try the following options to improve the performance of the join: