Configuring a remote Federate connector

The Connector APIs have been deprecated and are scheduled for removal in the next major release.

Siren Federate provides the capability to query data from a Federate cluster. through the remote clusters module, and the Siren Federate connector APIs .

Compatibility with security systems

To execute joins spanning several Federate clusters, set the following cluster- and index-level permissions on the clusters:

  • cluster:internal/federate/*

  • indices:data/read/mget

  • indices:data/read/msearch

  • indices:data/read/mtv

  • indices:data/read/open_point_in_time

  • indices:data/read/close_point_in_time

  • indices:data/read*

  • indices:admin/template/get

  • indices:admin/aliases/get

  • indices:admin/aliases/exists

  • indices:admin/get

  • indices:admin/exists

  • indices:admin/mappings/fields/get*

  • indices:admin/mappings/get*

  • indices:admin/mappings/federate/connector/get*

  • indices:admin/mappings/federate/connector/fields/get*

  • indices:admin/types/exists

  • indices:admin/validate/query

  • indices:monitor/settings/get

The remote Federate connector is compatible with the following security systems:

Before you begin

  1. Ensure that the remote clusters are configured as described in the Configuring remote clusters section of the Elasticsearch documentation.

  2. Install the Siren Federate plugin on the remote clusters.

  3. Set up the remote Federate clusters. For example, use the following settings:

    curl -X PUT http://localhost:9200/_cluster/settings -H 'Content-type: application/json' -d '
{
    "persistent": {
        "cluster": {
            "remote": {
                "remotefederate": {
                    "seeds": [
                        "127.0.0.1:9330"
                    ]
                }
            }
        }
    }
}
'

Procedure

In this procedure, we are using the example of a remote Federate cluster called remotefederate, which contains indices called logs-2019.01, logs-2019.02, …​, logs-2019.12, and so on.

  1. Define the datasource as an alias to the remote Federate cluster, by using the Siren Federate datasource API as follows:

    curl -X PUT http://localhost:9200/_siren/connector/datasource/remotefederateds -H 'Content-type: application/json' -d '
  {
    "federate": {
      "alias": "remotefederate"
    }
  }
  '

  2. Define a virtual index on the coordinator cluster that matches the wildcard index pattern logs-*, by using the Siren Federate virtual index API as follows:

    curl -X PUT http://localhost:9200/_siren/connector/index/logsvi -H 'Content-type: application/json' -d '
{
  "datasource": "remotefederateds",
  "resource": "logs-*",
  "key": "_id"
}
'

  3. Execute a join query. For example, the coordinator cluster contains an index called machines, which contains information about IP addresses on machines of interest. To find out about the logs that are associated to these machines, execute the following Federate join query:

    curl -X GET http://localhost:9200/siren/logsvi/_search -H 'Content-Type: application/json' -d '
{
    "query": {
        "join": {
            "indices": [
                "machines"
            ],
            "on": [
                "logs_ip_hash",
                "machines_ip_hash"
            ],
            "request": {
                "query": {
                    "match_all": {

                    }
                }
            }
        }
    }
}
'

    logs_ip_hash is the IP field in the index logsvi and machines_ip_hash is the IP field in the index machines.

    The API returns the following response:

    {
  "took": 150,
  "timed_out": false,
  "hits": {
    "total" : {
        "value": 1,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "logs-2019-11-12",
        "_id": "0",
        "_score": 2,
        "_source": {
          "date": "2019-11-12T12:12:12",
          "message": "trying out Siren"
        }
      }
    ]
  }
}

Known limitations for the Federate connector

To use Siren Federate with a remote Federate cluster, a coordinator Federate cluster must run version 7.7.1-20.0 or later, and the remote Federate cluster must run Siren Federate version 7.7.1-20.0 or later.