Maintaining Siren Investigate on Kubernetes

Changing the configuration

If you need to make changes to the Investigate configuration, do the following:

Edit the investigate.yml file as needed and recreate the investigate-config secret with the new version, for example:

kubectl -n siren delete secret/investigate-config
kubectl -n siren create secret generic investigate-config --from-file=investigate.yml=investigate.yml

To force the StatefulSet to apply the new configuration, increase the value of the CONFIG_VERSION environment variable in investigate-set.yaml and apply the manifest again.

...
        env:
          # The URL of the data cluster
          - name: ELASTICSEARCH_URL
            value: "https://siren-es-http:9200"
          # Increase this number when you want to force a statefulset upgrade
          # after changing the configuration secret, then apply the statefulset.
          - name: CONFIG_VERSION
            value: "3"
          - name: INSTANCE_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
...

Handling keys and certificates

If your Investigate configuration needs to have references to any local file, such as keys or certificates, you can create these as secrets and then mount them as volumes in your StatefulSet configuration.

Example

If you want to enable support for Investigate managed indices and you have generated the private RSA key required for it, do the following:

Upload the file as a secret:

kubectl -n siren create secret generic investigate-private-key --from-file=investigate.pem=investigate.pem

Add a volume to mount the key into all the containers created by the StatefulSet:

...
# Investigate containers
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: investigate
  labels:
    app: investigate
spec:
  serviceName: investigate
  replicas: 1
  selector:
    matchLabels:
      app: investigate
  template:
    metadata:
      labels:
        app: investigate
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: investigate
        ...
        env:
          - name: ELASTICSEARCH_URL
            value: "https://siren-es-http:9200"
          - name: CONFIG_VERSION
            value: "3"
        volumeMounts:
        - name: config
          mountPath: /opt/siren-investigate/config/investigate.yml
          readOnly: true
          subPath: investigate.yml
        # The Investigate private key secret mount
        - name: privatekey
          mountPath: /opt/siren-investigate/config/investigate.pem
          readOnly: true
          subPath: investigate.pem
      volumes:
      - name: config
        secret:
          defaultMode: 0644
          secretName: investigate-config
      - name: privatekey
        secret:
          defaultMode: 0644
          secretName: investigate-private-key

Reference the mounted secret in investigate.yml like any other file in the container, for example:

...
    shared_index_components:
      enabled: true
      dataspaceSignature:
        algorithm: 'RS256'
        privateKey:
          # Reference to the mounted secret
          key: '/opt/siren-investigate/config/investigate.pem'
          passphrase: 'password'
...

Upgrading the Investigate version

To upgrade to a later release of Investigate, do the following:

Take a snapshot of the Investigate system indices; .siren and .sirenaccess by default.
Block all traffic to the Investigate containers managed by the StatefulSet or temporarily terminate the StatefulSet.
Launch a Kubernetes job using the new Investigate version with the same configuration as your StatefulSet containers to run the upgrade command.
If the upgrade is successful, update the StatefulSet declaration to use the Docker image for the new version.
Remove the Kubernetes job.
Restore traffic to the Investigate containers.

Creating a Kubernetes job to upgrade Investigate to a later version

Create a new file named upgrade.yaml with the following contents:

# Investigate containers
apiVersion: batch/v1
kind: Job
metadata:
  name: investigate-upgrade
spec:
  template:
    spec:
      restartPolicy: Never

Copy the template specification from your StatefulSet to the upgrade.yaml file. Remove any port or probe and change the image attribute to use the new Investigate version, for example:

# Investigate containers
apiVersion: batch/v1
kind: Job
metadata:
  name: investigate-upgrade
spec:
  template:
    spec:
      restartPolicy: Never
      # Use the same specification as the stateful set containers, remembering to change the image attribute to use the version of Investigate you want to upgrade to.
      terminationGracePeriodSeconds: 30
      containers:
      - name: investigate

        # Specify here the newer Investigate release
        image: sirensolutions/siren-investigate:12.1.5

        imagePullPolicy: IfNotPresent
        # Do not specify any port or probe from the stateful set definition
        resources:
          limits:
            memory: 6Gi
            cpu: 2
          requests:
            memory: 6Gi
            cpu: 2
        env:
          # The URL of the data cluster. Make sure to set this to the same value as "elasticsearch.url" in the Investigate configuration file.
          - name: ELASTICSEARCH_URL
            value: "https://siren-es-http:9200"
          # Increase this number when you want to force a statefulset upgrade
          # after changing the configuration secret, then apply the statefulset again.
          - name: CONFIG_VERSION
            value: "1"
          - name: INSTANCE_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        volumeMounts:
        - name: config
          mountPath: /opt/siren-investigate/config/investigate.yml
          readOnly: true
          subPath: investigate.yml
      volumes:
      - name: config
        secret:
          defaultMode: 0644
          secretName: investigate-config

Add a custom command to the container specification to execute /opt/siren-investigate/bin/investigate upgrade -y. The command backs up the current Investigate indices to /opt/siren-investigate/data/backup-<date>. Delete the indices and proceed with the upgrade.

# Investigate containers
apiVersion: batch/v1
kind: Job
metadata:
  name: investigate-upgrade
spec:
  template:
    spec:
      restartPolicy: Never
      # Use the same specification as the stateful set containers, remembering to change the image attribute to use the version of Investigate you want to upgrade to.
      terminationGracePeriodSeconds: 30
      containers:
      - name: investigate
        image: sirensolutions/siren-investigate:12.1.5

        # Run the upgrade command.
        command:
        - sh
        - -c
        - |
          /opt/siren-investigate/bin/investigate upgrade -y

        imagePullPolicy: IfNotPresent
        ports:
        ...

Execute the job with kubectl apply -f upgrade.yaml.

Verify that the job is complete:

kubectl -n siren get jobs/investigate-upgrade

NAME                  COMPLETIONS   DURATION   AGE
investigate-upgrade   1/1           58s        2m26s

To ensure that there are no detected warnings or issues, verify the logs even if the job completed successfully, for example:
```
kubectl -n siren logs job/investigate-upgrade
```
When the upgrade has been performed, edit your StatefulSet to the image for the new Investigate version and apply it, for example:
```
kubectl -n siren apply -f investigate-set.yaml
```
Restore traffic to the StatefulSet and verify that it is running the correct Investigate version.
You can now delete the upgrade job and manifest.

When the Investigate upgrade command is executed, a local backup of the system indices is stored in /opt/siren-investigate/data/. To preserve this backup, mount a reclaimable persistent volume to /opt/siren-investigate/data in the job specification.

Next steps

For more information about the Siren Investigate configuration and upgrade processes, see the following: