Setting up record editing

Siren Investigate provides the ability to make manual changes to the records of your entity tables and searches. These changes can then be stored on the same index or in a separate revision index.

Enabling record editing helps in the process of data curation by providing a way to manually correct small errors in some records and test the fixes immediately.

Applied changes can be analyzed later by a data curator, to be applied at scale when importing a dataset.

Enabling record editing

Before you begin editing records in Siren Investigate, ensure that the required security settings are configured.

Revisions can be saved only if the index pattern contains a single index, which means that neither the characters that indicate multiple indices (,) nor a wildcard (*) are accepted in the index name.

  1. In the Data model app, select the Revisions tab.

  2. Select Editing is allowed on <name>.

  3. Specify how the revisions will be stored:

    • Use the same index: Configure the same index to store the edited records by overwriting the original data. The applied changes are permanent, and disabling record editing does not restore the original data.

    • Use a revision index: Configure a separate revision index. This does not overwrite the original index. You can easily revert all changes and restore the original data by disabling record editing.

      Storing edited records in a revision index is subject to the following limitations:

      • The revision index is joined with the original indices to produce the final view of the entity table. The speed of the join, therefore, depends on how many records are edited. Only the records in the user’s current dataspace are considered in the join. If you expect the number of revisions to exceed 15,000 records, it is recommended that you store the revisions in the same index to avoid impacting the system’s performance.

      • All edited records will score lower in search results, unless you specify a more accurate (but slower) search type in the siren:scoringSearchType advanced setting. This is due to the method that Elasticsearch uses to calculate scores across separate shards.

  4. Specify the fields that are allowed or denied for editing. Enter each value on a new line.

Use of an Elasticsearch index with no scoping is now deprecated in favor of Siren-managed dataspace scoped indices. For more information, see Configuring security for shared indices.

Revision visibility

If you are using the original index for storage, then revision visibility is the same as that of the original data. However, if you are configuring a separate revision index, you can choose between the following options:

  • Private: Revised data remains private to the dataspace that a record is revised in.

  • Shared: Revised data is visible from all dataspaces.

  • Shared revisions of a legacy index: This option is visible only for entity tables that have revision indices from earlier versions of Siren Investigate. It is selected when the revision index is a standard (not Siren-managed) Elasticsearch index.

The visibility of revised data is bound by the scope of the original data. This means that you cannot create shared revisions for data that is private to the dataspace, but you can create private revisions of shared data.

Revision index settings

If you select Use a revision index, you must complete the following fields:

  1. Name: Specify a name for the revision index.

    A revision index name must be unique - it cannot match the name of the original entity table. Revisions must also be stored in a single index. Therefore, the use of the wildcard character * and the comma , to indicate multiple indices is not supported. Index aliases are not supported either.

  2. Primary key: Select the primary key of your entity table or search, which is used to track records. You can view or discover the primary keys in your data set on the Fields tab.

  3. Click Create index.

Editing records

After record editing is configured, additional controls to edit records are available on all record tables and Record Table visualizations on that entity table or search.

For more information, see Editing records in record tables.

Adding data to entity tables that contain revisions

It is recommended that you do not attempt to overwrite edited records by importing data.

If you want to import data into an entity table that is enabled for revisions and has a primary key defined, be aware of the following considerations:

  • Using the same index: If you selected this option for the current entity table, the original field values are overwritten.

  • Using a revision index: The original record gets overwritten but remains hidden if the value of the 'primary key' field on the Revisions tab is the same. Only revisions are displayed in the Data tab.