Cross-Cluster Connections in Search with OpenSearch

Learn about cross-cluster connections in Search with OpenSearch.

You can use cross-cluster connections to search and analyze data across several OpenSearch clusters as though the data were available from a single source. Cross-cluster connections support cross-cluster search (CCS) and cross-cluster replication (CCR).

Cross-cluster connections are available by default with the Security plugin, but you must configure the participating clusters to allow connections and configure the required access permissions.

The meaning of outbound cluster and inbound cluster depends on whether you use CCS or CCR. The following table maps Search with OpenSearch terminology to OpenSearch roles.


Search with OpenSearch Term	OpenSearch Cross-Cluster Search Term	OpenSearch Cross-Cluster Replication Term
Outbound cluster	Coordinating cluster	Leader cluster/index
Inbound cluster	Remote cluster	Follower cluster/index

Note

The external OpenSearch open source project references in this topic link to OpenSearch version 3.6. Consult the OpenSearch project documentation for your cluster’s version. For cluster version requirements and compatibility limitations, see Version Requirements and Limitations.

Prerequisites

The following prerequisites apply to cross-cluster connections:

Permissions: You must have appropriate permissions for both the inbound and outbound clusters to perform cross-cluster operations. For the specific cluster permissions required, see Performing Cross-Cluster Replication.
IAM Policies: For information about required IAM policies and permissions, see IAM Policies.

Version Requirements

Cross-cluster connections have the following version requirements:

Cross-cluster search (CCS) is supported for clusters running OpenSearch 2.11 or later.
Cross-cluster replication (CCR) is supported for clusters running OpenSearch 2.15 or later.
For CCR, the follower cluster must be running a version equal to or newer than the leader cluster. This version compatibility limitation doesn't apply to CCS.

Limitations

Cross-cluster connections have the following limitations:

You can configure a maximum of five clusters per tenancy.
The following version compatibility requirements apply:
- Clusters running supported releases within the same OpenSearch major version can connect, even if their minor or patch versions differ.
- Clusters running different major versions can connect only when the major versions are consecutive.
- Clusters running the earlier major version must use the final minor release series of that major version. For example, an OpenSearch 3.6 cluster can connect to an OpenSearch 2.19.1 cluster. It can't connect to a cluster running an earlier 2.x release or an OpenSearch 1.x release.
Cross-cluster connections are supported between the following:
- Clusters in the same OC1 region.
- Clusters in different regions within the OC1 realm.

Configuring Cross-Cluster Connections

To set up CCS or CCR, first add an Inbound allowance on the cluster accepting the connection. After its work request succeeds and the cluster returns to Active, add an Outbound alias on the connecting cluster.

Use Adding a Cross-Cluster Connection in Search with OpenSearch for both initial Console actions. Use Updating a Cross-Cluster Connection in Search with OpenSearch to edit an existing Outbound connection and Deleting a Cross Cluster Connection from an OpenSearch Cluster to remove a connection.

For CLI and API setup, run ConfigureOutboundCluster on the accepting cluster, then UpdateOpensearchCluster on the connecting cluster. The following table identifies the targets.


Use Case	ConfigureOutboundCluster Target	UpdateOpensearchCluster Target
CCS	Inbound remote cluster	Outbound coordinating cluster
CCR	Outbound leader cluster	Inbound follower cluster

For CCS, the coordinating cluster connects to the remote cluster. For CCR, data is replicated from the leader to the follower, but the follower initiates the transport connection and pulls changes from the leader.

Note

The inboundClusterIds and outboundClusters arrays are complete lists of the connections to retain. Include existing entries that you want to keep. The following single-entry examples assume that no other connections are configured.

Cross-Cluster Search Configuration Example

Cluster A is the outbound coordinating cluster. Cluster B is the inbound remote cluster holding the indexes to search. Configure the connection in this order:

On Cluster B, allow Cluster A to connect:

oci opensearch cluster configureoutboundcluster \
  --opensearch-cluster-id "<cluster_b_ocid>" \
  --inbound-cluster-ids '["<cluster_a_ocid>"]' \
  --wait-for-state SUCCEEDED

Wait for the work request to succeed and Cluster B to return to Active.

On Cluster A, create the remote-b alias for Cluster B. Use Cluster A's current display name:

oci opensearch cluster update \
  --opensearch-cluster-id "<cluster_a_ocid>" \
  --display-name "<cluster_a_display_name>" \
  --outbound-cluster-config '{
  "isEnabled": true,
  "outboundClusters": [
    {
      "displayName": "remote-b",
      "seedClusterId": "<cluster_b_ocid>",
      "isSkipUnavailable": true,
      "mode": "SEARCH_AND_REPLICATION"
    }
  ]
}' \
  --wait-for-state SUCCEEDED

Wait for the work request to succeed and Cluster A to return to Active. Submit searches to Cluster A using remote-b as the remote alias.

Cross-Cluster Replication Configuration Example

Cluster A is the outbound leader cluster. Cluster B is the inbound follower cluster. The follower initiates the connection to pull changes from the leader. Configure the connection in this order:

On Cluster A, allow Cluster B to connect:

oci opensearch cluster configureoutboundcluster \
  --opensearch-cluster-id "<cluster_a_ocid>" \
  --inbound-cluster-ids '["<cluster_b_ocid>"]' \
  --wait-for-state SUCCEEDED

Wait for the work request to succeed and Cluster A to return to Active.

On Cluster B, create the leader-a alias for Cluster A. Use Cluster B's current display name:

oci opensearch cluster update \
  --opensearch-cluster-id "<cluster_b_ocid>" \
  --display-name "<cluster_b_display_name>" \
  --outbound-cluster-config '{
  "isEnabled": true,
  "outboundClusters": [
    {
      "displayName": "leader-a",
      "seedClusterId": "<cluster_a_ocid>",
      "isSkipUnavailable": true,
      "mode": "SEARCH_AND_REPLICATION"
    }
  ]
}' \
  --wait-for-state SUCCEEDED

Wait for the work request to succeed and Cluster B to return to Active. Use leader-a in replication requests sent to Cluster B.

The OCI Mode field does not select CCS or CCR. The OpenSearch operation that you perform with the alias determines whether you search or replicate.

Performing Cross-Cluster Search

The user performing a cross-cluster search requires OpenSearch permissions to query both the outbound cluster and the inbound cluster.

Here is an example of a cross-cluster search query:

curl -X GET "https://<outbound_cluster_opensearch_private_IP>:9200/<inbound_cluster_alias>:<index_name>/_search?q=title:Kubernetes&pretty"

For more information, see Cross-cluster search.

After both OCI work requests succeed, inspect the remote alias and run an authorized search against a known remote index. For the preceding CCS example, run these requests in OpenSearch Dashboards Dev Tools on Cluster A:

GET /_remote/info
GET /remote-b:<remote-index>/_search

The connected field indicates whether at least one transport connection is open. A value of false at an idle moment does not, by itself, prove that setup failed. Run a search and check its results. If skip_unavailable is true, an unavailable remote cluster can be skipped, so HTTP success without the expected hits is insufficient verification.

For connection status fields, see Remote Cluster Information API.

Performing Cross-Cluster Replication

With cross-cluster replication, you replicate indexes, mappings, and metadata from one OpenSearch cluster (outbound cluster) to another OpenSearch cluster (inbound cluster). After replication starts for a follower index, changes to its corresponding leader index are replicated to it. Use cross-cluster replication for the following scenarios:

To maintain a backup cluster that you can switch to if the primary cluster fails.
Provide faster query performance by keeping data closer to users in geographically distributed regions.
Distribute query and indexing loads across more than one cluster.

The following can impact cross-cluster replication performance:

Network latency between the outbound and inbound clusters.
The size and frequency of updates to the outbound index.
Resource availability on both the outbound cluster and the inbound clusters.

Starting and Verifying Replication

A cross-cluster connection does not start index replication. After both OCI work requests succeed, send the start-replication request to the follower cluster. In the preceding CCR example, run the following requests on Cluster B:

PUT /_plugins/_replication/<follower-index>/_start
{
  "leader_alias": "leader-a",
  "leader_index": "<leader-index>",
  "use_roles": {
    "leader_cluster_role": "<leader-role>",
    "follower_cluster_role": "<follower-role>"
  }
}

GET /_plugins/_replication/<follower-index>/_status

The leader_alias is the alias configured on the follower. The leader_index identifies an existing index on the leader. With the Security plugin enabled, supply appropriately mapped leader and follower roles.

Check for SYNCING, then verify that a document written to the leader index appears in the follower index. Alternatively, configure an auto-follow rule for matching future leader indexes.

For setup and role mapping, see Getting started with cross-cluster replication and Cross-cluster replication security. For start and status requests, see Cross-cluster replication API.

Permissions

To enable cross-cluster replication, you need cluster-level permissions for replication and index-level permissions for both the outbound cluster and inbound cluster. For more information, see your OpenSearch documentation under Permissions.

Replication Operations

The following table lists the replication operations you can perform. See Cross-cluster replication API for more information.


Operation	Description
Start replication	Begins the replication of an index from the outbound cluster to the inbound cluster.
Stop replication	Ends the replication and converts the inbound index to a standard index
Pause replication	Pauses replication of the outbound index.
Resume replication	Resumes replication of the outbound index.
Get replication status	Gets the status of index replication. Possible statuses are `SYNCING`, `BOOTSTRAPING`, `PAUSED`, and `REPLICATION NOT IN PROGRESS`. Use the syncing details to measure replication lag.
Get leader cluster stat	Gets information about replicated outbound indexes on a specified cluster.
Get follower cluster stats	Gets information about inbound indexes on a specified cluster.
Get auto-follow stats	Gets information about autofollow activity and any replication rules configured on the specified cluster.
Update settings	Updates settings on the inbound index.
Create replication rule	Automatically starts replication on indexes matching a specified pattern. If a new index on the outbound cluster matches the pattern, OpenSearch automatically creates a inbound index and begins replication. You can also use this API to update existing replication rules.
Delete replication rule	Deletes the specified replication rule. This operation prevents any new indexes from being replicated but doesn't stop existing replication already started based on the rule. Replicated indexes remain read-only until you stop replication.

Oracle Cloud Infrastructure Documentation