Oracle Cloud Infrastructure Documentation

Sync Data to a Vector Store

To sync data to a vector store, first, create one or more data scync connectors for a vector store in the OCI Generative AI service. Then run the command to perform the sync.

Important

This section applies only to vector stores created for unstructured data, where after you create the vector store, you must set up a connector to Object Storage to sync the data. For structured data, you point to an OCI Database tool and provide connection information when you create the vector store. See Creating a Vector Store.

(Optional) Add Metadata for Search Filtering

To make metadata available for search filtering, add metadata to the Object Storage files before performing data sync. Metadata added after a sync isn’t included in that sync unless you run data sync again. See Adding Metadata to Object Storage Files for Search Filtering

Step 1. Create a Vector Store Data Sync Connector

Learn about Vector Store Connectors and perform the Required IAM Policy for connectors before you perform this task.

Tip

After you create a data sync connector you can view its details and perform other tasks, such as updating or deleting it. Use the Actions menu (three dots) in the Console to access these tasks.

Create

  1. In the vector store detail page, select the vector store that you want to add a data sync connector to. If you need help finding the list page, see Listing Vector Stores.
  2. In the vector store detail page, select Create data sync connector

Basic Information

  1. Enter a name for the data sync connector.
    Start the name with a letter or underscore, followed by letters, numbers, hyphens, or underscores. The length can be 1 to 255 characters.
  2. (Optional) Enter a description for the data sync connector.
  3. Select a compartment to create the vector store in. The default compartment is the same as the list page, but you can select any compartment that you have permission to work in.

Data Bucket

  1. Select the compartment for the bucket that holds the data to be synchronized with this vector store.
  2. Select the bucket.
    When you select the bucket, all the files in the buckets lists in the page. Up to 1000 files and folders. Perform one of the following actions:
    • Select all in bucket
    • Select files and folders individually.
    Note

    If you select the files individually, you must select at least one file or folder.
  3. Select Create.

Step 2. Perform Data Sync

  1. Enter a name for the data sync.
    Start the name with a letter or underscore, followed by letters, numbers, hyphens, or underscores. The length can be 1 to 255 characters.
  2. (Optional) Enter a description for the data sync.
  3. Select Perform.

Step 3. Review Data Sync Jobs (Recommended)

  1. In the Vector stores list page, select the vector store that you want to see its sync jobs. If you need help finding the list page, see Listing Vector Stores.
  2. In the Data sync connectors tab, select the data sync connector that you want to see its sync jobs.
  3. Select the Data sync tab.
  4. In the listed Data Sync Jobs, select a job to see its details.
  5. Review the following information about the data sync job:
    • Total files synced
    • Last Updated
    • Files
      • Files created
      • Files updated
      • Files failed to process
      • Files ignored
      • Files in progress

Step 4. List Files

  1. In the Vector stores list page, select the vector store that you want to see its files. If you need help finding the list page, see Listing Vector Stores.
  2. Select the Files tab.
    All the files related to this vector store and their status is listed.