Oracle Cloud Infrastructure Documentation

Cohere Embed Multilingual Image 3

Review performance benchmarks for the cohere.embed-multilingual-image-v3.0 (Cohere Embed Multilingual Image 3) model hosted on one Embed Cohere unit of a dedicated AI cluster in OCI Generative AI.

  • See details for the model and review the following sections:
    • Available regions for this model.
    • Dedicated AI clusters for hosting this model.
  • Review the metrics.

Text Embeddings

This scenario applies only to the embedding models with text input. This scenario mimics embedding generation as part of the data ingestion pipeline of a vector database. In each scenario, all requests are the same size, which is 96 documents, each one with the same number of tokens. For example, for the scenario of 512 tokens mimics a collection of large PDF files, each file with 30,000+ words that a user would ingest into a vector DB.

64 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 64 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.10 0.10 9.86 607.39
2 0.10 0.10 9.66 1,192.60
4 0.11 0.11 9.30 2,295.80
8 0.11 0.11 8.69 4,296.59
24 0.15 0.15 6.69 9,900.74
48 0.19 0.19 5.10 15,114.13
96 0.28 0.28 3.43 20,339.92

128 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 128 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.10 0.10 9.78 1,243.08
2 0.10 0.10 9.55 2,428.37
4 0.11 0.11 9.16 4,660.31
8 0.12 0.12 8.42 8,561.42
24 0.15 0.15 6.31 19,255.04
48 0.21 0.21 4.62 28,172.80
96 0.31 0.31 3.01 36,706.87

256 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 256 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.10 0.10 9.65 2,493.61
2 0.10 0.10 9.37 4,840.64
4 0.11 0.11 8.81 9,107.30
8 0.12 0.12 7.88 16,286.08
24 0.17 0.17 5.50 34,074.47
48 0.24 0.24 3.85 47,783.85
96 0.43 0.43 2.16 53,652.55

512 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 512 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.10 0.10 9.21 4,791.85
2 0.11 0.11 8.74 9,094.38
4 0.12 0.12 7.76 16,165.29
8 0.15 0.15 6.39 26,582.37
24 0.23 0.23 4.03 50,284.92
48 0.37 0.37 2.44 61,014.65
96 0.63 0.63 1.42 71,048.17

1,024 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 1,024 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.09 0.09 8.96 9,348.26
2 0.10 0.10 8.25 17,233.64
4 0.12 0.12 6.87 28,670.18
8 0.15 0.15 5.37 44,877.47
24 0.28 0.28 2.86 71,733.22
48 0.53 0.53 1.52 76,050.39
96 0.99 0.99 0.80 80,597.64

2,048 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 2,048 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.11 0.11 8.16 17,069.42
2 0.13 0.13 7.17 30,023.10
4 0.15 0.15 5.82 48,737.45
8 0.21 0.21 4.21 70,449.76
24 0.48 0.48 1.79 90,090.74
48 0.93 0.93 0.92 92,553.18
96 1.68 1.68 0.51 101,574.61

8,096 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 8,096 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.15 0.15 5.40 44,724.35
2 0.20 0.20 4.05 67,118.88
4 0.31 0.31 2.59 85,752.57
8 0.59 0.59 1.41 93,369.42
24 1.56 1.56 0.53 105,492.68
48 2.84 2.84 0.29 115,098.50
96 5.27 5.27 0.15 121,706.97

32,000 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 32,000 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 0.38 0.38 1.90 62,367.46
2 0.57 0.57 1.30 85,209.04
4 1.05 1.05 0.75 97,631.79
8 1.91 1.91 0.42 109,411.13
24 5.12 5.12 0.16 122,915.27

128,000 Tokens

The following table shows the benchmarks for a scenario of 96 documents, 128,000 tokens per document.

The cohere.embed-multilingual-image-v3.0 model hosted on one Embed Cohere unit of a dedicated AI cluster.
Concurrency Time to First Token (TTFT)(second) Request-level Latency (second) Request-level Throughput (Request per second) (RPS) Total Throughput (tokens/second)
1 1.00 1.00 0.99 121,894.93
2 2.26 2.26 0.44 134,957.95
4 3.36 3.36 0.29 146,216.07

Image Embeddings

This scenario applies only to the embedding models with image input. In each scenario, I(M,N): Image with height Npx and width Mpx represents an image with the height of M and the width of N pixels. For example, I(1024,512) is an image with the height of 1,024 pixels and the width of 512 pixels.

I(512,512)

The following table shows hosting dedicated AI cluster benchmarks with the cohere.embed-multilingual-image-v3.0 hosted on one Embed Cohere unit of a dedicated AI cluster, in a scenario of an image with the height and width of 512 pixels.

Concurrency Request-level Latency (second) Request-level Throughput (Request per second) (RPS)
1 0.13 6.50
2 0.13 12.20
4 0.14 22.71
8 0.15 39.19
16 0.19 62.23
32 0.31 80.75
64 0.46 113.57
128 1.25 83.80
256 2.60 80.95

I(1024,512)

The following table shows hosting dedicated AI cluster benchmarks with the cohere.embed-multilingual-image-v3.0 hosted on one Embed Cohere unit of a dedicated AI cluster, in a scenario of an image with the height of 1,024 pixels and the width of 512 pixels.

Concurrency Request-level Latency (second) Request-level Throughput (Request per second) (RPS)
1 0.14 5.79
2 0.14 10.67
4 0.16 18.74
8 0.17 32.08
16 0.24 47.64
32 0.44 58.76
64 0.93 60.67
128 1.71 64.96
256 3.06 68.54

I(2048,2048)

The following table shows hosting dedicated AI cluster benchmarks with the cohere.embed-multilingual-image-v3.0 hosted on one Embed Cohere unit of a dedicated AI cluster, in a scenario of an image with the height and width of 2,048 pixels.

Concurrency Request-level Latency (second) Request-level Throughput (Request per second) (RPS)
1 0.26 2.82
2 0.30 4.77
4 0.29 10.43
8 0.34 18.14
16 0.57 21.93
32 1.09 25.44
64 2.08 26.99
128 4.14 26.24
256 10.17 23.60