Embedding models are the infrastructure layer of modern semantic search and retrieval-augmented generation (RAG). They convert text into dense vectors that encode semantic meaning, enabling systems to find relevant documents by meaning rather than keywords. The choice of embedding model directly impacts retrieval quality, cost, latency, and operational complexity. This guide walks ML engineers through the concrete trade-offs between OpenAI, Cohere, and open source options, with benchmark data and selection criteria that will remain relevant as the market matures.
Why this matters now
The embedding model landscape has stratified into three distinct categories with radically different economics and trade-offs. In 2024 and 2025, open source embedding models caught up to closed-source alternatives on most MTEB benchmarks, eliminating the traditional "pay for quality" calculus. Simultaneously, API costs for embeddings have become visible as a real line item in RAG applications that embed thousands of documents and process millions of queries monthly. Teams are now optimizing not just for retrieval quality but for the cost and latency of the entire RAG pipeline.
At the same time, vector databases have matured and deployed everywhere, making self-hosted embeddings increasingly viable for organizations that previously defaulted to cloud APIs. The decision is no longer binary. Most production systems now run a hybrid approach: using high-quality, fine-tuned embeddings for the core document collection and falling back to lighter models for query-time embedding or exploratory work. Understanding the spectrum of options is essential for making that hybrid strategy work.
The three categories: Commercial APIs, open source, and specialized models
The embedding market breaks into three tiers, each with a distinct cost and quality envelope.
Commercial APIs (OpenAI, Cohere) offer the highest retrieval quality and multilingual coverage out of the box. OpenAI's text-embedding-3-large consistently ranks at the top of MTEB leaderboards, and its multilingual performance is strong. Cohere's embed-english-v3.0 is competitive on English tasks and better priced at scale. Both offer API convenience, automatic model updates, and enterprise support. The trade-off is latency (100 to 500ms round-trip) and per-token costs that accumulate quickly in high-volume applications.
Open source models (BGE, E5, Jina, Voyage) can be self-hosted, fine-tuned, and run entirely on your infrastructure. The MTEB leaderboard is now dominated by open source models in the 7B to 1B parameter range. Inference latency is dramatically lower (5 to 50ms) and marginal cost approaches zero. The friction is operational: you manage the model, the infrastructure, the updates, and the fine-tuning pipeline. For organizations with the engineering capacity, the ROI is substantial. For smaller teams, the operational burden outweighs the savings.
Specialized models are smaller models fine-tuned for specific tasks (medical, legal, code) or optimized for speed-quality trade-offs. These often have lower latency and memory requirements but narrower scope. They are most useful when you have a well-defined domain and can justify custom fine-tuning.
Benchmark data: MTEB and what it actually predicts
MTEB (Massive Text Embedding Benchmark) is the standard evaluation suite for text embeddings across retrieval, clustering, pair classification, and semantic similarity tasks. The score is aggregated across 56 datasets. It is the closest approximation to a universal embedding quality metric, but it has important limitations.
As of late 2025, the top performers on MTEB are:
- OpenAI text-embedding-3-large: 64.3 aggregate score, 73.4 on retrieval tasks
- Cohere embed-english-v3.0: 64.1 aggregate, 73.2 on retrieval
- Jina v3 (open source): 63.8 aggregate, 72.9 on retrieval
- BGE m3-large (open source): 63.5 aggregate, 72.4 on retrieval
- E5-large-v2 (open source): 62.5 aggregate, 70.8 on retrieval
The gap between the top paid API (OpenAI) and the best open source model (Jina v3) is less than 1 percentage point on retrieval. This is the single most important fact for cost-conscious teams: you can no longer justify commercial embeddings purely on quality grounds. The decision hinges on operational cost, latency, and domain-specific performance.
MTEB is not perfectly predictive of your application's performance. A model's MTEB retrieval score correlates well with ranking quality on generic English corpora, but breaks down for:
- Domain-specific text (legal documents, medical notes, code)
- Short text and queries (titles, tags, user input)
- Cross-lingual retrieval and zero-shot transfer
- Hard negatives and dense corpora where top-100 recall matters more than global retrieval
Benchmark scores are necessary but not sufficient. Always run a small evaluation on your actual data before production rollout.
Cost analysis: Total cost of ownership for embedding
The simple per-token cost is misleading because it ignores volume, caching, and infrastructure amortization.
OpenAI pricing: text-embedding-3-small at $0.02 per 1M tokens and text-embedding-3-large at $0.13 per 1M tokens. For a typical RAG application that indexes 100,000 documents (10M tokens) and processes 10,000 queries per day (1M tokens daily), the monthly bill is roughly $600 for small embeddings or $3,900 for large. That is non-trivial but often worth it for quality and simplicity.
Cohere pricing: embed-english-v3.0 costs $0.10 per 1M tokens, placing it between OpenAI's two tiers. For the same hypothetical workload, monthly costs are $3,000. Cohere is priced for high-volume users and offers better discounts at scale.
Open source models: Marginal cost is zero once deployed. Initial costs are infrastructure (GPU, memory, storage), model serving, and engineering time to integrate and fine-tune. A single A100 GPU costs roughly $2,500 per month on-demand, can serve 100 concurrent embedding requests, and amortizes quickly if you are processing millions of embeddings monthly. For organizations embedding fewer than 10M tokens per month, the API is cheaper. For those embedding 100M+ tokens monthly, self-hosting becomes economically rational.
A hidden cost is caching and lookup. Embeddings for static documents (your knowledge base) should be precomputed once and stored in a vector database. Only queries need real-time embedding. This reduces API volume dramatically and makes even expensive closed-source embeddings cost-effective.
Latency and real-time inference trade-offs
Latency matters differently depending on the application. For batch indexing (embedding your knowledge base once), latency is irrelevant. For query-time embedding in a search interface, latency directly affects user experience.
OpenAI and Cohere APIs: Round-trip latency is typically 150 to 400ms from a US region. This includes network overhead, inference on their servers, and response transmission. In high-concurrency scenarios, queuing adds additional latency. This is acceptable for most search interfaces but becomes problematic if you are embedding high-frequency user input (real-time autocomplete, per-keystroke search).
Self-hosted open source models: Inference latency for a 300-token input on an A100 GPU ranges from 5 to 30ms depending on model size. E5-large and BGE-large are in the 15 to 25ms range. text-embedding-3-large is much larger (3072 dimensions) and would take 40 to 80ms on GPU. The operational latency (model loading, tokenization, batching) adds another 5 to 20ms. Self-hosted models are 5 to 20x faster for query embedding and enable truly real-time applications.
Latency optimization matters more than raw speed in production. Batch processing queries (embedding in batches of 32 or 64) reduces per-query latency and improves GPU utilization. Caching embeddings of common queries eliminates latency entirely. Most RAG systems should prioritize batch inference for documents and strategic caching for queries.
Multilingual performance and cross-lingual transfer
If your corpus spans multiple languages or you need to support users in different regions, embedding model choice becomes more constrained.
OpenAI text-embedding-3-large: Officially supports 100+ languages in a single model. Performance on non-English languages is materially lower than English (roughly 5 to 10 percentage points lower on MTEB multilingual retrieval tasks), but still competitive with specialized multilingual models. It is the safest choice for multilingual applications because OpenAI maintains coverage as languages evolve.
Cohere embed-english-v3.0: Also supports 100+ languages in a single model with similar multilingual performance to OpenAI. Cohere has separate language-specific models (embed-french, embed-german) optimized for each language, but the multilingual variant is generally preferred for simplicity.
Open source multilingual models: multilingual-e5-large, Jina v3, and BGE m3 all support 100+ languages. Performance on English-specific tasks is slightly lower than English-only variants (E5-large-v2 scores 62.5 on MTEB, but multilingual-e5-large scores 59.3). For non-English languages, multilingual models are usually on par with or better than commercial APIs. The trade-off is clarity: open source multilingual performance is less well-documented and may be uneven across languages.
For organizations with a dominant language and occasional cross-lingual queries, using language-specific embeddings (English-only E5 for English, separate models for other languages) is often more effective than a single multilingual model. This requires managing multiple models and increasing operational complexity but improves quality by 2 to 5 percentage points.
Practical dimensions: Size, speed, and storage constraints
Embedding dimensions determine the size of the vector and the amount of storage and compute required for similarity search.
OpenAI text-embedding-3-small produces 512-dimensional vectors. text-embedding-3-large produces 3072-dimensional vectors. Cohere produce 1024-dimensional vectors by default. Open source models vary: E5-large is 1024 dimensions, BGE-large is 1024 dimensions, Jina v3 is 8192 dimensions (but can be truncated). Smaller dimensions reduce storage costs and query latency in the vector database, but lose information. The MTEB leaderboard scores are for models at their full dimensionality, so truncating a model typically costs 1 to 3 percentage points of retrieval quality per 50% reduction in dimensions.
For most applications, 768 to 1024 dimensions is a practical sweet spot. It captures most of the semantic information, stores easily in a vector database, and runs fast on commodity hardware. 512 dimensions is aggressive and suitable only for high-volume applications where retrieval quality can be traded for speed. 3000+ dimensions (OpenAI's large variant) is overkill for most search applications and adds latency and cost with minimal quality improvement.
A concrete example: storing 1 million documents with 1024-dimensional embeddings requires roughly 4 GB of vector storage (1M * 1024 dimensions * 4 bytes per float32). Searching across 1 million vectors takes 5 to 20ms depending on the vector database index structure. With 3072 dimensions (OpenAI large), storage balloons to 12 GB and search latency increases proportionally. Unless you have a very hard retrieval problem, the larger model is not worth the operational overhead.
When this fails: Limitations and failure modes
Embedding models are robust but have known failure modes.
Long context decay: Embedding models compress long documents into a single vector, losing position information. The meaning of text at the end of a long document (over 512 tokens) gets attenuated. For document retrieval, this is usually acceptable because you split documents into chunks before embedding. For embedding entire search queries or user prompts, long context decay is not a factor.
Domain drift: A model trained primarily on web text will perform poorly on specialized text (medical journals, legal documents, source code). MTEB scores do not predict this. You will discover domain drift only by evaluating on your actual data. The solution is either to use a domain-specific model or fine-tune a general model on a small labeled dataset of your domain text.
Short text fragility: Embedding models trained on longer passages (the standard for MTEB) often struggle with very short text (1 to 3 word titles, hashtags, product names). The vector space becomes sparse and noisy for short queries. Solutions include using a separate short-text model, augmenting short text with synthetic context, or accepting lower retrieval quality on edge cases.
Semantic drift and out-of-distribution queries: A model trained on general English text will not embed ambiguous or adversarial inputs reliably. Misspellings, slang, and sarcasm are handled inconsistently. This is rarely a problem in practice because search interfaces and RAG systems are written by humans who tend to use clear language. But it is worth awareness in open-ended applications like customer support search.
API rate limits and cost surprises: OpenAI and Cohere have per-minute token rate limits that can unexpectedly block production traffic during traffic spikes. Cohere's generous free tier (100M tokens monthly) makes it easy to accidentally scale beyond the free tier. Budget monitoring and rate limiting on your side are essential.
Selection criteria: Picking the right model for your constraints
A decision tree for embedding model selection:
- Do you need real-time query embedding under 50ms latency? If yes, you must self-host. APIs cannot meet this requirement.
- Are you embedding more than 100M tokens monthly? If yes, self-hosting is economically rational unless you have no engineering capacity.
- Is your corpus domain-specific (medical, legal, code)? If yes, benchmark a domain-specific model against general models on your data. Domain-specific models often outperform general models by 5 to 10 percentage points even if they score lower on MTEB.
- Do you need multilingual support across more than 2 languages? If yes, use a multilingual model. OpenAI and Cohere are safest. Open source multilingual models work but require more evaluation.
- Can you afford $3,000 to $10,000 per month for API costs? If yes, OpenAI or Cohere is a reasonable choice for simplicity. No model management, no infrastructure to maintain.
- Do you have 1 to 2 GPU-hours of capacity available? If yes, self-hosting open source is viable and will save money within 3 months at scale.
The default choice for most organizations in 2026 is: (1) Use OpenAI text-embedding-3-small for initial prototyping and small-scale production; (2) Evaluate open source models (BGE m3-large, Jina v3, E5-large-v2) on your actual data at the 3-month mark; (3) If retrieval quality is similar, migrate to self-hosted to eliminate API costs. This hybrid path lets you start fast and optimize cost as volume scales.
Organizations with strong ML engineering should skip step 1 and start with open source. The setup is slightly harder (vector database, serving infrastructure) but the long-term cost and latency advantages are substantial. Conversely, organizations without GPU infrastructure or ML expertise should stay on APIs indefinitely. The convenience premium is worth the cost.
Production systems should always run a small evaluation before committing. Download a few thousand documents and their click-through or relevance labels. Embed them with both models. Measure Recall@10 and Recall@50 on your actual queries. A 1 to 2 percentage point difference in retrieval quality on your task is usually not worth switching models, but a 5+ percentage point difference is substantial and worth the operational cost of a new system.
Practical next steps
Start by identifying your embedding volume and latency requirements. Sketch out a simple cost model: (API tokens per month) * (price per million tokens). Cross-reference that against the cost of running a self-hosted model on your infrastructure. Most teams will find that APIs are cost-effective for the first 12 months and self-hosting becomes attractive only as volume scales into hundreds of millions of tokens monthly.
In parallel, benchmark the top 3 models on your actual corpus and queries. Do not rely on MTEB scores alone. A 30-minute evaluation on your data is worth weeks of reading benchmark leaderboards. Use that evaluation to pick your initial model. Plan a 6-month review to reassess as models improve and your requirements clarify. The embedding model landscape is moving fast, but it has stabilized enough that a 6-month window without re-evaluation is reasonable before you start leaving quality or cost on the table.
