embeddings-search — make and judge the vectors

You own the embedding technique layer: turn a corpus into searchable vectors, turn a question into a good retrieval, and measure whether that retrieval is any good. You stop the moment the right chunks come back, measured by a number. You do not assemble a prompt or generate an answer.

Route the adjacent surfaces away:

• Operating the store — collection schema, HNSW/IVFFlat tuning, metadata-filter path, quantization, ef_search recall knobs → ../vector-db/SKILL.md. You decide what vectors go in and how to query; vector-db decides how the store holds and serves them.
• The full retrieve → rerank → prompt → generate → answer loop and its groundedness / faithfulness eval → ../rag/SKILL.md.
• Pulling typed fields out of documents (invoice number, date, total) → ../structured-extraction/SKILL.md.
• Writing the prompt the model reasons with → ../prompt-engineering/SKILL.md.

1. Pick the embedding model

Decide on three axes: language coverage, quality tier (read MTEB but don't worship it), and cost — where cost is set by dimensions, because dims set storage and memory.

Model	Best when	Dims (Matryoshka)	Max input	~Price /1M tok	Query/doc asymmetry
OpenAI text-embedding-3-small	Cheap English/multi baseline	1536 (truncatable)	8191 tok	~$0.02	none required
OpenAI text-embedding-3-large	Higher quality, still API-simple	3072 (truncatable)	8191 tok	~$0.13	none required
Cohere embed-v4	Strong multilingual, API	up to 1536	long	API-priced	search_query vs search_document
Voyage voyage-3-large	Retrieval-specialised, top tasks	model-set	long	API-priced	yes (input_type)
Gemini Embedding	Tops MTEB English retrieval (~68.3)	truncatable	long	API-priced	yes (task type)
BGE-M3 / e5 (open)	Self-host, no per-token bill	1024 (BGE-M3)	long	self-host	yes (query: / passage:)

Quality anchor (mid-2026 MTEB English retrieval): Gemini ~68.3, Cohere embed-v4 ~65.2, OpenAI 3-large ~64.6, BGE-M3 ~63.0. MTEB is the standard comparison, not a verdict on your domain — references/models.md carries the full model matrix (dims, max tokens, price, input_type convention, Matryoshka support) and how to read MTEB without over-trusting it.

Two hard rules — each is a silent failure, no error, just worse results:

• Match the distance metric to the model. A cosine-trained model indexed or queried with L2 ranks silently wrong. Cosine → <=> in pgvector / Distance.COSINE in Qdrant. The index operator itself is vector-db's job; the requirement originates from the model, so state it in your config.
• Respect query/document asymmetry. Cohere, Voyage, Gemini, e5, BGE expect a different prompt or input_type for the query vs the stored passage. Embed both sides identically and recall silently drops.

Dimensions = cost. A 1024-dim float32 vector is 4 KB; at 10M docs that is 40 GB, and doubling dims doubles storage and memory. Matryoshka-trained models (OpenAI 3-*, Cohere, Gemini) let you truncate dims for graceful degradation — never re-embed the whole corpus just to shrink vectors.

2. Chunk the corpus

Start boring. Upgrade only when a number tells you to.

# Default recipe: recursive split, TOKEN-accurate count, 10–20% overlap.
import tiktoken
from langchain_text_splitters import RecursiveCharacterTextSplitter

enc = tiktoken.get_encoding("cl100k_base")
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base",
    chunk_size=512,      # tokens, not characters
    chunk_overlap=64,    # ~12% — keeps sentences from being cut mid-thought
)
chunks = splitter.split_text(document_text)

Counting by characters instead of tokens is the most common own-goal: 512 characters is ~100–130 tokens of English and far fewer of CJK, so "512" silently means different things per language and per model limit.

Upgrade ladder — graduate only when retrieval metrics (section 6) justify the added compute:

Strategy	When it pays	Cost
Recursive (default)	Always start here	lowest
Semantic (group by meaning)	Topic-mixed pages where fixed splits cut mid-idea; ~70% lift over naive in some benchmarks	one extra embed pass
Late chunking	Docs heavy with pronouns/anaphora ("it", "the company"); +10–12% on those	needs a long-context model
Contextual retrieval (prepend a heading/summary per chunk)	Chunks that aren't self-contained without their section	higher compute, more tokens stored

Embed the searchable text; store the rest as metadata. What you embed is what gets matched — don't bury the answer text under boilerplate, and don't embed raw HTML.

3. Construct the query

The query is half the retrieval. Embed it the way the model expects, then improve it only when recall data says you should.

# Asymmetric model: query and document use DIFFERENT input_type. Getting this wrong is silent.
q_vec   = embed(text=user_question, input_type="search_query")     # Cohere / Voyage
d_vec   = embed(text=passage,       input_type="search_document")
# e5 / BGE convention is a textual prefix instead:
#   query    -> "query: how do refunds work"
#   passage  -> "passage: Refunds are processed within 14 days…"

Query-side techniques, when each pays:

• Query rewriting — when user queries are terse or full of pronouns; normalise before embedding.
• HyDE (embed a hypothetical answer, not the question) — when questions are short and answers are long/technical, so the answer-shaped vector lands nearer the passage.
• Multi-query (fan out 3–4 paraphrases, union the hits) — when one phrasing under-recalls; costs N embeds and a dedup.

4. Hybrid + rerank

Dense and sparse fail in complementary ways: BM25 nails exact terms, IDs, SKUs, rare tokens; dense nails paraphrase. That is why exact-match queries return nothing while paraphrases work — the fix is adding sparse, not a bigger embedding model.

Fuse by rank, not score, with Reciprocal Rank Fusion so you never have to calibrate BM25 tf-idf magnitudes against cosine magnitudes per corpus:

# RRF: each doc scores 1/(k + rank) summed across the dense and sparse lists. k≈60.
def rrf(*ranked_lists, k=60):
    scores = {}
    for lst in ranked_lists:                 # fan-in 20–100 per list
        for rank, doc_id in enumerate(lst):  # rank is 0-based
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Then a cross-encoder reranker sits AFTER fusion: take top-50, score each against the original query, keep top-5 for downstream use.

• Current models: Cohere rerank-v4.0-pro / rerank-v4.0-fast (note rerank-3.5 is deprecated). Voyage rerank-2.5 (2025-08-11) is the first widely available instruction-following reranker — 32K-token context (8× Cohere v3.5), reports +7.94% accuracy vs Cohere v3.5 on a 93-dataset suite.
• A reranker raises precision but cannot recover a doc the retriever never returned. Recall is upstream. If the right chunk isn't in the top-50, no reranker saves you — fix recall first.

The fusion and index mechanics per engine (how Qdrant/Weaviate/pgvector run hybrid) are vector-db's: ../vector-db/SKILL.md.

5. Measure retrieval quality

This is the rigor of the skill. "Search is bad" is not actionable; "recall@10 is 0.62" is.

1. Build a golden query set — 30–50+ labeled query → relevant doc ids pairs drawn from real questions. This is the asset; everything else is reproducible from it.
2. Pick metrics — definitions, a runnable Python eval skeleton, golden-set construction and the A/B-one-change methodology are in references/evaluation.md:
   • recall@k — of the truly relevant docs, how many landed in the top-k. Catches "the right chunk never came back."
   • nDCG@k — rewards relevant docs ranked higher. Catches "right docs, wrong order."
   • MRR — how high the first relevant doc sits. Catches "the one answer is buried."
3. Move-the-number loop. Establish a baseline, change exactly one thing (model OR chunk size OR fusion OR reranker), re-measure on the same query set. Two changes at once and you learn nothing.

A retrieval change shipped without a before/after number on a golden set is a guess. verify.sh flags hybrid/rerank artifacts that mention no recall/nDCG/MRR for exactly this reason.

Anti-patterns

Anti-pattern	Why it bites	Do instead
Chunk size in characters	"512" means a different token count per language/model	Token-accurate count (tiktoken/model tokenizer)
Same input_type for query and document	Asymmetric models silently lose recall, no error	search_query vs search_document (or query:/passage:)
Cosine model indexed/queried with L2	Ranking is silently wrong	Match metric to model (cosine → <=>)
Add a reranker to fix bad recall	Reranker only reorders what retrieval returned	Fix recall (hybrid, chunking, model) first
No overlap on prose	Sentences cut mid-thought lose the answer	10–20% overlap
Tuning by eyeballing one query	One query isn't a measurement	Golden set + recall@k/nDCG before vs after
Trusting MTEB rank for your domain	Leaderboard ≠ your corpus/language	Eval the top 2–3 on your own golden set
Over-large dims "for safety"	Doubles storage/RAM, little recall gain	Right-size; truncate via Matryoshka
Re-embedding the corpus to change dims	Wasteful when the model is Matryoshka-trained	Truncate dims, don't re-embed
Embedding raw HTML/boilerplate	Match signal drowns in markup	Embed clean text; keep the rest as metadata