Cruxly

Cruxly › AI engineering › LLM Foundations

Embeddings and Similarity: Cosine, Dot Product, and What a Vector Encodes

the contrastive loss, cosine versus dot versus L2, and 50 million vectors in gigabytes

Open the interactive lesson

Prompt: "Search for 'not refundable' returns documents about refundable items. Your vector database has 50 million chunks at 1,024 dimensions and the infra lead asks how much RAM it needs. A colleague asks whether to use cosine or dot product and whether it matters. Explain what an embedding is, answer all three, and say what you would change."

40 minutes first. Compute cosine and squared distance for two unit vectors, then size the index in gigabytes. The trap is treating an embedding as a meaning-preserving compression of text; it is a vector trained to do one job well, and it is bad at others.

What an embedding model is trained to do, and what that implies

An embedding model is trained with a contrastive objective: pull the vectors of matching pairs together, push non-matching ones apart. The pairs are things like a question and the passage that answers it, or two paraphrases. Inside a training batch, every other example's passage serves as a negative. The usual loss, InfoNCE, is a softmax over the batch: for query i, the probability of picking its own passage i among all of them, with similarities divided by a temperature tau.

Work a batch of two. Similarities between queries (rows) and passages (columns) are [[0.8, 0.3], [0.2, 0.6]]; the diagonal is the true pair.

tau = 1.0:  query 1 softmax over [0.8, 0.3] = [0.622, 0.378]   -log(0.622) = 0.474
            query 2 softmax over [0.2, 0.6]  -> p(correct) = 0.599 -> 0.513   mean loss = 0.4935
tau = 0.1:  query 1 softmax over [8, 3]       = [0.993, 0.007]  mean loss = 0.0124

Lowering the temperature sharpens the softmax, so the same similarity gaps count for much more and the loss falls to near zero; the training effect is a pressure to widen those gaps. The loss only cares about ordering within a batch: matching pairs above non-matching ones. That has direct consequences for what the vector can and cannot hold.

An embedding is a vector for retrieval and comparison, tied to the model that made it. Vectors from two different models live in unrelated coordinate systems: you cannot compare them, mix them in one index, or upgrade the query model without re-embedding the corpus (RAG From Scratch, No Framework shows the mixed-version failure).

Cosine, dot product and L2: the same ranking on unit vectors

On unit-length vectors, cosine similarity, dot product and Euclidean distance rank candidates identically. For unit vectors a and b, |a - b|^2 = |a|^2 + |b|^2 - 2 a.b = 2 - 2 cos(a, b). Work it with a = [0.6, 0.8] and b = [0.8, 0.6]:

cosine = dot = 0.6*0.8 + 0.8*0.6 = 0.96
squared L2 = (0.6-0.8)^2 + (0.8-0.6)^2 = 0.04 + 0.04 = 0.08
check: 2 - 2*0.96 = 0.08

Higher cosine means smaller distance, monotonically, so any of the three give the same top-k. That is why "cosine or dot product?" has a short answer for most hosted embedding models, which return normalised vectors: it does not matter, so pick the one your index computes fastest (dot product on normalised vectors is a plain matrix multiply).

It matters when vectors are not normalised. Dot product mixes angle and length. Take a query [3, 0] and two candidates: v1 = [1, 1] and v2 = [0.9, 0].

dot:    q.v1 = 3.0     q.v2 = 2.7     -> v1 ranks first
cosine: q,v1 = 0.707   q,v2 = 1.000   -> v2 ranks first (same direction as the query)

Dot product rewards the longer vector even though it points a different way. Some models are trained for dot product on purpose and use vector length as a signal (popularity or document quality); others assume cosine. The rule is the model card's: use the metric the model was trained with, and if you are unsure, normalise and use cosine. Mismatched metric and model is a quiet quality bug because nothing errors; recall just drops.

Where you meet L2: index libraries often default to it, and the identity above means a normalised corpus makes it equivalent to cosine. Where you do not want normalisation: when lengths carry meaning you chose to keep, which you should know you did.

Cosine is also a score, not a probability. 0.82 means "close in this model's space", and its meaning differs by model and by corpus. A fixed threshold such as "above 0.8 is relevant" must be calibrated on labelled pairs for your model, and re-calibrated when you change it. Many models use only a narrow band of the possible range (anisotropy: vectors cluster in a cone), so scores of 0.7 to 0.9 can be common for unrelated text.

Asymmetric retrieval, and mixing models

Search is asymmetric: a short question and a long passage are different kinds of text, and good retrieval models are trained to know it. Many embedding models take an instruction or prefix, such as one marker for queries and another for documents, and they were trained with those markers. Omit them and quality drops, sometimes by a lot, with no error message. The model card lists the exact strings. Use one rule: the prefix you used to index must match what the model card says for documents, and the query side must use the query form.

Some tasks are symmetric: deduplication, clustering, and "find similar documents" compare like with like, so both sides use the same form. Mixing the two patterns up (a query prefix on documents) is a common silent bug.

Another approach to the question-versus-answer gap is to embed a hypothetical answer instead of the question: the HyDE paper (Gao et al., arXiv 2212.10496) generates a fake answer with an LLM and searches with its embedding. It can help when queries are very short or oddly phrased, but it adds an LLM call to every search and can anchor on a wrong guess, so treat it as an experiment to measure, not a default.

Can you mix vectors from two models? No, for the reason above. The practical consequences:

How to choose one is a separate topic: benchmarks such as MTEB (Muennighoff et al., arXiv 2210.07316) compare models across many tasks and are a start, but a model's average rank on a public benchmark is not your corpus. Build 100 labelled query-passage pairs from your own data and measure recall@k on your own queries; the retrieval lessons later in this track treat selection and evaluation in depth.

Dimension and storage: 50 million vectors in gigabytes

Index size is count times dimensions times bytes per number, and every lever is a multiplier on one of those. For 50 million chunks at 1,024 dimensions:

float32: 50e6 * 1024 * 4 bytes   = 204.8 GB
int8:    50e6 * 1024 * 1 byte    =  51.2 GB   (4x smaller)
binary:  50e6 * 1024 / 8 bytes   =   6.4 GB   (32x smaller)
256 dims, float32:  50e6 * 256 * 4 = 51.2 GB  (4x smaller than 1,024 dims)

Smaller sizing example: 1 million vectors at 1,536 dims in float32 is 6.14 GB. Those figures are for raw vectors. A graph index such as HNSW adds link overhead on top (the multiplier depends on parameters), and you still want memory headroom, replicas and room for growth; Multi-Tenant, Multilingual RAG at Scale covers the index choices and the post-filter trap. A brute-force scan of 50 million 1,024-dim vectors is about 2 x 50e6 x 1024 = 102 GFLOP per query, which is why you build an index instead.

Matryoshka representation learning (Kusupati et al., arXiv 2205.13147) trains the vector so its first k dimensions are a usable lower-dimensional embedding. You can truncate (and renormalise) to 256 or 512 dims and keep most of the quality, instead of training several models. The paper's abstract reports up to 14x smaller embeddings at the same accuracy on ImageNet-1K classification; text retrieval behaviour depends on the model, and only models trained this way tolerate truncation. Truncating a model that was not trained for it can hurt badly. Some hosted embedding APIs expose a dimensions option for such models; check the current docs of your provider.

What to say in the room: "Start from the recall target. Measure recall@10 at 1,024 dims, then at 256 and with int8, on a labelled sample from our corpus. Take the smallest configuration that keeps recall within the tolerance, often with a two-stage plan: cheap vectors to fetch 200 candidates, then a reranker on the originals or a cross-encoder for the final 10." The measured loss is a number you quote from your own test, not a figure from a paper.

:::warn Quantising or truncating changes the similarity scores, so thresholds calibrated on full vectors no longer apply. Recalibrate after any storage change. :::

Failure modes, privacy, and what else vectors are for

Name the failure by what the objective never required. Why does "not refundable" return refundable documents? Contrastive training rewarded topical closeness, and negation barely changes topic. With invented toy vectors, a reference passage might sit at cosine 0.997 from "refundable" and 0.911 from "not refundable": both are neighbours of the same region, and a small margin is easily outvoted by other terms. The cure is not a better threshold. Use hybrid retrieval with a lexical signal for exact words, a reranker that reads query and passage together (a cross-encoder sees "not" in context), and an evaluation set with negation cases in it.

Other known weak spots:

Embeddings can be inverted. Morris et al. (arXiv 2310.06816, EMNLP 2023) show an iterative method that recovers 92 percent of 32-token texts exactly from their embeddings, and it could recover full names from clinical notes. A vector database is therefore not an anonymisation step: treat embeddings of personal data as personal data (access control, retention and deletion), as discussed in Red-Teaming and Compliance as Code. The 92 percent is that paper's result on its own setup; real risk depends on the model and text length.

Vectors are useful far beyond search.

How to say it in the interview

"An embedding model is trained so that texts which should match land close together and others land far apart, so a vector encodes whatever distinctions that training objective needed, and not much else. On unit-length vectors cosine, dot product and L2 give the same ranking. Storage is count times dimensions times bytes: 50 million at 1,024 dims is 205 GB in float32, which is why I would quantise or truncate before I scaled hardware."

Checklist

Practice this interactively
Share this lessonWhatsAppLinkedInX

Related lessons