How Does a RAG Reranker Really Work?

When RAG retrieval disappoints, the advice AI engineers hear today is almost always “add a reranker”. Ask why a reranker works, and the answer usually stays at the architecture level: it is a cross-encoder, it applies attention over the query and the passage together, it is fine-tuned on relevance labels. All of that is true, and none of it says what the model actually learned. Push one level down, to terms a business partner could check, and the explanation usually stops.That gap matters. A team that cannot say in plain terms what the reranker does cannot defend the choice to use one, and cannot spot the cases where a keyword lookup would beat it for a fraction of the cost.This article gives the honest answer, the one you can hand to your business partner without waving hands. The reranker is not smarter than the embeddings step below it. It runs the same mechanism (statistical token association from training data), just conditioned differently (on the query-passage pair rather than each text independently). Once you see that, the “when to use a reranker” question stops being “add it because the tutorial did” and becomes “add it only when this specific tradeoff is worth paying for”.🧭 New to the series? Start with the map: Prompt, Context, Loop sets out the three engineering layers every RAG system is built on, the prompt (the call itself), the context (what fills the model’s window), the loop (when the next call fires and when it stops), and walks the whole series through that lens, article by article. It is the shortest way to see what is covered and where this one sits.This article sits in Part I, alongside the embeddings triptych (2A / 2B / 2C). - Image by author📓 Try the reranker on your own PDF at doc-intel/notebooks-vol1. The companion notebook loads a cross-encoder, applies it to a keyword-filtered top-K, and shows both the score and the tokens driving it. Change the query, watch which keywords carry the ranking.1. What data scientists say, and why it isn’t enoughAsk three data scientists what a reranker does and you get three answers, roughly:“It’s a cross-encoder. It scores the query-passage pair jointly and gives a relevance score.” Technically true, but the words cross-encoder and relevance are hiding what the model actually learned.“It applies attention over both texts, so it sees the interaction between them.” True at the architecture level, but architecture does not tell you what the model is doing with that attention.“It’s trained on relevance labels, so it learns which passages answer which questions.” Very close, but “learns which passages answer” is the wrong verb. The model does not learn to answer. It learns which tokens co-occurred.None of the three is wrong. All three are incomplete in a way that matters when you have to decide whether to keep the reranker in your pipeline, whether to fine-tune it on your corpus, or whether to replace it with something cheaper.The rest of this article walks that answer down to the mechanism, then names three consequences that change how you architect enterprise RAG.2. What actually happens inside a rerankerThe reranker is a specific kind of transformer, trained on a specific kind of data, that produces a specific kind of number. Each of those three pieces matters.2.1 The architecture: cross-encoder, not bi-encoderAn embedder (bi-encoder) reads the query alone, produces one vector. Reads a passage alone, produces one vector. Compares the two vectors by cosine. Each text is embedded independently, and the model never sees them together during scoring.A reranker (cross-encoder) reads the query and the passage together, as one concatenated input: [CLS] query [SEP] passage [SEP]. It runs BERT-style attention over the joint input, where every token can attend to every other token. It outputs a single relevance score.That “reads them together” is the whole architectural difference. Bi-encoder: two vectors, one comparison operation. Cross-encoder: one forward pass, one score. The joint attention is why the reranker feels smarter, and why it is 30 to 100 times slower per query.2.2 The training data: MS MARCO and its cousinsWhere does the reranker learn its scoring? From query-passage relevance pairs labeled by humans. The canonical dataset is MS MARCO (Bajaj et al. 2016, one million real Bing search queries with human-graded passage relevance). Others: Natural Questions (Google search + Wikipedia paragraphs), BEIR (a benchmark aggregator), TREC.Every training example is a triple: (query, passage, relevance_label). The model sees millions of these, and its weights adjust so that pairs labeled relevant get higher scores than pairs labeled not relevant.That is the sole learning signal. The model is never shown a question and asked to compose an answer; it is shown pairs, and it optimizes for a score that separates relevant pairs from non-relevant ones.Which raises the honest question: what pattern actually separates them in the training data?2.3 What the model really learns: keyword co-occurrence at the pair levelHere is the level down that rarely gets explained.The model looks at millions of (query, passage, relevance) triples and asks: what patterns in the joint token stream predict the relevance label? The dominant pattern is not “answering”. It is which query tokens tend to co-occur with which passage tokens in high-relevance pairs.Concretely, in MS MARCO the query “how to cancel my subscription” is labeled relevant against passages containing cancel, subscription, unsubscribe, terminate, end your membership. Millions of examples reinforce that when the query contains cancel, passages containing terminate or unsubscribe tend to be labeled relevant. The reranker’s weights absorb that association.So the “smart” reranker is doing keyword linking, at the query-passage pair level. It is a learned association table between query token neighborhoods and passage token neighborhoods, dressed up as a neural network score.The embedder does the same thing, but at each text independently. The reranker does it conditioned on the pair. Same mechanism, different conditioning.Second-order signals the reranker also picks up: positional patterns (a term appearing early in the passage often correlates with relevance), syntactic structure (subject-verb-object relations that link query tokens to passage tokens), the presence of definitional phrasing (“X is Y”). Those help, but they are second-order; the dominant signal is keyword co-occurrence.Why this frame matters: once you see the mechanism, the “will it work on my corpus?” question has a clear answer. If your corpus vocabulary and query vocabulary look like MS MARCO (general English, common web topics), the trained associations transfer, and the reranker feels magical. If your corpus vocabulary is specialized (insurance contracts, medical records, regulatory filings), the trained associations do not cover your domain, and the reranker inherits the same out-of-vocabulary failures as the embedder below it. No amount of “but it’s a cross-encoder” fixes that.3. The mechanism, shown: where the reranker wins, where it hits a wallSection 2 made a claim: the reranker is a learned association table between question-language and answer-language. That claim is testable. Take a handful of candidates, score them with three embedders (MiniLM, ada-002, text-embedding-3-large) and three cross-encoders (bge-base, bge-large, ms-marco-MiniLM), and read each row.3.1 Where it wins: the answer that does not repeat the questionAsk “What is the maximum coverage amount?” against three passages: the answer (“Cover is capped at 50,000 euros per year”), an echo that repeats the question’s words without answering (“The maximum coverage amount can be found in the benefits schedule”), and a distractor.Every embedder ranks the echo first; both bge rerankers flip the answer to the top. - Image by authorEvery embedder puts the echo first. It shares maximum, coverage, amount with the question, so its vector sits close. The answer shares almost nothing lexically, so it lands second or third. The two bge rerankers flip it: they read the question and the answer together, recognize that a “capped at X per year” passage answers a “maximum coverage amount” question, and lift it to #1. This is the reranker doing its one real job, bridging the question’s words to the answer’s words.It is not a one-off. The same flip reproduces on plain factoids:Same shape, general-knowledge version. bge lifts the answer over the echo, ms-marco keeps the echo on top. - Image by authorAcross a dozen queries of this shape (who wrote a play, the boiling point of water, the speed of light, the first president, plus the enterprise trio of deductible, notice period, coverage) the two bge rerankers rescue the answer to #1 where every embedder ranked an echo above it. The win is real and repeatable, on exactly one shape: a short factual answer that does not repeat the question, sitting behind an echo that does.Two honest caveats sit in the same two figures. First, not every reranker does it: ms-marco-MiniLM keeps the echo on top in both cases, the same lexical bias an embedder has. Second, when a strong embedder already answers the question (text-embedding-3-large gets several of these on its own), the reranker adds nothing over just using a better embedder.3.2 Where it hits a wall: your private vocabularyNow the case that decides the enterprise question. Ask “what’s the rule on contractor overtime?” where the answer uses the company’s own term, “non-employee labor compensated beyond 40h/week”, and never the word contractor.The answer never says “contractor”, it says “non-employee labor”. Every model, embedder and reranker alike, ranks it last. - Image by authorEvery column, embedder and reranker, ranks the answer last. The surface match (“Contractors are paid on a per-project basis”) wins. The reranker never saw contractor map to non-employee labor in MS MARCO, so its association table has no entry for it. The cross-attention it runs is real, but it can only fire on associations it learned, and this one it never learned.3.3 To clear that wall, you must already know the answerThe fix the literature offers is fine-tuning: feed the reranker labeled (question, passage, relevant) triples from your own domain until it learns that contractor maps to non-employee labor. But look at what labeling one of those triples requires. Someone who knows the domain has to point at the right passage and say this one answers the question. To point at it, they had to recognize that “non-employee labor beyond 40h/week” is what the answer looks like. That recognition is the answer keywords.So the training label and the dictionary entry carry the same information. For a “maximum coverage amount” question, labeling the answer means knowing the answer contains capped at, up to, a currency, per year. Writing the expert dictionary means typing exactly that: {capped at, up to, maximum, €, per year}. For the contractor case, labeling the pairs means knowing that contractor equals non-employee labor in this company, and the dictionary entry is that one line.The difference is the cost and the shape. The reranker needs hundreds of labeled pairs to generalize the mapping statistically, a retraining run, and it stays a black box scoring 0.83. The dictionary needs one line, fires deterministically, and shows the exact keyword that matched under audit. If you already know the answer well enough to label the data, you already know the answer keywords, and writing them down is the cheaper, auditable path. The reranker’s statistical learning only pays when the mapping is too broad to enumerate, which is the open web, not a bounded enterprise domain.4. Why the answer matters in enterpriseThree consequences flow from the honest answer, and each of them changes an architecture decision you may have made without noticing.4.1 The audit trail is opaqueA relevance score of 0.83 from a reranker is not defensible under scrutiny. A regulator asking why was this passage returned? gets “the reranker gave it 0.83” as an answer. That is not an audit trail. It is a black box that produced a number.Contrast with a keyword filter: the retrieved passage contains force majeure and pandemic. That statement is inspectable, replayable, and defensible. If the retrieval was wrong, you can trace which keyword was missing from the dictionary and add it. If a reranker was wrong, you shrug at the score and move on, or you retrain the whole thing.For enterprise use cases where retrieval decisions have compliance or contractual consequences (insurance underwriting, legal discovery, medical records, regulatory reporting), opacity is not a small tradeoff; it is a disqualifier.4.2 The cost is realA cross-encoder is 30 to 100 times slower per query than a bi-encoder. If your bi-encoder scores 1000 candidates in 20 ms, the reranker scores the same 1000 in 600 ms to 2 seconds. In practice, you do not rerank 1000 candidates: you take the bi-encoder’s top-20 or top-50 and rerank only those, which puts the added latency back in the 15 to 100 ms range, depending on the depth and the model.That is fine at low query volume. At 100 queries per second sustained, the reranker cost is a real operational line item: more GPU capacity, longer p99 latencies, more infrastructure to keep warm. The value it adds has to justify that cost, and that only happens when its trained associations genuinely cover your vocabulary. On out-of-domain enterprise corpora, it often does not.4.3 The vocabulary gap will show upEvery failure mode catalogued for embeddings on out-of-domain enterprise vocabulary applies to the reranker too, because it was trained on the same distribution (general web search). Force majeure and act of God are equivalent in an insurance contract but land in different neighborhoods in the reranker’s learned associations, because it saw them in different training contexts. Rescission was rare in MS MARCO. ShieldPro Elite was not there at all.Fine-tuning the reranker on your domain corpus helps, but only up to a point. You need labeled query-passage pairs from your domain to fine-tune, which is exactly what enterprise teams rarely have. And even a fine-tuned reranker inherits the same underlying mechanism: it still learns token associations, just from your smaller domain corpus, and the number of examples you can label rarely matches the millions MS MARCO provides.5. What to do instead, and when to keep the rerankerGiven the mechanism and the enterprise consequences, the question becomes: what earns the reranker’s slot in your pipeline?The default in enterprise RAG (per the series’ recommendation): a curated keyword dictionary maintained by domain experts. The expert already knows that force majeure equals act of God in this contract, that rescission is the formal term for what the user called cancellation, that ShieldPro Elite is the top-tier homeowners plan. Encoding that once in a versioned YAML dictionary and running keyword-based retrieval on top gives you:Auditable retrieval (the matched keywords are inspectable)Low latency (no LLM in the hot path, no GPU cost)Durability across model releases (the dictionary outlives every reranker version)Explainability to the business (they can read the dictionary)The reranker earns its slot in four specific cases. The first three are runtime slots, the fourth is not.In-domain distribution. Your corpus vocabulary and query vocabulary genuinely look like MS MARCO (general web, common English, high-frequency topics). Consumer FAQs, public-service portals, e-commerce help. The reranker’s trained associations transfer. Use it.Semantic re-ranking of a keyword-filtered top-K. After the keyword dictionary filters the corpus down to 20 candidates, the reranker can order them by contextual relevance. This is the same role Article 2C section 5.3 assigns to bi-encoder embeddings, and a cross-encoder does it more accurately at the cost of extra latency. Worth it when the top-K is small and the ordering matters.Compliance scenarios where the reranker’s score itself is the audit artefact. If your compliance framework requires “the model scored this passage above threshold X”, the score is the artefact, and the reranker fits the requirement.Offline, to discover what belongs in the dictionary. Run the reranker over a sample of real questions and read what it pulls up. Where it surfaces a mapping the dictionary does not have yet, you have a candidate alias. An expert confirms it or throws it out, and only the confirmed line ships. The model does the searching, the expert does the deciding, and what reaches production is the validated line, never the score. Article 2C gives embeddings the same treatment, and Article 16D runs this loop continuously at corpus scale, a failed search proposing the alias and an expert confirming it.The fourth case is the one that reframes the other three. Both paths do the same job, and the diagram below puts them side by side.The same table twice: learned on someone else’s corpus, or written by people who know the words. - Image by authorOutside those four cases, the reranker mostly adds cost: impressive in a demo, expensive in production, opaque under audit, and unable to compensate for the trained associations it does not have.One equivalence sits underneath all of it, and it is worth stating in a single line. A reranker is a keyword-association table that someone else trained on someone else’s corpus. Writing your own dictionary is the same job, done by the people who actually know the vocabulary, at a fraction of the cost and in a form an auditor can read. That equivalence stays invisible as long as the model is treated as magic. Open the box, as Section 2.3 did, and the choice makes itself: use the model to find candidate links, use the expert to validate them, and let the validated table be what production runs on.6. Sources and further readingThe reranker literature is dense and largely optimistic. Reading it against the article’s frame (“cross-encoders learn keyword association at the pair level, not comprehension”) is more useful than reading it as an unqualified endorsement.Same direction as the article:Nogueira & Cho, Passage Re-ranking with BERT, 2019 (arXiv:1901.04085). The paper that introduced cross-encoder reranking with BERT and set the pattern most current rerankers follow. Reads honestly about what the model learns.Khattab & Zaharia, ColBERT, SIGIR 2020 (arXiv:2004.12832). Late-interaction retrieval. Explicitly designed to preserve token-level signal that both embedders and cross-encoders lose, which is the strongest architectural signal that the token-level pattern is what actually matters.Different angle, different context:Bajaj et al., MS MARCO, 2016 (arXiv:1611.09268). The training data that shapes what almost every commercial reranker actually knows. Worth skimming to see the query and passage distribution the reranker’s associations come from.Muennighoff et al., MTEB: Massive Text Embedding Benchmark, EACL 2023 (arXiv:2210.07316). Includes reranker leaderboards. The leaderboard is measured on in-distribution benchmarks, which is exactly the case where the reranker looks good. It says less about what happens on your out-of-domain enterprise corpus.