What Is a Reranker? The Second Opinion in AI Search

A search returns results. But the order of those results decides whether an AI gives the right answer or a convincing wrong one. This is exactly where a reranker works: it takes the candidates a first search has found and sorts them again, this time with a full understanding of what the question actually means.

This article explains what a reranker is, what it really delivers in practice, and how Kasimir uses one, on its own infrastructure in Germany.

Two stages instead of one: retriever and reranker

Modern AI systems that access company knowledge follow the principle of retrieval-augmented generation, RAG for short. For every question, matching text passages are retrieved first, then the language model composes an answer from those passages. The quality of the answer depends directly on what this first step finds.

The first search, the retriever, has to be fast. It compares the question with thousands or millions of passages and may only take milliseconds. That is only possible with a trick: every passage is translated in advance into a numeric vector, a kind of coordinate of its meaning. The question gets the same treatment, and the search looks for the nearest coordinates. This is fast, but coarse. Question and passage are considered separately; they never see each other.

The reranker does the opposite. It receives only a handful of candidates, say the best forty, and reads each one together with the question. Question and text are present at the same time, so the model can relate them: Does this passage fit this question? Does it answer it, or does it merely happen to mention the same words? The result is a relevance score per candidate, and the list is re-sorted accordingly.

In technical terms the two approaches are called bi-encoder and cross-encoder. The bi-encoder encodes separately and scales. The cross-encoder encodes jointly and is precise. Because precision is expensive, it is applied only to the candidates that made the shortlist.

Why this matters so much in practice

A made-up but typical example. Someone asks: “Up to what amount can a team lead approve an invoice on their own?” The first search finds forty passages from an internal policy handbook. Many of them contain the words invoice, amount or approval. The one passage that actually answers the question lands in twelfth place, behind eleven passages of general wording.

Without a reranker, the language model receives those twelve passages. It has to find the right one itself while eleven others distract it. Sometimes that works. Sometimes the result is an answer that sounds plausible and comes from the wrong paragraph.

With a reranker, the same passage sits in first place with a relevance score close to one. The general passages score close to zero. And because the separation is so sharp, the system can simply drop it. The language model then only sees what truly belongs to the question.

Fewer but more relevant passages are not only faster. They are also safer. Research shows that language models systematically overlook information in the middle of a long prompt. Every passage that does not belong to the question displaces one that does.

What a reranker cannot do

A reranker only sorts what the first search handed it. If the right passage is not among the forty candidates, it cannot bring it forward. The first search therefore has to be broad enough; Kasimir combines a semantic search with a keyword search so that German compound words and standard numbers are found reliably.

And a reranker does not replace a language model. It decides what gets read, not what gets answered.

How Kasimir uses the reranker

Since September 2026, Kasimir re-scores all candidates from its document search with its own reranker. The model behind it is Qwen3-Reranker, an open model running on our own GPU infrastructure in Germany. No text passage leaves our systems for this step.

We measured two options beforehand: our own reranker and a commercial service via a European cloud. With forty passages, our own was faster, had no volume limit, and separated relevant from irrelevant far more sharply. The decision was easy.

We then measured what this changes about the answers. The basis was sixty real questions from day-to-day use, each one run with and without re-scoring, and the results compared blind.

The outcome is twofold. The ordering gets markedly better: where the top hit differs, the re-scored one is the more apt in 28 of 34 cases. The finished answers, however, were equally good in both variants. The language model apparently finds the right passage even when eleven others sit next to it.

So the gain lies elsewhere than you might first assume: around 60 percent less text goes to the model at the same answer quality. That saves cost and time and leaves more room for conversation history and attachments. We are writing this down openly because nobody else runs the counter-check.

In practice, instead of always receiving twelve passages, the language model now gets only those above a relevance threshold, usually between four and eight. The re-scoring costs around 150 milliseconds per question. If the reranker is unavailable, the previous order remains, so search works in every case.

Conclusion

A reranker is the second opinion in a search. The first search is fast and finds candidates; the reranker is thorough and selects. What it reliably delivers is the better order: the supporting passage sits at the top instead of somewhere down the list. Whether the finished answer also improves depends on the language model — and that can only be measured, not claimed. The second gain is certain: far less text in the context at the same quality. And all of it can run entirely on your own infrastructure, without documents leaving the company.

What this changes in practice for people using Kasimir is described in the changelog.