# Cross-Encoder Reranking

Cross-encoder reranking is an optional second-stage ranking technique that improves the relevance of documents returned by vector or hybrid search. A cross-encoder evaluates each query-document pair together and assigns more accurate relevance scores. The documents are reordered based on these scores, producing a more accurate final ranking while maintaining efficient retrieval. Applying a cross-encoder to a large set of rows is computationally expensive, so it is used as a second stage to rerank only the documents returned by the initial retrieval step.

For more information about first-stage retrieval, refer to [Working with Vector Data](https://docs.singlestore.com/db/v9.1/developer-resources/functional-extensions/working-with-vector-data.md) and [Hybrid Search](https://docs.singlestore.com/db/v9.1/developer-resources/functional-extensions/hybrid-search-reranking-full-text-and-vector-search-results.md).

A typical pipeline is:

1. Retrieve candidate documents using vector or hybrid search.

2. Score each query-document pair with a cross-encoder.

3. Reorder the candidates based on their relevance scores.

This approach balances the efficiency of initial retrieval with the improved ranking accuracy of a cross-encoder.

## Cross-Encoder Overview

Vector search uses embeddings generated by a bi-encoder model. Queries and documents are encoded separately into dense vectors, which are compared using a similarity metric such as cosine similarity or [dot product](https://docs.singlestore.com/db/v9.1/reference/sql-reference/vector-functions/dot-product.md). Because document embeddings can be generated ahead of time and stored in a vector index, retrieval is efficient even for very large collections.

A cross-encoder, in contrast, processes the query and document together to evaluate their semantic relationship directly. By jointly encoding each query-document pair, it captures interactions that embedding-based retrieval cannot, often producing more accurate relevance scores.

The main trade-off is performance. As every query-document pair must be evaluated independently, latency increases with the number of candidates. For this reason, cross-encoders are used to rerank only a small set (20-100) of top-ranked results. They are therefore less suitable for latency-sensitive or high-throughput workloads that require scoring large numbers of documents in real time. In production deployments, GPU acceleration is commonly used to reduce inference latency.

## Cross-Encoder Reranking Use Cases

Cross-encoder reranking is best suited to workloads where ranking quality is more important than minimizing latency. Common use cases include:

* User-facing search applications where high-quality results are a priority.
* Retrieval-augmented generation (RAG) pipelines that require highly relevant context for downstream LLMs.
* Complex semantic queries where more accurate relevance scoring improves the ordering of results.

## Example Implementation

Cross-encoder reranking can be implemented in several ways, including external application code, external functions, or Wasm UDFs. This example uses the external application approach with two steps:

1. SingleStore performs vector search with embeddings generated by the [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5)  bi-encoder model.

2. Python scripts (`rerank_example.py` and `compare_search_methods.py`) apply the [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) cross-encoder model to evaluate each query-document pair and rerank the retrieved candidates.

To understand the core concept, consider this simplified reranking workflow:

```python
query = "puzzle game with falling blocks"

# Step 1: Encode query with bi-encoder (from get_candidates method)
bi_encoder = SentenceTransformer("BAAI/bge-large-en-v1.5")
query_embedding = bi_encoder.encode(
    [query],
    normalize_embeddings=True,
    show_progress_bar=False,
)[0]
emb_json = json.dumps(query_embedding.tolist())

# Step 2: Retrieve candidates from SingleStore using vector search
with conn.cursor() as cur:
    cur.execute(
        """
        SELECT
            chunk_id,
            text,
            DOT_PRODUCT(embedding_1024, %s :> VECTOR(1024)) AS score
        FROM documents
        ORDER BY score DESC
        LIMIT %s
        """,
        (emb_json, 50),
    )
    candidates = cur.fetchall()

# Step 3: Rerank with cross-encoder (from rerank method)
cross_encoder = CrossEncoder("BAAI/bge-reranker-v2-m3")
pairs = [[query, text] for _, text, _ in candidates]
scores = cross_encoder.predict(
    pairs,
    batch_size=32,
    show_progress_bar=False,
)

# Step 4: Sort by cross-encoder scores
reranked = [
    (candidates[i][0], candidates[i][1], float(scores[i]))
    for i in range(len(candidates))
]
reranked.sort(key=lambda x: x[2], reverse=True)

# reranked[0] is now the most relevant result

```

**Note**: Many cross-encoder models are available, and new models continue to improve on existing approaches. Consider evaluating different models to find the one that best meets your performance requirements.

## Repository Structure

The full implementation is available in the [cross-encoder-reranking-example](https://github.com/singlestore-labs/cross-encoder-reranking-example) repository.The repository includes:

* Database setup
* Document loading and embedding generation
* Vector search
* Cross-encoder reranking scripts
* Before/after comparison examples

Refer to the repository [README](https://github.com/singlestore-labs/cross-encoder-reranking-example/blob/main/README.md) for setup and execution instructions.

## Example Results

The repository example uses a test dataset of Wikipedia video game articles. It includes scripts to download, chunk, and load the data into SingleStore.

Query: `puzzle game with falling blocks`

Before reranking (vector search):

| **Rank** | **Chunk** | **Score** | **Content**                                                                                                                                                 |
| -------- | --------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1        | 362       | 0.665     | Tetris generally has a consistent puzzle video game design. Gameplay consists of a rectangular field in which pieces consisting of four connected blocks... |
| 2        | 132       | 0.634     | Minecraft is a three-dimensional sandbox video game that has no required goals to accomplish...                                                             |
| 3        | 375       | 0.615     | See also: Brain Wall and Blokken, game shows based on Tetris...                                                                                             |

After reranking (cross-encoder):

| **Rank** | **Chunk** | **Score** | **Content**                                                                                                                                                                                                                |
| -------- | --------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1        | 361       | 0.993     | Tetris is a puzzle video game created by Alexey Pajitnov, a Soviet software engineer, in the mid-1980s. In Tetris, falling pieces consisting of four connected blocks, known as tetrominoes, must be sorted into a pile... |
| 2        | 362       | 0.712     | Tetris generally has a consistent puzzle video game design. Gameplay consists of a rectangular field...                                                                                                                    |
| 3        | 376       | 0.591     | Notes. References. Bibliography. Books. Ackerman, Dan (2016). The Tetris Effect: The Game that Hypnotized the World...                                                                                                     |

## Performance

Sample measurements on an Intel CPU with a 603-chunk dataset (from the example repository):

* Retrieve top-50 candidates: \~0.2s
* Rerank 50 candidates (CPU): \~25s
* Model loading (first run only): \~7s

These measurements reflect the computational cost of CPU-based inference and are not representative of production deployments, where batching and GPU acceleration can substantially reduce latency.

## Limitations

Cross-encoder reranking improves the ordering of retrieved results but does not improve recall. If a relevant document is not included in the initial candidate set, the cross-encoder cannot recover it.To increase coverage:

* Retrieve a larger candidate set
* Use hybrid search to combine semantic and full-text search
* Tune embeddings and retrieval parameters

Cross-encoder outputs should be interpreted as relative ranking signals within a single query, not as absolute relevance scores.

## Related Topics

* [Chunk Documents for Vector Search](https://docs.singlestore.com/db/v9.1/developer-resources/functional-extensions/chunk-documents-for-vector-search.md)

***

Modified at: September 10, 2026

Source: [/db/v9.1/developer-resources/functional-extensions/cross-encoder-reranking/](https://docs.singlestore.com/db/v9.1/developer-resources/functional-extensions/cross-encoder-reranking/)

(An index of the documentation is available at /llms.txt)