Cross-Encoder Reranking

Cross-Encoder Overview

Vector search uses embeddings generated by a bi-encoder model. Queries and documents are encoded separately into dense vectors, which are compared using a similarity metric such as cosine similarity or dot product. Because document embeddings can be generated ahead of time and stored in a vector index, retrieval is efficient even for very large collections.

A cross-encoder, in contrast, processes the query and document together to evaluate their semantic relationship directly. By jointly encoding each query-document pair, it captures interactions that embedding-based retrieval cannot, often producing more accurate relevance scores.

The main trade-off is performance. As every query-document pair must be evaluated independently, latency increases with the number of candidates. For this reason, cross-encoders are used to rerank only a small set (20-100) of top-ranked results. They are therefore less suitable for latency-sensitive or high-throughput workloads that require scoring large numbers of documents in real time. In production deployments, GPU acceleration is commonly used to reduce inference latency.

Cross-Encoder Reranking Use Cases

Cross-encoder reranking is best suited to workloads where ranking quality is more important than minimizing latency. Common use cases include:

• User-facing search applications where high-quality results are a priority.

• Retrieval-augmented generation (RAG) pipelines that require highly relevant context for downstream LLMs.

• Complex semantic queries where more accurate relevance scoring improves the ordering of results.

Example Implementation

Cross-encoder reranking can be implemented in several ways, including external application code, external functions, or Wasm UDFs. This example uses the external application approach with two steps:

1. SingleStore performs vector search with embeddings generated by the BAAI/bge-large-en-v1.5  bi-encoder model.

2. Python scripts (rerank_example.py and compare_search_methods.py) apply the BAAI/bge-reranker-v2-m3 cross-encoder model to evaluate each query-document pair and rerank the retrieved candidates.

To understand the core concept, consider this simplified reranking workflow:

query = "puzzle game with falling blocks"

# Step 1: Encode query with bi-encoder (from get_candidates method)
bi_encoder = SentenceTransformer("BAAI/bge-large-en-v1.5")
query_embedding = bi_encoder.encode(
    [query],
    normalize_embeddings=True,
    show_progress_bar=False,
)[0]
emb_json = json.dumps(query_embedding.tolist())

# Step 2: Retrieve candidates from SingleStore using vector search
with conn.cursor() as cur:
    cur.execute(
        """
        SELECT
            chunk_id,
            text,
            DOT_PRODUCT(embedding_1024, %s :> VECTOR(1024)) AS score
        FROM documents
        ORDER BY score DESC
        LIMIT %s
        """,
        (emb_json, 50),
    )
    candidates = cur.fetchall()

# Step 3: Rerank with cross-encoder (from rerank method)
cross_encoder = CrossEncoder("BAAI/bge-reranker-v2-m3")
pairs = [[query, text] for _, text, _ in candidates]
scores = cross_encoder.predict(
    pairs,
    batch_size=32,
    show_progress_bar=False,
)

# Step 4: Sort by cross-encoder scores
reranked = [
    (candidates[i][0], candidates[i][1], float(scores[i]))
    for i in range(len(candidates))
]
reranked.sort(key=lambda x: x[2], reverse=True)

# reranked[0] is now the most relevant result

Note: Many cross-encoder models are available, and new models continue to improve on existing approaches. Consider evaluating different models to find the one that best meets your performance requirements.

Limitations

Cross-encoder reranking improves the ordering of retrieved results but does not improve recall. If a relevant document is not included in the initial candidate set, the cross-encoder cannot recover it.To increase coverage:

• Retrieve a larger candidate set

• Use hybrid search to combine semantic and full-text search

• Tune embeddings and retrieval parameters

Cross-encoder outputs should be interpreted as relative ranking signals within a single query, not as absolute relevance scores.

Related Topics

• Chunk Documents for Vector Search