Chunk Documents for Vector Search

What is Chunking and Why is it Important?

Chunking is the process of breaking documents down into smaller chunks of text that are more suitable for embedding models and downstream LLMs. Selecting the right chunking strategy is critical for achieving high-quality search results.

When chunking documents, ensure that the chunks:

Are small enough to fit within the input limits of the embedding model, e.g., 256 - 8192 tokens, depending on the model. A token is typically 3-4 characters or ¾ of a word in English.
Contain enough context to remain meaningful.
Preserve semantics to obtain high-quality search results.

Chunking strategy directly impacts the quality of search results:

Chunks that are too large return vague matches.
Chunks that are too small lose context.
Chunks that split mid-sentence or mid-entity return fragments that confuse both people and LLMs.

High-quality vector similarity search depends on creating semantically coherent chunks that fit within an embedding model's limits. Consider where to split content (sentence, paragraph, or entity boundaries), how much overlap to include between consecutive chunks (typically 10–20%), and whether to preserve linguistic boundaries to avoid fragmenting meaning.

Note: If the text is short enough to fit comfortably within the embedding model's input limit (for example, product descriptions, tweets, or short articles), chunking is unnecessary and may reduce retrieval quality by splitting related context.

Choose a Chunking Strategy

Strategy	How it Works	When to Use
Fixed-size	Splits at a fixed character, word, or token count regardless of content.	Useful as a simple baseline or when guaranteed size limits are required. Fast, no dependencies, predictable. Splits mid-sentence and mid-entity with no semantic awareness; chunks may start and end in meaningless places.
Sentence-based	Splits at sentence boundaries detected by NLP tokenizers or regexes.	Useful for Q&A retrieval where the user asks a specific question with a relatively small answer. Each chunk is a complete thought. Produces chunks that correspond to natural reading boundaries, such as complete sentences. Sentence length varies significantly and chunk sizes are unpredictable. Adjacent sentences about the same topic may end up in different chunks.
Paragraph / structure	Splits on document structure such as blank lines, headings, chapter markers.	Useful for well-structured documents where the organization reflects the topics. Preserves semantically coherent sections that naturally belong together. Works best for well-structured documents. Less effective on raw, unstructured text. Paragraph sizes can vary significantly and may exceed embedding model limits.
Recursive	Tries splitting on paragraphs first, falls back to sentences, then words, then characters.	Useful as a general-purpose default. Adapts to content, respects structure when present, degrades gracefully. A solid default chunking approach that often outperforms more complex methods. Ultimately relies on character-count limits rather than semantic understanding.
Entity-preserving	Splits between sentences, but only where named entities (people, places, organizations) don't carry over into the next sentence. Keeps entities and their surrounding context in the same chunk.	Useful for documents where names, places, and organizations are central to search. For example, legal documents, news articles, bios. Chunks are self-contained and meaningful. Requires an NLP pipeline such as spaCy or Natural Language Toolkit (NLTK), is slower, and chunk sizes are less predictable. Can produce oversized chunks if entities span many sentences.
Semantic	Embeds each sentence in a document, compares adjacent embeddings, and identifies topic boundaries where similarity drops significantly.	Useful for documents where topics don't follow visual structure. For example, meeting transcripts and long emails. Topic-aware, regardless of text formatting. Expensive, as it requires an extra embedding API call per sentence for the initial chunking. May underperform recursive chunking and is sensitive to similarity threshold tuning.
Structure-aware (JSON/HTML/Markdown)	Parses the format's native structure (tags, keys, headings) and splits along those boundaries.	Useful for structured or semi-structured data. For example, Web pages, API responses, documents with markup. Preserves metadata and format semantics. Requires a different parser per format, and is less effective for long, unstructured text within a field.

Chunk Data in SingleStore

To support effective semantic or vector-based search, users may need to chunk document files or text stored in a table column when the text exceeds the embedding model’s input limits. This section demonstrates a complete workflow: chunk documents, load chunks into SingleStore, generate embeddings, and perform vector similarity search.

For example:

Python

# Chunk documents
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
    length_function=len,
)

chunks = splitter.split_text(text)

SQL

-- Load chunks into SingleStore
CREATE TABLE IF NOT EXISTS chunks (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    chunk_id BIGINT,
    text TEXT,
    source VARCHAR(255),
    strategy VARCHAR(50),
    length INT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    FULLTEXT KEY idx_text (text)
);

INSERT INTO chunks (chunk_id, text, source, strategy, length)
VALUES (...);

-- Generate embeddings
ALTER TABLE chunks ADD COLUMN embedding_1024 VECTOR(1024);

UPDATE chunks
SET embedding_1024 = ...
WHERE chunk_id = ...;

-- Vector similarity search
SELECT
    chunk_id,
    text,
    DOT_PRODUCT(embedding_1024, ... :> VECTOR(1024)) AS similarity
FROM chunks
ORDER BY similarity DESC
LIMIT 5;

This simplified example illustrates key steps in the workflow. For a complete, runnable implementation, refer to the vector-text-chunking repository.

The following example provides an end-to-end walkthrough using the recursive chunking strategy with sample data and configuration files.

Example: Using the Recursive Strategy

The recursive strategy tries splitting on paragraphs first, then sentences, then words, then characters. The following example uses a test document, test configuration, code, and example output to illustrate how recursive chunking works.

Sample document (test_doc.txt):

SingleStore is a distributed SQL database designed for modern data-intensive applications.
It combines the capabilities of a traditional relational database with the performance characteristics needed for real-time analytics and operational workloads.
The architecture uses a dual-storage engine approach with both rowstore and columnstore formats.
Rowstore is optimized for transactional OLTP workloads with fast inserts, updates, and point queries.
Columnstore is designed for analytical OLAP queries that scan large amounts of data efficiently.
The query optimizer in SingleStore automatically analyzes each query and chooses the optimal execution plan.
It considers multiple factors including data distribution across partitions, available indexes, query selectivity, and join order.
The optimizer can push computation down to storage nodes to minimize data movement.
Advanced features include native support for vector search using approximate nearest neighbor algorithms.
This enables AI and machine learning applications to perform semantic search over embeddings.
The database also supports geospatial data types, JSON operations, and full-text search capabilities.
SingleStore scales horizontally by distributing data across multiple nodes in a cluster.
The architecture separates compute and storage, allowing independent scaling of each layer.
Data is automatically partitioned and replicated for high availability and fault tolerance.

Step 1: Configure connection

Create a configuration file (config.json) with SingleStore database connection details and embedding settings. The dimension parameter specifies the size of the vector embeddings.

JSON

{
  "singlestore": {
    "host": "your-host.singlestore.com",
    "port": 3306,
    "user": "admin",
    "password": "your_password",
    "database": "chunking_test",
    "table_name": "chunks"
  },
  "embeddings": {
    "dimension": 1024
  }
}

Step 2: Chunk the document

Chunk the document using langchain_chunker.py, a Python script that uses LangChain text splitters. Full details about the available chunkers are provided later in the Chunkers section.

Options used in this command:

--strategy recursive: Uses recursive splitting, trying paragraphs, then sentences, then words, then characters

--size 500: Sets a target chunk size of 500 characters

--overlap 100: Adds a 100-character overlap between consecutive chunks to preserve context

--output: Specifies the output JSON file path

--stats: Displays statistics about the generated chunks

Shell

python chunkers/langchain_chunker.py data/test_doc.txt --strategy recursive --size 500 --overlap 100                                                  --output data/test_chunks.json --stats

Output:

Shell

Using recursive strategy...
✅ Created 5 chunks

📊 Statistics:
  total_chunks: 5
  avg_chunk_size: 291
  min_chunk_size: 255
  max_chunk_size: 327
  total_chars: 1,456
  overlap_estimate: 100
💾 Saved to data/test_chunks.json

Step 3: Load into SingleStore

Load the generated chunks into SingleStore. The load_chunks_s2.py script reads the configuration from config.json and automatically creates the database and table if they don't exist. It takes a single argument: the path to the chunks JSON file generated in Step 2.

Shell

python singlestore/load_chunks_s2.py data/test_chunks.json

Shell

Setting up database...
✅ Database 'chunking_test' ready
✅ Table 'chunks' ready
Loading chunks from data/test_chunks.json...
Found 5 chunks to load
  Loaded 5/5 chunks...
✅ Successfully loaded 5 chunks
Chunk Statistics:
Strategy             Count      Avg Length   Min      Max
------------------------------------------------------------
unknown              5          291          255      327
✅ All done! Your chunks are loaded in SingleStore.
Database: chunking_test
Table: chunks

Step 4: Generate embeddings

Generate vector embeddings with create_embeddings.py, which uses the sentence-transformers library. It reads configuration from config.json to determine the embedding dimension and automatically selects the appropriate embedding model. The embeddings are stored in a new column named embedding_1024 in the chunks table.

Shell

python singlestore/create_embeddings.py

Output:

Shell

 SingleStore 1024-Dimensional Vector Embeddings Generator
  ============================================================
  Using CPU for embeddings
  Loading embedding model: BAAI/bge-large-en-v1.5 for 1024 dimensions
  Model loaded. Embedding dimension: 1024
  Adding vector column 'embedding_1024' (dimension: 1024)...
  Vector column added successfully
  Skipping vector index creation
  Generating 1024-dim embeddings for 5 chunks...
     Device: cpu
     Batch size: 32
     Model: BAAI/bge-large-en-v1.5
  Embeddings generated and stored successfully!
     Processed: 5 chunks in 0.0 minutes
     Average rate: 2.4 chunks/second
  ...
  [Output truncated - test queries omitted]

Step 5: Vector Search

Perform vector similarity search using a natural language query. The --topk option returns the top matching chunks ranked by similarity score, and the --verify flag displays the actual chunk text along with similarity scores.

Shell

python singlestore/vector_search.py --topk "how does the query optimizer work" --verify

Output:

Shell

 Loading embedding model: BAAI/bge-large-en-v1.5 for 1024 dimensions...
  Connected to SingleStore
  Verification mode: Running once with text output
  Vector search for: 'how does the query optimizer work'
  ============================================================
  Results:
  1. Chunk 2 (similarity: 0.679)
     The query optimizer in SingleStore automatically analyzes each query and chooses
     the optimal execution plan. It considers multiple factors including data
     distribution across partitions, available indexes, query selectivity, and join order...
  2. Chunk 1 (similarity: 0.563)
     The architecture uses a dual-storage engine approach with both rowstore and
     columnstore formats. Rowstore is optimized for transactional OLTP workloads...
  3. Chunk 0 (similarity: 0.556)
     SingleStore is a distributed SQL database designed for modern data-intensive
     applications. It combines the capabilities of a traditional relational database...

Use the Repository

The preceding example uses scripts from the following repository:

https://github.com/singlestore-labs/vector-text-chunking

This repository provides end-to-end code for vector similarity search in SingleStore, including tools to:

Chunk document files.
Create embeddings from the chunks.
Load the embeddings and chunks into SingleStore.
Run a similarity search over the data.

It includes five chunking implementations (text_chunker.py, langchain_chunker.py, nltk_chunker.py, spacy_chunker.py, json_chunker.py), embedding generation scripts, vector search tools, sample data (pride_and_prejudice.txt) and an evaluation script (evaluate_chunks.py) to measure chunk quality.

Chunkers

The chunkers directory in the repository contains five standalone Python scripts for chunking text in files. These scripts use different chunking approaches as described in the following table.

Chunker File Name	Strategies	Options	Best For
`text_chunker.py`	Fixed-size: character count, word count Sentence-based: regex sentence splitting Paragraph/structure: paragraph count, chapter count, semantic blocks (groups paragraphs up to a maximum size, falls back to sentence splitting for oversized paragraphs)	Size in characters or words. Overlap supported. Chapter splitting for books.	Quick start. Pure Python, no dependencies. No ML model or ML overhead.
`langchain_chunker.py`	Fixed-size Recursive Sentence-based Semantic	Token counting via tiktoken or sentence-transformers tokenizer. Configurable overlap. Sentence detection via NLTK or spaCy.	Recursive mode is a solid general-purpose default. LangChain wrapper that supports multiple strategies. Recursive mode (splits on paragraphs → sentences → words → characters). Widely used in production RAG systems. Supports token-aware splitting (exact token limits) for meeting exact model limits. Supports embedding-based semantic splitting for topic-shift detection (requires OpenAI API key).
`nltk_chunker.py`	Fixed-size (tokens) Sentence-based (sentences) Entity-preserving (noun_phrases) Entity-preserving (entities / NER) Paragraph/structure (paragraphs)	Token counting via NLTK word tokenizer. Punkt sentence detection. Entity detection via NER or noun-phrase grammar. Overlap supported.	Linguistically-aware chunking using NLTK. Splits at sentence boundaries while preserving linguistic units. Best for not splitting mid-entity or mid-phrase and documents with important named entities.
`spacy_chunker.py`	Fixed-size (tokens) Sentence-Based Entity-preserving (groups sentences, keeps pronouns with their antecedents, respects entity boundaries) Entity-preserving (NER not split across chunks).	Full NLP pipeline. Semantic grouping keeps pronouns with antecedents. Token counting via spaCy tokenizer. Overlap supported.	Highest-quality NLP-aware splitting for entity-rich documents, but with a higher runtime cost. Similar to NLTK but adds pronoun / antecedent grouping. Documents with important names/entities that must not be split. NER-aware boundaries. Note: Requires Python 3.10 or later
`json_chunker.py`	Structure-aware	Reads NDJSON (article per line). Splits text fields at sentence boundaries. Preserves title/URL metadata per chunk.	Useful for structured data with text fields to be chunked. Preserves metadata per chunk. Purpose-built for Wikipedia NDJSON ingestion. Reads article-per-line format, chunks abstracts at sentence boundaries, preserves title/URL metadata per chunk.

Evaluation

The repository also contains a chunking-strategy comparison workflow in which chunks are generated with different strategies and then evaluated with an evaluation script. The evaluate_chunks.py evaluates chunking quality with various metrics. The metrics in the script include:

Size metrics, including count, mean size, median size, standard deviation, min size, max size, and size variance coefficient
Boundary quality, including ratios for complete sentences, starts-with-capital, ends-with-punctuation, broken quotes, and broken parentheses
Semantic coherence, including pronoun/entity ratios and average sentences per chunk
Overlap analysis between consecutive chunks
Information density

To learn more, try the sample workflow in the repository: generate chunks with a few different strategies, load them into SingleStore, run similarity searches, and compare the results to see how each approach affects retrieval quality.

Recommendations

Chunking strategy depends on the document type and search requirements. For most use cases, SingleStore recommends the recursive strategy as illustrated in langchain_chunker.py as a general-purpose default across various document types. Use 500-1000 character chunks with 100-200 character overlap (20% of chunk size).

Choose a Strategy

Consider these common document types when choosing a strategy for your workload:

General purpose documents: Recursive adapts to structure
Legal documents, news articles: Entity-preserving strategies (nltk or spacy) keep named entities intact
Emails, chat logs, transcripts: Semantic chunking detects topic shifts
JSON/structured data: json_chunker preserves metadata
Q&A systems: Sentence-based chunking provides complete thoughts

Optimize Results

Reduce chunk size (500-800 characters) if results seem too vague
Increase chunk size (1200 characters) if results appear fragmented
Switch to entity-preserving strategy if named entities are being split
Check size variance coefficient if chunk quality is inconsistent
Test with representative queries, as boundary quality metrics don't always correlate with search performance

Tie Chunks to Enclosing Documents

In many applications, retrieving chunks is sufficient and the chunks can be consumed directly by people, Retrieval-Augmented Generation (RAG) applications, or AI agents. In other cases, applications require returning the enclosing document that contains the chunk, rather than the chunk itself. A useful approach is to score documents by the maximum similarity score of any chunk in the document.

For example, given a table like this:

SQL

CREATE TABLE doc_chunks (
chunk_id BIGINT,
doc_id BIGINT /* id of enclosing document */,
chunk TEXT,
chunk_embedding VECTOR(1024)
);

The following query finds the top K documents with the maximum chunk score:

SQL

SET @qv = ('[...]'):>VECTOR(1024);
  
SELECT doc_id, MAX(chunk_embedding <*> @qv) AS max_score
FROM doc_chunks
GROUP BY doc_id
ORDER BY max_score DESC
LIMIT 5;

To use ANN indexing on the chunk_embedding column to speed up this query for large sets of chunks, consider using a two-stage approach. Retrieve a relatively large number of chunks (for example, 200), then select the top documents by maximum score (for example, 5).

The optimized query:

SQL

SET @qv = ('[...]'):>VECTOR(1024);
 
WITH top_chunks AS (
     SELECT doc_id, chunk_embedding <*> @qv AS score
     FROM doc_chunks
     ORDER BY score DESC
     LIMIT 200
  )

SELECT doc_id, MAX(score) AS max_score
FROM top_chunks
GROUP BY doc_id
ORDER BY max_score DESC
LIMIT 5;

This approach is more efficient for large datasets because the ANN index helps quickly find the top 200 candidate chunks, then the aggregation operates on a much smaller subset rather than scanning all chunks.

On this page

What is Chunking and Why is it Important?

Choose a Chunking Strategy

Chunk Data in SingleStore

Example: Using the Recursive Strategy

Step 1: Configure connection

Step 2: Chunk the document

Step 3: Load into SingleStore

Step 4: Generate embeddings

Step 5: Vector Search

Use the Repository

Chunkers

Evaluation

Recommendations

Choose a Strategy

Optimize Results

Tie Chunks to Enclosing Documents

Was this article helpful?

On this page

Was this article helpful?

Chunk Documents for Vector Search

On this page

What is Chunking and Why is it Important?

Choose a Chunking Strategy

Chunk Data in SingleStore

Example: Using the Recursive Strategy

Step 1: Configure connection

Step 2: Chunk the document

Step 3: Load into SingleStore

Step 4: Generate embeddings

Step 5: Vector Search

Use the Repository

Chunkers

Evaluation

Recommendations

Choose a Strategy

Optimize Results

Tie Chunks to Enclosing Documents

Related Topics

Was this article helpful?

On this page

Was this article helpful?