Fixed-size chunks, with overlap
Retrieval-augmented generation needs text cut into pieces small enough to embed, with enough overlap that an idea split between two chunks still appears whole in one of them. Ours does the simplest version of that well: a sliding window of a fixed size, moved forward by the size minus the overlap.
You get back one Markdown document with every chunk under its own numbered heading, ready to copy into the embedding step of your pipeline, whether it runs on OpenAI, Anthropic, Cohere, or any other embedding model.
What you control
- Chunk size in tokens: 500 by default, from 100 to 8,000
- Overlap in tokens: 50 by default, from 0 to 500
- Token counts are estimated at about 1.3 tokens per word, so a 500-token chunk holds about 385 words
- Chunks are cut on word boundaries, never in the middle of a word
- Each chunk is headed ## Chunk N with its approximate token count
- Chunks are separated by --- lines
Original line breaks and spacing are retained within each chunk. Boundaries use word counts, so they can divide sentences, headings or code blocks. For structure-aware boundaries, split the document on its headings first, then chunk each part.