Most RAG quality issues come from bad document chunking rather than the LLM model itself. Here is how to optimize retrieval:
1. Use Semantic Structure Splitting: Split Markdown and PDFs on header boundaries (`
## `) rather than arbitrary character counts to keep tables and code blocks intact.
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["
## ", "
### ", "
", "
", " "]
)
```
2. Combine Vector + Keyword Search: Pair vector embeddings with BM25 keyword search, then pass top results to a Cohere Reranker model. This catches both semantic context and exact product/code matches.
3. Parent-Child Indexing: Store 128-token chunks for vector retrieval, but return the surrounding 1024-token parent section to the LLM.