RAG Pipelines

Core Concepts

• Chunking: Chunk by semantics or document structure, not just fixed lengths. Avoid naive character-based chunking. Use semantic chunking (splitting at sentence/paragraph boundaries) or structure-aware chunking (e.g., Markdown header splitting).
• Embeddings: Select appropriate models (e.g. text-embedding-3-small).
• Retrieval: Use hybrid search (vector + keyword) and re-ranking for optimal precision. Do not rely purely on dense vector embeddings. Combine vector search (for semantic meaning) with keyword search (BM25 for exact matches/IDs/names) using Reciprocal Rank Fusion (RRF). Always add a cross-encoder reranking step after initial retrieval. Retrieve a larger pool of documents (e.g., top-20) and rerank them to select the top-5 most relevant chunks to inject into the prompt.
• Context Injection: Carefully format retrieved chunks into the prompt for maximum LLM attention. Instruct the LLM to cite specific retrieved chunks in its response. Reject answers that rely on pre-trained knowledge if the task requires strict adherence to retrieved context.