Retrieval-augmented generation, or RAG, is the most reliable way to make a language model answer questions about information it was never trained on. Instead of retraining the model, you fetch the most relevant passages from your own documents and hand them to the model alongside the question.
Step 1: Chunk your documents
Split documents into passages of roughly 300 to 800 tokens, keeping headings attached to their paragraphs. Chunks that are too small lose context; chunks that are too large dilute relevance. Store the source URL and title with every chunk so you can cite it later.
Step 2: Embed and index
Convert each chunk into an embedding — a list of numbers that captures its meaning — and store it in a vector database or a vector-enabled Postgres table. At query time, embed the user's question the same way and fetch the closest five to ten chunks.
Step 3: Write a grounded prompt
Tell the model to answer only from the supplied passages, to cite the source for each claim and to say "I don't know" when the answer is not present. This single instruction removes most hallucinations in practice.
Step 4: Evaluate before you ship
Write twenty real questions with known answers and run them after every change. Track whether the right passage was retrieved and whether the final answer was correct. Most RAG failures are retrieval failures, not model failures.
Key takeaways
- Step 1: Chunk your documents
- Step 2: Embed and index
- Step 3: Write a grounded prompt
- Step 4: Evaluate before you ship
