Dataloopr

Retrieval-Augmented Generation (RAG) has become a core pattern for building production-grade LLM applications that can answer questions using external, up-to-date, or private data while minimizing hallucinations.

1. RAG Fundamentals

What Is RAG and Why Is It Used?

RAG stands for Retrieval-Augmented Generation. It is an architecture that combines external information retrieval with generative LLMs (like GPT-4 or Llama 2) to produce answers grounded in a knowledge base, not just the LLM’s pretrained parameters.

• Retrieval: The system fetches relevant documents or passages from an external source (vector database, search index) based on a user query.
• Augmented Generation: The LLM receives both the original query and retrieved context, generating answers that cite or use this external information.

Problems RAG Solves

• Outdated Model Knowledge: LLMs are only as current as their training data. RAG enables access to new or updated information without retraining.
• Private/Proprietary Data Access: RAG connects LLMs to internal documents, wikis, databases, or customer records inaccessible to pretraining.
• Reducing Hallucinations: By grounding generation in retrieved facts, RAG reduces the risk of fabricated or unsupported answers.

RAG vs. Fine-Tuning vs. Prompt Engineering

RAG is not the only way to adapt LLMs to new tasks or data. Compare with two other approaches:

Summary: Use prompt engineering for quick wins, fine-tuning for deep customization, and RAG for robust, updatable, data-grounded systems.

2. RAG Architecture and Workflow

A RAG system is built from two distinct, but interconnected pipelines: the indexing pipeline (preparing the knowledge base) and the query pipeline (handling user questions).

RAG Architecture Diagram

Indexing Pipeline

1. Document Ingestion & Parsing: Load documents (PDFs, internal wikis, emails, etc.) and parse text.
2. Text Cleaning & Preprocessing: Remove headers, footers, boilerplate, and normalize Unicode or whitespace.
3. Document Chunking: Split documents into smaller, manageable chunks (e.g., 500-word passages).
4. Embedding Generation: Convert each chunk into a high-dimensional vector using an embedding model (e.g., OpenAI’s text-embedding-ada-002).
5. Storing in Index: Save vectors, text, and metadata (title, section, department, date, etc.) in a vector database for fast retrieval.

Query Pipeline

1. User Query Processing: Accept natural language question; preprocess (clean, normalize).
2. Query Embedding: Convert query to a vector (same embedding model as used for chunks).
3. Similarity Search & Top-k Retrieval: Find top-k most similar document vectors in the vector database (using cosine similarity or other metric).
4. Optional Filtering & Reranking: Apply metadata filters (e.g., department = HR), hybrid search, or rerank candidates for better relevance.
5. Context Construction: Collate the retrieved chunks into a context window, optionally with source attributions.
6. LLM Answer Generation: Send context and query to the LLM, which generates a grounded answer, often citing sources.

Practical Example: Annual Leave Policy Query

Suppose an employee asks: “How many paid vacation days do I get per year?” Here’s the end-to-end flow:

• Indexing: HR policies are ingested as PDFs, parsed, cleaned, and chunked into 300-word passages. Each chunk is embedded and stored in a vector database with metadata (department=HR, doc_type=policy).
• User Query: The employee’s question is embedded. The system retrieves top-3 similar chunks, filtering for department=HR and doc_type=policy.
• LLM Generation: The LLM receives the query and retrieved context. It generates an answer like: “You are entitled to 20 paid vacation days per year. (Source: 2023 Employee Handbook, Section 4.1).”

Stage Purposes and Failure Modes

• Parsing: Bad parsers can miss crucial text (e.g., tables in PDFs).
• Chunking: Poorly chosen chunk sizes may split key information or mix unrelated topics.
• Embedding: Outdated/poor embedding models may fail to capture meaning, especially with jargon.
• Retrieval: Low-quality or missing metadata can lead to irrelevant results.
• Context Construction: Too much or too little context reduces answer quality or causes truncation.
• LLM Generation: If context is irrelevant, LLM may hallucinate or refuse to answer.

3. Document Chunking

Why Chunk Documents?

LLMs have limited context windows, so you can’t feed entire documents. Chunking splits documents into retrievable, context-sized passages to maximize retrieval accuracy and answer quality.

Chunking Strategies

• Fixed-Size Chunking: Split by character or token count (e.g., every 512 tokens). Simple, but may break semantic units.
• Recursive (Sliding Window): Overlapping chunks to preserve context at boundaries.
• Semantic Chunking: Split by sentences, paragraphs, or sections to keep meaning intact.
• Structure-Aware: Use document structure (headings, table rows, FAQ pairs) to guide splits.

Chunk Size and Overlap

• Small Chunks: Higher precision, less context per chunk, may miss broader connections.
• Large Chunks: Higher recall, risk of mixing unrelated info, fewer retrievable units.
• Overlap: Adding overlap (e.g., 50 tokens) helps capture information spanning chunk boundaries.

Trade-offs

• Precision vs. Recall: Small, focused chunks improve precision; larger, overlapping chunks improve recall.
• Cost: More chunks mean higher storage and retrieval costs. Larger chunks risk exceeding LLM context windows.

Choosing Strategies by Document Type

• PDFs: Structure-aware or semantic (paragraphs, headings). Watch for tables and footnotes.
• Technical Docs: Split by section/header using Markdown or HTML tags.
• FAQs: Each Q&A pair is a chunk.
• Tables: One row or logical group per chunk; preserve column headers.

Python Example Using LangChain

from langchain.text_splitter import RecursiveCharacterTextSplitter

text = """Section 1: Policy Overview
Employees are entitled to annual leave...
Section 2: Eligibility
Full-time staff receive 20 days..."""

splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,   # max chars per chunk
    chunk_overlap=50, # overlap to preserve context
    separators=["\n\n", "\n", ".", " "]
)
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:\n{chunk}\n")

Parameters: chunk_size controls max size, chunk_overlap ensures info at boundaries is not lost, and separators guides splitting (by paragraphs, then sentences, etc.).

4. Embeddings and Similarity Search

What Are Embeddings?

Embeddings are dense vector representations of text that capture semantic meaning. They convert document chunks and queries into points in high-dimensional space, enabling similarity comparison.

• Example: “annual leave policy” and “paid vacation rules” are mapped to similar vectors, even though the words differ.

How Embeddings Are Used in RAG

• Document Chunks: Each chunk is embedded and stored in the vector database.
• User Query: The query is embedded at runtime. The system retrieves chunks with the highest similarity (closest vectors).

Similarity Metrics

• Cosine Similarity: Measures angle between vectors; insensitive to magnitude.
  Mathjax: \( \text{cosine}(A, B) = \frac{A \cdot B}{\|A\| \|B\|} \)
• Dot Product: Measures both direction and magnitude.
  Mathjax: \( A \cdot B = \sum_{i=1}^n A_i B_i \)
• Euclidean Distance: Straight-line distance between vectors.
  Mathjax: \( \|A - B\|_2 = \sqrt{\sum_{i=1}^n (A_i - B_i)^2} \)

Normalization

Cosine similarity requires normalized (unit-length) vectors for meaningful angular comparison. Dot product and Euclidean distance are sensitive to scale; embeddings are often normalized before indexing.

Numerical Example

import numpy as np

a = np.array([1, 2, 3])
b = np.array([1, 2, 2])

cos_sim = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
euclid = np.linalg.norm(a - b)
print(f"Cosine similarity: {cos_sim:.4f}")
print(f"Euclidean distance: {euclid:.4f}")

Semantic Similarity

Embeddings capture meaning, not just keywords. “CEO annual leave” and “chief executive vacation days” may be close in embedding space despite no shared words.

Limitations

• Embeddings often struggle with exact identifiers (e.g., invoice numbers, part codes), numbers, rare terms, or ambiguous queries (e.g., “Apple” - fruit or company).
• Keyword-based retrieval may be better for such cases (see hybrid search).

5. Retrieval Strategies

Dense Retrieval

Dense retrieval uses embedding-based similarity (cosine or dot product) to find semantically similar chunks. It excels at paraphrase, concept, and intent matching.

Sparse Retrieval

Sparse retrieval (e.g., BM25, TF-IDF) matches keywords using inverted indexes, focusing on exact term overlaps. It is strong for code, numbers, names, and rare terms.

Hybrid Retrieval

Hybrid retrieval combines dense and sparse methods. Results from both are merged, often using reciprocal rank fusion (RRF), which boosts candidates that rank highly in either method.

Metadata Filtering

At retrieval time, filter results by attributes such as department, date, or type (e.g., only HR policies after 2022).

Reranking

After retrieving candidates, use a cross-encoder or other relevance model to rerank for improved accuracy. Reranking is computationally expensive but boosts final relevance.

Bi-Encoder vs. Cross-Encoder

• Bi-Encoder: Embeds query and document independently. Fast, suitable for large-scale retrieval.
• Cross-Encoder: Processes query and candidate together (jointly), scoring each pair. More accurate for reranking, but slower and not scalable for full-corpus retrieval.

Reciprocal Rank Fusion (RRF)

RRF is a simple but effective method for combining ranked lists (from dense and sparse retrieval). For each candidate, its final score is:

\( \text{RRF Score} = \sum_{i=1}^{n} \frac{1}{k + r_i} \)

where \( r_i \) is the rank of the candidate in the \( i \)-th list, and \( k \) is a constant (typically 60) to avoid division by zero and dampen the influence of low-ranked results.

Retrieval Examples

• Exact Product Code: Query: “What is the warranty for item ABC12345?”
  • Sparse retrieval (BM25) will outperform dense retrieval, as the code is a rare exact match.
• Paraphrased Question: Query: “How many days off do employees get annually?”
  • Dense retrieval will outperform sparse, as it captures semantic similarity (“annual leave” ≈ “days off”).

Retrieval vs. Generation

• Retrieval surfaces relevant context; the LLM’s job is to synthesize and generate the correct answer from that context.
• Poor retrieval means even the best LLM will struggle to answer accurately.

6. Vector Databases and Indexing

What Is a Vector Database in RAG?

A vector database stores embeddings (vectors), original text chunks, and metadata for efficient similarity search. It is optimized for high-dimensional search and scaling to millions of documents.

• Stores: Vector (embedding), associated text, and metadata (e.g., department, creation date).
• Retrieves: Top-k most similar vectors to a query vector, often with filters or hybrid search.

Approximate Nearest Neighbor (ANN) Search

Exact similarity search is slow at scale. ANN algorithms trade some accuracy for much faster search:

• HNSW (Hierarchical Navigable Small World): Builds a multi-layer graph for efficient nearest neighbor search.
• IVF (Inverted File Index): Partitions vectors into clusters (centroids) and searches only the most promising clusters.

Both are widely used in open-source vector stores for fast retrieval in RAG.

Vector Database vs. RAG Orchestration Framework

• Vector Database: Handles storage, indexing, and retrieval of embeddings and metadata. Examples: FAISS, Chroma, Qdrant, Pinecone, Weaviate, Milvus.
• RAG Orchestration Framework: Manages the whole RAG pipeline—chunking, embedding, retrieval, context construction, LLM calls, etc. Examples: LangChain, LlamaIndex.

Comparing Popular Vector Databases

LangChain and LlamaIndex

• LangChain: Modular framework for building LLM pipelines; integrates with multiple vector stores, chunkers, retrievers, LLMs, and tools.
• LlamaIndex: Focused on data connectors, chunking, retrieval, and RAG evaluation; works with many vector stores.

Summary: Use a vector database for efficient retrieval; use orchestration frameworks to manage RAG workflow.

7. RAG Interview Questions and Answers

1. How does RAG differ from fine-tuning?

Answer: RAG augments LLMs with external, up-to-date knowledge at inference time by retrieving relevant documents and passing them as context to the LLM. Fine-tuning updates the LLM’s internal weights using new training data, which is expensive and static. RAG is better for dynamic, private, or frequently changing data; fine-tuning is best for custom tasks or style transfer.

• Follow-up: What are the risks of using RAG for tasks requiring strict formatting or style?

2. Explain the indexing and query pipelines in RAG.

Answer: The indexing pipeline processes raw documents: ingestion, cleaning, chunking, embedding, and indexing. The query pipeline takes a user query, embeds it, retrieves relevant chunks from the index, constructs context, and prompts the LLM to generate an answer.

• Follow-up: What could cause poor retrieval quality in the query pipeline?

3. Why is chunking necessary, and how would you select chunk size?

Answer: LLMs have limited context windows; chunking splits documents into manageable units for retrieval. Small chunks improve retrieval precision; large chunks increase recall but risk mixing unrelated info. Choose chunk size based on document structure, context window, and retrieval needs—experiment with overlap and semantic splits for best results.

• Follow-up: How would you chunk a table-heavy PDF differently from a Markdown FAQ?

4. What are embeddings, and how are they used in RAG?

Answer: Embeddings are dense vectors representing the semantic meaning of text. In RAG, both document chunks and user queries are embedded; similarity search retrieves chunks most relevant to the query for context grounding.

5. What is the difference between cosine similarity and dot product?

Answer: Cosine similarity measures the angle between vectors and is insensitive to their magnitude; dot product also considers magnitude. Cosine similarity is preferred for normalized embeddings, providing a bounded [-1, 1] score.

• Follow-up: When might Euclidean distance be a poor choice for semantic search?

6. Dense retrieval vs. sparse retrieval: when would you use each?

Answer: Dense retrieval (embeddings) excels at semantic similarity and paraphrase matching. Sparse retrieval (BM25) works best for exact matches, rare terms, codes, and numbers. Use dense for conceptual questions, sparse for precise entity lookups.

7. What is hybrid search, and how does RRF work?

Answer: Hybrid search combines dense and sparse retrieval to leverage both semantic and keyword matching. Reciprocal Rank Fusion (RRF) merges both result lists by rewarding candidates that rank highly in either, improving recall and relevance.

8. What is the difference between a bi-encoder and a cross-encoder?

Answer:

• Bi-Encoder: Encodes query and document separately, enabling fast large-scale retrieval but less nuanced scoring.
• Cross-Encoder: Jointly encodes query and document, allowing richer interaction and better reranking, but is too slow for full-corpus retrieval.

• Follow-up: Why is cross-encoder usually used for reranking, not initial retrieval?

9. What is reranking, and why does it improve retrieval?

Answer: Reranking reorders top candidates from initial retrieval using a more expensive, accurate relevance model (often a cross-encoder). It often corrects ranking errors from fast, approximate retrieval, leading to more relevant context for the LLM.

10. What is a vector database, and how does approximate nearest-neighbor search work?

Answer: A vector database stores and efficiently retrieves high-dimensional embeddings for RAG. Approximate nearest-neighbor (ANN) search (using HNSW, IVF, etc.) finds vectors similar to the query much faster than brute-force search, enabling real-time retrieval at scale.

• Follow-up: When would exact search be preferable to ANN?

11. A RAG system retrieves irrelevant documents. How would you troubleshoot it?

Answer:

• Check chunking: Are chunks too big, too small, or poorly aligned with meaning?
• Check embedding model: Is it outdated or unsuitable for your domain/jargon?
• Check similarity metric: Are vectors properly normalized? Is the metric appropriate?
• Check metadata and filtering: Are filters applied correctly?
• Inspect retrieved chunks and user queries for ambiguity or out-of-domain phrasing.

• Follow-up: How would you evaluate retrieval quality objectively?

Mastering these RAG fundamentals will help you design robust, production-ready AI systems and excel in LLM and GenAI interviews. In future parts, we’ll cover advanced topics such as evaluation, optimization, and prompt engineering for RAG.