RAG Evaluation Metrics, Frameworks, and Interview Questions

blog-cover-image

RAG Evaluation Metrics, Frameworks, and Interview Questions

Leon

Oct 24, 2026

Retrieval-Augmented Generation (RAG) systems are revolutionizing how large language models (LLMs) access up-to-date, domain-specific, or proprietary knowledge. However, evaluating RAG systems is far more intricate than assessing standard LLMs or retrieval systems.

1. Why RAG Evaluation Is Different

Unlike traditional ML models, RAG systems combine information retrieval and text generation. Evaluating only the final answer is insufficient because errors may arise from multiple components. Effective RAG evaluation must disentangle retrieval quality, generation quality, and end-to-end performance, diagnosing each layer separately.

Retrieval quality: Did the system fetch relevant documents or passages for the question?
Generation quality: Did the LLM produce an accurate, context-grounded, and helpful answer—free from hallucinations?
End-to-end quality: Did the whole system reliably answer the user's query?

A system may retrieve the right sources (high retrieval metrics) but generate a poor answer due to LLM hallucination or misunderstanding. Conversely, it might retrieve irrelevant sources but still answer correctly (by chance or because the LLM hallucinates the right answer). Thus, multi-layer evaluation with labeled test datasets, reference answers, expected source documents, and human or LLM-based review is essential.

2. Retrieval Evaluation Metrics

Retrieval metrics quantify how well the system finds supporting documents or passages. Below, we explain the most common metrics, each with intuition, formula, numerical example, interpretation, and limitations.

A. Precision@k

Intuition: Of the top k retrieved documents, what proportion are actually relevant?

\[ P@k = \frac{\text{Relevant documents in top } k}{k} \]

Example: For a query where the known relevant document IDs are {D2, D4}, and the system retrieves [D2, D5, D7] (k=3):
Relevant in top 3: D2
Precision@3 = 1/3 ≈ 0.33

Interpretation: High precision means few irrelevant documents, reducing the risk of distracting the LLM.
Limitations: Ignores relevant documents outside the top k. If fewer than k relevant documents exist, precision@k can never reach 1.

B. Recall@k

Intuition: Of all relevant documents, what proportion were retrieved in the top k?

\[ R@k = \frac{\text{Relevant documents in top } k}{\text{Total relevant documents}} \]

Example: With relevant {D2, D4} and system retrieves [D2, D5, D7]:
Relevant in top 3: D2
Recall@3 = 1/2 = 0.5

Interpretation: High recall means the system is less likely to miss necessary sources.
Limitations: If there are many relevant documents, recall is harder to maximize. High recall with low precision means many irrelevant documents are included.

C. Hit Rate@k

Intuition: Did at least one relevant document appear in the top-k results?

\[ \text{Hit Rate@k} = \begin{cases} 1, & \text{if at least one relevant document in top } k \\ 0, & \text{otherwise} \end{cases} \]

Example: Relevant {D2, D4}; system retrieves [D2, D5, D7]:
At least one (D2) is relevant ⇒ Hit Rate@3 = 1

Distinction: Hit Rate@k is binary per query. Recall@k is proportional and sensitive to the number of relevant documents.
Limitations: Does not consider how many relevant documents were retrieved, only whether at least one was found.

D. Mean Reciprocal Rank (MRR)

Intuition: How highly did the system rank the first relevant document?

\[ MRR = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{rank_i} \]

Where \(rank_i\) is the position of the first relevant document for query i; if no relevant document is retrieved, reciprocal rank is 0.

Example: For three queries:

Q1: Relevant {D2}, retrieved [D2, D5, D7] ⇒ First relevant at position 1 ⇒ 1/1 = 1.0
Q2: Relevant {D3}, retrieved [D5, D4, D3] ⇒ First relevant at position 3 ⇒ 1/3 ≈ 0.33
Q3: Relevant {D8}, retrieved [D2, D5, D7] ⇒ No relevant ⇒ 0

MRR = (1.0 + 0.33 + 0) / 3 ≈ 0.44

Interpretation: Higher MRR means relevant documents usually appear near the top.
Limitations: Only considers the first relevant document; ignores additional relevant documents.

E. Mean Average Precision (MAP)

Intuition: Measures how well a system ranks all relevant documents, rewarding those that rank them higher.

\[ AP = \frac{1}{\text{Total relevant}} \sum_{k=1}^{n} P@k \cdot rel_k \] \[ MAP = \frac{1}{N} \sum_{i=1}^N AP_i \]

Where \(rel_k = 1\) if the document at position k is relevant, 0 otherwise.

Example: For Q1, relevant {D2, D4}, retrieved [D2, D4, D7]:

At k=1: D2 is relevant ⇒ P@1 = 1/1 = 1
At k=2: D4 is relevant ⇒ P@2 = 2/2 = 1
At k=3: D7 not relevant

AP = (1 + 1) / 2 = 1.0

Interpretation: MAP considers both ranking and completeness across all relevant documents.
Difference from MRR: MRR only cares about the first relevant document; MAP considers all relevant documents' ranks.
Limitations: Requires binary relevance; ignores graded relevance.

F. Normalized Discounted Cumulative Gain (NDCG)

Intuition: Evaluates ranking quality with multi-level (graded) relevance; highly relevant documents near the top are most valuable.

\[ DCG@k = rel_1 + \sum_{i=2}^k \frac{rel_i}{\log_2(i+1)} \] \[ NDCG@k = \frac{DCG@k}{IDCG@k} \]

Where \(rel_i\) is the relevance score (e.g., 0, 1, 2), and \(IDCG\) is the DCG of the ideal ranking.

Example: Suppose relevance grades for top-3 retrieved: [2, 1, 0]

DCG@3 = 2 + (1 / log2(3)) + (0 / log2(4)) ≈ 2 + 0.6309 + 0 = 2.6309
Best possible (ideal): [2, 1, 0] ⇒ Same order ⇒ IDCG@3 = 2.6309
NDCG@3 = 2.6309 / 2.6309 = 1.0

Interpretation: NDCG close to 1 means near-perfect ranking of highly relevant documents at the top.
Limitations: Requires graded relevance labels; more complex to annotate and compute.

Metric	What it measures	Ranking considered?	When to use
Precision@k	Proportion of top-k retrieved that are relevant	No (just top-k)	Limit irrelevant context
Recall@k	Proportion of all relevant retrieved in top-k	No (just top-k)	Maximize coverage
Hit Rate@k	At least one relevant in top-k?	No	Binary coverage
MRR	How high first relevant appears	Yes (first relevant)	Single answer retrieval
MAP	Average precision over all relevant docs	Yes (all relevant)	Multi-label retrieval
NDCG	Ranking quality with graded relevance	Yes (graded positions)	Graded relevance, ranking-focused tasks

3. Generation and End-to-End Evaluation Metrics

Evaluating the output of the LLM in a RAG system involves assessing several dimensions, each capturing distinct aspects of answer quality. Definitions and scoring methods vary by project and framework.

Faithfulness / groundedness:
- Definition: Are claims in the answer supported by the retrieved context?
- Good: Cites facts present in context.
- Bad: Invents information not in the context.
- Assessment: Requires both answer and retrieved context.
Answer relevance:
- Definition: Does the answer address the user’s question?
- Good: Directly answers the question.
- Bad: Off-topic or generic response.
- Assessment: Needs question and answer.
Context relevance:
- Definition: Are retrieved documents useful for answering the question?
- Good: Retrieved passages provide supporting evidence.
- Bad: Only tangentially related or irrelevant passages.
- Assessment: Needs query, retrieved docs, and human/LLM judgment.
Answer correctness:
- Definition: Does the answer match reference or ground-truth answers?
- Good: Matches expected response.
- Bad: Incorrect or incomplete answer.
- Assessment: Needs question, answer, and reference answer.
Completeness:
- Definition: Does the answer cover all important parts of the question?
- Good: Addresses all sub-questions or facets.
- Bad: Only partially answers.
- Assessment: Needs question, answer, and possibly reference answer.
Citation accuracy:
- Definition: Do cited sources actually support the claims?
- Good: All citations are verifiable in retrieved context.
- Bad: Cites irrelevant or incorrect sources.
- Assessment: Needs answer, citations, and context.
Hallucination:
- Definition: Does the answer contain unsupported or fabricated claims?
- Good: All content is grounded in the context or general knowledge.
- Bad: Fabricates facts or quotes.
- Assessment: Needs answer, context, and possibly reference answer.

Note: Faithfulness and correctness are not interchangeable. An answer can be perfectly faithful (fully supported by the retrieved context), yet incorrect if the context itself is wrong or outdated. Conversely, a correct answer may pull from outside the retrieved context, reducing faithfulness but not correctness.

Some metrics require:

Reference answers (answer correctness, completeness).

Retrieved contexts (

Faithfulness, citation accuracy, hallucination).
Human or LLM judgment (faithfulness, completeness, citation accuracy, hallucination, answer relevance, context relevance).

Not all metrics are universally standardized. Implementation details, thresholds, and scoring rubrics vary—especially for faithfulness, hallucination, and citation accuracy. Many frameworks allow for both automated and manual (human/LLM) assessment.

4. Worked Evaluation Example

Let’s walk through a small, hypothetical dataset for three user queries. For consistency, all retrieval units will be document IDs.

Query	Relevant Doc IDs	Retrieved Top-3 Doc IDs	Generated Answer	Reference Answer
Q1: Who wrote "The Time Machine"?	{D2}	[D2, D5, D4]	"H.G. Wells wrote 'The Time Machine' in 1895."	"H.G. Wells"
Q2: What is the capital of France?	{D1, D3}	[D5, D3, D7]	"The capital of France is Paris."	"Paris"
Q3: Name a planet discovered in the 19th century.	{D6, D8}	[D5, D7, D10]	"Neptune is a planet discovered in 1846."	"Neptune"

1. Precision@3

Q1: Retrieved [D2, D5, D4]; relevant {D2}; relevant in top-3 = 1
Precision = 1/3 ≈ 0.33
Q2: Retrieved [D5, D3, D7]; relevant {D1, D3}; relevant in top-3 = 1 (D3)
Precision = 1/3 ≈ 0.33
Q3: Retrieved [D5, D7, D10]; relevant {D6, D8}; relevant in top-3 = 0
Precision = 0/3 = 0

Average Precision@3: (0.33 + 0.33 + 0) / 3 ≈ 0.22

2. Recall@3

Q1: 1 relevant in top-3, 1 total relevant ⇒ Recall = 1/1 = 1.0
Q2: 1 relevant in top-3, 2 total relevant ⇒ Recall = 1/2 = 0.5
Q3: 0 relevant in top-3, 2 total relevant ⇒ Recall = 0/2 = 0

Average Recall@3: (1.0 + 0.5 + 0) / 3 ≈ 0.5

3. Hit Rate@3

Q1: At least one relevant in top-3 ⇒ 1
Q2: Yes (D3) ⇒ 1
Q3: None ⇒ 0

Average Hit Rate@3: (1 + 1 + 0) / 3 ≈ 0.67

4. Reciprocal Rank and MRR

Q1: D2 at position 1 ⇒ Reciprocal rank = 1/1 = 1.0
Q2: D3 at position 2 ⇒ Reciprocal rank = 1/2 = 0.5
Q3: No relevant docs ⇒ Reciprocal rank = 0

MRR: (1.0 + 0.5 + 0) / 3 ≈ 0.5

5. NDCG Example (Graded Relevance)

Let’s assume graded relevance for Q2 (capital of France):

D1 (encyclopedia entry): relevance = 2
D3 (travel guide mentioning Paris): relevance = 1
D5, D7: relevance = 0

System retrieves [D5 (0), D3 (1), D7 (0)].

\[ DCG@3 = rel_1 + \frac{rel_2}{\log_2(3)} + \frac{rel_3}{\log_2(4)} \] \[ DCG@3 = 0 + \frac{1}{\log_2(3)} + \frac{0}{\log_2(4)} \approx 0 + 0.6309 + 0 = 0.6309 \] Best possible order: [D1 (2), D3 (1), D7 (0)] \[ IDCG@3 = 2 + \frac{1}{\log_2(3)} + 0 = 2 + 0.6309 = 2.6309 \] \[ NDCG@3 = 0.6309 / 2.6309 \approx 0.24 \]

6. Simple Faithfulness or Correctness Assessment

Q1: "H.G. Wells wrote 'The Time Machine' in 1895."
If D2 (retrieved) contains both the author and the year, answer is faithful and correct.
Q2: "The capital of France is Paris." D3 is a travel guide mentioning Paris, but D1 (most authoritative) was not retrieved, so possible partial faithfulness, but answer is correct.
Q3: "Neptune is a planet discovered in 1846." None of D6 or D8 (relevant docs) were retrieved, so the answer may be correct (if model hallucinated), but not faithful to context.

Key lessons: High recall does not guarantee a correct or faithful answer. Faithfulness depends on whether the retrieved context truly supports the generated answer.

5. Implementing Retrieval Metrics in Python


import numpy as np

# Test data: List of queries with relevant and retrieved doc IDs
queries = [
    {'relevant': {'D2'}, 'retrieved': ['D2', 'D5', 'D4']},
    {'relevant': {'D1', 'D3'}, 'retrieved': ['D5', 'D3', 'D7']},
    {'relevant': {'D6', 'D8'}, 'retrieved': ['D5', 'D7', 'D10']}
]

def precision_at_k(retrieved, relevant, k):
    retrieved_k = retrieved[:k]
    num_relevant = sum([1 for doc in retrieved_k if doc in relevant])
    return num_relevant / k

def recall_at_k(retrieved, relevant, k):
    if not relevant:
        return 0.0  # or np.nan, depending on convention
    retrieved_k = retrieved[:k]
    num_relevant = sum([1 for doc in retrieved_k if doc in relevant])
    return num_relevant / len(relevant)

def hit_rate_at_k(retrieved, relevant, k):
    retrieved_k = retrieved[:k]
    return int(any(doc in relevant for doc in retrieved_k))

def reciprocal_rank(retrieved, relevant):
    for idx, doc in enumerate(retrieved, 1):
        if doc in relevant:
            return 1.0 / idx
    return 0.0

def ndcg_at_k(retrieved, relevance_grades, k):
    dcg = 0.0
    for i in range(k):
        rel = relevance_grades[retrieved[i]] if i < len(retrieved) and retrieved[i] in relevance_grades else 0
        dcg += rel / np.log2(i + 2)
    # Ideal DCG
    ideal_order = sorted(relevance_grades.values(), reverse=True)
    idcg = 0.0
    for i in range(min(k, len(ideal_order))):
        idcg += ideal_order[i] / np.log2(i + 2)
    return dcg / idcg if idcg > 0 else 0.0

ks = 3
precisions = []
recalls = []
hits = []
rrs = []

# For NDCG, set up relevance grades for Q2
relevance_grades_q2 = {'D1': 2, 'D3': 1, 'D5': 0, 'D7': 0}

for i, q in enumerate(queries):
    precisions.append(precision_at_k(q['retrieved'], q['relevant'], ks))
    recalls.append(recall_at_k(q['retrieved'], q['relevant'], ks))
    hits.append(hit_rate_at_k(q['retrieved'], q['relevant'], ks))
    rrs.append(reciprocal_rank(q['retrieved'], q['relevant']))

print("Precision@3:", np.mean(precisions)) # ≈ 0.22
print("Recall@3:", np.mean(recalls))       # ≈ 0.5
print("Hit Rate@3:", np.mean(hits))        # ≈ 0.67
print("MRR:", np.mean(rrs))                # ≈ 0.5

# NDCG for Q2
print("NDCG@3 for Q2:", ndcg_at_k(queries[1]['retrieved'], relevance_grades_q2, ks)) # ≈ 0.24

Interpretation: The results match the calculations shown in the previous section. Precision is low (many irrelevant docs), recall is moderate (sometimes only part of the ground truth is retrieved), and MRR is moderate (sometimes relevant docs are not in the top spot). NDCG reflects that a highly relevant doc was not retrieved first.

Handling special cases:

If relevant is empty, recall is 0 (or undefined—decide per project).
If no relevant doc in retrieved, reciprocal rank is 0.

6. RAG Evaluation Frameworks and Tools

Choosing the right RAG evaluation tool depends on the workflow, depth of analysis, and need for experiment management or tracing. Here’s an updated comparison of leading tools:

Tool	Primary Purpose	RAG Evaluation Types	Metrics/Capabilities	Dataset & Experiment Management	Tracing/Observability	Key Advantages	Limitations	When to Use
Ragas	RAG evaluation library	Retrieval, generation, faithfulness, hallucination	Precision, recall, faithfulness (LLM), answer relevance, context relevance, hallucination	Dataset API, batch eval	No	Simple, open source, extensible	Limited experiment tracking, LLM cost for some metrics	Fast metric prototyping, simple pipelines
DeepEval	Eval framework for LLM/RAG	Retrieval, generation, custom metrics	Precision, recall, MRR, NDCG, LLM-based metrics, custom scoring via Python	Dataset versioning, experiment logging	No	Flexible, good for custom eval scripts	Manual experiment setup, no built-in tracing	Custom evaluation pipelines, R&D
TruLens	Tracing and evaluation for LLM apps	Retrieval, generation, LLM-based metrics	Faithfulness, context relevance, answer relevance (LLM), tracing, feedback records	Experiment tracking, dashboard	Yes	Integrated tracing + eval, supports LangChain/OpenAI	Some metrics require LLMs, complex for simple use	Debugging, tracing, feedback loops
LangSmith	Experiment management & tracing	Retrieval, generation, custom metrics	Custom evals (LLM/human), tracing, dataset management	Versioned datasets, runs, leaderboards	Yes	Full experiment lifecycle, tight LangChain integration	Best for LangChain users, SaaS pricing	Teams with LangChain stacks, production monitoring
Arize Phoenix	Observability for LLM/RAG	Retrieval, generation, analytics	Precision, recall, faithfulness, outlier detection, dashboards	Dataset upload, experiment tracking	Yes (deep traces, error analysis)	Great for production, rich dashboards	Setup required, cloud or self-hosted	Production RAG monitoring, analytics

Note: Metric names and implementations differ between tools; always check the official docs and API for details and version compatibility.

7. LLM-as-a-Judge Evaluation

When human labels are scarce, an LLM can serve as an automated judge for RAG outputs. This involves feeding the LLM a rubric (criteria), the question, the generated answer, the retrieved context, and optionally a reference answer.

Judge Inputs: User question, generated answer, retrieved context (and optionally reference answer or citations).
Rubric-based scoring: The LLM is prompted to assign a score based on a provided rubric (criteria and scale, e.g., 1–5 for faithfulness).
Pairwise comparison: The LLM compares two generated answers for the same question and context, indicating which is better and why.
Structured outputs: The LLM can be instructed to return a score, rationale/evidence, and optionally label specific errors (e.g., unsupported claims, missing info).

Benefits:

Scales to thousands of outputs when human review is too costly.
Can enforce detailed, consistent rubrics when prompted well.
Allows rapid iteration and ablation studies.

Challenges:

Position bias: The judge may prefer the first or longer answer.
Model bias: The judge may favor answers similar to its own style.
Prompt sensitivity: Results can vary with prompt wording.
Cost and reproducibility: Each judgment incurs LLM inference costs and may not be fully deterministic.
Human disagreement: LLM judges do not always match human preferences or task-specific nuances.

Example: Faithfulness Rubric (1–5)


You are an expert evaluator. Your task is to assess the faithfulness of an answer to a user question, using the provided retrieved context. Faithfulness means that all factual claims in the answer are directly supported by the retrieved context.
Assign a score from 1 to 5:
- 5: All claims are fully supported by the context, no unsupported information.
- 4: Most claims are supported; minor detail(s) not covered.
- 3: Some claims are supported, but significant parts are missing or unsupported.
- 2: Only a few claims are supported; most information is unsupported.
- 1: Answer is entirely unsupported by the context.
Provide evidence from the context that supports your decision.

A prompt like the above, along with the question, answer, and relevant context, will produce both a score and a rationale. This can be automated via most LLM APIs.

When human review remains necessary:

For high-stakes domains (medical, legal, etc.)
When calibrating or auditing LLM-judge reliability
For ambiguous or nuanced evaluation criteria
To validate new LLM judge prompts and scoring consistency

8. Building a Reliable RAG Evaluation Dataset

A robust evaluation dataset is the backbone of meaningful RAG assessment. It should represent real-world usage, edge cases, and failure modes.

Representative user queries: Use actual search logs, support tickets, or simulated queries from domain experts.
Reference answers: High-quality, trusted answers for each query (may be short or detailed).
Relevant document or chunk IDs: For each query, annotate which documents/passages are necessary or sufficient to answer the question.
Retrieved contexts: Store the actual top-k retrieved results per query per retrieval experiment.
Expected citations: If the RAG system outputs citations, annotate expected IDs or snippets.
Queries with insufficient info: Include queries where no answer exists in the knowledge base—these test the system’s ability to abstain or indicate uncertainty.
Multi-document questions: Include queries requiring synthesis from several sources.
Ambiguous queries and edge cases: Include questions that can be interpreted in multiple ways, or that probe for common errors (e.g., outdated info, entity disambiguation).
Frequently updated content: For time-sensitive domains, track whether the retrieved info is current.
Train/dev/test splits: Hold out a test set untouched during model development to prevent overfitting and leakage.

Preventing evaluation leakage:

Ensure queries are not seen by the system during training or retrieval index creation.
For synthetic queries (generated by an LLM), check for overlap with your document base to avoid trivial answers.
Mix synthetic and real queries for coverage, but do not rely solely on synthetic data.

Reusing benchmarks for ablation: When changing your embedding model, chunking strategy, retrieval method, or reranker, evaluate on the same benchmark dataset to fairly compare system changes.

9. Diagnosing RAG Evaluation Failures

Metric-driven debugging is key to RAG system improvement. Here are common failure scenarios and how to interpret and act on them:

1. High Recall@10 but Low Precision@10
- What it means: Most relevant documents are found, but many irrelevant ones are included.
- Does NOT establish: That the top-ranked docs are helpful for the LLM.
- Investigate: Retrieval scoring, index noise, precision-oriented rerankers.
- Experiment: Add a reranker or reduce k; analyze why irrelevant docs are scored highly.
2. High Retrieval Scores but Low Faithfulness
- What it means: System is retrieving relevant docs, but the LLM is hallucinating or ignoring context.
- Does NOT establish: That LLM is reading/using context correctly.
- Investigate: Prompt engineering, context length, and LLM’s ability to attend to context.
- Experiment: Try more explicit prompts, context highlighting, or better context truncation.
3. High Faithfulness but Low Answer Correctness
- What it means: LLM is accurately reflecting the context, but the context itself is wrong, outdated, or insufficient.
- Does NOT establish: That the knowledge base is complete/correct.
- Investigate: Content quality, KB updates, relevance annotation errors.
- Experiment: Review and improve content sources, update corpus, fix annotation errors.
4. High Hit Rate@5 but Poor MRR
- What it means: Relevant docs are somewhere in the top-5, but rarely at the top position(s).
- Does NOT establish: That the LLM is seeing the most useful context first.
- Investigate: Ranking algorithm, document scoring, position bias.
- Experiment: Tune retrieval scoring, test rerankers, analyze position of relevant docs.
5. High Retrieval Relevance but Incomplete Answers
- What it means: All needed sources are retrieved, but the LLM output omits key points.
- Does NOT establish: That the LLM can synthesize multi-doc answers well.
- Investigate: LLM prompt, output length limits, summarization capabilities.
- Experiment: Adjust prompts to encourage completeness, increase max tokens, test LLM variants.
6. Good Offline Metrics but Poor User Satisfaction
- What it means: Benchmarks look strong, but real users are unhappy.
- Does NOT establish: That offline metrics match real-world needs.
- Investigate: Dataset representativeness, real-world query drift, UI/UX issues.
- Experiment: Collect user feedback, expand test set, monitor live queries.
7. Correct Answers with Incorrect Citations
- What it means: The LLM provides the right answer, but cites sources that don’t support it.
- Does NOT establish: That the answer is grounded in the provided context.
- Investigate: How citations are attached/generated; whether context supports the claims cited.
- Experiment: Improve citation extraction, post-process answer-citation links, prompt the LLM for evidence explicitly.

10. RAG Evaluation Interview Questions and Answers

Why is RAG evaluation different from evaluating a traditional ML model?
Answer: Traditional ML models are typically evaluated on single-task outputs (e.g., accuracy, F1). RAG systems involve both retrieval and generation components, requiring separate and joint evaluation of retrieval quality (did we get the right documents?), generation quality (did we produce a good answer?), and end-to-end quality (did the system give the user what they wanted?).
Follow-up: How might you isolate retrieval vs. generation errors in a failed RAG response?
What is the difference between Precision@k and Recall@k?
Answer: Precision@k is the fraction of the top-k retrieved documents that are relevant; recall@k is the fraction of all relevant documents that appear in the top-k. Precision measures “purity,” recall measures “coverage.”
Example: If there are 2 relevant docs and 3 retrieved docs (only one is relevant), Precision@3 = 1/3; Recall@3 = 1/2.
How does Hit Rate@k differ from Recall@k?
Answer: Hit Rate@k is a binary metric: 1 if at least one relevant doc is in the top-k, 0 otherwise. Recall@k is proportional and counts how many of all relevant docs are retrieved.
Example: If there are 3 relevant docs and only one is retrieved in top-5, Hit Rate@5 = 1; Recall@5 = 1/3.
Explain Mean Reciprocal Rank (MRR) and when it is useful.
Answer: MRR averages the reciprocal of the rank of the first relevant document across queries. It’s especially useful when only one relevant document per query is needed (e.g., factoid QA).
How does NDCG differ from MRR?
Answer: NDCG (Normalized Discounted Cumulative Gain) supports multi-level (graded) relevance and rewards ranking highly relevant docs at the top. MRR only considers the first relevant doc and assumes binary relevance.
What is MAP (Mean Average Precision), and when would you use it?
Answer: MAP averages the precision at every position where a relevant document is found, over all queries. It’s valuable when multiple relevant docs are expected per query and both ranking and completeness matter.
What is faithfulness, and how does it differ from correctness?
Answer: Faithfulness measures whether the answer is supported by the retrieved context. Correctness checks if the answer matches ground-truth/reference answers. An answer can be faithful but incorrect (if context is wrong), or correct but unfaithful (if the answer is right but not grounded in the retrieved docs).
How would you evaluate RAG when reference answers are unavailable?
Answer: Use faithfulness, context relevance, and human or LLM-judge assessments to check if the answer is grounded in the context. You can also use question-answer consistency checks and user feedback.
How does LLM-as-a-judge work, and what are its limitations?
Answer: An LLM is prompted to score outputs based on a rubric (e.g., faithfulness, relevance) using the question, answer, and context. Benefits: scalable, consistent. Limitations: biases, prompt sensitivity, may not always agree with humans, cost.
Follow-up: How would you calibrate or validate LLM-judge scores?
How would you build a reliable RAG benchmark?
Answer: Collect representative queries, annotate reference answers and supporting document IDs, include edge cases (unanswerable, ambiguous), maintain train/dev/test splits, and periodically update with real user queries.
Why can high retrieval recall coexist with poor answer quality?
Answer: Retrieving all relevant docs (high recall) doesn’t guarantee the LLM will use them properly. The LLM might hallucinate or misunderstand context, or present incomplete/irrelevant answers.
How would you evaluate whether a reranker improves the system?
Answer: Compare retrieval metrics (MRR, NDCG, Precision@k) on a held-out benchmark before and after using the reranker. Also check downstream answer correctness/faithfulness to ensure gains propagate.
Follow-up: What if retrieval improves but answer quality declines?
How would you compare two RAG pipelines fairly?
Answer: Use the same benchmark dataset, queries, and evaluation metrics (retrieval, generation, end-to-end). Ensure no pipeline has seen test data during development. Optionally use human or LLM-judge for qualitative comparison.
How would you investigate a system with high faithfulness but low correctness?
Answer: Review the knowledge base for completeness and accuracy. High faithfulness means the LLM is using the context, but if correctness is low, the context may be incomplete or wrong. Update, correct, or expand the knowledge base.
<How would you balance retrieval quality, answer quality, latency, and evaluation cost?
Answer: Balancing these factors requires trade-offs:
- Retrieval quality can be improved with more sophisticated retrieval methods (reranking, hybrid search), but this often increases latency.
- Answer quality benefits from better prompts and more capable LLMs, which may be slower and more expensive.
- Latency can be reduced by limiting the number of retrieved documents, using faster models, or optimizing infrastructure, but this may hurt recall or answer completeness.
- Evaluation cost (especially with LLM-based or human-in-the-loop metrics) can be managed by sampling, batching, or using automated metrics where possible.
Practical Example: In a customer support chatbot, you might use a fast retrieval model for initial recall, then rerank only a small top-k for generation, and sample a subset of user interactions for expensive LLM or human review.
Follow-up: When would you accept slightly lower retrieval/answer quality for much faster response times?

Follow-up Questions for Selected Topics

Isolating retrieval vs. generation errors: How would you design an experiment to attribute RAG system failures to retrieval or generation?
LLM-judge calibration: How do you ensure that LLM-judge evaluations align with human experts? Would you use inter-rater agreement metrics?
Evaluating rerankers: What additional metrics or analyses could you use to detect if a reranker is overfitting to your benchmark dataset?
Latency vs. quality trade-off: What user scenarios justify higher latency (e.g., financial research) versus those demanding instant responses (e.g., autocomplete)?

Conclusion

RAG evaluation is a multi-layered process that requires careful metric selection, dataset construction, and diagnostic analysis. By understanding and applying the right retrieval and generation metrics—and leveraging modern frameworks and LLM-judge techniques—you can systematically improve your RAG pipelines and prepare for technical interviews or production deployments. Always test on representative, leakage-free datasets and combine automated and human evaluation for best results.

Next steps: Try implementing these metrics on your own RAG projects, experiment with different frameworks, and practice the above interview questions to deepen your expertise.