Evaluate retrieval and answer quality

Methods to understand

Build a question set covering ordinary, cross-passage, unanswerable, and false-premise cases. Evaluate whether retrieval finds the relevant material, and whether answers are correct, supported, and appropriately withheld. Record a baseline and retest each case.

Apply the method to a scenario

Query: "Does the project permit cross-department access to material?"
Record documents, sources, and answers; abstain without evidence.
Compare retrieval and citations before and after changes and record one failure.

Hands-on workshop

Build a RAG evaluation set that reveals failures.

1. Prepare at least ten normal, similar, cross-passage, unanswerable, and false-premise questions.
2. Record expected passages, retrieved passages, answers, and sources per question.
3. Score retrieval and answers separately and compare results against the same baseline.

const evaluation = [
  { question: "Who can read documents?", expectedIds: ["doc-a-01"], shouldAbstain: false },
  { question: "What must happen after document updates?", expectedIds: ["doc-a-02"], shouldAbstain: false },
  { question: "What is the price that the material does not specify?", expectedIds: [], shouldAbstain: true }
];
// Record for each case: retrievalPass、answerPass、citationPass、failureReason。

How to verify

Submit one example where retrieval succeeds but the answer fails and another where abstention is required.

Practice task

Choose a case related to your work and practice "Evaluate retrieval and answer quality". Submit inputs, steps, results, and verification records.

• Explain the task goal and inputs
• Provide actual results or runnable deliverables
• Identify errors, limits, and verification evidence