Learn RAG by Testing an LLM Against a Small Document Set

Understand the pipeline before adding tools

Separate loading documents, finding relevant passages and generating an answer. Preserve the document ID with each passage so a reader can trace the result. For this tiny set, start by selecting evidence manually; you do not need a vector database to discover whether the questions and reference answers make sense.

When you later automate retrieval, compare a simple text search with any embedding-based approach on the same questions. Keep the source collection and test sheet unchanged during the comparison. A more elaborate retrieval pipeline is useful only if your checks show a benefit that justifies its additional setup and failure modes.

Use the original practice documents

Download the room, equipment and workshop documents from the practice files box. They describe Northstar Learning Lab, a fictional organization with Seattle-local workshop times. The contact addresses use example.invalid and do not reach a real service. Document dates describe this exercise’s fixed version, not a current business policy.

The question sheet contains eight answerable questions and four unanswerable questions. For example, the Maple room holds 12 people. Finding the RAG study group’s room and its capacity requires both the workshop and room documents: Cedar, six people. The documents do not disclose a Wi-Fi password, weekend opening hours, a projector rental fee or employee benefits.

Score evidence separately from the answer

For every question, save the retrieved document IDs and passages before reading the generated response. Check whether those passages contain all required facts. Then compare the response with the reference answer: are the facts correct, does it answer the question and are its claims supported by the cited text? Finding a relevant document does not make every statement in an answer trustworthy.

For this exercise, an unanswerable question should return exactly “Not found in the provided documents.” Do not fill a gap using a plausible policy or outside knowledge. Missing information does not prove that a service is unavailable or an activity is forbidden. Keep the question sheet and reference answers out of the retrieval collection; otherwise you would be testing access to the answer key.

• Report answerable cases passed out of 8 and missing-information cases passed out of 4.
• Require both DOC-WORKSHOPS and DOC-ROOMS as evidence for the room-and-capacity question.
• Check Seattle local time and day of the week without inventing a calendar date or UTC offset.
• Record unsupported extra claims even when the main answer is correct.

Change one thing and investigate failures

Once you have a baseline, change one setting, such as passage size or the number of retrieved passages, and rerun all 12 questions. A retrieval failure means the necessary evidence did not reach the answering stage. An answer failure means the evidence was available but the response was wrong, incomplete or unsupported. Keep those counts separate so the next change addresses the actual problem.

The files and manual checks need no API key or software dependency. An automated version may require a model, libraries and paid services; confirm those separately. When comparing courses, look for document loading, source attribution, missing-answer handling and evaluation exercises, and inspect their stated editions. These 12 questions are a learning fixture, not proof of production reliability or performance on unfamiliar documents.