AI Feders original practice project: evaluate a small document assistant
Fixture version: 2026-10-15

Purpose and scope
Use exactly rooms.txt, equipment.txt and workshops.txt as the knowledge source.
Northstar Learning Lab is fictional. The addresses use example.invalid and
are not real contacts. The fixed document version is part of the exercise.
This is not an actual organization's current policy.
You can complete the evidence and answer checks manually with these text and
CSV files. No API key, software package or paid service is needed for that.
An automated RAG implementation is optional and requires your own environment.

Keep the test separate
questions.csv is an answer key and evaluation sheet, not a retrieval source.
Do not put it, these instructions or the guide into your retrieval collection.
There are 12 questions: 8 answerable and 4 unanswerable. The separator between
multiple source IDs or required facts inside a CSV cell is a vertical bar (|).
The CSV itself uses comma separators and should be read with a CSV parser.

Baseline workflow
1. Read the three documents. Keep each Document ID with any extracted passage.
2. For each question, choose the evidence before consulting the expected answer.
3. Save the retrieved document IDs, exact supporting text and proposed answer.
4. Compare with the expected answer and required facts. Record each failed
   check and whether it originated in retrieval or in the answer.
5. If you automate the pipeline, rerun all 12 questions after one change.

Exact checks
Q01: Maple capacity must be 12 people, supported by DOC-ROOMS.
Q02: The advance booking limit must be 14 days, supported by DOC-ROOMS.
Q03: One reservation's maximum duration must be 2 hours, from DOC-ROOMS.
Q04: USB-C presentation adapters must have a 3-day loan, from DOC-EQUIPMENT.
Q05: The return deadline must be before 16:00 Seattle local time on the due
     date, from DOC-EQUIPMENT. Do not change 'before' into 'at or before'.
Q06: The CSV workshop must be Wednesdays at 10:00 Seattle local time in Maple,
     from DOC-WORKSHOPS. Do not invent a calendar date or a UTC offset.
Q07: The RAG study group meets in Cedar (DOC-WORKSHOPS), and Cedar holds
     6 people (DOC-ROOMS). Both documents are required for this answer.
Q08-Q11: Return exactly 'Not found in the provided documents.' without the
     surrounding quote marks. No password, opening hours, rental fee or
     employee benefits are supplied. Do not invent them, search outside
     the supplied collection or claim their absence proves unavailability.
Q12: The organizer address must be learning@example.invalid, DOC-WORKSHOPS.

Scoring
For each answerable case, record three separate yes/no checks:
- Evidence: retrieved passages contain every required fact and identify all
  expected source IDs. Retrieving an ID without its relevant text is not enough.
- Correctness: the answer includes the reference facts and answers the question.
  Equivalent wording is allowed; matching a keyword alone is not a passing answer.
- Support: every factual claim in the answer is supported by its cited text.
An answerable case passes only if all three checks pass. Report passed / 8.
For each unanswerable case, require the exact missing-information answer and
no extra factual claim. Report passed / 4 separately; do not hide these failures
inside one combined score. Review the response, not just a substring match.

Failure diagnosis
Missing evidence at the answering step is a retrieval failure. Necessary
evidence present but a wrong, incomplete or unsupported response is an answer
failure. It is possible to have both. Record observations without guessing cause.
This small fixture does not test conflicting versions, permissions, malicious
documents, scale, model consistency or production reliability.
Keep the original files fixed; save any extended experiment as a separate copy.