Grounded Answering

Everything so far finds the right passages. The last stage hands them to a model in a way that keeps it honest. Three rules do most of the work:

Answer only from the sources. The retrieved chunks go into the prompt, with an explicit instruction to use nothing else.
Cite. Each chunk already has an id (notes.md#2). Making the model cite ids turns "trust me" into "check notes.md#2" — and makes retrieval failures visible when a claimed citation doesn't say what the answer claims.
Refuse honestly. Two failure modes, two guards. If even the best retrieval score is weak, the corpus doesn't cover the question — refuse before spending tokens on an LLM call. If the scores looked fine but the passages still don't answer it, the prompt instructs the model to say a fixed sentinel — NOT_IN_CONTEXT — instead of improvising. A sentinel beats prose ("Unfortunately I could not…") because your code can match it exactly and translate it to a UX-appropriate message.

Prompt assembly is plain string building, and the exact format is the graded function. Add it to rag.py:

def build_prompt(question, hits):
    lines = [
        "Answer the question using only the sources below.",
        "Cite every source you use by its bracketed id, like [notes.md#2].",
        "If the sources do not answer the question, reply exactly: NOT_IN_CONTEXT",
        "",
    ]
    for source, text in hits:
        lines.append(f"[{source}] {text}")
        lines.append("")
    lines.append(f"Question: {question}")
    return "\n".join(lines)

Instructions first, then one [id] text block per hit with blank lines between, then the question last (models weight the end of a prompt heavily — finish on the task, not on source #7). This is the same job LangChain's "stuff documents" chain or LlamaIndex's response synthesizer does: format retrieved context into a template. Frameworks make it configurable; the string logic is this.

Now the LLM client — the same injectable-callable seam as the CLI chatbot course, simplified for this course's shape. There, client(messages) -> str took the full conversation history, because each turn had to see everything said before. Here there's no multi-turn memory to replay — every question gets one assembled prompt and one answer — so the callable narrows to client(prompt) -> str, a single string in, a string back (pip install anthropic, or adapt to the OpenAI SDK the same way as in that course):

import anthropic

_anthropic = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY


def llm_client(prompt):
    response = _anthropic.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text

Close the loop:

REFUSAL = "I couldn't find anything about that in your documents."


def ask(question, k=4, min_score=0.25):
    hits = []
    for chunk_id, chunk, dist in retrieve(question, k):
        if 1.0 - dist >= min_score:          # cosine distance -> similarity
            hits.append((chunk_id, chunk))
    if not hits:
        return REFUSAL                        # guard 1: weak retrieval
    reply = llm_client(build_prompt(question, hits))
    if reply.strip() == "NOT_IN_CONTEXT":
        return REFUSAL                        # guard 2: model saw nothing useful
    return reply


if __name__ == "__main__":
    print("rag ready — type 'quit' to exit")
    while True:
        question = input("ask> ").strip()
        if question in ("quit", "exit"):
            break
        if question:
            print(ask(question))

Run it. Ask something your documents answer and check the citations point at real chunks. Then ask something they don't answer ("who won the 1998 World Cup?") and watch it refuse instead of hallucinating. That refusal is the difference between a demo and a system you can put in front of users. min_score is a knob you must tune against your own corpus and embedding model — try a few queries and watch the scores; there is no universal threshold.

The challenge grades build_prompt character-for-character. Pedantic? It's the point: prompts are interfaces. In production you'll diff prompts across versions, cache on their hashes, and write regression tests asserting a retrieved chunk landed in the right slot. "Roughly the right string" isn't a spec.

› Build the Prompt