Overview

In Training, context, and evidence, we supplied source material and examples in the prompt. Now we'd like to build on that experiment: how do we choose what goes into the prompt, and how do we make the answer useful to our code?

The Python code that imports predictions into the Galactic Archives expects one label per claim. The assistant returns a label, three paragraphs (all in perfectly formatted Markdown, of course), and an offer to help with anything else. Very polite. Less convenient for the next line of Python.

We'll make the output usable, then make more evidence available. These solve different problems: a valid JSON object can still contain an unsupported claim, and a retrieved record can still concern the wrong Zarbon.

Outcomes

After this lesson, you should be able to…

• distinguish an output instruction, JSON mode, and a JSON Schema request
• check completion status, output structure, and semantic correctness separately
• compare supplied evidence with retrieved records without updating model weights
• inspect retrieval errors, ambiguous questions, and conflicting sources
• explain how retrieval can select demonstrations as well as source material

Specify the prediction

Return to Galactic Archives record M1: Zarbon the Great ordered the finest purple caturra coffee beans on Tuesday. No arrival date is recorded. In the previous lesson, we asked for a label. Let's imagine that our code also needs the ID of the claim being evaluated, so we can match each prediction to the claim we submitted. We'll use integer ID 3 for the claim about the coffee shipment's arrival date, followed by one of SUPPORTED, CONTRADICTED, or UNKNOWN.

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field


class Prediction(BaseModel):
    """A claim label tied to an example ID."""

    model_config = ConfigDict(extra="forbid", strict=True)
    example_id: int = Field(description="ID of the claim being evaluated.")
    label: Literal["SUPPORTED", "CONTRADICTED", "UNKNOWN"] = Field(
        description="Judgment based only on the supplied record."
    )


print(Prediction.model_json_schema())

The schema describes the fields, their types, and the allowed labels. It does not contain a procedure for deciding whether a claim follows from a record.

Try validating {"example_id": 3, "label": "UNKNOWN"} with Prediction.model_validate_json(...). Then try "perhaps" as the label, a string instead of an integer ID, or an extra field. Predict which will fail.

Request the structure

After defining the claim and record, we can request a prediction with this structure:

response = client.chat.completions.create(
    model=model,
    messages=messages,
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "Prediction",
            "strict": True,
            "schema": Prediction.model_json_schema(),
        },
    },
    temperature=0,
    max_tokens=256,
    stream=False,
)

For a complete, runnable example, see classify in gateway_lab.py. It defines the claim and record, constructs the request, and validates the returned prediction.

The gateway and serving model must support the requested format. A schema option in a client library doesn't establish that every route can enforce it. This lesson uses a flat schema to keep both the request and any failure easy to inspect.

So can every LLM produce valid JSON?

Many language models can produce valid JSON, but asking for it doesn't guarantee that every response will parse. Training on code and examples of structured responses can help a model follow the requested format. That learned behavior is different from enforcing the format while generating an answer.

One common approach is constrained decoding. Recall that the model assigns scores to possible next token IDs. The serving software can use our schema to track which continuations are allowed. At each step, it blocks tokens that would make the output incompatible with the required structure, then selects from the remaining candidates. As more text is generated, the allowed continuations change. XGrammar's introduction walks through this process.

For our label field, the constraints can allow the three specified labels and exclude a value such as "perhaps". They can also enforce JSON punctuation and the expected field types. They can't decide which allowed label follows from the coffee order. We can still get a perfectly valid, perfectly wrong answer. Imperial paperwork has its limitations.

This constraint is applied during generation, without retraining the model. Its availability depends on the serving software, the model's integration, and which schema features that service supports. vLLM is one example of a serving system that provides this functionality. An API's structured output option doesn't establish how every provider implements it.

Even with format enforcement, a request can fail or generation can stop before the object is complete. Local validation checks the returned result. It doesn't constrain the model's generation or repair an incomplete response. That brings us to three separate checks.

Check the response

We need three types of checks before accepting a prediction:

1. Completion: Did the response finish? An output limit, refusal, or empty response is a different outcome from an incorrect label.
2. Structure: Does the content parse as JSON and match our schema? Parse and validate it locally.
3. Task correctness: Does the prediction refer to the expected claim, and does its label follow from the supplied evidence? A valid structure doesn't establish either of these.

# After the complete example's check of the completion status:
prediction = Prediction.model_validate_json(message.content or "")
if prediction.example_id != expected_id:
    raise ValueError("Prediction refers to the wrong example.")

For the claim “The coffee shipment arrived on Friday,” the expected label is UNKNOWN. A response of SUPPORTED would satisfy the schema but fail the task. Don't quietly replace invalid or incomplete responses with UNKNOWN: that would turn an application failure into an apparently meaningful prediction.

A real example: extracting information from interviews

The Archives are invented, but the distinction between structure and evidence matters in real applications. Kwak et al. [2025] describe extracting farm information from interview transcripts. Their LLM approach uses Pydantic data classes, JSON schemas, and constrained decoding, followed by separate field validation and checks against the source text.

Those checks still leave room for errors. Their string filter rejects values that share no words with the source, but even one shared word can let an unsupported extraction through. In one case, invented crop rotation information passed because it contained the word “years,” which also appeared in the transcript. The filter also didn't cover boolean or integer values.

The error analysis gives us a concrete reason to keep asking both questions: Does this value have the right type? And does the source support it?

Run the complete example

Clone or download the example repository, then open its folder. Install the dependencies in your configured Python environment.¹ Set GATEWAY_BASE_URL, GATEWAY_MODEL, and GATEWAY_API_KEY using the course settings and your own key (keep the key out of Git).

python -m pip install -e .
python gateway_lab.py classify --output classification.json

This command makes a model request. The output file records the request and response, including our fictional record, but not your API key. Depending on the project's settings, the model-gateway service may also log that content. Open classification.json and look at results.classify for the validated prediction and trace for the requests and responses. The fixed example is deliberately small. It is a compatibility example, not a benchmark.

Give the assistant evidence

Where was Zarbon the Great born? The model wasn't invited to the worldbuilding meeting. It might recognize the name from elsewhere or make a plausible guess. We'll move from one purchase order to a small collection about imperial history. Our supplied records establish this exercise's “facts.” We can inspect whether an answer follows from them without assuming what the model already knows about a name like Zarbon.

First, just provide the text

[great-birth] Zarbon the Great was born on the generation ship Aslan.
[great-conquest] Zarbon the Great conquered Alpha Centauri in 3125 (standard time).
[timid-birth] Zarbon the Timid was born on a mining station in the Kuiper belt.
[timid-life] Zarbon the Timid died in 4078.

Ask the model where Zarbon the Great was born, using only these records. Now ask where Zarbon the Great died. The records support the first answer and leave the second unspecified. A source ID attached to an invented answer would not make it supported, so be sure to check the cited record.

These facts remain in the request, not in newly trained parameters. For this tiny collection, supplying all four records is a reasonable baseline. Retrieval becomes useful when we need to select information from a larger collection.

When two departments disagree

For another challenge, add these notices from two imperial departments:

[earth-access-a] Office of Planetary Admissions, notice 17: Earth's archive access is provisional.
[earth-access-b] Records Office, notice 17: Earth's archive access is full.

Both notices concern the same access status, and neither gives a rule for which office takes precedence. An answer should report the conflict and cite both records, rather than quietly choosing the more welcoming department. Adding two sources doesn't help if the answer hides their disagreement.

Retrieve before generating

Retrieval-augmented generation (RAG) places retrieved material into the model's input. The retrieval can be lexical, embedding-based, or a combination. With a very large knowledge base to draw from, we might consider adopting a vector database. For this tutorial's scale, though, we can get by without it.

The repository's retrieve function scores each record by the number of distinct words it shares with the question. It uses stable record IDs to break ties, then selects up to two positive-scoring records. This intentionally simple baseline lets us inspect the whole selection process.

from gateway_lab import retrieve

hits = retrieve("Where was Zarbon the Great born?", k=2)
for hit in hits:
    # Keep each source ID beside the text it identifies.
    print(f"[{hit['id']}] {hit['text']}")

No gateway request happens in that block. Next, send those hits with the question:

python gateway_lab.py retrieve --output retrieval.json

For a complete, runnable example, see grounded_answer in gateway_lab.py. In retrieval.json, inspect results.retrieve.hits before judging results.retrieve.answer. The prompt asks for a supporting record ID and an acknowledgment when the records don't establish the answer.

What would embeddings change?

A colleague searches for “vacation,” but your document calls it “annual leave.” A search based only on shared words may miss the relevant passage.

Embedding retrieval maps documents and queries to vectors and compares them using a chosen similarity measure. The embedding model, its query/document formatting, and its training task all matter.

Don't mix vectors from different embedding models in one index and assume their coordinates are comparable. Also distinguish the embedding model from the model generating the final answer.

Retrieve examples as well as evidence

In the previous lesson, we used the same demonstrations for every claim. What if we had a larger collection of labeled examples? Some might be more useful for the current case than others. We could retrieve those examples and include them in the prompt, rather than always using a fixed set.

Consider a claim about whether a promised delivery actually arrived. An example that distinguishes a promise from a confirmed delivery might help more than one that happens to mention coffee. Similarity in wording or topic is one criterion. We can also select by the kind of decision, relevant metadata, or contrasting cases that clarify a difficult distinction.

Embeddings can help select either kind of material, but the selection criteria may differ. The demonstrations show how to approach the task, while the source passages establish what we can say about the current case. Both become part of the model's context, without updating its parameters.

We can test this by comparing fixed demonstrations with demonstrations selected for each input. For this test, keep the source passages, model, and generation settings the same. Retrieve from a separate example bank, keeping evaluation cases and their reference answers out of that bank. Inspect what was selected and whether it helped. The nearest examples aren't guaranteed to be the most useful ones.

For a concrete study of this approach, see Liu et al.'s What Makes Good In-Context Examples for GPT-3?.

Chunking: making things fit

A book generally won't fit in one embedding input. We split it into chunks (for example, overlapping paragraphs, sentences, or chapters), but the split changes what can be retrieved. Cutting an entity name away from the fact about it can make a passage difficult to interpret. Very large chunks can bury the answer in unrelated text. Keep source IDs and offsets so you can inspect the original context and evaluate these choices.

For our four facts, each record is already a convenient chunk. Don't add a chunking framework until there's something useful to split.

Optional: add a vector database

A system such as Chroma can store vectors and metadata and search a larger collection. If you're already using PostgreSQL, pgvector adds vector similarity search to that database. Neither chooses the right embedding model for our task or establishes that an answer is supported. Before moving the original example to an embedding route, verify course permission, input limits, vector dimensions, and the model's query instructions. Preserve stable IDs instead of using Python's hash(text), whose values can change between processes.

The next experiment is to compare the simple retriever with an embedding retriever on questions with known relevant records. Keep the corpus and evaluation questions fixed.

Evaluate the two stages

A user reports an incorrect answer. Before changing the prompt, we need to know whether the application supplied the right evidence and whether the model used it correctly.

Retrieved text is data, not automatically trusted instructions. A malicious passage may tell the application to ignore its task (prompt injection, which we can think of as an LLM's version of SQL injection). Supplying more text can add both useful evidence and new failure cases. Retrieval is not a truth filter.

We now know how to retrieve evidence, but what changes when the model can request an operation? We'll follow that request through local execution in Giving language models tools.

Check your understanding

For optional practice, try a few of these on your own or with a classmate.

1. How could a response satisfy the schema but fail to answer our question?
2. If the answer cites a real record ID, what else should we check?
3. What should the assistant do when the two Earth access notices conflict?

Resources

• Pydantic: JSON Schema and strict mode.
• Model Gateway basics.

Retrieval references

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, presents a learned retrieval/generation approach. Our tiny application illustrates the broader retrieve-then-generate pattern, not that paper's training procedure.
• Liu et al. (2022), What Makes Good In-Context Examples for GPT-3?, investigates selecting demonstrations by semantic similarity to the current input.
• Chroma and its documentation, for optional database work.
• pgvector, for vector similarity search in PostgreSQL.
• Return to the LLM unit.

References