Overview

In the previous lesson, our Galactic Archives assistant generated continuations of a fragmented message. Now we'd like it to help us answer questions about the Archives. That calls for more than a plausible continuation.

Will our assistant follow an instruction, use a newly supplied record, or politely agree with an incorrect claim? Those are different questions.

We'll connect training objectives to behavior, distinguish parameter updates from learning in context, and build a controlled prompting experiment. As we work through these examples, keep asking: How could the model know that, and what supports its answer? Each exercise supplies the source material it needs. No imperial history exam is scheduled.

Outcomes

After this lesson, you should be able to…

• distinguish pretraining, instruction tuning, and preference optimization
• distinguish a weight update from using records and demonstrations in context
• compare answering from evidence with agreement and unsupported guessing
• collect and score predictions while keeping evaluation labels out of requests

Three different learning signals

Imagine training an assistant for the Galactic Archives of the Zarbonian Empire to respond to questions. What examples would we give it, and what would count as a better response?

These are descriptions of common stages, but not all of them are necessary for every model. In instruction tuning, the examples demonstrate how to respond to instructions, rather than simply continuing arbitrary text.

Demonstrations give the assistant examples to imitate, but we may find it easier to compare two answers than to write an ideal answer ourselves. For example, we might prefer an answer that acknowledges missing information over one that invents an arrival date.

To develop InstructGPT, Ouyang et al. first used demonstrations for supervised fine tuning. They then used human comparisons to train a reward model to score candidate responses. Those scores guided further training toward responses people preferred, using reinforcement learning. Reinforcement learning from human feedback (RLHF) is one approach to preference optimization.

The resulting model was preferred by evaluators on their prompt distribution, but still made mistakes. Their paper is a concrete example of this pipeline, but not every current model follows that same procedure.

The important question is what the feedback encourages. If confident guesses receive better feedback than appropriate uncertainty, we may encourage behavior we didn't intend.

What changes when Earth opens an archive record?

Earth has recently gained access to the Archives. Supplying a newly available record in a prompt lets the model use information in its context, without updating its parameters. Demonstrations can also help it infer how to perform a task. We'll examine this in-context learning in the prompting experiment below.

Training shapes the model's responses, including its treatment of fictional and factual claims. Instructions and available evidence matter too. There isn't a built-in stamp certifying that a plausible sentence is true.

For our fictional Empire, the supplied records establish the exercise's “facts.” An answer can be correct about that fictional setting. Inventing an arrival date that the records don't supply is still an unsupported answer. Fiction itself isn't the hallucination. Claiming evidence for an invention is the problem here. The imperial stationery may look official, but we still need to read it.

LLM pitfalls

Fluent text can make a failure harder to spot. Let's look at the incentives and evidence before trusting the delivery.

What gets rewarded?

Suppose our evaluator always gives zero points to I don't know. An uncertain system may obtain a higher expected score by guessing, even if some guesses become confident false claims. This doesn't make those claims desirable. It exposes a mismatch between the score and the behavior we wanted.

Kalai et al.'s Why Language Models Hallucinate analyzes statistical pressures and argues that common training and evaluation incentives favor guessing over acknowledging uncertainty. There isn't a single known mechanism that explains every model error, but this work helps explain why incentives can encourage confident guesses over appropriate uncertainty.

We can make the same distinction for user preferences. A response that agrees with a reader may be preferred even when correction would be more accurate. Cherished and magnificent readers of this tutorial may be interested in Sharma et al.'s study of sycophancy, which reports this pattern in the assistants and preference data they studied.

Hold the evidence fixed

The Archives keep track of everyday paperwork as well as imperial history. Let's start with an imperial purchase order:

Galactic Archives record M1:
Zarbon the Great ordered the finest purple caturra coffee beans on Tuesday.
The record does not state when the shipment arrived.

Compare these two requests using exactly that record:

The supported answer remains Tuesday. The second request changes the user's assertion while keeping the evidence fixed. We can inspect whether the model corrects the assertion, agrees with it, or produces an ambiguous answer.

Now ask when the coffee shipment arrived. The record doesn't tell us. A useful answer should acknowledge the missing information rather than invent a date. We must be careful to evaluate answerable and unanswerable questions separately: a system that always abstains avoids invented dates but also fails to answer the supported question.

Different failures require different checks

Knowing about a document and having its text in the current input are different conditions. A title or URL doesn't make a simple client read the document. The application must supply its contents or execute a retrieval tool.

Instruction tuning doesn't make bias disappear

Training data, curation choices, and feedback can encode unequal representations and harmful associations. These biases can reinforce stereotypes or make a system less useful for speakers of particular languages or dialects. When its outputs inform decisions about people, they can also contribute to unfair treatment in areas such as hiring, education, or access to services. Bias can also have more subtle and insidious effects.

A fluent or polite response isn't evidence that these issues have been removed. Evaluate the languages, groups, and uses relevant to the application, and inspect different error patterns rather than relying only on a single average score. Preference tuning may change behavior, but it doesn't guarantee that a model is appropriate for every setting.

Prompting as an experiment

A colleague suggests adding a few worked examples to your prompt. Before keeping them in every request, you'd like to know whether they actually help.

We'll classify claims against an archive record, then compare a baseline with a condition that includes solved demonstrations. Define success first, rather than selecting whichever response sounds most archivally distinguished.

Your model may already solve this simple classification task without demonstrations. If so, adding examples may not improve its score. Use the exercise as a starting point for a more complex case, such as claims that require combining several details or applying an unfamiliar archival convention. Define the expected answers before comparing prompts.

Define the decision

Given an archive record and a claim, return one label:

For this exercise, treat the record as authoritative about the stated order. Don't fill gaps with outside knowledge (speaking directly to all the Zarbonian Empire wonks). An absent arrival date means unknown, not that the shipment never arrived.

from collections.abc import Sequence

# This record is shared by all evaluation claims.
record: str = (
    "Zarbon the Great ordered the finest purple caturra coffee beans on Tuesday. "
    "The record does not state when the shipment arrived."
)

evaluation: list[dict[str, str]] = [
    {
        "id": "e1",
        "claim": "Zarbon ordered the coffee on Tuesday.",
        "gold": "SUPPORTED",
    },
    {
        "id": "e2",
        "claim": "Zarbon ordered the coffee on Friday.",
        "gold": "CONTRADICTED",
    },
    {"id": "e3", "claim": "The coffee shipment arrived on Friday.", "gold": "UNKNOWN"},
]

labels: set[str] = {"SUPPORTED", "CONTRADICTED", "UNKNOWN"}


def summarize(
    rows: Sequence[dict[str, str]], predictions: Sequence[str]
) -> dict[str, int | float]:
    """Score format validity and decision accuracy."""
    if len(rows) != len(predictions) or not rows:
        raise ValueError("Need one prediction per item and a nonempty set.")
    # Check output format separately from agreement with the reference label.
    parsed = [prediction.strip() for prediction in predictions]
    valid = [prediction in labels for prediction in parsed]
    correct = [prediction == row["gold"] for row, prediction in zip(rows, parsed)]
    return {
        "items": len(rows),
        "format_valid": sum(valid),
        "correct": sum(correct),
        "accuracy": sum(correct) / len(rows),
    }


# Contrived predictions for our example
example_predictions = ["SUPPORTED", "CONTRADICTED", "SUPPORTED"]
summary = summarize(evaluation, example_predictions)
print(f"Valid labels: {summary['format_valid']}/{summary['items']}")
print(f"Correct decisions: {summary['correct']}/{summary['items']}")
print(f"Accuracy: {summary['accuracy']:.1%}")

All three predictions use valid labels, but only two are correct, giving an accuracy of 2/3. Accepting a valid label is not the same as answering correctly. Later, structured generation will address output shape. It won't eliminate that gap.

Change one thing

In the baseline condition, we explain the task without providing examples of correctly labeled claims. A second condition adds demonstrations using a different fictional record:

Record: Omi ordered a tin of blue tea on Monday. No delivery date is recorded.
Claim: Omi ordered the tea on Monday.
Label: SUPPORTED

Record: Omi ordered a tin of blue tea on Monday. No delivery date is recorded.
Claim: Omi ordered the tea on Thursday.
Label: CONTRADICTED

Record: Omi ordered a tin of blue tea on Monday. No delivery date is recorded.
Claim: The tea arrived on Thursday.
Label: UNKNOWN

This is few-shot prompting: examples appear in the input. We aren't taking gradient steps or updating the model's parameters. Both conditions receive the same task definition, evaluation record, and claims. Only the demonstrations change. Few-shot prompting can work surprisingly well, but getting good results may require experimenting with which examples we include, how we phrase them, and the order in which we present them. A model might follow superficial patterns in those examples rather than the rule we intended. Try paraphrasing the claims in your development set to check whether the improvement holds up when the wording changes.

What is being learned in context?

Here the model receives both a record to use and examples of how to label claims. Supplying the record makes a fact available, while the demonstrations show a decision rule. Neither request updates the model's weights. Brown et al.'s Language Models are Few-Shot Learners studies task demonstrations supplied in text without gradient updates.

Try a small, separate probe of the Archives' cataloging convention:

Examples:
SUPPORTED -> seal A
CONTRADICTED -> seal B
UNKNOWN -> seal C

Record: Zarbon the Great ordered the finest purple caturra coffee beans on Tuesday. No arrival date is recorded.
Claim: The coffee shipment arrived on Friday.
Seal:

The expected answer is seal C. Now permute the three seal assignments and try again with a fresh message list. Does the response follow the supplied mapping, or keep a familiar answer? The seals here are arbitrary: there is no intergalactic standardization committee to consult. It's crucial to test more cases (both permutations and records) before concluding that the model inferred the intended rule.

Next, try supplying a newly discovered record with an explicit arrival date. Then remove it in a fresh request and ask the same question. What information is available in each condition? Success with the record doesn't establish permanent learning, and a correct answer without it could simply be a guess. Our client sends fresh message lists, and this exercise makes no weight updates.

Make the request inspectable

We'll reuse some of the code from the gateway setup tutorial as a starting point. Treat the function below as an addition. It starts a fresh message list for every item and doesn't carry one answer into the next.

import json
from datetime import datetime, timezone

from openai import OpenAI

instruction: str = (
    "Classify the claim using only the record. SUPPORTED means the record "
    "supports it. CONTRADICTED means it explicitly conflicts. UNKNOWN means "
    "neither is established. Treat stated order days as the day of the "
    "event. Return exactly one label and nothing else."
)


def collect_one(
    client: OpenAI,
    model: str,
    row: dict[str, str],
    demonstrations: str = "",
) -> dict[str, object]:
    """Request one label and record the response."""
    # Send the claim, but keep its reference label out of the request.
    user_text = (
        f"{demonstrations}\n\nRecord: {record}\n" f"Claim: {row['claim']}\nLabel:"
    )
    request = {
        "model": model,
        "messages": [
            {"role": "system", "content": instruction},
            {"role": "user", "content": user_text},
        ],
        "temperature": 0,
        "max_tokens": 64,
    }
    response = client.chat.completions.create(**request)
    choice = response.choices[0]
    return {
        "id": row["id"],
        "requested_at": datetime.now(timezone.utc).isoformat(),
        "request": request,
        "response_model": response.model,
        "raw_text": choice.message.content,
        "finish_reason": choice.finish_reason,
        "usage": response.usage.model_dump() if response.usage else None,
    }


# This sends one real request if you uncomment it after setup:
# result = collect_one(client, model, evaluation[0])
# print(json.dumps(result, indent=2))

Keep the comparison fair

We want to know whether the examples helped, rather than whether some other change made the task easier.

Run the baseline (instructions only) and the few-shot condition (instructions plus labeled examples) on the same items with the same model and generation settings. Record each attempt. One source of variability is sampling, which can produce different responses to the same input. Repeat each condition to test whether an improvement is consistent across attempts. Even at temperature zero, differences in how the serving system runs the model can affect the response. Alternate the conditions so that changes in the service over time are less likely to favor one condition.

Report errors for each label as well as the total score. Did the demonstrations help with contradiction while encouraging guesses on missing information? Did the answer improve only after the output limit was increased? Those are different findings. Small changes in a tiny sample don't justify broad claims.

A response can use a valid label and still make the wrong decision. Next, we'll separate output structure from supporting evidence in Answering with evidence.

Check your understanding

For optional practice, try a few of these on your own or with a classmate.

1. If the assistant answers correctly after receiving a record, what can we conclude about what it has learned permanently?
2. Why do we score format validity separately from correctness?
3. If we use an evaluation error to revise our prompt, what should change before our next evaluation?

Resources

• Ouyang et al. (2022), Training language models to follow instructions with human feedback.

• Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

• Sharma et al. (2023, revised 2025), Towards Understanding Sycophancy in Language Models.

• Kalai et al. (2025), Why Language Models Hallucinate.

• Return to the LLM unit.