Overview

Suppose you ask a model about a paper you've just read. It gives you a confident, detailed answer. Has it actually read the paper? What changes if you supply the text yourself?

That's what we'll investigate in Part 1 of the paper review activity. We're not trying to coax out the best possible summary. We want to see what the model does with different inputs, check its claims, and make sense of what we find.

What can a citation tell the model?

A title or citation gives the model something to go on. It may have encountered related information during training^†, and it can also produce a plausible answer without knowing the paper's details. A fluent answer doesn't demonstrate that the model accessed the paper. Our simple API client doesn't browse, download a URL, or open a PDF just because we mention it.

We'll compare two conditions:

• Citation only: send the citation and your questions, without the paper text.
• Text provided: start fresh with the same citation and questions, now adding the paper text you selected.

The same model, the same questions, but different information to work with. Neither request needs tools.

How does the paper reach the model?

Remember the distinction between a model and the application around it? Our request contains the strings in messages, not the files sitting on our computer. Something has to read the paper and put its content into those messages. Here, we'll do that preparation ourselves.

Copy or extract the paper's text into a UTF-8 text file, then compare it with the original. This is worth taking a moment to check. Imagine a two-column page whose extracted text jumps between columns, or a results table whose numbers have lost their row labels. The model would receive that jumble, not the page you see in the PDF.

A local script that reads a text file before the request is simply preparing the input. The model isn't choosing to call a file-reading tool. We'll save that additional machinery for Part 2.

Same questions, different inputs

We'll build on the client and model variables from the gateway tutorial. Keep using that setup in the same Python session or script. First we'll assemble the citation-only input. Then we'll add the paper text to a separate input.

Start with the citation

Replace the placeholders with your verified citation and the questions you prepared for the activity. We'll number the questions once so that we can reuse the same wording in both conditions:

citation = "REPLACE_WITH_YOUR_VERIFIED_CITATION_AND_VERSION"
questions = [
    "REPLACE_WITH_YOUR_FIRST_QUESTION",
    "REPLACE_WITH_YOUR_SECOND_QUESTION",
    "REPLACE_WITH_YOUR_THIRD_QUESTION",
]
question_text = "\n".join(
    f"{number}. {question}" for number, question in enumerate(questions, start=1)
)

# Condition A includes the citation, but not the paper text.
citation_messages = [{
    "role": "user",
    "content": f"Paper: {citation}\n\nQuestions:\n{question_text}",
}]

We haven't sent anything yet. citation_messages is just the list we'll give the client when we're ready to make the request.

Now add the paper text

Put your inspected text in paper.txt beside the script, or adjust the path to where you actually saved it. This time, our message will include the citation, the paper text, and the same questions:

from pathlib import Path

# Inspect this file before sending it. No PDF extraction happens in this line.
paper_text = Path("paper.txt").read_text(encoding="utf-8")
if not paper_text.strip():
    raise ValueError("The paper text is empty. Check the file before continuing.")

# Condition B starts fresh. It does not include Condition A's answer.
text_messages = [{
    "role": "user",
    "content": (
        f"Paper: {citation}\n\n"
        f"Provided paper text:\n{paper_text}\n\n"
        f"Questions:\n{question_text}"
    ),
}]

What makes this a fresh conversation? The message list determines the history supplied in this workflow. We're reusing client, but not sending Condition A's answer along with Condition B's input. Creating another OpenAI client isn't necessary to keep those histories separate.

Save before you revise

Suppose the model invents a result. It's tempting to correct it straight away, but then we'd lose track of what it said before we stepped in. Let's keep the initial request and answer before trying to improve anything.

The helper below sends one condition at a time and saves the exact messages and initial response in a JSON file. You don't need to memorize the bookkeeping. The important pieces are what we sent, which model and settings we used, and what came back. Choose a new output filename for each attempt so that later repairs don't overwrite the initial evidence.

import json
from datetime import datetime, timezone

settings = {"max_tokens": 1024, "stream": False}

def send_and_save(messages, output_path):
    # Reserve a new file before sending, rather than overwrite an earlier run.
    with Path(output_path).open("x", encoding="utf-8") as output:
        started_at = datetime.now(timezone.utc).isoformat()
        try:
            response = client.chat.completions.create(
                model=model,
                messages=messages,
                **settings,
            )
        except Exception as error:
            record = {
                "started_at": started_at,
                "model": model,
                "settings": settings,
                "messages": messages,
                "error_type": type(error).__name__,
            }
            json.dump(record, output, ensure_ascii=False, indent=2)
            raise

        record = {
            "started_at": started_at,
            "model": model,
            "settings": settings,
            "messages": messages,
            "response": response.model_dump(mode="json"),
        }
        json.dump(record, output, ensure_ascii=False, indent=2)

    choice = response.choices[0]
    print("Finish reason:", choice.finish_reason)
    print(choice.message.content)

Now send the citation-only input and inspect the answer before continuing:

send_and_save(citation_messages, "condition-a-initial.json")

Once you've kept that initial answer, send the text-provided input:

send_and_save(text_messages, "condition-b-initial.json")

These files give us a record to inspect, not a replacement for CM's submission instructions. The helper opens each file with "x", which refuses to overwrite an existing file. If that filename is already taken, choose a new one rather than discard the earlier attempt.

Did we get a complete answer? Check the finish reason. The output cap of 1024 is an example, not an assignment requirement or a guarantee that the model can finish within that limit. If the response ends with length, keep it as part of your initial record and document any later attempt separately. Check the model's limits, and record available revision information from its gateway card too. A returned model name doesn't necessarily identify an exact weights revision.

If the request fails, the helper saves the attempted input and error type before raising the error again. Keep the request ID and safe diagnostic information too, if available. A blocked request is a reason to ask for help, not to invent a response for the record.

What if the paper doesn't fit?

We might have a perfectly readable text file that is still too long for the model's context. That context has to accommodate the input and generated output. Character counts aren't token counts, and the appropriate tokenizer depends on the model. Don't silently slice the paper until the request succeeds. We'd be changing the evidence available to the model without keeping track of it.

Count before you cut

Fortunately, we don't have to guess from the number of characters. When enabled for your project and assigned model, the gateway's Tokenizer view lets you count tokens and inspect how the text is split into token IDs and pieces.

To check the text-provided input from our example:

1. Open your course project in the gateway and choose Tokenizer > Tokenize.
2. Select the assigned model you're using for both conditions.
3. Paste the full message into Input, including the citation and questions, not just the paper. In our example, that's text_messages[0]["content"].
4. Choose User chat message as the input format and leave special tokens at their serving defaults. That includes the serving model's chat formatting, rather than counting only the plain text.
5. Set Reserve output tokens to the output cap you plan to use (1024 in our example), then select Tokenize.

The result shows the input token count and how much context remains after that output reservation. A plain-text count of the paper alone would miss the citation, questions, and chat formatting. Count what you actually intend to send. If you add conversation history later, that has to be counted too.

For this activity, we're checking what fits the assigned model, not choosing between models. The result can help us decide whether to send the whole paper or work section by section with that same model.

This is a useful budget check, but not a guarantee that generation will succeed. The tokenizer utility also has its own input limits. If it rejects a long input, or tokenization isn't available for your assigned model, ask for help rather than quietly switch models or trim away evidence.

Work with sections when needed

For Part 1, we can supply the text all at once or work section by section, following the instructions in CM. If you choose sections, be explicit about the context supplied for each request. A fresh message containing only Results doesn't also contain Methods from your previous request. The model doesn't automatically carry that earlier exchange forward.

You can include selected earlier material or explicit conversation history, but those take up context too. Record what you sent and what you left out. Replacing omitted sections with an LLM-generated summary would change the input again. Don't let that change disappear from your record.

Was the answer supported?

Now we have something to compare. Pick a concrete claim and look for its support in the paper. Can you point to the section, table, or figure that supports it? Does the model's wording go further than that evidence allows?

Try to separate what happened from what you suspect caused it. A scrambled table and an incorrect reported number can occur together, but that alone doesn't establish that the extraction problem caused this particular error. Record both observations and be clear about what remains uncertain.

If the answers are accurate, that's something to investigate too. You don't need to manufacture errors to have a useful comparison. Which claims can you verify? What changed when you supplied the text? What would you still check before relying on an answer?

Use CM's template and rubric to organize what you find. The model's answers are part of the evidence, not a substitute for your own interpretation and reflection.

What will tools change in Part 2?

So far, we've decided which text to send before making a request. What if the model could request an operation to get more information? In Part 2, we'll use the same assigned model with tools. As we saw in the gateway tutorial, the model proposes an operation, the client decides whether to execute it, and the client returns the result in a later request.

We'll examine what that additional machinery changes. For now, keep Part 1 free of model-selected tools, automatic retrieval, and agent loops. Let's first understand what happens when we supply the information ourselves.