Overview

Large language models (LLMs) get the "large" part of their name from their scale, including billions of parameters (the numerical values learned during training). Those values have to live somewhere! For example, a model with 70 billion parameters stored at 16 bits each needs about 140 GB of memory just for its parameters, before we account for the working memory needed to generate a response. That's far more than a typical laptop can muster.¹

There's also the work of computing each response. GPUs are useful here because they can perform many of the model's calculations in parallel. Shared servers with this hardware let us use demanding models without each person needing their own machine capable of running them. Smaller models and quantized models (which store weights using fewer bits) can make local use practical too.

Now imagine wanting to experiment with several models. We need somewhere to store their weights, enough memory for whichever ones we keep loaded, and the computing power to run them. We don't have to keep them all loaded at once, but switching between them can mean more downloads, setup, and waiting. Sharing model servers can spare each person from repeating all of that work.

Suppose we want to use several models, each running on a different machine. Without a shared entry point, our code may need a different server address and access setup for each one. Moving a model to another machine could mean changing the clients that use it. How can we keep those details out of each client's code?

A model gateway sits between clients and model servers. Clients send requests to the gateway, which checks access and routes each request to the selected model. We can use one interface without having to track where each model runs. That doesn't make the models interchangeable, but it simplifies how we reach them.

You don't need to specify which model host you're targeting. Just tell the gateway which model you want to "talk" with by using its model ID. The gateway handles routing your request to the server hosting that model.

Outcomes

After this tutorial, you should be able to…

• distinguish a model, a gateway, and the client or harness around a model
• find your course project and use your own API key without exposing it
• list available models and make a small OpenAI-compatible request
• count a planned chat input and check space for its requested output
• recognize what document access, conversation history, and tools require
• inspect completion status and ask for help with access or request limits

A shared API interface

How does our code "talk" with a model? An API (application programming interface) gives one program a way to communicate with another. Instead of typing into a chat window, we can send messages from Python and work with the response in our code.

OpenAI's API has become a widely adopted reference for this kind of interface. Other services (ex. Hugging Face's Text Generation Inference) offer OpenAI-compatible APIs, allowing us to use familiar request formats and client libraries with models served elsewhere. Think of it as a shared convention, rather than a promise that every service supports every feature in exactly the same way.

The model is not the whole application

A chat application often involves all sorts of things like assembling prompts, extracting files, managing history, and executing tools. All of that affects what information reaches the model. It helps to distinguish three pieces:

An LLM ("the model") doesn't magically do all of those things. They involve an interplay between the model's outputs (including proposed decisions or actions) and the rest of the application. The application controls which actions actually happen and what information gets sent back to the model.

If we want the model to work with a document or an earlier conversation, something has to supply that information. For example, putting a title, URL, or local filename in a prompt doesn't make our simple Python client open the document and send its contents.

Tools

What would change if we wanted the model to use a tool^†? We could describe an available function in a request and let the model propose calling it. Our client decides whether to execute the requested operation and, if so, sends the result back to the model:

graph TD
  C["Your client"] -->|messages| G["Gateway: access and routing"]
  G -->|input| M["Model"]
  M -->|answer or tool call| G
  G -->|response| C
  C -->|execute allowed call| T["Your tools"]
  T -->|result for next request| C

The gateway connects the client to the model, but it does not execute our local tools. The application decides which tools to offer and which proposed calls to carry out. Asking for an operation isn't the same as having executed it.

We'll explore that loop separately. We can start with ordinary messages, without adding tool execution to the mix.

Our course gateway

With those distinctions in mind, let's connect to the gateway we'll be using in the course. It organizes access into projects. Your project determines which models you can use, and your own project key lets your Python client make requests. The models run elsewhere, so you don't need to download them or run them on your laptop.

Before you start

Use your configured Python environment and your own course account. You'll find the gateway address, assigned project, and model identifier in Class Maestro (CM). We'll use those values below, rather than hard-code a particular model into the tutorial.

Find your project and create a key

Follow the gateway link in CM and sign in with your course account. Open your assigned project. Don't see it? Ask for help. Borrowing someone else's key or choosing an unrelated project won't fix your course access.

Open the project's Keys tab and create your own key. Signing in through your browser gets you into the gateway's interface. The key is what lets your code make API requests.

Set up the Python client

We'll use the openai Python package to handle the requests. If it isn't already installed, install it in your existing course environment:

python -m pip install openai

Our client needs three things: where to send requests, which model to use, and our key. We'll keep these in environment variables, which our Python code can read without putting the key into the script itself.

Replace the two placeholders below with the API URL ending in /v1 and the assigned model ID from CM. Python's getpass asks for your key without showing it as you type or putting it into shell history. This example works in Bash and Bourne-style shells:

export GATEWAY_BASE_URL='PASTE_THE_ASSIGNED_API_URL_ENDING_IN_/v1'
export GATEWAY_MODEL='PASTE_THE_ASSIGNED_MODEL_ID'

GATEWAY_API_KEY=$(python -c 'import getpass; print(getpass.getpass("Gateway key: "))')
export GATEWAY_API_KEY

Run Python from that same terminal so it inherits these variables. If you're using a notebook, there's one more detail to keep in mind:

Make a small request

Before sending a long document, let's check that we can make a small request. We'll connect to the gateway, check that our assigned model is available, and ask a short question:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["GATEWAY_BASE_URL"],
    api_key=os.environ["GATEWAY_API_KEY"],
    timeout=120,
    max_retries=0,
)

# Check access to the assigned model before sending a request.
model = os.environ["GATEWAY_MODEL"]
available = {item.id for item in client.models.list().data}
if model not in available:
    raise RuntimeError("The assigned model is not available to this key.")

response = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Explain a held-out evaluation in two sentences."}],
    max_tokens=256,
    stream=False,
)
choice = response.choices[0]
print("Finish reason:", choice.finish_reason)
print(choice.message.content)

Notice how the earlier distinctions show up in the code. base_url tells the client where to send the request, api_key authorizes it, and model says which model we want to talk with. Changing the model ID doesn't require us to find that model's server address.

The gateway takes care of access checks and routing, but we still have choices to make about the input and output. Use the model assigned in CM, even if your key permits other models. Read its card to see its limits and the available evidence about what it supports. When documenting a run, record the model ID and any available version information.

What did we actually send? The messages list contains our conversation. Each message has a role and content. Here, there's just one user message. The model's answer comes back as an assistant message. Calling it an assistant doesn't make its answer authoritative, of course!

We use the SDK's Chat Completions interface because our gateway supports that interface. OpenAI's own service and our gateway need not support every API feature or model setting in the same way.

There are no tools in this request, and no earlier conversation is included. If we make another request, the client doesn't automatically add this exchange to it. We have to supply any history we want the model to receive.

What about max_tokens=256? That's a cap on the generated output, not the size of the input we can send. Check the printed finish reason too. A value of length tells us the response reached a length limit, so we should check whether it was cut short rather than assume we received a complete answer.

If you omit max_tokens, the gateway currently defaults to 2,048 output tokens. Set it explicitly when comparing runs so the intended budget is clear. The model may stop before reaching that cap.

Count tokens before a longer request

A model's context window must accommodate both the input and generated output. Our messages, conversation history, and chat formatting take up space before the model starts answering. max_tokens caps the output, but it doesn't enlarge the context window.

Let's imagine a model has a 4,096-token context window. If our messages use 3,000 tokens and we ask for up to 1,500 output tokens, the planned total is 4,500 tokens. That exceeds 4,096. We either need a smaller output budget or a different, explicitly chosen input. These numbers illustrate the calculation. Check the limits for your assigned model.

The gateway also adds token_limits to each chat model in /v1/models. Inspect the assigned model's entry to find the gateway output ceiling, default budget, and context window evidence:

assigned = next(item for item in client.models.list().data if item.id == model)
print(assigned.model_dump()["token_limits"])

These additional fields extend the OpenAI-compatible response with gateway-specific token limits. OpenAI's own API does not provide these fields.

A context value of null means unknown. Its status can also mark a stale observation or configured fallback. The gateway's output ceiling is separate from the model's total context, and may be higher than the model can actually use for a given request.

Despite its name, reserve_output_tokens doesn't reserve anything on the server. It tells the tokenizer how much planned output to include in this calculation:

remaining_tokens = context_limit - count - reserve_output_tokens

Here, count is the number of input tokens. fits is true when the result is zero or greater. In the example above, the result is -404 and fits is false. The counting call generates no answer and doesn't change a later generation request.

Count and generate with the same output limit

Tokenization must be enabled for your project and supported by the assigned model. If the Tokenizer view is missing or access is denied, ask your instructor. It isn't necessary to create another key: these requests use the same project bearer key as generation.

The example below uses the environment variables and client from the earlier example. We use httpx, an HTTP library installed with the OpenAI client, to call the gateway's own utility endpoints. Those endpoints start with /api/v1, whereas the OpenAI-compatible generation endpoint starts with /v1.

import httpx

# GATEWAY_BASE_URL ends in /v1; these utilities use /api/v1 instead.
gateway_url = os.environ["GATEWAY_BASE_URL"].rstrip("/").removesuffix("/v1")
headers = {"Authorization": "Bearer " + os.environ["GATEWAY_API_KEY"]}

messages = [
    {"role": "user", "content": "Explain a held-out evaluation in two sentences."}
]
output_budget = 1024

with httpx.Client(base_url=gateway_url, headers=headers, timeout=120) as gateway:
    discovery = gateway.get("/api/v1/tokenizer/models")
    discovery.raise_for_status()
    supported = any(
        item["id"] == model and item["capabilities"]["tokenize"]
        for item in discovery.json()["data"]
    )
    if not supported:
        raise RuntimeError("Ask your instructor about tokenization for this model.")

    counted = gateway.post(
        "/api/v1/tokenize",
        json={
            "model": model,
            "messages": messages,
            "reserve_output_tokens": output_budget,
        },
    )
    counted.raise_for_status()
    budget = counted.json()

print("Input tokens:", budget["count"])
print("Total context limit:", budget["context_limit"])
print("Gateway output ceiling:", budget["gateway_output_cap"])
print("Space left after planned input and output:", budget["remaining_tokens"])
print("Fits:", budget["fits"])
if not budget["fits"]:
    raise RuntimeError("Revise the input or output budget before generating.")

# This is a separate generation request. Repeat the output budget explicitly.
response = client.chat.completions.create(
    model=model,
    messages=messages,
    max_tokens=output_budget,
    stream=False,
)
print(response.choices[0].message.content)

Use the same model and full message list for both calls. Include any conversation history and, when using tools, the same tool definitions. Counting just a paper's text would miss the instructions, questions, and chat formatting around it.

This utility currently supports text/chat counting through supported vLLM models. It doesn't account for image inputs, response schemas, or every model-specific reasoning setting. fits: true is a token limit check, not a guarantee that generation will succeed. Tokenization also has its own input and returned-token limits. Ask for help if it can't count your intended input.

When a request fails

If the small request doesn't work, start with the basics:

• Is this the right project?
• Does the model ID match the one you were given?
• Is your key still active?

A typo or access problem is easier to isolate with a short prompt than with an entire paper.

For a context overflow, the gateway returns HTTP 400 with code: "context_length_exceeded". For example, requesting 4,097 output tokens from a model with a 4,096-token context window produces:

{
  "error": {
    "message": "This model's context limit is 4096 tokens, shared by input and output. Requested output: 4097 tokens. Reduce the messages or max_tokens.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "context_length_exceeded",
    "details": {
      "requested_output_tokens": 4097,
      "context_limit": 4096
    }
  }
}

An oversized input gets the same error code, with param: "messages". When the backend reports only a lower bound, fields such as prompt_tokens_at_least make that distinction explicit. Counts appear when the backend supplies them.

Exceeding the gateway's separate output cap returns HTTP 400 with code: "token_limit" and the requested and allowed output counts.^†

For a longer request, check the input and output limits. If you cut information to fit within the limit, you also change what information is available to the model to generate its response. You may need to get creative with how you break down the task to meet context constraints.

The client does not automatically retry token limit or byte size errors. Change the request first. We set max_retries=0 in this example so temporary failures also return immediately, making each attempt easier to inspect.

Still stuck? Keep the error and request ID, if available, and ask for help. Leave the key out of anything you share. Once you get an answer, check its substantive claims against the source. Getting a response tells us the request worked, not that the answer is right.