Overview

In this unit, we'll look at Large Language Models (LLMs). 👀

Outcomes

By the end of this unit, you'll be able to …

• describe the structure of an LLM

• identify pitfalls of LLMs

• describe the pretraining tasks used for LLMs

• create and use retrieval augmented generation (RAG) to improve prompting

• instruct LLMs to use tools

Large Language Models (LLMs)

In 2025, Large Language Models (LLMs) are a foundational technology in natural language processing (NLP). They have seen widespread adoption beyond academia, because they have such broad application (often without requiring task-specific training).

The dominant architecture for LLMs are decoder-only transformers with absolute positional embeddings and byte-pair encoding (BPE) for tokenization.

Tokenization revisited for LLMs

In earlier courses, we examined tokenization as a way to divide text into useful units for processing. LLM tokenization adds a model-specific layer to this: the model does not receive words directly. It receives token IDs from a fixed tokenizer vocabulary.

Before going further, review the linked tutorial:

• How LLMs See Text: Tokenization

Pay particular attention to the distinction between word-like tokens and model tokenizer tokens. A model token may be a word, a word piece, punctuation, whitespace-sensitive fragment, byte-like piece, or something that looks linguistically unnatural.

This distinction affects several practical questions:

• context length: the model's limit is measured in tokens, not words
• cost: many hosted APIs charge by input and output tokens
• multilingual behavior: the same amount of visible text may require more tokens in one language than another
• prompt formatting: spaces, punctuation, and line breaks can change the token sequence
• next-token prediction: the model predicts a distribution over token IDs, not directly over words

The associated Etude activity asks you to inspect tokenizations, compare word-level and model-level token counts, and connect token IDs to the softmax-based decoding examples below.

Training

LLMs are pretrained on extremely large text corpora (ex. 15+ trillion tokens) using self-supervised learning and next token prediction.

Once pretrained on large corpora, LLMs then typically are put through one or more rounds of supervised fine-tuning (SFT) to learn to following instructions (Instruction-tuning; ex. question answering). Their behavior can then be further refined using techniques like Reinforcement Learning from Human Feedback (RLHF) to ensure the most likely model output is also the most preferred output by some criteria (ex. politeness, factuality, etc.).

Scaling

The "large" in LLMs is in reference to the model's number of trainable parameters. Large has become a moving target. Previously, billions of parameters was considered "large", but LLMs like GPT-5 are speculated to have trillion of paramters!

The reason LLMs keep getting larger is due to observations about parameter and data scaling¹ [2020] where researchers found that larger models trained on even more data appear to continue to improve in performance on a wide range of NLP tasks. While there are some non-profits such as AI2 training very large language models and providing real research contributions about the process, most of these very large language models today are closed-source and proprietary. Many of the details² related to how they were trained and the costs to train them are unknown to the public. The cost to train GPT-3 in 2020 was estimated at more than 4 million dollars (USD).

Can small language models be effective?

Very large language models (ex. Claude Sonnet 4.5, GPT-4, etc.) have broad capabilities, but most of them cannot be directly run on commodity hardware (ex. your laptop). Even for inference (i.e., generating predictions), they typically require massive data centers with specialized hardware and a hefty power supply.

One exciting area of research is how to make smaller language models (order of 1 million to $\leq$ 4 billion parameters) approximate the performance of their much larger siblings. In this tutorial, we'll look at several techniques for enhancing the capabilities of SmolLM2 [2025], a family of open weight and permissively licensed small language models ranging in size from 135M to 1.7B parameters.

Setup

We'll be taking a code-first approach, so that you can try these techniques out for yourself on your own machine.

To get started, create a virtual environment and install the specified dependencies:

# see https://mamba.readthedocs.io/en/latest/installation/micromamba-installation.html
mamba create -y -n "llms" python=3.12
mamba activate llms
# install the educational library 
# built for this lesson
# from the main (default) branch
pip install "git+https://github.com/myedibleenso/learn-llms.git@main#egg=learn-llms[all]"

In this lesson, we'll be using two containerized services: one for serving LLMs and another as a vector database.

Ollama

In this lesson, we'll be using Ollama to interact with LLMs. This is far from the only option available, but it's adequate for this lesson. Ollama is a popular model serving framework that exposes an OpenAI-compatible API. Due to the popularity of the OpenAI-compatible API (mostly due to OpenAI's first mover advantage) This makes it easy to integrate with many other libraries.

Launch the following docker container:

# NOTE: if you're on a Mac with an M1/M2 chip,
# you may want to install this directly
# via https://ollama.com/download instead of using Docker
docker run --rm -p 11434:11434 ollama/ollama:latest ollama serve

ChromaDB

In this lesson, we'll be using ChromaDB as a vector database. This is far from the only option available, but it's adequate for this lesson.

Launch the following docker container:

docker run -it -p 8002:8000 chromadb/chroma:0.6.3 run --host 0.0.0.0

Prompting

Despite all those parameters, LLMs are trained to do something very simple (at least conceptually): predict the next token in a sequence of text. The input to an LLM is typically referred to as a prompt.

from clu.ling.learn.llms import env
from clu.ling.learn.llms.ollama import create_client
from ollama import Client
import typing

# Our ollama instance could be running anywhere.
# For instance, a beefy server
client = create_client(host=env.OLLAMA_HOST)


# Let's create a simple function to make it easier to prompt our LLM and get its response.
def get_response(
    prompt: str,
    model="smollm2:1.7b",
    client: Client = create_client(host=env.OLLAMA_HOST),
    temperature: float = 1.0,
    seed: typing.Optional[int] = 42,
    verbose: bool = True,
) -> str:
    """Query our LLM and return its response"""
    options = {"temperature": temperature}
    if seed:
        options["seed"] = seed
    response = client.chat(
        model, messages=[{"role": "user", "content": prompt}], options=options
    )
    if verbose:
        print(f"prompt:\n{prompt}\n")
    return response.message.content


# ensure the model is available
client.pull("smollm2:1.7b")

print_response = lambda resp: print(f"response:\t{resp}")

# Correct (a full sentence).
print_response(get_response(prompt="What is the capital of Japan?", temperature=1))

Unsurprisingly, the structure of a prompt (i.e., our input) has a large effect on the output that gets generated by the LLM. Providing the LLM with additional context and instructions can help guide the model to produce outputs closer to the user's expectations:

# Good work!
print_response(
    get_response(
        prompt="What is the capital of Japan? Provide a single word answer with no punctuation.",
        temperature=1,
    )
)

# Japanese language output, but nonsense
print_response(
    get_response(
        prompt="What is the capital of Japan? Answer in Japanese.", temperature=1
    )
)

Prompt roles

LLM prompts often incorporate multiple messages with different roles. The most common roles are system, user, assistant, and tool.

• system: A system prompt or message is a persistent set of instructions for the model. Often the system prompt sets a persona (ex. "You are a research assistant", "You're an Elvis impersonator. […]") and expectations about tone and response structure.
• user: A user prompt or message capture's the user's input/query (a simple prompt)
• assistant: An assistant prompt or message is used to record a model's prior responses or encode examples (ex. few-shot in context learning)
• tool: A tool prompt or message is used to record the output of a tool call (ex. the result of executing a function or API call)

These role-based messages are interleaved to encode a conversation history.

In-context learning

One of the key strengths of LLMs is that they can infer how to perform tasks simply by being provided with examples in their prompt context. This method of prompting is is known as in-context learning (ICL).

Depending on the number of examples provided, in-context learning can be categorized as zero-shot (no examples) one-shot (a single example provided), few-shot, or many-shot.

Decoding strategies

The next token selection process is known as decoding. Various strategies exist.

Greedy decoding

In greedy decoding, the model simply selects the most probable next token at each step. Often this output can be boring and repetitive.

Beam search

In beam search, multiple candidate sequences (beams) are preserved throughout generation. These sequences can then be scored and ranked by some criteria to select the best options. Beam search produces superior output to greedy search, but it can be computationally expensive.

Sampling

LLMs can be coaxed into generating more diverse outputs through sampling. In a determinstic setting, an LLM will choose the most probable next token from its softmax layer each and every time. For more variation, one can employ a sampling strategy.

Top-k sampling

In top-$k$ sampling, a random number generator is used to select a token from the $k$ most probable tokens predicted by the model.

Temperature

It's possible to manipulate the skew of probabilities produced by softmax using by adjusting the "temperature" ($T$ in the formula below):

$$softmax(x_{i}) = \frac{e^{(x_i / T)}}{\sum_{j} e^{(x_{j} / T)}}$$

import torch

def logits_from_probs(probabilities: torch.Tensor) -> torch.Tensor:
    """Convert probabilities to log probs (i.e., logits)"""
    return torch.log(probabilities)

def climate_controlled_softmax(logits: torch.Tensor, temperature: float = 1.0) -> torch.tensor:
    """Applies softmax with temperature (T)
    """
    if temperature <= 0.0:
        raise ValueError("Temperature must be a positive value.")
    # NOTE: OG softmax (T=1)
    scaled_logits = logits / temperature
    return torch.softmax(scaled_logits, dim=-1)


probs = torch.tensor([0.75, 0.15, 0.20])
logits = logits_from_probs(probs)
climate_controlled_softmax(logits, temperature=4)

Temperature is typically used in conjunction with a sampling method (ex. top-$k$ sampling). The probabilities of the predicted tokens are adjusted by some temperature $T$ before sampling. Increasing the temperature beyond 1 increases the likelihood of selecting less probable tokens which in turn produces more diverse outputs. As $T$ increases, LLM output becomes less coherent (more random). Conversely, lowering $T$ below 1 makes the model's output more deterministic.

LLM pitfalls

There are several pitfalls to be aware of when working with LLMs.

Hallucinations

LLMs have seen tons of text, but they don't know everything. In some sense, their knowledge is "frozen in time", as their training data only includes text up to some date. When prompted around things outside the realm of their training, LLMs will generate a (confident sounding) response, but the response is incorrect (ex. non-factual or unrelated). These incorrect outputs are commonly referred to as hallucinations.

Hallucinations pose a serious challenge for real-world applications of LLMs. They can be mitigated (though not eliminated!) through improved prompting or context engineering (techniques for enriching context). Contextual enrichment can draw from external knowledges bases or the output of tools (ex. forms of code exeuction like interacting with a search engine, a database, or even a file system).

Bias

LLMs reflect the bias present in their training data ("you are what you eat"). Given that they're trained on large swaths of the internet, some of that bias can be more overt and explicit.

Various techniques exist to mitigate bias in LLMs (training data curation, SFT, RLHF, etc.), but debiasing remains an open research problem. Users should take care in evaluating the setting in which an LLM will be used (ex. resume screening, legal advice, etc.) and make plans for monitoring their output.

Retrieval Augmented Generation (RAG)

A general technique for enriching the context of an LLM is to retrieve information from some source and include it in the prompt. This approach is known as Retrieval Augmented Generation (RAG). A populare approach to RAG is to embed knowledge and store it in a vector database. At query time, the prompt is intercepted, optionall transformed, embedded, and queried against the vector database. The text associated with the top $k$ nearest embeddings are then injected into the prompt as additional context, allowing the LLM to generate a response that synthesizes this information.

Embedding knowledge is one part science and another part art. One has to select an appropriate embedding model (ex. an embedding model trained to encode questions and their answers near one another in some vector space). Most embedding models are transformer-based and have a limit on their input size. Something like a book will need to be chunked into smaller pieces before each piece can be embedded. Decisions about where to chunk the text (ex. sentences, $n$-grams, prompting an LLM to chunk text for a particular task, etc.) will influence the quality of the embeddings. Overchunking or underchunking can lead to poor retrieval performance (key context may get scattered across multiple chunks or lost in the noise of larger passage). Retrieval strategies may need to be aware of the chunking strategy used to assemble more cohesive context.

Let's examine an example using RAG to encode information that would otherwise be unknown to our LLM.

from clu.ling.learn.llms import env
from clu.ling.learn.llms.ollama import create_client
from ollama import Client
from chromadb import Documents, EmbeddingFunction, Embeddings
from chromadb import Documents, EmbeddingFunction, Embeddings
import chromadb
import typing

# Our ollama instance could be running anywhere.
# For instance, a beefy server
client = create_client(host=env.OLLAMA_HOST)

# ensure the model is available
client.pull("smollm2:1.7b")


# Let's create a simple function to make it easier to prompt our LLM and get its response.
def get_response(
    prompt: str,
    client: Client = create_client(host=env.OLLAMA_HOST),
    model="smollm2:1.7b",
    temperature: float = 1.0,
    seed: typing.Optional[int] = 42,
    verbose: bool = True,
) -> str:
    """Query our LLM and return its response"""
    options = {"temperature": temperature}
    if seed:
        options["seed"] = seed
    response = client.chat(
        model, messages=[{"role": "user", "content": prompt}], options=options
    )
    if verbose:
        print(f"prompt:\n{prompt}\n")
    return response.message.content


print_response = lambda resp: print(f"response:\t{resp}")

query = "Where was Zarbon the Great born?"

print_response(get_response(client, prompt=query))

# Problem: "Zarbon the Great" is a character I invented for this example.
# While it may have associations with other fictional characters,
# there is no way the LLM can "know" anything about this character.
# What if we provided it with more context?

# imagine we have a knowledge base of facts about
# characters in our universe ...
galactic_facts = [
    "Zarbon the Great was born on the generation ship Aslan.",
    "Zarbon the Great is remembered for his conquest of Alpha Centauri 3125 (standard time).",
    "Zarborn the Timid was born on a small mining station in the middle of the Kuiper belt.",
    "Zarbon the Timid suffered from Gravity Sickness until their death in 4078.",
]

# We'll enrich our context using a form of retrieval augmented generation (RAG).  Our retrieval method will be embedding-based.  First, we'll embed these "facts" and store them in a vector database.
# To construct our prompt for our LLM, we'll query against this vector database and then include $$k$$ of our top hits in our prompt as additional context...

# Problem: How should we embed our "facts" and query?

# Option 1: embed ahead of time with ollama,
# and pass the embeddings to our vector store
# embeddings = client.embed(model="all-minilm:latest", input=galactic_facts).embeddings
#
# Option 2: create an embedding function
# we can also provide our vector store
# with an embedding function which
# will take care of the embedding process.
# we can use ollama's API to embed:


class LlamaEmbeddingFunction(EmbeddingFunction[Documents]):
    """Custom ChromaDB embedding function for use with sentence-transformer models"""

    def __init__(self, ollama_client: Client, model_name: str = "all-minilm:latest"):
        self.client = ollama_client
        self.model_name = model_name

    def name(self) -> str:
        return self.model_name

    def __call__(self, docs: Documents) -> Embeddings:
        # embed each document
        return self.client.embed(model=self.model_name, input=docs).embeddings


# Option 3: something like
# sentence-transformers would provide
# more flexibility for embedding

# We'll use ChromaDB as our vector store

# Our ChromaDB instance could be running anywhere
vectordb = chromadb.HttpClient(
    host=env.CHROMADB_HOST,
    port=env.CHROMADB_PORT,
)

# Next, we'll create a "collection" of embeddings.
# Collections help us to organize our knowledge
# and selectively query
# NOTE: this collection can be customized.
# see https://docs.trychroma.com/docs/collections/configure#hnsw-index-configuration
collection = vectordb.get_or_create_collection(
    "galactic-facts",
    configuration={
        "hnsw": {
            # we'll use cosine distance (i.e., 1 - cosine sim)
            "space": "cosine",
        }
    },
    embedding_function=LlamaEmbeddingFunction(ollama_client=client),
)

# we need to assign a unique ID to each piece of text
# that we embed
id_for = lambda doc: f"doc-{hash(doc)}"
# replace if exists, else add new record
collection.upsert(
    # we need to produce unique IDs for each record
    # that we add to our vector store
    ids=[id_for(gf) for gf in galactic_facts],
    # our documents
    # the embedding function we defined will take care
    # of actually embedding these
    documents=galactic_facts,
)

# what are the 2 most similar (in terms of cosine distance) documents
# to our query?
results = collection.query(
    query_texts=[query],
    n_results=2,  # Number of nearest matches to retrieve
    include=["documents", "distances"],
)

# let's check our results.
# NOTE: a lower distance score is a closer match
for hit, dist in zip(results["documents"][0], results["distances"][0]):
    print(f"{dist:.2f}:\t{hit}")

rag_prompt = f"""
{query}

Use the following information to answer the question:

{'\n'.join(f'- {hit}' for hit in results["documents"][0])}
""".strip()

print_response(get_response(client, prompt=rag_prompt))

# That's pretty good, but the response is a bit overly detailed (a matter of opinion).
# Let's see if we can further guide the output...
refined_rag_prompt = f"""
{query}

Use the following information to answer the question. Be as succinct as possible:

{'\n'.join(f'- {hit}' for hit in results["documents"][0])}
""".strip()

print_response(get_response(client, prompt=refined_rag_prompt))

# Not bad for a small language model!
# If we constrain our output (logit biasing, structured output, etc.), we can probably do even better.

Not bad for a small language model! If we constrain our output (logit biasing, structured output, etc.), we can probably do even better.

Using tools

What if LLMs could execute code and call APIs? While they can't take action on their own, when provided with a definition of a tool, they can learn to predict what tool should be called and how (ex. function parameters).

"tool-calling" LLMs are implemented using an agentic loop:

1. An LLM is provided with a prompt including a set of tool definitions.
2. THe LLM generates a response predicting which tool(s) to call and how (i.e., parameter values).
3. Some system evaluates the LLM response, executes the predicted tool calls, and appends the tool responses to the LLM's context.
4. The LLM generates a new reponse based on this enriched context (a growing history of tool calls and responses)

The process repeats until some stopping criterion is met (ex. $k$ tools calls made, no tool call generated, etc.) and the final response is returned.

Let's look at an example…

from datetime import datetime
from clu.ling.learn.llms import env
from clu.ling.learn.llms.ollama import create_client
from ollama import Client, ChatResponse
import ollama
import typing

# Our ollama instance could be running anywhere.
# For instance, a beefy server
client = create_client(host=env.OLLAMA_HOST)

# ensure the model is available
client.pull("smollm2:1.7b")


# Not that the docstring of the function
# is used as context for our tool def
def get_current_local_time() -> str:
    """
    get current local time

    Returns:
        str: current (24-hour clock) time as H:M
    """
    current_local_time = datetime.now()
    return current_local_time.strftime("%H:%m")


response = client.chat(
    "smollm2:1.7b",
    messages=[{"role": "user", "content": "What is the current local time?"}],
    tools=[get_current_local_time],
    think=False,
)

# ChatResponse(model='smollm2:1.7b', created_at='???', done=True, done_reason='stop', total_duration=249113625, load_duration=47981958, prompt_eval_count=265, prompt_eval_duration=49427167, eval_count=29, eval_duration=143076750, message=Message(role='assistant', content='', thinking=None, images=None, tool_name=None, tool_calls=[ToolCall(function=Function(name='get_current_local_time', arguments={}))]), logprobs=None)


# We'll use an **agentic loop**.  An **agentic loop** is an approach to problem solving where an LLM makes a series of tool calls and self-reflections ("thinking") to work or "reason" toward a solution.

# Agent loop
# adapted from https://docs.ollama.com/capabilities/tool-calling#multi-turn-tool-calling-agent-loop
tools = [get_current_local_time]
messages = [{"role": "user", "content": "What is the current local time?"}]
while True:
    response = client.chat(
        "smollm2:1.7b", messages=messages, tools=[get_current_local_time], think=False
    )
    messages.append(response.message)
    tool_map = {fn.__name__: fn for fn in tools}
    if response.message.tool_calls:
        for tc in response.message.tool_calls:
            if tc.function.name in tool_map:
                print(
                    f"Calling {tc.function.name} with arguments {tc.function.arguments}"
                )
                tool = tool_map[tc.function.name]
                result = tool(**tc.function.arguments)
                print(f"Result: {result}")
                # add the tool result to the messages
                messages.append(
                    {
                        "role": "tool",
                        "tool_name": tc.function.name,
                        "content": str(result),
                    }
                )
    else:
        # end the loop when there are no more tool calls
        break
    # continue the loop with the updated messages


# let's refactor...


def agentic_loop(
    prompt: str,
    tools: list[typing.Callable] = [],
    client: Client = create_client(host=env.OLLAMA_HOST),
    model: str = "smollm2:1.7b",
    simple: bool = True,
    temperature: float = 0.1,
    seed: typing.Optional[int] = 42,
    verbose: bool = False,
) -> typing.Union[list[ChatResponse], str]:

    tool_map = {fn.__name__: fn for fn in tools}
    options = {"temperature": temperature}
    if seed:
        options["seed"] = seed
    messages = [{"role": "user", "content": prompt}]
    responses = []
    while True:
        response = client.chat(
            model,
            messages=messages,
            tools=tools,
            # think=True,
            options=options,
        )
        responses.append(response)
        # print(f"Thinking: {response.message.thinking}")
        res = responses[-1].message.content
        if verbose:
            print(f"Messages (history):\t{messages}")
            print(f"Reponse: {res}")
        if response.message.tool_calls:
            for tc in response.message.tool_calls:
                if tc.function.name in tool_map:
                    print(
                        f"Calling {tc.function.name} with arguments {tc.function.arguments}"
                    )
                    result = tool_map[tc.function.name](**tc.function.arguments)
                    print(f"Result: {result}")
                    # add the tool result to the messages
                    messages.append(
                        {
                            "role": "tool",
                            "tool_name": tc.function.name,
                            "content": str(result),
                        }
                    )
        else:
            # end the loop when there are no more tool calls
            break
    if verbose:
        print(f"Messages (history):\t{messages}")
    final_response = client.chat(
        model,
        messages=messages,
        # NOTE: We'll disable tools for the last call.
        # tools=tools,
        options=options,
    )
    responses.append(final_response)
    res = final_response.message.content
    messages.append({"role": "assistant", "content": res})
    if verbose:
        print(f"Messages (history):\t{messages}")
    return res if simple else responses


# should be straightforward with the tool output...
agentic_loop(
    prompt="What is the current local time?",
    tools=[get_current_local_time],
    temperature=1.1,
)

# building upon the tool output...
agentic_loop(
    prompt="Provide the local time in 12-hour format",
    tools=[get_current_local_time],
    temperature=1.1,
)

Model Context Protocol

Model Context Protocol (MCP) is an open-source standard introduced by Anthropic to make tools and resource access methods more portable across various LLM serving frameworks. You can learn more about MCP at https://modelcontextprotocol.io.

If you want to build an MCP server in Python, you may want to take a look at the official Python MCP SDK.

Usage is conceptually quite similar to the example above. An MCP server exposes a set of tool definitions (ex. typically something like OpenAPI specification). Each tool definition is registered with the LLM as context in its prompt (equivalent to tool_map in the example above) and an agentic loop is executed. Matching tool calls are executed against the MCP server and the results are appended to the LLM's context for iterations of the loop.

Constrained decoding

LLM output can be constrained by 1) biasing or masking its logits (ex. Grammar-Constrained Decoding (GCD)), 2) providing a detailed specification for the output, or 3) some combination.

Structured output

One simple but effective technique for guiding the output of an LLM is to provide a detailed specification of the desired output format in the prompt. If the output format is a JSON schema definition, we can validate the output against the schema and instantiate classes directly from the LLM responses that are valid.

from clu.ling.learn.llms.ollama import *
from clu.ling.learn.llms import env, tools
from pydantic import BaseModel, Field
import typing


# same as clu.ling.learn.llms.structured_output.Time
# Why BaseModel?  Any class extending 
# a Pydantic BaseModel can generate a json schema
class Time(BaseModel):
  """
  Represents time in a 24 hour format
  """
  hour: int = Field(..., description="the hour in 24-hour format", le=24)
  minute: int = Field(..., description="minutes", le=59)

res = agentic_loop(
  client=create_client(host=env.OLLAMA_HOST), prompt="What is the local time?", 
  tools=[tools.get_current_local_time], 
  temperature=0.1,
  # our structured output 
  response_model=Time
)
# a Time instance!
assert isinstance(res, Time)
print(res)

Practice

Try to put these ideas into practice for your shared task and class competition submissions.

References