What a Language Model Actually Does

Imagine asking an AI assistant to turn your meeting notes into an email. The draft comes back clear, polite, ready to send. It also promises delivery on Friday. You read it again. Nobody promised Friday.

You asked for help with the wording, not permission to make a new commitment. Yet there it is, written with exactly the same confidence as everything that came from your notes. How does the assistant get the tone right and the facts wrong in the same paragraph?

Behind the chat is a language model: a system trained on text to generate text. The assistant is the application around it, which may also search the web, open files, or use other tools. The model supplies the writing, including that Friday nobody agreed to. To see why, let’s follow the email from your notes to the send button.

It learned by predicting what came next

Finish this sentence: “Please find attached the…” You probably thought of an invoice, a report, or a proposal. A pineapple would need explaining. Change the opening to “For the fruit salad, chop the…” and the pineapple fits. The words before the gap change what you expect after it.

A similar exercise is central to training the models behind many chat assistants, but it is not a matter of filling the gap at random. Given part of a text, the model assigns probabilities to possible continuations. Training adjusts its internal numerical settings to give more probability to the continuation in the training text. Those settings are called parameters; large models can have billions of them.

Repeat this across enormous amounts of text and the model learns much more than common phrases. Making good predictions also involves relationships among ideas, facts, and steps in an explanation. The resulting model can translate, explain, and solve some kinds of problems. Later training helps shape it into an assistant that follows instructions and responds usefully. None of that guarantees a verified source behind every sentence.

That is how it can produce the tone and structure of a follow-up email without having any record of your meeting. Predicting the language of a commitment is not the same task as verifying that someone made it. For your commitments, it needs your notes, and it still has to use them correctly.

If you correct the date in the chat, the model can use the correction in its next reply. An ordinary conversation does not rewrite those learned settings. Whether the assistant remembers the correction next week depends on what the application saves and supplies later.

How your notes become something it can work with

Now you paste in the notes. You see sentences; the model processes numerical representations of them. First, the text is divided into pieces called tokens. A token may be a whole word, part of one, or a punctuation mark. It is a unit for processing text, not necessarily a unit of meaning: one word may take several tokens.

Each token gets an identifying number, which is used to look up a list of numerical values called an embedding. Learned during training, those values provide a starting representation, not a dictionary definition. Successive layers of calculation transform it using the surrounding text. In “the bank approved the loan” and “the river overflowed its bank,” the surrounding words are what distinguish the two uses of “bank.”

It also needs information about order. “The client called the supplier” and “The supplier called the client” contain the same words, but change who made the call. The model’s calculations must account for that difference.

This also explains why page count is a poor guide to how much text a model can handle: its text capacity is measured in tokens. An application may impose separate limits on file size. But converting the notes into numbers is only the beginning. The useful work depends on the relationships the model calculates from them.

The answer is assembled, not looked up

When you hit send, the application gathers your request, the notes, perhaps earlier messages, and instructions of its own. The model works with what it receives, not with everything you know about the meeting. Using a trained model this way is called inference: this is when it puts what it learned to work on your request.

How does it connect one part of the notes with another? In the transformer design used by these models, attention calculates weights that determine how information from different text positions is combined. This can help connect “it” with “the supplier” in “The supplier needs more time. It will confirm the schedule tomorrow.” Attention works alongside other calculations, across many layers; it is not a separate reader identifying who did what.

Attention here is a calculation, not a promise of careful reading. The model may connect enough of the notes to write a good summary and still miss “the date is not confirmed.” That one omission can change what the email promises.

The model generates the email rather than retrieving a finished answer. It calculates probabilities for the next token, using the available context and its learned parameters. The system selects a token according to those probabilities and its generation settings, adds it to the answer, and repeats. This is next-token prediction. The calculation may draw on a long passage even though it produces one token at a time. The probability describes a continuation of the text, not a verdict on whether a claim is true.

Back to Friday. Once “We will deliver on Friday” appears, “Please send your comments by Thursday” makes a plausible next sentence. Now the client has a deadline too. The model can sometimes correct itself, but choosing the next piece does not itself check whether a promise was made. A paragraph can hold together perfectly around an invented fact.

You may also notice that asking again produces different wording. The system can take the highest-probability token or sample from the probability distribution, giving some alternatives a chance. A setting called temperature adjusts that distribution: lower values favor the leading candidates more strongly. That tends to reduce variation, but it does not verify facts. The assistant can give you the same wrong date twice.

But I already told it that

Suppose you had already said not to promise a date, ten messages earlier. You can still scroll up and read the instruction. Why didn’t the assistant use it?

There is a limit to how much text the model can work with at once, counting the material it receives and the answer it generates. That space is called the context window. A desk is a useful image for the limit: only so many papers fit. It says nothing about how well their contents will be used. A larger context window increases capacity, not necessarily reliability.

The application decides what goes on that desk. It may send recent messages, summarize older ones, or select passages from an uploaded file. The chat you see on screen is not necessarily the text the model receives next. Your instruction might still be visible to you and missing from the material sent to it.

Or the instruction may have arrived intact and the model failed to follow it. Information can be present without adequately influencing the answer; even its position in a long text can affect performance. That is different from information being absent. Before adding more pages or repeating yourself, separate three questions: did the material fit, was it sent, and did the answer use it correctly?

What to trust it with

Calling this “just predicting text” understates what the process can accomplish. A model can apply learned relationships to material it never encountered during training: translate a new passage, explain it differently, or organize scattered notes into a clear email. Those are real capabilities. New wording alone does not establish human-like understanding, any more than fluent wording establishes accuracy. The useful question is what the result gets right and how we can check it.

A false or unsupported claim presented as fact is commonly called a hallucination. Our invented Friday is one example, not an explanation of every such error. The model might supply a date absent from the notes, or turn “Friday, if the client approves” into an unconditional promise. In the second case, the source was there; the condition was lost. Neither error has to sound uncertain.

Tools can improve the answer: a document search can supply the agreement, a calendar can supply a date, and a calculator can compute a total. But the assistant might retrieve an outdated agreement or attribute one project’s deadline to another. Tools provide evidence and specific capabilities; the system still has to use them correctly. A citation helps when you can open it and find support for the claim, not merely because a link appears beside the answer.

For our email, start with the actual notes and a clear request: “Draft the follow-up. Include only commitments that appear in these notes. If a date is missing, mark it pending.” That gives the assistant a useful way to handle the gap instead of filling it.

Then compare the draft with the notes. Did the names survive? Are the dates there? Did “we could” become “we will”? A better instruction helps, but it does not replace that check. If the company wants to automate this, deciding who or what checks the commitments is part of the work.

My measure is simple: does it still save time once I include the review? A draft that needs a few checks can be a real help. One that makes me reconstruct the whole meeting may have moved the work instead of reducing it.

Keep the better wording. Before you send it, confirm that Friday was actually promised. The client will expect the delivery, not an explanation of how a language model works.

Source notes

These primary sources support the explanation. The meeting and the email are hypothetical examples. Details vary across models and applications.

• How the model connects parts of a text. Vaswani et al. introduced the transformer design, including its attention mechanism and how it represents word order. Source: “Attention Is All You Need”, 2017.
• Why words are split into pieces. Sennrich, Haddow, and Birch describe a method for representing text with pieces smaller than whole words. Source: “Neural Machine Translation of Rare Words with Subword Units”, 2015.
• What predicting the next piece can achieve. Brown et al. describe GPT-3’s training and test results; OpenAI documents GPT-4’s capabilities and limitations. Sources: Brown et al., 2020; OpenAI, 2023.
• Using long contexts. Liu et al. found that the position of relevant information affected performance in the models and tasks they tested. Fitting more text does not ensure it will all be used well. Source: “Lost in the Middle”, 2023.