How does a chatbot actually write a sentence?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, we saw data, chips and better methods make AI suddenly good at recognising pictures. But a chatbot writes back whole sentences, in good grammar, on almost any topic. So what is it actually doing? The answer sounds far too simple. It guesses what comes next, then does it again, and again ^1,2. Let's see why that trick goes so far, and where it breaks.

A guessing game 0:28

The idea is far older than chatbots. In 1951, the American mathematician Claude Shannon asked a person to guess a passage of text one letter at a time, guessing again after each miss ³. The passage began, there is no reverse on a motorcycle ³. Out of a hundred and two letters and spaces, the guesser got seventy-nine right on the very first try ³. Shannon's point was that anyone who speaks a language carries an enormous, unspoken knowledge of what tends to come next ³.

A machine made of books 1:03

Three years earlier, Shannon had run the game in reverse, to write text instead of guessing it ⁴. He opened a book at random, found a word, and wrote down the word that came after it. Then he opened another page, found that new word, and copied whatever followed, over and over ⁴. The result began: the head and in frontal attack on an English writer that the character of this point is therefore another method ⁴. Each little stretch sounds like English, but the whole is nonsense ⁴.

A tiny model you can check 1:36

We can build Shannon's machine in a few lines of code. Give it five short sentences about kookaburras, magpies and gum trees, and count which word followed which. After "the", it saw kookaburra three times, old twice, sun twice, and a few others once. Now write one word at a time, choosing each next word in proportion to those counts. Ours wrote: the old magpie laughed at the hot sun. That sentence is in none of the five. Run it again, and you might get: the sun sat in the kookaburra. Every pair of words is fine, but the whole is silly, just as Shannon found ⁴.

From counting to a network 2:20

Counting only works for short stretches ⁴. Look back ten words instead of one, and almost every stretch is new, so there's nothing to count ⁵. In 2003, Yoshua Bengio and colleagues called this the curse of dimensionality, and proposed a fix: train a neural network to predict the next word instead ⁵. That's the network from episodes three and four. Today's chatbots still do that same job, at vastly greater scale ^1,6,7.

Not letters, not words 2:52

But a next what, exactly? Letters would make every text very long, and whole words can't cover every name, typo and new word ^8,1. So language models use something in between, called tokens ^8,1. A token is a chunk of text: often a whole common word, sometimes a piece of a word, sometimes a single character ^9,7.

A real split 3:15

Here's a real tokenizer at work. This is the one OpenAI published with its GPT-2 model in 2019, run on a sentence of ours ^1,10. It cuts the sentence into fourteen tokens. Common words like laughed and afternoon come whole, and so do Adelaide and unbelievably. But kookaburra, a rarer word, comes out in five pieces ^8,10. Each token is really a number, and those numbers are all the network ever sees ^10,7.

Where tokens come from 3:49

The method behind it was adapted in 2015 by Rico Sennrich, Barry Haddow and Alexandra Birch, for machine translation, from an old data-compression trick called byte pair encoding ⁸. GPT-2 borrowed it four years later ¹. Start with single characters. Find the pair that sits side by side most often in a mountain of text, glue it into one new token, and repeat, tens of thousands of times ^8,1. GPT-2 ended up with about fifty thousand tokens ¹. Meta's Llama 3, in April 2024, used about a hundred and twenty-eight thousand ¹¹. As of October 2026, Google and OpenAI both put a token at roughly four characters of English ^12,9.

Scores for every token 4:41

Now the prediction itself. Feed in the tokens so far, and the network gives back a score for every token in its vocabulary ¹. For GPT-2, that's about fifty thousand scores, every single time ¹. A step called softmax turns those scores into probabilities that add up to one ¹³. Take: the kookaburra sat in the old gum. With made-up scores for five candidates, tree comes out at about seventy-seven per cent, trees at ten, and three others share the rest ¹³.

Pick one 5:16

So which does it write? The obvious choice is the top one, every time. That's called greedy decoding, and it tends to get stuck, repeating itself ². One 2019 study called the result bland and strangely repetitive ¹⁴. The alternative is to sample: pick at random, but weighted by the probabilities, like a die with more faces for likelier tokens ². Usually it lands on tree, but now and then on something else.

Temperature 5:48

How weighted is the die? That's set by a number called temperature. Before the softmax, every score is divided by it ¹³. A low temperature sharpens the differences. At nought point two, tree gets virtually every roll, and as it nears zero, it's the same as always picking the top token ². A high temperature flattens them. At two, tree drops to about forty-nine per cent, and even stump gets more than eight ¹³. Ten rolls each: tree every time at the low end, a mix at the high end.

The dial in real products 6:24

Companies that sell these models describe the dial differently. Mistral's documentation suggests keeping it between nought and nought point seven ¹⁵. Google's says its newest Gemini models ignore custom temperatures altogether ⁶. And both Google and Anthropic warn that even at zero, the same question can still give slightly different answers ^7,16. That's as of October 2026, and it changes often.

One token at a time 6:53

Now the loop. The chosen token is stuck on the end of the text. Then the whole thing, your question plus every token written so far, goes back through the model to get the next one ^1,2. Pick, append, repeat, until it picks a token that means stop ¹⁰. So the answer isn't composed first and then typed out ^1,2. It's built as it goes, which is why replies can appear a few words at a time, and companies even measure the time to the first token ⁷.

So what's a large language model? 7:25

A language model is anything that predicts the next token like this ^1,5. Large means the network has a huge number of parameters, the adjustable numbers from episodes three and four, trained on a huge amount of text ¹⁷. There's no official cut-off ¹⁷. GPT-2, in 2019, had one and a half billion parameters ¹. GPT-3, in 2020, had a hundred and seventy-five billion ¹⁸. Meta's largest Llama 3.1, in July 2024, had four hundred and five billion ¹⁹. The models behind chatbots such as ChatGPT, Gemini, Claude and DeepSeek, and open models such as Meta's Llama and Mistral's, are of this kind ^{17,9,15,7,11,20}.

Why it's so powerful 8:14

The GPT-2 researchers' bet was that to predict the next token well across everything people write, a model would pick up grammar, facts, styles and formats along the way ¹. In 2019, they found their model starting to answer questions and translate without being trained for either ¹. In 2020, GPT-3 could do many tasks from just a few examples typed into the prompt ¹⁸.

Parrot, or something more? 8:41

Does that mean it understands? Researchers disagree ^21,22. In 2021, the linguist Emily Bender, Timnit Gebru and colleagues called a language model a stochastic parrot, stitching together patterns of words it has seen, without any reference to meaning ²¹. In 2023, Kenneth Li and colleagues trained a model only to predict legal moves in the board game Othello ²². It was never shown the board, yet they found evidence it had built an inner picture of one anyway ²². That debate is still running.

No reverse gear 9:17

Now the limits. Remember Shannon's motorcycle: there's no reverse ³. In the basic loop, once a token is written, everything after is predicted from it, and nothing goes back to fix it ^1,2. One workaround is to write the working out first ²³. In 2022, researchers found that asking for intermediate steps improved reasoning in large models ²³. Since then, some models have been trained to do it at length before answering: OpenAI's o1 series, in 2024, then DeepSeek's R1, in January 2025 ²⁰. And a likely sentence is not always a true one, but that's episode nine.

Tokens hide the letters 10:02

The tokens themselves cause trouble too. Typed on its own, strawberry reaches GPT-2 as three chunks; in the middle of a sentence, it's a single token ¹⁰. Either way, the model never directly sees the letters ¹⁰. In a December 2024 study, most of the language models tested couldn't count a letter that appears more than twice in a word, though researchers disagree on how much tokens are to blame ^24,25. Numbers get chopped up as well, here into one-two-three, four-five, six-seven ¹⁰. In a 2024 study, just adding commas to numbers, so the chunks lined up properly, improved some of OpenAI's models' arithmetic a lot ²⁶.

The part we skipped 10:47

So that's how a chatbot writes a sentence: tokens in, a probability for every possible next token, one weighted pick, and round again ^1,2,13. But we treated the network itself as a closed box. Take this sentence: the trophy didn't fit in the suitcase because it was too big. To predict what comes next, a model has to work out what "it" refers to. How does it decide which earlier words matter? What is attention, and why did one 2017 paper change everything?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, "Language Models are… — cdn.openai.com
2. P. von Platen, "How to generate text: using different decoding methods for language… — huggingface.co
3. C. E. Shannon, "Prediction and Entropy of Printed English", Bell System Technical… — sites.socsci.uci.edu
4. C. E. Shannon, "A Mathematical Theory of Communication", Bell System Technical… — people.math.harvard.edu
5. Y. Bengio, R. Ducharme, P. Vincent, C. Jauvin, "A Neural Probabilistic Language… — jmlr.org
6. Google AI for Developers, "Prompt design strategies" (Gemini API docs: temperature,… — ai.google.dev
7. Anthropic, Claude Docs "Glossary": Pretraining, Temperature, Tokens, TTFT (read 15… — platform.claude.com
8. R. Sennrich, B. Haddow, A. Birch, "Neural Machine Translation of Rare Words with… — arxiv.org
9. Google AI for Developers, "Understand and count tokens" (Gemini API docs, last updated… — ai.google.dev
10. OpenAI, gpt-2 repository: src/encoder.py (byte-level BPE) and the published 124M… — github.com
11. Meta AI, "Introducing Meta Llama 3" (18 April 2024) — ai.meta.com
12. OpenAI, tiktoken README (fast BPE tokeniser; lossless; about 4 bytes per token) — github.com
13. G. Hinton, O. Vinyals, J. Dean, "Distilling the Knowledge in a Neural Network",… — arxiv.org
14. A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi, "The Curious Case of Neural Text… — arxiv.org
15. Mistral AI, API reference: chat completion, temperature (read 15 October 2026) — docs.mistral.ai
16. Google Cloud, Vertex AI "Content generation parameters" (temperature; last updated 15… — cloud.google.com
17. W. X. Zhao et al., "A Survey of Large Language Models", arXiv:2303.18223 (v19, March 2026) — arxiv.org
18. T. B. Brown et al., "Language Models are Few-Shot Learners" (GPT-3), arXiv:2005.14165… — arxiv.org
19. Meta AI, "Introducing Llama 3.1" (23 July 2024) — ai.meta.com
20. DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via… — arxiv.org
21. E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, "On the Dangers of… — doi.org
22. K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, M. Wattenberg, "Emergent World… — arxiv.org
23. J. Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language… — arxiv.org
24. T. Fu, R. Ferrando, J. Conde, C. Arriaga, P. Reviriego, "Why Do Large Language Models… — arxiv.org
25. A. Cosma, S. Ruseti, E. Radoi, M. Dascalu, "The Strawberry Problem: Emergence of… — arxiv.org
26. A. K. Singh, DJ Strouse, "Tokenization counts: the impact of tokenization on… — arxiv.org

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.