What is 'attention', and why did one 2017 paper change everything?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, a chatbot wrote one token at a time, and we treated the network inside as a closed box. We stopped on this sentence. The trophy didn't fit in the suitcase because it was too big. What was too big? The trophy, obviously. Now change one word: because it was too small. Now "it" is the suitcase, and you switched without noticing. How does a machine make that switch? With a trick called attention, and a 2017 paper that built a new kind of network from it ¹.

A test built to be hard 0:37

That sentence isn't ours. In 2011, the computer scientist Hector Levesque published it as part of a test for machines, the Winograd Schema Challenge, set out in full a year later with Ernest Davis and Leora Morgenstern ^2,3. Each puzzle is a pair of sentences that differ by a word or two, where the answer flips ². The answer is obvious to a person, but hard to find with simple statistics over text ².

Why "it" matters 1:08

It's named after Terry Winograd, whose 1970 thesis used a similar pair, and his point was translation ⁴. In French, the word for "it" depends on the noun it points to ^5,4. Trophée is masculine, so "it" becomes il, and valise, the suitcase, is feminine, so it becomes elle ⁶. Google's researchers used this kind of sentence in 2017 to show off their new model ⁵.

Words as places on a map 1:38

So how does a network hold a word in the first place? It turns each one into a list of numbers, called an embedding, which you can picture as a position on a map ⁷. The map is learned from text, on an old idea from linguistics: words found in similar company tend to mean similar things ⁷. On our toy map, kookaburra sits near magpie, far from trophy. Real maps use far more than two numbers. In GPT-3, in 2020, each token got twelve thousand, two hundred and eighty-eight ⁸.

Sums with meanings 2:14

In 2013, Tomas Mikolov and colleagues, first at Microsoft Research and then at Google, showed how much those maps can hold ^9,10. Their method, word2vec, learned word positions from one point six billion words of text in under a day ^7,9. And you could do sums with meanings. Take king, subtract man, add woman, and you land closest to queen ¹⁰. On our toy map, the arrow from man to woman, moved onto king, lands beside queen.

The fine print 2:49

The famous test quietly leaves the input words out of the possible answers ⁹. When Malvina Nissim and colleagues let them back in, the nearest word was often just king ¹¹. And maps learned from what people write soak up our stereotypes: a 2016 study found man is to computer programmer as woman is to homemaker ¹². Nissim's team argued that the analogy test can exaggerate some biases, while agreeing they're real ¹¹.

One word, one spot 3:19

Here's the catch for our trophy. A map like this gives each word one fixed spot, whatever sentence it's in ^7,13. "It" lands in the same place every time, and on its own it says little more than 'some thing' ¹³. The network needs a way to rewrite each word's numbers using the words around it, and that is exactly the job attention does ^5,13.

Reading one word at a time 3:43

Before 2017, most of the best translation networks read a sentence one word after another, carrying a running summary along ^1,5. They're called recurrent networks ¹. Squeezing a whole sentence into one summary was a bottleneck, so in 2014 Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio let the translator look back over every input word, and focus on the ones that mattered ¹⁴. That was attention, but still bolted onto a recurrent network ^1,14.

Attention is all you need 4:18

In June 2017, eight researchers, most of them at Google, asked: what if attention was the whole thing ¹? Their paper threw the recurrent part out, and built a network from attention alone, called the Transformer ¹. The paper lists all eight as equal contributors, in random order ¹.

Every word looks at the others 4:39

For every word, the network asks: which other words matter to me, and how much ^5,13? It scores each one, turns the scores into percentages, and rewrites its own numbers as a blend of theirs, in those proportions ^1,13. If "it" puts most of its attention on trophy, its numbers fill up with trophy ¹³. The name is borrowed from human attention, but this is arithmetic, not awareness.

Query, key and value 5:06

How is a score worked out? Each word makes three new lists of numbers from its embedding ^1,13. A query: what am I looking for? A key: what do I offer? And a value: what I'll pass on if chosen ¹³. The score is how well one word's query lines up with another's key: multiply them number by number, add up, and shrink the result a little ¹. Then softmax, from last episode, turns scores into percentages ¹. Nobody writes the tables that make queries, keys and values. They're learned in training, like every other weight ^1,13.

Worked example: "it" 5:46

Let's run it, with made-up keys and queries to show the mechanics. In GPT-2's tokenizer, our sentence is thirteen tokens ¹⁵. In a chatbot, each token can only look back, because later ones haven't been written yet ¹. So when the model reaches "it", there's no "big" yet. In our example, "it" splits its attention, about thirty per cent on trophy and thirty on suitcase. It holds both possibilities open, which is how a standard textbook describes this moment too ¹³.

Then "big" arrives 6:21

Now "big" arrives and looks back. Its query asks, roughly, what's the thing being put in? That lines up with trophy's key, so trophy gets about eighty-five per cent of the attention, and suitcase under one. Swap in "small", and the query asks for the container instead, so suitcase gets the eighty-five per cent. One word changed, and the attention flipped. In a real model, these numbers are learned, spread across hundreds of dimensions and many layers ^1,13.

Many heads, many layers 6:56

One pass of this is called an attention head ¹. The 2017 model ran eight heads side by side, each free to learn a different kind of link, and stacked six layers of them ¹. GPT-3, in 2020, had ninety-six heads in each of ninety-six layers ⁸. And because attention alone ignores word order, dog bites man would look like man bites dog, so in the 2017 model, each word also gets a position signal, built from waves ¹.

How do we know? 7:28

Do real heads track things like "it"? The 2017 paper showed two heads apparently linking "its" to "the Law", the noun it referred to ¹. In 2019, Sarthak Jain and Byron Wallace, testing mostly recurrent models with attention, argued that attention weights largely don't explain a model's choices ¹⁶. Months later, Sarah Wiegreffe and Yuval Pinter replied that it depends what you mean by explanation, in a paper called Attention is not not Explanation ¹⁷.

Why it trains so fast 8:04

So what made it take over so fast? A big part was speed ^1,5. A recurrent network must finish word one before word two, before word three ^1,5. Attention scores every pair of words at once, as one big multiplication of tables of numbers, exactly the work the GPUs from episode five are built for ^1,13. The authors reported a record English-to-German translation score after three and a half days on eight GPUs, a fraction of the cost of the best earlier models ¹.

The T in GPT 8:38

In 2018, OpenAI's first GPT used a Transformer that only looks back, and Google's BERT one that reads both ways ^18,19. By 2020, transformers were sorting images too, cut into patches like words ²⁰. In reports from 2023 to 2025, the makers of GPT-4, Gemini, Llama, Mistral, DeepSeek and Claude all describe their models as transformers ^{21,22,23,24,25,26}. And in April 2025, an analysis by the journal Nature ranked the 2017 paper the seventh most-cited of this century ^27,28.

The trophy test falls 9:17

And the trophy test? By 2019, systems built on big pre-trained transformers, then trained on puzzles like these, scored better than ninety per cent ²⁹. In 2022, a review co-written by two of the test's creators called its defeat sobering ²⁹. It was a mistake, they wrote, to assume you need common sense to resolve pronouns, and there's little evidence these systems reason with common sense consistently well ²⁹. OpenAI's 2023 GPT-4 report, by contrast, listed a big Winograd-style test under 'commonsense reasoning' ²¹. So passing the test didn't settle the question it was built to ask ²⁹.

The price of looking at everything 10:02

There's a price. Every token scores every other, so the work grows with the square of the length ^1,13. Our thirteen tokens make a hundred and sixty-nine pairs. GPT-3's two thousand and forty-eight tokens make about four million, in every head of every layer ⁸. A million tokens would make a trillion ¹. Yet as of October 2026, Google and Anthropic say some of their models take a million tokens ^30,31. OpenAI said the same of its GPT-4.1 models in April 2025, and that month Meta said ten million for one model ^32,33. Some models take shortcuts: in Mistral's 2023 model, each layer looks back over only recent tokens ²⁴. And long isn't the same as well used: a 2023 study found models were worst at using information buried in the middle ³⁴.

Is attention really all you need? 10:59

Not everyone thinks the title is the last word. In December 2023, Albert Gu and Tri Dao proposed a model called Mamba, which drops attention altogether, so its cost grows only in a straight line with length ³⁵. Whether something else takes over is an open question.

Next 11:20

So attention lets every word look at the others and borrow their meaning, and doing it all at once let these networks learn on GPUs at enormous scale ^1,5. But learn from what, exactly? And how does a network that predicts the next token become a helpful assistant? How do you turn the whole internet into a chatbot?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I.… — arxiv.org
2. H. J. Levesque, E. Davis, L. Morgenstern, "The Winograd Schema Challenge", Proc. KR 2012 — cs.nyu.edu
3. H. J. Levesque, "The Winograd Schema Challenge", AAAI Spring Symposium on Logical… — commonsensereasoning.org
4. E. Davis, "Winograd Schemas and Machine Translation", arXiv:1608.01884 (2016) — arxiv.org
5. J. Uszkoreit, "Transformer: A Novel Neural Network Architecture for Language… — research.google
6. Wiktionnaire, "trophée" (nom masculin) and "valise" (nom féminin), read 15 October 2026 — fr.wiktionary.org
7. D. Jurafsky, J. H. Martin, Speech and Language Processing (3rd ed. draft, 19 August… — web.stanford.edu
8. T. B. Brown et al., "Language Models are Few-Shot Learners" (GPT-3), arXiv:2005.14165… — arxiv.org
9. T. Mikolov, K. Chen, G. Corrado, J. Dean, "Efficient Estimation of Word… — arxiv.org
10. T. Mikolov, W. Yih, G. Zweig, "Linguistic Regularities in Continuous Space Word… — aclanthology.org
11. M. Nissim, R. van Noord, R. van der Goot, "Fair is Better than Sensational: Man is to… — arxiv.org
12. T. Bolukbasi, K.-W. Chang, J. Zou, V. Saligrama, A. Kalai, "Man is to Computer… — arxiv.org
13. D. Jurafsky, J. H. Martin, Speech and Language Processing (3rd ed. draft, 19 August… — web.stanford.edu
14. D. Bahdanau, K. Cho, Y. Bengio, "Neural Machine Translation by Jointly Learning to… — arxiv.org
15. OpenAI, gpt-2 repository: src/encoder.py and the published 124M vocabulary files… — github.com
16. S. Jain, B. C. Wallace, "Attention is not Explanation", NAACL 2019 (arXiv:1902.10186,… — arxiv.org
17. S. Wiegreffe, Y. Pinter, "Attention is not not Explanation", EMNLP 2019… — arxiv.org
18. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, "Improving Language… — cdn.openai.com
19. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, "BERT: Pre-training of Deep… — arxiv.org
20. A. Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image… — arxiv.org
21. OpenAI, "GPT-4 Technical Report", arXiv:2303.08774 (March 2023) — arxiv.org
22. Gemini Team, Google, "Gemini: A Family of Highly Capable Multimodal Models",… — arxiv.org
23. Llama Team, AI @ Meta, "The Llama 3 Herd of Models", arXiv:2407.21783 (July 2024) — arxiv.org
24. A. Q. Jiang et al., "Mistral 7B", arXiv:2310.06825 (October 2023) — arxiv.org
25. DeepSeek-AI, "DeepSeek-V3 Technical Report", arXiv:2412.19437 (December 2024) — arxiv.org
26. J. Lindsey et al. (Anthropic), "On the Biology of a Large Language Model", Transformer… — transformer-circuits.pub
27. H. Pearson, H. Ledford, M. Hutson, R. Van Noorden, "Exclusive: the most-cited papers… — nature.com
28. University of Toronto, "U of T research on AI, global health among 25 most cited of… — gro.utoronto.ca
29. V. Kocijan, E. Davis, T. Lukasiewicz, G. Marcus, L. Morgenstern, "The Defeat of the… — arxiv.org
30. Google AI for Developers, "Long context" (Gemini API docs, last updated 13 October 2026) — ai.google.dev
31. Anthropic, Claude Docs "Context windows" (read 15 October 2026) — platform.claude.com
32. Meta AI, "The Llama 4 herd: The beginning of a new era of natively multimodal AI… — ai.meta.com
33. OpenAI, "Introducing GPT-4.1 in the API" (14 April 2025; read via Wayback 20250414192522) — openai.com
34. N. F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts",… — arxiv.org
35. A. Gu, T. Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces",… — arxiv.org

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.