Why did AI suddenly get good after 2012?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, we watched a network learn by passing the blame for its mistakes backwards. That recipe, backpropagation, was made famous by a 1986 paper ^1,2. Then, after an early burst of excitement, it spent years on the sidelines ². In the late 1990s, as three of its champions later wrote, neural networks were "largely forsaken" by machine learning ². Then, in 2012, one of them suddenly beat everything else at recognising photos ^3,4. So if the recipe was ready in 1986, why did it take until 2012?

A photo is just numbers 0:41

Start with why photos are hard. To a computer, a photo is a grid of numbers, one brightness value per dot, or three for colour. Slide the same object one dot to the right, and a whole set of those numbers changes, though the object hasn't. So before 2012, the best systems used features designed by people, recipes for spotting edges and corners, with a learned classifier on top ^4,3.

The boom before the winter 1:10

And AI had already boomed and bust once more ⁵. In the early 1980s, the hot idea was the expert system: piles of IF-THEN rules, written down by people from what experts knew ⁵. One, called XCON, configured computer orders for the company DEC from 1980, and by 1989 DEC estimated that XCON and its sister systems were saving it over forty million dollars a year ⁵. Membership of the main American AI society peaked at 16,421 in 1987 ⁵. By 1996, it was down to between four and five thousand ⁵.

Why rules broke 1:52

Hand-written rules are brittle ⁵. They break on problems just outside what they were written for, and they don't know what they don't know ⁵. A story told about the medical system MYCIN makes the point ⁵. Someone entered a male patient who'd had amniocentesis, a test done only in pregnancy, and it accepted that without complaint ⁵. Nobody had written the rule that men don't get pregnant ⁵. Promises outran results, and by the late 1980s people were calling it an AI winter ⁵.

Out in the cold 2:25

Neural networks didn't vanish. By the late 1990s, one was reading more than a tenth of all the cheques written in the United States ². But most researchers had moved on ². It was widely believed that a deep network would get stuck in a poor solution while training ². A few groups kept going, and from about 2006 they started getting results ^2,6.

A child's eyes 2:51

One researcher thought the missing piece was data ⁷. Fei-Fei Li pointed out that a child's eyes take in a new view about every 200 milliseconds ⁷. At five a second, twelve waking hours a day, that's about 237 million pictures by age three, by our own count ⁷. Her version: hundreds of millions, and nobody tells the child how to see ⁷. The standard collections of labelled pictures held tens of thousands at most ³.

Building ImageNet 3:23

In 2007, Li and her colleague Kai Li at Princeton started ImageNet ⁷. By Li's account, they downloaded nearly a billion pictures, then paid people online to sort and label them ^7,8. Almost fifty thousand people in 167 countries took part ⁷. Colleagues advised her to do something more useful for her career, and she once joked she'd reopen the dry-cleaning shop that had paid her way through college ⁷. By 2009, ImageNet had 3.2 million labelled pictures in over five thousand categories ⁸.

How big a jump 4:01

From 2010, ImageNet ran a yearly contest with 1,000 categories and about 1.2 million training photos ⁹. The contest it grew out of, called PASCAL, had 20 categories and about twenty-two thousand labelled images in 2012 ⁹. ImageNet's had 1.4 million in all: about sixty-six times the pictures ⁹. And instead of one category called dog, there were 120 dog breeds ⁹.

Five guesses 4:33

The score was the top-five error ^4,9. The system makes five guesses for each photo, and it only counts as wrong if the right answer isn't among them ^4,9. That's fair when a photo holds several things ⁹. In 2010, the winner got about 28 per cent wrong ⁹. In 2011, the winner still got about 26 per cent wrong ⁹.

Two thousand and twelve 4:57

In 2012, a University of Toronto team, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, entered a deep neural network later called AlexNet ^3,9,6. Trained only on the contest's own photos, it got sixteen per cent wrong, or fifteen with extra ImageNet pictures ⁴. The next best entry got twenty-six per cent wrong ⁴. Per thousand photos, that's about 164 misses instead of 262, more than a third fewer ⁴.

What a convolution does 5:34

AlexNet was a convolutional network, an idea rooted in Kunihiko Fukushima's work in the early 1980s, and in Yann LeCun's cheque readers ^2,7. A convolution is just a small grid of weights, a filter, sliding across the picture ². At each spot, multiply the pixels under it by the weights, and add them up. Work it through, and this filter scores twenty-one where dark meets bright, and zero elsewhere, so it's an edge detector. The network learns its filters for itself, and stacked layers turn edges into patterns, then parts, then objects ².

Ingredient two: game hardware 6:14

Every one of those positions can be worked out at the same time as all the others. For AlexNet's first layer alone, that's about a hundred million per picture, by our count ³. That's the kind of job graphics chips are good at, as we saw in the Silicon series. AlexNet trained for five to six days on two GeForce GTX 580s, gaming cards whose standard model had launched at 499 US dollars ^3,10,11. Its authors wrote it would improve "simply by waiting for faster GPUs and bigger datasets" ³.

Sixteen thousand cores 6:51

Around the same time, a team from Google and Stanford took another road ¹². They trained a network on ten million frames from YouTube videos, using a thousand ordinary computers with sixteen thousand processor cores, for three days ¹². Nobody labelled anything, yet parts of the network came to respond to faces, and to cat faces ¹². It was a different task, so the results don't compare. But one road used a room full of computers, for a network about seventeen times bigger, and the other used two graphics cards ^12,3.

Ingredient three: better methods 7:27

The third ingredient was better methods ^3,2. One was a simpler switch inside each neuron, called ReLU: if the input is negative, output zero, otherwise pass it straight through ³. The older S-shaped curves flatten out, so learning crawls ³. In one test, ReLU trained six times faster ³. Another was dropout: in training, neurons in some layers are each switched off at random, half the time, so none can lean on a particular partner ³. That helps stop the network memorising the practice exam ³.

Was 2012 really the start? 8:04

Not everyone thinks 2012 was the start ⁶. A 2022 study of AI's computing history, by researchers at Epoch AI, argues that 2010 fits better ⁶. Networks had run on graphics chips since 2006, and GPU networks had already won a traffic-sign contest ⁶. Speech recognition switched to deep learning between 2009 and 2012 ^6,2. The ingredients had been building for years ⁶. 2012 was when computer vision couldn't ignore them ^2,6.

Then it kept falling 8:39

Once the field switched, the error kept dropping ⁹. By 2014, the winner was below seven per cent ⁹. In 2015, a Microsoft team's much deeper network got under four per cent ¹³. It's often said machines beat people that year. But the human figure, about five per cent, came from one trained person checking 1,500 photos, while a less-practised second person, on fewer photos, got twelve per cent wrong ⁹.

Compute took off 9:10

And then the computing poured in ⁶. It's counted in floating-point operations, or FLOP: single multiplications or additions of numbers with fractions, like 3.14 ¹⁴. A 2022 study found that before 2010, the compute used to train notable AI systems doubled about every twenty months ⁶. Since then, about every six ⁶. Over ten years, that's sixty-four times more, against about a million times more ⁶. In May 2024, the research group Epoch AI put the growth at four to five times a year ¹⁵.

From two cards to sixteen thousand 9:50

AlexNet's training is estimated at about five hundred million billion operations ¹⁶. The biggest version of Meta's Llama 3.1, released in July 2024, took about 81 million times that, using Meta's own figure, on up to 16,384 GPUs ^17,16. That's about four point seven times more each year, in line with Epoch's four-to-five ^15,16. The biggest runs of 2025 are estimated at roughly ten times more again, though companies mostly don't publish the figures ¹⁶.

Not just brute force 10:28

But it wasn't only brute force ^14,18. A 2020 study by two OpenAI researchers found that by 2019, matching AlexNet took 44 times less computing than in 2012, thanks to better methods ¹⁴. Moore's law alone would have given about eleven times ¹⁴. The computer scientist Rich Sutton argued in 2019 that methods which use more computing win in the end ¹⁹. The cognitive scientist Gary Marcus had warned in 2018 that deep learning was data-hungry and would need other ideas too ²⁰. That debate is still going.

The recipe today 11:07

The data had costs too: in 2019, the ImageNet team reported problems in its own categories of people, which we'll come back to next season ²¹. But the recipe stuck. Huge datasets, warehouses of AI chips and better methods are behind today's chatbots, as of 2026, such as ChatGPT, Claude, DeepSeek, Gemini, Llama and Mistral ^16,22. Pictures were the 2012 breakthrough, but language needed more. So next time: how does a chatbot actually write a sentence?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. D. E. Rumelhart, G. E. Hinton, R. J. Williams, "Learning representations by… — nature.com
2. Y. LeCun, Y. Bengio, G. Hinton, "Deep learning", Nature 521, 436–444 (28 May 2015)… — cs.toronto.edu
3. A. Krizhevsky, I. Sutskever, G. E. Hinton, "ImageNet Classification with Deep… — proceedings.neurips.cc
4. ImageNet: "Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)" results (Task… — image-net.org
5. N. J. Nilsson, The Quest for Artificial Intelligence: A History of Ideas and… — ai.stanford.edu
6. J. Sevilla, L. Heim, A. Ho, T. Besiroglu, M. Hobbhahn, P. Villalobos, "Compute Trends… — arxiv.org
7. Fei-Fei Li, "How we're teaching computers to understand pictures", TED2015 (March… — ted.com
8. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-Fei, "ImageNet: A Large-Scale… — image-net.org
9. O. Russakovsky et al., "ImageNet Large Scale Visual Recognition Challenge",… — arxiv.org
10. Legit Reviews, "NVIDIA GeForce GTX 580 GF110 Fermi Video Card Review", 9 Nov 2010 — legitreviews.com
11. Gaming Nexus, "GeForce GTX 580 announced", 9 Nov 2010 — gamingnexus.com
12. Q. V. Le et al., "Building high-level features using large scale unsupervised… — arxiv.org
13. K. He, X. Zhang, S. Ren, J. Sun (Microsoft Research), "Deep Residual Learning for… — arxiv.org
14. D. Hernandez, T. B. Brown, "Measuring the Algorithmic Efficiency of Neural Networks",… — arxiv.org
15. J. Sevilla, E. Roldán, "Training Compute of Frontier AI Models Grows by 4-5x per… — epoch.ai
16. Epoch AI, "Data on AI Models" (notable models dataset, updated 14 Oct 2026; accessed… — epoch.ai
17. Llama Team, AI @ Meta, "The Llama 3 Herd of Models", arXiv:2407.21783 (July 2024) — arxiv.org
18. E. Erdil, T. Besiroglu, "Algorithmic progress in computer vision", arXiv:2212.05153… — arxiv.org
19. R. Sutton, "The Bitter Lesson", 13 March 2019 — incompleteideas.net
20. G. Marcus, "Deep Learning: A Critical Appraisal", arXiv:1801.00631 (Jan 2018) — arxiv.org
21. K. Yang, K. Qinami, L. Fei-Fei, J. Deng, O. Russakovsky, "Towards Fairer Datasets:… — arxiv.org
22. A. Ho, T. Besiroglu, E. Erdil et al., "Algorithmic progress in language models",… — arxiv.org

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.