Why does AI need so many chips, and so much power?

The full story

The episode's narration, word for word. Numbers link to the sources below; timestamps play that moment.

The question 0:00

Last time, we watched an agent work in a loop, calling a model again and again. Every one of those calls is arithmetic, done on a chip, in a building, drawing power from a grid. In 2025, the world's data centres used about four hundred and eighty-five terawatt-hours of electricity, and the International Energy Agency expects that to roughly double by 2030 ¹. So why does AI need so many chips, and so much power?

Multiply, then add 0:32

Start inside the network. Back in episode three, a neuron multiplied each input by a weight and added up the results ². Here's a tiny layer, with three inputs and two neurons. Two times one, one times minus two, three times one: add them, and the first neuron says three. The second says four. That's six multiply-and-adds, one for every weight ². A whole network is mostly this, over and over ².

Counting the work 1:05

For a model like this one, its size tells you its workload ³. Every weight costs about two operations, a multiply and an add, for every token it writes ³. Meta's largest Llama 3.1, from July 2024, has four hundred and five billion weights ⁴. That's about eight hundred and ten billion operations per token, by our count ^3,4. A five-hundred-token reply takes about four hundred trillion ^3,4.

Why a graphics chip 1:36

The good news is that, within a layer, none of those sums waits for any other ². The Silicon series showed how graphics chips exploit exactly that: thousands of simple units, all working at once. Nvidia's H100, from 2022, has special units, called Tensor Cores, that do small blocks of multiply-adds in one step ⁵. Without its sparsity trick, it does about a thousand trillion sixteen-bit operations a second ^6,5. Use eight-bit numbers instead, and it does twice as many ⁶. Google's first AI chip, the TPU, used eight-bit maths for the same reason ².

Fetching costs more than maths 2:19

But the arithmetic isn't the expensive part ⁷. In 2014, the Stanford engineer Mark Horowitz tallied the energy of each step on chips made with a then-common 45-nanometre process ⁷. A sixteen-bit multiply took about one picojoule, a million-millionth of a joule ⁷. Fetching a number from main memory took hundreds of times more, by our count ⁷. Think of a chef: chopping is quick, but the walk to the cool room isn't. The analogy breaks, because memory delivers many numbers at once, but the cost of moving data is real ^7,8.

Answering, one token at a time 2:57

Now, using a trained model is called inference ⁸. To write each new token, a model like this has to read every one of its weights from memory ⁸. At sixteen bits, that's two bytes each, so this Llama's weights fill about eight hundred and ten gigabytes ^8,4. One H100 holds eighty ⁶. Meta wrote that the model didn't fit on one eight-GPU server, so it spread it over sixteen chips, or shrank most of them to eight bits ⁴.

Why it serves crowds 3:30

And here's the twist. An H100 can do about two hundred and ninety-five operations for every byte it reads from memory, by our count ^6,5. One person's next token needs only about one ^3,8. So for a single conversation, the arithmetic would sit idle more than ninety-nine per cent of the time ^6,8. The fix is batching: answer many people at once, so each weight, once fetched, is used for all of them ⁸. That's why these services run best at huge scale ⁸.

Training: six per weight 4:03

Training costs more per token ³. The forward pass is two operations per weight, and passing the blame backwards, from episode four, takes about twice that again, so six in all ³. Llama 3.1's biggest model trained on fifteen point six trillion tokens ⁴. Six, times four hundred and five billion, times that, gives about thirty-eight trillion trillion operations ^3,4. That's the total from episode five, and now you know where it comes from ⁴.

Fifty-four days 4:37

Meta's paper tells what that took ⁴. It used up to sixteen thousand H100s, each rated at seven hundred watts ⁴. In one fifty-four-day stretch, the job was interrupted 466 times ⁴. Most of those were unplanned, and about seventy-eight per cent of the unplanned ones were down to hardware, confirmed or suspected, mostly GPUs ⁴. Even the weather showed up: the run's speed rose and fell by one to two per cent each day, slowing in the midday heat ⁴.

Swinging the grid 5:11

And when tens of thousands of GPUs pause together, say to save their progress, the data centre's power use can swing by tens of megawatts in an instant ⁴. Meta's engineers wrote that this was stretching the limits of the power grid ⁴. Chips don't just use power, they jolt it about.

The training bill 5:31

How much energy is that? Meta's model card lists about thirty-one million GPU-hours to train the largest model ⁹. At seven hundred watts each, that's about twenty-two gigawatt-hours, by our rough count ⁹. Real chips often draw less than their rating, but cooling and other equipment add more ^10,11.

Train once, answer billions 5:52

But a model is trained once and then used over and over ¹². When Google measured its own machine learning energy, in one week of April in 2019, 2020 and 2021, about three-fifths went on answering and two-fifths on training ¹². Machine learning was ten to fifteen per cent of Google's total energy use ¹². That's one company, years ago, but it shows where the cost can end up.

The buildings 6:21

All this happens in data centres, which the Silicon series toured. The International Energy Agency says their electricity use rose seventeen per cent in 2025, to about four hundred and eighty-five terawatt-hours ^13,1. Electricity use by data centres focused on AI grew by half ¹. That's like a steady fifty-five gigawatts, around the clock, by our count ¹. The agency projects about nine hundred and fifty terawatt-hours by 2030, around three per cent of the world's electricity ¹. That's a projection, not a promise.

Closer to home 6:59

In Australia, the energy market operator forecast in August 2026 that data centres on the national grid, which covers the eastern states and South Australia, will go from about five terawatt-hours a year to about thirty-four within a decade ¹⁴. That would take them from under three per cent of the grid's consumption to about thirteen ¹⁴. It names AI and cloud services as the drivers ¹⁴. But it also notes that more than forty per cent of data centre projects since 2025 have dropped out or slipped back ¹⁴.

What does one question cost? 7:35

So what about one question to a chatbot? A widely quoted 2023 estimate said about three watt-hours, but it assumed very long questions and answers, on older chips ¹⁰. In February 2025, the research group Epoch AI estimated about a third of a watt-hour for a typical ChatGPT query, from public information ¹⁰. In June 2025, OpenAI's chief executive Sam Altman put ChatGPT's average at about a third too, without saying how ¹⁵. In August 2025, Google said its median Gemini text prompt used about a quarter of a watt-hour, counting idle machines and cooling ¹⁶. In July 2025, Mistral published carbon and water per answer instead, from a study with France's ecological transition agency ¹⁷. Google's and OpenAI's figures haven't been checked independently ^16,15. Other makers, including Anthropic, which makes Claude, hadn't published a per-prompt figure as of October 2026 ¹⁸.

A kettle's worth 8:44

Let's put that in everyday terms, assuming a third of a watt-hour per simple text prompt ¹⁰. Heating a litre of water from twenty degrees to the boil takes at least ninety-three watt-hours, and a real kettle uses a bit more ^19,20. So one kettle boil is roughly three hundred prompts, by our count ^19,20. A ten-watt LED globe would use a prompt's worth in about two minutes, by our count ¹⁰. An hour of streaming video in 2019, counting the screen it played on, was about two hundred and fifty prompts ²¹. Those prompt figures leave out your own phone or computer ¹⁶.

Not all prompts are small 9:23

But that's a short, simple question ¹⁰. Paste in a long report, about a hundred thousand tokens, and Epoch's estimate rises to about forty watt-hours, almost half a kettle ¹⁰. The International Energy Agency says making video, long reasoning and agent tasks like last episode's can use hundreds or thousands of times more energy per request than simple text ¹.

Small each, big together 9:50

In July 2025, OpenAI said ChatGPT was getting two and a half billion prompts a day ²². At a third of a watt-hour each, that's about eight hundred and fifty megawatt-hours a day, by our count ^15,22. Over a year, that's well under a tenth of a per cent of all data-centre electricity ¹. And the energy to train that one Llama model would run those prompts for about twenty-five days ^9,22. So simple chat is a small slice ¹. The growth comes from many more people using AI, and from heavier uses that are spreading fast ¹.

Cheaper each, more in total 10:30

Both trends are real at once. Google says the energy per median prompt fell thirty-three times in the year to May 2025 ¹⁶. Yet total demand keeps climbing ^1,13. What that means for emissions, for water, for local grids and for who pays, is the subject of a whole episode next season.

Looking back 10:52

So that's the season. We started with a word everyone means differently. We saw machines learn from examples, built from weighted sums, improved by walking downhill on their errors. Data, chips and better methods made them good after 2012. Chatbots predict one token at a time, guided by attention, then shaped to be helpful. That's why they can sound sure and be wrong. Pictures come out of noise, agents work in loops, and almost all of it is multiply-and-add, at enormous scale, running on electricity.

Next season 11:30

Next season, we turn from how AI works to what it means for people, from the evidence rather than the headlines. And the first question is one you may have asked yourself. Is AI going to take my job?

Sources

Every factual claim in the episode is tied to one of these. Spotted an error? Tell us.

1. IEA, Key Questions on Energy and AI (2026), Executive summary — iea.org
2. K. Sato, C. Young (Google), "An in-depth look at Google's first Tensor Processing Unit… — cloud.google.com
3. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown et al. (Johns Hopkins University,… — arxiv.org
4. Llama Team, AI @ Meta, "The Llama 3 Herd of Models", arXiv:2407.21783 (July 2024),… — arxiv.org
5. Nvidia Technical Blog, "NVIDIA Hopper Architecture In-Depth" (22 March 2022) — developer.nvidia.com
6. Nvidia, H100 Tensor Core GPU product page and specifications (accessed 15 Oct 2026) — nvidia.com
7. M. Horowitz, "1.1 Computing's energy problem (and what we can do about it)", ISSCC… — gwern.net
8. Nvidia Technical Blog, "Mastering LLM Techniques: Inference Optimization" (17 Nov 2023) — developer.nvidia.com
9. Meta, Llama 3.1 model card ("Training Energy Use", GPU hours per model; accessed 15… — github.com
10. J. You, "How much energy does ChatGPT use?", Epoch AI, Gradient Updates (7 Feb 2025) — epoch.ai
11. Uptime Institute, Global Data Center Survey 2025 (Report 180, July 2025) — datacenter.uptimeinstitute.com
12. D. Patterson, J. Gonzalez, U. Hölzle, Q. Le, C. Liang et al., "The Carbon Footprint of… — arxiv.org
13. IEA news, "Data centre electricity use surged in 2025, even with tightening… — iea.org
14. AEMO, 2026 Electricity Statement of Opportunities for the NEM (August 2026),… — aemo.com.au
15. S. Altman, "The Gentle Singularity" (10 June 2025) — blog.samaltman.com
16. Google Cloud Blog, "Measuring the environmental impact of AI inference" (21–22 Aug 2025) — cloud.google.com
17. Mistral AI, "Our contribution to a global environmental standard for AI" (22 July 2025) — mistral.ai
18. G. Pakula (Celsius), "Which AI uses the least energy? ChatGPT vs Claude vs Gemini"… — projetcelsius.com
19. HyperPhysics (Georgia State University), "Specific Heat" (water 4.186 J/g·°C) — hyperphysics.phy-astr.gsu.edu
20. D. M. Murray, J. Liao, L. Stankovic, V. Stankovic (University of Strathclyde),… — strathprints.strath.ac.uk
21. G. Kamiya, IEA commentary, "The carbon footprint of streaming video: fact-checking the… — iea.org
22. TechCrunch, "ChatGPT users send 2.5 billion prompts a day" (21 July 2025; OpenAI's… — techcrunch.com

Researched and scripted with AI assistance, fact-checked claim by claim, with synthetic narration and diagrams drawn in code. How we make episodes.