What's in an AI Data Center?

And why does it use so much power?

Everything You Need To Know About AI Data Centers: Part 2

AI data centers are enormous, occupying vast amounts of space—but what’s actually inside them? Why do they need to be so massive?

First of all, an AI data center is an advanced version of a traditional data center. AI requires significantly more computing power than conventional data centers, which are mainly used for storage, email services, websites, and applications. As a result, AI data centers have a much higher computing density and are therefore considerably larger.

What’s inside an AI data center?

At the heart of every AI data center are clusters of high-density GPUs. Unlike traditional data centers, which rely primarily on CPUs, AI workloads require GPUs to process massive amounts of data in parallel.

A single GPU doesn’t take up much space, but AI data centers contain thousands of them, packed into server racks. Those racks quickly fill entire buildings. And some of those buildings are true monsters.

The largest category is known as hyperscale data centers. These facilities typically consume 100 MW or more of power—roughly equivalent to the electricity used by 80,000 U.S. homes. The largest hyperscale projects announced so far exceed 10 GW, meaning they would consume more electricity than 8 million homes, or roughly twice the total power demand of some U.S. states.

And that brings us to another question.

If AI data centers contain so many GPUs, and today’s average AI server rack consumes between 80 and 100 kilowatts (kW) of power, how much electricity can these facilities—and hyperscalers—actually use?

Feeding the AI Beast

Now we’re getting to one of the hottest topics surrounding AI data centers.

In 2020, OpenAI’s GPT-3 model was trained using 10,000 NVIDIA V100 GPUs in a Microsoft data center over approximately 15 days. The training consumed an estimated 1.29 gigawatt-hours (GWh) of electricity—enough to power about one million homes for an hour.

Just a few years later, GPT-4 reportedly required roughly 40 times more energy: around 50 GWh. That’s enough electricity to power approximately 5,000 homes for an entire year, or about 0.02% of California’s annual electricity consumption.

Looking ahead, next-generation AI servers expected around 2028 may consume 1 megawatt (MW) each—roughly enough electricity to power 1,000 average U.S. homes—while fitting into a space no larger than a refrigerator.

How is it even possible for something that small to consume so much energy?

Each GPU inside an AI server contains billions of transistors. NVIDIA’s Blackwell GPU, for example, contains more than 200 billion transistors. Every time those transistors switch on and off, they generate heat. Pack thousands of these chips together, and the result is an enormous amount of concentrated heat.

At full load, engineers sometimes describe the heat flux from these chips as approaching extreme levels, requiring highly advanced cooling systems. Those cooling systems rely on significant amounts of clean water to keep the hardware operating safely.

But we’re now closer to 2030 than we are to 2020. What happens as hyperscale AI infrastructure continues to expand? Will electricity demand continue to grow? Will even more water be needed for cooling?

These are important questions—and ones we should all be asking. Find the answers to these and many other questions surrounding AI data centers in our Report.

Our families, friends, and communities need to understand what’s happening. This isn’t just another environmental talking point. AI infrastructure has real implications for energy, water, local communities, and the electrical grid.

We also encourage you to explore the data on our map, ask questions, and join the conversation on social media and in your community.

Don't stop here. Read the rest of the series.