Compute: the chips, clouds and data centers behind AI

What compute is and why it scales#

Compute is the arithmetic a computer does to train or run an AI model, counted in floating-point operations (FLOP): single multiplications and additions on numbers with fractional parts.123 A 2020 study by Kaplan and colleagues found that language-modeling performance improves smoothly and predictably as compute, model size (how many weights a model has) and data grow together, which gives developers a reason to keep scaling.456

That study tracked loss, a score of how badly a model predicts text, and loss fell as compute rose.74 Training and running a model come down mostly to matrix multiplication: multiplying big tables of numbers, such as the weights, against each other.82 GPUs and other AI chips pack many multiply-and-add circuits side by side, and their speed is counted in FLOP per second (FLOP/s).910 More weights and more data each add arithmetic, so growing both raises the compute bill.11 In a paper revised in June 2024, Pablo Villalobos and colleagues at Epoch AI argued that public human-written text cannot sustain such scaling beyond this decade.51211

In 2022, Hoffmann and colleagues found that model size and training data should grow equally, unlike the 2020 study, which had advised putting extra compute mostly into bigger models.1354 To test this, they gave a model called Chinchilla the same compute as an earlier model, Gopher, but far fewer weights and far more data.1413 The researchers reported that Chinchilla outperformed Gopher and much larger models such as GPT-3 across a wide range of tests.13

In October 2026, eleven companies pledged $2.4 billion in compute credits and science tools to the Genesis Mission Consortium, a Department of Energy initiative.1516

GPUs and accelerators#

AI runs on GPUs and other specialized chips because a neural network, the layered set of weights inside an AI model, mostly does matrix multiplication.2 That work splits into many small, separate calculations.2 A CPU, the ordinary processor, races through tasks one after another, while a GPU, a chip suited to graphics, works on thousands at once.17

Since its Volta generation, Nvidia's GPUs have had tensor cores, circuits built just for matrix multiplication, and other AI chips have similar units.1819 They also use low-precision formats such as FP16 and FP8, which store each number in fewer bits to save memory and time.2021 Often the limit is memory bandwidth, how fast data reaches the math units, so AI chips use high-bandwidth memory (HBM).2223 Fast links such as Nvidia's NVLink and InfiniBand networking let many chips share one job like a single machine.2024 CUDA, Nvidia's GPU programming platform, and libraries like cuDNN spare researchers from writing their own fast code for each new chip.2526

Google's early Tensor Processing Unit (TPU) was a custom ASIC, a chip made for one job, built around one large matrix-multiply unit to run trained neural networks.102718 It left out CPU and GPU features that raise average speed but make response times less predictable, keeping it small and low power.10 In 2017, Google reported the TPU ran its inference work about 15 to 30 times faster on average than the CPU and GPU of its day.1018 The paper started from a view held by many chip designers: big gains must now come from chips built for one kind of task.10

Data centers and power#

An AI data center is a building full of servers, storage and network gear, plus the power and cooling systems that keep it running.2829 It often needs huge amounts of firm power, meaning power that is steady and always available, which adds pressure to grids already facing capacity, reliability and affordability challenges.3031

In December 2024, the US Department of Energy released a Lawrence Berkeley National Laboratory report that projected data center energy usage could double or triple by 2028.32 In January 2025, the department wrote that steep growth in data center loads can affect regional grids, and that demand forecasts keep shifting.30

Hyperscalers are firms such as Google, Amazon, Meta and Microsoft that design, build and run massive data centers.31 Since 2024, hyperscalers have announced a series of high-profile deals with nuclear utilities, vendors and reactor designers, and a June 2026 Carnegie Endowment paper said nuclear power has entered the conversation as one of the few sources capable of delivering the firm, high-density electricity that data centers need.3133 The same paper found that the companies preferred to buy nuclear power as customers rather than own or develop plants.31 The paper said the deals signal enthusiasm, but that it is still unclear whether big tech is willing to make big bets on nuclear power.31

Training versus inference: building a model versus running it#

Training builds a model's weights in one large, costly job, while inference runs the finished model to answer requests and costs money again with every answer.3435

A training run ties up many chips, servers and network links for its whole length, so the chips need fast connections to each other.3637 Inference, also called serving, is different: the model writes its answer token by token, and each step waits mostly on reading the weights from memory, not on arithmetic.3839 So inference hardware is judged on memory bandwidth, meaning how fast a chip reads its memory, and on output per dollar and per watt.2235 Models that think longer before answering write more tokens, and every new user adds requests, so inference spending grows with use.4041

In August 2025, Google researchers published a study that measured the energy, carbon emissions and water used by inference in Google's system for serving its Gemini assistant.4142 They reported that the median text prompt in the Gemini app used 0.24 watt-hours of energy, which Google compared to watching nine seconds of television.4142 That figure covers serving one median text prompt, not training the model, and the cost comes back with every new prompt.4143 Experts quoted by The Verge, including Shaolei Ren, who has studied the water use of AI, said the study left out key data.42

Neoclouds: GPU-rental companies#

A neocloud is a company that rents out GPUs, the chips used to train and run AI models, as a service, separate from the big cloud companies. Our story.4445

A neocloud buys GPUs, runs them in data centers, and rents them by the hour to clients who would rather not buy their own. Our story.464745 Like any company buying costly equipment, it spreads each chip's cost over that chip's expected useful life, a practice called depreciation.46

Nebius, a neocloud based in Amsterdam, signed a deal worth up to $19.4 billion to supply Microsoft with computing power from a data center in New Jersey.47 Microsoft runs its own cloud, Azure, but it turned to outside providers to cover a shortage of capacity for AI work.47 The case shows that neoclouds sell to the big cloud companies as well as to AI developers.47

In August 2026, neocloud rents for Nvidia's H100 chip were stabilizing in a range below what the big cloud companies charged.45