Data centers, the windowless, anonymous, boxy structures that few people even notice, have become a rare unifying force across the polarized American landscape this year. Everyone seems to hate them.

The explosion in data center construction has been fueled by intense interest in artificial intelligence, a technology that demands unprecedented amounts of computing power. Fitch Group Inc. estimates that the top five hyperscalers will spend $750 billion on data center construction this year, with three-quarters earmarked for AI.

But skepticism about AI’s impact on jobs, combined with perceptions about data centers’ disruptive impacts on communities, power grids and water supplies, have stoked a lot of outrage.

A Heatmap survey published last week found that three-quarters of Americans now oppose the construction of new data centers in their communities. Activists have organized more than 130 protests across dozens of states. The issue has even become a talking point in the upcoming midterm elections.

It’s hard to argue with the facts and alarming projections about data centers’ power and water consumption and associated environmental impact.

In 2023, data centers consumed roughly 4.4% of all electricity in the United States, a figure projected to triple by 2028. Data center power consumption is on track to double globally by 2030, reaching an amount equivalent to Japan’s entire annual electricity consumption. Some computer scientists estimate that data centers could consume up to 20% of the world’s electricity by 2035.

This appetite for power is mirrored by an equally voracious demand for water needed to keep rows of high-density server racks from overheating. A single large data center can consume up to 5 million gallons of water daily, equivalent to the use of a city of 50,000 people. The United Nations estimates global AI demand will consume Denmark’s total annual water usage next year.

Mechanical differences

Why does AI consume so much energy? In short, it’s because the mechanics of AI processing defy the traditional economics of software and cloud computing. Software companies write code once and host it on multitenant servers where the incremental cost of adding new users is negligible.

Generative AI has shattered that model, forcing the technology sector to cope with new challenges in thermodynamics, material limits and resource scarcity. Behind the clean interface of modern chatbots lies a physical infrastructure of unprecedented scale, a ravenous mix of processors, high-speed fiber networks, water-cooling towers and electric grids operating at their physical limits.

“Traditional software has very low marginal cost because computation happens primarily on the user’s device or cheaper multitenant infrastructure,” said Andrew Marshall, vice president of developer relations at Yugabyte Inc., maker of a distributed PostgreSQL database. “AI inference incurs a real computational cost for every interaction.”

That means AI is unlikely ever to approach the scalable economics of traditional software. “Conventional software runs one code path for up to millions of users,” Marshall said. “An AI application runs a different computation for each one. That’s what makes it worth paying for, and it’s what defeats the caching and code reuse that give software its margins.”

A single query submitted to an AI assistant such as OpenAI PBC’s ChatGPT requires up to 10 times more electricity than a traditional Google search, according to the International Energy Agency. Generating a five-second video using generative AI models consumes as much electricity as running a household microwave oven nonstop for over an hour.

AI is also currently feeling the impact of Jevons Paradox, a sobering phenomenon first observed by 19th-century economist William Stanley Jevons: Even if processing becomes more efficient, demand increases to the point that total usage outweighs the per-unit savings, raising overall consumption.

Hyperscalers, neocloud providers and dozens of startups are working on ways to make AI processing less energy-intensive, but “demand is still skyrocketing for more and more capacity,” said Ramesh Chettuvetty, senior vice president of AI product and business at Lightbits Labs Ltd., which is building an intelligent cache orchestration engine. Although Lightbits Labs says its technology can improve the capacity of existing graphics processing units up to 16-fold, “there is not going to be any impact on [total] GPU capacity,” he said. “It’s not possible to meet demand right now.”

AI is also so new that forecasting demand is, at best, an educated guess. Though model providers have been slashing token costs over the past two years, AI projects remain a highly unpredictable operational expense rather than a fixed capital asset. Gartner Inc. predicts that at least half of generative AI projects will overrun their budgets through 2028.

All of this raises the question of why AI is so expensive and whether technical innovations can push it further down the price/performance curve relative to traditional software. The short answer: Not likely.

GPU tax

At the root of the AI cost crisis is the physical hardware required to execute high-performance calculations. Unlike conventional software that runs on general-purpose central processing units, generative AI models rely almost exclusively on specialized parallel processors, primarily graphics processing units, or Google LLC’s tensor processing units, to run AI workloads more efficiently. This specialization has created acute supply chain constraints, transforming AI infrastructure from a software engineering challenge into a highly capital-intensive hardware acquisition race.

The high cost of GPUs is the result of two interrelated factors: semiconductor fabrication costs and the physical limits of memory. High-performance chips require hundreds of steps and specialized resources before they ever reach a server rack. For example, the chips processing AI workloads must be fabricated using “ultrapure” water to rinse away microscopic silicon residue created during manufacturing. A single semiconductor fabrication facility can consume up to 10 million gallons of ultrapure water daily. Because it takes roughly 1.5 gallons of tap water to produce a gallon of ultrapure water, a typical chip factory draws 15 million gallons of municipal water every day, equivalent to the use of about 33,000 households.

Then there’s the “memory wall.” An LLM doesn’t just process a query; it must load its entire weight matrix — consisting of hundreds of billions of parameters — into local memory. To handle the intense data transfer rates required for these operations, hardware manufacturers must deploy specialized, expensive high-bandwidth memory that stacks memory chips vertically to speed up data transfer.

This has triggered a brutal global resource squeeze. AI companies are currently purchasing an estimated 70% of the world’s supply of high-end computer memory, according to The Atlantic, leading to severe shortages elsewhere. The prices of consumer computer memory and hard-drive storage have skyrocketed as a result, with the costs of some laptops rising by as much as 50%.

Despite the astronomical cost of these hardware clusters — a single high-end GPU can cost tens of thousands of dollars — the actual efficiency of the hardware in production is surprisingly low. GPU clusters often operate at an average utilization rate of just 10% to 12%.

That’s a direct consequence of memory starvation. Because GPUs compute data much faster than memory architectures can supply it, the processors spend a lot of their active clock cycles waiting for memory to be fetched.

To compensate, operators frequently overprovision their capacity, reserving clusters at every layer of the stack to ensure they can handle sudden, unpredictable spikes in traffic. The result is a lopsided cost structure with operators paying full price for continuous, maximum-wattage hardware capacity while utilizing only a small part of it.

“AI gets expensive because you reserve capacity at every layer and use a fraction of it,” said Yugabyte’s Marshall. “Demand is spiky, and nobody wants to be the layer that runs out.”

Arthur Rasmusson, director of AI architecture at Lightbits Labs, recalled working with an LLM provider that ran a billion-dollar cluster at 10% utilization most of the time to handle occasional traffic surges. “You might be shocked,” he said, at typical utilization rates.

Training vs. inference

Model training is the most resource-intensive stage of the AI lifecycle, but it is not the biggest user of power and water over time. Training involves feeding massive datasets into a neural network to adjust its billions of parameters, a process that requires thousands of high-end processors running at maximum capacity for months.

Training’s power requirements are staggering. The Economist reported that Meta Platforms Inc.’s Llama 3.1 model required 27.5 gigawatt-hours of energy to train, enough power to supply 7,500 American homes for a year.

But training is a fixed, onetime capital event that can be amortized over millions of future transactions. Inference, which is the processing of live workloads, is the much bigger expense. Jefferies Financial Group Inc. analyst Brent Thill has estimated that inference accounts for 96% of the energy consumed in AI data centers, according to The Economist.

That cost is a direct consequence of two architectural limitations in modern deep learning: quadratic complexity and the autoregressive execution loop.

Traditional software is resource-efficient because it typically scales logarithmically or linearly. That means that as the input size grows, the computational time required to process it increases slowly.

LLMs scale quadratically, meaning that they must calculate the mathematical relationship between every single word or token in a prompt and every other word. Doubling the size of the input document therefore quadruples the amount of memory and computation required.

That scaling penalty is exacerbated by autoregression, a technique in which a model predicts the next data point in a sequence using its own past outputs as inputs. Autoregression compensates for the fact that computers can’t reason like humans by mimicking the thought process using probability.

Humans can formulate an entire sentence in their heads, but an LLM must execute a complete pass through its neural network to predict a single next token or chunk of data that the model uses to read, write and process information. The output is appended to the previous text, and the entire combined string is fed back into the model to predict the next token. It assumes that the future will follow past patterns.

That means that to generate a 1,000-token response, the GPU must run its billions of parameters through a mathematical loop 1,000 times. Every interaction is a resource-intensive computation that can’t be easily cached or bypassed. The result is that long conversations, document summaries or multiturn software development tasks can quickly become extremely computationally intensive.

The simplest way to reduce costs and power consumption is to ask the AI model to do less. Loading a model up with millions of data points is essentially wasting GPU capacity on calculations that could be done on a desktop computer.

“The fact that a model can ingest hundreds of thousands or millions of tokens does not mean it should,” said Varqa Abyaneh, founder and CEO of Opetek Ltd., the developer of an AI reasoning system for capital markets.

LLMs are good at tasks like understanding ambiguous questions, decomposing complex problems and selecting analytical approaches, he said.  Conventional computers are good at performing calculations across millions of data points and can do so at much lower cost.

Opetek’s AI reasoning system, called Arius, separates the data the model requires from data that can be processed more cheaply in a Python program or Excel. The approach has yielded over 90% cost reductions in some financial scenarios.

Abyaneh said the approach can be used in any data-intensive scenario. “The goal should not be to minimize reasoning,” he said. “It should be to spend reasoning where reasoning creates value.”

Agentic frontier

The financial and physical demands of AI inference are magnified with the ongoing shift from simple, human-driven chat interfaces to autonomous agentic workflows. Unlike chatbots that wait for a prompt, AI agents can operate independently, carrying out multistep business workflows, executing tool calls and interacting directly with other software.

This causes a dramatic expansion in the volume of variable tokens. Human users are physically limited by how quickly they can read and type, but a machine-to-machine agentic loop can execute thousands of transactions in seconds. Anthropic PBC has estimated that multi-agent systems consume about 15 times as many tokens as a single-turn human chat session. International Data Corp. has projected that the number of actively deployed AI agents worldwide will exceed 1 billion by 2029, about 40 times as many as were in use last year.

The reason agents are so rapacious is that they don’t actually think but loop repeatedly through the same data. “If an engineering assistant is tasked with fixing an application bug, it runs a build, encounters a failure, and invokes local tools to investigate,” Avichay Har-Tuv, finops team lead at CloudZone Inc., wrote in an article reviewed by SiliconANGLE.

“To make a decision, it pulls thousands of lines of verbose container logs, deep JSON structural payloads and identical database schemas, moving the entire block back into the cloud LLM’s context window,” he wrote. “If the first fix fails, the agent repeats the loop.”

Each time that happens, the agent retransmits the same database schemas and metadata across the network to a remote endpoint.  “The overwhelming majority of data transmitted during these multiturn sessions is not high-value logical code or intellectual property, but infrastructure noise,” Har-Tuv wrote.

Token waste

In a video tutorial on the YouTube channel Computerphile, Michael Pound, an associate professor of computer science at the University of Nottingham, demonstrated how such “token waste” can consume 60,000 to 100,000 tokens in a matter of minutes for a simple bug fix. Although tokens cost only a fraction of a cent each, the costs – and power demands – can quickly add up across hundreds of tasks.

“It’s easy for an agent to spin up a lot of cycles of time without you even asking for it,” said Dave McCarthy, group vice president of cloud and datacenter infrastructure at IDC. “There aren’t many circuit breakers.”

Human inefficiency doesn’t help. AI is so new that few organizations have reconfigured the data and processes needed to help models perform at peak efficiency. Gartner said the budget overruns it forecast will be largely the result of fundamental deficiencies such as poor architectural designs and a lack of operational controls.

“If an AI system doesn’t know what a field means, which metric is authoritative, where the data came from or whether it can be trusted, it has to figure those things out while it’s working,” said Animesh Kumar, co-founder and chief technology officer of The Modern Data Company Inc., creator of a platform that contextualizes data. “Retries, unnecessary context and using a powerful model for a relatively simple task all add more compute.”

Fragmented data sources introduce overhead by requiring AI models to pull information from multiple databases, increasing duplication and consuming tokens, said Michael Gale, chief marketing officer at EnterpriseDB Corp., which sells a commercial version of the open-source PostgreSQL database management system.

Gale likens a DBMS to a refrigerator, which is only opened and closed occasionally. AI essentially accesses the refrigerator constantly, drawing power each time.

“In an AI world, you have to pull data literally 86,000 seconds a day,” he said. He estimates that organizations can cut inference costs by 10% by vectorizing their data, allowing multiple sequential operations to execute in parallel. “If you can solve the energy consumption issue at the data layer, it gives you a lot more agility to handle some of the bigger stuff,” he said.

Mitigations

Data center operators and model developers are acutely aware that the current brute-force approach to scaling AI is economically and environmentally unsustainable. Numerous efforts are underway to reduce power demands without degrading accuracy or performance.

The most promising of these in the short term is quantization. In standard machine learning, model parameters are stored as highly precise 32-bit floating-point numbers. Quantization compresses these weights down to as little as four bits. Reducing the size of each parameter can shrink a model’s memory footprint by over 80%, allowing networks to run on cheaper hardware without a meaningful drop in output quality.

At the architectural level, mixture of experts designs are delivering substantial operational savings in some cases. Instead of activating a massive, monolithic neural network for every query, an MoE model divides its parameters into specialized sub-networks or “experts.” When a user submits a query, a routing algorithm determines which expert is best suited for the task and activates only that specific pathway.

If a user asks a coding question, for example, only the programming experts light up, leaving most of the network dormant. Google LLC researchers have estimated that MoE can reduce computation and data transfer volumes by 10- to 100-fold.

Model distillation uses a massive, high-performing model as a “teacher” to train a highly efficient, compact “student” model that can execute specific tasks at a small fraction of the cost. The student model learns to match the teacher’s detailed probability patterns rather than processing raw data labels, allowing it to capture the deeper reasoning and nuances of the larger model.

Microsoft Corp. researcher Alexia Jolicoeur-Martineau has pioneered tiny recursive models that achieved success on complex logic tasks in biology and electrical engineering. Her work has demonstrated that highly structured, small-scale architectures can solve well-defined problems without the overhead of massive foundation models.

“Not every enterprise task requires the largest or most powerful model,” said The Modern Data Co.’s Kumar. “Matching the model to the task can allow smaller or specialized models to handle a significant amount of work at lower cost.” GPU king Nvidia Corp. estimates that up to 70% of current LLM queries could be handled by SLMs without a meaningful drop in performance.

IDC’s McCarthy said the relative newness of AI means organizations are struggling to understand how to use the technology efficiently and wasting resources in the process. “Every time there’s a new technology wave, we see people throwing the kitchen sink at it,” he said. “You don’t always need the latest and greatest GPU or model. But organizations don’t have a lot of history to work with.”

Some of the techniques that can yield the greatest efficiency benefits are already well understood. Retrieval-augmented generation reduces unnecessary steps by providing contextually relevant information. Persistent memory allows agents to reuse prior work rather than reconstruct context from scratch. Both are tried-and-tested techniques that cut processing overhead. “The greatest efficiency gains come from eliminating duplicated retrieval and repeated inference without weakening the quality of the context provided,” said Yugabyte’s Marshall.

Context optimization layers are also showing promise. Working from the assumption that prompts often contain a lot of redundant information, content optimization techniques intercept and streamline data before it ever reaches the LLM. The open-source Project Headroom pre-processes heavy payloads locally, strips out syntax boilerplate, isolates log files and substitutes lightweight cryptographic hashes for long text streams to reduce token consumption up to 95% without affecting accuracy.

Selecting the right model can also significantly impact costs and power consumption. “The answer is not to use less AI; it’s to be smarter about where you use it, match the right models to each use case, and architect efficient context management and data processing,” said Gonçalo Borrêga, senior director of product management at OutSystems Inc. “Flexibility is imperative. If another model can do the same job at a lower cost six months from now, companies should be able to switch without rebuilding their entire system.”

Users should also take return on investment into account. “What matters is whether value grows faster than cost,” Borrêga said. “If an agent can reduce a two-hour process to three minutes, paying for inference can still produce a strong return.”

Next-generation infrastructure

Although software optimizations help, achieving greater efficiencies depends more on overhauling physical infrastructure and computing hardware. Promising advances there include specialized silicon architectures, predictive memory management, carbon-aware grid scheduling and advanced thermodynamic engineering.

A major hardware initiative is to shift processing from general-purpose GPUs to specialized application-specific integrated circuits and TPUs. Google, which has been co-designing its own TPUs for over a decade, says its Ironwood custom inference chip is 30 times more energy-efficient than its earliest models.

Tech giants are also transitioning their data center fleets to direct liquid cooling and liquid immersion systems, which completely submerge servers in nonconductive synthetic oil that conducts heat but not electricity.

Numerous startups are rethinking how software interacts with hardware to eliminate processing bottlenecks. Two-year-old startup Mindbeam AI Inc. recently released an open-source inference framework that it says can run LLMs on commodity consumer CPUs, bypassing the GPU bottleneck entirely for certain workloads.

Mindbeam’s approach constrains neural network weights to just three values, eliminating the complex floating-point multiplication operations that chew up processing cycles. The firm said its approach delivers a 17- to 96-fold improvement in CPU throughput, while slashing memory consumption.

Lightbits Labs, ScaleFlux Inc. and FarmGPU Inc. are collaborating on an architecture to ease AI inference bottlenecks caused by limited GPU memory. LightInferra software stores and reuses key-value cache data across nonvolatile memory express storage and managed GPU inference infrastructure to predict when data will be needed and moves it closer to processors. Lightbits says it can triple inference requests on existing GPUs while cutting power and infrastructure costs by 65%.

Inferra by Lightbits Labs is a “predictive prefetch” algorithm that analyzes upstream and downstream signals to predict what data the processor will need next and streams granular memory blocks only when needed. The company, which will release Inferra on Sept. 9, said it can raise GPU utilization from an average of 10% to 12% to more than 75% while enabling 16 times more concurrent inference sessions on existing GPU infrastructure.

Groq Inc. has raised $650 million for a chip design called a language processing unit that accelerates AI inference by bypassing GPU bottlenecks, delivering extreme token-generation speeds. SambaNova Inc. has raked in $1 billion to build an inference chip that it said can speed GPU processing up to fivefold.

Cerebras Systems Inc. is hoping to steal some of Nvidia Corp.’s market share with a wafer-scale architecture that bypasses the memory constraints that slow conventional GPUs by keeping model weights in on-chip memory. The company says its approach can boost throughput by 500%.

Despite efforts to displace GPUs, “they retain a major advantage because of their mature software ecosystem, flexibility and ability to support rapidly changing models,” said Arun Chandrasekaran, distinguished VP analyst at Gartner Inc.

AI giants are also shifting from static grid consumption to carbon-aware computing. Nvidia claims to have significantly reduced GPU power requirements in its latest Vera Rubin platform by compressing the numerical values that an AI model calculates to determine how strongly each word or token should relate to other tokens when generating a response, thereby minimizing unnecessary calculations and data movement. Its DSX MaxLPS framework dynamically coordinates power limits across racks and workloads, enabling data center operators to run up to 40% more GPUs within the same power budget.

Google’s system for Carbon-Intelligent Compute Management, for another example, automatically analyzes day-ahead carbon intensity forecasts and generates schedules that limit computing resources available to flexible background workloads during peak grid strain.

The road ahead

As promising as these initiatives are, they may ultimately founder on the shoals of Jevon’s Paradox. In its most recent earnings announcement, Nvidia noted that its growth is constrained by supply, indicating that current demand is nearly limitless.

“Better inference efficiency and hardware improvements can offset” some of the growth in power demand, said Gartner’s Chandrasekaran. However, “the likely outcome is more AI compute needs overall, even if the cost and energy consumed per prompt or task continue to fall.”

The complexity of AI processing also defies simple solutions. An example is the release of DeepSeek V3 in late 2024.  It was initially hailed as an environmental and financial breakthrough, thanks to algorithmic optimizations that allowed the model’s final training run to be completed ten times faster than comparable models, with a proportionate drop in power and inference costs.

Yet any potential energy savings were immediately swallowed by the introduction of “reasoning” models such as DeepSeek R1, The Economist noted.  Because those models use a methodical approach called Type 2 thinking, breaking a problem down, testing multiple approaches and validating its work before settling on an answer, they require significantly more processing time per query. The efficiency gains of V3 were quickly eaten up by the extended thinking times of R1.

Agents have a similar effect. Because they can search the web, write code and execute multistep tasks, a single request consumes orders of magnitude more energy than a simple chat query. Existing measurement frameworks don’t yet account for the impact of idle-machine overhead, data-center cooling and network transport.

That means the economic and environmental costs of AI likely can’t be solved at any single layer of the stack. Improving efficiency requires a coordinated, full-stack approach that links hardware design, software optimization, data management and energy supply. Only then can the industry transition from the brute-force scaling of the past to an efficient, sustainable utility model.

“The question in front of us is ‘Can we make the marginal cost of AI low as we scale AI usage?’” Chandrasekaran said. “We haven’t solved for this yet.”

Image: SiliconANGLE/Google Flow