UPDATED 15:56 EDT / SEPTEMBER 24 2012

NEWS

Big Data: Polluter or Environmentalist?

Here’s what we know. Data volumes are growing exponentially – a.k.a. Big Data. And we need hardware – lots of hardware – to store it, crunch it and deliver all that data to hungry end-users – both business users and consumers.

The problem, as pointed out in a New York Times piece published over the weekend, is that all that hardware requires significant power to operate, including industrial cooling equipment and back-up generators that spew diesel exhaust and other pollutants into the atmosphere. Meanwhile, datacenter operators that prize continuous uptime over energy efficiency routinely run their operations at full power even while many servers sit idle or significantly underutilized.

And as data volumes continue to grow, so will the number of polluting datacenters. This, says the Times, is “sharply at odds with [the information industry’s] image of sleek efficiency and environmental friendliness.”

I don’t wholly agree with this assessment and here’s why. Yes, sprawling datacenters in the age of cloud computing do create significantly more pollution than your average one-family home or even big city apartment building. But overall, as even the Times points out, all told datacenters consume just 2% of the nation’s energy resources. And, while there is a long way to go, the industry is making significant strides in improving datacenter efficiency.

Speaking more specifically to Big Data, there are a number of initiatives underway to improve the efficiency of hardware used in Hadoop deployments. In a research note last month, I wrote about two of them. One is Project Serengeti from VMware, which allows administrators to virtualize Hadoop deployments on existing hardware. This both improves node efficiency and often removes the need to purchase new hardware.

The other is Pervasive Software’s DataRush products. As I wrote then:

Together, DataRush and RushAnalyzer abstract away the complexity of parallelizing Hadoop jobs, give users the ability to monitor i/o and CPU usage in real-time, and mitigates memory constraints. The results are server efficiency rates as high as 80%, according to Inbar, and the ability to more quickly and iteratively perform Big Data Analytics.

NASA Center for Climate Simulation’s Datacenter

Another point that the Times neglects to make is that much of the data science and analytics being done to combat climate change and to further environmental sustainability is supported by these very same datacenters. The NASA Center for Climate Simulation, for example, has built a custom Big Data platform to crunch multiple petabytes of weather data to better predict and understand global climate change. The NCCS datacenter consumes massive amounts of power, but the insights its researchers uncover through Big Data Analytics might just lead to ways to limit or even reverse global warming. I’d say that’s more than a fair trade.

Of course, not every Big Data project’s goal is to promote the greater good. But there are a number of such projects in the climate sciences as well as healthcare and education industries.

I applaud the Times for covering this issue, as it’s an important one. But let’s keep our perspective realistic. While Big Data does add its fair share of pollutants to the atmosphere, the industry is both developing more efficient computing methods and tackling some of the thorniest environmental challenges facing mankind that have the potential to more than make up for any direct impact datacenter operations have on the atmosphere.


A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU