UPDATED 08:00 EDT / DECEMBER 09 2015

NEWS

Data is infinite; the issue is cost, says Wikibon’s George Gilbert

The oft-cited study claiming 40 percent annual data growth is wrong, writes Wikibon Big Data Analyst George Gilbert in “Getting Ready for Big data 3.0 and the IoT”. Data volume is essentially limitless; the limiting factor is the cost of storing it, he writes

Traditional systems-of-record data, created and captured by humans, has historically cost roughly $1 billion per terabyte. It demands expensive scale-up storage and databases. Machine-generated data is different. It is huge in volume and cheap to produce. And because each datum is never updated and carries a unique time stamp, new data is simply appended to the end of the file.

This kind of data is typified by log data, which are ideal for capture in a Hadoop cluster running on cheap, commodity storage in the public cloud, Gilbert writes. And since virtually everything these days constantly generates logs, there is a lot of that kind of data to capture.

Companies that want to start learning how to capture and analyze machine-generated data for value would be advised to start with the logs from their existing applications. Properly collecting and analyzing this data to better manage the applications requires many of the skills, infrastructure and processes needed to support Internet of Things (IoT) tomorrow. And it can pay dividends in better application optimization and management. The easiest and least expensive infrastructure is Hadoop running on a public cloud.

In the second half of this two-part series Gilbert will discuss how to deliver near real-time responsiveness while building toward converged analytics, which enables any type of analytics to run on any type of data.

Image via Geralt

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU