The oft-cited study claiming 40 percent annual data growth is wrong, writes Wikibon Big Data Analyst George Gilbert in “Getting Ready for Big data 3.0 and the IoT”. Data volume is essentially limitless; the limiting factor is the cost of storing it, he writes

Traditional systems-of-record data, created and captured by humans, has historically cost roughly $1 billion per terabyte. It demands expensive scale-up storage and databases. Machine-generated data is different. It is huge in volume and cheap to produce. And because each datum is never updated and carries a unique time stamp, new data is simply appended to the end of the file.

This kind of data is typified by log data, which are ideal for capture in a Hadoop cluster running on cheap, commodity storage in the public cloud, Gilbert writes. And since virtually everything these days constantly generates logs, there is a lot of that kind of data to capture.

Companies that want to start learning how to capture and analyze machine-generated data for value would be advised to start with the logs from their existing applications. Properly collecting and analyzing this data to better manage the applications requires many of the skills, infrastructure and processes needed to support Internet of Things (IoT) tomorrow. And it can pay dividends in better application optimization and management. The easiest and least expensive infrastructure is Hadoop running on a public cloud.

In the second half of this two-part series Gilbert will discuss how to deliver near real-time responsiveness while building toward converged analytics, which enables any type of analytics to run on any type of data.

Image via Geralt