Just as businesses have started to get the hang of software and its benefits and challenges, a new age is upon them. The tech world started to revolve around data and the enterprise is following closely, trying to determine how to drive value from what started as buzz words and to then move closer to palpable results. Use cases, companies boasting about impressive results, research, surveys, and expert opinions show up left and right but, as it happens in the Age of Data, there’s an information overload that is hard to sort, analyze, and put to good use.

“In the technology world, and in society in general, we are exiting the ‘Age of Software’ and entering the ‘Age of Data.’ Data is now the central asset around which all businesses and applications revolve. The focus is no longer on the software application but on the lifeblood of data beneath the application skin. Data dominates,” says Mike Hoskins, the CTO at Actian.

Do we need ALL the data for quality data?

The first issue that arises is the quality of data. With data gathered from everywhere, humans, transactions, and all the machines you can think of, from cars to machines and light sensors, the amount is overwhelming. When one thinks of Big Data, they picture an overwhelming, unstructured volume of information that’s hard to store, more so to clean, organize, and analyze. When one adds the real-time requirement to putting data in to get actionable information out, the headaches kick in.

As the “garbage in, garbage out” saying goes, data quality is indeed the root of all problems. But the question of quality when analyzing petabytes of information is quite different than fixing the usually man-made errors that define small data sets. Cleaning up and sorting an information tsunami before analyzing it might be a nearly impossible approach to solve problem. On the other hand, that impressive amount of data might be the solution, as the errors might be lost in this huge sea and have no significant effect on end results.

“When collected and analyzed in near real-time, this information can help any business make faster and better decisions. Who wouldn’t benefit from that?” Hoskins wonders. And the examples of huge success are there to back him up. Just look at the benefits and potential improvements Big Data and analytics bring to the healthcare industry! Or the use cases for astronomy.

Yet data still needs to be collected, stored, cleaned, analyzed, transformed, so the infrastructure supporting that is paramount.

“The challenge is that the traditional software stack absolutely cannot scale to deal with the sheer diversity, speed and volume of these streams of data. The Age of Data has exposed the dead end that is conventional software architecture. We need completely new, modern, super-scaling architecture that can scale up and out to leverage the phenomenal power of commodity multicore hardware and enable organizations to affordably take action on their big data,” Hoskins explains.

NetApp says clustered storage is the answer to tomorrow’s data storage needs, while flash caching seems to be the answer to the need of acceleration in analytics. Virtualization helps, as well as data and software-led approaches to designing infrastructure. The cloud plays its part in everything, even helping cars get smarter. But we’re not there yet, as the amount of data is growing exponentially.

Bring security to the data

The one thing that is clear to everyone is that data is valuable. Even hackers are sure of that, that’s why they are targeting valuable data in their attacks. Security is the main conversation topics at all data related events and that’s how it should be, as no one, no matter how big and powerful, is safe. Just look at the recent breach at Apple or the Nexus leak. Companies such as HP focus on Big Data in their security updates, this new age requires a shift in how we approach security. The most effective solution in keeping data safe would be to bring security to the data.

Going hand in hand with security, there is the much talked about matter of privacy. We do like to share our data online, we like it when it’s put to good use, but we still need to maintain our privacy. Given recent developments, we are entitled to fear about how our data is being used. We’re still digesting the news of NSA spying on US citizens and businesses, when AT&T announced it plans to sell customer data to marketers through an update of its privacy policy. While being concerned is healthy, too much concern can hinder information sharing and prevent significant breakthroughs, especially in healthcare. Finding the middle ground is key but will we get there or stay paranoid in a world where trust seems to be a hard to acquire commodity?

If we think of privacy concerns, changing approaches in how we see and design hardware and software infrastructure, and the fact that most Big Data tools focus on allowing non-technical employees use it to create business value by offering effective and friendly means to view and understand information, the biggest challenge remains a human-related one: a cultural change across different generations. IT departments, sales people, management, they all need to rethink their values, their processes, and their priorities to make full use of what the Age of Data unveils in terms of potential.