The path to 2011 has been filled with numerous new security threats—and some old ones that have learned new tricks—as the Internet continues to expand, computers get faster, storage gets smarter, and connectivity increases these threats will continue to evolve and security must do so as well. This is the classic Red Queen’s race and the next stage will probably involve the deployment of massively-structured analysis of behavior overseeing systems security.

Conventional security is designed around a paradigm of watching traffic flowing through a gate, or training users to spot potentially untrustworthy transactions in relationship to them—that’s the level of firewalls, intrusion detection, and personal security. However, attackers are getting smarter about how they reach into systems, they subvert human social queues, they trick firewalls into allowing them through, or they just simply walk in the door with filched authentication.

With the advent of Big Data we have companies storing even more logs and information than before—especially now that storage is an easy thing—but with more data doesn’t always come more understanding. Highly distributed frameworks such as Hadoop have come in to resolve this problem by allowing IT Departments to analyze (in real time) giant heaps of data flowing through the traffic of their infrastructure in order to make things more efficient, but they can also be used to track and discover things that might be out of place: like an attacker.

Ellen Messmer from TechWorld asked the question, “Can Big Data be used to catch network invaders?” and like her respondents, I agree that it can; but it will require a shift in the current security paradigm to make good use of it.

Scott Crawford, analyst with consultancy Enterprise Management Associates, thinks so. “Statistical analysts will identify anomalies but not understand the security,” he commented during an analysts panel at the recent RSA Conference in San Francisco on the topic of Big Data and how it could help security.

Crawford predicted eventually there will emerge “a market for security algorithms” for Big Data. He noted firms such as Red Lambda and Palantir are tackling this today in maths-heavy analysis aimed at spotting anomalies.

We’ve already seen some products that deploy extremely similar algorithms such as RSA’s threat-detection product NetWitness and HP ArcSight SIM.

Detecting an intrusion after-the-fact is the worst time to discover it; but it’s equally important to get in its way and prevent the intruder from doing damage at that point. It also means that false alarms may have an impact on worker productivity and efficiency in general. As a result, it will become important to look at the business logic of anyone using Hadoop or an RSA product to track behavior do so as transparently as possible so that when an apparent threat arises it doesn’t send the entire process into a tizzy.

What we’ll need is firms to rise up and produce good infrastructure for providing and developing security via Big Data because currently most IT Departments may collect between 1 TB and 1 petabyte of data, but they don’t know what to do with it. According to a report from LogLogic the barrier to Big Data with security is hodgepodge legacy systems and IT firms not ready to deploy it.

In a survey of over 200 security officers (titles ranging from directors to CISOs), LogLogic found that only 27% of respondents clearly understand what big data means.

Still, respondents at least know they need to worry about it—half responded that they are concerned about managing big data. The problem is that many still don’t have a centralized viewpoint to do so—59% are either not managing log data, using disparate systems, or using antiquated spreadsheets. At the same time, the amount of IT data organizations are producing is growing exponentially—with 62% producing more than 1 TB and up to 1 petabyte of data.

Products like Hadoop will probably work extremely well even with legacy systems that store different data types as long as they can all be siloed in the same place and a distributed analysis system developed to understand each one. However, this will open up new faults in the system that an attacker could exploit. Security needs to be holistic and aware of the entire system at once in order to function properly—but with the rise of mobile devices, corporate networks thrust into the cloud, and other agile shifts in technology it will become increasingly necessary to enable a more distributed business intelligence and security that can handle not just system security but personal security.

Ideally, in the case of cloud-security tied with Big Data, it will be about being able to authenticate transactions between systems and not just authenticating the people. With real-time Big Data analysis running, transactions can be watched and judged based on the patterns that they follow and greater scrutiny can be called down when transactions deviate from known patterns or anomalies start cropping up in operations.

Often, it’s been discovered, the rise of anomalies presages an attack or an outright failure.

With Big Data analysis, IT security professionals could stay that one extra step ahead of the bad guys. We just need someone to step up and start developing a solution that will let them do that.