The Apache Software Foundation is now incubating a project backed by Cloudera Inc. and Intel Corp. that aims to bolster cyber security with Big Data analytics and machine learning.

Previously known as the Open Network Insights (ONI) initiative, the project is now called Apache Spot, since it uses machine learning to sniff out bad traffic amongst good data. It can also study network traffic to characterize its unique behavior using the open source distributed storage and processing software Hadoop, which helps it to discover if any anomalies are present.

Intel launched the project back in February on Cloudera’s cloud computing platform. The project is also backed by eBay Inc., as well as smaller security firms like Anomoli Inc., Centrify Corp., Cloudwick Inc., Cybraics Inc., Endgame Inc., Jask, Streamsets Inc. and Webroot Inc.

Apache Spot works by storing large amounts of information within Hadoop, then using Apache Spark to process data from deep packet inspection of domain name system (DNS) traffic, connections and log files from proxies. Spot then leverages its machine learning capabilities to build models of networked systems and learn how they communicate, drawing on billions of collected events which are filtered for noise. With this, Spot is able to create a shortlist of the most likely security threats a network will face.

One primary use-case for Spot is to help organizations reduce the “mean time to incident detection and resolution” (MTTR), which is a key metric for measuring the efficiency of cyber security. In a blog post, Cloudera software engineer Mark Grover and systems engineer Morris Hicks explained that Spot is able to boost MTTR because it provides a central storage capability that houses all of the data needed to facilitate an investigation.

Spot helps out with those investigations too, by gathering all of the characteristics of a given IP address and building a timeline of all conversations that originated with it. This data is then used to create storyboards of the threat event with interactive visualizations.

Collaborating against threats

In addition, Spot draws on what its creators call “common open data models” for security information, that are designed to foster collaboration between enterprises to ward off new threats, by comparing them against historical data sets for greater insights. This capability is perhaps the most interesting of Spot’s abilities, as enterprises traditionally haven’t shared much information about security issues – unlike the hackers themselves, who regularly share information on underground forums.

“The idea is, let’s create a common data model that any application developer can take advantage of to bring new analytic capabilities to bear on cyber security problems,” said Mike Olson, Cloudera co-founder and chief strategy officer, as the project was unveiled at Strata+Hadoop World in New York. “This is a big deal, and could have a huge impact around the world.”

The open data models for network, endpoint and users allow Spot to integrate cross-application data to provide better enterprise visibility and analytics. Organizations can also share analytics more easily via those open data models as new threats are discovered.

“The open source community is the perfect environment for Apache Spot to take a collective, peer-driven approach to fighting cybercrime,” Ron Kasabian, vice president and general manager for Intel’s Analytics and Artificial Intelligence Solutions Group, said in a statement. “The combined expertise of contributors will help further Apache Spot’s open data model vision and provide the grounds for collaboration on the world’s toughest and constantly evolving challenges in cyber security analytics.”

Apache Spot is available to download from the Github open source repository.