This Week in Big Data: IBM & Hadoop, and the Public Sector

This past week in big data has seen some astronomical developments, for the most part from the Square Kilometer Array telescope. The project is kicking into high gear now that the the various international acidic institutes involved in the project are starting to turn to the IT industry for the solutions they need.

DDN was hired to provide the in-memory processing and storage, and IBM is collaborating with ASTRON to setup the data munching software on top. Almost $45 million is funneled into this particularl puzzle piece nicknamed DOME–one small part of a 1.5 billion euro initiative to explore the origins of the cosmos.

Just a day later, we had equally meaningful news from IBM with the announcement of two new software solutions. The tenth generation B2 and InfoSphere Warehouse feature several improvements over their respective predecessors, including advanced timeline cross-indexing and Hadoop support.

A new startup called Kontagent also has some new ideas concerning the analytics engine. It announced kSuite DataMine, a hosted Hive-based platform designed to help marketers and game developers receive full insight from their data. The way the company is realizing this is quite unique, as Matt Weinberger explained in his post earlier this week.

The last part of this recap on big data news in the past seven days is dedicated to a $250 million Pentagon venture aiming to improve the processes of decrypting encoded transmits using big data.  The NSA-Central Security Service Commercial Solutions Center, abbreviated to the somewhat less awkwardly-pronounced codename of NCSC is getting more funding, along with CINDER. The Cyber-Insider Threat team is exploring new techniques to detect network intrusions and cyber espionage.

There’s also Insight, a third undertaking that aims to improve decision making via the means of machine learning.