Attivio has added new connectors to its technology through integration of its active intelligence engine with data sources such as Hadoop, as well as data warehouses from vendors including HP Vertica, EMC Greenplum, IBM Netezza and Oracle.

The Attivio enhancement is another example of how big data is changing the way we do business and forcing a new perspective about the intersection of operations and development. This reality is new for everyone, offering us a unique perspective about the ways that IT is transforming.

This year’s OSCON conference in Portland is a window into this new world and why the knowledge being shared is a chance for even the most laggard of IT shops to catch up and keep pace with the rapid innovation cycle now underway.

Let’s first take a quick look at Attivio. The Newton, Ma. company is at the intersection of where business intelligence meets search. Its technology connects into data warehouses and pulls out information that can be used to solve customer problems.

Attivio is designed as a platform. Its middleware is integrated by ISVs and systems integrators.  It supports REST, XML, RSS feeds, HTTP, WSDL and more.  Its reputation comes from the way it can take unstructured content such as emails, documents and other text items and make sense of it.

But demand is increasing for more data integrations. As Wikibon’s Jeff Kelly points out, sensor data and log files from the data center are of interest to customers who wish to know the secrets that comes when the right algorithms are developed.

Attivio has been getting great reviews for the past few years. But the data demands are far different now than in 2009. Increasingly, how business intelligence is gleamed from disparate data sources requires a lot of work before it even enters a recommendation engine such as that from Attivio. Namely that means hiring developers to get the data right in the first place so it can be properly analyzed and acted upon.

And right now, the demand is deep for knowledge about how to get the most of big data. So much  so that the OSCON conference this week has an entire event within in it called OSCON Data. What are attendees learning about? Tracks cover such topics as analytics and visualization, Hadoop, NoSQL and real-time and streaming.

The track descriptions make it evident that much of this is new for the developer. For example, “Taming the Big Data Firehose,” is described as:

The term Big Data describes a new class of database applications that need to process massive data volumes in two disparate states – real time and historical. In either state, the requirements of Big Data applications vastly exceed the capabilities of traditional, one-size-fits-all database systems.

That description is a basic one. It goes on:

Most Big Data applications require MPP scale-out architectures and have the following characteristics: 1. A “fire hose” data source such as an HTTP streams, sensor grid or other machine-generated data 2. A real-time database capable of ingesting, organizing and managing high volume inputs 3. A persistent data storage and analysis infrastructure capable of managing petabyte+ historical databases In this talk, we will introduce a simple formula for all Big Data applications: Big Data = Fast Data + Deep Data. Through a use-case format, we will discuss the specialized requirements for real-time (“fast”) and analytic (“deep”) data management. We’ll also explore ways in which popular business intelligence solutions can be used to implement real-time and historical analytics.

The developers are here to learn new theories and practices which requires a different type of automated infrastructure.  But is IT willing to make the change and open its infrastructure?

Dave Cahill writes on Wikibon:

The DevOps movement emanating from cloud and big data is a 100% application-driven mentality that is striving for a programmatic infrastructure. In theory the goal is for the application/workload, whether physical or virtual, to communicate “these are my needs”, and the storage can respond.

This doesn’t happen today because storage admins won’t let it. Old school storage guys are all about control and tuning. They back into the IOPs they need by controlling spindles and data placement. They struggle mightily to relinquish this control.

The cloud mentality is 100% in conflict with this old school storage dinosaur viewpoint. These next-gen storage platforms are more application-aware. The more granular tuning at the platform level yields more purpose-built solutions that are designed for a particular workload or customer set. Conceptually this is very different from the general purpose era that has preceded it where vendors built boxes with all these knobs that could be tuned by admins. The more knobs the better. The new generation of systems gets rid of the “knobs”, takes whatever I/O you throw at it, and ensures a good enough experience for the targeted workload or customer set. I call the differentiation between the two, experience-based versus approach-based IT.

Services Angle

It’s not too late for IT. They need developers and developers needs them. Big data is new to everyone as demonstrated here at OSCON.  To grasp it will require a broader way of thinking about data and how it drives business.

Attivio  customers face a new reality.  Big data success will come by building applications on top of Attivio. That means customers will be looking to service providers with that expertise. Judging from OSCON, it’s still a small pool of developers with the knowledge to build big data tools. That knowledge is growing but it may take some time for the developer community to evolve. As that community does grow look for a new generation of big data use cases to take shape in everything from product development to customer service.