Big data promises to revolutionize business. But often business opportunities are fleeting. A retailer, for example, can only use information on up-selling a customer while that customer is in the store or on the retailer’s Web site. Once the customer checks out or leaves, the opportunity is gone. New online business use cases such as ad-tech, in which advertisers bid to get their ads in front of individual consumers based on that consumer’s value to the advertiser at that moment, are built on microsecond response, in this case to a user log-on to a Web site like Facebook or Google.

The problem is that the core Big Data technologies, starting with Hadoop, are not designed for sub-millisecond response. The answer, writes Wikibon Big Data Analyst Jeff Kelly,  are in-memory databases such as Aerospike, Inc.’s Aerospike, IBM DB2 with BLU Acceleration, Pivotal Software, Inc.’s GemFire, Oracle Database In-Memory, and SAP’s HANA. The first two of those are hybrid systems that tier data across DRAM, solid-state (flash) and spinning disk storage; the other three are pure DRAM databases. They often leverage new design approaches such as columnar architecture, data-skipping and high rates of data compression to further increase analytical performance.

In-memory databases are not necessarily new, but the falling cost of computer memory combined with the advent of new business use cases that make extremely fast in-line data analysis more valuable are driving them into traditional as well as born-on-the-cloud business infrastructures. SAP, for instance, has made HANA the core of its architecture going forward, meaning that SAP users will need to implement it to support new versions of SAP business applications. Any operational business process that requires sub-millisecond analytic performance is a candidate for these very fast platforms, Kelly writes in ”In-Memory Databases Put the Action in Actionable Insights.”

While SAP HANA and other new database engines are totally in-memory, the hybrid systems, when architected correctly, can provide the performance of in-memory databases at a lower overall price point. The in-memory portion can hold the most recent data and support in-line analytics, while the lower tiers can support applications that do not demand sub-millisecond response. They compliment rather than replace Hadoop and NoSQL alternatives, as well as traditional relational database systems. These latter systems are used by data scientists for deep data analytics, and increasingly for predictive analytics that identify longer-term market trends and business opportunities. These, Kelly writes, are obviously valuable. But in-line analytics, which is often supported by in-memory databases, “is often where the ‘action’ occurs, where transactions take place and where the real money is made.”

photo credit: Striking Photography by Bo Insogna via photopin cc