This week the Wikibon community sat down with Dag Liodden, Co-Founder and CTO of Tapad, a two-year old company that provides cross-platform digital advertising technology, to discuss the challenges and opportunities for real-time analytics and decision making with big data. The session began with an explanation of Tapad’s solution, which processes around 150,000 ad impressions per second during peak usage times to select the right ad, for the right user and right device. According to Liodden, it was apparent very early that a traditional relational database wasn’t the best option for achieving the data volume, latency/throughput requirements and access patterns necessary for Tapad.

Tapad elected to use a key-value based NoSQL repository after evaluating multiple options, but Liodden made it clear the company wasn’t just jumping on the NoSQL bandwagon. NoSQL was the best fit for the organization’s needs. Although some NoSQL store data in RAM, the amount data Tapad stores and processes makes using RAM exclusively a very expensive option. Tapad leverages a hybrid solution; it stores index keys in memory and the associated data on solid state drives (SSD) using the Aerospike NoSQL repository.

The company has been using the SSDs for a year and a half. The strategy cost much less than the terabytes of memory that would have been necessary manage Tapad’s data. In addition, SSDs makes scaling across multiple servers more efficient. When data is stored in RAM, bringing a new server online for failover requires reading a massive amount into memory, which would introduce significant latency.

Liodden explained the complex multi-step, data driven decision making process Tapad performs with each request in about 100 milliseconds. Like other industries, technology advancements in big data, modern CPU architecture and storage has allowed the advertising technology market to evolve quickly. Liodden believes Tapad’s architectural strategies could be useful for any organization that needs to process large datasets in real time. He recommends Aerospike as an option for businesses that are using Memcache, but need the data to be persistent.

He cautions that company’s NoSQL-based infrastructure is not a silver bullet for data; it supports a very specific, real-time operational use case. It is not an analytics solution. Organizations should not become enamored by big data and NoSQL hype and try to apply it to every problem. Find the right architecture for the problem.