From its origins in a Silicon Valley basement over a decade ago, Pepperdata Inc. has grown to become a cloud cost optimization solution for major companies such as Citibank N.A. and T‑Mobile USA Inc.

The company specializes in helping enterprises manage big data performance using technology created originally to handle on-premises workloads, many of which now operate in the cloud.

“The company built a bunch of extremely good IP around large-scale collection of time series data,” said Ash Munshi (pictured, left), chief executive officer of Pepperdata. “The kind of data that we collected would be the equivalent of what Uber collects on all of its cars and all of the movements of its cars today. And we were doing that roughly 10 years ago.”

Munshi spoke with Lisa Martin, industry analyst for theCUBE, SiliconANGLE Media’s livestreaming studio, during the “Analytics and Cost Optimization” AWS Startup Showcase. He was joined by Kirk Lewis (right), senior solution engineer at Pepperdata, and they discussed the company’s big data cost and capacity optimization solutions for small to large businesses around the world. (* Disclosure below.)

From MapReduce to Spark

Pepperdata’s original vision grew from the programming model MapReduce. The company’s founders played a pivotal role in building the programming application for large-scale computation, according to Munshi, and Pepperdata’s subsequent shift to Apache Spark as an engine for large-scale data processing allowed it to tackle performance challenges inherent in the big data world.

“Spark is now the infrastructure for being able to do a lot of large-scale distributed computation,” Munshi said. “Pepperdata provided the metrics that were required to run something efficiently, how to monitor it, how to make sure things were running properly and then be able to do that for mere mortals and let enterprises make that work. We started getting some good traction.”

In a report prepared by Pepperdata last year, the company found that Spark had emerged as the number one big data application running on Kubernetes. The market was embracing Kubernetes while migrating toward microservices architectures, and Pepperdata saw an opportunity to move optimization on the big data side into the microservices world.

“There’s been a trend afoot which basically says, ‘Let’s take all of this big data stuff, all of this microservices stuff, and unify it under Kubernetes,’” Munshi said. “That’s exactly what we’ve done. We’re about to introduce a set of products that essentially take our time-tested, proven, scalable capabilities with some of the largest companies in the world, move that into the Kubernetes space and offer the same kind of optimization.”

Finding optimal capacity

In addition to optimizing big data in microservices, Pepperdata has also been focused on helping enterprises optimize cluster resources and recapture wasted capacity. This has become more of an issue in network operations as companies rely on nodes to run an optimal number of containers and are increasingly looking for new ways to save money.

“There is a gap between what is asked for in terms of resources and what is actually used dynamically,” Munshi said. “As a result, that gap allows us to put more capacity effectively or create virtual capacity on a given node. The core technology that does the cost savings is Capacity Optimizer.”

Capacity Optimizer is Pepperdata’s solution for automatically optimizing cluster resources and realizing price/performance improvement on top of AWS autoscaling. The Pepperdata offering identifies where more work can be done and adds tasks to nodes with available resources.

“We know that there’s always a usage curve there,” said Lewis, in a demonstration of Capacity Optimizer during the interview. “The problem is there’s always going to be that delta between what you’ve asked for and what you’ve used which quantifies the waste. We augment the autoscaler in a way that says don’t launch any more nodes until Capacity Optimizer packs that node to about 80%.”

Interest in Capacity Optimizer is being driven by enterprise desire to control costs as cloud expenses have climbed. It is a dynamic that Pepperdata has seen before when its customers sought expense management in on-premises environments before moving to the cloud.

“It turns out that the stuff we did on-prem for controlling cost was applicable to the stuff on the cloud as well,” Munshi said. “We are in fact seeing from the smallest companies to the largest companies a great desire to be able to go and save money. You don’t do everything because of cost optimization, but you make sure that you’re actually managing your business in a way that is not a runaway cost structure.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the “Analytics and Cost Optimization” AWS Startup Showcase event:

(* Disclosure: Pepperdata Inc. sponsored this segment of theCUBE. Neither Pepperdata nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE