What’s more redundant than another big data software tool? Another one from Apache, maybe?

Don’t be so quick to wave off newcomers like Apache Flink, said Dean Wampler (pictured), vice president of fast data engineering at Lightbend Inc.

“If you’d asked me a year ago, I would have said, ‘Well, I’m not sure we really need Flink. Spark seems to meet all our needs,'”  Wampler said at Flink Forward 2017 in San Francisco, California. Today, the need for smaller, more focused data tools for different companies’ needs is clear, he told George Gilbert (@ggilbert41), host of theCUBE, SiliconANGLE Media’s mobile live streaming studio. (*Disclosure below.)

While Spark is superbly engineered, it covers vast data analytics terrain, which means more things to break, some of which may not be needed to begin with, Wampler said.

“Having one tool that’s focused just on processing streams is often a good idea,” he said, adding that Flink fills this niche and hones in on advanced problems like late-arriving data and windowing semantics.

Trimming fat in the stack

Not that Flink is the sole answer to any company’s data strategy; they will have to pastiche a number of tools together, Wampler said. But choosing targeted and easy-to-wield ones can lighten the overall burden of managing them.

“The reality is we always have to integrate a bunch of tools together, and it’s that integration that’s usually the hard part,” he said.

A mish-mash of tools can be particularity confusing when the cause of a problem must be traced. “How do I know why this thing’s misbehaving when maybe it’s something upstream that’s misbehaving? That sort of thing,” Wampler said.

Lightbend uses machine learning and artificial intelligence to auto-detect and correct those problems for management, Wampler said. This is available through the company’s application development platform, which bundles Flink, Spark and other tools.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of Flink Forward 2017. (*Disclosure: TheCUBE is a paid media partner at Flink Forward. The conference sponsor, data Artisans, does not have editorial oversight of content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE