What’s the surest way to advance a technology in a short time? Give it away — to an open-source community. Seminal big data software library Apache Hadoop gained momentum in open source, and today, most disruptive big data development is springing from open source as well.

“If people have the community traction, that is the new benchmark,” said John Furrier (@furrier) (pictured, left), co-host of theCUBE, SiliconANGLE Media’s mobile livestreaming studio. This is evident at SiliconANGLE’s and theCUBE’s BigData NYC 2017 event, where Furrier and co-host James Kobielus (@jameskobielus) (pictured, right) discussed the community edge.

Yahoo Inc. just open-sourced its big data search and recommendation software Vespa, following its hugely popular 2006 Hadoop contribution. It clearly believes Vespa can evolve via open-source developer brains just as Hadoop did.

“As the community model grows up, you’re starting to see a renaissance of real creative developers,” Furrier said.

These developers are not just working out implementation kinks; they’re innovating at a level that makes a difference for applications. “Real creative competition — in a renaissance, that’s really the key,” Furrier stated.

The renaissance will be automated

Much new development branches out from big data per se, into artificial intelligence, machine learning and internet of things. “Data professionals and developers are moving toward new frameworks like TensorFlow,” Kobielus said. TensorFlow is Google’s open-source deep learning framework. Caffe and Theano are additional open-source deep learning technologies with bustling communities around them.

Some of the most exciting work happening in open-source (and at Stanford University) revolves around automating the acquisition of data needed to train machine learning models. Many would like to see deep learning tools and methods operationalized, enabling what some call DataOps or InsightOps (IBM’s term), Kobielus pointed out.

“I think what are coming into being are DevOps frameworks to span the entire life cycle of the creation and the training and deployment and iteration of AI,” he said.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of BigData NYC 2017.

Photo: SiliconANGLE