As data scales up, businesses face challenges of formatting unstructured data, aggregation from multiple sources, resiliency and security. This has given way to the growth of Fluent Bit, an open-source log processor and forwarder that allows enterprises to collect any data-like metrics and logs from different sources, enrich them with filters and send them to various destinations.

Fluent Bit is a subproject of Fluentd data collector, which is the CNCF’s fourth project after Kubernetes, Prometheus and OpenTracing. It allows for a unified logging layer to be implemented in cloud native architectures. The biggest difference between the two log processors is that Fluent Bit is a lightweight version of Fluentd.

“Fluentd is established in the market, and [with] Fluent Bit we’re getting around 2 million Docker Hub downloads every single day, so nowadays the traction of the project is incredible,” said Eduardo Silva (pictured), principle engineer at Treasure Data and project maintainer at Fluent Bit. “[Fluent Bit] is mostly used to collect logs from the files … and from mostly a Kubernetes environment is able to process all this information and metadata and solve all the problems of ‘how do I collect my data, how do I make sure that the data has the right context?’”

Silva spoke with John Furrier, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the recent KubeCon + CloudNativeCon. They discussed Fluent Bit’s gain in popularity, what’s new for the log processor, and the features and improvements expected in the coming months. (* Disclosure below.)

Being vendor-neutral is key

One of the great differentiators of Fluentd and Fluent Bit log processors is that they are vendor-neutral —companies will not be locked into one vendor, according to Silva. The idea is to give customers the freedom to decide where and when to send data, in addition to controlling expenses with this process.

“You can go to the market; they would find maybe all the tools for logging or tools for metrics because there’re a ton of them, but I think that not all of them can say, ‘We are vendor-neutral.’ Not all of them can offer this flexibility to the use,” Silva explained.

The projects have taken the approach that data is agnostic about how it comes and where it comes from, he added. They handle the tasks of pulling and receiving the data from multiple systems, transforming it into a meaningful set of fields and eventually streaming the output to a defined destination for storage.

The focus for this year and the next is to improve performance and speed up data processing. But there are also some new features to be released.

“Where we’re going next right now, there are two major areas: One of them is distributed extreme processing, the capabilities [that] put this intelligence on the edge,” he said. “[And] we’re going to provide this year the ability to write your own plug-ins in Wasm, WebAssembly.”

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of KubeCon + CloudNativeCon. (* Disclosure: Cloud Native Computing Foundation sponsored this segment of theCUBE. Neither CNCF nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE