In the pre-cloud world of computing, there was a perception that everything processed in the cloud was just like it was processed in a data center, only in a vague “over there” location rather than on computers racked away in the nearest closet.

Perhaps that was true; however, since the mass adaptation of cloud with its inherent capabilities and flexibilities (not to mention, containers and the internet of things) have led to a data explosion. It has been becoming increasingly difficult for organizations to locate and process the important data they have lurking in data lakes, and even to separate critical data from useless data.

“I mean, we cover everything from large financials, telephone companies, e-commerce businesses. And the drive to adopt agile and cloud and software-defined in the enterprise has driven complexity to the point where the poor, old human brain is just out of luck,” said Phil Tee (pictured), co-founder and chief executive officer of Moogsoft Inc.

As an innovator in artificial intelligence operations, or AIOps, Moogsoft helps enterprises to scale and detect problems before they happen, allowing them to use their data in a much more efficient manner.

Tee spoke with John Walls (@JohnWalls21), co-host of theCUBE, SiliconANGLE Media’s mobile livestreaming studio, and guest host Justin Warren (@jpwarren), chief analyst at PivotNine Pty Ltd., during AWS re:Invent in Las Vegas. They discussed how AIOps can help organizations to get full value from their data repositories. (* Disclosure below.)

Information science sifts through data for you

To put some context on the amount of data that many organizations are handling today, before the cloud and virtualization, within a typical enterprise, a high event rate would have been 100 to 200 events a second. Nowadays, for an average Moogsoft customer, it’s more like 1,000 to 2,000 events a second, or even 10,000 to 20,000 events a second, according to Tee. This is why legacy systems really struggle with that amount of data at those processing speeds.

“Job one is, if you accept, and I certainly do, that most of that data is junk, most of it is inconsequential, you’ve got to have an algorithmic way of getting rid of that,” Tee said.

The old-fashioned manner of handling this “sifting” was by making a series of lists, with instructions such as “ignore if it’s of a certain severity” or “ignore because it comes from this list of hosts.” This sort of handling by lists is a slow and cumbersome process; there’s nothing agile about it.

Moogsoft’s approach is to use information science to measure the semantic and informational content of an event to see whether it’s saying something of import. By using this technique, it’s possible to eliminate as much as 90 to 95 percent of the inbound data, according to Tee. This narrows the data lake down to a point where it can be processed in real time through much more computer-intensive AI algorithms to get that high-quality indication of an instance or a potential instance.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of AWS re:Invent. (* Disclosure: Moogsoft Inc. sponsored this segment of theCUBE. Neither Moogsoft nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE