UPDATED 14:30 EDT / DECEMBER 27 2018

CLOUD

AIOps: pulling critical data out of data lakes

In the pre-cloud world of computing, there was a perception that everything processed in the cloud was just like it was processed in a data center, only in a vague “over there” location rather than on computers racked away in the nearest closet.

Perhaps that was true; however, since the mass adaptation of cloud with its inherent capabilities and flexibilities (not to mention, containers and the internet of things) have led to a data explosion. It has been becoming increasingly difficult for organizations to locate and process the important data they have lurking in data lakes, and even to separate critical data from useless data.

“I mean, we cover everything from large financials, telephone companies, e-commerce businesses. And the drive to adopt agile and cloud and software-defined in the enterprise has driven complexity to the point where the poor, old human brain is just out of luck,” said Phil Tee (pictured), co-founder and chief executive officer of Moogsoft Inc.

As an innovator in artificial intelligence operations, or AIOps, Moogsoft helps enterprises to scale and detect problems before they happen, allowing them to use their data in a much more efficient manner.

Tee spoke with John Walls (@JohnWalls21), co-host of theCUBE, SiliconANGLE Media’s mobile livestreaming studio, and guest host Justin Warren (@jpwarren), chief analyst at PivotNine Pty Ltd., during AWS re:Invent in Las Vegas. They discussed how AIOps can help organizations to get full value from their data repositories. (* Disclosure below.)

Information science sifts through data for you

To put some context on the amount of data that many organizations are handling today, before the cloud and virtualization, within a typical enterprise, a high event rate would have been 100 to 200 events a second. Nowadays, for an average Moogsoft customer, it’s more like 1,000 to 2,000 events a second, or even 10,000 to 20,000 events a second, according to Tee. This is why legacy systems really struggle with that amount of data at those processing speeds.

“Job one is, if you accept, and I certainly do, that most of that data is junk, most of it is inconsequential, you’ve got to have an algorithmic way of getting rid of that,” Tee said.

The old-fashioned manner of handling this “sifting” was by making a series of lists, with instructions such as “ignore if it’s of a certain severity” or “ignore because it comes from this list of hosts.” This sort of handling by lists is a slow and cumbersome process; there’s nothing agile about it.

Moogsoft’s approach is to use information science to measure the semantic and informational content of an event to see whether it’s saying something of import. By using this technique, it’s possible to eliminate as much as 90 to 95 percent of the inbound data, according to Tee. This narrows the data lake down to a point where it can be processed in real time through much more computer-intensive AI algorithms to get that high-quality indication of an instance or a potential instance.

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of AWS re:Invent. (* Disclosure: Moogsoft Inc. sponsored this segment of theCUBE. Neither Moogsoft nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE

A message from John Furrier, co-founder of SiliconANGLE:

Your vote of support is important to us and it helps us keep the content FREE.

One click below supports our mission to provide free, deep, and relevant content.  

Join our community on YouTube

Join the community that includes more than 15,000 #CubeAlumni experts, including Amazon.com CEO Andy Jassy, Dell Technologies founder and CEO Michael Dell, Intel CEO Pat Gelsinger, and many more luminaries and experts.

“TheCUBE is an important partner to the industry. You guys really are a part of our events and we really appreciate you coming and I know people appreciate the content you create as well” – Andy Jassy

THANK YOU