Experts advocate that a proactive approach to data resilience requires identifying threats at the first possible opportunity before they have a chance to infiltrate a system and wreak havoc.

Recently, IBM Corp. scientists and engineers developed new techniques for scanning data as it arrives inside a flash array, measuring the entropy of the data — the degree of randomness or disorder in a system.

IBM applied that concept to data to help identify specific kinds of encrypted data that could be evidence of malware. The new techniques are responding to a “huge societal problem” that exists right now, according to Andy Walls (pictured), fellow, chief technology officer and chief architect of IBM FlashSystems at IBM Corp.

“We’ve got whole organization crime areas and nation-states that are seeing the money they can make in ransomware and going after various kinds of companies and governments and wreaking havoc to try to make money,” Walls said. “We need to do whatever we can at every part of the stack. Even the block storage where the data is stored needs to do what it can to try to detect these intrusions as soon as possible.”

Walls discussed those techniques with theCUBE industry analyst Dave Vellante at IBM Storage Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. (* Disclosure below.)

Highly capable adversaries

Organizations shouldn’t wonder if they will be attacked, but when. The challenge involves highly capable adversaries with a black box in black storage. What were some of the challenges that IBM scientists and engineers had to go through given those challenges? This is the start of a journey, according to Walls.

“Our main job is to store the data. However, as that data comes in, we can run some tests on it and do something that’s called Shannon entropy. It was developed way back in 1948,” he said.

Shannon entropy aims to determine the disorder in the data and how random the data is. That is a perfect way of determining if the data’s being encrypted or not, according to Walls. IBM has dedicated some of its processing power in the controllers of the flash system to bring the data in, test it, sample it and look at what the entropy is for that data for each volume.

“We look at the volumes differently. Each one is different, because it represents an application from the system or part of an application. So we keep each volume separate, we determine its entropy, but we go a step beyond that,” Walls said. “We also are looking at how the compressibility is changing.”

With a look at the compressibility and a look at the entropy, the company then combines the two and tests to see if the data is changing, according to Walls. If a customer had a volume that had a compressibility of 50%, for example, what would be seen is, for some time, the entropy of the data coming in would be high enough to raise an alert that something was going on, according to Walls.

“Now, it’s not necessarily that it’s a ransomware attack. It may be somebody’s turned encryption on in the application and the storage administrator doesn’t know,” he said. “But the storage administrator needs to know because his compressibility is changing. He needs to know that he might need to allocate more storage. So we’re detecting anomalies, as well as looking for ransomware attacks.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the IBM Storage Summit:

(* Disclosure: IBM Corp. sponsored this segment of theCUBE. Neither IBM nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE