Approaches to business data overload range from “If I stick my head in the sand I can’t see the problem,” to panic attacks from data overwhelm, to the seemingly rational attempt to centralize and control everything.

None of these work.

Ignoring data altogether is a security nightmare, not knowing where to start is ineffectual, and tight control contradicts modern dispersed data accessibility. Worse, these approaches defeat the true purpose of data collection: creating business value.

Thankfully, there’s a lifeline for companies floundering in the self-created data swamps.

“If an organization knows where its data is, whose data it is, what it is and also who has access to it, they can start to make actionable decisions based on this new data intelligence,” said William Murphy (pictured), vice president of business development and technology alliances at BigID Inc.

Murphy spoke with John Walls, host of theCUBE, SiliconANGLE Media’s livestreaming studio, during the AWS Startup Showcase Event: Innovators in Cloud Data. They discussed the importance of data discovery in cloud-scale data management. (* Disclosure below.)

Data discovery is the foundation that enables companies to gain value from data

As it passes its fifth anniversary, BigID is celebrating being an Amazon Web Services Inc. showcase innovator and part of the AWS Partner Network Global Startup Program. Late last year, the company tipped across the $1 billion valuation mark to reach unicorn status, as investors validated its innovative stance on data governance, security and privacy in a regulated world.

BigID was built on the precept that organizations needed to have a better understanding of their data. “Starting in 2016 with the authorship of the European privacy law, organizations had to treat data differently than they did before,” Murphy stated.

The masses of data companies had accumulated suddenly became an issue. Secure storage costs mounted, and high-profile breaches caused reputational damage alongside financial loss. By giving companies an understanding across their data assets and the ability to locate and access specific information easily, BigID moves data from being a financial and security liability to an asset that can be leveraged to drive business value, according to Murphy.

“We’re empowering organizations to handle their data [whether it is] sensitive, personal [or] regulated,” he said.

BigID controls data sprawl and speedily scans ever-expanding data reserves

One BigID customer, which Murphy doesn’t name due to privacy concerns, was very concerned about sensitive data sprawl across its intellectual property. BigID provided custom tags and classifications, giving the company visibility into what data was where and who had access.

“They could see across their cloud environment, across their on-premise environment, across shared drives … where sensitive data had sprawl, where it had moved, who had access to it,” Murphy said.

Thanks to this knowledge, the company realigned its storage, content management and data governance strategies, moving sensitive data to lock it down on higher levels and create more access control while also creating easier share access to less sensitive data.

Another customer was struggling with how to understand its massive and constantly growing data reserves.

“If we create this data map, if we create this data inventory, it’s going to be out of date day one. As soon as we say, it’s complete, we’ve already added more,” Murphy said, recalling the company’s concerns. BigID was able to scan the customer’s entire AWS environment in a matter of months, including new data as it was added.

“Our Hyperscan capability … was a gamechanger for them to understand what data they had and where it is, whose it is, in a way that they never thought they could keep up with,” Murphy said.

Data discovery is a concern for all companies across all industry verticals, regardless of size. And one lesson BigID has learned from AWS is the importance of focusing on customer needs from the global enterprise level to small independent businesses. Their new SmallID solution is a cloud native, lightweight version of BigID that has been developed exclusively through AWS for small and medium customers in the commercial space.

“They can start to build a foundation of understanding their data, for protection, for security and for privacy,” Murphy said.

The BigID-AWS partnership

AWS and BigID share a customer-obsessed culture, according to Murphy, who noted that the cloud mega-corporation is “very forward-looking for an organization as large as they are.” That progressive stance shows in AWS’ support of startups like BigID, with Amazon understanding that fostering innovation benefits the greater ecosystem by increasing the solutions available for all.

An example of this comes with cloud-scale data security, where BigID’s startup innovation builds on AWS’ corporate investment.

“Companies like AWS are spending a lot more money on security than most organizations can,” Murphy said. “They have huge security teams; they’re building massive infrastructure. And then on top of that [ecosystem partners] can use products like BigID to make themselves more secure from outside threats and inside threats as well.”

BigID connects with over 12 different AWS native applications, including object storage service S3, cloud data warehouse Redshift, serverless interactive query service Athena, and real-time data processing and analysis service Kinesis Data Streams, as well as third-party apps built on AWS by cloud data platforms, such as Snowflake Inc. and Databricks Inc.

“The technical teams and department teams [at AWS] have been an enormous part of our success there,” Murphy said, giving a shoutout to the support provided by the AWS Partner Solutions program and associated Independent Service Provider Accelerate co-sell program.

Retrieving keys from the data dumpster

Imagining sensitive data assets of personal keys, Murphy draws an apt analogy to illustrate the importance of data discovery.

“I live in Manhattan,” he said. “If I were to collect all the keys of everybody that lived in a 10-block radius around me and put it into a dumpster, and keep doing that, I would theoretically know where all the keys were. They’re in the dumpster. Now, if somebody asked me, “I’d like my keys back,” I’d have a really hard time giving them because I’ve got to sort through 10,000 people’s keys.”

Through data discovery, BigID creates an index for this “data dumpster,” tagging data “keys” and making them easy to relocate.

Another way in which this analogy applies to cloud-scale data management is the ability to gain value from secure, anonymous data. The keys in the dumpster are not connected to personal information, such as a name or address, but they do hold a lot of clues to the owners’ personality and social status.

“Are you in an old building? Are you in a new building? Do you have a bike? Do you have a car? Do you have a gym locker? There’s all sorts of information,” Murphy said. “You can gain a lot of theoretically innocuous but valuable information from the data that’s there while not compromising the sensitive data.”

While humans could sort through 10,000 keys and label them, it would be an arduous task. With big data reserves in the terabytes, the limits of human organizational scale have long been passed.

“Let machine learning take care of things that it can,” Murphy advised. “Don’t waste human hours that are precious and expensive unnecessarily, and make better decisions based on that data.”

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase Event: Innovators in Cloud Data. (* Disclosure: BigID Inc. sponsored this CUBE Conversation. Neither BigID nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE