Two years of a global pandemic brought an acceleration of businesses to the cloud, followed by another important wave of digital transformation. The move to the cloud has also been accompanied by a need to leverage data even more effectively in the enterprise.

Cloud provider Amazon Web Services Inc. is in prime position to observe this transformation with its extensive customer and partner ecosystem. Data is driving the process of reinvention for many firms as the private sector emerges from the disruption caused by a virus that swept the world.

“The first wave of reinvention was really driven by the cloud as customers were able to transform technology,” said Rahul Pathak (pictured), vice president of analytics at AWS. “We are seeing another wave of reinvention, which involves companies reinventing their businesses with data. Our customers are looking at ways to put data to work; it’s about making better decisions, finding new efficiencies, and finding opportunities to succeed at scale. It’s really about the survival of the most informed.”

Pathak spoke with theCUBE industry analyst John Furrier during the opening session of the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event, an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed the fundamental importance of data as code, integrating machine learning within cloud systems, the rise of serverless adoption, and how customers must deal with massive amounts of data. (* Disclosure below.)

Focus on machine learning

The data as code movement allows enterprises to manage data programmatically, with the capability to automate pipelines, version information, and share it in collaborative models across clouds. This fits neatly with AWS’ focus on machine learning within many elements of its cloud-based portfolio.

“The idea of data as code and bringing some of the repeatability of processes from software development to how people build data applications is absolutely fundamental,” Pathak said. “It’s especially so in machine learning where you need to think about the explainability of a model. These ideas are showing up in all stages of the data workflow.”

To this end, AWS has pursued a deliberate strategy to infuse many of its core enterprise tools with machine learning capabilities. Its Athena interactive query technology and Redshift data warehouse offering are just two examples of how the cloud giant has built machine learning into its enterprise data solutions.

“We’re trying to bring machine learning closer to data,” Pathak explained. “We want to provide a comprehensive set of capabilities for ingestion and cataloging, analytics and machine learning. It’s about unifying data wherever it lives, connecting it so customers can build a complete picture of their business.”

Big trend in serverless

There is another layer of technology that AWS has added to its tools within the past year, and this involves the use of serverless services. The company believes that customers want to run code, manage data, and integrate applications without the burden of managing servers.

“Serverless has been another big trend for us,” Pathak said. “Within analytics, everything that we offer has a serverless option. We announced serverless Redshift, serverless EMR, serverless Kinesis and Kafka. The goal here is to take away the need to manage infrastructure for customers so they can focus on driving differentiated business value.”

Realizing differentiated business value can be a significant challenge for enterprises when the sheer volume of data exceeds a legacy infrastructure’s ability to manage it. Pathak offered a couple of customer use cases that exemplified the scale on which today’s critical applications must run.

The social media platform Pinterest Inc. has 431 million monthly active users and leverages AWS technology to scale daily log searches by a factor of 3x or more. The Financial Industry Regulatory Authority, or FINRA, uses Amazons EMR Spark and Hadoop services to process a massive number of records.

“They ingest 250 billion records per day,” Pathak noted. “FINRA processes 20 TB of data daily running across tens of thousands of nodes, looking for fraud and bad actors in the market.”

The AWS ecosystem of startup companies is pursuing a wide range of data-driven opportunities in a space where companies are looking for effective business solutions. Enterprises need to deal with an avalanche of information and build infrastructure that maximizes this digital asset to the fullest.

“There is no slowdown in sight for the volume of data that we’re generating,” Pathak said. “As long as we and the startups we work with continue to push on making things better, deal with more data cheaply, make it easier to get inside, and maintain a super high bar in security, investments in these areas will just pay off.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event:

(* Disclosure: TheCUBE is a paid media partner for the AWS Startup Showcase: “Data as Code — The Future of Enterprise Data and Analytics” event. Neither AWS, a sponsor for theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE