Amazon Web Services Inc.’s popular neural network development platform Amazon SageMaker is getting a major refresh, with a host of new capabilities that will support the integration of faster structured query language analytics, petabyte-scale data processing and more besides.

The new updates, announced at the annual AWS extravaganza re:Invent, are designed to transform Amazon SageMaker into a more comprehensive, fully integrated platform for artificial intelligence development.

They include the new SageMaker Unified Studio, which is a portal through which customers can access data from across their organization, along with various AI and machine learning development tools, and the SageMaker Catalog, which hosts a collection of powerful large language models and other developmental artifacts.

Meanwhile, SageMaker is getting its very own data platform, called SageMaker Lakehouse, which unifies data from multiple data lakes, warehouses and operational databases and applications, making it easier for developers to access.

The company launched Amazon SageMaker back in 2017, long before the current AI development craze that was inspired by OpenAI’s ChatGPT, and it has since become the cloud infrastructure giant’s primary AI application development platform. It hosts a glut of tools for building AI applications, making it possible to create neural networks, train them, deploy and monitor their performance, fine-tune them and perform various other essential tasks in one place

Every AI developer tool in one place

SageMaker Unified Studio is available in preview now and represents a major evolution of the platform, giving users access to a simplified environment through which they can access all of their data and put it to use in AI systems. It unifies all of the tools found in Amazon’s previously disparate ecosystem of developer studios, query editors and visual tools found in platforms such as Amazon Bedrock, Amazon EMR, Amazon Redshift, AWS Glue and the existing SageMaker Studio, the company said. It’s all about making everything easier to access, so users can discover and prepare data, create queries and code, and build AI models all in one place.

Besides having everything in one place, developers will also benefit from Amazon Q Developer, an AI-powered assistant that can aid in data discovery. They can ask questions such as what data they should be looking at to get a better idea of their organization’s product sales, and immediately obtain the answers they need, AWS said.

Amazon Bedrock’s integrated development environment is also integrated in SageMaker Unified Studio, making it possible to build AI applications using an extensive library of high-performance foundation models, together with various pre-made AI agents, knowledge bases, guardrails and workflows.

The company said it’s integrating everything in response to the way customers are using SageMaker today. It explained that it has seen how most users also leverage its data analytics tools to support the tasks they’re doing with SageMaker, and so it just makes sense to bring them all under one hood to enable easier access.

A giant data repository

As for the SageMaker Catalog, it’s built atop Amazon DataZone and provides access to hundreds of approved AI models together with safeguards such as granular access controls and AI guardrails. They can prevent AI applications from exhibiting toxic or biased behavior.

With the arrival of SageMaker Lakehouse, SageMaker users gain the ability to centralize access to the underlying data assets that power their AI models, along with analytics capabilities. One advantage of this setup is that it makes it easier to combine data from multiple sources, such as in Amazon S3 data lakes, Redshift data warehouses or other, federated data sources. SageMaker Lakehouse itself can be accessed via the SageMaker Unified Studio.

It also makes it simpler for users to query data, as SageMaker Lakehouse is compatible with the open Apache Iceberg data standard, meaning the information within it can be explored with various SQL analytics tools.

The pharmaceutical giant F. Hoffmann-La Roche AG has been using SageMaker Lakehouse in early access, and says it was able to eliminate data silos and make information easier to access, without any complicated data movement procedures. As a result, it’s seeing a 40% reduction in data processing times, it said.

Easier app integrations

As it strives to make life easier for developers, AWS is also announcing what it says are “zero-ETL integrations” with various third-party software-as-a-service applications. For instance, customers can integrate Amazon Aurora MySQL, PostgreSQL and other kinds of databases directly with Amazon SageMaker, without needing to build any complex data pipelines first.

The zero-ETL integrations eliminate the complex “extract, transact and load” process that’s traditionally required to change the format of data found in one application to meet the requirements of another. Essentially, AWS does all of this itself, and it’s a big benefit because building data pipelines can be a time-consuming and error-prone process that creates major headaches, even for the biggest organizations.

AWS Vice President of Data and AI Swami Sivasubramanian said the convergence of AI and data analytics means that companies are reliant on “increasingly interconnected” data sources, hence the need to make all of that information more accessible.

“Many customers are already using combinations of our purpose-built analytics and machine learning tools, such as Amazon SageMaker, Amazon EMR, Amazon Redshift, and more,” he said. “The next generation of SageMaker brings together those capabilities, along with some exciting new features, to give customers all the tools they need for data processing, SQL analytics, model development and training, directly within it.”

Image: AWS