Data management startup Immuta Inc. today announced it’s updating its platform, enabling data scientists using the widely adopted data processing software Apache Spark to build algorithms that adhere to strict policies on governance and privacy.

Immuta’s data management platform focuses on preparing data for use in artificial intelligence and machine learning projects. Meanwhile, Apache Spark is a big-data processing framework that’s commonly used for large-scale structured query language, batch and stream processing.

Last September, Immuta updated its platform with new differential privacy controls and personalized data access designed to anonymize data and control who is able to access it. Now Immuta is extending these capabilities to data from Apache Spark with native policy enforcement controls, enabling businesses to process, secure and audit Spark data on a “massive scale,” the company said.

The idea is that organizations can use SparkSQL data to power machine learning algorithms while remaining in compliance with regulations such as Europe’s General Data Protection Act, which goes into effect next month. Under the GDPR, businesses will be required to protect the personal data and privacy of EU citizens for transactions that occur within EU member states. Those that fail to comply could face fines of millions of dollars.

Native policy enforcement within Spark gives organizations a way to “proactively manage and apply policy controls” to their SparkSQL data, said Immuta Chief Technology Officer Steve Touw. “We are enabling organizations to significantly increase the speed and scale at which they process, secure and audit data. This is key to meeting compliance and regulatory requirements from the GDPR, HIPAA and other laws.”

Touw said Immuta’s new capabilities can be integrated into Spark clusters without the need to configure drivers, meaning that businesses can access their Spark data without compliance worries in a matter of minutes.

Immuta is also adding a new Automated Governance Reports feature to version 2.1 of its platform. As the name suggests, this allows users to generate instant reports on data activities taking place within their workflows, providing insights such as who, when and how people are accessing company data.

Immuta’s new capabilities were given a warm welcome by David Cook, the chief information security officer at Databricks Inc., which is the lead developer of Apache Spark. He said the updates help to expand Spark’s role as a unified analytics engine for AI workloads.

“Regulated data access and dynamic policy enforcement are key enablers for Apache Spark users to build algorithms that are legal and ethical,” Cook said. “By making these innovations available for the Spark ecosystem, more enterprises can automate access to their data and manage the risk of cloud-based data science programs.”

Image: Pexels/Pixabay