Data engineers and architects are finding themselves in high demand because data is increasingly used to drive business decisions. To keep up with the speed and scale at which data is made available, as well as the heightening regulations that protect it, organizations are rethinking how data engineers perform their tasks and efficiently manage a secure data pipeline.

As part of their day-to-day job, data engineers are required to ensure that the data they release to analysts is only that which the analysts are allowed to see. In this way, compliance can complicate the data transformation work that analysts need to do.

“Data analysts downstream are hamstrung to a certain extent and bottlenecked by requesting data engineers do some of this transformation work for them … that bottleneck could be a back breaker for organizations,” said Steve Touw (pictured), co-founder and chief technology officer of Immuta Inc. “You need to tie transformation with compliance in order to streamline the analytics in your organization.”

Touw spoke with Keith Townsend, guest host of theCUBE, SiliconANGLE Media’s livestreaming studio, during AWS re:Invent. They discussed the increasingly cumbersome tasks of data engineers, the paradigms of legacy roles based access control vs. attribution-based access control, the ability to share data more flexibly with third parties, and more. (* Disclosure below.)

Separating policy from platform

At the heart of this discussion is the term “role-based access control,” which has been in use for years and involves aggregating user roles and then building rules around those roles.

“We believe that the roles-based access control paradigm is just broken,” Touw said. “We’ve got customers with thousands of roles that they’re trying to manage, to slice up the data all different ways they need to.”

Immuta offers a more dynamic attribute-based access control solution and policy-based access control solution that enables organizations to dynamically enforce policy by separating who the user is from the policy that needs to be enforced and having that execute at runtime.

Touw offered an analogy to role-based access control, explaining that it is like writing code without being able to use variables. Instead of writing the same block of code over and over again with slight changes based on role, attribute-based access control allows for more variables and the policy is decided at runtime based on who the users are and what they’re doing.

The dynamic nature of attribute-based access control lends itself to the public cloud. Because Immuta enforces policy at the data layer, it allows analysts the flexibility to use the tools at their disposal, such as those within AWS. 

“It all comes down to scalability,” Touw said. “For the same reasons you separate storage from compute … you want to separate your policy from your platform.” 

Watch the complete video interview below, and be sure to check out more of SiliconANGLE’s and theCUBE’s coverage of AWS re:Invent. (* Disclosure: Immuta Inc. sponsored this segment of theCUBE. Neither Immuta nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE Media