These days companies often issue new releases and updates of their software several times a day to millions of users in the cloud, and no matter how much they try to make sure it will run flawlessly, it’s nearly impossible to guarantee it.

That’s all the more true today because companies are increasingly using many microservices, or pieces of applications, in their software, making the discovery of problems extremely difficult and time-consuming. The basic question large companies are asking is simple, according to Google Cloud product manager Andrew Phillips: “How do we release more quickly but not break stuff?”

Google LLC and Netflix Inc. think they have the answer in a new service they’re delivering via an open source license today: Kayenta. It’s essentially a more automated way for companies to test their new release against the existing release that works OK and determine quickly whether the new version will work OK too.

In more technical terms, Kayenta is an automated “canary analysis” service. As Google describes canary analysis — not surprisingly named after the idea of a canary in a coal mine for detecting deadly gases — it’s a software release deployment process in which a change is partially rolled out and then evaluated against the current deployment called a baseline to make sure the change is operating at least as well as the baseline, using a variety of metrics.

Kayenta works with Spinnaker, a commonly used open-source continuous software delivery platform for releasing software changes very quickly on multiple clouds. Spinnaker was developed originally by Netflix for its own operations, which are about as demanding as any on the internet, and the streaming video company worked with Google to turn it and now Kayenta into open-source services anyone can use.

Although Google’s interest is likely in part because it wants to make it easier for companies to deploy new software quickly on its cloud, Kayenta can be used more widely. It does a rapid statistical analysis of the new release, based on metrics the user chooses, such as processor and memory usage and error rates, and provides an aggregate score for the “canary”:

That’s in stark contrast to common current practice, in which an engineer looks at logs and graphs manually on analytics dashboards to assess whether the new release will work well, which introduces a lot of errors and doesn’t even work for very large deployments of software multiple times a day.

“Most of the tooling in this space is either written for wizards… or it’s so restrictive that you can’t really use it [for your system],” Phillips said in an interview. “Kayenta is a reimagining of Netflix’s version. [It] takes this advanced piece of technology and makes it work at scale.”

But it’s also “pluggable,” meaning it’s open for other software tools to plug into it, such as metrics and monitoring services. For now, it supports Prometheus, Google’s Stackdriver, Datadog and Netflix’s Atlas, but Phillips implied that it will eventually support others such as Amazon Web Services Inc.’s CloudWatch.

Netflix and Google have high hopes for Kayenta. Greg Burrell, a senior reliability engineer at Netflix, said he expects Kayenta to be making thousands of canary “judgments” per day by the end of the year. There’s a lot more technical detail on Netflix’s post today.

Photo: Animals Town