Security of the software supply chain has gained significant attention over the past year. Two major cybersecurity attacks — SolarWinds and Kaseya — proved sharp reminders to reexamine every component of software development and deployment, including what they are and where they came from.

The first signs of major supply chain vulnerability actually showed up over four years ago when the malware wiper NotPetya was launched against Ukraine in 2017. NotPetya attackers, believed to be threat actors in the Russian military, allegedly injected malicious code into accounting software owned by a Ukrainian company. The result was an estimated $10 billion in damages that impacted organizations across Asia, Europe and the Americas.

From a cost standpoint, NotPetya was a mere drop in the bucket. Incident response costs to repair damage from the SolarWinds breach are estimated to exceed $100 billion, while additional costs for recovery from the Kaseya attack earlier this year are still being calculated. Bottom line: The cost of software supply chain vulnerability has risen exponentially and open source is proving to be a prime target.

“We’re starting to see an increase in attacks,” Luke Hinds, security engineering lead, Office of the CTO, at Red Hat Inc., said during a recent interview with theCUBE, SiliconANGLE Media’s livestreaming studio. “There’s a recent statistic that came out … a 620% increase since last year of supply chain attacks involving the open-source ecosystem. So, things are certainly ramping up.”

A multifaceted problem

The recent incidents prompted the White House to issue an executive order in June designed to protect government systems from software supply chain attacks. Even more significantly, concerns around software security have galvanized some of the largest enterprise computing firms in the world to find technology solutions that reduce the potential for future exploits.

Part of the problem is that not all of the world’s software sits behind locked gates. In fact, 90% of information technology leaders say they are using enterprise open-source tools, according to Red Hat’s “The State of Enterprise Open Source” report, and 79% expect use of open-source software to increase over the next two years.

The challenge of protecting open-source software, which is built by many contributors, is akin to securing a bank vault where everyone is free to come and go as they please. What’s inside is extremely valuable, but there are no guards or locks on the vault door.

“This is a multifaceted problem,” said Sandy Carielli, principal analyst at Forrester Research, in an interview with SiliconANGLE for this story. “Enterprises have a desire to put in additional controls and analysis for software supply chain security to reduce risk. One of the problems is that open source is so pervasive.”

Open-source initiatives

Red Hat Inc. has offered a few ideas around how to address this challenge, and it starts with a project called Sigstore, which automates digital signing and verification of software elements. The goal is to provide a free tool that will verify software’s provenance, thus allowing developers to use open-source solutions securely. The bank vault is still wide open, but only verified, trusted sources are allowed to touch what’s inside.

Sigstore is a project originally prototyped at Red Hat, and the company has positioned it as an “open answer” to software supply chain trust and security. Red Hat has been actively seeking to galvanize industry support around the digital signature solution.

“We have big plans with Sigstore, and that’s where partnership comes into the picture,” said Chris Wright, senior vice president and chief technology officer at Red Hat, in remarks at the Linux Foundation’s Open Source Summit this fall. “So far, the community counts more than 675 commits, more than 465 members and 20 organizations. To offer all of these capabilities and become a widely used free service, we need to establish trust.”

This pursuit of partnerships and trust has led Red Hat and IBM to join with a host of other large enterprise players in supporting the Open Source Security Foundation. The collaboration, led by the Linux Foundation, has already raised $10 million from the two firms and other notables, including Amazon, Cisco Systems, Dell Technologies, Microsoft, Google, Oracle and VMware.

The initiative brings together multiple projects under a single umbrella with the charter to find and fix security vulnerabilities in open-source software. Open-source visionary Brian Behlendorf, who was a co-creator of the Apache web server, is OpenSSF’s general manager.

“These groups know their stacks are made up of largely open-source software, so they’re looking to pay it forward to these indirect dependencies,” Behlendorf said in a recent interview. “By improving the baseline of open source, they’re going to get better quality code in the end.”

Cloud tools and ClusterFuzz

Container security is also a part of the software supply chain discussion. Aside from supporting OpenSSF, IBM announced in July that it would extend Sigstore with an ability to verify the origin and integrity of a Kubernetes manifest with cryptographic signing. The aim is to ensure a resource has not been modified maliciously before being applied to a Kubernetes cluster.

IBM has also automated the practice of consulting Common Vulnerabilities and Exposures databases to reduce the risk of using compromised packages and binaries in applications. The company’s Anaconda Repository for IBM Cloud Pak automates this work for IT administrators.

Amazon Web Services Inc. offers two separate tools that can assist in monitoring supply chain security. One is AWS CloudTrail, which provides a log of all actions that have occurred inside a user’s AWS environment. The other is Amazon CloudWatch, a monitoring and observability service for developers and IT managers. Logs, metrics and events are collected to provide a unified view of AWS resource use.

Google has recently made an enhancement to its open-source ClusterFuzz tool, with a focus on improving software supply chain security. In early November, the search giant announced the release of “ClusterFuzzLite,” which throws random data at computer programs to identify bugs that could be leveraged by threat actors.

The latest enhancement is an outgrowth of Google’s OSS-Fuzz project launched five years ago to inject quality assurance into open-source initiatives. One interesting sidebar to Google’s most recent announcement involved the data it released. The company noted that OSS-Fuzz had identified 6,500 open-source vulnerabilities and corrected 21,000 functional bugs to date.

Bill of materials solution

Creating a system for verified signatures and implementing tools to spot dangerous behavior or fix vulnerabilities are steps in the right direction, but there are indications from U.S. government officials that more transparency will be needed. The Executive Order issued this year by the White House called for a software bill of materials, or SBOM.

This approach is similar to what is often found in the manufacture of goods. Much as manufacturers provide an inventory of all items contained in a specific product, a software BOM would provide a list of all open-source and third-party elements contained in a codebase.

Baseline information for an SBOM has been developed by the National Telecommunications and Information Administration. At a minimum, the government is seeking to make the SBOM a requirement for providing software to federal agencies. Whether it will be adopted as a universal standard within the tech industry remains an open question.

“I’m a big fan of the software bill of materials initiatives that are underway,” said Forrester’s Carielli. “We need controls to track and manage software. Making new versions of open-source software less vulnerable is an important initiative, but we can’t confuse that as some sort of silver bullet for all software supply chain security.”

How the tech industry ultimately responds to vulnerabilities in the software security supply chain is a work in progress. Initiatives such as Sigstore to address concerns around open-source security have been a good place to start, but trust can be a tricky proposition in a software ecosystem based on sharing and global use of common tools. Yet, it will likely take trust as part of the open-source ethos to provide an answer for today’s security conundrum.

“We pull each other’s code all the time, which demonstrates that high level of implicit trust,” Red Hat’s Wright said in a recent blog post. “Transparency, partnership, trust. That’s how we’ve gotten it done in the past, and that’s how we’ll continue to do it in the future.”

Image: Sergey Nivens