Fastly Inc. has shared more details about the brief but widespread outage in its content delivery network on Tuesday that took many of the internet’s most popular websites and cloud services offline.

Publicly traded Fastly helps companies speed up page load times for their users. Its content delivery network stores copies of a website’s content on servers spread around the world.  Fastly downloads the content onto visitors’ browsers from the nearest server, which is often closer to the user than the infrastructure on which the original copy of the website is hosted. The result is an often significant decrease in loading times.

Fastly’s outage on Tuesday left users unable to access popular news sites, music streaming platforms, payment apps and the U.K. government’s information portal, among many other services. The company disclosed shortly after the incident began that it wasn’t the result of a cyberattack. However, it didn’t share too many other details until now. 

Nick Rockwell, Fastly’s senior vice president of engineering and infrastructure, wrote in a blog post today that the outage was the result of a software bug introduced into the content delivery network as part of a May 12 update. The bug was triggered by what the  executive described as a “valid customer configuration change.”

“On May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances,” Rockwell detailed. “Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.”

The executive also shared a timeline of the outage. It began at 5:47 a.m. EDT and was detected within a minute by Fastly’s engineers, who then set to work on identifying what caused the disruption. They spotted the customer configuration change that set off the outage after about 40 minutes. Fastly’s services began coming back online at 6:36 a.m. and most were operational by 7 a.m.

“Once the immediate effects were mitigated, we turned our attention to fixing the bug and communicating with our customers,” Rockwell wrote. “We created a permanent fix for the bug and began deploying it at 17:25.”

Fastly, the executive wrote, is currently in the process of rolling out the fix to all its systems. The company’s plan is to take a closer look at its software testing and quality assurance processes to understand why they didn’t catch the bug. Fastly’s engineers also plan to evaluate its incident response procedures to find ways of speeding up the pace of outage remediation efforts.

Rockwell concluded by stating that “even though there were specific conditions that triggered this outage, we should have anticipated it. We provide mission-critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support.”

Fastly’s shares climbed more than 10% on Tuesday despite the outage, but offset some of the increase today and closed down 2.7%. Shares of rival content delivery network operators Cloudflare Inc. and Akamai Technologies Inc. jumped as well on Tuesday. 

Photo: Fastly