Twitter Inc. has open-sourced an internally developed tool for discovering performance anomalies and usage spikes in its network that were too brief to be noticed by its regular observability and metrics systems.

In a blog post Monday, Twitter site reliability engineer Brian Martin said Rezolus is a “high-resolution systems performance telemetry agent” that’s now available to download via its GitHub repository.

Twitter uses Rezolus to help it quantify workload characteristics, diagnose runtime performance issues and gather data for network optimization, Martin said.

“Rezolus provides a collection of signals to help us make sense of fine-grained runtime behavior,” he wrote. “We’ve found it particularly helpful in understanding and optimizing performance. With a single agent, we’re able to get telemetry from a wide range of sources. To our knowledge, no other open source project offers such comprehensive insight in a single package.”

Martin explained that Twitter built Rezolus in order to be able to observe its systems’ performance on a “fine-grained timescale.” The problem was that its engineers regularly came across “seconds-long performance anomalies” in its network. But they were unable to diagnose the cause of these problems because their existing telemetry systems returned a low sample rate relative to the length of those anomalies.

In order to diagnose network problems, sampling rates must be at least twice the duration of the shortest anomaly in order to accurately reflect its intensity. So the company built Rezolus in order to measure network performance degradation more precisely.

“One time, several services were experiencing repeatedly degraded success rates for a few minutes at a time,” Martin explained. “These services each found that they were being throttled by a backend service. The team responsible for that service didn’t see anything on the existing telemetry that they could use to figure out what was happening during the minutes where throttling was occurring. But knowing that throttling decisions are made on a finer timescale than the default telemetry collection, they began to suspect sub-minutely bursts.”

By deploying Rezolus to its backend service, Martin’s team was able to find out exactly what had happened.

“We were able to see that even though average request rates weren’t significantly elevated, there were bursts of over five times the baseline traffic during which CPU utilization was bursting to 100%,” Martin wrote,” We were also able to identify exactly when they happened. With the additional telemetry from Rezolus, we were able to correlate with the backend service logs and determine the source of the spikes.”

Martin said Twitter had decided to open-source Rezolus in order to help other companies facing similar network performance issues and build a larger community around the project.

Image: edar/Pixabay