A post this week on Cisco Talos’ blog literally shows the power of one person’s grit and determination.

Joe Marshall, a cyberthreat researcher at Cisco Systems Inc.’s threat intelligence research team, was seeing what was happening in Ukraine, where Russian attacks on its electric grid had left millions of its citizens without power last winter. “What started as a chance presentation this year turned into a multi-national, multi-company global team of power grid security practitioners who had never worked together before,” he wrote. Marshall pulled in assistance from Cisco along with various U.S. and Ukrainian government experts to help rebuild these systems and keep the grid running.

A year ago, nearly half of the grid had been destroyed. One of the problems with resurrecting the grid is that Ukraine’s high-voltage lines rely on GPS satellites for time synchronization – and one of the first attacks by Russia was to jam these signals to prevent Ukrainian forces from targeting Russian troop locations.

When GPS signals can’t reach power controls, these synchronization efforts fail and the utility can’t manage its loads between different parts of the system. This means that the entire stability of a power grid becomes a problem, and it can’t balance loads and supplies. Americans are only vaguely aware of this problem, until a major storm causes an outage.

GPS signals are broadcast all over the globe and have an accuracy of 100 nanoseconds. They are received by everyone’s cell phone, for example, until the signals are jammed.

Marshall had a chance meeting with staff from the utility where he first heard about the problem, and offhandedly suggested it purchase atomic clocks to synchronize its signals. One of the staffers retorted that this wasn’t really a workable solution, given their expense and difficulty in supplying these units.

But the issue stayed with Marshall, who spent time talking to various Cisco staffers who were specialists in the internet of things. One of its team members suggested a way to use Cisco’s Industrial Ethernet switch networking gear as the basis of a solution for the power company in Ukraine.

“The switches do not have atomic clocks for holdover accuracy – but they have a good enough clock, able to measure time accurately in microseconds, to sustain an accurate time sync, for Ukraine’s power purposes,” Marshall wrote. But the off-the-shelf versions still needed modification.

Cisco needed to have its switches handle the hours-long GPS outages, which required new firmware and recovery algorithms along with a way to recognize when GPS signals were blocked. They set to work and came up with the needed modifications.

The blog post is interesting in what Marshall doesn’t mention. Normally, Talos blogs are packed full of cybersecurity issues such as describing distributed denial-of-service or malware campaigns.

The story of the Ukrainian power grid has nothing to do with these exploits, but how the various people worked together to provide a custom solution. As Marshall wrote, “Our ultimate goal was to keep the lights on in Ukraine, and help make the lives of Ukrainians who are living in an active war zone, just that little bit easier.”

Image: Cisco/Talos