Software testing before deployment can be a long, winding and tedious process because software development itself is a complex process. At Facebook Inc., where apps are used by hundreds of millions of users a day, testing before release is of utmost importance.

To ease this process, Facebook engineers today announced the launch of Sapienz, an intelligent automated testing system designed to develop test cases for software rolling off the production line.

Sapienz technology uses artificial intelligence and computational search algorithms to sample the space of all possible tests using what’s called search-based software testing.

Using this principle, Sapienz builds a model of the software under test, through user interface interaction, and harvests good tests for subsequent use. The system is a hybrid approach that combines low-level individual events, as seen from a “code view,” and higher-level patterns of related events known as “motif events.” By combining knowledge of these approaches, tests can capture the chain of events of low-level events as they cascade into high-level patterns.

Also important, the Sapienz system works through the user interface, the same way users and testing engineers would test the system, making them easy for a human to replicate. This reduces the chances of false positives common to many automated testing systems. But it also makes it much more difficult for a computational system to model.

Above is a video of proof-of-concept Sapienz in action on Android devices during its early stages of development. It shows the system automating tests across multiple devices by rapidly launching apps and running through the UI. At the end of this process, Sapienz would report back how the software and devices reacted to its virtual fingertips running swiftly over the software.

“We first deployed Sapienz at Facebook in September 2017,” Facebook Engineering Manager Mark Harman (pictured) and Facebook Software Engineer Ke Mao said in a blog post, “and have since scaled it to meet the challenge of testing millions of lines of code in Facebook’s Android app.”

Mark Harman and Ke Mao did a presentation about Sapienz today at Facebook’s F8 developer conference. The video is available online with more details for developers.

Sapienz now executes tens of thousands of test cases on Facebook apps daily and it automates the design and deployment and reporting for each. In merely a few months after its deployment it has cut the time it takes to diagnose and fix issues within hours, sometimes minutes.

Facebook also revealed that Sapienz has an extremely low false-positive rate. Engineers reported that 75 percent of Sapienz reports were actionable and resulted in fixes.

As part of software development and deployment, the tool reports directly into the continuous integration DevOps platform Phabicator, a code-review tool used by Facebook engineers to ensure code quality and rapid deployment. The Sapienz tool does so by commenting just like a human engineer would. Since its process is automated, engineers pushing code through Phabicator receive real-time reviews and commentary from Sapienz as if they had a team of testers in the next room.

Automated testing is an important segment of the software development industry. Numerous services have developed at-scale testing that tries to trigger issues the way users would, such as Sauce Labs Inc. building a pool of devices to test against or Rainforest QA crowdsourcing actual humans to do scale testing. Automated strategies have emerged from numerous parts of the industry including CA Technologies, Microsoft Inc.’s Xamarin and and many others.

According to a report from Zion Market Research, the automated market testing market is expected to reach $55 million by 2022. This expectation is driven by what ZMR says is a trend towards digital transformation across the globe and the need for companies to ease the high costs and time consuming nature of human testing. Automated tests also allow companies such as Facebook to test broad distributions quickly and reduce the total time to test. That saves time and money on testing while also reducing headaches by preventing more bugs from reaching deployment.

The Sapienz system grew out of an academic paper and was brought into production at Facebook-scale in just 17 months. Harman and Mao said they believe this represents an example of how academic researchers and engineers can collaborate to take ideas from the drawing board into the real world to create big benefits.

“We are just at the beginning of our journey with Sapienz at Facebook and expect over the next few years to implement and share new applications and feature enhancements,” said Harman and Mao. “We also expect that search-based software engineering will find many new exciting and innovative applicators, both here at Facebook and in the wider tech sector as a whole.”

The Sapienz technology is currently being extended to test other Android and iOS apps in the Facebook product line.

Photo: Facebook