This is the second installment of a two-part series on how a team from the Institute for Human and Machine Cognition (IHMC) qualified to join 13 other teams to compete in the upcoming Defense Advanced Research Projects Agency (DARPA) Robotics Challenge (DRC) Trials.  The event will take place Dec. 20 and 21 at the Homestead-Miami Speedway in Homestead, Fla., where spectators can watch robots being tested on the capabilities that would enable them to provide assistance in future natural and man-made disasters.

The IHMC is a not-for-profit research institute of the Florida University System that develops robotics software and hardware with a goal of leveraging and extending human capabilities. The first installment of “Robots Engineered by Agile Teams…” discussed the team’s agile software process, its focus on test-driven development and how, with continuous integration (CI), multiple developers added changes concurrently with testing done in concert.

No time for downtime – overcoming ‘Robot time’ constraints

The IHMC team regularly faces a workflow bottleneck unique to robotics development – a thing they like to call ‘robot time’. Testing in a virtualized environment can only approximate the physics and behavior of a robot operating in the real world, so the team must regularly test their code on the real robot. Traditionally this involved each sub-team scheduling face-to-face time with the robot at the beginning of the week, during which they would quickly iterate through as many test -> fail -> analyze -> develop cycles as possible:

The problem with this workflow is that it made for an inefficient use of the team’s scarcest resource: million-dollar robots, of which they have just one! IHMC is working with the Boston Dynamics‘ Atlas robot, an anthropomorphic robot specifically designed to negotiate terrain and operate tools in a manner similar to a human, The team powered down the robot during the lengthy analysis and development phases, waiting for the new code to be built and deployed before the next Atlas test run. With just a few weeks left until the competition and swathes of tests to run, the team could not afford to waste this much ‘robot time’.

They switched to a new workflow optimized for maximizing the time spent actually running tests on the robot. As a result, the operators no longer had to halt testing to perform analysis and development when the robot did something unexpected. Instead, they raise an issue in Atlassian JIRA, issues management software, and immediately moved on to another test run, keeping the robot operating more or less continuously.

Prioritizing tasks with an issue-focused workflow

After the tests were run, developers used their Atlassian JIRA Agile backlog to triage issues that occurred during the trial. “Blocker” issues that hold up trial progress were often pulled into the current sprint while less critical issues were left in the backlog for potential inclusion in the next iteration. Switching from immediate investigation to a calculated approach allowed the IHMC team to prioritize and schedule issues so they could target higher value tasks earlier.

So how did the IHMC team actually tackle these tasks? The issues contained a brief description of the problem, including environmental metadata such as the SVN revision number and an Atlassian JIRA custom field with references to the log data captured when the issue occurred. As a web application developer, when I think of log data, I think of a rolling text file that’s a few hundred kilobytes in length, or at worst a few megabytes if it hasn’t been rotated recently. Not so here.

The “logging computer” is a beast of a machine that captures about a gigabyte of data per minute while the robot is running. This includes video feeds from the robot’s fisheye cameras, output data from the LIDAR, telemetry from various sensors mounted on the robot, feeds from static and human-operated video cameras observing the robot and log statements from the robot’s control program and operator interface. Depending on how long the robot is active, the computer can captures up to a terabyte of data on any given day!

This logging data is crucial for developers assigned to investigate an issue affecting the robot. The telemetry captured from the real robot is used to reproduce issues offline in the Simulation Construction Set, which allows developers to code and test fixes locally on their own computers before verifying them later on the real robot. Often the robot failure was encoded as a regression test run by Atlassian Bamboo to ensure that the same issue didn’t crop up again later.

With the help of Atlassian JIRA and Atlassian Bamboo, this new issue-focused workflow yields a greater amount of robot time for the team which, in turn, means higher quality software driving Atlas. This means more points at the DARPA Robotics Challenge in Miami and, ultimately, one step closer to building functional robots to assist in recovery from natural and man-made operations, potentially saving hundreds if not thousands of human lives.

Will it pay off? The physical DRC trials are open to the public at the Miami-Homestead Speedway, so come along and check it out for yourself! If you can’t make it to Florida, you can always follow along at http://www.theroboticschallenge.org/ and on DARPA’s YouTube channel DARPA TV. Either way, this is sure to be an exciting competition that could have a drastic impact on the recovery and health of our communities in the future.

About the Author

Tim Pettersen is a developer at Atlassian. He’s spent the last few years working on developer tools, most recently the hot new enterprise Git hosting solution Atlassian Stash. Tim’s passions in software are pluggability, API design and integration. When he’s not speaking at conferences, Tim enjoys hacking on anything android, git or realtime related.