The professional outsourcing firm Capgemini SE has developed a solution designed to identify sperm whales accurately using computer vision using image processing and cloud computing to look at the animals’ tail fins or “flukes.”

Whale flukes can be used to identify individual animals because the tails gain unique features over their lives – similar to human fingerprints – depending on the whale’s unique lived experiences.

Scientists can then use photographs of the flukes when the whales dive under the water to document key information about the animals, such as behaviors, migratory paths and other movement patterns.

Sperm whales are not the easiest animals to watch because of their long dive times and it’s unknown how many there are in the world. It is currently thought the population is currently in the hundreds of thousands. Although commercial whaling has ceased and the species is protected worldwide, fishing-line entanglement and collisions with ships represent the greatest threats to them.

The process of matching flukes from photographs can prove to be tedious, difficult and very labor-intensive.

Lisa Steiner, a marine biologist and renowned expert for sperm whale research in the Azores, an archipelago of islands in the mid-Atlantic near Portugal, told SiliconANGLE in an interview that she began long ago with black-and-white photos set on her living room floor laid out like a rogues gallery of whale flukes that she would pore over, looking for likely suspects in an attempt to make identifications.

Modern advances in software have done away with that, of course, but the tedium remained.

“I have to manually assist the program to pick out the contour for each half of the fluke,” Steiner said. “If the photos are good, this process doesn’t take very long. However, if there isn’t a lot of contrast between the fluke and background or there is a lot of glare on the edge of the fluke, I have to follow the contour manually.”

A Capgemini employee went on a whale spotting expedition with Steiner and experienced this awkward method firsthand and felt she could improve it. Thus, she proposed replacing the dated software with something that could automate away the identification process using computer vision and machine learning.

To make this happen, Capgemini created the theme for the 2020 annual Global Data Science Challenge, which would use Amazon Web Services Inc.’s machine learning capabilities.

As a result, nearly 700 teams from across Capgemini participated in the challenge. After several months the winning team, called “AI Sensing,” presented a solution consisting of a pre-trained deep neural network trained on 4,500 pictures containing flukes of more than 2,200 individual whales.

Reaching out to AWS and the cloud

The model runs on a graphics processing unit-based Amazon Elastic Compute Cloud compute cluster using Amazon SageMaker. That’s an AWS service that helps everyday developers and data scientists build, train and deploy machine learning models quickly. Capgemini presented the new solution at AWS re:Invent 2020, a 3-week virtual annual developer conference put on by Amazon.

In three hours, the platform can compare a photo to all images in the entire database and, according to Capgemini, the solution has an accuracy rate of almost 97.5%.

“We are optimistic about the potential of machine learning and its ability to materially benefit society,” said Mike Miller, director of Amazon Web Services’ AI Devices division. “This unique solution for Lisa Steiner is a testament to Capgemini’s capabilities, and the potential of machine learning to positively impact whale conservation and solve some of the world’s toughest challenges.”

With access to this new solution, Steiner has been freed up to focus her time on tasks such as migration tracking and habitat protection – and she needs to take less time spent pouring over photos in an attempt to tag them to individual whales.

The system even matched a whale that had not been “seen” for almost two decades because of changes in the contour of its fluke.

“I found quite a cool match that had not been detected previously, due to some changes in the contour,” said Steiner. “This goes back to 2002, so there were three separate IDs for this animal, instead of one. If I had just clicked match on the first photo, I wouldn’t have found it.”

Image: Capgemini