Thanks to the anonymity of the internet, it is next to impossible to effectively fight online harassment, as services like Twitter have repeatedly learned. Jigsaw, a subsidiary of Alphabet Inc. (Google), wants to change that by using machine learning to weed out caustic people in online communities.

In a recent feature in Wired, Jigsaw founder and President Jared Cohen explained that his company has created a new tool called Conversation AI, which he says is capable of recognizing abusive language with far greater accuracy than keyword filters or other similar programs.

“I want to use the best technology we have at our disposal to begin to take on trolling and other nefarious tactics that give hostile voices disproportionate weight,” Cohen told Wired. “To do everything we can to level the playing field.”

According to Cohen, the goal behind Conversation AI is to eliminate the self-censorship caused when people receive death threats online for voicing their opinions, which Cogen says is an unfortunately common occurrence, especially toward women and minorities.

To teach Conversation AI how to recognize abusive language, Jigsaw first fed it with language data from a number of sources, including 17 million comments from New York Times articles and 130,000 text snippets from discussions on Wikimedia. Jigsaw then had panels of 10 random people read through the data and decide whether they constituted person threats or not. From there, the company fed that data into Google’s TensorFlow machine learning program.

Cohen admits that Conversation AI is still not perfect, and according to Google, the AI still has a roughly 10 percent rate of false positives. In these cases, Conversation AI could accidentally censor legitimate conversations, resulting the exact opposite of what Jigsaw’s goal of anti-censorship.

“When you’re looking at curbing online harassment and at free expression, there’s a tension between the two,” Cohen said. “We don’t claim to have all the answers.”

The New York Times and Wikimedia will both be the first to test out Conversation AI’s capabilities. Wikimedia has not yet stated how it plans to use Jigsaw’s new program, but the New York Times says it will put Conversation AI to work as the first line of defense in moderating comments on its web articles.

Reforming toxic users with AI

Harassment is a problem in pretty much every online community, and multiplayer gaming is no different. Over a year ago, Tencent-owned Riot Games introduced a system for its popular online game, League of Legends, that shares a lot in common with Jigsaw’s Conversation AI.

Like Jigsaw, Riot took real world language data that had been evaluated by real humans, in this case the millions of people who play League of Legends. An auto-moderator system called Tribunal generated public case files containing chat logs and game data for players who had been reported for toxic behavior, and users could then review those files and vote on whether or not they were actually toxic or merely standard video game trash talk.

Riot then used machine learning to compare the language data with the human evaluations to create a system that, like Conversation AI, can automatically detect negative behaviors and issue bans and other punishments based on that information.

Photo by hang_in_there