An MIT researcher, Associate Professor Devavrat Shah, says he has developed an algorithm that can accurately predict what topics will trend on Twitter. Shah claims a 95 percent accuracy rate during testing of his model and has been predicting trends hours before they appear on Twitter’s list.

Wikibon Analyst Jeff Kelly says that the new algorithm relies more on the ability to ingest
larger amounts of data across distributed systems. In traditional algorithms and data models, certain human assumptions come into play. He said, “So to the extent that you can take out the human element in the algorithms and models that you apply to data, the more accurate they’re likely to be.”

Twitter is currently using their own algorithms to display what’s trending in real-time. They’re always looking to improve the algorithm and process because it translates to dollars for them. The more accurate they are and the earlier they can display what’s trending, the more they can charge for ads and sponsors.

Kelly said that Twitter could possibly use this to improve its existing service and consider building into their own models or they could take the position that their own algorithm is superior. He said, “With big data and machine learning, it’s all about adapting, so I have no doubt that Twitter and the data scientist team there is always looking for ways to improve their algorithm.”

Shah’s algorithm could have potential usage in many areas, including stock markets, natural disasters, or even combatting cybercrime. See the whole segment with Kristin Feledy and Jeff Kelly on the Morning NewsDesk show.