Predicting Twitter trends with 95 percent accuracy

Massacusetts Institue of Technology (MIT) Associate Professor Devavrat Shah and his student, Stanislav Nikolov, claim to have found an algorithm that can predict trending topics on Twitter as much as five hours in advance —and with an accuracy of 95 percent, according to the MIT News. 

 

"(The algorithm) combs through data in a sample set — in this case, data about topics that previously did and did not trend — and tries to find meaningful patterns. What distinguishes it is that it’s nonparametric, meaning that it makes no assumptions about the shape of patterns," the MIT report said.

 

Shah's and Nikolov's new algorithm used an initial training set of 200 topics that did trend, and 200 topics that didn't. 

 

The algorithm then compares changes in the number of tweets about new topics over time to the changes of every sample in the training set over time. Each sample gives a weighted "vote" on whether or not a topic will trend, and the combined weighted votes then give a probabilistic estimate of the possibility that a topic will trend.

 

Twitter trends are a goldmine

 

Twitter's "Trending Topics" list is highly prized by product makers, advertising agencies, and media organizations as a source of insight into current public sentiment as well as for free publicity.

 

"(Twitter) could charge a premium for ads linked to popular topics, but it also represents a new approach to statistical analysis that could, in theory, apply to any quantity that varies over time: the duration of a bus ride, ticket sales for films, maybe even stock prices," the MIT report said.

 

"People go to social-media sites to find out what's happening now," explained Ashish Goel, a member of Twitter's technical advisory board and an associate professor of management science at Stanford University.

 

"So speeding up the process (of analysis) is something that is very useful," he said.

 

Waiting for the 'jump'

 

Shah explained that, unlike the new algorithm, standard prediction models usually require specific parameters to look out for so that computers know when to alert researchers that the number of tweets about a topic is about to "jump".

 

"The problem with this is... there are a thousand things that could happen. (So we) just let the data decide," Shah said.

 

Shah said that the algorithm's accuracy can be expected to improve if given a larger training set of data.

 

To offset the greater cost of computation needed for a larger data set, Shah hopes to utilize cloud computing to distribute the workload among several machines. —TJD, GMA News