New software can predict future news

In the not-so-distant future, software may be able to predict the future —by looking at clues in past and present-day news articles.

 

Such software can predict when and where disease outbreaks might occur based on two decades of New York Times articles and other online data, according to Technology Review.

 

“I truly view this as a foreshadowing of what’s to come. Eventually this kind of work will start to have an influence on how things go for people,” said Eric Horvitz, distinguished scientist and codirector at Microsoft Research.

 

The research for the new software comes from Microsoft and the Technion-Israel Institute of Technology, the Technology Review said.

 

Horvitz did the research in collaboration with Kira Radinsky, a PhD researcher at the Technion-Israel Institute.

 

Once perfected, the system could help aid organizations and others be more proactive in tackling disease outbreaks or other problems.

 

Horvitz said the tests suggested a refined version could be used in real settings, such as in helping government aid agencies plan humanitarian response and readiness.

 

“We’ve done some reaching out and plan to do some follow-up work with such people,” he said.

 

Striking results

 

When tested on historical data, the system provides "striking results," Technology Review said.

 

It said reports of droughts in Angola in 2006 triggered a warning about possible cholera outbreaks in the country, "because previous events had taught the system that cholera outbreaks were more likely in years following droughts."

 

A second warning about cholera in Angola was triggered by news reports of large storms in Africa in early 2007.

 

Less than a week later, reports appeared that cholera had become established.

 

Similar tests involving forecasts of disease, violence, and a significant numbers of deaths saw the system’s warnings correct between 70 to 90 percent of the time.

 

While Horvitz and Radinsky are not the first to consider using online news and other data to forecast future events, they say they make use of more data sources that allows their system to be more general-purpose.

 

22 years of archives

 

The system uses 22 years of New York Times archives, from 1986 to 2007, as well as data from the Internet to learn about what leads up to major news events.

 

Radinsky said one useful source was DBpedia, a structured form of the information inside Wikipedia constructed using crowdsourcing.

 

Other sources included WordNet, which helps software understand the meaning of words, and OpenCyc, a database of common knowledge.

 

“We can understand, or see, the location of the places in the news articles, how much money people earn there, and even information about politics,” Radinsky said.

 

With all this information, researchers get valuable context not available in news articles, and which is necessary to figure out general rules for what events precede others.

 

But Horvitz said Microsoft does not have plans to commercialize the research for now, though the project will continue.

 

Market for predictive tools

 

Technology Review said there is already a small market for predictive tools.

 

It cited a startup called Recorded Future, which makes predictions about future events harvested from forward-looking statements online and other sources.

 

Recorded Future includes government intelligence agencies among its customers.

 

Its founder and CEO Christopher Ahlberg said the new research is “good work” that shows how predictions can be made using hard data.

 

But Ahlberg added turning the prototype system into a product would require further development.

 

Meanwhile, Horvitz said the software can continue to learn from human nature and many aspects of the environment, which he said have stayed the same.

 

“I’m personally interested in getting data further back in time,” he said. — TJD, GMA News