When asked to predict the future, a wise individual from a galaxy far, far away once said, “Difficult to see; always in motion is the future.” What Yoda understood is that the complex and dynamic nature of real-world events requires a sophisticated approach to forecasting. Recently Kairos Research was tapped by Raytheon-BBN Technologies to help develop just such an approach as part of an ongoing project funded by the Intelligence Advanced Research Projects Activity’s (IARPA’s) Hybrid Forecasting Competition (HFC) program. Kairos is working closely with Raytheon-BBN and other team members to create and test novel hybrid human-machine forecasting tools that optimally combine human and AI predictions.
When it comes to forecasting, humans (including professional intelligence analysts) have long stood as the gold standard. Human analysts can hop nimbly between topics, forecasting everything from global conflicts to flu outbreaks. But humans are susceptible to a wide range of cognitive biases: they tend to assign too much weight to salient but irrelevant details, and they often discount important historical trends. In contrast, “machine” forecasters (a term that includes statistical prediction algorithms) are tireless and immune to many human biases. But algorithms are no panacea. Often they are limited in the types of events they can forecast, and they can rely too much on past data while remaining blind to game-changing developments (such as recent events in the news).
According to IARPA’s website, “Hybrid approaches hold promise for combining the strengths of these two approaches while mitigating their individual weaknesses.” To realize this potential, HFC research teams “will integrate human and machine forecasting contributions in novel ways. These systems will compete in a multi-year competition to identify approaches that may enable the Intelligence Community (IC) to radically improve the accuracy and timeliness of geopolitical forecasts.”
Consistent with these goals, Kairos scientists are testing innovative software tools that enable human forecasters to create simple algorithms that automatically update their forecasts in response to changing information. In addition, the Kairos team is pioneering new methods for integrating multiple forecasters’ predictions into a single “crowd” forecast using deep neural networks, as well as using machine learning to match individual forecasters to specific prediction problems based on their unique expertise and interests.
“There are many possible ways to meld human and AI predictions,” said Dr. Colin Widmer, one of the lead Kairos contributors to the project. “The HFC program provides us with a great opportunity to try out different hybridization concepts in the context of a long-term forecasting tournament. Unlike a lot of laboratory research, which involves small-scale studies and ‘toy’ problems, the IARPA competition yields thousands of objectively scoreable forecasts involving real-world geopolitical events – so we can tell if our innovations are actually making a difference.”
Although the forecasting competition is still ongoing – and the future (as Yoda said) is difficult to see – ground-breaking research projects like HFC increase the likelihood that tomorrow’s prediction systems will outperform today’s forecasters, human or otherwise.
