Human insight remains the gold standard in many knowledge curation tasks. At the same time, rapid advances in AI – in particular large language models (LLMs) – have the potential to augment and complement human expertise, allowing for the rapid curation of large datasets at scale. To explore the promise of human-AI interaction in large-scale knowledge curation, Kairos Research recently kicked off a new DARPA-funded project called “Combining Human Insight with Machine Efficiency for Robust Annotation at Scale (CHIMERAS).” Like its mythological namesake, CHIMERAS envisions a hybrid methodology for data curation, melding the best aspects of human and machine cognition. According to project lead Dr. Cara Widmer, “the latest class of generative LLMs are adept at quickly extracting and summarizing key information from complex documents, but they typically lack the deep causal understanding possessed by human experts.” She further explained, “CHIMERAS seeks to harness human causal knowledge to guide and scaffold the curation process. But just as importantly, the results of this process can lead to new insights, causing experts to revise and improve their theoretical models.”
CHIMERAS resides within DARPA’s Collaborative Knowledge Curation (CKC) portfolio, which is itself part of the broader DARPA Advanced Research Concepts (ARC) initiative. According to DARPA’s website, the central question posed by CKC is, “How can we partially automate knowledge curation to help analysts and decision-makers gain and maintain awareness in complicated, interdependent systems?” The CHIMERAS project seeks to examine this question in the domain of economic sanctions. While sanctions are a critical tool in economic statecraft, their effectiveness can be difficult to measure and predict. Well-curated datasets can facilitate the development of better predictive models, both qualitative and quantitative.
