Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)
Authors:
Sabrina Toro,
Anna V Anagnostopoulos,
Sue Bello,
Kai Blumberg,
Rhiannon Cameron,
Leigh Carmody,
Alexander D Diehl,
Damion Dooley,
William Duncan,
Petra Fey,
Pascale Gaudet,
Nomi L Harris,
Marcin Joachimiak,
Leila Kiani,
Tiago Lubiana,
Monica C Munoz-Torres,
Shawn O'Neil,
David Osumi-Sutherland,
Aleix Puig,
Justin P Reese,
Leonore Reiser,
Sofia Robb,
Troy Ruem**,
James Seager,
Eric Sid
, et al. (5 additional authors not shown)
Abstract:
Background: Ontologies are fundamental components of informatics infrastructure in domains such as biomedical, environmental, and food sciences, representing consensus knowledge in an accurate and computable form. However, their construction and maintenance demand substantial resources and necessitate substantial collaboration between domain experts, curators, and ontology experts. We present Dyna…
▽ More
Background: Ontologies are fundamental components of informatics infrastructure in domains such as biomedical, environmental, and food sciences, representing consensus knowledge in an accurate and computable form. However, their construction and maintenance demand substantial resources and necessitate substantial collaboration between domain experts, curators, and ontology experts. We present Dynamic Retrieval Augmented Generation of Ontologies using AI (DRAGON-AI), an ontology generation method employing Large Language Models (LLMs) and Retrieval Augmented Generation (RAG). DRAGON-AI can generate textual and logical ontology components, drawing from existing knowledge in multiple ontologies and unstructured text sources.
Results: We assessed performance of DRAGON-AI on de novo term construction across ten diverse ontologies, making use of extensive manual evaluation of results. Our method has high precision for relationship generation, but has slightly lower precision than from logic-based reasoning. Our method is also able to generate definitions deemed acceptable by expert evaluators, but these scored worse than human-authored definitions. Notably, evaluators with the highest level of confidence in a domain were better able to discern flaws in AI-generated definitions. We also demonstrated the ability of DRAGON-AI to incorporate natural language instructions in the form of GitHub issues.
Conclusions: These findings suggest DRAGON-AI's potential to substantially aid the manual ontology construction process. However, our results also underscore the importance of having expert curators and ontology editors drive the ontology generation process.
△ Less
Submitted 12 June, 2024; v1 submitted 17 December, 2023;
originally announced December 2023.
Models, Markets, and the Forecasting of Elections
Authors:
Rajiv Sethi,
Julie Seager,
Emily Cai,
Daniel M. Benjamin,
Fred Morstatter
Abstract:
We examine probabilistic forecasts for battleground states in the 2020 US presidential election, using daily data from two sources over seven months: a model published by The Economist, and prices from the PredictIt exchange. We find systematic differences in accuracy over time, with markets performing better several months before the election, and the model performing better as the election appro…
▽ More
We examine probabilistic forecasts for battleground states in the 2020 US presidential election, using daily data from two sources over seven months: a model published by The Economist, and prices from the PredictIt exchange. We find systematic differences in accuracy over time, with markets performing better several months before the election, and the model performing better as the election approached. A simple average of the two forecasts performs better than either one of them overall, even though no average can outperform both component forecasts for any given state-date pair. This effect arises because the model and the market make different kinds of errors in different states: the model was confidently wrong in some cases, while the market was excessively uncertain in others. We conclude that there is value in using hybrid forecasting methods, and propose a market design that incorporates model forecasts via a trading bot to generate synthetic predictions. We also propose and conduct a profitability test that can be used as a novel criterion for the evaluation of forecasting performance.
△ Less
Submitted 25 May, 2021; v1 submitted 6 February, 2021;
originally announced February 2021.
