A prototype hybrid prediction market for estimating replicability of published work
Authors:
Tatiana Chakravorti,
Robert Fraleigh,
Timothy Fritton,
Michael McLaughlin,
Vaibhav Singh,
Christopher Griffin,
Anthony Kwasnica,
David Pennock,
C. Lee Giles,
Sarah Rajtmajer
Abstract:
We present a prototype hybrid prediction market and demonstrate the avenue it represents for meaningful human-AI collaboration. We build on prior work proposing artificial prediction markets as a novel machine-learning algorithm. In an artificial prediction market, trained AI agents buy and sell outcomes of future events. Classification decisions can be framed as outcomes of future events, and acc…
▽ More
We present a prototype hybrid prediction market and demonstrate the avenue it represents for meaningful human-AI collaboration. We build on prior work proposing artificial prediction markets as a novel machine-learning algorithm. In an artificial prediction market, trained AI agents buy and sell outcomes of future events. Classification decisions can be framed as outcomes of future events, and accordingly, the price of an asset corresponding to a given classification outcome can be taken as a proxy for the confidence of the system in that decision. By embedding human participants in these markets alongside bot traders, we can bring together insights from both. In this paper, we detail pilot studies with prototype hybrid markets for the prediction of replication study outcomes. We highlight challenges and opportunities, share insights from semi-structured interviews with hybrid market participants, and outline a vision for ongoing and future work.
△ Less
Submitted 1 March, 2023;
originally announced March 2023.
A Synthetic Prediction Market for Estimating Confidence in Published Work
Authors:
Sarah Rajtmajer,
Christopher Griffin,
Jian Wu,
Robert Fraleigh,
Laxmaan Balaji,
Anna Squicciarini,
Anthony Kwasnica,
David Pennock,
Michael McLaughlin,
Timothy Fritton,
Nishanth Nakshatri,
Arjun Menon,
Sai Ajay Modukuri,
Rajal Nivargi,
Xin Wei,
C. Lee Giles
Abstract:
Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and behavioral sciences literature. We demonstrate our system and detail our findings using a collection of known replication projects. We suggest that this work lays the…
▽ More
Explainably estimating confidence in published scholarly work offers opportunity for faster and more robust scientific progress. We develop a synthetic prediction market to assess the credibility of published claims in the social and behavioral sciences literature. We demonstrate our system and detail our findings using a collection of known replication projects. We suggest that this work lays the foundation for a research agenda that creatively uses AI for peer review.
△ Less
Submitted 23 December, 2021;
originally announced January 2022.
