Scientific Text Analysis with Robots applied to observatory proposals
Authors:
T. Jerabkova,
H. M. J. Boffin,
F. Patat,
D. Dorigo,
F. Sogni,
F. Primas
Abstract:
To test the potential disruptive effect of Artificial Intelligence (AI) transformers (e.g., ChatGPT) and their associated Large Language Models on the time allocation process, both in proposal reviewing and grading, an experiment has been set-up at ESO for the P112 Call for Proposals. The experiment aims at raising awareness in the ESO community and build valuable knowledge by identifying what fut…
▽ More
To test the potential disruptive effect of Artificial Intelligence (AI) transformers (e.g., ChatGPT) and their associated Large Language Models on the time allocation process, both in proposal reviewing and grading, an experiment has been set-up at ESO for the P112 Call for Proposals. The experiment aims at raising awareness in the ESO community and build valuable knowledge by identifying what future steps ESO and other observatories might need to take to stay up to date with current technologies. We present here the results of the experiment, which may further be used to inform decision-makers regarding the use of AI in the proposal review process. We find that the ChatGPT-adjusted proposals tend to receive lower grades compared to the original proposals. Moreover, ChatGPT 3.5 can generally not be trusted in providing correct scientific references, while the most recent version makes a better, but far from perfect, job. We also studied how ChatGPT deals with assessing proposals. It does an apparent remarkable job at providing a summary of ESO proposals, although it doesn't do so good to identify weaknesses. When looking at how it evaluates proposals, however, it appears that ChatGPT systematically gives a higher mark than humans, and tends to prefer proposals written by itself.
△ Less
Submitted 3 July, 2024;
originally announced July 2024.
The First Results of Distributed Peer Review at ESO Show Promising Outcomes
Authors:
Tereza Jerabkova,
Ferdinando Patat,
Francesca Primas,
Dario Dorigo,
Fabio Sogni,
Lucas Astolfi,
Thomas Bierwirth,
Michael Pruemm
Abstract:
The European Southern Observatory (ESO) implemented a new paradigm called Distributed Peer Review (DPR) as part of its proposal evaluation process in Period 110. Under DPR, Principal Investigators who submit proposals agree to review a certain number of proposals submitted by their peers and accept that their own proposal(s) are reviewed by their peers who have also submitted proposals in the same…
▽ More
The European Southern Observatory (ESO) implemented a new paradigm called Distributed Peer Review (DPR) as part of its proposal evaluation process in Period 110. Under DPR, Principal Investigators who submit proposals agree to review a certain number of proposals submitted by their peers and accept that their own proposal(s) are reviewed by their peers who have also submitted proposals in the same cycle. This article presents a brief overview of the DPR process at ESO, and its outcomes based on data from periods 110 and 111.
△ Less
Submitted 16 May, 2023;
originally announced May 2023.
