Assessing Large Language Models on Climate Information
Authors:
Jannis Bulian,
Mike S. Schäfer,
Afra Amini,
Heidi Lam,
Massimiliano Ciaramita,
Ben Gaiarin,
Michelle Chen Hübscher,
Christian Buck,
Niels G. Mede,
Markus Leippold,
Nadine Strauß
Abstract:
As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM genera…
▽ More
As Large Language Models (LLMs) rise in popularity, it is necessary to assess their capability in critically relevant domains. We present a comprehensive evaluation framework, grounded in science communication research, to assess LLM responses to questions about climate change. Our framework emphasizes both presentational and epistemological adequacy, offering a fine-grained analysis of LLM generations spanning 8 dimensions and 30 issues. Our evaluation task is a real-world example of a growing number of challenging problems where AI can complement and lift human performance. We introduce a novel protocol for scalable oversight that relies on AI Assistance and raters with relevant education. We evaluate several recent LLMs on a set of diverse climate questions. Our results point to a significant gap between surface and epistemological qualities of LLMs in the realm of climate communication.
△ Less
Submitted 28 May, 2024; v1 submitted 4 October, 2023;
originally announced October 2023.
