Towards Measuring the Representation of Subjective Global Opinions in Language Models
Authors:
Esin Durmus,
Karina Nguyen,
Thomas I. Liao,
Nicholas Schiefer,
Amanda Askell,
Anton Bakhtin,
Carol Chen,
Zac Hatfield-Dodds,
Danny Hernandez,
Nicholas Joseph,
Liane Lovitt,
Sam McCandlish,
Orowa Sikder,
Alex Tamkin,
Janel Thamkul,
Jared Kaplan,
Jack Clark,
Deep Ganguli
Abstract:
Large language models (LLMs) may not equitably represent diverse global perspectives on societal issues. In this paper, we develop a quantitative framework to evaluate whose opinions model-generated responses are more similar to. We first build a dataset, GlobalOpinionQA, comprised of questions and answers from cross-national surveys designed to capture diverse opinions on global issues across dif…
▽ More
Large language models (LLMs) may not equitably represent diverse global perspectives on societal issues. In this paper, we develop a quantitative framework to evaluate whose opinions model-generated responses are more similar to. We first build a dataset, GlobalOpinionQA, comprised of questions and answers from cross-national surveys designed to capture diverse opinions on global issues across different countries. Next, we define a metric that quantifies the similarity between LLM-generated survey responses and human responses, conditioned on country. With our framework, we run three experiments on an LLM trained to be helpful, honest, and harmless with Constitutional AI. By default, LLM responses tend to be more similar to the opinions of certain populations, such as those from the USA, and some European and South American countries, highlighting the potential for biases. When we prompt the model to consider a particular country's perspective, responses shift to be more similar to the opinions of the prompted populations, but can reflect harmful cultural stereotypes. When we translate GlobalOpinionQA questions to a target language, the model's responses do not necessarily become the most similar to the opinions of speakers of those languages. We release our dataset for others to use and build on. Our data is at https://huggingface.co/datasets/Anthropic/llm_global_opinions. We also provide an interactive visualization at https://llmglobalvalues.anthropic.com.
△ Less
Submitted 11 April, 2024; v1 submitted 28 June, 2023;
originally announced June 2023.
Modelling heterogeneous outcomes in multi-agent systems
Authors:
Orowa Sikder
Abstract:
A broad set of empirical phenomenon in the study of social, economic and machine behaviour can be modelled as complex systems with averaging dynamics. However many of these models naturally result in consensus or consensus-like outcomes. In reality, empirical phenomenon rarely converge to these and instead are characterized by rich, persistent variation in the agent states. Such heterogeneous outc…
▽ More
A broad set of empirical phenomenon in the study of social, economic and machine behaviour can be modelled as complex systems with averaging dynamics. However many of these models naturally result in consensus or consensus-like outcomes. In reality, empirical phenomenon rarely converge to these and instead are characterized by rich, persistent variation in the agent states. Such heterogeneous outcomes are a natural consequence of a number of models that incorporate external perturbation to the otherwise convex dynamics of the agents. The purpose of this paper is to formalize the notion of heterogeneity and demonstrate which classes of models are able to achieve it as an outcome, and therefore are better suited to modelling important empirical questions. We do so by determining how the topology of (time-varying) interaction networks restrict the space of possible steady-state outcomes for agents, and how this is related to the study of random walks on graphs. We consider a number of intentionally diverse examples to demonstrate how the results can be applied.
△ Less
Submitted 2 July, 2020;
originally announced July 2020.
When facts fail: Bias, polarisation and truth in social networks
Authors:
Orowa Sikder,
Robert E. Smith,
Pierpaolo Vivo,
Giacomo Livan
Abstract:
Online social networks provide users with unprecedented opportunities to engage with diverse opinions. At the same time, they enable confirmation bias on large scales by empowering individuals to self-select narratives they want to be exposed to. A precise understanding of such tradeoffs is still largely missing. We introduce a social learning model where most participants in a network update thei…
▽ More
Online social networks provide users with unprecedented opportunities to engage with diverse opinions. At the same time, they enable confirmation bias on large scales by empowering individuals to self-select narratives they want to be exposed to. A precise understanding of such tradeoffs is still largely missing. We introduce a social learning model where most participants in a network update their beliefs unbiasedly based on new information, while a minority of participants reject information that is incongruent with their preexisting beliefs. This simple mechanism generates permanent opinion polarization and cascade dynamics, and accounts for the aforementioned tradeoff between confirmation bias and social connectivity through analytic results. We investigate the model's predictions empirically using US county-level data on the impact of Internet access on the formation of beliefs about global warming. We conclude by discussing policy implications of our model, highlighting the downsides of debunking and suggesting alternative strategies to contrast misinformation.
△ Less
Submitted 21 January, 2020; v1 submitted 26 August, 2018;
originally announced August 2018.
