-
Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation
Authors:
Eilam Shapira,
Reut Apel,
Moshe Tennenholtz,
Roi Reichart
Abstract:
Recent advances in Large Language Models (LLMs) have spurred interest in designing LLM-based agents for tasks that involve interaction with human and artificial agents. This paper addresses a key aspect in the design of such agents: Predicting human decision in off-policy evaluation (OPE), focusing on language-based persuasion games, where the agent's goal is to influence its partner's decisions t…
▽ More
Recent advances in Large Language Models (LLMs) have spurred interest in designing LLM-based agents for tasks that involve interaction with human and artificial agents. This paper addresses a key aspect in the design of such agents: Predicting human decision in off-policy evaluation (OPE), focusing on language-based persuasion games, where the agent's goal is to influence its partner's decisions through verbal messages. Using a dedicated application, we collected a dataset of 87K decisions from humans playing a repeated decision-making game with artificial agents. Our approach involves training a model on human interactions with one agents subset to predict decisions when interacting with another. To enhance off-policy performance, we propose a simulation technique involving interactions across the entire agent space and simulated decision makers. Our learning strategy yields significant OPE gains, e.g., improving prediction accuracy in the top 15% challenging cases by 7.1%. Our code and the large dataset we collected and generated are submitted as supplementary material and publicly available in our GitHub repository: https://github.com/eilamshapira/HumanChoicePrediction
△ Less
Submitted 28 February, 2024; v1 submitted 17 May, 2023;
originally announced May 2023.
-
MeeQA: Natural Questions in Meeting Transcripts
Authors:
Reut Apel,
Tom Braude,
Amir Kantor,
Eyal Kolman
Abstract:
We present MeeQA, a dataset for natural-language question answering over meeting transcripts. It includes real questions asked during meetings by its participants. The dataset contains 48K question-answer pairs, extracted from 422 meeting transcripts, spanning multiple domains. Questions in transcripts pose a special challenge as they are not always clear, and considerable context may be required…
▽ More
We present MeeQA, a dataset for natural-language question answering over meeting transcripts. It includes real questions asked during meetings by its participants. The dataset contains 48K question-answer pairs, extracted from 422 meeting transcripts, spanning multiple domains. Questions in transcripts pose a special challenge as they are not always clear, and considerable context may be required in order to provide an answer. Further, many questions asked during meetings are left unanswered. To improve baseline model performance on this type of questions, we also propose a novel loss function, \emph{Flat Hierarchical Loss}, designed to enhance performance over questions with no answer in the text. Our experiments demonstrate the advantage of using our approach over standard QA models.
△ Less
Submitted 15 May, 2023;
originally announced May 2023.
-
Designing an Automatic Agent for Repeated Language based Persuasion Games
Authors:
Maya Raifer,
Guy Rotman,
Reut Apel,
Moshe Tennenholtz,
Roi Reichart
Abstract:
Persuasion games are fundamental in economics and AI research and serve as the basis for important applications. However, work on this setup assumes communication with stylized messages that do not consist of rich human language. In this paper we consider a repeated sender (expert) -- receiver (decision maker) game, where the sender is fully informed about the state of the world and aims to persua…
▽ More
Persuasion games are fundamental in economics and AI research and serve as the basis for important applications. However, work on this setup assumes communication with stylized messages that do not consist of rich human language. In this paper we consider a repeated sender (expert) -- receiver (decision maker) game, where the sender is fully informed about the state of the world and aims to persuade the receiver to accept a deal by sending one of several possible natural language reviews. We design an automatic expert that plays this repeated game, aiming to achieve the maximal payoff. Our expert is implemented within the Monte Carlo Tree Search (MCTS) algorithm, with deep learning models that exploit behavioral and linguistic signals in order to predict the next action of the decision maker, and the future payoff of the expert given the state of the game and a candidate review. We demonstrate the superiority of our expert over strong baselines, its adaptability to different decision makers, and that its selected reviews are nicely adapted to the proposed deal.
△ Less
Submitted 31 December, 2021; v1 submitted 11 May, 2021;
originally announced May 2021.
-
Predicting Decisions in Language Based Persuasion Games
Authors:
Reut Apel,
Ido Erev,
Roi Reichart,
Moshe Tennenholtz
Abstract:
Sender-receiver interactions, and specifically persuasion games, are widely researched in economic modeling and artificial intelligence. However, in the classic persuasion games setting, the messages sent from the expert to the decision-maker (DM) are abstract or well-structured signals rather than natural language messages. This paper addresses the use of natural language in persuasion games. For…
▽ More
Sender-receiver interactions, and specifically persuasion games, are widely researched in economic modeling and artificial intelligence. However, in the classic persuasion games setting, the messages sent from the expert to the decision-maker (DM) are abstract or well-structured signals rather than natural language messages. This paper addresses the use of natural language in persuasion games. For this purpose, we conduct an online repeated interaction experiment. At each trial of the interaction, an informed expert aims to sell an uninformed decision-maker a vacation in a hotel, by sending her a review that describes the hotel. While the expert is exposed to several scored reviews, the decision-maker observes only the single review sent by the expert, and her payoff in case she chooses to take the hotel is a random draw from the review score distribution available to the expert only. We also compare the behavioral patterns in this experiment to the equivalent patterns in similar experiments where the communication is based on the numerical values of the reviews rather than the reviews' text, and observe substantial differences which can be explained through an equilibrium analysis of the game. We consider a number of modeling approaches for our verbal communication setup, differing from each other in the model type (deep neural network vs. linear classifier), the type of features used by the model (textual, behavioral or both) and the source of the textual features (DNN-based vs. hand-crafted). Our results demonstrate that given a prefix of the interaction sequence, our models can predict the future decisions of the decision-maker, particularly when a sequential modeling approach and hand-crafted textual features are applied. Further analysis of the hand-crafted textual features allows us to make initial observations about the aspects of text that drive decision making in our setup
△ Less
Submitted 31 March, 2022; v1 submitted 17 December, 2020;
originally announced December 2020.
-
Predicting human decisions with behavioral theories and machine learning
Authors:
Ori Plonsky,
Reut Apel,
Eyal Ert,
Moshe Tennenholtz,
David Bourgin,
Joshua C. Peterson,
Daniel Reichman,
Thomas L. Griffiths,
Stuart J. Russell,
Evan C. Carter,
James F. Cavanagh,
Ido Erev
Abstract:
Predicting human decision-making under risk and uncertainty represents a quintessential challenge that spans economics, psychology, and related disciplines. Despite decades of research effort, no model can be said to accurately describe and predict human choice even for the most stylized tasks like choice between lotteries. Here, we introduce BEAST Gradient Boosting (BEAST-GB), a novel hybrid mode…
▽ More
Predicting human decision-making under risk and uncertainty represents a quintessential challenge that spans economics, psychology, and related disciplines. Despite decades of research effort, no model can be said to accurately describe and predict human choice even for the most stylized tasks like choice between lotteries. Here, we introduce BEAST Gradient Boosting (BEAST-GB), a novel hybrid model that synergizes behavioral theories, specifically the model BEAST, with machine learning techniques. First, we show the effectiveness of BEAST-GB by describing CPC18, an open competition for prediction of human decision making under risk and uncertainty, in which BEAST-GB won. Second, we show that it achieves state-of-the-art performance on the largest publicly available dataset of human risky choice, outperforming purely data-driven neural networks, indicating the continued relevance of BEAST theoretical insights in the presence of large data. Third, we demonstrate BEAST-GB's superior predictive power in an ensemble of choice experiments in which the BEAST model alone falters, underscoring the indispensable role of machine learning in interpreting complex idiosyncratic behavioral data. Finally, we show BEAST-GB also displays robust domain generalization capabilities as it effectively predicts choice behavior in new experimental contexts that it was not trained on. These results confirm the potency of combining domain-specific theoretical frameworks with machine learning, underscoring a methodological advance with broad implications for modeling decisions in diverse environments.
△ Less
Submitted 18 April, 2024; v1 submitted 15 April, 2019;
originally announced April 2019.
-
Characterizing Efficient Referrals in Social Networks
Authors:
Reut Apel,
Elad Yom-Tov,
Moshe Tennenholtz
Abstract:
Users of social networks often focus on specific areas of that network, leading to the well-known "filter bubble" effect. Connecting people to a new area of the network in a way that will cause them to become active in that area could help alleviate this effect and improve social welfare.
Here we present preliminary analysis of network referrals, that is, attempts by users to connect peers to ot…
▽ More
Users of social networks often focus on specific areas of that network, leading to the well-known "filter bubble" effect. Connecting people to a new area of the network in a way that will cause them to become active in that area could help alleviate this effect and improve social welfare.
Here we present preliminary analysis of network referrals, that is, attempts by users to connect peers to other areas of the network. We classify these referrals by their efficiency, i.e., the likelihood that a referral will result in a user becoming active in the new area of the network. We show that by using features describing past experience of the referring author and the content of their messages we are able to predict whether referral will be effective, reaching an AUC of 0.87 for those users most experienced in writing efficient referrals. Our results represent a first step towards algorithmically constructing efficient referrals with the goal of mitigating the "filter bubble" effect pervasive in on line social networks.
△ Less
Submitted 1 May, 2018;
originally announced May 2018.
