-
How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments
Authors:
Jen-tse Huang,
Eric John Li,
Man Ho Lam,
Tian Liang,
Wenxuan Wang,
Youliang Yuan,
Wenxiang Jiao,
Xing Wang,
Zhaopeng Tu,
Michael R. Lyu
Abstract:
Decision-making, a complicated task requiring various types of abilities, presents an excellent framework for assessing Large Language Models (LLMs). Our research investigates LLMs' decision-making capabilities through the lens of a well-established field, Game Theory. We focus specifically on games that support the participation of more than two agents simultaneously. Subsequently, we introduce o…
▽ More
Decision-making, a complicated task requiring various types of abilities, presents an excellent framework for assessing Large Language Models (LLMs). Our research investigates LLMs' decision-making capabilities through the lens of a well-established field, Game Theory. We focus specifically on games that support the participation of more than two agents simultaneously. Subsequently, we introduce our framework, GAMA-Bench, including eight classical multi-agent games. We design a scoring scheme to assess a model's performance in these games quantitatively. Through GAMA-Bench, we investigate LLMs' robustness, generalizability, and enhancement strategies. Results reveal that while GPT-3.5 shows satisfying robustness, its generalizability is relatively limited. However, its performance can be improved through approaches such as Chain-of-Thought. Additionally, we conduct evaluations across various LLMs and find that GPT-4 outperforms other models on GAMA-Bench, achieving a score of 60.5. Moreover, Gemini-1.0-Pro and GPT-3.5 (0613, 1106, 0125) demonstrate similar intelligence on GAMA-Bench. The code and experimental results are made publicly available via https://github.com/CUHK-ARISE/GAMABench.
△ Less
Submitted 25 April, 2024; v1 submitted 18 March, 2024;
originally announced March 2024.
-
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
Authors:
Jen-tse Huang,
Wenxuan Wang,
Eric John Li,
Man Ho Lam,
Shujie Ren,
Youliang Yuan,
Wenxiang Jiao,
Zhaopeng Tu,
Michael R. Lyu
Abstract:
Large Language Models (LLMs) have recently showcased their remarkable capacities, not only in natural language processing tasks but also across diverse domains such as clinical medicine, legal consultation, and education. LLMs become more than mere applications, evolving into assistants capable of addressing diverse user requests. This narrows the distinction between human beings and artificial in…
▽ More
Large Language Models (LLMs) have recently showcased their remarkable capacities, not only in natural language processing tasks but also across diverse domains such as clinical medicine, legal consultation, and education. LLMs become more than mere applications, evolving into assistants capable of addressing diverse user requests. This narrows the distinction between human beings and artificial intelligence agents, raising intriguing questions regarding the potential manifestation of personalities, temperaments, and emotions within LLMs. In this paper, we propose a framework, PsychoBench, for evaluating diverse psychological aspects of LLMs. Comprising thirteen scales commonly used in clinical psychology, PsychoBench further classifies these scales into four distinct categories: personality traits, interpersonal relationships, motivational tests, and emotional abilities. Our study examines five popular models, namely text-davinci-003, gpt-3.5-turbo, gpt-4, LLaMA-2-7b, and LLaMA-2-13b. Additionally, we employ a jailbreak approach to bypass the safety alignment protocols and test the intrinsic natures of LLMs. We have made PsychoBench openly accessible via https://github.com/CUHK-ARISE/PsychoBench.
△ Less
Submitted 22 January, 2024; v1 submitted 2 October, 2023;
originally announced October 2023.
-
Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench
Authors:
Jen-tse Huang,
Man Ho Lam,
Eric John Li,
Shujie Ren,
Wenxuan Wang,
Wenxiang Jiao,
Zhaopeng Tu,
Michael R. Lyu
Abstract:
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situa…
▽ More
Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes five LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4 and LLaMA-2. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, dubbed EmotionBench, is made openly accessible via https://github.com/CUHK-ARISE/EmotionBench. We aspire to contribute to the advancement of LLMs regarding better alignment with the emotional behaviors of human beings, thereby enhancing their utility and applicability as intelligent assistants.
△ Less
Submitted 24 April, 2024; v1 submitted 7 August, 2023;
originally announced August 2023.
-
Revisiting the Reliability of Psychological Scales on Large Language Models
Authors:
Jen-tse Huang,
Wenxuan Wang,
Man Ho Lam,
Eric John Li,
Wenxiang Jiao,
Michael R. Lyu
Abstract:
Recent research has extended beyond assessing the performance of Large Language Models (LLMs) to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, i…
▽ More
Recent research has extended beyond assessing the performance of Large Language Models (LLMs) to examining their characteristics from a psychological standpoint, acknowledging the necessity of understanding their behavioral characteristics. The administration of personality tests to LLMs has emerged as a noteworthy area in this context. However, the suitability of employing psychological scales, initially devised for humans, on LLMs is a matter of ongoing debate. Our study aims to determine the reliability of applying personality assessments to LLMs, explicitly investigating whether LLMs demonstrate consistent personality traits. Analyzing responses under 2,500 settings reveals that gpt-3.5-turbo shows consistency in responses to the Big Five Inventory, indicating a high degree of reliability. Furthermore, our research explores the potential of gpt-3.5-turbo to emulate diverse personalities and represent various groups, which is a capability increasingly sought after in social sciences for substituting human participants with LLMs to reduce costs. Our findings reveal that LLMs have the potential to represent different personalities with specific prompt instructions. By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field. We have made our experimental results and the corresponding code openly accessible via https://github.com/CUHK-ARISE/LLMPersonality.
△ Less
Submitted 28 December, 2023; v1 submitted 31 May, 2023;
originally announced May 2023.
-
XUV Frequency Combs via Femtosecond Enhancement Cavities
Authors:
Arthur K Mills,
TJ Hammond,
Matthew HC Lam,
David J Jones
Abstract:
We review the current state of tabletop extreme ultraviolet (XUV) sources based on high harmonic generation (HHG) in femtosecond enhancement cavities (fsEC). Recent developments have enabled generation of high photon flux (1014 photons/sec) in the XUV, at high repetition rates (>50 MHz) and spanning the spectral region from 40 nm - 120 nm. This level of performance has enabled precision spectrosco…
▽ More
We review the current state of tabletop extreme ultraviolet (XUV) sources based on high harmonic generation (HHG) in femtosecond enhancement cavities (fsEC). Recent developments have enabled generation of high photon flux (1014 photons/sec) in the XUV, at high repetition rates (>50 MHz) and spanning the spectral region from 40 nm - 120 nm. This level of performance has enabled precision spectroscopy with XUV frequency combs and promises further applications in XUV spectroscopic and photoemission studies. We discuss the theory of operation and experimental details of the fsEC and XUV generation based on HHG, including current technical challenges to increasing the photon flux and maximum photon energy produced by this type of system. Current and future applications for these sources are also discussed.
△ Less
Submitted 2 June, 2012;
originally announced June 2012.
