-
Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT
Authors:
Amirhossein Abaskohi,
Sara Baruni,
Mostafa Masoudi,
Nesa Abbasi,
Mohammad Hadi Babalou,
Ali Edalat,
Sepehr Kamahi,
Samin Mahdizadeh Sani,
Nikoo Naghavian,
Danial Namazifard,
Pouya Sadeghi,
Yadollah Yaghoobzadeh
Abstract:
This paper explores the efficacy of large language models (LLMs) for Persian. While ChatGPT and consequent LLMs have shown remarkable performance in English, their efficiency for more low-resource languages remains an open question. We present the first comprehensive benchmarking study of LLMs across diverse Persian language tasks. Our primary focus is on GPT-3.5-turbo, but we also include GPT-4 a…
▽ More
This paper explores the efficacy of large language models (LLMs) for Persian. While ChatGPT and consequent LLMs have shown remarkable performance in English, their efficiency for more low-resource languages remains an open question. We present the first comprehensive benchmarking study of LLMs across diverse Persian language tasks. Our primary focus is on GPT-3.5-turbo, but we also include GPT-4 and OpenChat-3.5 to provide a more holistic evaluation. Our assessment encompasses a diverse set of tasks categorized into classic, reasoning, and knowledge-based domains. To enable a thorough comparison, we evaluate LLMs against existing task-specific fine-tuned models. Given the limited availability of Persian datasets for reasoning tasks, we introduce two new benchmarks: one based on elementary school math questions and another derived from the entrance exams for 7th and 10th grades. Our findings reveal that while LLMs, especially GPT-4, excel in tasks requiring reasoning abilities and a broad understanding of general knowledge, they often lag behind smaller pre-trained models fine-tuned specifically for particular tasks. Additionally, we observe improved performance when test sets are translated to English before inputting them into GPT-3.5. These results highlight the significant potential for enhancing LLM performance in the Persian language. This is particularly noteworthy due to the unique attributes of Persian, including its distinct alphabet and writing styles.
△ Less
Submitted 2 April, 2024;
originally announced April 2024.
-
A Cartesian Closed Category for Random Variables
Authors:
Pietro Di Gianantonio,
Abbas Edalat
Abstract:
We present a novel, yet rather simple construction within the traditional framework of Scott domains to provide semantics to probabilistic programming, thus obtaining a solution to a long-standing open problem in this area. Unlike current main approaches that employ some probability measures or continuous valuations on non-standard or rather complex structures, we use the Scott domain of random va…
▽ More
We present a novel, yet rather simple construction within the traditional framework of Scott domains to provide semantics to probabilistic programming, thus obtaining a solution to a long-standing open problem in this area. Unlike current main approaches that employ some probability measures or continuous valuations on non-standard or rather complex structures, we use the Scott domain of random variables from a standard sample space -- the unit interval or the Cantor space -- to any given Scott domain. The map taking any such random variable to its corresponding probability distribution provides an effectively given, Scott continuous surjection onto the probabilistic power domain of the underlying Scott domain, establishing a new basic result in classical domain theory. We obtain a Cartesian closed category by enriching the category of Scott domains to capture the equivalence of random variables on these domains. The construction of the domain of random variables on this enriched category forms a strong commutative monad, which is suitable for defining the semantics of probabilistic programming.
△ Less
Submitted 11 June, 2024; v1 submitted 18 February, 2024;
originally announced February 2024.
-
Systematic Literature Review: Computational Approaches for Humour Style Classification
Authors:
Mary Ogbuka Kenneth,
Foaad Khosmood,
Abbas Edalat
Abstract:
Understanding various humour styles is essential for comprehending the multifaceted nature of humour and its impact on fields such as psychology and artificial intelligence. This understanding has revealed that humour, depending on the style employed, can either have therapeutic or detrimental effects on an individual's health and relationships. Although studies dedicated exclusively to computatio…
▽ More
Understanding various humour styles is essential for comprehending the multifaceted nature of humour and its impact on fields such as psychology and artificial intelligence. This understanding has revealed that humour, depending on the style employed, can either have therapeutic or detrimental effects on an individual's health and relationships. Although studies dedicated exclusively to computational-based humour style analysis remain somewhat rare, an expansive body of research thrives within related task, particularly binary humour and sarcasm recognition. In this systematic literature review (SLR), we survey the landscape of computational techniques applied to these related tasks and also uncover their fundamental relevance to humour style analysis. Through this study, we unveil common approaches, illuminate various datasets and evaluation metrics, and effectively navigate the complex terrain of humour research. Our efforts determine potential research gaps and outlined promising directions. Furthermore, the SLR identifies a range of features and computational models that can seamlessly transition from related tasks like binary humour and sarcasm detection to invigorate humour style classification. These features encompass incongruity, sentiment and polarity analysis, ambiguity detection, acoustic nuances, visual cues, contextual insights, and more. The computational models that emerge contain traditional machine learning paradigms, neural network architectures, transformer-based models, and specialised models attuned to the nuances of humour. Finally, the SLR provides access to existing datasets related to humour and sarcasm, facilitating the work of future researchers.
△ Less
Submitted 30 January, 2024;
originally announced February 2024.
-
A Multilingual Virtual Guide for Self-Attachment Technique
Authors:
Alicia Jiayun Law,
Ruoyu Hu,
Lisa Alazraki,
Anandha Gopalan,
Neophytos Polydorou,
Abbas Edalat
Abstract:
In this work, we propose a computational framework that leverages existing out-of-language data to create a conversational agent for the delivery of Self-Attachment Technique (SAT) in Mandarin. Our framework does not require large-scale human translations, yet it achieves a comparable performance whilst also maintaining safety and reliability. We propose two different methods of augmenting availab…
▽ More
In this work, we propose a computational framework that leverages existing out-of-language data to create a conversational agent for the delivery of Self-Attachment Technique (SAT) in Mandarin. Our framework does not require large-scale human translations, yet it achieves a comparable performance whilst also maintaining safety and reliability. We propose two different methods of augmenting available response data through empathetic rewriting. We evaluate our chatbot against a previous, English-only SAT chatbot through non-clinical human trials (N=42), each lasting five days, and quantitatively show that we are able to attain a comparable level of performance to the English SAT chatbot. We provide qualitative analysis on the limitations of our study and suggestions with the aim of guiding future improvements.
△ Less
Submitted 25 October, 2023;
originally announced October 2023.
-
Pure Bayesian Nash equilibrium for Bayesian games with multidimensional vector Types and linear payoffs
Authors:
Sébastien Huot,
Abbas Edalat
Abstract:
We study $n$-agent Bayesian Games with $m$-dimensional vector types and linear payoffs, also called Linear Multidimensional Bayesian Games. This class of games is equivalent with $n$-agent, $m$-game Uniform Multigames. We distinguish between games that have a discrete type space and those with a continuous type space. More specifically, we are interested in the existence of pure Bayesian Nash Equi…
▽ More
We study $n$-agent Bayesian Games with $m$-dimensional vector types and linear payoffs, also called Linear Multidimensional Bayesian Games. This class of games is equivalent with $n$-agent, $m$-game Uniform Multigames. We distinguish between games that have a discrete type space and those with a continuous type space. More specifically, we are interested in the existence of pure Bayesian Nash Equilibrium for such games and efficient algorithms to find them. For continuous priors we suggest a methodology to perform Nash Equilibrium search in simple cases. For discrete priors we present algorithms that can handle two actions and two players games efficiently. We introduce the core concept of threshold strategy and, under some mild conditions, we show that these games have at least one pure Bayesian Nash Equilibrium. We illustrate our results with several examples like Double Game Prisoner Dilemna (DGPD), Chicken Game and Sustainable Adoption Decision Problem (SADP).
△ Less
Submitted 21 October, 2023;
originally announced October 2023.
-
From Words and Exercises to Wellness: Farsi Chatbot for Self-Attachment Technique
Authors:
Sina Elahimanesh,
Shayan Salehi,
Sara Zahedi Movahed,
Lisa Alazraki,
Ruoyu Hu,
Abbas Edalat
Abstract:
In the wake of the post-pandemic era, marked by social isolation and surging rates of depression and anxiety, conversational agents based on digital psychotherapy can play an influential role compared to traditional therapy sessions. In this work, we develop a voice-capable chatbot in Farsi to guide users through Self-Attachment (SAT), a novel, self-administered, holistic psychological technique b…
▽ More
In the wake of the post-pandemic era, marked by social isolation and surging rates of depression and anxiety, conversational agents based on digital psychotherapy can play an influential role compared to traditional therapy sessions. In this work, we develop a voice-capable chatbot in Farsi to guide users through Self-Attachment (SAT), a novel, self-administered, holistic psychological technique based on attachment theory. Our chatbot uses a dynamic array of rule-based and classification-based modules to comprehend user input throughout the conversation and navigates a dialogue flowchart accordingly, recommending appropriate SAT exercises that depend on the user's emotional and mental state. In particular, we collect a dataset of over 6,000 utterances and develop a novel sentiment-analysis module that classifies user sentiment into 12 classes, with accuracy above 92%. To keep the conversation novel and engaging, the chatbot's responses are retrieved from a large dataset of utterances created with the aid of Farsi GPT-2 and a reinforcement learning approach, thus requiring minimal human annotation. Our chatbot also offers a question-answering module, called SAT Teacher, to answer users' questions about the principles of Self-Attachment. Finally, we design a cross-platform application as the bot's user interface. We evaluate our platform in a ten-day human study with N=52 volunteers from the non-clinical population, who have had over 2,000 dialogues in total with the chatbot. The results indicate that the platform was engaging to most users (75%), 72% felt better after the interactions, and 74% were satisfied with the SAT Teacher's performance.
△ Less
Submitted 25 March, 2024; v1 submitted 13 October, 2023;
originally announced October 2023.
-
Recursive Solution of Initial Value Problems with Temporal Discretization
Authors:
Abbas Edalat,
Amin Farjudian,
Yiran Li
Abstract:
We construct a continuous domain for temporal discretization of differential equations. By using this domain, and the domain of Lipschitz maps, we formulate a generalization of the Euler operator, which exhibits second-order convergence. We prove computability of the operator within the framework of effectively given domains. The operator only requires the vector field of the differential equation…
▽ More
We construct a continuous domain for temporal discretization of differential equations. By using this domain, and the domain of Lipschitz maps, we formulate a generalization of the Euler operator, which exhibits second-order convergence. We prove computability of the operator within the framework of effectively given domains. The operator only requires the vector field of the differential equation to be Lipschitz continuous, in contrast to the related operators in the literature which require the vector field to be at least continuously differentiable. Within the same framework, we also analyze temporal discretization and computability of another variant of the Euler operator formulated according to Runge-Kutta theory. We prove that, compared with this variant, the second-order operator that we formulate directly, not only imposes weaker assumptions on the vector field, but also exhibits superior convergence rate. We implement the first-order, second-order, and Runge-Kutta Euler operators using arbitrary-precision interval arithmetic, and report on some experiments. The experiments confirm our theoretical results. In particular, we observe the superior convergence rate of our second-order operator compared with the Runge-Kutta Euler and the common (first-order) Euler operators.
△ Less
Submitted 16 September, 2023; v1 submitted 10 January, 2023;
originally announced January 2023.
-
A Language for Evaluating Derivatives of Functionals Using Automatic Differentiation
Authors:
Pietro Di Gianantonio,
Abbas Edalat,
Ran Gutin
Abstract:
We present a simple functional programming language, called Dual PCF, that implements forward mode automatic differentiation using dual numbers in the framework of exact real number computation. The main new feature of this language is the ability to evaluate correctly up to the precision specified by the user -- in a simple and direct way -- the directional derivative of functionals as well as fi…
▽ More
We present a simple functional programming language, called Dual PCF, that implements forward mode automatic differentiation using dual numbers in the framework of exact real number computation. The main new feature of this language is the ability to evaluate correctly up to the precision specified by the user -- in a simple and direct way -- the directional derivative of functionals as well as first order functions. In contrast to other comparable languages, Dual PCF also includes the recursive operator for defining functions and functionals. We provide a wide range of examples of Lipschitz functions and functionals that can be defined in Dual PCF. We use domain theory both to give a denotational semantics to the language and to prove the correctness of the new derivative operator using logical relations. To be able to differentiate functionals -- including on function spaces equipped with their compact-open topology that do not admit a norm -- we develop a domain-theoretic directional derivative that is Scott continuous and extends Clarke's subgradient of real-valued locally Lipschitz maps on Banach spaces to real-valued continuous maps on Hausdorff topological vector spaces. Finally, we show that we can express arbitrary computable linear functionals in Dual PCF.
△ Less
Submitted 18 November, 2023; v1 submitted 12 October, 2022;
originally announced October 2022.
-
An Empathetic AI Coach for Self-Attachment Therapy
Authors:
Lisa Alazraki,
Ali Ghachem,
Neophytos Polydorou,
Foaad Khosmood,
Abbas Edalat
Abstract:
In this work, we present a new dataset and a computational strategy for a digital coach that aims to guide users in practicing the protocols of self-attachment therapy. Our framework augments a rule-based conversational agent with a deep-learning classifier for identifying the underlying emotion in a user's text response, as well as a deep-learning assisted retrieval method for producing novel, fl…
▽ More
In this work, we present a new dataset and a computational strategy for a digital coach that aims to guide users in practicing the protocols of self-attachment therapy. Our framework augments a rule-based conversational agent with a deep-learning classifier for identifying the underlying emotion in a user's text response, as well as a deep-learning assisted retrieval method for producing novel, fluent and empathetic utterances. We also craft a set of human-like personas that users can choose to interact with. Our goal is to achieve a high level of engagement during virtual therapy sessions. We evaluate the effectiveness of our framework in a non-clinical trial with N=16 participants, all of whom have had at least four interactions with the agent over the course of five days. We find that our platform is consistently rated higher for empathy, user engagement and usefulness than the simple rule-based framework. Finally, we provide guidelines to further improve the design and performance of the application, in accordance with the feedback received.
△ Less
Submitted 31 January, 2024; v1 submitted 17 September, 2022;
originally announced September 2022.
-
Sentiment Analysis of Persian-English Code-mixed Texts
Authors:
Nazanin Sabri,
Ali Edalat,
Behnam Bahrak
Abstract:
The rapid production of data on the internet and the need to understand how users are feeling from a business and research perspective has prompted the creation of numerous automatic monolingual sentiment detection systems. More recently however, due to the unstructured nature of data on social media, we are observing more instances of multilingual and code-mixed texts. This development in content…
▽ More
The rapid production of data on the internet and the need to understand how users are feeling from a business and research perspective has prompted the creation of numerous automatic monolingual sentiment detection systems. More recently however, due to the unstructured nature of data on social media, we are observing more instances of multilingual and code-mixed texts. This development in content type has created a new demand for code-mixed sentiment analysis systems. In this study we collect, label and thus create a dataset of Persian-English code-mixed tweets. We then proceed to introduce a model which uses BERT pretrained embeddings as well as translation models to automatically learn the polarity scores of these Tweets. Our model outperforms the baseline models that use Naïve Bayes and Random Forest methods.
△ Less
Submitted 25 February, 2021;
originally announced February 2021.
-
Prior Independent Equilibria and Linear Multi-dimensional Bayesian Games
Authors:
Abbas Edalat,
Samira Hossein Ghorban
Abstract:
We show that a Bayesian strategy map profile is a Bayesian Nash Equilibrium independent of any prior if and only if the Bayesian strategy map profile, evaluated at any type profile, is the Nash equilibrium of the so-called local deterministic game corresponding to that type profile. We call such a Bayesian game type-regular. We then show that an m-dimensional n-agent Bayesian game whose utilities…
▽ More
We show that a Bayesian strategy map profile is a Bayesian Nash Equilibrium independent of any prior if and only if the Bayesian strategy map profile, evaluated at any type profile, is the Nash equilibrium of the so-called local deterministic game corresponding to that type profile. We call such a Bayesian game type-regular. We then show that an m-dimensional n-agent Bayesian game whose utilities are linearly dependent on the types of the agents is equivalent, following a normalisation of the type space of each agent into the (m-1)-simplex, to a simultaneous competition in nm so-called basic n-agent games. If the game is own-type-linear, i.e., the utility of each player only depends linearly on its own type, then the Bayesian game is equivalent to a simultaneous competition in m basic n-agent games, called a multi-game. We then prove that an own-type-linear Bayesian game is type-regular if it is type-regular on the vertices of the (m-1)-simplex, a result which provides a large class of type-regular Bayesian maps. The class of m-dimensional own-type-linear Bayesian games can model, via their equivalence with multi-games, simultaneous decision-making in m different environments. We show that a two dimensional own-type-linear Bayesian game can be used to give a new model of the Prisoner's Dilemma (PD) in which the prosocial tendencies of the agents are considered as their types and the two agents play simultaneously in the PD as well as in a prosocial game. This Bayesian game addresses the materialistic and the prosocial tendencies of the agents. Similarly, we present a new two dimensional Bayesian model of the Trust game in which the type of the two agents reflect their prosocial tendency or trustfulness, which leads to more reasonable Nash equilibria. We finally consider an example of such multi-environment decision making in production by several companies in multi-markets.
△ Less
Submitted 8 April, 2018;
originally announced April 2018.
-
Multi-games and a double game extension of the Prisoner's Dilemma
Authors:
Abbas Edalat,
Ali Ghoroghi,
Georgios Sakellariou
Abstract:
We propose a new class of games, called Multi-Games (MG), in which a given number of players play a fixed number of basic games simultaneously. In each round of the MG, each player will have a specific set of weights, one for each basic game, which add up to one and represent the fraction of the player's investment in each basic game. The total payoff for each player is then the convex combination…
▽ More
We propose a new class of games, called Multi-Games (MG), in which a given number of players play a fixed number of basic games simultaneously. In each round of the MG, each player will have a specific set of weights, one for each basic game, which add up to one and represent the fraction of the player's investment in each basic game. The total payoff for each player is then the convex combination, with the corresponding weights, of the payoffs it obtains in the basic games. The basic games in a MG can be regarded as different environments for the players. When the players' weights for the different games in MG are private information or types with given conditional probability distributions, we obtain a particular class of Bayesian games. We show that for the class of so-called completely pure regular Double Game (DG) with finite sets of types, the Nash equilibria (NE) of the basic games can be used to compute a Bayesian Nash equilibrium of the DG in linear time with respect to the number of types of the players. We study a DG for the Prisoner's Dilemma (PD) by extending the PD with a second so-called Social Game (SG), generalising the notion of altruistic extension of a game in which players have different altruistic levels (or social coefficients). We study two different examples of Bayesian games in this context in which the social coefficients have a finite set of values and each player only knows the probability distribution of the opponent's social coefficient. In the first case we have a completely pure regular DG for which we deduce a Bayesian NE. Finally, we use the second example to compare various strategies in a round-robin tournament of the DG for PD, in which the players can change their social coefficients incrementally from one round to the next.
△ Less
Submitted 26 June, 2012; v1 submitted 22 May, 2012;
originally announced May 2012.
-
An Extension of Gleason's Theorem for Quantum Computation
Authors:
A. Edalat
Abstract:
We develop a synthesis of Turing's paradigm of computation and von Neumann's quantum logic to serve as a model for quantum computation with recursion, such that potentially non-terminating computation can take place, as in a quantum Turing machine. This model is based on the extension of von Neumann's quantum logic to partial states, defined here as sub-probability measures on the Hilbert space,…
▽ More
We develop a synthesis of Turing's paradigm of computation and von Neumann's quantum logic to serve as a model for quantum computation with recursion, such that potentially non-terminating computation can take place, as in a quantum Turing machine. This model is based on the extension of von Neumann's quantum logic to partial states, defined here as sub-probability measures on the Hilbert space, equipped with the natural pointwise partial ordering. The sub-probability measures allow a certain probability for the non-termination of the computation. We then derive an extension of Gleason's theorem and show that, for Hilbert spaces of dimension greater than two, the partial order of sub-probability measures is order isomorphic with the collection of partial density operators, i.e., trace class positive operators with trace between zero and one, equipped with the usual partial ordering induced from positive operators. We show that the expected value of a bounded observable with respect to a partial state can be defined as a closed bounded interval, which extends the classical definition of expected value.
△ Less
Submitted 11 November, 2003;
originally announced November 2003.
