-
The effect of diversity on group decision-making
Authors:
Georgi Karadzhov,
Andreas Vlachos,
Tom Stafford
Abstract:
We explore different aspects of cognitive diversity and its effect on the success of group deliberation. To evaluate this, we use 500 dialogues from small, online groups discussing the Wason Card Selection task - the DeliData corpus. Leveraging the corpus, we perform quantitative analysis evaluating three different measures of cognitive diversity. First, we analyse the effect of group size as a pr…
▽ More
We explore different aspects of cognitive diversity and its effect on the success of group deliberation. To evaluate this, we use 500 dialogues from small, online groups discussing the Wason Card Selection task - the DeliData corpus. Leveraging the corpus, we perform quantitative analysis evaluating three different measures of cognitive diversity. First, we analyse the effect of group size as a proxy measure for diversity. Second, we evaluate the effect of the size of the initial idea pool. Finally, we look into the content of the discussion by analysing discussed solutions, discussion patterns, and how conversational probing can improve those characteristics. Despite the reputation of groups for compounding bias, we show that small groups can, through dialogue, overcome intuitive biases and improve individual decision-making. Across a large sample and different operationalisations, we consistently find that greater cognitive diversity is associated with more successful group deliberation. Code and data used for the analysis are available in the repository: https://github.com/gkaradzhov/cognitive-diversity-groups-cogsci24.
△ Less
Submitted 20 May, 2024; v1 submitted 2 February, 2024;
originally announced February 2024.
-
What makes you change your mind? An empirical investigation in online group decision-making conversations
Authors:
Georgi Karadzhov,
Tom Stafford,
Andreas Vlachos
Abstract:
People leverage group discussions to collaborate in order to solve complex tasks, e.g. in project meetings or hiring panels. By doing so, they engage in a variety of conversational strategies where they try to convince each other of the best approach and ultimately reach a decision. In this work, we investigate methods for detecting what makes someone change their mind. To this end, we leverage a…
▽ More
People leverage group discussions to collaborate in order to solve complex tasks, e.g. in project meetings or hiring panels. By doing so, they engage in a variety of conversational strategies where they try to convince each other of the best approach and ultimately reach a decision. In this work, we investigate methods for detecting what makes someone change their mind. To this end, we leverage a recently introduced dataset containing group discussions of people collaborating to solve a task. To find out what makes someone change their mind, we incorporate various techniques such as neural text classification and language-agnostic change point detection. Evaluation of these methods shows that while the task is not trivial, the best way to approach it is using a language-aware model with learning-to-rank training. Finally, we examine the cues that the models develop as indicative of the cause of a change of mind.
△ Less
Submitted 25 July, 2022;
originally announced July 2022.
-
Leaf: Multiple-Choice Question Generation
Authors:
Kristiyan Vachev,
Momchil Hardalov,
Georgi Karadzhov,
Georgi Georgiev,
Ivan Koychev,
Preslav Nakov
Abstract:
Testing with quiz questions has proven to be an effective way to assess and improve the educational process. However, manually creating quizzes is tedious and time-consuming. To address this challenge, we present Leaf, a system for generating multiple-choice questions from factual text. In addition to being very well suited for the classroom, Leaf could also be used in an industrial setting, e.g.,…
▽ More
Testing with quiz questions has proven to be an effective way to assess and improve the educational process. However, manually creating quizzes is tedious and time-consuming. To address this challenge, we present Leaf, a system for generating multiple-choice questions from factual text. In addition to being very well suited for the classroom, Leaf could also be used in an industrial setting, e.g., to facilitate onboarding and knowledge sharing, or as a component of chatbots, question answering systems, or Massive Open Online Courses (MOOCs). The code and the demo are available on https://github.com/KristiyanVachev/Leaf-Question-Generation.
△ Less
Submitted 22 January, 2022;
originally announced January 2022.
-
Generating Answer Candidates for Quizzes and Answer-Aware Question Generators
Authors:
Kristiyan Vachev,
Momchil Hardalov,
Georgi Karadzhov,
Georgi Georgiev,
Ivan Koychev,
Preslav Nakov
Abstract:
In education, open-ended quiz questions have become an important tool for assessing the knowledge of students. Yet, manually preparing such questions is a tedious task, and thus automatic question generation has been proposed as a possible alternative. So far, the vast majority of research has focused on generating the question text, relying on question answering datasets with readily picked answe…
▽ More
In education, open-ended quiz questions have become an important tool for assessing the knowledge of students. Yet, manually preparing such questions is a tedious task, and thus automatic question generation has been proposed as a possible alternative. So far, the vast majority of research has focused on generating the question text, relying on question answering datasets with readily picked answers, and the problem of how to come up with answer candidates in the first place has been largely ignored. Here, we aim to bridge this gap. In particular, we propose a model that can generate a specified number of answer candidates for a given passage of text, which can then be used by instructors to write questions manually or can be passed as an input to automatic answer-aware question generators. Our experiments show that our proposed answer candidate generation model outperforms several baselines.
△ Less
Submitted 29 August, 2021;
originally announced August 2021.
-
DeliData: A dataset for deliberation in multi-party problem solving
Authors:
Georgi Karadzhov,
Tom Stafford,
Andreas Vlachos
Abstract:
Group deliberation enables people to collaborate and solve problems, however, it is understudied due to a lack of resources. To this end, we introduce the first publicly available dataset containing collaborative conversations on solving a well-established cognitive task, consisting of 500 group dialogues and 14k utterances. In 64% of these conversations, the group members are able to find a bette…
▽ More
Group deliberation enables people to collaborate and solve problems, however, it is understudied due to a lack of resources. To this end, we introduce the first publicly available dataset containing collaborative conversations on solving a well-established cognitive task, consisting of 500 group dialogues and 14k utterances. In 64% of these conversations, the group members are able to find a better solution than they had identified individually, and in 43.8% of the groups who had a correct answer as their final solution, none of the participants had solved the task correctly by themselves. Furthermore, we propose a novel annotation schema that captures deliberation cues and release all 14k utterances annotated with it. Finally, we use the proposed dataset to develop and evaluate two methods for generating deliberation utterances. The data collection platform, dataset and annotated corpus are publicly available at https://delibot.xyz.
△ Less
Submitted 16 April, 2023; v1 submitted 11 August, 2021;
originally announced August 2021.
-
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)
Authors:
Marcos Zampieri,
Preslav Nakov,
Sara Rosenthal,
Pepa Atanasova,
Georgi Karadzhov,
Hamdy Mubarak,
Leon Derczynski,
Zeses Pitenis,
Çağrı Çöltekin
Abstract:
We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, En…
▽ More
We present the results and main findings of SemEval-2020 Task 12 on Multilingual Offensive Language Identification in Social Media (OffensEval 2020). The task involves three subtasks corresponding to the hierarchical taxonomy of the OLID schema (Zampieri et al., 2019a) from OffensEval 2019. The task featured five languages: English, Arabic, Danish, Greek, and Turkish for Subtask A. In addition, English also featured Subtasks B and C. OffensEval 2020 was one of the most popular tasks at SemEval-2020 attracting a large number of participants across all subtasks and also across all languages. A total of 528 teams signed up to participate in the task, 145 teams submitted systems during the evaluation period, and 70 submitted system description papers.
△ Less
Submitted 30 September, 2020; v1 submitted 12 June, 2020;
originally announced June 2020.
-
What Was Written vs. Who Read It: News Media Profiling Using Text Analysis and Social Media Context
Authors:
Ramy Baly,
Georgi Karadzhov,
Jisun An,
Haewoon Kwak,
Yoan Dinkov,
Ahmed Ali,
James Glass,
Preslav Nakov
Abstract:
Predicting the political bias and the factuality of reporting of entire news outlets are critical elements of media profiling, which is an understudied but an increasingly important research direction. The present level of proliferation of fake, biased, and propagandistic content online, has made it impossible to fact-check every single suspicious claim, either manually or automatically. Alternati…
▽ More
Predicting the political bias and the factuality of reporting of entire news outlets are critical elements of media profiling, which is an understudied but an increasingly important research direction. The present level of proliferation of fake, biased, and propagandistic content online, has made it impossible to fact-check every single suspicious claim, either manually or automatically. Alternatively, we can profile entire news outlets and look for those that are likely to publish fake or biased content. This approach makes it possible to detect likely "fake news" the moment they are published, by simply checking the reliability of their source. From a practical perspective, political bias and factuality of reporting have a linguistic aspect but also a social context. Here, we study the impact of both, namely (i) what was written (i.e., what was published by the target medium, and how it describes itself on Twitter) vs. (ii) who read it (i.e., analyzing the readers of the target medium on Facebook, Twitter, and YouTube). We further study (iii) what was written about the target medium on Wikipedia. The evaluation results show that what was written matters most, and that putting all information sources together yields huge improvements over the current state-of-the-art.
△ Less
Submitted 9 May, 2020;
originally announced May 2020.
-
SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identification
Authors:
Sara Rosenthal,
Pepa Atanasova,
Georgi Karadzhov,
Marcos Zampieri,
Preslav Nakov
Abstract:
The widespread use of offensive content in social media has led to an abundance of research in detecting language such as hate speech, cyberbullying, and cyber-aggression. Recent work presented the OLID dataset, which follows a taxonomy for offensive language identification that provides meaningful information for understanding the type and the target of offensive messages. However, it is limited…
▽ More
The widespread use of offensive content in social media has led to an abundance of research in detecting language such as hate speech, cyberbullying, and cyber-aggression. Recent work presented the OLID dataset, which follows a taxonomy for offensive language identification that provides meaningful information for understanding the type and the target of offensive messages. However, it is limited in size and it might be biased towards offensive language as it was collected using keywords. In this work, we present SOLID, an expanded dataset, where the tweets were collected in a more principled manner. SOLID contains over nine million English tweets labeled in a semi-supervised fashion. We demonstrate that using SOLID along with OLID yields sizable performance gains on the OLID test set for two different models, especially for the lower levels of the taxonomy.
△ Less
Submitted 24 September, 2021; v1 submitted 29 April, 2020;
originally announced April 2020.
-
Tanbih: Get To Know What You Are Reading
Authors:
Yifan Zhang,
Giovanni Da San Martino,
Alberto Barrón-Cedeño,
Salvatore Romeo,
Jisun An,
Haewoon Kwak,
Todor Staykovski,
Israa Jaradat,
Georgi Karadzhov,
Ramy Baly,
Kareem Darwish,
James Glass,
Preslav Nakov
Abstract:
We introduce Tanbih, a news aggregator with intelligent analysis tools to help readers understanding what's behind a news story. Our system displays news grouped into events and generates media profiles that show the general factuality of reporting, the degree of propagandistic content, hyper-partisanship, leading political ideology, general frame of reporting, and stance with respect to various c…
▽ More
We introduce Tanbih, a news aggregator with intelligent analysis tools to help readers understanding what's behind a news story. Our system displays news grouped into events and generates media profiles that show the general factuality of reporting, the degree of propagandistic content, hyper-partisanship, leading political ideology, general frame of reporting, and stance with respect to various claims and topics of a news outlet. In addition, we automatically analyse each article to detect whether it is propagandistic and to determine its stance with respect to a number of controversial topics.
△ Less
Submitted 4 October, 2019;
originally announced October 2019.
-
Automatic Fact-Checking Using Context and Discourse Information
Authors:
Pepa Atanasova,
Preslav Nakov,
Lluís Màrquez,
Alberto Barrón-Cedeño,
Georgi Karadzhov,
Tsvetomila Mihaylova,
Mitra Mohtarami,
James Glass
Abstract:
We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims, and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms o…
▽ More
We study the problem of automatic fact-checking, paying special attention to the impact of contextual and discourse information. We address two related tasks: (i) detecting check-worthy claims, and (ii) fact-checking claims. We develop supervised systems based on neural networks, kernel-based support vector machines, and combinations thereof, which make use of rich input representations in terms of discourse cues and contextual features. For the check-worthiness estimation task, we focus on political debates, and we model the target claim in the context of the full intervention of a participant and the previous and the following turns in the debate, taking into account contextual meta information. For the fact-checking task, we focus on answer verification in a community forum, and we model the veracity of the answer with respect to the entire question--answer thread in which it occurs as well as with respect to other related posts from the entire forum. We develop annotated datasets for both tasks and we run extensive experimental evaluation, confirming that both types of information ---but especially contextual features--- play an important role.
△ Less
Submitted 4 August, 2019;
originally announced August 2019.
-
Evaluating Variable-Length Multiple-Option Lists in Chatbots and Mobile Search
Authors:
Pepa Atanasova,
Georgi Karadzhov,
Yasen Kiprov,
Preslav Nakov,
Fabrizio Sebastiani
Abstract:
In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links'' metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, al…
▽ More
In recent years, the proliferation of smart mobile devices has lead to the gradual integration of search functionality within mobile platforms. This has created an incentive to move away from the "ten blue links'' metaphor, as mobile users are less likely to click on them, expecting to get the answer directly from the snippets. In turn, this has revived the interest in Question Answering. Then, along came chatbots, conversational systems, and messaging platforms, where the user needs could be better served with the system asking follow-up questions in order to better understand the user's intent. While typically a user would expect a single response at any utterance, a system could also return multiple options for the user to select from, based on different system understandings of the user's intent. However, this possibility should not be overused, as this practice could confuse and/or annoy the user. How to produce good variable-length lists, given the conflicting objectives of staying short while maximizing the likelihood of having a correct answer included in the list, is an underexplored problem. It is also unclear how to evaluate a system that tries to do that. Here we aim to bridge this gap. In particular, we define some necessary and some optional properties that an evaluation measure fit for this purpose should have. We further show that existing evaluation measures from the IR tradition are not entirely suitable for this setup, and we propose novel evaluation measures that address it satisfactorily.
△ Less
Submitted 25 May, 2019;
originally announced May 2019.
-
Multi-Task Ordinal Regression for Jointly Predicting the Trustworthiness and the Leading Political Ideology of News Media
Authors:
Ramy Baly,
Georgi Karadzhov,
Abdelrhman Saleh,
James Glass,
Preslav Nakov
Abstract:
In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: (i) trustworthiness estimation (on a 3-point scale) and (ii) political ideology detection (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles. In particular, we propose a multi-task ordinal regression framework that models the two p…
▽ More
In the context of fake news, bias, and propaganda, we study two important but relatively under-explored problems: (i) trustworthiness estimation (on a 3-point scale) and (ii) political ideology detection (left/right bias on a 7-point scale) of entire news outlets, as opposed to evaluating individual articles. In particular, we propose a multi-task ordinal regression framework that models the two problems jointly. This is motivated by the observation that hyper-partisanship is often linked to low trustworthiness, e.g., appealing to emotions rather than sticking to the facts, while center media tend to be generally more impartial and trustworthy. We further use several auxiliary tasks, modeling centrality, hyperpartisanship, as well as left-vs.-right bias on a coarse-grained scale. The evaluation results show sizable performance gains by the joint models over models that target the problems in isolation.
△ Less
Submitted 31 March, 2019;
originally announced April 2019.
-
Predicting Factuality of Reporting and Bias of News Media Sources
Authors:
Ramy Baly,
Georgi Karadzhov,
Dimitar Alexandrov,
James Glass,
Preslav Nakov
Abstract:
We present a study on predicting the factuality of reporting and bias of news media. While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media. These are under-studied but arguably important research problems, both in their own right and as a prior for fact-checking systems. We experiment with a large list of news we…
▽ More
We present a study on predicting the factuality of reporting and bias of news media. While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media. These are under-studied but arguably important research problems, both in their own right and as a prior for fact-checking systems. We experiment with a large list of news websites and with a rich set of features derived from (i) a sample of articles from the target news medium, (ii) its Wikipedia page, (iii) its Twitter account, (iv) the structure of its URL, and (v) information about the Web traffic it attracts. The experimental results show sizable performance gains over the baselines, and confirm the importance of each feature type.
△ Less
Submitted 1 October, 2018;
originally announced October 2018.
-
We Built a Fake News & Click-bait Filter: What Happened Next Will Blow Your Mind!
Authors:
Georgi Karadzhov,
Pepa Gencheva,
Preslav Nakov,
Ivan Koychev
Abstract:
It is completely amazing! Fake news and click-baits have totally invaded the cyber space. Let us face it: everybody hates them for three simple reasons. Reason #2 will absolutely amaze you. What these can achieve at the time of election will completely blow your mind! Now, we all agree, this cannot go on, you know, somebody has to stop it. So, we did this research on fake news/click-bait detection…
▽ More
It is completely amazing! Fake news and click-baits have totally invaded the cyber space. Let us face it: everybody hates them for three simple reasons. Reason #2 will absolutely amaze you. What these can achieve at the time of election will completely blow your mind! Now, we all agree, this cannot go on, you know, somebody has to stop it. So, we did this research on fake news/click-bait detection and trust us, it is totally great research, it really is! Make no mistake. This is the best research ever! Seriously, come have a look, we have it all: neural networks, attention mechanism, sentiment lexicons, author profiling, you name it. Lexical features, semantic features, we absolutely have it all. And we have totally tested it, trust us! We have results, and numbers, really big numbers. The best numbers ever! Oh, and analysis, absolutely top notch analysis. Interested? Come read the shocking truth about fake news and click-bait in the Bulgarian cyber space. You won't believe what we have found!
△ Less
Submitted 10 March, 2018;
originally announced March 2018.
-
Fact Checking in Community Forums
Authors:
Tsvetomila Mihaylova,
Preslav Nakov,
Lluis Marquez,
Alberto Barron-Cedeno,
Mitra Mohtarami,
Georgi Karadzhov,
James Glass
Abstract:
Community Question Answering (cQA) forums are very popular nowadays, as they represent effective means for communities around particular topics to share information. Unfortunately, this information is not always factual. Thus, here we explore a new dimension in the context of cQA, which has been ignored so far: checking the veracity of answers to particular questions in cQA forums. As this is a ne…
▽ More
Community Question Answering (cQA) forums are very popular nowadays, as they represent effective means for communities around particular topics to share information. Unfortunately, this information is not always factual. Thus, here we explore a new dimension in the context of cQA, which has been ignored so far: checking the veracity of answers to particular questions in cQA forums. As this is a new problem, we create a specialized dataset for it. We further propose a novel multi-faceted model, which captures information from the answer content (what is said and how), from the author profile (who says it), from the rest of the community forum (where it is said), and from external authoritative sources of information (external support). Evaluation results show a MAP value of 86.54, which is 21 points absolute above the baseline.
△ Less
Submitted 8 March, 2018;
originally announced March 2018.
-
Fully Automated Fact Checking Using External Sources
Authors:
Georgi Karadzhov,
Preslav Nakov,
Lluis Marquez,
Alberto Barron-Cedeno,
Ivan Koychev
Abstract:
Given the constantly growing proliferation of false claims online in recent years, there has been also a growing research interest in automatically distinguishing false rumors from factually true claims. Here, we propose a general-purpose framework for fully-automatic fact checking using external sources, tap** the potential of the entire Web as a knowledge source to confirm or reject a claim. O…
▽ More
Given the constantly growing proliferation of false claims online in recent years, there has been also a growing research interest in automatically distinguishing false rumors from factually true claims. Here, we propose a general-purpose framework for fully-automatic fact checking using external sources, tap** the potential of the entire Web as a knowledge source to confirm or reject a claim. Our framework uses a deep neural network with LSTM text encoding to combine semantic kernels with task-specific embeddings that encode a claim together with pieces of potentially-relevant text fragments from the Web, taking the source reliability into account. The evaluation results show good performance on two different tasks and datasets: (i) rumor detection and (ii) fact checking of the answers to a question in community question answering forums.
△ Less
Submitted 1 October, 2017;
originally announced October 2017.
-
Classes of Measures Generated by Capacities
Authors:
G. Dafni,
G. E. Karadzhov,
J. Xiao
Abstract:
We introduce classes of measures in the half-space $\mathbf{R}^{n+1}_+,$ generated by Riesz, or Bessel, or Besov capacities in $\mathbf{R}^n$, and give a geometric characterization as Carleson-type measures.
We introduce classes of measures in the half-space $\mathbf{R}^{n+1}_+,$ generated by Riesz, or Bessel, or Besov capacities in $\mathbf{R}^n$, and give a geometric characterization as Carleson-type measures.
△ Less
Submitted 2 October, 2006;
originally announced October 2006.
-
Infinite dimensional families of locally nonsolvable partial differential operators
Authors:
Michael Christ,
Georgi Karadzhov,
Detlef Müller
Abstract:
Local solvability is analyzed for natural families of partial differential operators having double characteristics. In some families the set of all operators that are not locally solvable is shown to have both infinite dimension and infinite codimension.
Local solvability is analyzed for natural families of partial differential operators having double characteristics. In some families the set of all operators that are not locally solvable is shown to have both infinite dimension and infinite codimension.
△ Less
Submitted 11 December, 1995;
originally announced December 1995.
-
Local Solvability For a Class of Partial Differential Operators With Double Characteristics
Authors:
Michael Christ,
Georgi Karadzhov
Abstract:
A necessary and sufficient condition for local solvability is presented for the linear partial differential operators $-X^2-Y^2+ia(x)[X,Y]$ in $\bold R^3=\{(x,y,t)\}$, where $X=\partial_x,\; Y=\partial_y+x^k\partial_t$, and $a\in C^{\infty}(\bold R^1)$ is real valued, for each positive integer $k$.
A necessary and sufficient condition for local solvability is presented for the linear partial differential operators $-X^2-Y^2+ia(x)[X,Y]$ in $\bold R^3=\{(x,y,t)\}$, where $X=\partial_x,\; Y=\partial_y+x^k\partial_t$, and $a\in C^{\infty}(\bold R^1)$ is real valued, for each positive integer $k$.
△ Less
Submitted 5 December, 1995;
originally announced December 1995.
