-
Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish Memes
Authors:
Shreyash Mishra,
S Suryavardan,
Megha Chakraborty,
Parth Patwa,
Anku Rani,
Aman Chadha,
Aishwarya Reganti,
Amitava Das,
Amit Sheth,
Manoj Chinnakotla,
Asif Ekbal,
Srijan Kumar
Abstract:
Analyzing memes on the internet has emerged as a crucial endeavor due to the impact this multi-modal form of content wields in sha** online discourse. Memes have become a powerful tool for expressing emotions and sentiments, possibly even spreading hate and misinformation, through humor and sarcasm. In this paper, we present the overview of the Memotion 3 shared task, as part of the DeFactify 2…
▽ More
Analyzing memes on the internet has emerged as a crucial endeavor due to the impact this multi-modal form of content wields in sha** online discourse. Memes have become a powerful tool for expressing emotions and sentiments, possibly even spreading hate and misinformation, through humor and sarcasm. In this paper, we present the overview of the Memotion 3 shared task, as part of the DeFactify 2 workshop at AAAI-23. The task released an annotated dataset of Hindi-English code-mixed memes based on their Sentiment (Task A), Emotion (Task B), and Emotion intensity (Task C). Each of these is defined as an individual task and the participants are ranked separately for each task. Over 50 teams registered for the shared task and 5 made final submissions to the test set of the Memotion 3 dataset. CLIP, BERT modifications, ViT etc. were the most popular models among the participants along with approaches such as Student-Teacher model, Fusion, and Ensembling. The best final F1 score for Task A is 34.41, Task B is 79.77 and Task C is 59.82.
△ Less
Submitted 12 September, 2023;
originally announced September 2023.
-
Findings of Factify 2: Multimodal Fake News Detection
Authors:
S Suryavardan,
Shreyash Mishra,
Megha Chakraborty,
Parth Patwa,
Anku Rani,
Aman Chadha,
Aishwarya Reganti,
Amitava Das,
Amit Sheth,
Manoj Chinnakotla,
Asif Ekbal,
Srijan Kumar
Abstract:
With social media usage growing exponentially in the past few years, fake news has also become extremely prevalent. The detrimental impact of fake news emphasizes the need for research focused on automating the detection of false information and verifying its accuracy. In this work, we present the outcome of the Factify 2 shared task, which provides a multi-modal fact verification and satire news…
▽ More
With social media usage growing exponentially in the past few years, fake news has also become extremely prevalent. The detrimental impact of fake news emphasizes the need for research focused on automating the detection of false information and verifying its accuracy. In this work, we present the outcome of the Factify 2 shared task, which provides a multi-modal fact verification and satire news dataset, as part of the DeFactify 2 workshop at AAAI'23. The data calls for a comparison based approach to the task by pairing social media claims with supporting documents, with both text and image, divided into 5 classes based on multi-modal relations. In the second iteration of this task we had over 60 participants and 9 final test-set submissions. The best performances came from the use of DeBERTa for text and Swinv2 and CLIP for image. The highest F1 score averaged for all five classes was 81.82%.
△ Less
Submitted 12 September, 2023; v1 submitted 19 July, 2023;
originally announced July 2023.
-
Factify 2: A Multimodal Fake News and Satire News Dataset
Authors:
S Suryavardan,
Shreyash Mishra,
Parth Patwa,
Megha Chakraborty,
Anku Rani,
Aishwarya Reganti,
Aman Chadha,
Amitava Das,
Amit Sheth,
Manoj Chinnakotla,
Asif Ekbal,
Srijan Kumar
Abstract:
The internet gives the world an open platform to express their views and share their stories. While this is very valuable, it makes fake news one of our society's most pressing problems. Manual fact checking process is time consuming, which makes it challenging to disprove misleading assertions before they cause significant harm. This is he driving interest in automatic fact or claim verification.…
▽ More
The internet gives the world an open platform to express their views and share their stories. While this is very valuable, it makes fake news one of our society's most pressing problems. Manual fact checking process is time consuming, which makes it challenging to disprove misleading assertions before they cause significant harm. This is he driving interest in automatic fact or claim verification. Some of the existing datasets aim to support development of automating fact-checking techniques, however, most of them are text based. Multi-modal fact verification has received relatively scant attention. In this paper, we provide a multi-modal fact-checking dataset called FACTIFY 2, improving Factify 1 by using new data sources and adding satire articles. Factify 2 has 50,000 new data instances. Similar to FACTIFY 1.0, we have three broad categories - support, no-evidence, and refute, with sub-categories based on the entailment of visual and textual data. We also provide a BERT and Vison Transformer based baseline, which achieves 65% F1 score in the test set. The baseline codes and the dataset will be made available at https://github.com/surya1701/Factify-2.0.
△ Less
Submitted 2 October, 2023; v1 submitted 7 April, 2023;
originally announced April 2023.
-
Memotion 3: Dataset on Sentiment and Emotion Analysis of Codemixed Hindi-English Memes
Authors:
Shreyash Mishra,
S Suryavardan,
Parth Patwa,
Megha Chakraborty,
Anku Rani,
Aishwarya Reganti,
Aman Chadha,
Amitava Das,
Amit Sheth,
Manoj Chinnakotla,
Asif Ekbal,
Srijan Kumar
Abstract:
Memes are the new-age conveyance mechanism for humor on social media sites. Memes often include an image and some text. Memes can be used to promote disinformation or hatred, thus it is crucial to investigate in details. We introduce Memotion 3, a new dataset with 10,000 annotated memes. Unlike other prevalent datasets in the domain, including prior iterations of Memotion, Memotion 3 introduces Hi…
▽ More
Memes are the new-age conveyance mechanism for humor on social media sites. Memes often include an image and some text. Memes can be used to promote disinformation or hatred, thus it is crucial to investigate in details. We introduce Memotion 3, a new dataset with 10,000 annotated memes. Unlike other prevalent datasets in the domain, including prior iterations of Memotion, Memotion 3 introduces Hindi-English Codemixed memes while prior works in the area were limited to only the English memes. We describe the Memotion task, the data collection and the dataset creation methodologies. We also provide a baseline for the task. The baseline code and dataset will be made available at https://github.com/Shreyashm16/Memotion-3.0
△ Less
Submitted 2 October, 2023; v1 submitted 17 March, 2023;
originally announced March 2023.
-
Ruuh: A Deep Learning Based Conversational Social Agent
Authors:
Sonam Damani,
Nitya Raviprakash,
Umang Gupta,
Ankush Chatterjee,
Meghana Joshi,
Khyatti Gupta,
Kedhar Nath Narahari,
Puneet Agrawal,
Manoj Kumar Chinnakotla,
Sneha Magapu,
Abhishek Mathur
Abstract:
Dialogue systems and conversational agents are becoming increasingly popular in the modern society but building an agent capable of holding intelligent conversation with its users is a challenging problem for artificial intelligence. In this demo, we demonstrate a deep learning based conversational social agent called "Ruuh" (facebook.com/Ruuh) designed by a team at Microsoft India to converse on…
▽ More
Dialogue systems and conversational agents are becoming increasingly popular in the modern society but building an agent capable of holding intelligent conversation with its users is a challenging problem for artificial intelligence. In this demo, we demonstrate a deep learning based conversational social agent called "Ruuh" (facebook.com/Ruuh) designed by a team at Microsoft India to converse on a wide range of topics. Ruuh needs to think beyond the utilitarian notion of merely generating "relevant" responses and meet a wider range of user social needs, like expressing happiness when user's favorite team wins, sharing a cute comment on showing the pictures of the user's pet and so on. The agent also needs to detect and respond to abusive language, sensitive topics and trolling behavior of the users. Many of these problems pose significant research challenges which will be demonstrated in our demo. Our agent has interacted with over 2 million real world users till date which has generated over 150 million user conversations.
△ Less
Submitted 22 October, 2018;
originally announced October 2018.
-
Relevance Scoring of Triples Using Ordinal Logistic Classification - The Celosia Triple Scorer at WSDM Cup 2017
Authors:
Nausheen Fatma,
Manoj K. Chinnakotla,
Manish Shrivastava
Abstract:
In this paper, we report our participation in the Task 2: Triple Scoring of WSDM Cup challenge 2017. In this task, we were provided with triples of "type-like" relations which were given human-annotated relevance scores ranging from 0 to 7, with 7 being the "most relevant" and 0 being the "least relevant". The task focuses on two such relations: profession and nationality. We built a system which…
▽ More
In this paper, we report our participation in the Task 2: Triple Scoring of WSDM Cup challenge 2017. In this task, we were provided with triples of "type-like" relations which were given human-annotated relevance scores ranging from 0 to 7, with 7 being the "most relevant" and 0 being the "least relevant". The task focuses on two such relations: profession and nationality. We built a system which could automatically predict the relevance scores for unseen triples. Our model is primarily a supervised machine learning based one in which we use well-designed features which are used to a make a Logistic Ordinal Regression based classification model. The proposed system achieves an overall accuracy score of 0.73 and Kendall's tau score of 0.36.
△ Less
Submitted 22 December, 2017;
originally announced December 2017.
-
Deep Feature Fusion Network for Answer Quality Prediction in Community Question Answering
Authors:
Sai Praneeth Suggu,
Kushwanth N. Goutham,
Manoj K. Chinnakotla,
Manish Shrivastava
Abstract:
Community Question Answering (cQA) forums have become a popular medium for soliciting direct answers to specific questions of users from experts or other experienced users on a given topic. However, for a given question, users sometimes have to sift through a large number of low-quality or irrelevant answers to find out the answer which satisfies their information need. To alleviate this, the prob…
▽ More
Community Question Answering (cQA) forums have become a popular medium for soliciting direct answers to specific questions of users from experts or other experienced users on a given topic. However, for a given question, users sometimes have to sift through a large number of low-quality or irrelevant answers to find out the answer which satisfies their information need. To alleviate this, the problem of Answer Quality Prediction (AQP) aims to predict the quality of an answer posted in response to a forum question. Current AQP systems either learn models using - a) various hand-crafted features (HCF) or b) use deep learning (DL) techniques which automatically learn the required feature representations.
In this paper, we propose a novel approach for AQP known as - "Deep Feature Fusion Network (DFFN)" which leverages the advantages of both hand-crafted features and deep learning based systems. Given a question-answer pair along with its metadata, DFFN independently - a) learns deep features using a Convolutional Neural Network (CNN) and b) computes hand-crafted features using various external resources and then combines them using a deep neural network trained to predict the final answer quality. DFFN achieves state-of-the-art performance on the standard SemEval-2015 and SemEval-2016 benchmark datasets and outperforms baseline approaches which individually employ either HCF or DL based techniques alone.
△ Less
Submitted 26 June, 2016; v1 submitted 22 June, 2016;
originally announced June 2016.
