-
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers
Authors:
Osman Batur İnce,
Tanin Zeraati,
Semih Yagcioglu,
Yadollah Yaghoobzadeh,
Erkut Erdem,
Aykut Erdem
Abstract:
Neural networks have revolutionized language modeling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains a topic of debate. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harness…
▽ More
Neural networks have revolutionized language modeling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains a topic of debate. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harnessing the power of dataset cartography (Swayamdipta et al., 2020). By strategically identifying a subset of compositional generalization data using this approach, we achieve a remarkable improvement in model accuracy, yielding enhancements of up to 10% on CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminating the need for hyperparameter tuning while consistently achieving superior performance. Our findings highlight the untapped potential of dataset cartography in unleashing the full capabilities of compositional generalization within Transformer models. Our code is available at https://github.com/cyberiada/cartography-for-compositionality.
△ Less
Submitted 18 October, 2023;
originally announced October 2023.
-
Diversity dilemmas: uncovering gender and nationality biases in graduate admissions across top North American computer science programs
Authors:
Ghazal Kalhor,
Tanin Zeraati,
Behnam Bahrak
Abstract:
Although different organizations have defined policies towards diversity in academia, many argue that minorities are still disadvantaged in university admissions due to biases. Extensive research has been conducted on detecting partiality patterns in the academic community. However, in the last few decades, limited research has focused on assessing gender and nationality biases in graduate admissi…
▽ More
Although different organizations have defined policies towards diversity in academia, many argue that minorities are still disadvantaged in university admissions due to biases. Extensive research has been conducted on detecting partiality patterns in the academic community. However, in the last few decades, limited research has focused on assessing gender and nationality biases in graduate admission results of universities. In this study, we collected a novel and comprehensive dataset containing information on approximately 14,000 graduate students majoring in computer science (CS) at the top 25 North American universities. We used statistical hypothesis tests to determine whether there is a preference for students' gender and nationality in the admission processes. In addition to partiality patterns, we discuss the relationship between gender/nationality diversity and the scientific achievements of research teams. Consistent with previous studies, our findings show that there is no gender bias in the admission of graduate students to research groups, but we observed bias based on students' nationality.
△ Less
Submitted 29 August, 2023; v1 submitted 1 February, 2023;
originally announced February 2023.
-
UTNLP at SemEval-2022 Task 6: A Comparative Analysis of Sarcasm Detection Using Generative-based and Mutation-based Data Augmentation
Authors:
Amirhossein Abaskohi,
Arash Rasouli,
Tanin Zeraati,
Behnam Bahrak
Abstract:
Sarcasm is a term that refers to the use of words to mock, irritate, or amuse someone. It is commonly used on social media. The metaphorical and creative nature of sarcasm presents a significant difficulty for sentiment analysis systems based on affective computing. The methodology and results of our team, UTNLP, in the SemEval-2022 shared task 6 on sarcasm detection are presented in this paper. W…
▽ More
Sarcasm is a term that refers to the use of words to mock, irritate, or amuse someone. It is commonly used on social media. The metaphorical and creative nature of sarcasm presents a significant difficulty for sentiment analysis systems based on affective computing. The methodology and results of our team, UTNLP, in the SemEval-2022 shared task 6 on sarcasm detection are presented in this paper. We put different models, and data augmentation approaches to the test and report on which one works best. The tests begin with traditional machine learning models and progress to transformer-based and attention-based models. We employed data augmentation based on data mutation and data generation. Using RoBERTa and mutation-based data augmentation, our best approach achieved an F1-sarcastic of 0.38 in the competition's evaluation phase. After the competition, we fixed our model's flaws and achieved an F1-sarcastic of 0.414.
△ Less
Submitted 28 June, 2022; v1 submitted 18 April, 2022;
originally announced April 2022.
