-
Identifying Fairness Issues in Automatically Generated Testing Content
Authors:
Kevin Stowe,
Benny Longwill,
Alyssa Francis,
Tatsuya Aoyama,
Debanjan Ghosh,
Swapna Somasundaran
Abstract:
Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Spe…
▽ More
Natural language generation tools are powerful and effective for generating content. However, language models are known to display bias and fairness issues, making them impractical to deploy for many use cases. We here focus on how fairness issues impact automatically generated test content, which can have stringent requirements to ensure the test measures only what it was intended to measure. Specifically, we review test content generated for a large-scale standardized English proficiency test with the goal of identifying content that only pertains to a certain subset of the test population as well as content that has the potential to be upsetting or distracting to some test takers. Issues like these could inadvertently impact a test taker's score and thus should be avoided. This kind of content does not reflect the more commonly-acknowledged biases, making it challenging even for modern models that contain safeguards. We build a dataset of 601 generated texts annotated for fairness and explore a variety of methods for classification: fine-tuning, topic-based classification, and prompting, including few-shot and self-correcting prompts. We find that combining prompt self-correction and few-shot learning performs best, yielding an F1 score of 0.79 on our held-out test set, while much smaller BERT- and topic-based models have competitive performance on out-of-domain data.
△ Less
Submitted 1 May, 2024; v1 submitted 23 April, 2024;
originally announced April 2024.
-
Lessons Learned from a Citizen Science Project for Natural Language Processing
Authors:
Jan-Christoph Klie,
Ji-Ung Lee,
Kevin Stowe,
Gözde Gül Şahin,
Nafise Sadat Moosavi,
Luke Bates,
Dominic Petrak,
Richard Eckart de Castilho,
Iryna Gurevych
Abstract:
Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how…
▽ More
Many Natural Language Processing (NLP) systems use annotated corpora for training and evaluation. However, labeled data is often costly to obtain and scaling annotation projects is difficult, which is why annotation tasks are often outsourced to paid crowdworkers. Citizen Science is an alternative to crowdsourcing that is relatively unexplored in the context of NLP. To investigate whether and how well Citizen Science can be applied in this setting, we conduct an exploratory study into engaging different groups of volunteers in Citizen Science for NLP by re-annotating parts of a pre-existing crowdsourced dataset. Our results show that this can yield high-quality annotations and attract motivated volunteers, but also requires considering factors such as scalability, participation over time, and legal and ethical issues. We summarize lessons learned in the form of guidelines and provide our code and data to aid future work on Citizen Science.
△ Less
Submitted 25 April, 2023;
originally announced April 2023.
-
Controlled Language Generation for Language Learning Items
Authors:
Kevin Stowe,
Debanjan Ghosh,
Mengxuan Zhao
Abstract:
This work aims to employ natural language generation (NLG) to rapidly generate items for English language learning applications: this requires both language models capable of generating fluent, high-quality English, and to control the output of the generation to match the requirements of the relevant items. We experiment with deep pretrained models for this task, develo** novel methods for contr…
▽ More
This work aims to employ natural language generation (NLG) to rapidly generate items for English language learning applications: this requires both language models capable of generating fluent, high-quality English, and to control the output of the generation to match the requirements of the relevant items. We experiment with deep pretrained models for this task, develo** novel methods for controlling items for factors relevant in language learning: diverse sentences for different proficiency levels and argument structure to test grammar. Human evaluation demonstrates high grammatically scores for all models (3.4 and above out of 4), and higher length (24%) and complexity (9%) over the baseline for the advanced proficiency model. Our results show that we can achieve strong performance while adding additional control to ensure diverse, tailored content for individual users.
△ Less
Submitted 28 November, 2022;
originally announced November 2022.
-
Metaphor Generation with Conceptual Map**s
Authors:
Kevin Stowe,
Tuhin Chakrabarty,
Nanyun Peng,
Smaranda Muresan,
Iryna Gurevych
Abstract:
Generating metaphors is a difficult task as it requires understanding nuanced relationships between abstract concepts. In this paper, we aim to generate a metaphoric sentence given a literal expression by replacing relevant verbs. Guided by conceptual metaphor theory, we propose to control the generation process by encoding conceptual map**s between cognitive domains to generate meaningful metap…
▽ More
Generating metaphors is a difficult task as it requires understanding nuanced relationships between abstract concepts. In this paper, we aim to generate a metaphoric sentence given a literal expression by replacing relevant verbs. Guided by conceptual metaphor theory, we propose to control the generation process by encoding conceptual map**s between cognitive domains to generate meaningful metaphoric expressions. To achieve this, we develop two methods: 1) using FrameNet-based embeddings to learn map**s between domains and applying them at the lexical level (CM-Lex), and 2) deriving source/target pairs to train a controlled seq-to-seq generation model (CM-BART). We assess our methods through automatic and human evaluation for basic metaphoricity and conceptual metaphor presence. We show that the unsupervised CM-Lex model is competitive with recent deep learning metaphor generation systems, and CM-BART outperforms all other models both in automatic and human evaluations.
△ Less
Submitted 2 June, 2021;
originally announced June 2021.
-
The challenges of temporal alignment on Twitter during crises
Authors:
Aniket Pramanick,
Tilman Beck,
Kevin Stowe,
Iryna Gurevych
Abstract:
Language use changes over time, and this impacts the effectiveness of NLP systems. This phenomenon is even more prevalent in social media data during crisis events where meaning and frequency of word usage may change over the course of days. Contextual language models fail to adapt temporally, emphasizing the need for temporal adaptation in models which need to be deployed over an extended period…
▽ More
Language use changes over time, and this impacts the effectiveness of NLP systems. This phenomenon is even more prevalent in social media data during crisis events where meaning and frequency of word usage may change over the course of days. Contextual language models fail to adapt temporally, emphasizing the need for temporal adaptation in models which need to be deployed over an extended period of time. While existing approaches consider data spanning large periods of time (from years to decades), shorter time spans are critical for crisis data. We quantify temporal degradation for this scenario and propose methods to cope with performance loss by leveraging techniques from domain adaptation. To the best of our knowledge, this is the first effort to explore effects of rapid language change driven by adversarial adaptations, particularly during natural and human-induced disasters. Through extensive experimentation on diverse crisis datasets, we analyze under what conditions our approaches outperform strong baselines while highlighting the current limitations of temporal adaptation methods in scenarios where access to unlabeled data is scarce.
△ Less
Submitted 9 November, 2022; v1 submitted 17 April, 2021;
originally announced April 2021.
-
Ranking Creative Language Characteristics in Small Data Scenarios
Authors:
Julia Siekiera,
Marius Köppel,
Edwin Simpson,
Kevin Stowe,
Iryna Gurevych,
Stefan Kramer
Abstract:
The ability to rank creative natural language provides an important general tool for downstream language understanding and generation. However, current deep ranking models require substantial amounts of labeled data that are difficult and expensive to obtain for different domains, languages and creative characteristics. A recent neural approach, the DirectRanker, promises to reduce the amount of t…
▽ More
The ability to rank creative natural language provides an important general tool for downstream language understanding and generation. However, current deep ranking models require substantial amounts of labeled data that are difficult and expensive to obtain for different domains, languages and creative characteristics. A recent neural approach, the DirectRanker, promises to reduce the amount of training data needed but its application to text isn't fully explored. We therefore adapt the DirectRanker to provide a new deep model for ranking creative language with small data. We compare DirectRanker with a Bayesian approach, Gaussian process preference learning (GPPL), which has previously been shown to work well with sparse data. Our experiments with sparse training data show that while the performance of standard neural ranking approaches collapses with small training datasets, DirectRanker remains effective. We find that combining DirectRanker with GPPL increases performance across different settings by leveraging the complementary benefits of both models. Our combined approach outperforms the previous state-of-the-art on humor and metaphor novelty tasks, increasing Spearman's $ρ$ by 14% and 16% on average.
△ Less
Submitted 23 October, 2020;
originally announced October 2020.
-
How to Best Predict the Daily Number of New Infections of Covid-19
Authors:
Bernd Skiera,
Lukas Jürgensmeier,
Kevin Stowe,
Iryna Gurevych
Abstract:
Knowledge about the daily number of new infections of Covid-19 is important because it is the basis for political decisions resulting in lockdowns and urgent health care measures. We use Germany as an example to illustrate shortcomings of official numbers, which are, at least in Germany, disclosed only with several days of delay and severely underreported on weekends (more than 40%). These shortco…
▽ More
Knowledge about the daily number of new infections of Covid-19 is important because it is the basis for political decisions resulting in lockdowns and urgent health care measures. We use Germany as an example to illustrate shortcomings of official numbers, which are, at least in Germany, disclosed only with several days of delay and severely underreported on weekends (more than 40%). These shortcomings outline an urgent need for alternative data sources. The other widely cited source provided by the Center for Systems Science and Engineering at Johns Hopkins University (JHU) also deviates for Germany on average by 79% from the official numbers. We argue that Google Search and Twitter data should complement official numbers. They predict even better than the original values from Johns Hopkins University and do so several days ahead. These two data sources could also be used in parts of the world where official numbers do not exist or are perceived to be unreliable.
△ Less
Submitted 8 April, 2020;
originally announced April 2020.
-
Metaphoric Paraphrase Generation
Authors:
Kevin Stowe,
Leonardo Ribeiro,
Iryna Gurevych
Abstract:
This work describes the task of metaphoric paraphrase generation, in which we are given a literal sentence and are charged with generating a metaphoric paraphrase. We propose two different models for this task: a lexical replacement baseline and a novel sequence to sequence model, 'metaphor masking', that generates free metaphoric paraphrases. We use crowdsourcing to evaluate our results, as well…
▽ More
This work describes the task of metaphoric paraphrase generation, in which we are given a literal sentence and are charged with generating a metaphoric paraphrase. We propose two different models for this task: a lexical replacement baseline and a novel sequence to sequence model, 'metaphor masking', that generates free metaphoric paraphrases. We use crowdsourcing to evaluate our results, as well as develo** an automatic metric for evaluating metaphoric paraphrases. We show that while the lexical replacement baseline is capable of producing accurate paraphrases, they often lack metaphoricity, while our metaphor masking model excels in generating metaphoric sentences while performing nearly as well with regard to fluency and paraphrase quality.
△ Less
Submitted 28 February, 2020;
originally announced February 2020.
-
Sufficiently Small $\barθ$ in $SU(3)^3 \times S_3$ Unification Model
Authors:
K. Chalut,
H. Cheng,
P. H. Frampton,
K. Stowe,
T. Yoshikawa
Abstract:
Since CP violation in weak decays is successfully described by the KM mechanism, the strong CP problem cannot easily be accommodated. This leads us to reconsider the issue. If the axion and massless up quark are abandoned, we must extend the standard model. Extension to $SU(3)^3 \times S_3$ unification leads to the following situation: {\it if} CP is a high-energy symmetry and the appropriate sy…
▽ More
Since CP violation in weak decays is successfully described by the KM mechanism, the strong CP problem cannot easily be accommodated. This leads us to reconsider the issue. If the axion and massless up quark are abandoned, we must extend the standard model. Extension to $SU(3)^3 \times S_3$ unification leads to the following situation: {\it if} CP is a high-energy symmetry and the appropriate symmetry-breaking hierarchy of scales is in place, then the $\barθ$ parameter of the QCD sub-theory is guaranteed to be sufficiently small. We find $\barθ < 10^{-11}$ while the empirical limit from the neutron electric dipole moment requires only that $\barθ < 1.3 \times 10^{-10}$.
△ Less
Submitted 8 April, 2002; v1 submitted 5 April, 2002;
originally announced April 2002.
