-
When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learning
Authors:
Orion Weller,
Kevin Seppi,
Matt Gardner
Abstract:
Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL)…
▽ More
Transfer learning (TL) in natural language processing (NLP) has seen a surge of interest in recent years, as pre-trained models have shown an impressive ability to transfer to novel tasks. Three main strategies have emerged for making use of multiple supervised datasets during fine-tuning: training on an intermediate task before training on the target task (STILTs), using multi-task learning (MTL) to train jointly on a supplementary task and the target task (pairwise MTL), or simply using MTL to train jointly on all available datasets (MTL-ALL). In this work, we compare all three TL methods in a comprehensive analysis on the GLUE dataset suite. We find that there is a simple heuristic for when to use one of these techniques over the other: pairwise MTL is better than STILTs when the target task has fewer instances than the supporting task and vice versa. We show that this holds true in more than 92% of applicable cases on the GLUE dataset and validate this hypothesis with experiments varying dataset size. The simplicity and effectiveness of this heuristic is surprising and warrants additional exploration by the TL community. Furthermore, we find that MTL-ALL is worse than the pairwise methods in almost every case. We hope this study will aid others as they choose between TL methods for NLP tasks.
△ Less
Submitted 17 May, 2022;
originally announced May 2022.
-
Exploring the Relationship Between Algorithm Performance, Vocabulary, and Run-Time in Text Classification
Authors:
Wilson Fearn,
Orion Weller,
Kevin Seppi
Abstract:
Text classification is a significant branch of natural language processing, and has many applications including document classification and sentiment analysis. Unsurprisingly, those who do text classification are concerned with the run-time of their algorithms, many of which depend on the size of the corpus' vocabulary due to their bag-of-words representation. Although many studies have examined t…
▽ More
Text classification is a significant branch of natural language processing, and has many applications including document classification and sentiment analysis. Unsurprisingly, those who do text classification are concerned with the run-time of their algorithms, many of which depend on the size of the corpus' vocabulary due to their bag-of-words representation. Although many studies have examined the effect of preprocessing techniques on vocabulary size and accuracy, none have examined how these methods affect a model's run-time. To fill this gap, we provide a comprehensive study that examines how preprocessing techniques affect the vocabulary size, model performance, and model run-time, evaluating ten techniques over four models and two datasets. We show that some individual methods can reduce run-time with no loss of accuracy, while some combinations of methods can trade 2-5% of the accuracy for up to a 65% reduction of run-time. Furthermore, some combinations of preprocessing techniques can even provide a 15% reduction in run-time while simultaneously improving model accuracy.
△ Less
Submitted 8 April, 2021;
originally announced April 2021.
-
Humor Detection: A Transformer Gets the Last Laugh
Authors:
Orion Weller,
Kevin Seppi
Abstract:
Much previous work has been done in attempting to identify humor in text. In this paper we extend that capability by proposing a new task: assessing whether or not a joke is humorous. We present a novel way of approaching this problem by building a model that learns to identify humorous jokes based on ratings gleaned from Reddit pages, consisting of almost 16,000 labeled instances. Using these rat…
▽ More
Much previous work has been done in attempting to identify humor in text. In this paper we extend that capability by proposing a new task: assessing whether or not a joke is humorous. We present a novel way of approaching this problem by building a model that learns to identify humorous jokes based on ratings gleaned from Reddit pages, consisting of almost 16,000 labeled instances. Using these ratings to determine the level of humor, we then employ a Transformer architecture for its advantages in learning from sentence context. We demonstrate the effectiveness of this approach and show results that are comparable to human performance. We further demonstrate our model's increased capabilities on humor identification problems, such as the previously created datasets for short jokes and puns. These experiments show that this method outperforms all previous work done on these tasks, with an F-measure of 93.1% for the Puns dataset and 98.6% on the Short Jokes dataset.
△ Less
Submitted 31 August, 2019;
originally announced September 2019.
-
Automatic Evaluation of Local Topic Quality
Authors:
Jeffrey Lund,
Piper Armstrong,
Wilson Fearn,
Stephen Cowley,
Courtni Byun,
Jordan Boyd-Graber,
Kevin Seppi
Abstract:
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improve the quality of these token-level topic assignments, have been evaluated only w…
▽ More
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improve the quality of these token-level topic assignments, have been evaluated only with respect to global metrics. We propose a task designed to elicit human judgments of token-level topic assignments. We use a variety of topic model types and parameters and discover that global metrics agree poorly with human assignments.
Since human evaluation is expensive we propose a variety of automated metrics to evaluate topic models at a local level. Finally, we correlate our proposed metrics with human judgments from the task on several datasets. We show that an evaluation based on the percent of topic switches correlates most strongly with human judgment of local topic quality. We suggest that this new metric, which we call consistency, be adopted alongside global metrics such as topic coherence when evaluating new topic models.
△ Less
Submitted 17 May, 2019;
originally announced May 2019.
-
Why Didn't You Listen to Me? Comparing User Control of Human-in-the-Loop Topic Models
Authors:
Varun Kumar,
Alison Smith-Renner,
Leah Findlater,
Kevin Seppi,
Jordan Boyd-Graber
Abstract:
To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments. These approaches extend previously proposed frameworks, including constraints and informed prior-based methods. Users should have a sense of control in HLTM systems, so we propose a control metric to mea…
▽ More
To address the lack of comparative evaluation of Human-in-the-Loop Topic Modeling (HLTM) systems, we implement and evaluate three contrasting HLTM modeling approaches using simulation experiments. These approaches extend previously proposed frameworks, including constraints and informed prior-based methods. Users should have a sense of control in HLTM systems, so we propose a control metric to measure whether refinement operations' results match users' expectations. Informed prior-based methods provide better control than constraints, but constraints yield higher quality topics.
△ Less
Submitted 3 June, 2019; v1 submitted 23 May, 2019;
originally announced May 2019.
-
Cross-referencing using Fine-grained Topic Modeling
Authors:
Jeffrey Lund,
Piper Armstrong,
Wilson Fearn,
Stephen Cowley,
Emily Hales,
Kevin Seppi
Abstract:
Cross-referencing, which links passages of text to other related passages, can be a valuable study aid for facilitating comprehension of a text. However, cross-referencing requires first, a comprehensive thematic knowledge of the entire corpus, and second, a focused search through the corpus specifically to find such useful connections. Due to this, cross-reference resources are prohibitively expe…
▽ More
Cross-referencing, which links passages of text to other related passages, can be a valuable study aid for facilitating comprehension of a text. However, cross-referencing requires first, a comprehensive thematic knowledge of the entire corpus, and second, a focused search through the corpus specifically to find such useful connections. Due to this, cross-reference resources are prohibitively expensive and exist only for the most well-studied texts (e.g. religious texts). We develop a topic-based system for automatically producing candidate cross-references which can be easily verified by human annotators. Our system utilizes fine-grained topic modeling with thousands of highly nuanced and specific topics to identify verse pairs which are topically related. We demonstrate that our system can be cost effective compared to having annotators acquire the expertise necessary to produce cross-reference resources unaided.
△ Less
Submitted 17 May, 2019;
originally announced May 2019.
-
Preprocessor Selection for Machine Learning Pipelines
Authors:
Brandon Schoenfeld,
Christophe Giraud-Carrier,
Mason Poggemann,
Jarom Christensen,
Kevin Seppi
Abstract:
Much of the work in metalearning has focused on classifier selection, combined more recently with hyperparameter optimization, with little concern for data preprocessing. Yet, it is generally well accepted that machine learning applications require not only model building, but also data preprocessing. In other words, practical solutions consist of pipelines of machine learning operators rather tha…
▽ More
Much of the work in metalearning has focused on classifier selection, combined more recently with hyperparameter optimization, with little concern for data preprocessing. Yet, it is generally well accepted that machine learning applications require not only model building, but also data preprocessing. In other words, practical solutions consist of pipelines of machine learning operators rather than single algorithms. Interestingly, our experiments suggest that, on average, data preprocessing hinders accuracy, while the best performing pipelines do actually make use of preprocessors. Here, we conduct an extensive empirical study over a wide range of learning algorithms and preprocessors, and use metalearning to determine when one should make use of preprocessors in ML pipeline design.
△ Less
Submitted 23 October, 2018;
originally announced October 2018.
-
Sculpt, Deploy, Repeat: Fast Prototy** of Interactive Physical Objects
Authors:
Michael Jones,
Kevin Seppi
Abstract:
Building a deployable PhysiComp that merges form and function typically involves a significant investment of time and skill in digital electronics, 3D modeling and mechanical design. We aim to help designers quickly create prototypes by removing technical barriers in that process. Other methods for constructing PhysiComp prototypes either lack fidelity in representing shape and function or are con…
▽ More
Building a deployable PhysiComp that merges form and function typically involves a significant investment of time and skill in digital electronics, 3D modeling and mechanical design. We aim to help designers quickly create prototypes by removing technical barriers in that process. Other methods for constructing PhysiComp prototypes either lack fidelity in representing shape and function or are confined to use in the studio next to a workstation, camera or projector system. Software 3D CAD tools can be used to design the shape but do not provide immediate tactile feedback on fit and feel. In this work, sculpting around 3D printed replicas of electronics combines electronics and form in a fluid design environment. The sculptures are scanned, modified for assembly and then printed on a 3D printer. Using this process, functional prototypes can be created with about 4 hours of focused effort over a day and a half with most of that time spent waiting for the 3D printer. The process lends itself to concurrent exploration of several designs and to rapid iteration. This allows the design process to converge quickly to a PhysiComp that is comfortable and useful.
△ Less
Submitted 18 September, 2017;
originally announced September 2017.
