Showing 1–19 of 19 results for author: Bloodgood, M

Early Forecasting of Text Classification Accuracy and F-Measure with Active Learning

Authors: Thomas Orth, Michael Bloodgood

Abstract: When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stop** methods to minimize the cost of data annotation. An important capability for improving the utility of stop** methods is to effectively forecast the performance of the text classification models. Forecasting can be… ▽ More When creating text classification systems, one of the major bottlenecks is the annotation of training data. Active learning has been proposed to address this bottleneck using stop** methods to minimize the cost of data annotation. An important capability for improving the utility of stop** methods is to effectively forecast the performance of the text classification models. Forecasting can be done through the use of logarithmic models regressed on some portion of the data as learning is progressing. A critical unexplored question is what portion of the data is needed for accurate forecasting. There is a tension, where it is desirable to use less data so that the forecast can be made earlier, which is more useful, versus it being desirable to use more data, so that the forecast can be more accurate. We find that when using active learning it is even more important to generate forecasts earlier so as to make them more useful and not waste annotation effort. We investigate the difference in forecasting difficulty when using accuracy and F-measure as the text classification system performance metrics and we find that F-measure is more difficult to forecast. We conduct experiments on seven text classification datasets in different semantic domains with different characteristics and with three different base machine learning algorithms. We find that forecasting is easiest for decision tree learning, moderate for Support Vector Machines, and most difficult for neural networks. △ Less

Submitted 11 April, 2020; v1 submitted 20 January, 2020; originally announced January 2020.

Comments: 8 pages, 9 figures, 2 tables; published in Proceedings of the IEEE 14th International Conference on Semantic Computing (ICSC), San Diego, CA, USA, pages 77-84, February 2020

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2020 IEEE 14th International Conference on Semantic Computing (ICSC), pages 77-84, San Diego, CA, USA, February 2020. IEEE

The Use of Unlabeled Data versus Labeled Data for Stop** Active Learning for Text Classification

Authors: Garrett Beatty, Ethan Kochis, Michael Bloodgood

Abstract: Annotation of training data is the major bottleneck in the creation of text classification systems. Active learning is a commonly used technique to reduce the amount of training data one needs to label. A crucial aspect of active learning is determining when to stop labeling data. Three potential sources for informing when to stop active learning are an additional labeled set of data, an unlabeled… ▽ More Annotation of training data is the major bottleneck in the creation of text classification systems. Active learning is a commonly used technique to reduce the amount of training data one needs to label. A crucial aspect of active learning is determining when to stop labeling data. Three potential sources for informing when to stop active learning are an additional labeled set of data, an unlabeled set of data, and the training data that is labeled during the process of active learning. To date, no one has compared and contrasted the advantages and disadvantages of stop** methods based on these three information sources. We find that stop** methods that use unlabeled data are more effective than methods that use labeled data. △ Less

Submitted 22 April, 2019; v1 submitted 25 January, 2019; originally announced January 2019.

Comments: 8 pages, 4 figures, 3 tables; published in Proceedings of the IEEE 13th International Conference on Semantic Computing (ICSC), Newport Beach, CA, USA, pages 287-294, January 2019

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 287-294, Newport Beach, CA, USA, January 2019. IEEE

Stop** Active Learning based on Predicted Change of F Measure for Text Classification

Authors: Michael Altschuler, Michael Bloodgood

Abstract: During active learning, an effective stop** method allows users to limit the number of annotations, which is cost effective. In this paper, a new stop** method called Predicted Change of F Measure will be introduced that attempts to provide the users an estimate of how much performance of the model is changing at each iteration. This stop** method can be applied with any base learner. This m… ▽ More During active learning, an effective stop** method allows users to limit the number of annotations, which is cost effective. In this paper, a new stop** method called Predicted Change of F Measure will be introduced that attempts to provide the users an estimate of how much performance of the model is changing at each iteration. This stop** method can be applied with any base learner. This method is useful for reducing the data annotation bottleneck encountered when building text classification systems. △ Less

Submitted 22 April, 2019; v1 submitted 25 January, 2019; originally announced January 2019.

Comments: 8 pages, 12 tables; published in Proceedings of the 2019 IEEE 13th International Conference on Semantic Computing (ICSC), Newport Beach, CA, USA, pages 47-54, January 2019

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2019 IEEE 13th International Conference on Semantic Computing (ICSC), pages 47-54, Newport Beach, CA, USA, January 2019. IEEE

Impact of Batch Size on Stop** Active Learning for Text Classification

Authors: Garrett Beatty, Ethan Kochis, Michael Bloodgood

Abstract: When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a learning curve perspective, it remains an open question how batch size impacts meth… ▽ More When using active learning, smaller batch sizes are typically more efficient from a learning efficiency perspective. However, in practice due to speed and human annotator considerations, the use of larger batch sizes is necessary. While past work has shown that larger batch sizes decrease learning efficiency from a learning curve perspective, it remains an open question how batch size impacts methods for stop** active learning. We find that large batch sizes degrade the performance of a leading stop** method over and above the degradation that results from reduced learning efficiency. We analyze this degradation and find that it can be mitigated by changing the window size parameter of how many past iterations of learning are taken into account when making the stop** decision. We find that when using larger batch sizes, stop** methods are more effective when smaller window sizes are used. △ Less

Submitted 16 May, 2018; v1 submitted 24 January, 2018; originally announced January 2018.

Comments: 2 pages, 1 table; published in Proceedings of the IEEE 12th International Conference on Semantic Computing (ICSC 2018), Laguna Hills, CA, USA, pages 306-307, January 2018

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pages 306-307, Laguna Hills, CA, USA, January 2018. IEEE

Support Vector Machine Active Learning Algorithms with Query-by-Committee versus Closest-to-Hyperplane Selection

Authors: Michael Bloodgood

Abstract: This paper investigates and evaluates support vector machine active learning algorithms for use with imbalanced datasets, which commonly arise in many applications such as information extraction applications. Algorithms based on closest-to-hyperplane selection and query-by-committee selection are combined with methods for addressing imbalance such as positive amplification based on prevalence stat… ▽ More This paper investigates and evaluates support vector machine active learning algorithms for use with imbalanced datasets, which commonly arise in many applications such as information extraction applications. Algorithms based on closest-to-hyperplane selection and query-by-committee selection are combined with methods for addressing imbalance such as positive amplification based on prevalence statistics from initial random samples. Three algorithms (ClosestPA, QBagPA, and QBoostPA) are presented and carefully evaluated on datasets for text classification and relation extraction. The ClosestPA algorithm is shown to consistently outperform the other two in a variety of ways and insights are provided as to why this is the case. △ Less

Submitted 16 May, 2018; v1 submitted 24 January, 2018; originally announced January 2018.

Comments: 8 pages, 7 figures, 3 tables; published in Proceedings of the IEEE 12th International Conference on Semantic Computing (ICSC 2018), Laguna Hills, CA, USA, pages 148-155, January 2018

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pages 148-155, Laguna Hills, CA, USA, January 2018. IEEE

Using Global Constraints and Reranking to Improve Cognates Detection

Authors: Michael Bloodgood, Benjamin Strauss

Abstract: Global constraints and reranking have not been used in cognates detection research to date. We propose methods for using global constraints by performing rescoring of the score matrices produced by state of the art cognates detection systems. Using global constraints to perform rescoring is complementary to state of the art methods for performing cognates detection and results in significant perfo… ▽ More Global constraints and reranking have not been used in cognates detection research to date. We propose methods for using global constraints by performing rescoring of the score matrices produced by state of the art cognates detection systems. Using global constraints to perform rescoring is complementary to state of the art methods for performing cognates detection and results in significant performance improvements beyond current state of the art performance on publicly available datasets with different language pairs and various conditions such as different levels of baseline state of the art performance and different data size conditions, including with more realistic large data size conditions than have been evaluated with in the past. △ Less

Submitted 19 August, 2017; v1 submitted 24 April, 2017; originally announced April 2017.

Comments: 10 pages, 6 figures, 6 tables; published in the Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1983-1992, Vancouver, Canada, July 2017

ACM Class: I.2.6; I.2.7; I.5.1; I.5.4

Journal ref: In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1983-1992, Vancouver, Canada, July 2017. Association for Computational Linguistics

Filtering Tweets for Social Unrest

Authors: Alan Mishler, Kevin Wonus, Wendy Chambers, Michael Bloodgood

Abstract: Since the events of the Arab Spring, there has been increased interest in using social media to anticipate social unrest. While efforts have been made toward automated unrest prediction, we focus on filtering the vast volume of tweets to identify tweets relevant to unrest, which can be provided to downstream users for further analysis. We train a supervised classifier that is able to label Arabic… ▽ More Since the events of the Arab Spring, there has been increased interest in using social media to anticipate social unrest. While efforts have been made toward automated unrest prediction, we focus on filtering the vast volume of tweets to identify tweets relevant to unrest, which can be provided to downstream users for further analysis. We train a supervised classifier that is able to label Arabic language tweets as relevant to unrest with high reliability. We examine the relationship between training data size and performance and investigate ways to optimize the model building process while minimizing cost. We also explore how confidence thresholds can be set to achieve desired levels of performance. △ Less

Submitted 1 April, 2017; v1 submitted 20 February, 2017; originally announced February 2017.

Comments: 7 pages, 8 figures, 3 tables; published in Proceedings of the 2017 IEEE 11th International Conference on Semantic Computing (ICSC), San Diego, CA, USA, pages 17-23, January 2017

ACM Class: H.3.3; I.2.6; I.2.7; I.5.4

Journal ref: In Proceedings of the 2017 IEEE 11th International Conference on Semantic Computing (ICSC), pages 17-23, San Diego, CA, USA, January 2017. IEEE

Data Cleaning for XML Electronic Dictionaries via Statistical Anomaly Detection

Authors: Michael Bloodgood, Benjamin Strauss

Abstract: Many important forms of data are stored digitally in XML format. Errors can occur in the textual content of the data in the fields of the XML. Fixing these errors manually is time-consuming and expensive, especially for large amounts of data. There is increasing interest in the research, development, and use of automated techniques for assisting with data cleaning. Electronic dictionaries are an i… ▽ More Many important forms of data are stored digitally in XML format. Errors can occur in the textual content of the data in the fields of the XML. Fixing these errors manually is time-consuming and expensive, especially for large amounts of data. There is increasing interest in the research, development, and use of automated techniques for assisting with data cleaning. Electronic dictionaries are an important form of data frequently stored in XML format that frequently have errors introduced through a mixture of manual typographical entry errors and optical character recognition errors. In this paper we describe methods for flagging statistical anomalies as likely errors in electronic dictionaries stored in XML format. We describe six systems based on different sources of information. The systems detect errors using various signals in the data including uncommon characters, text length, character-based language models, word-based language models, tied-field length ratios, and tied-field transliteration models. Four of the systems detect errors based on expectations automatically inferred from content within elements of a single field type. We call these single-field systems. Two of the systems detect errors based on correspondence expectations automatically inferred from content within elements of multiple related field types. We call these tied-field systems. For each system, we provide an intuitive analysis of the type of error that it is successful at detecting. Finally, we describe two larger-scale evaluations using crowdsourcing with Amazon's Mechanical Turk platform and using the annotations of a domain expert. The evaluations consistently show that the systems are useful for improving the efficiency with which errors in XML electronic dictionaries can be detected. △ Less

Submitted 11 April, 2016; v1 submitted 25 February, 2016; originally announced February 2016.

Comments: 8 pages, 4 figures, 5 tables; published in Proceedings of the 2016 IEEE Tenth International Conference on Semantic Computing (ICSC), Laguna Hills, CA, USA, pages 79-86, February 2016

ACM Class: I.5.1; I.5.4; G.3; I.2.7; I.2.6

Journal ref: In Proceedings of the 2016 IEEE Tenth International Conference on Semantic Computing (ICSC), pages 79-86, Laguna Hills, CA, USA, February 2016. IEEE

Analysis of Stop** Active Learning based on Stabilizing Predictions

Authors: Michael Bloodgood, John Grothendieck

Abstract: Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stop** active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the… ▽ More Within the natural language processing (NLP) community, active learning has been widely investigated and applied in order to alleviate the annotation bottleneck faced by developers of new NLP systems and technologies. This paper presents the first theoretical analysis of stop** active learning based on stabilizing predictions (SP). The analysis has revealed three elements that are central to the success of the SP method: (1) bounds on Cohen's Kappa agreement between successively trained models impose bounds on differences in F-measure performance of the models; (2) since the stop set does not have to be labeled, it can be made large in practice, hel** to guarantee that the results transfer to previously unseen streams of examples at test/application time; and (3) good (low variance) sample estimates of Kappa between successive models can be obtained. Proofs of relationships between the level of Kappa agreement and the difference in performance between consecutive models are presented. Specifically, if the Kappa agreement between two models exceeds a threshold T (where $T>0$), then the difference in F-measure performance between those models is bounded above by $\frac{4(1-T)}{T}$ in all cases. If precision of the positive conjunction of the models is assumed to be $p$, then the bound can be tightened to $\frac{4(1-T)}{(p+1)T}$. △ Less

Submitted 23 April, 2015; originally announced April 2015.

Comments: 10 pages, 8 tables; appeared in Proceedings of the Seventeenth Conference on Computational Natural Language Learning, August 2013

ACM Class: I.5.1; I.5.4; G.3; I.2.7; I.2.6

Journal ref: In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 10-19, Sofia, Bulgaria, August 2013. Association for Computational Linguistics

Statistical modality tagging from rule-based annotations and crowdsourcing

Authors: Vinodkumar Prabhakaran, Michael Bloodgood, Mona Diab, Bonnie Dorr, Lori Levin, Christine D. Piatko, Owen Rambow, Benjamin Van Durme

Abstract: We explore training an automatic modality tagger. Modality is the attitude that a speaker might have toward an event or state. One of the main hurdles for training a linguistic tagger is gathering training data. This is particularly problematic for training a tagger for modality because modality triggers are sparse for the overwhelming majority of sentences. We investigate an approach to automatic… ▽ More We explore training an automatic modality tagger. Modality is the attitude that a speaker might have toward an event or state. One of the main hurdles for training a linguistic tagger is gathering training data. This is particularly problematic for training a tagger for modality because modality triggers are sparse for the overwhelming majority of sentences. We investigate an approach to automatically training a modality tagger where we first gathered sentences based on a high-recall simple rule-based modality tagger and then provided these sentences to Mechanical Turk annotators for further annotation. We used the resulting set of training data to train a precise modality tagger using a multi-class SVM that delivers good performance. △ Less

Submitted 3 March, 2015; originally announced March 2015.

Comments: 8 pages, 6 tables; appeared in Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, July 2012; In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 57-64, Jeju, Republic of Korea, July 2012. Association for Computational Linguistics

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 57-64, Jeju, Republic of Korea, July 2012. Association for Computational Linguistics

Use of Modality and Negation in Semantically-Informed Syntactic MT

Authors: Kathryn Baker, Michael Bloodgood, Bonnie J. Dorr, Chris Callison-Burch, Nathaniel W. Filardo, Christine Piatko, Lori Levin, Scott Miller

Abstract: This paper describes the resource- and system-building efforts of an eight-week Johns Hopkins University Human Language Technology Center of Excellence Summer Camp for Applied Language Exploration (SCALE-2009) on Semantically-Informed Machine Translation (SIMT). We describe a new modality/negation (MN) annotation scheme, the creation of a (publicly available) MN lexicon, and two automated MN tagge… ▽ More This paper describes the resource- and system-building efforts of an eight-week Johns Hopkins University Human Language Technology Center of Excellence Summer Camp for Applied Language Exploration (SCALE-2009) on Semantically-Informed Machine Translation (SIMT). We describe a new modality/negation (MN) annotation scheme, the creation of a (publicly available) MN lexicon, and two automated MN taggers that we built using the annotation scheme and lexicon. Our annotation scheme isolates three components of modality and negation: a trigger (a word that conveys modality or negation), a target (an action associated with modality or negation) and a holder (an experiencer of modality). We describe how our MN lexicon was semi-automatically produced and we demonstrate that a structure-based MN tagger results in precision around 86% (depending on genre) for tagging of a standard LDC data set. We apply our MN annotation scheme to statistical machine translation using a syntactic framework that supports the inclusion of semantic annotations. Syntactic tags enriched with semantic annotations are assigned to parse trees in the target-language training texts through a process of tree grafting. While the focus of our work is modality and negation, the tree grafting procedure is general and supports other types of semantic information. We exploit this capability by including named entities, produced by a pre-existing tagger, in addition to the MN elements produced by the taggers described in this paper. The resulting system significantly outperformed a linguistically naive baseline model (Hiero), and reached the highest scores yet reported on the NIST 2009 Urdu-English test set. This finding supports the hypothesis that both syntactic and semantic information can improve translation quality. △ Less

Submitted 5 February, 2015; originally announced February 2015.

Comments: 28 pages, 13 figures, 2 tables; appeared in Computational Linguistics, 38(2):411-438, 2012

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: Computational Linguistics, 38(2):411-438, 2012

Rapid Adaptation of POS Tagging for Domain Specific Uses

Authors: John E. Miller, Michael Bloodgood, Manabu Torii, K. Vijay-Shanker

Abstract: Part-of-speech (POS) tagging is a fundamental component for performing natural language tasks such as parsing, information extraction, and question answering. When POS taggers are trained in one domain and applied in significantly different domains, their performance can degrade dramatically. We present a methodology for rapid adaptation of POS taggers to new domains. Our technique is unsupervised… ▽ More Part-of-speech (POS) tagging is a fundamental component for performing natural language tasks such as parsing, information extraction, and question answering. When POS taggers are trained in one domain and applied in significantly different domains, their performance can degrade dramatically. We present a methodology for rapid adaptation of POS taggers to new domains. Our technique is unsupervised in that a manually annotated corpus for the new domain is not necessary. We use suffix information gathered from large amounts of raw text as well as orthographic information to increase the lexical coverage. We present an experiment in the Biological domain where our POS tagger achieves results comparable to POS taggers specifically trained to this domain. △ Less

Submitted 31 October, 2014; originally announced November 2014.

Comments: 2 pages, 2 tables; appeared in Proceedings of the HLT-NAACL BioNLP Workshop on Linking Natural Language and Biology, June 2006

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the HLT-NAACL BioNLP Workshop on Linking Natural Language and Biology, pages 118-119, New York, New York, June 2006. Association for Computational Linguistics

A random forest system combination approach for error detection in digital dictionaries

Authors: Michael Bloodgood, Peng Ye, Paul Rodrigues, David Zajic, David Doermann

Abstract: When digitizing a print bilingual dictionary, whether via optical character recognition or manual entry, it is inevitable that errors are introduced into the electronic version that is created. We investigate automating the process of detecting errors in an XML representation of a digitized print dictionary using a hybrid approach that combines rule-based, feature-based, and language model-based m… ▽ More When digitizing a print bilingual dictionary, whether via optical character recognition or manual entry, it is inevitable that errors are introduced into the electronic version that is created. We investigate automating the process of detecting errors in an XML representation of a digitized print dictionary using a hybrid approach that combines rule-based, feature-based, and language model-based methods. We investigate combining methods and show that using random forests is a promising approach. We find that in isolation, unsupervised methods rival the performance of supervised methods. Random forests typically require training data so we investigate how we can apply random forests to combine individual base methods that are themselves unsupervised without requiring large amounts of training data. Experiments reveal empirically that a relatively small amount of data is sufficient and can potentially be further reduced through specific selection criteria. △ Less

Submitted 30 October, 2014; originally announced October 2014.

Comments: 9 pages, 7 figures, 10 tables; appeared in Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, April 2012

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data, pages 78-86, Avignon, France, April 2012. Association for Computational Linguistics

Bucking the Trend: Large-Scale Cost-Focused Active Learning for Statistical Machine Translation

Authors: Michael Bloodgood, Chris Callison-Burch

Abstract: We explore how to improve machine translation systems by adding more translation data in situations where we already have substantial resources. The main challenge is how to buck the trend of diminishing returns that is commonly encountered. We present an active learning-style data solicitation algorithm to meet this challenge. We test it, gathering annotations via Amazon Mechanical Turk, and find… ▽ More We explore how to improve machine translation systems by adding more translation data in situations where we already have substantial resources. The main challenge is how to buck the trend of diminishing returns that is commonly encountered. We present an active learning-style data solicitation algorithm to meet this challenge. We test it, gathering annotations via Amazon Mechanical Turk, and find that we get an order of magnitude increase in performance rates of improvement. △ Less

Submitted 21 October, 2014; originally announced October 2014.

Comments: 11 pages, 14 figures; appeared in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 2010

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 854-864, Uppsala, Sweden, July 2010. Association for Computational Linguistics

Using Mechanical Turk to Build Machine Translation Evaluation Sets

Authors: Michael Bloodgood, Chris Callison-Burch

Abstract: Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon's Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professi… ▽ More Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon's Mechanical Turk (MTurk) to make MT test sets cheaply. We find that MTurk can be used to make test sets much cheaper than professionally-produced test sets. More importantly, in experiments with multiple MT systems, we find that the MTurk-produced test sets yield essentially the same conclusions regarding system performance as the professionally-produced test sets yield. △ Less

Submitted 20 October, 2014; originally announced October 2014.

Comments: 4 pages, 2 tables; appeared in Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, June 2010

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, pages 208-211, Los Angeles, California, June 2010. Association for Computational Linguistics

Semantically-Informed Syntactic Machine Translation: A Tree-Grafting Approach

Authors: Kathryn Baker, Michael Bloodgood, Chris Callison-Burch, Bonnie J. Dorr, Nathaniel W. Filardo, Lori Levin, Scott Miller, Christine Piatko

Abstract: We describe a unified and coherent syntactic framework for supporting a semantically-informed syntactic approach to statistical machine translation. Semantically enriched syntactic tags assigned to the target-language training texts improved translation quality. The resulting system significantly outperformed a linguistically naive baseline model (Hiero), and reached the highest scores yet reporte… ▽ More We describe a unified and coherent syntactic framework for supporting a semantically-informed syntactic approach to statistical machine translation. Semantically enriched syntactic tags assigned to the target-language training texts improved translation quality. The resulting system significantly outperformed a linguistically naive baseline model (Hiero), and reached the highest scores yet reported on the NIST 2009 Urdu-English translation task. This finding supports the hypothesis (posed by many researchers in the MT community, e.g., in DARPA GALE) that both syntactic and semantic information are critical for improving translation quality---and further demonstrates that large gains can be achieved for low-resource languages with different word order than English. △ Less

Submitted 24 September, 2014; originally announced September 2014.

Comments: 10 pages, 7 figures, 3 tables; appeared in Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas (AMTA), October 2010

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas (AMTA), Denver, Colorado, October 2010

A Method for Stop** Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stop**

Authors: Michael Bloodgood, K. Vijay-Shanker

Abstract: A survey of existing methods for stop** active learning (AL) reveals the needs for methods that are: more widely applicable; more aggressive in saving annotations; and more stable across changing datasets. A new method for stop** AL based on stabilizing predictions is presented that addresses these needs. Furthermore, stop** methods are required to handle a broad range of different annotatio… ▽ More A survey of existing methods for stop** active learning (AL) reveals the needs for methods that are: more widely applicable; more aggressive in saving annotations; and more stable across changing datasets. A new method for stop** AL based on stabilizing predictions is presented that addresses these needs. Furthermore, stop** methods are required to handle a broad range of different annotation/performance tradeoff valuations. Despite this, the existing body of work is dominated by conservative methods with little (if any) attention paid to providing users with control over the behavior of stop** methods. The proposed method is shown to fill a gap in the level of aggressiveness available for stop** AL and supports providing users with control over stop** behavior. △ Less

Submitted 17 September, 2014; originally announced September 2014.

Comments: 9 pages, 3 figures, 5 tables; appeared in Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), June 2009

ACM Class: I.2.6; I.2.7; I.5.1; I.5.4; G.3

Journal ref: In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 39-47, Boulder, Colorado, June 2009. Association for Computational Linguistics

Taking into Account the Differences between Actively and Passively Acquired Data: The Case of Active Learning with Support Vector Machines for Imbalanced Datasets

Authors: Michael Bloodgood, K. Vijay-Shanker

Abstract: Actively sampled data can have very different characteristics than passively sampled data. Therefore, it's promising to investigate using different inference procedures during AL than are used during passive learning (PL). This general idea is explored in detail for the focused case of AL with cost-weighted SVMs for imbalanced data, a situation that arises for many HLT tasks. The key idea behind t… ▽ More Actively sampled data can have very different characteristics than passively sampled data. Therefore, it's promising to investigate using different inference procedures during AL than are used during passive learning (PL). This general idea is explored in detail for the focused case of AL with cost-weighted SVMs for imbalanced data, a situation that arises for many HLT tasks. The key idea behind the proposed InitPA method for addressing imbalance is to base cost models during AL on an estimate of overall corpus imbalance computed via a small unbiased sample rather than the imbalance in the labeled training data, which is the leading method used during PL. △ Less

Submitted 16 September, 2014; originally announced September 2014.

Comments: 4 pages, 5 figures; appeared in Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 137-140, Boulder, Colorado, June 2009. Association for Computational Linguistics

ACM Class: I.2.6; I.2.7; I.5.1; I.5.4

Journal ref: Proceedings of HLT: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Short Papers, pages 137-140, Boulder, Colorado, June 2009. Association for Computational Linguistics

An Approach to Reducing Annotation Costs for BioNLP

Authors: Michael Bloodgood, K. Vijay-Shanker

Abstract: There is a broad range of BioNLP tasks for which active learning (AL) can significantly reduce annotation costs and a specific AL algorithm we have developed is particularly effective in reducing annotation costs for these tasks. We have previously developed an AL algorithm called ClosestInitPA that works best with tasks that have the following characteristics: redundancy in training material, bur… ▽ More There is a broad range of BioNLP tasks for which active learning (AL) can significantly reduce annotation costs and a specific AL algorithm we have developed is particularly effective in reducing annotation costs for these tasks. We have previously developed an AL algorithm called ClosestInitPA that works best with tasks that have the following characteristics: redundancy in training material, burdensome annotation costs, Support Vector Machines (SVMs) work well for the task, and imbalanced datasets (i.e. when set up as a binary classification problem, one class is substantially rarer than the other). Many BioNLP tasks have these characteristics and thus our AL algorithm is a natural approach to apply to BioNLP tasks. △ Less

Submitted 12 September, 2014; originally announced September 2014.

Comments: 2 pages, 1 figure, 5 tables; appeared in Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing at ACL (Association for Computational Linguistics) 2008

ACM Class: I.2.7; I.2.6; I.5.1; I.5.4

Journal ref: In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, pages 104-105, Columbus, Ohio, June 2008. Association for Computational Linguistics