Showing 1–11 of 11 results for author: Setiawan, H

Automating Behavioral Testing in Machine Translation

Authors: Javier Ferrando, Matthias Sperber, Hendra Setiawan, Dominic Telaar, Saša Hasan

Abstract: Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine Translation (MT) is currently restricted to largely handcrafted tests covering a limited range of capabilities and languages. To address this limitation, we propose to use Large La… ▽ More Behavioral testing in NLP allows fine-grained evaluation of systems by examining their linguistic capabilities through the analysis of input-output behavior. Unfortunately, existing work on behavioral testing in Machine Translation (MT) is currently restricted to largely handcrafted tests covering a limited range of capabilities and languages. To address this limitation, we propose to use Large Language Models (LLMs) to generate a diverse set of source sentences tailored to test the behavior of MT models in a range of situations. We can then verify whether the MT model exhibits the expected behavior through matching candidate sets that are also generated using LLMs. Our approach aims to make behavioral testing of MT systems practical while requiring only minimal human effort. In our experiments, we apply our proposed evaluation framework to assess multiple available MT systems, revealing that while in general pass-rates follow the trends observable from traditional accuracy-based metrics, our method was able to uncover several important differences and potential bugs that go unnoticed when relying only on accuracy. △ Less

Submitted 2 November, 2023; v1 submitted 5 September, 2023; originally announced September 2023.

One Wide Feedforward is All You Need

Authors: Telmo Pessoa Pires, António V. Lopes, Yannick Assogba, Hendra Setiawan

Abstract: The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model's parameters, it is hi… ▽ More The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model's parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big. △ Less

Submitted 21 October, 2023; v1 submitted 4 September, 2023; originally announced September 2023.

Comments: Accepted at WMT23 (EMNLP 2023)

Accurate Knowledge Distillation with n-best Reranking

Authors: Hendra Setiawan

Abstract: We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudo-labels for student model's training data from top n-best hypotheses and leverage a diverse set of models with different inductive biases, objective functions or architectures, including some publicly-available large language models, to pick the highest-quality hypothe… ▽ More We propose utilizing n-best reranking to enhance Sequence-Level Knowledge Distillation (Kim and Rush, 2016) where we extract pseudo-labels for student model's training data from top n-best hypotheses and leverage a diverse set of models with different inductive biases, objective functions or architectures, including some publicly-available large language models, to pick the highest-quality hypotheses as labels. The effectiveness of our proposal is validated through experiments on the WMT'21 German-English and Chinese-English translation tasks. Our results demonstrate that utilizing pseudo-labels generated by our n-best reranker leads to a significantly more accurate student model. In fact, our best student model achieves comparable accuracy to a large translation model from (Tran et al., 2021) with 4.7 billion parameters, while having two orders of magnitude fewer parameters. △ Less

Submitted 12 June, 2024; v1 submitted 19 May, 2023; originally announced May 2023.

Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data

Authors: Mozhdeh Gheini, Tatiana Likhomanenko, Matthias Sperber, Hendra Setiawan

Abstract: Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an ab… ▽ More Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points. △ Less

Submitted 19 December, 2022; originally announced December 2022.

End-to-End Speech Translation for Code Switched Speech

Authors: Orion Weller, Matthias Sperber, Telmo Pires, Hendra Setiawan, Christian Gollan, Dominic Telaar, Matthias Paulik

Abstract: Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and transla… ▽ More Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation. To evaluate model performance on this task, we create a novel ST corpus derived from existing public data sets. We explore various ST architectures across two dimensions: cascaded (transcribe then translate) vs end-to-end (jointly transcribe and translate) and unidirectional (source -> target) vs bidirectional (source <-> target). We show that our ST architectures, and especially our bidirectional end-to-end architecture, perform well on CS speech, even when no CS training data is used. △ Less

Submitted 11 April, 2022; originally announced April 2022.

Comments: Accepted to Findings of ACL 2022

Consistent Transcription and Translation of Speech

Authors: Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, Matthias Paulik

Abstract: The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applicati… ▽ More The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. While high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy. △ Less

Submitted 28 August, 2020; v1 submitted 24 July, 2020; originally announced July 2020.

Comments: Accepted at TACL (pre-MIT Press publication version); added dataset link

Variational Neural Machine Translation with Normalizing Flows

Authors: Hendra Setiawan, Matthias Sperber, Udhay Nallasamy, Matthias Paulik

Abstract: Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the… ▽ More Variational Neural Machine Translation (VNMT) is an attractive framework for modeling the generation of target translations, conditioned not only on the source sentence but also on some latent random variables. The latent variable modeling may introduce useful statistical dependencies that can improve translation accuracy. Unfortunately, learning informative latent variables is non-trivial, as the latent space can be prohibitively large, and the latent codes are prone to be ignored by many translation models at training time. Previous works impose strong assumptions on the distribution of the latent code and limit the choice of the NMT architecture. In this paper, we propose to apply the VNMT framework to the state-of-the-art Transformer and introduce a more flexible approximate posterior based on normalizing flows. We demonstrate the efficacy of our proposal under both in-domain and out-of-domain conditions, significantly outperforming strong baselines. △ Less

Submitted 28 May, 2020; originally announced May 2020.

Comments: To appear in 2020 Association for Computational Linguistics (ACL) as a short paper

The S$π$RIT Time Projection Chamber

Authors: J. Barney, J. Estee, W. G. Lynch, T. Isobe, G. Jhang, M. Kurata-Nishimura, A. B. McIntosh, T. Murakami, R. Shane, S. Tangwancharoen, M. B. Tsang, G. Cerizza, M. Kaneko, J. W. Lee, C. Y. Tsang, R. Wang, C. Anderson, H. Baba, Z. Chajecki, M. Famiano, R. Hodges-Showalter, B. Hong, T. Kobayashi, P. Lasko, J. Łukasik , et al. (15 additional authors not shown)

Abstract: The SAMURAI Pion Reconstruction and Ion-Tracker Time Projection Chamber (S$π$RIT TPC) was designed to enable measurements of heavy ion collisions with the SAMURAI spectrometer at the RIKEN Radioactive Isotope Beam Factory and provide constraints on the Equation of State of neutron-rich nuclear matter. The S$π$RIT TPC has a 50.5 cm drift length and an 86.4 cm $\times$ 134.4 cm pad plane with 12,096… ▽ More The SAMURAI Pion Reconstruction and Ion-Tracker Time Projection Chamber (S$π$RIT TPC) was designed to enable measurements of heavy ion collisions with the SAMURAI spectrometer at the RIKEN Radioactive Isotope Beam Factory and provide constraints on the Equation of State of neutron-rich nuclear matter. The S$π$RIT TPC has a 50.5 cm drift length and an 86.4 cm $\times$ 134.4 cm pad plane with 12,096 pads that are equipped with the Generic Electronics for TPCs readout electronics. The S$π$RIT TPC allows excellent reconstruction of particles and provides isotopic resolution for pions and other light charged particles across a wide range of energy losses and momenta. Details of the S$π$RIT TPC are presented, along with discussion of the TPC performance based on cosmic ray and experimental data. △ Less

Submitted 21 May, 2020; originally announced May 2020.

Comments: 12 pages, 20 figures

On Determining Dead Layer and Detector Thicknesses for a Position-Sensitive Silicon Detector

Authors: J. Manfredi, Jenny Lee, W. G. Lynch, C. Y. Niu, M. B. Tsang, C. Anderson, J. Barney, K. W. Brown, Z. Chajecki, K. P. Chan, G. Chen, J. Estee, Z. Li, C. Pruitt, A. M. Rogers, A. Sanetullaev, H. Setiawan, R. Showalter, C. Y. Tsang, J. R. Winkelbauer, Z. Xiao, Z. Xu

Abstract: In this work, two particular properties of the position-sensitive, thick silicon detectors (known as the "E" detectors) in the High Resolution Array (HiRA) are investigated: the thickness of the dead layer on the front of the detector, and the overall thickness of the detector itself. The dead layer thickness for each E detector in HiRA is extracted using a measurement of alpha particles emitted f… ▽ More In this work, two particular properties of the position-sensitive, thick silicon detectors (known as the "E" detectors) in the High Resolution Array (HiRA) are investigated: the thickness of the dead layer on the front of the detector, and the overall thickness of the detector itself. The dead layer thickness for each E detector in HiRA is extracted using a measurement of alpha particles emitted from a $^{212}$Pb pin source placed close to the detector surface. This procedure also allows for energy calibrations of the E detectors, which are otherwise inaccessible for alpha source calibration as each one is sandwiched between two other detectors. The E detector thickness is obtained from a combination of elastically scattered protons and an energy-loss calculation method. Results from these analyses agree with values provided by the manufacturer. △ Less

Submitted 18 January, 2018; originally announced January 2018.

Comments: Accepted for publication in Nuclear Instruments and Methods in Physics Research

Pion Production in Rare Isotope Collisions

Authors: M. B. Tsang, J. Estee, H. Setiawan, W. G. Lynch, J. Barney, M. B. Chen, G. Cerizza, P. Danielewicz, J. Hong, P. Morfouace, R. Shane, S. Tangwancharoen, K. Zhu, T. Isobe, M. Kurata-Nishimura, J. Lukasik, T. Murakami, the SπRIT collaboration

Abstract: Pion energy spectra are presented for central collisions of neutron-rich 132Sn+124Sn and neutron-deficient 108Sn+112Sn systems using simulations with Boltzmann-Uehling-Uhlenbeck transport model. These calculations, which incorporate isospin-dependent mean field potentials for relevant baryons and mesons, display a sensitivity to the pion spectra that could allow significant constraints on the dens… ▽ More Pion energy spectra are presented for central collisions of neutron-rich 132Sn+124Sn and neutron-deficient 108Sn+112Sn systems using simulations with Boltzmann-Uehling-Uhlenbeck transport model. These calculations, which incorporate isospin-dependent mean field potentials for relevant baryons and mesons, display a sensitivity to the pion spectra that could allow significant constraints on the density dependence of the symmetry energy and its mean field potential at supra-saturation densities. The predicted sensitivity increases with the isospin asymmetry of the total system and decreases with incident energy. △ Less

Submitted 12 March, 2017; v1 submitted 20 December, 2016; originally announced December 2016.

Journal ref: Phys. Rev. C 95, 044614 (2017)

Statistical Machine Translation Features with Multitask Tensor Networks

Authors: Hendra Setiawan, Zhongqiang Huang, Jacob Devlin, Thomas Lamar, Rabih Zbib, Richard Schwartz, John Makhoul

Abstract: We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non-local translation phenomena. Second, we augment the architecture of the neural network with tensor layers that capture important higher-order interaction among th… ▽ More We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non-local translation phenomena. Second, we augment the architecture of the neural network with tensor layers that capture important higher-order interaction among the network units. Third, we apply multitask learning to estimate the neural network parameters jointly. Each of our proposed methods results in significant improvements that are complementary. The overall improvement is +2.7 and +1.8 BLEU points for Arabic-English and Chinese-English translation over a state-of-the-art system that already includes neural network features. △ Less

Submitted 1 June, 2015; originally announced June 2015.

Comments: 11 pages (9 content + 2 references), 2 figures, accepted to ACL 2015 as a long paper