-
Lepton-neutron interaction and S-wave low energy parameters
Authors:
Jaume Carbonell,
Tobias Frederico
Abstract:
A lepton-neutron potential in configuration space is obtained. It is based on the Coulomb plus hyperfine interaction Hamiltonian integrated over the neutron charge and magnetic densities. Different parametrisations of the neutron electromagnetic form factors are compared. It is given in the operator form with a central, spin-spin, tensor and spin-orbit terms. The potentials for lowest partial wave…
▽ More
A lepton-neutron potential in configuration space is obtained. It is based on the Coulomb plus hyperfine interaction Hamiltonian integrated over the neutron charge and magnetic densities. Different parametrisations of the neutron electromagnetic form factors are compared. It is given in the operator form with a central, spin-spin, tensor and spin-orbit terms. The potentials for lowest partial waves states are presented. We compute the lepton-neutron lepton ($ln$) low-energy parameters for the S-waves, estimate the zero-energy cross sections for higher angular momentum states, and point out a possible divergence in the partial wave summation due to the spin-orbit potential.
△ Less
Submitted 3 May, 2024;
originally announced May 2024.
-
Comparison of $\bar{\hbox{N}}\hbox{N}$ optical models
Authors:
Jaume Carbonell,
Guillaume Hupin,
Sławomir Wycech
Abstract:
We compare the strong part of the $\bar{\hbox{N}}\hbox{N}$ interaction obtained by the Nijmegen partial wave analysis and the results of some of the most popular $\bar{\hbox{N}}\hbox{N}$ optical potentials in configuration space. We have found severe discrepancies in most of the partial waves, especially above $p_{Lab}$=400 MeV/c where the partial wave analysis displays a resonant-like structure i…
▽ More
We compare the strong part of the $\bar{\hbox{N}}\hbox{N}$ interaction obtained by the Nijmegen partial wave analysis and the results of some of the most popular $\bar{\hbox{N}}\hbox{N}$ optical potentials in configuration space. We have found severe discrepancies in most of the partial waves, especially above $p_{Lab}$=400 MeV/c where the partial wave analysis displays a resonant-like structure in the $^{31}$S$_0$ and $^{33}$P$_0$ waves. Some theoretical difficulties to interpret this behaviour in terms of dynamical resonances are pointed pout and an alternative explanation is suggested. A much better stability is observed in the low energy parameters, apart from some discrepancies due to the presence of near-threshold quasi-bound states in particular waves. Large deviations have also been found between the corresponding potentials, at short and medium-range ($r\gtrsim 1$ fm) distances.
△ Less
Submitted 26 September, 2023;
originally announced September 2023.
-
Scaling of the $^{19}$B two-neutron halo properties close to unitarity
Authors:
Emiko Hiyama,
Rimantas Lazauskas,
Jaume Carbonell,
Tobias Frederico
Abstract:
We explore the description of the bound $^{19}$B isotope in terms of a $^{17}$B+n+n three-body system where the two-body subsystems $^{17}$B+n and neutron-neutron (nn) have virtual states close to the continuum. Dimensionless scaling functions for the root-mean-square (rms) radii are defined and studied for different parameters of the neutron-core potential and considering three different models f…
▽ More
We explore the description of the bound $^{19}$B isotope in terms of a $^{17}$B+n+n three-body system where the two-body subsystems $^{17}$B+n and neutron-neutron (nn) have virtual states close to the continuum. Dimensionless scaling functions for the root-mean-square (rms) radii are defined and studied for different parameters of the neutron-core potential and considering three different models for neutron-neutron interaction. The scaling functions for the radii are rooted on the universal behavior of three-body systems close to the Efimov limit and depend only on dimensionless quantities formed by the two-neutron separation energies and scattering lengths. Our results show in practice the model independence of these scaling functions close to unitarity. We provide an estimation of the different rms relative separation distances between the constituents, as well as of the proton and matter radii.
△ Less
Submitted 25 August, 2022; v1 submitted 24 August, 2022;
originally announced August 2022.
-
Low energy structures in nuclear reactions with 4n in the final state
Authors:
Rimantas Lazauskas,
Emiko Hiyama,
Jaume Carbonell
Abstract:
We present a reaction model to describe the fast removal of the $α$-particle core in $^8$He nucleus with eventual emission of four neutrons. The obtained four neutron energy distributions allows to explain the sharp low energy peak observed by studying the missing mass spectra of four neutrons in [Nature Vol. 606, p. 678], as a consequence of dineutron-dineutron correlations.
We present a reaction model to describe the fast removal of the $α$-particle core in $^8$He nucleus with eventual emission of four neutrons. The obtained four neutron energy distributions allows to explain the sharp low energy peak observed by studying the missing mass spectra of four neutrons in [Nature Vol. 606, p. 678], as a consequence of dineutron-dineutron correlations.
△ Less
Submitted 14 February, 2023; v1 submitted 15 July, 2022;
originally announced July 2022.
-
$^7$H ground state as a $^3$H+4n resonance
Authors:
Emiko Hiyama,
Rimantas Lazauskas,
Jaume Carbonell
Abstract:
We have investigated the possible existence of a $^7$H resonant state, considered as a five-body system consisting of a $^3$H core with four valence neutrons. To this aim, an effective n-$^3$H potential is constructed in order to reproduce the low energy elastic neutron scattering on $^3$H phase shifts and the $^5$H resonant ground state in terms of $^3$H-n-n system. The variational Gaussian Expan…
▽ More
We have investigated the possible existence of a $^7$H resonant state, considered as a five-body system consisting of a $^3$H core with four valence neutrons. To this aim, an effective n-$^3$H potential is constructed in order to reproduce the low energy elastic neutron scattering on $^3$H phase shifts and the $^5$H resonant ground state in terms of $^3$H-n-n system. The variational Gaussian Expansion Method is used to solve the 5-body Schrödinger equation, while the resonant state parameters were estimated by means of the stabilization method. We have not found any sign of a narrow low energy resonance in the vicinity of $^3$H+4n threshold. However, we have identified a very broad structure at $E_R\approx 9$ MeV above this threshold, which corresponds to the $^7$H J$^π$=1/2$^+$ ground state. In the vicinity of this state, we have also identified a broad structure corresponding to the ground state of $^6$H isotope with quantum numbers $J^π=2^-$.
△ Less
Submitted 11 July, 2022;
originally announced July 2022.
-
Protonium annihilation densities in a unitary coupled channel model
Authors:
Emanuel Ydrefors,
Jaume Carbonell
Abstract:
We consider a unitary coupled channel model to describe the low energy proton-antiproton scattering and the lower Coulomb-like protonium states. The existence of deeper quasi-bound states of nuclear nature is found to be a consequence of the experimental data. The properties of these states as well as the protonium annihilation densities are described and the difference with respect to the optical…
▽ More
We consider a unitary coupled channel model to describe the low energy proton-antiproton scattering and the lower Coulomb-like protonium states. The existence of deeper quasi-bound states of nuclear nature is found to be a consequence of the experimental data. The properties of these states as well as the protonium annihilation densities are described and the difference with respect to the optical models description are manifested.
△ Less
Submitted 16 October, 2021;
originally announced October 2021.
-
Antiproton-deuteron hydrogenic states in optical models
Authors:
Rimantas Lazauskas,
Jaume Carbonell
Abstract:
By solving the Faddeev equations for the ppn system, we compute the antiproton-deuteron level shifts and widths for the lowest hydrogenic states as well as the corresponding pd scattering lengths and volumes. The pd annihilation densities are obtained and compared to the nuclear density of deuterium. The validity of the Trueman relation for composite particles is studied. The strong part of NN int…
▽ More
By solving the Faddeev equations for the ppn system, we compute the antiproton-deuteron level shifts and widths for the lowest hydrogenic states as well as the corresponding pd scattering lengths and volumes. The pd annihilation densities are obtained and compared to the nuclear density of deuterium. The validity of the Trueman relation for composite particles is studied. The strong part of NN interaction is described by two different optical models, including the pp-nn coupling and n-p mass difference, while for NN several realistic interactions are used.
△ Less
Submitted 3 August, 2021;
originally announced August 2021.
-
The quest for light multineutron systems
Authors:
F. Miguel Marques,
Jaume Carbonell
Abstract:
The long history of the research concerning the possible existence of bound or resonant states in light multineutron systems, essentially $^3$n and $^4$n, is reviewed. Both the experimental and the theoretical points of view have been considered, with the aim of showing a clear picture of all the different detection and calculation techniques that have been used, with particular emphasis in the is…
▽ More
The long history of the research concerning the possible existence of bound or resonant states in light multineutron systems, essentially $^3$n and $^4$n, is reviewed. Both the experimental and the theoretical points of view have been considered, with the aim of showing a clear picture of all the different detection and calculation techniques that have been used, with particular emphasis in the issues that have been found. Finally, some aspects of the present and future research in this field are discussed.
△ Less
Submitted 22 February, 2021;
originally announced February 2021.
-
Hybrid nature of the abnormal solutions of the Bethe-Salpeter equation in the Wick-Cutkosky model
Authors:
J. Carbonell,
V. A. Karmanov ans H. Sazdjian
Abstract:
In the Wick-Cutkosky model, where two scalar massive constituents interact by means of the exchange of a scalar massless particle, the Bethe-Salpeter equation has solutions of two types, called "normal" and "abnormal". In the non-relativistic limit, the normal solutions correspond to the usual Coulomb spectrum, whereas the abnormal ones do not have non-relativistic counterparts -- they are absent…
▽ More
In the Wick-Cutkosky model, where two scalar massive constituents interact by means of the exchange of a scalar massless particle, the Bethe-Salpeter equation has solutions of two types, called "normal" and "abnormal". In the non-relativistic limit, the normal solutions correspond to the usual Coulomb spectrum, whereas the abnormal ones do not have non-relativistic counterparts -- they are absent in the Schrödinger equation framework. We have studied, in the formalism of the light-front dynamics, the Fock-space content of the abnormal solutions. It turns out that, in contrast to the normal ones, the abnormal states are dominated by the massless exchange particles (by 90 \% or more), what provides a natural explanation of their decoupling from the two-body Schrödinger equation. Assuming that one of the massive constituents is charged, we have calculated the electromagnetic elastic form factors of the normal and abnormal states, as well as the transition form factors. The results on form factors confirm the many-body nature of the abnormal states, as found from the Fock-space analysis. The abnormal solutions have thus properties similar to those of hybrid states, made here essentially of two massive constituents and several or many massless exchange particles. They could also be interpreted as the Abelian scalar analogs of the QCD hybrid states. The question of the validity of the ladder approximation of the model is also examined.
△ Less
Submitted 10 January, 2021;
originally announced January 2021.
-
19B isotope as a 17B-n-n three-body cluster close to unitary limit
Authors:
J. Carbonell,
E. Hiyama,
R. Lazauskas,
F. M. Marqués
Abstract:
We describe 19B in terms of a 17B-n-n three-body system, where the two-body subsystems 17B-n and n-n are unbound (virtual) states close to the unitary limit. The energy of 19B ground state is well reproduced and two low-lying resonances are predicted. Their eventual link with the Efimov physics is discussed. This model can be extended to describe the recently discovered resonant states in 20,21B.
We describe 19B in terms of a 17B-n-n three-body system, where the two-body subsystems 17B-n and n-n are unbound (virtual) states close to the unitary limit. The energy of 19B ground state is well reproduced and two low-lying resonances are predicted. Their eventual link with the Efimov physics is discussed. This model can be extended to describe the recently discovered resonant states in 20,21B.
△ Less
Submitted 24 December, 2020;
originally announced December 2020.
-
Efficient Meta Lifelong-Learning with Limited Memory
Authors:
Zirui Wang,
Sanket Vaibhav Mehta,
Barnabás Póczos,
Jaime Carbonell
Abstract:
Current natural language processing models work well on a single task, yet they often fail to continuously learn new tasks without forgetting previous ones as they are re-trained throughout their lifetime, a challenge known as lifelong learning. State-of-the-art lifelong language learning methods store past examples in episodic memory and replay them at both training and inference time. However, a…
▽ More
Current natural language processing models work well on a single task, yet they often fail to continuously learn new tasks without forgetting previous ones as they are re-trained throughout their lifetime, a challenge known as lifelong learning. State-of-the-art lifelong language learning methods store past examples in episodic memory and replay them at both training and inference time. However, as we show later in our experiments, there are three significant impediments: (1) needing unrealistically large memory module to achieve good performance, (2) suffering from negative transfer, (3) requiring multiple local adaptation steps for each test example that significantly slows down the inference speed. In this paper, we identify three common principles of lifelong learning methods and propose an efficient meta-lifelong framework that combines them in a synergistic fashion. To achieve sample efficiency, our method trains the model in a manner that it learns a better initialization for local adaptation. Extensive experiments on text classification and question answering benchmarks demonstrate the effectiveness of our framework by achieving state-of-the-art performance using merely 1% memory size and narrowing the gap with multi-task learning. We further show that our method alleviates both catastrophic forgetting and negative transfer at the same time.
△ Less
Submitted 6 October, 2020;
originally announced October 2020.
-
Soft Gazetteers for Low-Resource Named Entity Recognition
Authors:
Shruti Rijhwani,
Shuyan Zhou,
Graham Neubig,
Jaime Carbonell
Abstract:
Traditional named entity recognition models use gazetteers (lists of entities) as features to improve performance. Although modern neural network models do not require such hand-crafted features for strong performance, recent work has demonstrated their utility for named entity recognition on English data. However, designing such features for low-resource languages is challenging, because exhausti…
▽ More
Traditional named entity recognition models use gazetteers (lists of entities) as features to improve performance. Although modern neural network models do not require such hand-crafted features for strong performance, recent work has demonstrated their utility for named entity recognition on English data. However, designing such features for low-resource languages is challenging, because exhaustive entity gazetteers do not exist in these languages. To address this problem, we propose a method of "soft gazetteers" that incorporates ubiquitously available information from English knowledge bases, such as Wikipedia, into neural named entity recognition models through cross-lingual entity linking. Our experiments on four low-resource languages show an average improvement of 4 points in F1 score. Code and data are available at https://github.com/neulab/soft-gazetteers.
△ Less
Submitted 4 May, 2020;
originally announced May 2020.
-
Improving Candidate Generation for Low-resource Cross-lingual Entity Linking
Authors:
Shuyan Zhou,
Shruti Rijhwani,
John Wieting,
Jaime Carbonell,
Graham Neubig
Abstract:
Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. The first step of (X)EL is candidate generation, which retrieves a list of plausible candidate entities from the target-language KB for each mention. Approaches based on resources from Wikipedia have proven successful in the realm of relati…
▽ More
Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. The first step of (X)EL is candidate generation, which retrieves a list of plausible candidate entities from the target-language KB for each mention. Approaches based on resources from Wikipedia have proven successful in the realm of relatively high-resource languages (HRL), but these do not extend well to low-resource languages (LRL) with few, if any, Wikipedia pages. Recently, transfer learning methods have been shown to reduce the demand for resources in the LRL by utilizing resources in closely-related languages, but the performance still lags far behind their high-resource counterparts. In this paper, we first assess the problems faced by current entity candidate generation methods for low-resource XEL, then propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios. The methods are simple, but effective: we experiment with our approach on seven XEL datasets and find that they yield an average gain of 16.9% in Top-30 gold candidate recall, compared to state-of-the-art baselines. Our improved model also yields an average gain of 7.9% in in-KB accuracy of end-to-end XEL.
△ Less
Submitted 3 March, 2020;
originally announced March 2020.
-
StructSum: Summarization via Structured Representations
Authors:
Vidhisha Balachandran,
Artidoro Pagnoni,
Jay Yoon Lee,
Dheeraj Rajagopal,
Jaime Carbonell,
Yulia Tsvetkov
Abstract:
Abstractive text summarization aims at compressing the information of a long source document into a rephrased, condensed summary. Despite advances in modeling techniques, abstractive summarization models still suffer from several key challenges: (i) layout bias: they overfit to the style of training corpora; (ii) limited abstractiveness: they are optimized to copying n-grams from the source rather…
▽ More
Abstractive text summarization aims at compressing the information of a long source document into a rephrased, condensed summary. Despite advances in modeling techniques, abstractive summarization models still suffer from several key challenges: (i) layout bias: they overfit to the style of training corpora; (ii) limited abstractiveness: they are optimized to copying n-grams from the source rather than generating novel abstractive summaries; (iii) lack of transparency: they are not interpretable. In this work, we propose a framework based on document-level structure induction for summarization to address these challenges. To this end, we propose incorporating latent and explicit dependencies across sentences in the source document into end-to-end single-document summarization models. Our framework complements standard encoder-decoder summarization models by augmenting them with rich structure-aware document representations based on implicitly learned (latent) structures and externally-derived linguistic (explicit) structures. We show that our summarization framework, trained on the CNN/DM dataset, improves the coverage of content in the source documents, generates more abstractive summaries by generating more novel n-grams, and incorporates interpretable sentence-level structures, while performing on par with standard baselines.
△ Less
Submitted 16 February, 2021; v1 submitted 1 March, 2020;
originally announced March 2020.
-
Description of Four- and Five-Nucleon Systems by Solving Faddeev-Yakubovsky Equations in Configuration Space
Authors:
Rimantas Lazauskas,
Jaume Carbonell
Abstract:
The Faddeev Yakubovsky equations constitute a rigorous formulation of the quantum mechanical N body problem in the framework of non relativistic dynamics. They allow the exact solutions of the Schrodinger equation for bound and scattering states to be obtained. In this review, we will present the general formalism as well as the numerical tools we use to solve Faddeev Yakubovsky equations in confi…
▽ More
The Faddeev Yakubovsky equations constitute a rigorous formulation of the quantum mechanical N body problem in the framework of non relativistic dynamics. They allow the exact solutions of the Schrodinger equation for bound and scattering states to be obtained. In this review, we will present the general formalism as well as the numerical tools we use to solve Faddeev Yakubovsky equations in configuration space. We will consider in detail the description of the four and five nucleon systems based on modern realistic nuclear Hamiltonians. Recent achievements in this domain will be summarized. Some of the still controversial issues related with the nuclear Hamiltonians as well as the numerical methods traditionally employed to solve few nucleon problems will be highlighted.
△ Less
Submitted 14 February, 2020;
originally announced February 2020.
-
Harnessing Code Switching to Transcend the Linguistic Barrier
Authors:
Ashiqur R. KhudaBukhsh,
Shriphani Palakodety,
Jaime G. Carbonell
Abstract:
Code mixing (or code switching) is a common phenomenon observed in social-media content generated by a linguistically diverse user-base. Studies show that in the Indian sub-continent, a substantial fraction of social media posts exhibit code switching. While the difficulties posed by code mixed documents to further downstream analyses are well-understood, lending visibility to code mixed documents…
▽ More
Code mixing (or code switching) is a common phenomenon observed in social-media content generated by a linguistically diverse user-base. Studies show that in the Indian sub-continent, a substantial fraction of social media posts exhibit code switching. While the difficulties posed by code mixed documents to further downstream analyses are well-understood, lending visibility to code mixed documents under certain scenarios may have utility that has been previously overlooked. For instance, a document written in a mixture of multiple languages can be partially accessible to a wider audience; this could be particularly useful if a considerable fraction of the audience lacks fluency in one of the component languages. In this paper, we provide a systematic approach to sample code mixed documents leveraging a polyglot embedding based method that requires minimal supervision. In the context of the 2019 India-Pakistan conflict triggered by the Pulwama terror attack, we demonstrate an untapped potential of harnessing code mixing for human well-being: starting from an existing hostility diffusing \emph{hope speech} classifier solely trained on English documents, code mixed documents are utilized as a bridge to retrieve \emph{hope speech} content written in a low-resource but widely used language - Romanized Hindi. Our proposed pipeline requires minimal supervision and holds promise in substantially reducing web moderation efforts.
△ Less
Submitted 15 June, 2020; v1 submitted 30 January, 2020;
originally announced January 2020.
-
Structure and EM form factors of purely relativistic systems
Authors:
V. A. Karmanov,
J. Carbonell,
H. Sazdjian
Abstract:
The Bethe-Salpeter equation for two massive scalar particles interacting by scalar massless exchange has solutions of two types, which differ from each other by their behavior in the non-relativistic limit: the normal solutions which turn into the Coulomb ones and the "abnormal" solutions. The latter ones have no non-relativistic counterparts and disappear in the non-relativistic limit. We studied…
▽ More
The Bethe-Salpeter equation for two massive scalar particles interacting by scalar massless exchange has solutions of two types, which differ from each other by their behavior in the non-relativistic limit: the normal solutions which turn into the Coulomb ones and the "abnormal" solutions. The latter ones have no non-relativistic counterparts and disappear in the non-relativistic limit. We studied the composition of all these states. It turns out that the normal states, even for large binding energy, are dominated by two massive particles. Whereas, the contribution of the two-body sector into the abnormal states, even for small binding energy, is of the order of 1% only; they are dominated by an indefinite number of the massless particles. The elastic electromagnetic form factors for both normal and abnormal states, as well as the transition ones between them, are calculated.
△ Less
Submitted 2 January, 2020;
originally announced January 2020.
-
Low-energy neutron scattering on light nuclei and $^{19}$B as a $^{17}$B-$n$-$n$ three-body system in the unitary limit
Authors:
Jaume Carbonell,
Emiko Hiyama,
Rimantas Lazauskas,
F. Miguel Marqués
Abstract:
We consider the evolution of the neutron-nucleus scattering length for the lightest nuclei. We show that, when increasing the number of neutrons in the target nucleus, the strong Pauli repulsion is weakened and the balance with the attractive nucleon-nucleon interaction results into a resonant virtual state in $^{18}$B.
We describe $^{19}$B in terms of a $^{17}$B-$n$-$n$ three-body system where…
▽ More
We consider the evolution of the neutron-nucleus scattering length for the lightest nuclei. We show that, when increasing the number of neutrons in the target nucleus, the strong Pauli repulsion is weakened and the balance with the attractive nucleon-nucleon interaction results into a resonant virtual state in $^{18}$B.
We describe $^{19}$B in terms of a $^{17}$B-$n$-$n$ three-body system where the two-body subsystems $^{17}$B-$n$ and $n$-$n$ are unbound (virtual) states close to the unitary limit. The energy of $^{19}$B ground state is well reproduced and two low-lying resonances are predicted. Their eventual link with the Efimov physics is discussed. This model can be extended to describe the recently discovered resonant states in $^{20,21}$B.
△ Less
Submitted 11 December, 2019;
originally announced December 2019.
-
Optimizing Data Usage via Differentiable Rewards
Authors:
Xinyi Wang,
Hieu Pham,
Paul Michel,
Antonios Anastasopoulos,
Jaime Carbonell,
Graham Neubig
Abstract:
To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that "adapts" to its current learning state and estimates the importance of each training data instance. Trainin…
▽ More
To acquire a new skill, humans learn better and faster if a tutor, based on their current knowledge level, informs them of how much attention they should pay to particular content or practice problems. Similarly, a machine learning model could potentially be trained better with a scorer that "adapts" to its current learning state and estimates the importance of each training data instance. Training such an adaptive scorer efficiently is a challenging problem; in order to precisely quantify the effect of a data instance at a given time during the training, it is typically necessary to first complete the entire training process. To efficiently optimize data usage, we propose a reinforcement learning approach called Differentiable Data Selection (DDS). In DDS, we formulate a scorer network as a learnable function of the training data, which can be efficiently updated along with the main model being trained. Specifically, DDS updates the scorer with an intuitive reward signal: it should up-weigh the data that has a similar gradient with a dev set upon which we would finally like to perform well. Without significant computing overhead, DDS delivers strong and consistent improvements over several strong baselines on two very different tasks of machine translation and image classification.
△ Less
Submitted 16 June, 2021; v1 submitted 22 November, 2019;
originally announced November 2019.
-
Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
Authors:
Zirui Wang,
Jiateng Xie,
Ruochen Xu,
Yiming Yang,
Graham Neubig,
Jaime Carbonell
Abstract:
Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and…
▽ More
Learning multilingual representations of text has proven a successful method for many cross-lingual transfer learning tasks. There are two main paradigms for learning such representations: (1) alignment, which maps different independently trained monolingual representations into a shared space, and (2) joint training, which directly learns unified multilingual representations using monolingual and cross-lingual objectives jointly. In this paper, we first conduct direct comparisons of representations learned using both of these methods across diverse cross-lingual tasks. Our empirical results reveal a set of pros and cons for both methods, and show that the relative performance of alignment versus joint training is task-dependent. Stemming from this analysis, we propose a simple and novel framework that combines these two previously mutually-exclusive approaches. Extensive experiments demonstrate that our proposed framework alleviates limitations of both approaches, and outperforms existing methods on the MUSE bilingual lexicon induction (BLI) benchmark. We further show that this framework can generalize to contextualized representations such as Multilingual BERT, and produces state-of-the-art results on the CoNLL cross-lingual NER benchmark.
△ Less
Submitted 17 February, 2020; v1 submitted 10 October, 2019;
originally announced October 2019.
-
Voice for the Voiceless: Active Sampling to Detect Comments Supporting the Rohingyas
Authors:
Shriphani Palakodety,
Ashiqur R. KhudaBukhsh,
Jaime G. Carbonell
Abstract:
The Rohingya refugee crisis is one of the biggest humanitarian crises of modern times with more than 600,000 Rohingyas rendered homeless according to the United Nations High Commissioner for Refugees. While it has received sustained press attention globally, no comprehensive research has been performed on social media pertaining to this large evolving crisis. In this work, we construct a substanti…
▽ More
The Rohingya refugee crisis is one of the biggest humanitarian crises of modern times with more than 600,000 Rohingyas rendered homeless according to the United Nations High Commissioner for Refugees. While it has received sustained press attention globally, no comprehensive research has been performed on social media pertaining to this large evolving crisis. In this work, we construct a substantial corpus of YouTube video comments (263,482 comments from 113,250 users in 5,153 relevant videos) with an aim to analyze the possible role of AI in hel** a marginalized community. Using a novel combination of multiple Active Learning strategies and a novel active sampling strategy based on nearest-neighbors in the comment-embedding space, we construct a classifier that can detect comments defending the Rohingyas among larger numbers of disparaging and neutral ones. We advocate that beyond the burgeoning field of hate-speech detection, automatic detection of \emph{help-speech} can lend voice to the voiceless people and make the internet safer for marginalized communities.
△ Less
Submitted 6 January, 2020; v1 submitted 8 October, 2019;
originally announced October 2019.
-
Hope Speech Detection: A Computational Analysis of the Voice of Peace
Authors:
Shriphani Palakodety,
Ashiqur R. KhudaBukhsh,
Jaime G. Carbonell
Abstract:
The recent Pulwama terror attack (February 14, 2019, Pulwama, Kashmir) triggered a chain of escalating events between India and Pakistan adding another episode to their 70-year-old dispute over Kashmir. The present era of ubiquitious social media has never seen nuclear powers closer to war. In this paper, we analyze this evolving international crisis via a substantial corpus constructed using comm…
▽ More
The recent Pulwama terror attack (February 14, 2019, Pulwama, Kashmir) triggered a chain of escalating events between India and Pakistan adding another episode to their 70-year-old dispute over Kashmir. The present era of ubiquitious social media has never seen nuclear powers closer to war. In this paper, we analyze this evolving international crisis via a substantial corpus constructed using comments on YouTube videos (921,235 English comments posted by 392,460 users out of 2.04 million overall comments by 791,289 users on 2,890 videos). Our main contributions in the paper are three-fold. First, we present an observation that polyglot word-embeddings reveal precise and accurate language clusters, and subsequently construct a document language-identification technique with negligible annotation requirements. We demonstrate the viability and utility across a variety of data sets involving several low-resource languages. Second, we present an analysis on temporal trends of pro-peace and pro-war intent observing that when tensions between the two nations were at their peak, pro-peace intent in the corpus was at its highest point. Finally, in the context of heated discussions in a politically tense situation where two nations are at the brink of a full-fledged war, we argue the importance of automatic identification of user-generated web content that can diffuse hostility and address this prediction task, dubbed \emph{hope-speech detection}.
△ Less
Submitted 24 February, 2020; v1 submitted 11 September, 2019;
originally announced September 2019.
-
Learning Rhyming Constraints using Structured Adversaries
Authors:
Harsh Jhamtani,
Sanket Vaibhav Mehta,
Jaime Carbonell,
Taylor Berg-Kirkpatrick
Abstract:
Existing recurrent neural language models often fail to capture higher-level structure present in text: for example, rhyming patterns present in poetry. Much prior work on poetry generation uses manually defined constraints which are satisfied during decoding using either specialized decoding procedures or rejection sampling. The rhyming constraints themselves are typically not learned by the gene…
▽ More
Existing recurrent neural language models often fail to capture higher-level structure present in text: for example, rhyming patterns present in poetry. Much prior work on poetry generation uses manually defined constraints which are satisfied during decoding using either specialized decoding procedures or rejection sampling. The rhyming constraints themselves are typically not learned by the generator. We propose an alternate approach that uses a structured discriminator to learn a poetry generator that directly captures rhyming constraints in a generative adversarial setup. By causing the discriminator to compare poems based only on a learned similarity matrix of pairs of line ending words, the proposed approach is able to successfully learn rhyming patterns in two different English poetry datasets (Sonnet and Limerick) without explicitly being provided with any phonetic information.
△ Less
Submitted 15 September, 2019;
originally announced September 2019.
-
A Little Annotation does a Lot of Good: A Study in Bootstrap** Low-resource Named Entity Recognizers
Authors:
Aditi Chaudhary,
Jiateng Xie,
Zaid Sheikh,
Graham Neubig,
Jaime G. Carbonell
Abstract:
Most state-of-the-art models for named entity recognition (NER) rely on the availability of large amounts of labeled data, making them challenging to extend to new, lower-resourced languages. However, there are now several proposed approaches involving either cross-lingual transfer learning, which learns from other highly resourced languages, or active learning, which efficiently selects effective…
▽ More
Most state-of-the-art models for named entity recognition (NER) rely on the availability of large amounts of labeled data, making them challenging to extend to new, lower-resourced languages. However, there are now several proposed approaches involving either cross-lingual transfer learning, which learns from other highly resourced languages, or active learning, which efficiently selects effective training data based on model predictions. This paper poses the question: given this recent progress, and limited human annotation, what is the most effective method for efficiently creating high-quality entity recognizers in under-resourced languages? Based on extensive experimentation using both simulated and real human annotation, we find a dual-strategy approach best, starting with a cross-lingual transferred model, then performing targeted annotation of only uncertain entity spans in the target language, minimizing annotator effort. Results demonstrate that cross-lingual transfer is a powerful tool when very little data can be annotated, but an entity-targeted annotation strategy can achieve competitive accuracy quickly, with just one-tenth of training data.
△ Less
Submitted 23 August, 2019;
originally announced August 2019.
-
The Faddeev-Yakubovsky symphony
Authors:
Rimantas Lazauskas,
Jaume Carbonell
Abstract:
We briefly summarize the main steps leading to the Faddeev-Yakubovsky equations in configuration space for N=3, 4 and 5 interacting particles.
We briefly summarize the main steps leading to the Faddeev-Yakubovsky equations in configuration space for N=3, 4 and 5 interacting particles.
△ Less
Submitted 13 August, 2019;
originally announced August 2019.
-
CMU-01 at the SIGMORPHON 2019 Shared Task on Crosslinguality and Context in Morphology
Authors:
Aditi Chaudhary,
Elizabeth Salesky,
Gayatri Bhat,
David R. Mortensen,
Jaime G. Carbonell,
Yulia Tsvetkov
Abstract:
This paper presents the submission by the CMU-01 team to the SIGMORPHON 2019 task 2 of Morphological Analysis and Lemmatization in Context. This task requires us to produce the lemma and morpho-syntactic description of each token in a sequence, for 107 treebanks. We approach this task with a hierarchical neural conditional random field (CRF) model which predicts each coarse-grained feature (eg. PO…
▽ More
This paper presents the submission by the CMU-01 team to the SIGMORPHON 2019 task 2 of Morphological Analysis and Lemmatization in Context. This task requires us to produce the lemma and morpho-syntactic description of each token in a sequence, for 107 treebanks. We approach this task with a hierarchical neural conditional random field (CRF) model which predicts each coarse-grained feature (eg. POS, Case, etc.) independently. However, most treebanks are under-resourced, thus making it challenging to train deep neural models for them. Hence, we propose a multi-lingual transfer training regime where we transfer from multiple related languages that share similar typology.
△ Less
Submitted 23 July, 2019;
originally announced July 2019.
-
Modeling $^{19}$B as a $^{17}$B-n-n three-body system in the unitary limit
Authors:
Emiko Hiyama,
Rimantas Lazauskas,
F. Miguel Marqués,
Jaume Carbonell
Abstract:
We present a model description of the bound $^{17}$B isotope in terms of a $^{17}$B-n-n three-body system where the two-body subsystems $^{17}$B-n and n-n are unbound (virtual) states close to the unitary limit. The $^{17}$B ground state is well described in terms of two-body potentials only, and two low-lying resonances are predicted. Their eventual link with the Efimov physics is discussed. Thi…
▽ More
We present a model description of the bound $^{17}$B isotope in terms of a $^{17}$B-n-n three-body system where the two-body subsystems $^{17}$B-n and n-n are unbound (virtual) states close to the unitary limit. The $^{17}$B ground state is well described in terms of two-body potentials only, and two low-lying resonances are predicted. Their eventual link with the Efimov physics is discussed. This model can be naturally used to describe the recently discovered resonant states in $^{20,21}$B.
△ Less
Submitted 3 July, 2019;
originally announced July 2019.
-
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Authors:
Zhilin Yang,
Zihang Dai,
Yiming Yang,
Jaime Carbonell,
Ruslan Salakhutdinov,
Quoc V. Le
Abstract:
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we p…
▽ More
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
△ Less
Submitted 2 January, 2020; v1 submitted 19 June, 2019;
originally announced June 2019.
-
Domain Adaptation of Neural Machine Translation by Lexicon Induction
Authors:
Junjie Hu,
Mengzhou Xia,
Graham Neubig,
Jaime Carbonell
Abstract:
It has been previously noted that neural machine translation (NMT) is very sensitive to domain shift. In this paper, we argue that this is a dual effect of the highly lexicalized nature of NMT, resulting in failure for sentences with large numbers of unknown words, and lack of supervision for domain-specific words. To remedy this problem, we propose an unsupervised adaptation method which fine-tun…
▽ More
It has been previously noted that neural machine translation (NMT) is very sensitive to domain shift. In this paper, we argue that this is a dual effect of the highly lexicalized nature of NMT, resulting in failure for sentences with large numbers of unknown words, and lack of supervision for domain-specific words. To remedy this problem, we propose an unsupervised adaptation method which fine-tunes a pre-trained out-of-domain NMT model using a pseudo-in-domain corpus. Specifically, we perform lexicon induction to extract an in-domain lexicon, and construct a pseudo-parallel in-domain corpus by performing word-for-word back-translation of monolingual in-domain target sentences. In five domains over twenty pairwise adaptation settings and two model architectures, our method achieves consistent improvements without using any in-domain parallel sentences, improving up to 14 BLEU over unadapted models, and up to 2 BLEU over strong back-translation baselines.
△ Less
Submitted 2 June, 2019;
originally announced June 2019.
-
Bound states of relativistic nature
Authors:
V. A. Karmanov,
J. Carbonell,
H. Sazdjian
Abstract:
Bethe-Salpeter equation, for massless exchange and large fine structure constant $α>π/4$, in addition to the Balmer series, provides another (abnormal) series of energy levels which are not given by the Schrödinger equation. So strong field can be created by a point-like charge $Z>107$. The nuclei with this charge, though available, they are far from to be point-like that weakens the field. Theref…
▽ More
Bethe-Salpeter equation, for massless exchange and large fine structure constant $α>π/4$, in addition to the Balmer series, provides another (abnormal) series of energy levels which are not given by the Schrödinger equation. So strong field can be created by a point-like charge $Z>107$. The nuclei with this charge, though available, they are far from to be point-like that weakens the field. Therefore, the abnormal states of this origin hardly exist.
We analyze the more realistic case of exchange by a massive particle when the large value of coupling constant is typical for the strong interaction. It turns out that this interaction still generates a series of abnormal relativistic states. The properties of these solutions are studied. Their existence in nature seems possible.
△ Less
Submitted 7 March, 2019;
originally announced March 2019.
-
Ab initio calculations of 5H resonant states
Authors:
R. Lazauskas,
E. Hiyama,
J. Carbonell
Abstract:
By solving the 5-body Faddeev-Yakubovsky equations in configuration space with realistic nuclear Hamiltonians we have studied the resonant states of $^5$H isotope. Two different methods, allowing to bypass the exponentially diverging boundary conditions, have been employed providing consistent results. The existence of $^5$H broad J$^π$=1/2$^+$,3/2$^+$,5/2$^+$ states as S-matrix poles has been con…
▽ More
By solving the 5-body Faddeev-Yakubovsky equations in configuration space with realistic nuclear Hamiltonians we have studied the resonant states of $^5$H isotope. Two different methods, allowing to bypass the exponentially diverging boundary conditions, have been employed providing consistent results. The existence of $^5$H broad J$^π$=1/2$^+$,3/2$^+$,5/2$^+$ states as S-matrix poles has been confirmed and compared with the, also calculated, resonant states in $^4$H isotope. We have established that the positions of these resonances only mildly depend on the nuclear interaction model.
△ Less
Submitted 27 February, 2019;
originally announced February 2019.
-
The ARIEL-CMU Systems for LoReHLT18
Authors:
Aditi Chaudhary,
Siddharth Dalmia,
Junjie Hu,
Xinjian Li,
Austin Matthews,
Aldrian Obaja Muis,
Naoki Otani,
Shruti Rijhwani,
Zaid Sheikh,
Nidhi Vyas,
Xinyi Wang,
Jiateng Xie,
Ruochen Xu,
Chunting Zhou,
Peter J. Jansen,
Yiming Yang,
Lori Levin,
Florian Metze,
Teruko Mitamura,
David R. Mortensen,
Graham Neubig,
Eduard Hovy,
Alan W Black,
Jaime Carbonell,
Graham V. Horwood
, et al. (5 additional authors not shown)
Abstract:
This paper describes the ARIEL-CMU submissions to the Low Resource Human Language Technologies (LoReHLT) 2018 evaluations for the tasks Machine Translation (MT), Entity Discovery and Linking (EDL), and detection of Situation Frames in Text and Speech (SF Text and Speech).
This paper describes the ARIEL-CMU submissions to the Low Resource Human Language Technologies (LoReHLT) 2018 evaluations for the tasks Machine Translation (MT), Entity Discovery and Linking (EDL), and detection of Situation Frames in Text and Speech (SF Text and Speech).
△ Less
Submitted 24 February, 2019;
originally announced February 2019.
-
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Authors:
Zihang Dai,
Zhilin Yang,
Yiming Yang,
Jaime Carbonell,
Quoc V. Le,
Ruslan Salakhutdinov
Abstract:
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not…
▽ More
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
△ Less
Submitted 2 June, 2019; v1 submitted 9 January, 2019;
originally announced January 2019.
-
Characterizing and Avoiding Negative Transfer
Authors:
Zirui Wang,
Zihang Dai,
Barnabás Póczos,
Jaime Carbonell
Abstract:
When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task. However, when transferring knowledge from a less related source, it may inversely hurt the target performance, a phenomenon known as negative transfer. Despite its pervasiveness, negative transfer is usually described in an informal manner, lack…
▽ More
When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task. However, when transferring knowledge from a less related source, it may inversely hurt the target performance, a phenomenon known as negative transfer. Despite its pervasiveness, negative transfer is usually described in an informal manner, lacking rigorous definition, careful analysis, or systematic treatment. This paper proposes a formal definition of negative transfer and analyzes three important aspects thereof. Stemming from this analysis, a novel technique is proposed to circumvent negative transfer by filtering out unrelated source data. Based on adversarial networks, the technique is highly generic and can be applied to a wide range of transfer learning algorithms. The proposed approach is evaluated on six state-of-the-art deep transfer methods via experiments on four benchmark datasets with varying levels of difficulty. Empirically, the proposed method consistently improves the performance of all baseline methods and largely avoids negative transfer, even when the source data is degenerate.
△ Less
Submitted 4 October, 2019; v1 submitted 23 November, 2018;
originally announced November 2018.
-
Zero-shot Neural Transfer for Cross-lingual Entity Linking
Authors:
Shruti Rijhwani,
Jiateng Xie,
Graham Neubig,
Jaime Carbonell
Abstract:
Cross-lingual entity linking maps an entity mention in a source language to its corresponding entry in a structured knowledge base that is in a different (target) language. While previous work relies heavily on bilingual lexical resources to bridge the gap between the source and the target languages, these resources are scarce or unavailable for many low-resource languages. To address this problem…
▽ More
Cross-lingual entity linking maps an entity mention in a source language to its corresponding entry in a structured knowledge base that is in a different (target) language. While previous work relies heavily on bilingual lexical resources to bridge the gap between the source and the target languages, these resources are scarce or unavailable for many low-resource languages. To address this problem, we investigate zero-shot cross-lingual entity linking, in which we assume no bilingual lexical resources are available in the source low-resource language. Specifically, we propose pivot-based entity linking, which leverages information from a high-resource "pivot" language to train character-level neural entity linking models that are transferred to the source low-resource language in a zero-shot manner. With experiments on 9 low-resource languages and transfer through a total of 54 languages, we show that our proposed pivot-based framework improves entity linking accuracy 17% (absolute) on average over the baseline systems, for the zero-shot scenario. Further, we also investigate the use of language-universal phonological representations which improves average accuracy (absolute) by 36% when transferring between languages that use different scripts.
△ Less
Submitted 9 November, 2018;
originally announced November 2018.
-
Neural Cross-Lingual Named Entity Recognition with Minimal Resources
Authors:
Jiateng Xie,
Zhilin Yang,
Graham Neubig,
Noah A. Smith,
Jaime Carbonell
Abstract:
For languages with no annotated resources, unsupervised transfer of natural language processing models such as named-entity recognition (NER) from resource-rich languages would be an appealing capability. However, differences in words and word order across languages make it a challenging problem. To improve map** of lexical items across languages, we propose a method that finds translations base…
▽ More
For languages with no annotated resources, unsupervised transfer of natural language processing models such as named-entity recognition (NER) from resource-rich languages would be an appealing capability. However, differences in words and word order across languages make it a challenging problem. To improve map** of lexical items across languages, we propose a method that finds translations based on bilingual word embeddings. To improve robustness to word order differences, we propose to use self-attention, which allows for a degree of flexibility with respect to word order. We demonstrate that these methods achieve state-of-the-art or competitive NER performance on commonly tested languages under a cross-lingual setting, with much lower resource requirements than past approaches. We also evaluate the challenges of applying these methods to Uyghur, a low-resource language.
△ Less
Submitted 11 September, 2018; v1 submitted 29 August, 2018;
originally announced August 2018.
-
Towards Semi-Supervised Learning for Deep Semantic Role Labeling
Authors:
Sanket Vaibhav Mehta,
Jay Yoon Lee,
Jaime Carbonell
Abstract:
Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, the neural models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. The paper proposes a semi-supervised semantic role labeling method that outperforms the state-of-the-art in limited SRL training corpora. The method is based…
▽ More
Neural models have shown several state-of-the-art performances on Semantic Role Labeling (SRL). However, the neural models require an immense amount of semantic-role corpora and are thus not well suited for low-resource languages or domains. The paper proposes a semi-supervised semantic role labeling method that outperforms the state-of-the-art in limited SRL training corpora. The method is based on explicitly enforcing syntactic constraints by augmenting the training objective with a syntactic-inconsistency loss component and uses SRL-unlabeled instances to train a joint-objective LSTM. On CoNLL-2012 English section, the proposed semi-supervised training with 1%, 10% SRL-labeled data and varying amounts of SRL-unlabeled data achieves +1.58, +0.78 F1, respectively, over the pre-trained models that were trained on SOTA architecture with ELMo on the same SRL-labeled data. Additionally, by using the syntactic-inconsistency loss on inference time, the proposed model achieves +3.67, +2.1 F1 over pre-trained model on 1%, 10% SRL-labeled data, respectively.
△ Less
Submitted 28 August, 2018;
originally announced August 2018.
-
Adapting Word Embeddings to New Languages with Morphological and Phonological Subword Representations
Authors:
Aditi Chaudhary,
Chunting Zhou,
Lori Levin,
Graham Neubig,
David R. Mortensen,
Jaime G. Carbonell
Abstract:
Much work in Natural Language Processing (NLP) has been for resource-rich languages, making generalization to new, less-resourced languages challenging. We present two approaches for improving generalization to low-resourced languages by adapting continuous word representations using linguistically motivated subword units: phonemes, morphemes and graphemes. Our method requires neither parallel cor…
▽ More
Much work in Natural Language Processing (NLP) has been for resource-rich languages, making generalization to new, less-resourced languages challenging. We present two approaches for improving generalization to low-resourced languages by adapting continuous word representations using linguistically motivated subword units: phonemes, morphemes and graphemes. Our method requires neither parallel corpora nor bilingual dictionaries and provides a significant gain in performance over previous methods relying on these resources. We demonstrate the effectiveness of our approaches on Named Entity Recognition for four languages, namely Uyghur, Turkish, Bengali and Hindi, of which Uyghur and Bengali are low resource languages, and also perform experiments on Machine Translation. Exploiting subwords with transfer learning gives us a boost of +15.2 NER F1 for Uyghur and +9.7 F1 for Bengali. We also show improvements in the monolingual setting where we achieve (avg.) +3 F1 and (avg.) +1.35 BLEU.
△ Less
Submitted 28 August, 2018;
originally announced August 2018.
-
Towards more Reliable Transfer Learning
Authors:
Zirui Wang,
Jaime Carbonell
Abstract:
Multi-source transfer learning has been proven effective when within-target labeled data is scarce. Previous work focuses primarily on exploiting domain similarities and assumes that source domains are richly or at least comparably labeled. While this strong assumption is never true in practice, this paper relaxes it and addresses challenges related to sources with diverse labeling volume and dive…
▽ More
Multi-source transfer learning has been proven effective when within-target labeled data is scarce. Previous work focuses primarily on exploiting domain similarities and assumes that source domains are richly or at least comparably labeled. While this strong assumption is never true in practice, this paper relaxes it and addresses challenges related to sources with diverse labeling volume and diverse reliability. The first challenge is combining domain similarity and source reliability by proposing a new transfer learning method that utilizes both source-target similarities and inter-source relationships. The second challenge involves pool-based active learning where the oracle is only available in source domains, resulting in an integrated active transfer learning framework that incorporates distribution matching and uncertainty sampling. Extensive experiments on synthetic and two real-world datasets clearly demonstrate the superiority of our proposed methods over several baselines including state-of-the-art transfer learning methods.
△ Less
Submitted 5 July, 2018;
originally announced July 2018.
-
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
Authors:
George Philipp,
Jaime G. Carbonell
Abstract:
For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We introduce the nonlinearity coefficient (NLC), a measurement of the complexity of…
▽ More
For a long time, designing neural architectures that exhibit high performance was considered a dark art that required expert hand-tuning. One of the few well-known guidelines for architecture design is the avoidance of exploding gradients, though even this guideline has remained relatively vague and circumstantial. We introduce the nonlinearity coefficient (NLC), a measurement of the complexity of the function computed by a neural network that is based on the magnitude of the gradient. Via an extensive empirical study, we show that the NLC is a powerful predictor of test error and that attaining a right-sized NLC is essential for optimal performance.
The NLC exhibits a range of intriguing and important properties. It is closely tied to the amount of information gained from computing a single network gradient. It is tied to the error incurred when replacing the nonlinearity operations in the network with linear operations. It is not susceptible to the confounders of multiplicative scaling, additive bias and layer width. It is stable from layer to layer. Hence, we argue that the NLC is the first robust predictor of overfitting in deep networks.
△ Less
Submitted 30 January, 2019; v1 submitted 31 May, 2018;
originally announced June 2018.
-
Equation for the Nakanishi weight function using the inverse Stieltjes transform
Authors:
V. A. Karmanov,
J. Carbonell,
T. Frederico
Abstract:
The bound state Bethe-Salpeter amplitude was expressed by Nakanishi in terms of a smooth weight function g. By using the generalized Stieltjes transform, we derive an integral equation for the Nakanishi function g for a bound state case. It has the standard form g= Vg, where V is a two-dimensional integral operator. The prescription for obtaining the kernel V starting with the kernel K of the Beth…
▽ More
The bound state Bethe-Salpeter amplitude was expressed by Nakanishi in terms of a smooth weight function g. By using the generalized Stieltjes transform, we derive an integral equation for the Nakanishi function g for a bound state case. It has the standard form g= Vg, where V is a two-dimensional integral operator. The prescription for obtaining the kernel V starting with the kernel K of the Bethe-Salpeter equation is given.
△ Less
Submitted 15 February, 2018;
originally announced February 2018.
-
The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions
Authors:
George Philipp,
Dawn Song,
Jaime G. Carbonell
Abstract:
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exp…
▽ More
Whereas it is believed that techniques such as Adam, batch normalization and, more recently, SeLU nonlinearities "solve" the exploding gradient problem, we show that this is not the case in general and that in a range of popular MLP architectures, exploding gradients exist and that they limit the depth to which networks can be effectively trained, both in theory and in practice. We explain why exploding gradients occur and highlight the *collapsing domain problem*, which can arise in architectures that avoid exploding gradients.
ResNets have significantly lower gradients and thus can circumvent the exploding gradient problem, enabling the effective training of much deeper networks. We show this is a direct consequence of the Pythagorean equation. By noticing that *any neural network is a residual network*, we devise the *residual trick*, which reveals that introducing skip connections simplifies the network mathematically, and that this simplicity may be the major cause for their success.
△ Less
Submitted 6 April, 2018; v1 submitted 15 December, 2017;
originally announced December 2017.
-
Nonparametric Neural Networks
Authors:
George Philipp,
Jaime G. Carbonell
Abstract:
Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch. In this paper, we address the problem of automatically finding a good network size during a single training cycle. We introduce *nonparametric neural networks*, a non-probabilistic framework for conducting o…
▽ More
Automatically determining the optimal size of a neural network for a given task without prior information currently requires an expensive global search and training many networks from scratch. In this paper, we address the problem of automatically finding a good network size during a single training cycle. We introduce *nonparametric neural networks*, a non-probabilistic framework for conducting optimization over all possible network sizes and prove its soundness when network growth is limited via an L_p penalty. We train networks under this framework by continuously adding new units while eliminating redundant units via an L_2 penalty. We employ a novel optimization algorithm, which we term *adaptive radial-angular gradient descent* or *AdaRad*, and obtain promising results.
△ Less
Submitted 14 December, 2017;
originally announced December 2017.
-
Asymmetric Variational Autoencoders
Authors:
Guoqing Zheng,
Yiming Yang,
Jaime Carbonell
Abstract:
Variational inference for latent variable models is prevalent in various machine learning problems, typically solved by maximizing the Evidence Lower Bound (ELBO) of the true data likelihood with respect to a variational distribution. However, freely enriching the family of variational distribution is challenging since the ELBO requires variational likelihood evaluations of the latent variables. I…
▽ More
Variational inference for latent variable models is prevalent in various machine learning problems, typically solved by maximizing the Evidence Lower Bound (ELBO) of the true data likelihood with respect to a variational distribution. However, freely enriching the family of variational distribution is challenging since the ELBO requires variational likelihood evaluations of the latent variables. In this paper, we propose a novel framework to enrich the variational family by incorporating auxiliary variables to the variational family. The resulting inference network doesn't require density evaluations for the auxiliary variables and thus complex implicit densities over the auxiliary variables can be constructed by neural networks. It can be shown that the actual variational posterior of the proposed approach is essentially modeling a rich probabilistic mixture of simple variational posterior indexed by auxiliary variables, thus a flexible inference model can be built. Empirical evaluations on several density estimation tasks demonstrates the effectiveness of the proposed method.
△ Less
Submitted 9 July, 2018; v1 submitted 20 November, 2017;
originally announced November 2017.
-
Convolutional Normalizing Flows
Authors:
Guoqing Zheng,
Yiming Yang,
Jaime Carbonell
Abstract:
Bayesian posterior inference is prevalent in various machine learning problems. Variational inference provides one way to approximate the posterior distribution, however its expressive power is limited and so is the accuracy of resulting approximation. Recently, there has a trend of using neural networks to approximate the variational posterior distribution due to the flexibility of neural network…
▽ More
Bayesian posterior inference is prevalent in various machine learning problems. Variational inference provides one way to approximate the posterior distribution, however its expressive power is limited and so is the accuracy of resulting approximation. Recently, there has a trend of using neural networks to approximate the variational posterior distribution due to the flexibility of neural network architecture. One way to construct flexible variational distribution is to warp a simple density into a complex by normalizing flows, where the resulting density can be analytically evaluated. However, there is a trade-off between the flexibility of normalizing flow and computation cost for efficient transformation. In this paper, we propose a simple yet effective architecture of normalizing flows, ConvFlow, based on convolution over the dimensions of random input vector. Experiments on synthetic and real world posterior inference problems demonstrate the effectiveness and efficiency of the proposed method.
△ Less
Submitted 9 July, 2018; v1 submitted 6 November, 2017;
originally announced November 2017.
-
Gradient-based Inference for Networks with Output Constraints
Authors:
Jay Yoon Lee,
Sanket Vaibhav Mehta,
Michael Wick,
Jean-Baptiste Tristan,
Jaime Carbonell
Abstract:
Practitioners apply neural networks to increasingly complex problems in natural language processing, such as syntactic parsing and semantic role labeling that have rich output structures. Many such structured-prediction problems require deterministic constraints on the output values; for example, in sequence-to-sequence syntactic parsing, we require that the sequential outputs encode valid trees.…
▽ More
Practitioners apply neural networks to increasingly complex problems in natural language processing, such as syntactic parsing and semantic role labeling that have rich output structures. Many such structured-prediction problems require deterministic constraints on the output values; for example, in sequence-to-sequence syntactic parsing, we require that the sequential outputs encode valid trees. While hidden units might capture such properties, the network is not always able to learn such constraints from the training data alone, and practitioners must then resort to post-processing. In this paper, we present an inference method for neural networks that enforces deterministic constraints on outputs without performing rule-based post-processing or expensive discrete search. Instead, in the spirit of gradient-based training, we enforce constraints with gradient-based inference (GBI): for each input at test-time, we nudge continuous model weights until the network's unconstrained inference procedure generates an output that satisfies the constraints. We study the efficacy of GBI on three tasks with hard constraints: semantic role labeling, syntactic parsing, and sequence transduction. In each case, the algorithm not only satisfies constraints but improves accuracy, even when the underlying network is state-of-the-art.
△ Less
Submitted 22 April, 2019; v1 submitted 26 July, 2017;
originally announced July 2017.
-
Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network
Authors:
Adams Wei Yu,
Lei Huang,
Qihang Lin,
Ruslan Salakhutdinov,
Jaime Carbonell
Abstract:
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-norma…
▽ More
In this paper, we propose a generic and simple strategy for utilizing stochastic gradient information in optimization. The technique essentially contains two consecutive steps in each iteration: 1) computing and normalizing each block (layer) of the mini-batch stochastic gradient; 2) selecting appropriate step size to update the decision variable (parameter) towards the negative of the block-normalized gradient. We conduct extensive empirical studies on various non-convex neural network optimization problems, including multi-layer perceptron, convolution neural networks and recurrent neural networks. The results indicate the block-normalized gradient can help accelerate the training of neural networks. In particular, we observe that the normalized gradient methods having constant step size with occasionally decay, such as SGD with momentum, have better performance in the deep convolution neural networks, while those with adaptive step sizes, such as Adam, perform better in recurrent neural networks. Besides, we also observe this line of methods can lead to solutions with better generalization properties, which is confirmed by the performance improvement over strong baselines.
△ Less
Submitted 23 April, 2018; v1 submitted 16 July, 2017;
originally announced July 2017.
-
Modelling double charge exchange response function for tetraneutron system
Authors:
Rimantas Lazauskas,
Emiko Hiyama,
Jaume Carbonell
Abstract:
This work is an attempt to model the $4n$ response function of a recent RIKEN experimental study of the double charge exchange $^4$He($^8$He,$^8$Be)$^4$n reaction in order to put in evidence an eventual enhancement mechanism of the zero energy cross section, including a near-threshold resonance. This resonance can indeed be reproduced only by adding to the standard nuclear Hamiltonian an unphysica…
▽ More
This work is an attempt to model the $4n$ response function of a recent RIKEN experimental study of the double charge exchange $^4$He($^8$He,$^8$Be)$^4$n reaction in order to put in evidence an eventual enhancement mechanism of the zero energy cross section, including a near-threshold resonance. This resonance can indeed be reproduced only by adding to the standard nuclear Hamiltonian an unphysically large T=3/2 attractive 3n-force which destroys the neighboring nuclear chart. No other mechanisms like cusps or related structures were found.
△ Less
Submitted 22 May, 2017;
originally announced May 2017.
-
Bound state equation for the Nakanishi weight function
Authors:
J. Carbonell,
T. Frederico,
V. A. Karmanov
Abstract:
The bound state Bethe-Salpeter amplitude was expressed by Nakanishi using a two-dimensional integral representation, in terms of a smooth weight function $g$, which carries the detailed dynamical information. A similar, but one-dimensional, integral representation can be obtained for the Light-Front wave function in terms of the same weight function $g$. By using the generalized Stieltjes transfor…
▽ More
The bound state Bethe-Salpeter amplitude was expressed by Nakanishi using a two-dimensional integral representation, in terms of a smooth weight function $g$, which carries the detailed dynamical information. A similar, but one-dimensional, integral representation can be obtained for the Light-Front wave function in terms of the same weight function $g$. By using the generalized Stieltjes transform, we first obtain $g$ in terms of the Light-Front wave function in the complex plane of its arguments. Next, a new integral equation for the Nakanishi weight function $g$ is derived for a bound state case. It has the standard form $g= N g$, where $N$ is a two-dimensional integral operator. We give the prescription for obtaining the kernel $ N$ starting with the kernel $K$ of the Bethe-Salpeter equation. The derivation is valid for any kernel given by an irreducible Feynman amplitude.
△ Less
Submitted 13 April, 2017;
originally announced April 2017.
-
Co-Clustering for Multitask Learning
Authors:
Keerthiram Murugesan,
Jaime Carbonell,
Yiming Yang
Abstract:
This paper presents a new multitask learning framework that learns a shared representation among the tasks, incorporating both task and feature clusters. The jointly-induced clusters yield a shared latent subspace where task relationships are learned more effectively and more generally than in state-of-the-art multitask learning methods. The proposed general framework enables the derivation of mor…
▽ More
This paper presents a new multitask learning framework that learns a shared representation among the tasks, incorporating both task and feature clusters. The jointly-induced clusters yield a shared latent subspace where task relationships are learned more effectively and more generally than in state-of-the-art multitask learning methods. The proposed general framework enables the derivation of more specific or restricted state-of-the-art multitask methods. The paper also proposes a highly-scalable multitask learning algorithm, based on the new framework, using conjugate gradient descent and generalized \textit{Sylvester equations}. Experimental results on synthetic and benchmark datasets show that the proposed method systematically outperforms several state-of-the-art multitask learning methods.
△ Less
Submitted 2 March, 2017;
originally announced March 2017.
