Computer Science > Computation and Language
[Submitted on 30 Aug 2019 (v1), last revised 6 May 2020 (this version, v2)]
Title:Encoders Help You Disambiguate Word Senses in Neural Machine Translation
View PDFAbstract:Neural machine translation (NMT) has achieved new state-of-the-art performance in translating ambiguous words. However, it is still unclear which component dominates the process of disambiguation. In this paper, we explore the ability of NMT encoders and decoders to disambiguate word senses by evaluating hidden states and investigating the distributions of self-attention. We train a classifier to predict whether a translation is correct given the representation of an ambiguous noun. We find that encoder hidden states outperform word embeddings significantly which indicates that encoders adequately encode relevant information for disambiguation into hidden states. Decoders could provide further relevant information for disambiguation. Moreover, the attention weights and attention entropy show that self-attention can detect ambiguous nouns and distribute more attention to the context. Note that this is a revised version. The content related to decoder hidden states has been updated.
Submission history
From: Gongbo Tang [view email][v1] Fri, 30 Aug 2019 15:00:19 UTC (118 KB)
[v2] Wed, 6 May 2020 15:13:57 UTC (123 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.