-
arXiv:2208.10247 [pdf, ps, other]
Generalized Attention Mechanism and Relative Position for Transformer
Abstract: In this paper, we propose generalized attention mechanism (GAM) by first suggesting a new interpretation for self-attention mechanism of Vaswani et al. . Following the interpretation, we provide description for different variants of attention mechanism which together form GAM. Further, we propose a new relative position representation within the framework of GAM. This representation can be easily… ▽ More
Submitted 23 July, 2022; originally announced August 2022.
Comments: 6 pages