A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation
Authors:
Tae Jun Ham,
Sung Jun Jung,
Seonghak Kim,
Young H. Oh,
Yeonhong Park,
Yoonho Song,
Jung-Hun Park,
Sanghee Lee,
Kyoung Park,
Jae W. Lee,
Deog-Kyoon Jeong
Abstract:
With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network prim…
▽ More
With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.
△ Less
Submitted 21 February, 2020;
originally announced February 2020.
Ink-Jet Printed Graphene Electronics
Authors:
F. Torrisi,
T. Hasan,
W. Wu,
Z. Sun,
A. Lombardo,
T. Kulmala,
G. W. Hshieh,
S. J. Jung,
F. Bonaccorso,
P. J. Paul,
D. P. Chu,
A. C. Ferrari
Abstract:
We demonstrate ink-jet printing as a viable method for large area fabrication of graphene devices. We produce a graphene-based ink by liquid phase exfoliation of graphite in N-Methylpyrrolidone. We use it to print thin-film transistors, with mobilities up to~95cm^2V^(-1)s(-1), as well as transparent and conductive patterns, with~80 % transmittance and~30kOhm/sq sheet resistance. This paves the way…
▽ More
We demonstrate ink-jet printing as a viable method for large area fabrication of graphene devices. We produce a graphene-based ink by liquid phase exfoliation of graphite in N-Methylpyrrolidone. We use it to print thin-film transistors, with mobilities up to~95cm^2V^(-1)s(-1), as well as transparent and conductive patterns, with~80 % transmittance and~30kOhm/sq sheet resistance. This paves the way to all-printed, flexible and transparent graphene devices on arbitrary substrates
△ Less
Submitted 21 November, 2011;
originally announced November 2011.
