Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition
Authors:
Timo Lohrenz,
Patrick Schwarz,
Zhengyang Li,
Tim Fingscheidt
Abstract:
Recently, attention-based encoder-decoder (AED) models have shown high performance for end-to-end automatic speech recognition (ASR) across several tasks. Addressing overconfidence in such models, in this paper we introduce the concept of relaxed attention, which is a simple gradual injection of a uniform distribution to the encoder-decoder attention weights during training that is easily implemen…
▽ More
Recently, attention-based encoder-decoder (AED) models have shown high performance for end-to-end automatic speech recognition (ASR) across several tasks. Addressing overconfidence in such models, in this paper we introduce the concept of relaxed attention, which is a simple gradual injection of a uniform distribution to the encoder-decoder attention weights during training that is easily implemented with two lines of code. We investigate the effect of relaxed attention across different AED model architectures and two prominent ASR tasks, Wall Street Journal (WSJ) and Librispeech. We found that transformers trained with relaxed attention outperform the standard baseline models consistently during decoding with external language models. On WSJ, we set a new benchmark for transformer-based end-to-end speech recognition with a word error rate of 3.65%, outperforming state of the art (4.20%) by 13.1% relative, while introducing only a single hyperparameter.
△ Less
Submitted 15 December, 2021; v1 submitted 2 July, 2021;
originally announced July 2021.
Towards the Interactive Effects of Demand Response Participation on Electricity Spot Market Price
Authors:
Saeed Mohajeryami,
Milad Doostan,
Seyedmahdi Moghadasi,
Peter Schwarz
Abstract:
The electricity market is threatened by supply scarcity, which may lead to very sharp price spikes in the spot market. On the other hand, demand-side's activities could effectively mitigate the supply scarcity and absorb most of these shocks and therefore smooth out the price volatility. In this paper, the positive effects of employing demand response programs on the spot market price are investig…
▽ More
The electricity market is threatened by supply scarcity, which may lead to very sharp price spikes in the spot market. On the other hand, demand-side's activities could effectively mitigate the supply scarcity and absorb most of these shocks and therefore smooth out the price volatility. In this paper, the positive effects of employing demand response programs on the spot market price are investigated. A demand-price elasticity based model is used to simulate the customer reaction function in the presence of a real time pricing. The demand achieve by DR program is used to adjust the spot market price by using a price regression model. SAS software is used to run the multiple linear regression model and MATLAB is used to simulate the demand response model. The approach is applied on one week data in summer 2014 of Connecticut in New England ISO. It could be concluded from the results of this study that applying DR program smooths out most of the price spikes in the electricity spot market and considerably reduces the customers' electricity cost.
△ Less
Submitted 22 August, 2016;
originally announced August 2016.
