Showing 1–1 of 1 results for author: Banaszczak, J
-
WER we are and WER we think we are
Authors:
Piotr Szymański,
Piotr Żelasko,
Mikolaj Morzy,
Adrian Szymczak,
Marzena Żyła-Hoppe,
Joanna Banaszczak,
Lukasz Augustyniak,
Jan Mizgajski,
Yishay Carmiel
Abstract:
Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR s…
▽ More
Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB'05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.
△ Less
Submitted 7 October, 2020;
originally announced October 2020.
