Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclaw
Authors:
Jan Chorowski,
Grzegorz Ciesielski,
Jarosław Dzikowski,
Adrian Łańcucki,
Ricard Marxer,
Mateusz Opala,
Piotr Pusz,
Paweł Rychlikowski,
Michał Stypułkowski
Abstract:
We present a number of low-resource approaches to the tasks of the Zero Resource Speech Challenge 2021. We build on the unsupervised representations of speech proposed by the organizers as a baseline, derived from CPC and clustered with the k-means algorithm. We demonstrate that simple methods of refining those representations can narrow the gap, or even improve upon the solutions which use a high…
▽ More
We present a number of low-resource approaches to the tasks of the Zero Resource Speech Challenge 2021. We build on the unsupervised representations of speech proposed by the organizers as a baseline, derived from CPC and clustered with the k-means algorithm. We demonstrate that simple methods of refining those representations can narrow the gap, or even improve upon the solutions which use a high computational budget. The results lead to the conclusion that the CPC-derived representations are still too noisy for training language models, but stable enough for simpler forms of pattern matching and retrieval.
△ Less
Submitted 22 June, 2021;
originally announced June 2021.
Aligned Contrastive Predictive Coding
Authors:
Jan Chorowski,
Grzegorz Ciesielski,
Jarosław Dzikowski,
Adrian Łańcucki,
Ricard Marxer,
Mateusz Opala,
Piotr Pusz,
Paweł Rychlikowski,
Michał Stypułkowski
Abstract:
We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned. In this way, the prediction netw…
▽ More
We investigate the possibility of forcing a self-supervised model trained using a contrastive predictive loss to extract slowly varying latent representations. Rather than producing individual predictions for each of the future representations, the model emits a sequence of predictions shorter than that of the upcoming representations to which they will be aligned. In this way, the prediction network solves a simpler task of predicting the next symbols, but not their exact timing, while the encoding network is trained to produce piece-wise constant latent codes. We evaluate the model on a speech coding task and demonstrate that the proposed Aligned Contrastive Predictive Coding (ACPC) leads to higher linear phone prediction accuracy and lower ABX error rates, while being slightly faster to train due to the reduced number of prediction heads.
△ Less
Submitted 22 June, 2021; v1 submitted 24 April, 2021;
originally announced April 2021.
