ProsAudit, a prosodic benchmark for self-supervised speech models
Authors:
Maureen de Seyssel,
Marvin Lavechin,
Hadrien Titeux,
Arthur Thomas,
Gwendal Virlet,
Andrea Santos Revilla,
Guillaume Wisniewski,
Bogdan Ludusan,
Emmanuel Dupoux
Abstract:
We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, and an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inser…
▽ More
We present ProsAudit, a benchmark in English to assess structural prosodic knowledge in self-supervised learning (SSL) speech models. It consists of two subtasks, their corresponding metrics, and an evaluation dataset. In the protosyntax task, the model must correctly identify strong versus weak prosodic boundaries. In the lexical task, the model needs to correctly distinguish between pauses inserted between words and within words. We also provide human evaluation scores on this benchmark. We evaluated a series of SSL models and found that they were all able to perform above chance on both tasks, even when evaluated on an unseen language. However, non-native models performed significantly worse than native ones on the lexical task, highlighting the importance of lexical knowledge in this task. We also found a clear effect of size with models trained on more data performing better in the two subtasks.
△ Less
Submitted 1 June, 2023; v1 submitted 23 February, 2023;
originally announced February 2023.
