Exploring compressibility of transformer based text-to-music (TTM) models
Authors:
Vasileios Moschopoulos,
Thanasis Kotsiopoulos,
Pablo Peso Parada,
Konstantinos Nikiforidis,
Alexandros Stergiadis,
Gerasimos Papakostas,
Md Asif Jalal,
Jisi Zhang,
Anastasios Drosou,
Karthikeyan Saravanan
Abstract:
State-of-the art Text-To-Music (TTM) generative AI models are large and require desktop or server class compute, making them infeasible for deployment on mobile phones. This paper presents an analysis of trade-offs between model compression and generation performance of TTM models. We study compression through knowledge distillation and specific modifications that enable applicability over the var…
▽ More
State-of-the art Text-To-Music (TTM) generative AI models are large and require desktop or server class compute, making them infeasible for deployment on mobile phones. This paper presents an analysis of trade-offs between model compression and generation performance of TTM models. We study compression through knowledge distillation and specific modifications that enable applicability over the various components of the TTM model (encoder, generative model and the decoder). Leveraging these methods we create TinyTTM (89.2M params) that achieves a FAD of 3.66 and KL of 1.32 on MusicBench dataset, better than MusicGen-Small (557.6M params) but not lower than MusicGen-small fine-tuned on MusicBench.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
