-
Improving Diffusion Models's Data-Corruption Resistance using Scheduled Pseudo-Huber Loss
Authors:
Artem Khrapov,
Vadim Popov,
Tasnima Sadekova,
Assel Yermekova,
Mikhail Kudinov
Abstract:
Diffusion models are known to be vulnerable to outliers in training data. In this paper we study an alternative diffusion loss function, which can preserve the high quality of generated data like the original squared $L_{2}$ loss while at the same time being robust to outliers. We propose to use pseudo-Huber loss function with a time-dependent parameter to allow for the trade-off between robustnes…
▽ More
Diffusion models are known to be vulnerable to outliers in training data. In this paper we study an alternative diffusion loss function, which can preserve the high quality of generated data like the original squared $L_{2}$ loss while at the same time being robust to outliers. We propose to use pseudo-Huber loss function with a time-dependent parameter to allow for the trade-off between robustness on the most vulnerable early reverse-diffusion steps and fine details restoration on the final steps. We show that pseudo-Huber loss with the time-dependent parameter exhibits better performance on corrupted datasets in both image and audio domains. In addition, the loss function we propose can potentially help diffusion models to resist dataset corruption while not requiring data filtering or purification compared to conventional training algorithms.
△ Less
Submitted 25 March, 2024;
originally announced March 2024.
-
A Semi-Supervised Deep Learning Approach to Dataset Collection for Query-By-Humming Task
Authors:
Amantur Amatov,
Dmitry Lamanov,
Maksim Titov,
Ivan Vovk,
Ilya Makarov,
Mikhail Kudinov
Abstract:
Query-by-Humming (QbH) is a task that involves finding the most relevant song based on a hummed or sung fragment. Despite recent successful commercial solutions, implementing QbH systems remains challenging due to the lack of high-quality datasets for training machine learning models. In this paper, we propose a deep learning data collection technique and introduce Covers and Hummings Aligned Data…
▽ More
Query-by-Humming (QbH) is a task that involves finding the most relevant song based on a hummed or sung fragment. Despite recent successful commercial solutions, implementing QbH systems remains challenging due to the lack of high-quality datasets for training machine learning models. In this paper, we propose a deep learning data collection technique and introduce Covers and Hummings Aligned Dataset (CHAD), a novel dataset that contains 18 hours of short music fragments, paired with time-aligned hummed versions. To expand our dataset, we employ a semi-supervised model training pipeline that leverages the QbH task as a specialized case of cover song identification (CSI) task. Starting with a model trained on the initial dataset, we iteratively collect groups of fragments of cover versions of the same song and retrain the model on the extended data. Using this pipeline, we collect over 308 hours of additional music fragments, paired with time-aligned cover versions. The final model is successfully applied to the QbH task and achieves competitive results on benchmark datasets. Our study shows that the proposed dataset and training pipeline can effectively facilitate the implementation of QbH systems.
△ Less
Submitted 2 December, 2023;
originally announced December 2023.
-
Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
Authors:
Vadim Popov,
Ivan Vovk,
Vladimir Gogoryan,
Tasnima Sadekova,
Mikhail Kudinov,
Jiansheng Wei
Abstract:
Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We pres…
▽ More
Voice conversion is a common speech synthesis task which can be solved in different ways depending on a particular real-world scenario. The most challenging one often referred to as one-shot many-to-many voice conversion consists in copying the target voice from only one reference utterance in the most general case when both source and target speakers do not belong to the training dataset. We present a scalable high-quality solution based on diffusion probabilistic modeling and demonstrate its superior quality compared to state-of-the-art one-shot voice conversion approaches. Moreover, focusing on real-time applications, we investigate general principles which can make diffusion models faster while kee** synthesis quality at a high level. As a result, we develop a novel Stochastic Differential Equations solver suitable for various diffusion model types and generative tasks as shown through empirical studies and justify it by theoretical analysis.
△ Less
Submitted 4 August, 2022; v1 submitted 28 September, 2021;
originally announced September 2021.
-
Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
Authors:
Vadim Popov,
Ivan Vovk,
Vladimir Gogoryan,
Tasnima Sadekova,
Mikhail Kudinov
Abstract:
Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by graduall…
▽ More
Recently, denoising diffusion probabilistic models and generative score matching have shown high potential in modelling complex data distributions while stochastic calculus has provided a unified point of view on these techniques allowing for flexible inference schemes. In this paper we introduce Grad-TTS, a novel text-to-speech model with score-based decoder producing mel-spectrograms by gradually transforming noise predicted by encoder and aligned with text input by means of Monotonic Alignment Search. The framework of stochastic differential equations helps us to generalize conventional diffusion probabilistic models to the case of reconstructing data from noise with different parameters and allows to make this reconstruction flexible by explicitly controlling trade-off between sound quality and inference speed. Subjective human evaluation shows that Grad-TTS is competitive with state-of-the-art text-to-speech approaches in terms of Mean Opinion Score. We will make the code publicly available shortly.
△ Less
Submitted 5 August, 2021; v1 submitted 13 May, 2021;
originally announced May 2021.
-
Towards security recommendations for public-key infrastructures for production environments in the post-quantum era
Authors:
S. E. Yunakovsky,
M. Kot,
N. O. Pozhar,
D. Nabokov,
M. A. Kudinov,
A. Guglya,
E. O. Kiktenko,
E. Kolycheva,
A. Borisov,
A. K. Fedorov
Abstract:
Quantum computing technologies pose a significant threat to the currently employed public-key cryptography protocols. In this paper, we discuss the impact of the quantum threat on public key infrastructures (PKIs), which are used as a part of security systems for protecting production environments. We analyze security issues of existing models with a focus on requirements for a fast transition to…
▽ More
Quantum computing technologies pose a significant threat to the currently employed public-key cryptography protocols. In this paper, we discuss the impact of the quantum threat on public key infrastructures (PKIs), which are used as a part of security systems for protecting production environments. We analyze security issues of existing models with a focus on requirements for a fast transition to post-quantum solutions. Although our primary focus is on the attacks with quantum computing, we also discuss some security issues that are not directly related to the used cryptographic algorithms but are essential for the overall security of the PKI. We attempt to provide a set of security recommendations regarding the PKI from the viewpoints of attacks with quantum computers.
△ Less
Submitted 23 June, 2021; v1 submitted 4 May, 2021;
originally announced May 2021.
-
Security analysis of the W-OTS$^+$ signature scheme: Updating security bounds
Authors:
M. A. Kudinov,
E. O. Kiktenko,
A. K. Fedorov
Abstract:
In this work, we discuss in detail a flaw in the original security proof of the W-OTS${^+}$ variant of the Winternitz one-time signature scheme, which is an important component for various stateless and stateful many-time hash-based digital signature schemes. We update the security proof for the W-OTS${^+}$ scheme and derive the corresponding security level. Our result is of importance for the sec…
▽ More
In this work, we discuss in detail a flaw in the original security proof of the W-OTS${^+}$ variant of the Winternitz one-time signature scheme, which is an important component for various stateless and stateful many-time hash-based digital signature schemes. We update the security proof for the W-OTS${^+}$ scheme and derive the corresponding security level. Our result is of importance for the security analysis of hash-based digital signature schemes.
△ Less
Submitted 22 July, 2021; v1 submitted 18 February, 2020;
originally announced February 2020.
-
Advanced attribute-based protocol based on the modified secret sharing scheme
Authors:
M. A. Kudinov,
A. A. Chilikov,
E. O. Kiktenko,
A. K. Fedorov
Abstract:
We construct a new protocol for attribute-based encryption with the use of the modification of the standard secret sharing scheme. In the suggested modification of the secret sharing scheme, only one master key for each user is required that is achieved by linearly enlarging public parameters in the access formula. We then use this scheme for designing an attribute-based encryption protocol relate…
▽ More
We construct a new protocol for attribute-based encryption with the use of the modification of the standard secret sharing scheme. In the suggested modification of the secret sharing scheme, only one master key for each user is required that is achieved by linearly enlarging public parameters in the access formula. We then use this scheme for designing an attribute-based encryption protocol related to some access structure in terms of attributes. We demonstrate that the universe of possible attributes does not affect the resulting efficiency of the scheme. The security proofs for both constructions are provided.
△ Less
Submitted 23 November, 2020; v1 submitted 6 December, 2019;
originally announced December 2019.
-
Proof-of-forgery for hash-based signatures
Authors:
E. O. Kiktenko,
M. A. Kudinov,
A. A. Bulychev,
A. K. Fedorov
Abstract:
In the present work, a peculiar property of hash-based signatures allowing detection of their forgery event is explored. This property relies on the fact that a successful forgery of a hash-based signature most likely results in a collision with respect to the employed hash function, while the demonstration of this collision could serve as convincing evidence of the forgery. Here we prove that wit…
▽ More
In the present work, a peculiar property of hash-based signatures allowing detection of their forgery event is explored. This property relies on the fact that a successful forgery of a hash-based signature most likely results in a collision with respect to the employed hash function, while the demonstration of this collision could serve as convincing evidence of the forgery. Here we prove that with properly adjusted parameters Lamport and Winternitz one-time signatures schemes could exhibit a forgery detection availability property. This property is of significant importance in the framework of crypto-agility paradigm since the considered forgery detection serves as an alarm that the employed cryptographic hash function becomes insecure to use and the corresponding scheme has to be replaced.
△ Less
Submitted 8 September, 2021; v1 submitted 30 May, 2019;
originally announced May 2019.
-
Detecting brute-force attacks on cryptocurrency wallets
Authors:
E. O. Kiktenko,
M. A. Kudinov,
A. K. Fedorov
Abstract:
Blockchain is a distributed ledger, which is protected against malicious modifications by means of cryptographic tools, e.g. digital signatures and hash functions. One of the most prominent applications of blockchains is cryptocurrencies, such as Bitcoin. In this work, we consider a particular attack on wallets for collecting assets in a cryptocurrency network based on brute-force search attacks.…
▽ More
Blockchain is a distributed ledger, which is protected against malicious modifications by means of cryptographic tools, e.g. digital signatures and hash functions. One of the most prominent applications of blockchains is cryptocurrencies, such as Bitcoin. In this work, we consider a particular attack on wallets for collecting assets in a cryptocurrency network based on brute-force search attacks. Using Bitcoin as an example, we demonstrate that if the attack is implemented successfully, a legitimate user is able to prove that fact of this attack with a high probability. We also consider two options for modification of existing cryptocurrency protocols for dealing with this type of attacks. First, we discuss a modification that requires introducing changes in the Bitcoin protocol and allows diminishing the motivation to attack wallets. Second, an alternative option is the construction of special smart-contracts, which reward the users for providing evidence of the brute-force attack. The execution of this smart-contract can work as an automatic alarm that the employed cryptographic mechanisms, and (particularly) hash functions, have an evident vulnerability.
△ Less
Submitted 19 December, 2019; v1 submitted 15 April, 2019;
originally announced April 2019.
-
Fine-tuning of Language Models with Discriminator
Authors:
Vadim Popov,
Mikhail Kudinov
Abstract:
Cross-entropy loss is a common choice when it comes to multiclass classification tasks and language modeling in particular. Minimizing this loss results in language models of very good quality. We show that it is possible to fine-tune these models and make them perform even better if they are fine-tuned with sum of cross-entropy loss and reverse Kullback-Leibler divergence. The latter is estimated…
▽ More
Cross-entropy loss is a common choice when it comes to multiclass classification tasks and language modeling in particular. Minimizing this loss results in language models of very good quality. We show that it is possible to fine-tune these models and make them perform even better if they are fine-tuned with sum of cross-entropy loss and reverse Kullback-Leibler divergence. The latter is estimated using discriminator network that we train in advance. During fine-tuning probabilities of rare words that are usually underestimated by language models become bigger. The novel approach that we propose allows us to reach state-of-the-art quality on Penn Treebank: perplexity decreases from 52.4 to 52.1. Our fine-tuning algorithm is rather fast, scales well to different architectures and datasets and requires almost no hyperparameter tuning: the only hyperparameter that needs to be tuned is learning rate.
△ Less
Submitted 15 January, 2019; v1 submitted 12 November, 2018;
originally announced November 2018.
-
Differentially Private Distributed Learning for Language Modeling Tasks
Authors:
Vadim Popov,
Mikhail Kudinov,
Irina Piontkovskaya,
Petr Vytovtov,
Alex Nevidomsky
Abstract:
One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general…
▽ More
One of the big challenges in machine learning applications is that training data can be different from the real-world data faced by the algorithm. In language modeling, users' language (e.g. in private messaging) could change in a year and be completely different from what we observe in publicly available data. At the same time, public data can be used for obtaining general knowledge (i.e. general model of English). We study approaches to distributed fine-tuning of a general model on user private data with the additional requirements of maintaining the quality on the general data and minimization of communication costs. We propose a novel technique that significantly improves prediction quality on users' language compared to a general model and outperforms gradient compression methods in terms of communication efficiency. The proposed procedure is fast and leads to an almost 70% perplexity reduction and 8.7 percentage point improvement in keystroke saving rate on informal English texts. We also show that the range of tasks our approach is applicable to is not limited by language modeling only. Finally, we propose an experimental framework for evaluating differential privacy of distributed training of language models and show that our approach has good privacy guarantees.
△ Less
Submitted 6 March, 2018; v1 submitted 20 December, 2017;
originally announced December 2017.
