-
Speech Robust Bench: A Robustness Benchmark For Speech Recognition
Authors:
Muhammad A. Shah,
David Solans Noguero,
Mikko A. Heikkila,
Nicolas Kourtellis
Abstract:
As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 69 input perturbations which are intended to simulate…
▽ More
As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 69 input perturbations which are intended to simulate various corruptions that ASR models may encounter in the physical and digital world. We use SRB to evaluate the robustness of several state-of-the-art ASR models and observe that model size and certain modeling choices such as discrete representations, and self-training appear to be conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females, and observed noticeable disparities in the model's robustness across subgroups. We believe that SRB will facilitate future research towards robust ASR models, by making it easier to conduct comprehensive and comparable robustness evaluations.
△ Less
Submitted 8 March, 2024;
originally announced March 2024.
-
Differentially private partitioned variational inference
Authors:
Mikko A. Heikkilä,
Matthew Ashman,
Siddharth Swaroop,
Richard E. Turner,
Antti Honkela
Abstract:
Learning a privacy-preserving model from sensitive data which are distributed across multiple devices is an increasingly important problem. The problem is often formulated in the federated learning context, with the aim of learning a single global model while kee** the data distributed. Moreover, Bayesian learning is a popular approach for modelling, since it naturally supports reliable uncertai…
▽ More
Learning a privacy-preserving model from sensitive data which are distributed across multiple devices is an increasingly important problem. The problem is often formulated in the federated learning context, with the aim of learning a single global model while kee** the data distributed. Moreover, Bayesian learning is a popular approach for modelling, since it naturally supports reliable uncertainty estimates. However, Bayesian learning is generally intractable even with centralised non-private data and so approximation techniques such as variational inference are a necessity. Variational inference has recently been extended to the non-private federated learning setting via the partitioned variational inference algorithm. For privacy protection, the current gold standard is called differential privacy. Differential privacy guarantees privacy in a strong, mathematically clearly defined sense.
In this paper, we present differentially private partitioned variational inference, the first general framework for learning a variational approximation to a Bayesian posterior distribution in the federated learning setting while minimising the number of communication rounds and providing differential privacy guarantees for data subjects.
We propose three alternative implementations in the general framework, one based on perturbing local optimisation runs done by individual parties, and two based on perturbing updates to the global model (one using a version of federated averaging, the second one adding virtual parties to the protocol), and compare their properties both theoretically and empirically.
△ Less
Submitted 18 April, 2023; v1 submitted 23 September, 2022;
originally announced September 2022.
-
Tight Accounting in the Shuffle Model of Differential Privacy
Authors:
Antti Koskela,
Mikko A. Heikkilä,
Antti Honkela
Abstract:
Shuffle model of differential privacy is a novel distributed privacy model based on a combination of local privacy mechanisms and a secure shuffler. It has been shown that the additional randomisation provided by the shuffler improves privacy bounds compared to the purely local mechanisms. Accounting tight bounds, however, is complicated by the complexity brought by the shuffler. The recently prop…
▽ More
Shuffle model of differential privacy is a novel distributed privacy model based on a combination of local privacy mechanisms and a secure shuffler. It has been shown that the additional randomisation provided by the shuffler improves privacy bounds compared to the purely local mechanisms. Accounting tight bounds, however, is complicated by the complexity brought by the shuffler. The recently proposed numerical techniques for evaluating $(\varepsilon,δ)$-differential privacy guarantees have been shown to give tighter bounds than commonly used methods for compositions of various complex mechanisms. In this paper, we show how to obtain accurate bounds for adaptive compositions of general $\varepsilon$-LDP shufflers using the analysis by Feldman et al. (2021) and tight bounds for adaptive compositions of shufflers of $k$-randomised response mechanisms, using the analysis by Balle et al. (2019). We show how to speed up the evaluation of the resulting privacy loss distribution from $\mathcal{O}(n^2)$ to $\mathcal{O}(n)$, where $n$ is the number of users, without noticeable change in the resulting $δ(\varepsilon)$-upper bounds. We also demonstrate looseness of the existing bounds and methods found in the literature, improving previous composition results significantly.
△ Less
Submitted 31 January, 2022; v1 submitted 1 June, 2021;
originally announced June 2021.
-
Differentially private cross-silo federated learning
Authors:
Mikko A. Heikkilä,
Antti Koskela,
Kana Shimizu,
Samuel Kaski,
Antti Honkela
Abstract:
Strict privacy is of paramount importance in distributed machine learning. Federated learning, with the main idea of communicating only what is needed for learning, has been recently introduced as a general approach for distributed learning to enhance learning and improve security. However, federated learning by itself does not guarantee any privacy for data subjects. To quantify and control how m…
▽ More
Strict privacy is of paramount importance in distributed machine learning. Federated learning, with the main idea of communicating only what is needed for learning, has been recently introduced as a general approach for distributed learning to enhance learning and improve security. However, federated learning by itself does not guarantee any privacy for data subjects. To quantify and control how much privacy is compromised in the worst-case, we can use differential privacy.
In this paper we combine additively homomorphic secure summation protocols with differential privacy in the so-called cross-silo federated learning setting. The goal is to learn complex models like neural networks while guaranteeing strict privacy for the individual data subjects. We demonstrate that our proposed solutions give prediction accuracy that is comparable to the non-distributed setting, and are fast enough to enable learning models with millions of parameters in a reasonable time.
To enable learning under strict privacy guarantees that need privacy amplification by subsampling, we present a general algorithm for oblivious distributed subsampling. However, we also argue that when malicious parties are present, a simple approach using distributed Poisson subsampling gives better privacy.
Finally, we show that by leveraging random projections we can further scale-up our approach to larger models while suffering only a modest performance loss.
△ Less
Submitted 10 July, 2020;
originally announced July 2020.
-
Differentially Private Markov Chain Monte Carlo
Authors:
Mikko A. Heikkilä,
Joonas Jälkö,
Onur Dikmen,
Antti Honkela
Abstract:
Recent developments in differentially private (DP) machine learning and DP Bayesian learning have enabled learning under strong privacy guarantees for the training data subjects. In this paper, we further extend the applicability of DP Bayesian learning by presenting the first general DP Markov chain Monte Carlo (MCMC) algorithm whose privacy-guarantees are not subject to unrealistic assumptions o…
▽ More
Recent developments in differentially private (DP) machine learning and DP Bayesian learning have enabled learning under strong privacy guarantees for the training data subjects. In this paper, we further extend the applicability of DP Bayesian learning by presenting the first general DP Markov chain Monte Carlo (MCMC) algorithm whose privacy-guarantees are not subject to unrealistic assumptions on Markov chain convergence and that is applicable to posterior inference in arbitrary models. Our algorithm is based on a decomposition of the Barker acceptance test that allows evaluating the Rényi DP privacy cost of the accept-reject choice. We further show how to improve the DP guarantee through data subsampling and approximate acceptance tests.
△ Less
Submitted 17 June, 2019; v1 submitted 29 January, 2019;
originally announced January 2019.
-
CH abundance gradient in TMC-1
Authors:
Aleksi Suutarinen,
Wolf Dietrich Geppert,
Jorma Harju,
Arto Heikkilä,
Stephan Hotzel,
Mika Juvela,
Tom J. Millar,
Catherine Walsh,
Jan Gerard Amos Wouterloot
Abstract:
We observed the 9-cm Lambda-doubling lines of CH along the dense filament of TMC-1. The CH column densities were compared with the total H2 column densities derived using the 2MASS NIR data and previously published SCUBA maps and with OH column densities derived using previous observations with Effelsberg. We also modelled the chemical evolution of TMC-1 adopting physical conditions typical of dar…
▽ More
We observed the 9-cm Lambda-doubling lines of CH along the dense filament of TMC-1. The CH column densities were compared with the total H2 column densities derived using the 2MASS NIR data and previously published SCUBA maps and with OH column densities derived using previous observations with Effelsberg. We also modelled the chemical evolution of TMC-1 adopting physical conditions typical of dark clouds using the UMIST Database for Astrochemistry gas-phase reaction network to aid the interpretation of the observed OH/CH abundance ratios. The CH column density has a clear peak in the vicinity of the cyanopolyyne maximum of TMC-1. The fractional CH abundance relative to H2 increases steadily from the northwestern end of the filament where it lies around 1.0e-8, to the southeast where it reaches a value of 2.0e-8. The OH and CH column densities are well correlated, and we obtained OH/CH abundance ratios of ~ 16 - 20. These values are clearly larger than what has been measured recently in diffuse interstellar gas and is likely to be related to C to CO conversion at higher densities. The good correlation between CH and OH can be explained by similar production and destruction pathways. We suggest that the observed CH and OH abundance gradients are mainly due to enhanced abundances in a low-density envelope which becomes more prominent in the southeastern part and seems to continue beyond the dense filament. An extensive envelope probably signifies an early stage of dynamical evolution, and conforms with the detection of a large CH abundance in the southeastern part of the cloud. The implied presence of other simple forms of carbon in the gas phase provides a natural explanation for the observation of "early-type" molecules in this region.
△ Less
Submitted 31 May, 2011; v1 submitted 30 May, 2011;
originally announced May 2011.
-
Sub-millimeter Observations of Giant Molecular Clouds in the Large Magellanic Cloud: Temperature and Density as Determined from J=3-2 and J=1-0 transitions of CO
Authors:
T. Minamidani,
N. Mizuno,
Y. Mizuno,
A. Kawamura,
T. Onishi,
T. Hasegawa,
K. Tatematsu,
M. Ikeda,
Y. Moriguchi,
N. Yamaguchi,
J. Ott,
T. Wong,
E. Muller,
J. L. Pineda,
A. Hughes,
L. Staveley-Smith,
U. Klein,
A. Mizuno,
S. Nikolić,
R. S. Booth,
A. Heikkilä,
L. -A. Nyman,
M. Lerner,
G. Garay,
S. Kim
, et al. (4 additional authors not shown)
Abstract:
We have carried out sub-mm 12CO(J=3-2) observations of 6 giant molecular clouds (GMCs) in the Large Magellanic Cloud (LMC) with the ASTE 10m sub-mm telescope at a spatial resolution of 5 pc and very high sensitivity. We have identified 32 molecular clumps in the GMCs and revealed significant details of the warm and dense molecular gas with n(H2) $\sim$ 10$^{3-5}$ cm$^{-3}$ and Tkin $\sim$ 60 K.…
▽ More
We have carried out sub-mm 12CO(J=3-2) observations of 6 giant molecular clouds (GMCs) in the Large Magellanic Cloud (LMC) with the ASTE 10m sub-mm telescope at a spatial resolution of 5 pc and very high sensitivity. We have identified 32 molecular clumps in the GMCs and revealed significant details of the warm and dense molecular gas with n(H2) $\sim$ 10$^{3-5}$ cm$^{-3}$ and Tkin $\sim$ 60 K. These data are combined with 12CO(J=1-0) and 13CO(J=1-0) results and compared with LVG calculations. We found that the ratio of 12CO(J=3-2) to 12CO(J=1-0) emission is sensitive to and is well correlated with the local Halpha flux. We interpret that differences of clump propeties represent an evolutionary sequence of GMCs in terms of density increase leading to star formation.Type I and II GMCs (starless GMCs and GMCs with HII regions only, respectively) are at the young phase of star formation where density does not yet become high enough to show active star formation and Type III GMCs (GMCs with HII regions and young star clusters) represents the later phase where the average density is increased and the GMCs are forming massive stars. The high kinetic temperature correlated with \Halpha flux suggests that FUV heating is dominant in the molecular gas of the LMC.
△ Less
Submitted 23 October, 2007;
originally announced October 2007.
-
Molecular line study of evolution in protostellar cloud cores
Authors:
S. Kontinen,
J. Harju,
A. Heikkila,
L. K. Haikala
Abstract:
Two dense dark cloud cores representing different stages of dynamical evolution were observed in a number of molecular spectral lines. One of the cores, Cha- MMS1 in the Chamaeleon cloud I contains a Class 0 protostar, whereas the other, CrA C in the R Coronae Australis cloud, is pre-stellar. We find that the cores have very different chemical compositions. Cha-MMS1 exhibits characteristics of s…
▽ More
Two dense dark cloud cores representing different stages of dynamical evolution were observed in a number of molecular spectral lines. One of the cores, Cha- MMS1 in the Chamaeleon cloud I contains a Class 0 protostar, whereas the other, CrA C in the R Coronae Australis cloud, is pre-stellar. We find that the cores have very different chemical compositions. Cha-MMS1 exhibits characteristics of so-called `early-type' chemistry with high abundan- ces of carbon-chain molecules such as HC3N, but it also has a large N2H+ abundance, which is expected only to build up at later stages. In contrast, none of the carbon-chain molecules were detected in CrA C. CrA C has a higher SO abundance than Cha-MMS1, which implies that it is chemically `older' than Cha- MMS1. The most striking difference between the two cores is seen in the HC3N/SO abundance ratio, which is at least three orders of magnitude higher in Cha-MMS than in CrA C. This result is surprising since starless cores are usually thought to be chemically younger than star-forming cores. Because of the high N2H+ abundance, we suggest that Cha-MMS1 represents the `late-time cyanopolyyne peak'. This would also be a more natural explanation for the carbon-chain molecules than the `early-time' picture in view of the fact that the core is presently collapsing to form a star. Thus, the dynamical evolution in CrA C seems to have been very slow compared with that of Cha-MMS1, and we discuss possible reasons for this.
△ Less
Submitted 9 August, 2000;
originally announced August 2000.
