-
NanoBatch Privacy: Enabling fast Differentially Private learning on the IPU
Authors:
Edward H. Lee,
Mario Michael Krell,
Alexander Tsyplikhin,
Victoria Rege,
Errol Colak,
Kristen W. Yeom
Abstract:
Differentially private SGD (DPSGD) has recently shown promise in deep learning. However, compared to non-private SGD, the DPSGD algorithm places computational overheads that can undo the benefit of batching in GPUs. Micro-batching is a common method to alleviate this and is fully supported in the TensorFlow Privacy library (TFDP). However, it degrades accuracy. We propose NanoBatch Privacy, a ligh…
▽ More
Differentially private SGD (DPSGD) has recently shown promise in deep learning. However, compared to non-private SGD, the DPSGD algorithm places computational overheads that can undo the benefit of batching in GPUs. Micro-batching is a common method to alleviate this and is fully supported in the TensorFlow Privacy library (TFDP). However, it degrades accuracy. We propose NanoBatch Privacy, a lightweight add-on to TFDP to be used on Graphcore IPUs by leveraging batch size of 1 (without microbatching) and gradient accumulation. This allows us to achieve large total batch sizes with minimal impacts to throughput. Second, we illustrate using Cifar-10 how larger batch sizes are not necessarily optimal from a privacy versus utility perspective. On ImageNet, we achieve more than 15x speedup over TFDP versus 8x A100s and significant speedups even across libraries such as Opacus. We also provide two extensions: 1) DPSGD for pipelined models and 2) per-layer clip** that is 15x faster than the Opacus implementation on 8x A100s. Finally as an application case study, we apply NanoBatch training for use on private Covid-19 chest CT prediction.
△ Less
Submitted 2 June, 2022; v1 submitted 24 September, 2021;
originally announced September 2021.
-
Studying the potential of Graphcore IPUs for applications in Particle Physics
Authors:
Lakshan Ram Madhan Mohan,
Alexander Marshall,
Samuel Maddrell-Mander,
Daniel O'Hanlon,
Konstantinos Petridis,
Jonas Rademacker,
Victoria Rege,
Alexander Titterton
Abstract:
This paper presents the first study of Graphcore's Intelligence Processing Unit (IPU) in the context of particle physics applications. The IPU is a new type of processor optimised for machine learning. Comparisons are made for neural-network-based event simulation, multiple-scattering correction, and flavour tagging, implemented on IPUs, GPUs and CPUs, using a variety of neural network architectur…
▽ More
This paper presents the first study of Graphcore's Intelligence Processing Unit (IPU) in the context of particle physics applications. The IPU is a new type of processor optimised for machine learning. Comparisons are made for neural-network-based event simulation, multiple-scattering correction, and flavour tagging, implemented on IPUs, GPUs and CPUs, using a variety of neural network architectures and hyperparameters. Additionally, a Kálmán filter for track reconstruction is implemented on IPUs and GPUs. The results indicate that IPUs hold considerable promise in addressing the rapidly increasing compute needs in particle physics.
△ Less
Submitted 20 August, 2020;
originally announced August 2020.
-
Design and Implementation of an Antenna Model for the Cooja simulator
Authors:
Vishwesh Rege
Abstract:
COOJA is a network simulator developed for wireless sensor networks. It can be used for high-level algorithm development as well as low-level device driver implementations for accurate simulation of wireless sensor networks before deployment. However, in a simulation Cooja assumes that the nodes are only equipped with omnidirectional antennas. There is currently no support for directional antennas…
▽ More
COOJA is a network simulator developed for wireless sensor networks. It can be used for high-level algorithm development as well as low-level device driver implementations for accurate simulation of wireless sensor networks before deployment. However, in a simulation Cooja assumes that the nodes are only equipped with omnidirectional antennas. There is currently no support for directional antennas. Due to the growing interest in the use of directional or smart antennas in wireless sensor networks, a model that can support directional antennas is essential for the realistic simulations of protocols relying on directional communication. This paper presents work on extending COOJA with a directional antenna model.
△ Less
Submitted 8 December, 2015;
originally announced October 2016.
