Showing 1–9 of 9 results for author: Reisser, M

Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Data

Authors: Mahdi Morafah, Matthias Reisser, Bill Lin, Christos Louizos

Abstract: The proliferation of edge devices has brought Federated Learning (FL) to the forefront as a promising paradigm for decentralized and collaborative model training while preserving the privacy of clients' data. However, FL struggles with a significant performance reduction and poor convergence when confronted with Non-Independent and Identically Distributed (Non-IID) data distributions among partici… ▽ More The proliferation of edge devices has brought Federated Learning (FL) to the forefront as a promising paradigm for decentralized and collaborative model training while preserving the privacy of clients' data. However, FL struggles with a significant performance reduction and poor convergence when confronted with Non-Independent and Identically Distributed (Non-IID) data distributions among participating clients. While previous efforts, such as client drift mitigation and advanced server-side model fusion techniques, have shown some success in addressing this challenge, they often overlook the root cause of the performance reduction - the absence of identical data accurately mirroring the global data distribution among clients. In this paper, we introduce Gen-FedSD, a novel approach that harnesses the powerful capability of state-of-the-art text-to-image foundation models to bridge the significant Non-IID performance gaps in FL. In Gen-FedSD, each client constructs textual prompts for each class label and leverages an off-the-shelf state-of-the-art pre-trained Stable Diffusion model to synthesize high-quality data samples. The generated synthetic data is tailored to each client's unique local data gaps and distribution disparities, effectively making the final augmented local data IID. Through extensive experimentation, we demonstrate that Gen-FedSD achieves state-of-the-art performance and significant communication cost savings across various datasets and Non-IID settings. △ Less

Submitted 13 May, 2024; originally announced May 2024.

Comments: International Workshop on Federated Foundation Models for the Web 2024 (FL@FM-TheWebConf'24)

A Mutual Information Perspective on Federated Contrastive Learning

Authors: Christos Louizos, Matthias Reisser, Denis Korzhenkov

Abstract: We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and user verification; by adding a user verification loss to each client's local SimCLR loss we recover a lower bound to the global multi-view mutual information. To accommodate for t… ▽ More We investigate contrastive learning in the federated setting through the lens of SimCLR and multi-view mutual information maximization. In doing so, we uncover a connection between contrastive representation learning and user verification; by adding a user verification loss to each client's local SimCLR loss we recover a lower bound to the global multi-view mutual information. To accommodate for the case of when some labelled data are available at the clients, we extend our SimCLR variant to the federated semi-supervised setting. We see that a supervised SimCLR objective can be obtained with two changes: a) the contrastive loss is computed between datapoints that share the same label and b) we require an additional auxiliary head that predicts the correct labels from either of the two views. Along with the proposed SimCLR extensions, we also study how different sources of non-i.i.d.-ness can impact the performance of federated unsupervised learning through global mutual information maximization; we find that a global objective is beneficial for some sources of non-i.i.d.-ness but can be detrimental for others. We empirically evaluate our proposed extensions in various tasks to validate our claims and furthermore demonstrate that our proposed modifications generalize to other pretraining methods. △ Less

Submitted 3 May, 2024; originally announced May 2024.

Comments: Published as a conference paper at ICLR 2024

Hyperparameter Optimization through Neural Network Partitioning

Authors: Bruno Mlodozeniec, Matthias Reisser, Christos Louizos

Abstract: Well-tuned hyperparameters are crucial for obtaining good generalization behavior in neural networks. They can enforce appropriate inductive biases, regularize the model and improve performance -- especially in the presence of limited data. In this work, we propose a simple and efficient way for optimizing hyperparameters inspired by the marginal likelihood, an optimization objective that requires… ▽ More Well-tuned hyperparameters are crucial for obtaining good generalization behavior in neural networks. They can enforce appropriate inductive biases, regularize the model and improve performance -- especially in the presence of limited data. In this work, we propose a simple and efficient way for optimizing hyperparameters inspired by the marginal likelihood, an optimization objective that requires no validation data. Our method partitions the training data and a neural network model into $K$ data shards and parameter partitions, respectively. Each partition is associated with and optimized only on specific data shards. Combining these partitions into subnetworks allows us to define the ``out-of-training-sample" loss of a subnetwork, i.e., the loss on data shards unseen by the subnetwork, as the objective for hyperparameter optimization. We demonstrate that we can apply this objective to optimize a variety of different hyperparameters in a single training run while being significantly computationally cheaper than alternative methods aiming to optimize the marginal likelihood for neural networks. Lastly, we also focus on optimizing hyperparameters in federated learning, where retraining and cross-validation are particularly challenging. △ Less

Submitted 28 April, 2023; originally announced April 2023.

Comments: Published as a conference paper at ICLR 2023

Quantization Robust Federated Learning for Efficient Inference on Heterogeneous Devices

Authors: Kartik Gupta, Marios Fournarakis, Matthias Reisser, Christos Louizos, Markus Nagel

Abstract: Federated Learning (FL) is a machine learning paradigm to distributively learn machine learning models from decentralized data that remains on-device. Despite the success of standard Federated optimization methods, such as Federated Averaging (FedAvg) in FL, the energy demands and hardware induced constraints for on-device learning have not been considered sufficiently in the literature. Specifica… ▽ More Federated Learning (FL) is a machine learning paradigm to distributively learn machine learning models from decentralized data that remains on-device. Despite the success of standard Federated optimization methods, such as Federated Averaging (FedAvg) in FL, the energy demands and hardware induced constraints for on-device learning have not been considered sufficiently in the literature. Specifically, an essential demand for on-device learning is to enable trained models to be quantized to various bit-widths based on the energy needs and heterogeneous hardware designs across the federation. In this work, we introduce multiple variants of federated averaging algorithm that train neural networks robust to quantization. Such networks can be quantized to various bit-widths with only limited reduction in full precision model accuracy. We perform extensive experiments on standard FL benchmarks to evaluate our proposed FedAvg variants for quantization robustness and provide a convergence analysis for our Quantization-Aware variants in FL. Our results demonstrate that integrating quantization robustness results in FL models that are significantly more robust to different bit-widths during quantized on-device inference. △ Less

Submitted 22 June, 2022; originally announced June 2022.

An Expectation-Maximization Perspective on Federated Learning

Authors: Christos Louizos, Matthias Reisser, Joseph Soriaga, Max Welling

Abstract: Federated learning describes the distributed training of models across multiple clients while kee** the data private on-device. In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters. We show that with simple Gaussian priors and a ha… ▽ More Federated learning describes the distributed training of models across multiple clients while kee** the data private on-device. In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters. We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting. This perspective on FedAvg unifies several recent works in the field and opens up the possibility for extensions through different choices for the hierarchical model. Based on this view, we further propose a variant of the hierarchical model that employs prior distributions to promote sparsity. By similarly using the hard-EM algorithm for learning, we obtain FedSparse, a procedure that can learn sparse neural networks in the federated learning setting. FedSparse reduces communication costs from client to server and vice-versa, as well as the computational costs for inference with the sparsified network - both of which are of great practical importance in federated learning. △ Less

Submitted 19 November, 2021; originally announced November 2021.

DP-REC: Private & Communication-Efficient Federated Learning

Authors: Aleksei Triastcyn, Matthias Reisser, Christos Louizos

Abstract: Privacy and communication efficiency are important challenges in federated training of neural networks, and combining them is still an open problem. In this work, we develop a method that unifies highly compressed communication and differential privacy (DP). We introduce a compression technique based on Relative Entropy Coding (REC) to the federated setting. With a minor modification to REC, we ob… ▽ More Privacy and communication efficiency are important challenges in federated training of neural networks, and combining them is still an open problem. In this work, we develop a method that unifies highly compressed communication and differential privacy (DP). We introduce a compression technique based on Relative Entropy Coding (REC) to the federated setting. With a minor modification to REC, we obtain a provably differentially private learning algorithm, DP-REC, and show how to compute its privacy guarantees. Our experiments demonstrate that DP-REC drastically reduces communication costs while providing privacy guarantees comparable to the state-of-the-art. △ Less

Submitted 7 December, 2021; v1 submitted 9 November, 2021; originally announced November 2021.

Federated Mixture of Experts

Authors: Matthias Reisser, Christos Louizos, Efstratios Gavves, Max Welling

Abstract: Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be subopt… ▽ More Federated learning (FL) has emerged as the predominant approach for collaborative training of neural network models across multiple users, without the need to gather the data at a central location. One of the important challenges in this setting is data heterogeneity, i.e. different users have different data characteristics. For this reason, training and using a single global model might be suboptimal when considering the performance of each of the individual user's data. In this work, we tackle this problem via Federated Mixture of Experts, FedMix, a framework that allows us to train an ensemble of specialized models. FedMix adaptively selects and trains a user-specific selection of the ensemble members. We show that users with similar data characteristics select the same members and therefore share statistical strength while mitigating the effect of non-i.i.d data. Empirically, we show through an extensive experimental evaluation that FedMix improves performance compared to using a single global model across a variety of different sources of non-i.i.d.-ness. △ Less

Submitted 14 July, 2021; originally announced July 2021.

Relaxed Quantization for Discretized Neural Networks

Authors: Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, Max Welling

Abstract: Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activ… ▽ More Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification. △ Less

Submitted 3 October, 2018; originally announced October 2018.

Decreasing nuclear volume concentrates DNA and enforces transcription factor-chromatin associations during Zebrafish genome activation

Authors: Matthias Reisser, J. Christof M. Gebhardt

Abstract: Zygotic genome activation (ZGA), the onset of transcription after initial quiescence, is a major developmental step in many species, which occurs after ten cell divisions in Zebrafish embryos. How transcription factor-chromatin interactions evolve during early development to support ZGA is largely unknown. We established single molecule tracking in live develo** Zebrafish embryos using reflected… ▽ More Zygotic genome activation (ZGA), the onset of transcription after initial quiescence, is a major developmental step in many species, which occurs after ten cell divisions in Zebrafish embryos. How transcription factor-chromatin interactions evolve during early development to support ZGA is largely unknown. We established single molecule tracking in live develo** Zebrafish embryos using reflected light-sheet microscopy to visualize the general transcription factor TATA-binding protein (TBP), and developed a novel data acquisition and analysis scheme to extract kinetic information during fast cell cycles. The chromatin-bound fraction of TBP increases during early development, compatible with increasing transcriptional activity. By quantifying TBP and DNA concentrations and their binding kinetics, we device a physical model of how the nuclear volume, which decreases during early development, enhances TBP-chromatin associations. Our single molecule data suggest that the shrinking nucleus is a major driving force and timer of ZGA in Zebrafish embryos. △ Less

Submitted 10 October, 2017; originally announced October 2017.

Journal ref: Nature Communications, 9(1), 5218 (2018)