Initialisation and Topology Effects in Decentralised Federated Learning

Abstract

Fully decentralised federated learning enables collaborative training of individual machine learning models on distributed devices on a communication network while kee** the training data localised. This approach enhances data privacy and eliminates both the single point of failure and the necessity for central coordination. Our research highlights that the effectiveness of decentralised federated learning is significantly influenced by the network topology of connected devices. We propose a strategy for uncoordinated initialisation of the artificial neural networks, which leverages the distribution of eigenvector centralities of the nodes of the underlying communication network, leading to a radically improved training efficiency. Additionally, our study explores the scaling behaviour and choice of environmental parameters under our proposed initialisation strategy. This work paves the way for more efficient and scalable artificial neural network training in a distributed and uncoordinated environment, offering a deeper understanding of the intertwining roles of network structure and learning dynamics.

1 Introduction

The traditional centralised approach to machine learning has shown great progress in the last few decades. This approach, while practical, comes at a cost in terms of systemic data privacy risks and centralisation overhead [1, 2, 3]. To alleviate these issues, the federated learning framework was proposed where each node (client) updates a local machine learning model using local data and only shares its model parameters with a centralised server, which in turn aggregates these individual models into one model and redistributes it to each node [1].

While this approach reduces the data privacy risk by eliminating data sharing with the centralised server, it still maintains a singular point of failure and puts a heavy communication burden on the central server node [4, 5]. Decentralised federated learning aims to provide an alternative approach that maintains data privacy but removes the need for a centralised server. This involves the set of nodes (clients) updating their local models based on the local data, but directly communicating with one another through a communication network. Each node then updates its local model by aggregating those of the neighbourhood [4, 6]. The efficiency of this approach is impacted by several kinds of inhomogeneities [6] characterising the network structure, initial conditions, learning data, and temporal irregularities. In this paper, we focus on the first two of these.

Motivation.

Decentralised federated learning immediately raises two distinct new issues compared to the centralised federated learning approach. First, that the initialisation and operations of the nodes have to be performed in an uncoordinated manner, as the role of coordination previously lay with the server.

Second, that the effect of structure and possible heterogeneities in the communication network structure is poorly understood. In the case of centralised federated learning, the communication network is organised as a simple star graph. In a decentralised setting, however, the network might be an emergent result of, e.g., the social network of the users of the devices, a distributed peer-discovery protocol or a hand-engineered topology comprising of IoT devices. Each of these assumptions leads to a different network topology with wildly different characteristics. Many network topologies modelling real-world phenomena, unlike a star graph, have diameters that monotonically scale up with the number of nodes, inducing an inherent latency in the communication of information between nodes that are not directly connected. Structural heterogeneities, e.g., the dimensionality of the network, degree heterogeneity and heterogeneities in other centrality measures also play important roles in the evolution of the information-sharing processes on networks [7]. This makes network heterogeneities primary candidates for analysis of any decentralised system.

Contribution.

In this manuscript, we focus on two issues: First, we show that a naïve application of commonly used methods of artificial neural network initialisation results in subpar performance when training deep neural networks coupled in a communication network and we propose an alternative uncoordinated initialisation method to resolve this issue. Second, we show that the proposed initialisation causes a significant improvement in different communication network topologies and demonstrate how the topology affects the scaling properties of the learning process.

2 Related works

Decentralised federated learning [4] comes as a natural next step in the development of the field of federated learning to save communication costs, improve robustness and privacy [1]. This approach has been used in application areas such as object recognition in medical images [8, 9] and other industrial settings [10, 11]. It has also been extended by novel optimisation and aggregation methods [12, 5, 13] and theoretical advances in terms of convergence analysis [14].

The structure of complex networks, central to decentralised federated learning by coding the communication structure between connected devices [15], can embody various heterogeneities. These have been found to be a crucial factor in understanding a variety of complex systems that involve many entities communicating or interacting together. For example, the role of degree distribution [16, 17], high clustering [18, 19] or existence of flat or hierarchical community structures [20, 21] in networks of real-world phenomena has been understood and analysed for decades. Recent advances in network modelling have extended heterogeneities in networks from structural to incorporate spatial [22] and also temporal heterogeneities, induced by patterns of, e.g., spatial constraints or bursty or self-exciting activity of the nodes [23, 24, 25, 26]. In the decentralised federated learning settings, the matter of structural heterogeneities of the underlying communication network has only been very recently subjected to systemic studies. Notably, Vogels et al. [27] analyse the effect of topology on optimal learning rate and Palmieri et al. [28] analyse the differences among individual training curves of specific nodes (e.g., high-degree hubs versus peripheries) for Barabási–Albert networks and stochastic block models with two blocks [7].

On the matter of parameter initialisation in federated learning, recent studies have focused on the effect of starting all nodes from a homogeneous set of parameters [6] or the parameters of an independently pre-trained model [29]. Historically, artificial neural networks were initialised from random uniform or Gaussian distribution with scales set based on heuristics and trial and error [30] or for specific activation functions [31]. The advent of much deeper architectures and widespread use of non-linear activation functions such as ReLU or Tanh led to a methodical understanding of the role of initial parameters to avoid exploding or diminishing activations and gradients. Glorot and Bengio [32] proposed a method based on certain simplifying assumptions about the non-linearities used. Later, He et al. [33] defined a more general framework for use with a wider variety of non-linearities, which was used for training the ResNet image recognition architecture [34].

In this work, we will be extending the same approach for effective initialisation of artificial neural network parameters to the decentralised setting, where the parameters are affected not only by the optimisation based on the training data but also due to interactions with other nodes of the communication network.

3 Preliminaries

In our setup, we used a simple decentralised federated learning system with an iterative process. Nodes are connected through a predetermined static, undirected communication network $G=(\mathcal{V},\mathcal{E})$ where $\mathcal{E}\subseteq\{\{v_{i},v_{j}\}|v_{i},v_{j}\in\mathcal{V}\text{ and }v_{i}\neq v_{j}\}$. All nodes train the same artificial neural network architecture $h(w_{i},\cdot)$ with $i$ indicating the parameters of the $i$-th node. Moreover, at time zero, we assume that $w_{i}\neq w_{j}\forall i,j\in\mathcal{V}$. $n=|\mathcal{V}|$ indicates the number of nodes (sometimes referred to as the system size), while $k_{i}=|\mathcal{N}_{i}|$ indicates the degree of a node $i$ defined as the number of its neighbours, and $p(k)$ is the degree distribution. For a node that is reached by following a random link, $q(k)$ is the distribution of the number of other links to that node, known as the excess degree distribution. The adjacency matrix is indicated by $A$.

Nodes $v_{i}\in\mathcal{V}$ initialise their local model parameters based on one of the following strategies: (1) random initialisation with no gain correction, an uncoordinated approach where each node draws their initial parameters independently based on a strategy optimised for isolated centralised training, e.g., from Ref. [33]; or (2) random initialisation with gain correction, which is our proposed initialisation strategy that re-scales initial parameter distributions from (1) based on the topology of the communication network. This approach will be explored in detail in Section 4.

Each node $i$ of the communication network holds a labelled local dataset $D_{i}$ such that $D_{i}\cap D_{j}=\emptyset,\forall i,j\in\mathcal{V}$ which are a portion of the same global dataset $D=\bigcup_{i=1}^{|\mathcal{V}|}D_{i}$. Each node can access its private data but not those of other nodes and we assume that the local datasets do not change over time. All nodes train the same artificial neural network architecture $h(w_{i},\cdot)$ with $w_{i}\in\mathbb{R}^{p}$ indicating the parameters of the $i$-th node solving the following Empirical Risk Minimisation problem:

$$\min_{w}F(w_{i};D_{i}), F(w_{i};D_{i})=\frac{1}{|D_{i}|}\sum_{(x,y)\in D_{i}}\ell(y,h(w_{i};x))$$ (1)

with $\ell$ representing the loss function, which can be convex or non-convex. Moreover, at time zero we assume that $w_{i}\neq w_{j}\forall i,j\in\mathcal{V}$ The nodes perform one or more batches of local training on their local data using an optimiser. At each iteration $t$, called here a communication round, the $i$-th node updates its parameters (weights and biases) using parameter values communicated by the nodes in its neighbourhood through the decentralised version of FedAvg (here called DecAvg):

$$w_{i}=\beta_{i}w_{i}+\sum_{\forall j\in\mathcal{N}_{i}}\beta_{j}w_{j},\quad\beta_{i}=\frac{|D_{i}|}{|D_{i}|+\sum_{\forall j\in\mathcal{N}_{i}}|D_{j}|},\forall i\in\mathcal{V}\,.$$ (2)

Since both the iid and non-iid distributions used in the manuscript provide on expectation the same number of total items per node, we can assume $\beta_{i}\approx 1/(k_{i}+1)$. In this paper, we use a combination of different datasets, iid and non-iid distribution of labelled data, neural network architectures (i.e., MLP, CNN) and optimisers (i.e., SGD, Adam). For more information, refer to Appendix A.

Expected values are indicated using the brackets around the random variable, e.g., $\langle k\rangle$ is the mean degree. Standard deviation is shown using $\sigma(\ldots)$. The $d$ parameters of node $i$ are indicated by vector $w_{i}$ of size $d$, sometimes arranged in a $d\times n$ matrix $W=\{w_{j,i}\}$. $\sigma_{ap}$ indicates the mean standard deviation across columns of $W$, i.e. $\sum_{i=1}^{n}\sigma(w_{*,i})/n$ where $w_{*,i}$ denotes column $i$ of $W$, while $\sigma_{an}$ indicates the mean standard deviation across rows, i.e. $\sum_{j=1}^{d}\sigma(w_{j,*})/d$, where $w_{j,*}$ denotes row $j$ of $W$. Artificial neural networks are usually initialised with parameters that are drawn from different sets of distributions, e.g., weights of each layer are drawn from a separate distribution. In this case, a vector $w_{i}$ can be formed for one specific set of parameters, drawn from a zero-mean distribution with standard deviation $\sigma_{init.}$.

4 Uncoordinated initialisation of artificial neural networks

Unlike in centralised federated learning, it is unwarranted to assume that in a massive decentralised federated learning setup, all nodes can negotiate and agree on initial values for model parameters in a manner that each parameter is exactly equal across all nodes. A fully uncoordinated method for selecting initial values for the model parameters means that each node should be able to draw initial values for their model parameters independently with few or no communication in a manner that is robust to possible communication errors.

It has been shown that judicious choice of initialisation strategy can enable training of much deeper artificial neural networks [32, 33, 34]. Specifically, a good parameter initialisation method leads to initial parameters that neither increase nor decrease activation values for consecutive layers exponentially [33, 32]. In the case of decentralised federated learning, this proves more challenging, as the aggregation step changes the distribution of parameters, meaning that the optimal initial value distributions are not only a function of the machine learning model architecture, but also affected by the communication network structure.

Empirically, we observe (Figure 1 dashed lines) that the decentralised, uncoordinated initialisation of nodes using the method proposed by He et al. [33] results in progressively poorer performance in the federated setting as the number of nodes grows, while the proposed uncoordinated initialisation performs similarly to the coordinated homogeneous strategy. Figure 1(b,d,f,h) shows this as scaling of the loss trajectory as a function of communication rounds with the number of nodes as $n^{\mu}$, with values of $\mu$ observed in range $0.4\leq\mu\leq 1$ depending on the experimental setting.

Figure 1: Mean test cross-entropy loss with the proposed initialisation (solid lines) compared to the initialisation method proposed in He et al. [33] without re-scaling (dashed lines). The decentralised federated learning process on nodes connected through (a,b) fully-connected (complete) networks with MNIST classification task on a simple multilayer perceptron with iid data distribution (c,d) Barabási–Albert networks with average degree 4, with the So2Sat LCZ42 classification task, using a simple convolutional architecture, Zipf data distribution, $\alpha=1.8$, (e,f) random 4-regular networks with CIFAR-10 classification task with VGG16 architecture and (g,h) same configuration as (a,b) but using Adam optimiser with decoupled weight decay. The results show that without the proposed re-scaling of the parameters, the mean test loss has a plateau lasting a number of rounds linear or sub-linear to the system size, while our uncoordinated proposed initialisation method performs on par with the coordinated homogeneous initialisation. Bottom row (b,d,f,h) shows the empirical scaling of the test loss time trajectory of the independent [33] method initialisation with system size, with exponents ranging from 0.4 to 1. Error bars represent 95% confidence intervals.

In more realistic settings, it is often the case that, either due to faults, technical limitations or deliberate choice, not all communication channels between nodes stay open at all times, or that not all nodes participate at every round of communication. We analyse this by assuming each connection or node is active at each point in time with a probability $p$. In Figure 2 we observe that our initialisation method performs favourably compared to the initialisation method of He et al. [33] even at very low values of occupation probability $p$, with each node arriving at an eventually consistent state much earlier.

Figure 2: Mean cross-entropy test loss as a function of communication rounds for a fully connected communication network $n=64$ using the MNIST dataset with 512 items per node. Each (a) connection or (b) node are active at each round with probability $p$. Note that inactive nodes still perform local training, but are in effect momentarily isolated from the network. The proposed initialisation is displayed with solid lines and the independent initialisation method of He et al. [33] with dashed lines. Even at fairly low values of $p$, the system as a whole has a much better overall learning trajectory with our proposed parameter initialisation method compared to that of He et al. [33]. Error bars represent 95% confidence intervals.

To understand the general characteristics of the learning process we propose a simplified numerical model: an iterative process, where each of the $n$ network nodes has a vector of $d$ parameters drawn from a Gaussian distribution with standard deviation $\sigma_{\text{init.}}$ and mean zero. At each iteration, similar to the federated averaging step, each node updates its parameter vector by averaging its immediate neighbourhood, then, to mimic the effects of the local training step all node parameters are updated by adding a Gaussian distributed noise with zero mean and standard deviation $\sigma_{\text{noise}}$.

This model mimics the general behaviour of the decentralised learning system at the early stages of the process, since we can assume that the changes to the parameters as a result of the local learning process are generally negligible compared to the changes in parameters due to the aggregation steps. Simulations of the decentralised federated learning process (Figure 3(a,b)) can provide evidence for this assumption.

Figure 3: (a) Mean magnitude of change in parameters due to training and aggregation independently as well as the total change, as well as the mean cosine similarity of the changes during training and aggregation. In the early rounds of the iterative process, the vector of change due to the aggregation is several orders of magnitude larger than that of the training. Additionally, the cosine similarity trajectory indicates the orthogonality of these vectors in the early rounds, supporting the numeric model assumption that the early evolution of the system is dominated by the aggregation step. Additionally evolution of standard deviation of $\sigma_{an}$ and $\sigma_{ap}$ on (b) the distributed learning process with actual ANNs and (c) the numerical simplified model show similar early-stage dynamics. Values were calculated by (a,b) running or (c) numerically modelling the decentralised federated learning process on random 32-regular $n=256$ networks. Panels (a,b) were performed with 80 training samples per node, 1 epoch per communication round. Error bars represent 95% confidence intervals.

The results from the simplified numeric model for random $k$-regular graphs predict, as shown in Figure 3(c), that the standard deviation of the value of the same parameter across nodes averaged for all parameters, which we call $\sigma_{an}$, will decrease to some value close to the standard deviation of noise (simulating changes due to local training). Meanwhile the standard deviation of the parameters of the same node average across all nodes, $\sigma_{ap}$, will decrease only to a factor of $1/\sqrt{n}$ of the original standard deviation $\sigma_{\text{init.}}$. Note that for artificial neural networks each layer’s initial parameters are usually drawn from distributions with different values of $\sigma_{\text{init.}}$, based on the number of inputs and outputs of each layer and other considerations. The analysis here can be applied to each batch of parameters drawn from the same distribution, e.g., to weights of the same layer, independently.

Two of the dynamics visualised in Figure 3(b,c) stand out in particular. First, the value towards which $\sigma_{ap}$ approaches can allow us to select the initial distribution of parameters $\sigma_{\text{init.}}$ in a way that after stabilisation of $\sigma_{ap}$, the neural network models would on expectation have an optimal parameter distribution. In Section 4.1, we show that this compression can be calculated for any graph based on the distribution of eigenvector centralities of the nodes [7], with the case of graphs with uniform centralities giving a factor of $1/\sqrt{n}$. Second, the time to reach the steady state for $\sigma_{an}$ plays an important role since this determines the number of rounds required before the improvements of the learning process start in earnest. This is because the magnitude of the changes to parameters due to the learning process (modelled by noise in the numerical model) becomes comparable to those of the aggregation process. In Section 4.3 we show that this “stabilisation” time scales similar, up to a constant factor, to the mixing time of lazy random walks on the graph.

Algorithm 1 Decentralised federated training cycle along with theproposed initialisation steps.

1:given number local batches $b$, Optimiser $Opt()$, set of neighbours $\mathcal{N}$.

2:estimate $\left\|v_{\text{steady}}\right\|$ based on a sketch of the communication network, vide Section 4.2.

3:for layer $l$ with parameters $\omega^{l}_{0}$ in $w_{0}$ and activation function $f_{l}$ do

4:     initialise $\omega^{l}_{0}$ based on an initialisation method suitable for the architecture, e.g. He et al. [33]

5:     scale $\omega^{l}_{0}$ by a factor of $\left\|v_{\text{steady}}\right\|^{-1}$

6:end for

7:repeat

8:     $t\leftarrow t+1$

9:     $g_{t-1}\leftarrow\text{BatchGrads}(w_{t-1})$

10:     $w_{t}\leftarrow Opt(g_{t-1},w_{t-1})$

11:     if $t\mod b=0$ then

12:         send parameters $w_{t}$ to neighbours

13:         receive parameters of all neighbours as $w^{i}_{t}\,\forall i\in\mathcal{N}$

14:         aggregate neighbourhood parameters as in Equation 2$\triangleright$ DecAvg aggregation

15:         re-initialise optimiser state.

16:     end if

17:until stop** criteria are met

4.1 The compression of node parameters

For the simplified model we can analytically estimate the steady state values for $\sigma_{an}$ and $\sigma_{ap}$, as well as the scaling of the number of rounds to arrive at these values using methods from finite-state discrete-time Markov chains. Let $A$ be the adjacency matrix of our underlying graph $G$. We construct a right stochastic matrix $A^{\prime}$ where

$$A^{\prime}_{ij}=\frac{A_{ij}+I_{ij}}{\sum_{k}A_{kj}+I_{kj}}\,,$$ (3)

where $I$ is the identity matrix. This corresponds to the Markov transition matrix of random walks on graph $G$, if the random walker can stay at the same node or take one of the links connected to that node with equal probability for each possible action. This formulation can also be seen on the basis of the DecAvg aggregation in Equation 2. If we arrange all initial node parameters in a $d\times n$ matrix $W_{\text{init.}}$, the parameters at round $t$ are determined by $W_{\text{init.}}A^{\prime t}+\sum^{t-1}_{\tau=0}N_{\tau}A^{\prime\tau}$, where $N_{\tau}$ is a random $d\times n$ noise matrix with each index drawn from $\mathcal{N}(0,\sigma_{\text{noise}}^{2})$.

Assuming that the graph $G$ is connected, the matrix $A^{\prime t}$ would converge to a matrix where each row is the steady state vector of the Markov matrix $A^{\prime}$, the eigenvector corresponding to the largest eigenvalue 1, normalised to sum to 1. If the steady state vector is given as $v_{\text{steady}}$ the variance contribution of the term $W_{\text{init.}}A^{\prime t}$ along each row (i.e., expected variance of parameters of each node) is given as $\sigma_{\text{init.}}^{2}\left\|v_{\text{steady}}\right\|^{2}$. It can be trivially shown, from a direct application of the Cauchy–Schwarz inequality $|\langle\vec{1},v_{\text{steady}}\rangle|\leq\|\vec{1}\|\|v_{\text{steady}}\|\,$ that for connected networks $\langle\vec{1},v_{\text{steady}}\rangle=\sum v_{\text{steady}}=1$, the $\left\|v_{\text{steady}}\right\|^{2}$ term has a minimum value of $1/n$. This is achieved for random regular networks and other network models where nodes have uniformly distributed eigenvector centralities, such as Erdős–Rényi networks and lattices on $d$-dimensional tori.

Given that the noises are drawn independently, the variance contribution of the noise term $\sum_{\tau=0}^{t-1}N_{\tau}A^{\prime\tau}$ has an upper-bound of $t\sigma^{2}_{\text{noise}}$. If $t\sigma^{2}_{\text{noise}}\ll\sigma^{2}_{\text{init}}/n$, then the standard deviation across parameters at round $t$ can be approximated by $\lim_{t\rightarrow\infty}\sigma_{ap}\approx\sigma_{\text{init.}}\left\|v_{\text{steady}}\right\|$. For a large connected random $k$-regular network, this reduces to $\lim_{t\rightarrow\infty}\sigma_{ap}=\sigma_{\text{init.}}/\sqrt{n}$. Generally for other networks, numerical solution for $\left\|v_{\text{steady}}\right\|$ can be obtained by calculating sum-normalised eigenvector centralities of the original network after adding self-loops to all nodes, with weights equal to the node degree.

These results, combined with the existing analyses on the role of artificial neural network parameters and their effect on diminishing or exploding gradients [33] suggest that it is reasonable to take into account the compression of the node parameters (e.g., the $1/\sqrt{n}$ factor for $k$-regular networks) when initialising the parameters, and we show in Section 4.2 that even a rough estimate of this factor would be quite effective in practice. Depending on the choice of architecture and optimiser, and especially for large networks with hundreds of nodes, this re-scaling of initial parameters can play a sizeable role in the efficacy of the training process. In our experiments, we took this into account by multiplying a gain factor of $\sqrt{n}$ in the standard deviation of layer parameters suggested by He et al. [33].

Note that $v_{\text{steady}}$ is simply the sum-normalised vector of eigenvector centralities of the communication network nodes with a self-loop added to all nodes with a weight equal to the degree of that node. Each element of that vector specifies the probability of a random walk to end up on that specific node, if the random walk process has equal probability of taking any one of the edges or staying on the node. This means that the value of this gain is a factor of the system size and the distribution of network centralities. The practical application of this is explored in Section 4.2.

The numeric model applies with minimal changes to directed and weighted communication networks, similar to connected undirected networks. In the case of a strongly connected directed network, the convergence is guaranteed since the stochastic matrix $A^{\prime}$ is aperiodic due to the existence of the self-loops. For the case of a weighted communication network, the weights are reflected in the graph adjacency matrix $A$, with the provision that a diagonal matrix of the weights each node assigns to its own weights should be used in Equation 3 instead of the identity matrix $I$.

4.2 Estimating parameter scaling factor $\left\|v_{\text{steady}}\right\|$

Calculating the scaling factor $\left\|v_{\text{steady}}\right\|$ based on a perfect knowledge of the entire communication network is trivial. In real-world scenarios, however, it is often the case that we can only rely on each node’s imperfect knowledge of the connectivity network during initialisation. In this section, we explore a few scenarios to illustrate how this affects the process described.

Often, while the full topology of the communication network might not be known to each node, the network might have emerged as a result of a central organising principle. For example, assume that the communication network is formed through a peer discovery system where a new node is assigned to be connected to existing nodes. If the assignment probability is a linear function of their current degree (i.e., a node with more neighbours is more likely to be recommended to a new node) then the resulting network would be an example of the preferential attachment process, with a power-law degree sequence [35]. Similarly, if the communication networks are based off of real-world social relationships or physical distance of the nodes, then the network would have properties similar to those observed in social or geometric networks, respectively. If such information about the network formation principles are known beforehand, fewer variables need to be estimated.

Let’s take, for example, a communication network formed based on randomly establishing connections between two nodes, or formed one based on a Barabási–Albert preferential attachment process. In these cases, with only an estimate of the number of nodes $n$, it is possible to estimate the scaling factor $\left\|v_{\text{steady}}\right\|$. While this information is not necessarily available to all nodes, a simple application of a gossip protocol can provide an estimate to all nodes. It is important to note that the estimate for $n$ need not be exact. Figure 4(a) shows that even a significant over- or underestimation of the number of nodes would still advantage our proposed initialisation method.

Figure 4: The effect of incomplete information in form of over- or under-estimating (a) the number of nodes or (b) the $\left\|v_{\text{steady}}\right\|$ size exponent (vide Figure 5(a,b)) on our proposed initialisation method. Note that our proposed initialisation still performs significantly better than unscaled independent initialisation, shown with dashed lines. Performed on the MLP configuration with MNIST dataset, on fully connected network. Error bars represent 95% confidence intervals

If no information on the network topology is known in advance, it is possible to arrive at a best guess by polling a sample of the network (perhaps through a gossip protocol) for a degree distribution. The scaling of $\left\|v_{\text{steady}}\right\|$ with system size in random networks with different distributions are illustrated in Figure 5 where the value of the scaling factor exponent is derived for Erdős–Rényi, $k$-regular, Barabási–Albert and heavy-tail degree distribution configuration model random networks with the same size and (on expectation) the same number of links. The value of $\left\|v_{\text{steady}}\right\|$ is simply $n^{-\alpha}$ where $\alpha$ is the scaling factor exponent in Figure 5(a,b).

Furthermore, we show empirically that $\left\|v_{\text{steady}}\right\|$ is independent of the degree assortativity of the network. This is done by rewiring a network using the edge swap method, i.e. selecting two edges and swap** the endpoints. This is performed through simulated annealing: a specific target value for degree-assortativity is set and random edge-swaps are accepted or rejected based on their utility and a temperature variable that decreases slowly over time, until the network converges to the desired target assortativity. The results in Figure 5(c) show that $\left\|v_{\text{steady}}\right\|$ stays the same after rewiring.

Figure 5: (a,b) The effect of heterogeneous distribution of centralities in the scaling factor $\left\|v_{\text{steady}}\right\|$ from the simplified numerical model. Homogeneous random networks (Erdős–Rényi $G(n,p)$ networks and random $k$-regular networks) display $\left\|v_{\text{steady}}\right\|\approx 1/\sqrt{n}$, while Barabási–Albert networks and configuration model heavy-tail degree distribution networks (with degree distribution $p(k)\sim k^{-\gamma}$) show this factor scaling exponentially with the number of nodes with different exponents that is itself a function of $\gamma$. (c) Degree distribution preserving rewiring of Erdős–Rényi network ($n=2048$) to produce networks with various values of degree assortativity $\rho$ shows that $\left\|v_{\text{steady}}\right\|$ is not affected by degree assortativity. Error bars represent 95% confidence intervals.

4.3 Initial stabilisation time

The stabilisation time of $\sigma_{an}$, the number of communication rounds until the blue curve in Figure 3(b,c) flattens out, determines the number of rounds where local training has a negligible effect on the parameters. Understanding the scaling of stabilisation time with the number of nodes and other environmental parameters is important, as before this stabilisation the aggregation process dominates the local training process by several orders of magnitude (Figure 3(a)), inhibiting effective training.

Deriving the scaling of number of stabilisation rounds with number of nodes $n$ is a matter of calculating the mixing time of the Markov matrix $A^{\prime}$. The problem is remarkably close to the lazy random walk, where at each step the walker might stay with probability $1/2$ or select one of the links for the next transition. However, in our case this staying probability is lower or equal to that of lazy random walk, being equal to $1/(k_{i}+1)$ where $k_{i}$ is the degree of node $i$. It has been shown that, since the staying probabilities are bounded in $(0,1)$, the mixing time of the random walk process described here grows asymptotically with that of lazy random walk up to a constant factor [36, Corollary 9.5].

Mixing time of lazy random-walks on graphs is a subject of active study. Lattices on $d$-dimensional tori have a mixing time with an upper bound at $d^{2}l^{2}$ [37, Theorem 5.5] where $l\propto n^{1/d}$ is the linear system size. Connected random $k$-regular networks, as expander graphs, have a mixing time of $O(\log n)$ [38, 39], while connected supercritical Erdős–Rényi ($G(n,m)$ and $G(n,p)$) graphs (with average degree larger than 1) have lazy random walk mixing times of $O(\log^{2}n)$ [40, 41]. Generally, for a given Markov matrix the convergent rate can be estimated based on its spectral gap [42].

5 Conclusion

Here we introduced a fully uncoordinated, decentralised artificial neural network initialisation method that provides a significantly improved training trajectory, while solely relying on the macroscopic properties of the communication network. We also showed that the initial stages of the uncoordinated decentralised federated learning process are governed by dynamics similar to those of the lazy random walk on graphs. Furthermore, we also showed empirically (Appendix B) that when using the proposed initialisation method, the test loss of the decentralised federated learning system can approach that of a centralised system with the same total number of training samples, and that the final outcome, in terms of the best test loss achieved, is fairly robust to different network densities and momentary communication failures, and it can benefit from more frequent communication between the nodes.

5.1 Limitations

In this work, we have not considered an unequal allocation of computation power among the nodes to focus solely on the role of the initialisation and the network. In real-world settings, these are often combined or correlated with network properties such as the degree or other centrality measures, which might affect the efficacy of the decentralised federated learning process. Understanding the combination and interactions of these properties with network features adds another layer of interdependency and complexity to the problem, which most certainly was not addressable without first studying the simpler case presented here. The prospect of extending this work to these more complex settings is interesting to consider.

The federated learning process presented here does not support heterogeneous machine learning architectures between nodes. We expect this to become more prominent with the advances in edge computing and device availability. We also did not consider possible heterogeneities in node-to-node communication patterns, such as burstiness or diurnal pattern, which has been shown to affect the rapidity of other network dynamics like spreading and percolation processes [23, 25, 26].

Additionally, some artificial neural network architectures utilise batch normalisation [43], which seemingly greatly limits (but does not eliminate [33]) the vanishing/exploding gradients issue that necessitates careful parameter initialisation. It is important to note that this choice greatly reduces options in the choice of architecture or risks introducing gradient explosion at initial training steps, making deep networks of arbitrary structure prohibitively difficult to train [44]. Careful parameter initialisation, on the other hand, provides a more generalisable solution.

Our work enables uncoordinated decentralised federated learning that can efficiently train a model using all the data available to all nodes without having the nodes share data directly with a centralised server or with each other. While this enables or streamlines novel use cases, it is important to note that trained machine learning models themselves could be exploited to extract information about the training data [45, 46]. It is therefore important to not view federated learning as a panacea for data privacy issues but to view direct data sharing as the weakest link in data privacy.

Appendix A Datasets, implementation and experimental architecture

The artificial neural network architectures employed in this manuscript are a simple feedforward neural network with four fully connected layers, consisting of 512, 256, and 128 neurons in three hidden layers, followed by an output layer of size 10, and employs ReLU activation functions after each layer except the output layer, a simple feedforward neural network consisting of 3 2D convolutional layers with 32, 64 and 64 output channels, each with 3 kernels and one pixel padding of zeros, followed by two fully connected linear hidden layers of size 128 and 64 and one output layer and the VGG16 architecture [47]. Of course, as discussed in the literature, the effects of the initialisation method would be even more visible in deeper neural network architectures [33, 32].

Our experiments will be performed on subsets of the MNIST digit classification task [48], the So2Sat LCZ42 dataset for local climate zone classification [49] and the CIFAR-10 image classification dataset [50], distributed between nodes either iid or non-iid based on a Zipf distribution. In terms of local optimisation, we tested stochastic gradient decent with momentum and Adam with decoupled weight decay, although empirical evidence (Figure 3) hints that the the effects of local optimisation are only non-negligible compared to those of the aggregation at a longer time-scale than the one mainly of interest in this manuscript.

The MNIST dataset was released under the Creative Commons 3.0 Attribution-ShareAlike license. Available at http://yann.lecun.com/exdb/mnist/.

The So2Sat LCZ42 dataset was released under the Creative Commons 4.0 Attribution licence, at https://mediatum.ub.tum.de/1613658. In our use case, we used a random subset of the random split of the third version, only including the 10 bands from the Sentinel-2 satellite to artificially simulate a more realistic, data-poor scenario.

The CIFAR-10 (Canadian Institute for Advanced Research, 10 classes) dataset was released under the Creative Commons 0 version 1.0, available at https://www.cs.toronto.edu/~kriz/cifar.html.

Table 1 shows in brief the infromation about the configurations used for producing figures in the manuscript.

Table 1: Configurations used in this manuscript. Stochastic gradient descent used momentum $m=0.5$, while Adam optimiser with decoupled weight decay was initialised with parameter $\beta_{1}=0.9$, $\beta_{2}=0.999$, $\epsilon=10^{-8}$ and $\lambda=10^{-2}$. Both Optimisers used a learning rate of $10^{-3}$. All configurations used a minibatch size of 16 and 8 minibatches of local training per communication round.

Cfg.	Dataset	Architecture	Comm. net.	Optimiser	Data dist.	Items per node
A	MNIST	MLP	Full	SGD	iid	512
B	So2Sat	CNN+MLP	BA (m=8)	SGD	Zipf ($\alpha=1.8$)	1024
C	CIFAR-10	VGG-16	4-regular	SGD	iid	512
D	MNIST	MLP	Full	AdamW	iid	512

Runtimes for and configuration used is reported in Table 2 for the purpose of reproduction.

The implementation of the full-fidelity simulated decentralised federated learning system is available for the purposes of reproduction under the MIT open-source license at https://anonymous.4open.science/r/sat-9D08.

Table 2: Median runtime (in minutes) for a single realisation of each configuration, and the total runtime of all realisations for each configuration. Scaling refers to Figure 1, Estimates to Figure 4, and Probs to Figure 2. Runtimes were measured on AMD MI250X GPUs with two realisations running concurrently per graphics compute die (GCD) to increase efficiency. Configuration labels refer to Table 1. Note that if the preliminary experiments and the experiments presented solely in the appendices are taken into account, the total computation costs increases to roughly 10000–12000 GCD-hours.

Configuration	Size (nodes)	Median runtime (mins)	Total runtime (hours)
A (Scaling)	8	9.3	10.1
	16	20.1	22.4
	32	51.2	55.1
	64	137.1	148.3
A (Estimates)	64	136.2	642.4
A (Probs)	64	77.4	1466
B (Scaling)	32	95.0	405.2
	64	177.7	757.7
	128	340.2	1499.4
	256	716.8	1530.1
C (Scaling)	8	47.5	25.7
	16	92.9	49.7
	32	167.1	90.0
	64	314.9	169.87
D (Scaling)	8	10.54	5.5
	16	23.05	12.0
	32	51.42	27.3
	64	126.8	68.2
Total runtime			6985.0
Total GCD-hours			3492.5

Appendix B Scalability and the role of exogenous and endogenous decentralised federated learning parameters

As shown before in Figure 1, the choice of initialisation strategy significantly affects the behaviour of the system when varying the environment parameter such as the number of nodes. In this section, we will briefly discuss the effect of network topology on the learning trajectory of the system, then systematically analyse the role of different environmental parameters such as the system size (number of nodes), the communication network density, the training sample size and the frequency of communication between nodes in the trajectory of the decentralised federated learning, when using the initialisation method proposed in Section 4. As most of these quantities are involved in some form of cost–benefit trade-off, understanding the changes in behaviour due to each one can allow a better grasp of the system behaviour at larger scales.

For the rest of this section, however, we limited the analysis to a single topology, random $k$-regular networks, to focus on a more in-depth analysis of the role of environmental parameters other than the network topology, such as the system size, frequency of communication, and network density.

For the purposes of this section, we make the simplifying assumption that the communication time is negligible compared to the training time. In some cases, we introduce “wall-clock equivalent” values, indicating the computation time spent by an individual node up to communication round $t$, multiplied by the number of training mini-batches of training between two rounds of communication. This “wall-clock equivalent” can be seen as a linear scaling of the communication rounds $t$.

Figure 6: Trajectory of mean test cross-entropy loss over communication rounds for (a) connected random $k$-regular networks with $n=64$ nodes and different values for degree $k$, with 80 balanced training samples per node, (b) 32-regular random network with different number of total labelled training samples, balanced across classes, assigned to each item, (c) with different number of nodes and (d) with different number of local epochs between communications. In all panels, the horizontal dashed lines correspond to the best test loss of a central system with the same amount of the total number of training samples as the entire decentralised federated learning system simulated. Error bars represent 95% confidence intervals. The horizontal axes in (b,d) are scaled to show the “wall-clock equivalent”, a value linearly comparable to the total computation cost of a single node until round $t$.

Network density.

The number of links in the communication network directly increases the communication burden on the nodes. Our results (Figure 6(a) show that while a very small value for the average degree affects the rapidity of the training convergence disproportionately, as long as the average degree is significantly larger than the critical threshold for connectivity, i.e., for random network models with average excess degree $\langle q(k)\rangle\gg 1$, the trajectory will be quite consistent across different network densities. Note that, although in Section 4.3 we were mostly concerned with the scaling of the initial mixing time with the number of nodes, in many cases this would also benefit from a higher average degree. Also note that average degrees close to the critical threshold might not prove practical or desirable for the communication network in the first place, as the network close to the critical threshold is highly susceptible to fragmentation with the cutting off of even very few links.

Training samples per node.

Assuming that each device is capable of performing training on a constant rate of mini-batches per unit of time, more training samples per node increase the total amount of training data, while also linearly increasing the training time for every epoch. Our results (Figure 6(b)) show that (1) the test loss approaches that of a centralised system with the same number of total training samples, and (2) that the trajectory of test loss with effective wall-clock time remains consistent.

System size and total computation cost.

The number of nodes in the network affects the training process in multiple ways. If a larger system size is synonymous with a proportionate increase in the total number of training samples available to the system as a whole, it is interesting to see if the system is capable of utilising those in the same way as an increase in the number of items per node would. Our results (Figure 6(c)) show that if the increase in size coincides with an increase in the total number of items, the system is able to effectively utilise these, always approaching the test loss limit of a centralised system with the same total data.

Another aspect is that an increase in the number of nodes would mean an increase in the total computation cost, so it would be interesting to analyse if this increase (without a corresponding increase in the total amount of data) would result in any improvements in the learning trajectory. In short, our results in Figure 7 show that if the same amount of data is spread across more nodes, each node will have to train on roughly the same number of minibatches to arrive at a similar test loss, and that this result is even consistent with the learning trajectory of the centralised single node scenario.

Figure 7: (a) Trajectory of mean test cross-entropy loss over wall-clock time equivalent over 32-regular random graphs and for an isolated node while kee** the total number of training samples across the whole system constant. Each node was assigned training samples balanced across 10 classes, with a total of 40 960 training samples divided equally across the nodes. Error bars represent 95% confidence intervals. The horizontal dashed line corresponds to the best test loss of a central system with the same total amount of training samples as the entire decentralised federated learning system simulated. The sloped dashed line shows the power-law trajectory of loss with equivalent to wall-clock time consistent with results from Henighan et al. [51]. (b) “Slowdown” as a result of splitting the same total amount of training samples across more nodes. Values are calculated as $\tau_{1}(0.11)/\tau_{n}(0.11)$ where $\tau_{n}(l)$ is the wall-clock time equivalent (values of horizontal axis from panel (a)) required for a system with $n$ nodes to reach cross-entropy loss value $l$.

Communication frequency.

Finally, we consider the role of frequency of communication in the trajectory of loss, manifested as the number of local training epochs between communications. It has been shown in the context of decentralised parallel stochastic gradient descent that a higher frequency of communications increases the efficacy of the training process, as it prevents a larger drift [13]. While a similar phenomenon in the context of an uncoordinated decentralised federated learning seems plausible, showing this relationship empirically on a system of reasonable size was fraught with difficulties due to the issues discussed in Figure 1. Utilising the proposed initialisation method enables this and allows us to confirm (Figure 6(d)) that while more frequent communication increases the communication burden on the entire network, more frequent communication translates to both a lower final test loss as well as faster convergence.