Stochastic gradient descent-based inference for dynamic network models with attractors

Abstract

In Coevolving Latent Space Networks with Attractors (CLSNA) models, nodes in a latent space represent social actors, and edges indicate their dynamic interactions. Attractors are added at the latent level to capture the notion of attractive and repulsive forces between nodes, borrowing from dynamical systems theory. However, CLSNA reliance on MCMC estimation makes scaling difficult, and the requirement for nodes to be present throughout the study period limit practical applications. We address these issues by (i) introducing a Stochastic gradient descent (SGD) parameter estimation method, (ii) develo** a novel approach for uncertainty quantification using SGD, and (iii) extending the model to allow nodes to join and leave over time. Simulation results show that our extensions result in little loss of accuracy compared to MCMC, but can scale to much larger networks. We apply our approach to the longitudinal social networks of members of US Congress on the social media platform X. Accounting for node dynamics overcomes selection bias in the network and uncovers uniquely and increasingly repulsive forces within the Republican Party.

Keywords: Longitudinal social networks; Attractors; Partisan polarization; Dynamic networks analysis; Co-evolving network model

1 Introduction

We consider the problem of modeling dynamic networks, a collection of network graphs $G_{t}$ indexed over times $t$. In particular, we focus on a specific class of temporal network models known as CLSNA models, first developed in Zhu et al. (2023). In a CLSNA model, the relations and interactions between nodes and certain attributes of the nodes influence each other as they co-evolve over time.

Latent space network models are a commonly used class of models for static networks (Hoff et al., 2002) where the probability of a relation between actors depends on the positions of individuals in an unobserved “social space” or latent space. Several dynamic latent space network models have been proposed in the literature, which involve embedding nodes of temporal networks into a latent Euclidean space, thus allowing each actor to have a temporal trajectory in the latent space. Earlier approaches modeling the transition and evolution of actors dictate that latent positions evolve over time in a Markov fashion (Sarkar and Moore, 2005; Sewell and Chen, 2015a, b, 2016). In the CLSNA model (Zhu et al., 2023), temporal evolution of latent positions depends on network connectivity via the presence of attractors (a concept that is fundamental to dynamical systems) at the latent level. The CLSNA model has been shown to be effective at disentangling positive (attractive) and negative (repulsive) forces among political elites and the public when applied to longitudinal social networks from the social media platforms X and Reddit (Zhu et al., 2023).

The CLSNA model has adeptly captured the nuances of polarization in American politics, offering valuable insights into the past decade. Despite its efficacy, the model’s potential could be further realized by advancing the inference methods to address the challenges of scalability. The challenges of scalability arise in two key aspects. The first aspect is an increasing number of nodes. The second is changes to sets of nodes over time.

The first aspect addresses the reliance on MCMC, as used in Zhu et al. (2023), for estimating model parameters and latent positions. This approach, even though it is quite accurate, becomes prohibitively computationally expensive when scaling to networks with more than a couple hundred nodes. Various methods have been proposed to reduce computational costs for dynamic latent position network models. Sarkar and Moore (2005) introduced a two-stage procedure to optimize the likelihood: multidimensional scaling followed by conjugate gradient update rule. Raftery et al. (2012) used the case-control likelihood approximation to speed up the estimation algorithm. In Liu and Chen (2021), the authors propose to use a variational inference algorithm for estimating the dynamic latent position of the network model.

For the second aspect, the original formulation of CLSNA models assumes that all nodes are present at all time points. In the real world of dynamic networks, the sets of nodes tend to change over time. Sewell and Chen (2015b) develop a model for dynamic networks in which only a subset of edges are observed and the edges are missing at random. Zhu et al. (2023) handles the issue by pre-processing the data and kee** a subset of congress members who are present during the whole period of time studied.

In this paper, we directly address the above two challenges of scalability. Specifically, we (i) develop a Stochastic Gradient Descent (SGD) parameter estimation method that allows us to scale to larger networks; (ii) accompany that by a novel corresponding approach to variance estimation that builds on the notion of Laplace approximation; and (iii) implement these advances within an extension of the CLSNA framework that allows nodes to join and leave the network. Motivated by Laplace’s approximation we replace the true posterior probability distribution by a Gaussian, with the mean at the MAP solution and precision matrix determined by the observed Fisher information. Rue et al. (2009) developed the class of INLA algorithms to perform Laplace’s approximation when the precision matrix is sparse, for instance in the case of the conditional independence network structure in state-space models and Gaussian Markov Random Fields (GMRFs). However, in the class of CLSNA models that we study here, the assumption of sparsity does not remain applicable, necessitating the novel development we offer here.

We use the Congressional Hashtag Network from the platform formerly known as Twitter, now referred to as “X”, as detailed in Zhu et al. (2023). The network was constructed from tweets by US congress-persons from 2010 to 2020. Each year, a binary network was formed, wherein nodes represented sitting members of Congress. Edges between two nodes indicated that their common hashtag usage exceeded the annual average among all pairs of congresspersons. We will revisit the X network to highlight the importance of the two aspects of scalability we propose in this paper: a flexible extended model that accommodates varying sets of nodes, and a fast and scalable SGD-based model inference method. We note that the simplicity and effectiveness of the proposed SGD-based inference and uncertainty quantification developed in this paper is potentially a valuable methodology for generic statistical models. We plan to investigate this direction in future works.

The rest of the paper is organized as follows. Section 2 reviews the standard CLSNA model in Zhu et al. (2023) and presents the extended CLSNA model that enables us to accommodate real world dynamic networks with nodes entering and leaving the networks. Section 3 introduces the two-stage SGD-based inference algorithm: the SGD-based point estimation method and SGD-based variance estimation method. Section 4 provides results from simulations with factorial designs to assess the capability of the proposed algorithm under different settings. In Section 5, we report the results from fitting the extended model to the X Congressional Hashtag Network.

2 Extended CLSNA model

The CLSNA model of Zhu et al. (2023) assumes that all nodes are present at all time points. In the real world of dynamic networks, the sets of nodes tend to change over time. Take the X Congressional Hashtag Network as an example. Figure 1 shows numbers of re-elected and newly elected Democratic and Republican congress members in the X network. We observe significant number of changes in the sets of nodes present at each time because congress members only serve a fixed-year term before they are considered for reelection and because newly elected or sitting congress members decided to open an X account.

In order to use the original CLSNA model, Zhu et al. (2023) keep a subset of congress members who were in office and on X during the whole period of time. After applying this filtering criteria, the number of nodes was reduced from around 500 to 207.

By develo** a model that accommodates nodes entering and leaving the network, we could make inference based on the full data set. As a result, we are able to obtain latent position estimates at each time and estimates of the attractive or repulsive forces between a given node and its neighbors (influencing its movement over time) that are unbiased by factors like time in office and early technology adoption.

Figure 1: Numbers of re-elected and newly elected Democratic and Republican congress members in the X hashtag network

2.1 Model Definition

Let $G_{t}=(V_{t},E_{t})$ be a network evolving in (discrete) time $t$, with vertex set $V_{t}$ and edge set $E_{t}$. We allow for $V_{t}$ to vary over time. Let $Y_{t}$ be the (random) adjacency matrix at time $t$ corresponding to $G_{t}$. Data is in the form of time series of adjacency matrices $\{y_{t}:t=1,\cdots,T\}$, where $y_{t,ij}=1$ if there is an edge between node $i$ and node $j$ at time $t$ and $y_{t,ij}=0$ otherwise.

We model $Y_{t}$ using a latent space approach, with attractors added in the latent level to capture the notions of attractive and repulsive forces. In addition, we want to accommodate nodes entering and leaving the network. Let $z_{t,i}\in\mathbb{R}^{p}$ be a time-indexed latent (i.e., unobserved) position for node $i$ in $p$-dimensional Euclidean space, and let $z_{t}=\{z_{t,i}\}$. Assume that each of the $N_{t}$ nodes falls into one of two groups, i.e., Democratic and Republican, with known node label $\pi(i)\in\mathcal{C}$ for node $i$, where $\mathcal{C}=\{1,2\}$ is the set of group labels.

Formally, we define our model as follows:

$$Y_{t,ij}|p_{t,ij}\sim Bernoulli(p_{t,ij})$$ (1)

where at time $t=1$,

$$\hbox{logit}(p_{t,ij})=\alpha-s(\bm{z}_{t,i},\bm{z}_{t,j})$$ (2)

$$Z_{t,i}\sim Normal(0,\tau^{2}I_{p})\enskip,$$ (3)

and at time $t\geq 2$, if node $i$ is absent at time $t-1$,

$$\hbox{logit}(p_{t,ij})=\alpha+\delta Y_{t-1,ij}-s(\bm{z}_{t,i},\bm{z}_{t,j})$$ (4)

$$Z_{t,i}\sim Normal(\mu_{t,i},\phi^{2}I_{p})$$ (5)

$$\mu_{t,i}=\overline{z}^{\pi(i)}_{t-1}\enskip,$$ (6)

or else at time $t\geq 2$, if node $i$ is present at time $t-1$,

$$\hbox{logit}(p_{t,ij})=\alpha-s(\bm{z}_{t,i},\bm{z}_{t,j})$$ (7)

$$Z_{t,i}|Z_{t-1,i}=z_{t-1,i}\sim Normal(\mu_{t,i},\sigma^{2}I_{p})$$ (8)

$$\mu_{t,i}={z}_{t-1,i}+\gamma^{\pi(i)}_{w}A_{i}^{w}(z_{t-1},Y_{t-1})+\gamma_{b}A_{i}^{b}(z_{t-1},Y_{t-1})\enskip.$$ (9)

Here $s(\cdot,\cdot)$ is a similarity function, and $A_{i}^{w}$ and $A_{i}^{b}$ are the two attractor functions for node $i$ in $Y_{t-1}$. Euclidean distance is used as the similarity function, $s(z_{t,i},z_{t,j})=||z_{t,i}-z_{t,j}||_{2}$, and the two attractors for node $i$ are defined as follows,

$$A_{i}^{w}(\bm{z}_{t-1},Y_{t-1})=\bar{\bm{z}}^{1}_{t-1,i}-\bm{z}_{t-1,i},\,\bar{\bm{z}}^{1}_{t-1,i}=\frac{1}{|S_{1}^{(i)}|}\sum_{j\in S_{1}^{(i)}}\bm{z}_{t-1,j}$$ (10)

$$A_{i}^{b}(\bm{z}_{t-1},Y_{t-1})=\bar{\bm{z}}^{2}_{t-1,i}-\bm{z}_{t-1,i},\,\bar{\bm{z}}^{2}_{t-1,i}=\frac{1}{|S_{2}^{(i)}|}\sum_{j\in S_{2}^{(i)}}\bm{z}_{t-1,j}\enskip,$$ (11)

which are the discrepancies of $\bm{z}_{t-1,i}$ from two local averages at time $t-1$. The sets $S_{1}^{(i)}$ and $S_{2}^{(i)}$ are defined based on group membership and network connectivity as follows:

1. $S_{1}^{(i)}=\{j\in\mathcal{N}\setminus i\,|\,Y_{ij}=1,\pi(i)=\pi(j)\}$, neighbors of node $i$ in the same group

2. $S_{2}^{(i)}=\{j\in\mathcal{N}\setminus i\,|\,Y_{ij}=1,\pi(i)\neq\pi(j)\}$, neighbors of node $i$ in a different group.

Note that even though it is not explicitly mentioned in the notation, the sets $S_{1}^{(i)}$ and $S_{2}^{(i)}$ also depend on time because the connectivity sets are allowed to change with time. When $S_{1}^{(i)}$ or $S_{2}^{(i)}$ or both are empty, we set $A_{i}^{w}(\bm{z}_{t-1},Y_{t-1})=0$, or $A_{i}^{b}(\bm{z}_{t-1},Y_{t-1})=0$ or both to be zero respectively. This implies that if node $i$ is not connected to any members of a specific group, it receives no force from that group.

Note that we automatically recover the original CLSNA model when $V_{t}\equiv V$ is fixed over time. In this case, at time $t\geq 2$, a node $i$ is always present at time $t-1$ and (4) and (5) are redundant in that case.

In this proposed extended model, each node lies in a $p$-dimensional Euclidean latent space. The smaller the distance between two nodes in the latent space, the greater their probability of being connected, as in (1), (2), (4), and (7). The expressions in Eqs. (10) and (11) capture the discrepancy between the current latent position of node $i$ and the average of that of its current neighbors in groups $1$ and $2$, respectively. The corresponding parameters $\gamma^{w}_{1},\gamma^{w}_{2},$ and $\gamma^{b}$ represent attractive and repulsive forces respectively. CLSNA model allows the network connectivity to enter the temporal evolution of latent positions in the form of attractors. Specifically, in the proposed model the evolution of latent positions for each node $i$ from $t-1$ to $t$ is modeled by the normal transition distribution in Eq. (9), the mean vector of which depends not only on the latent position of itself at time $t-1$, but also on the two local averages, one from its neighbors in the same group, the other from its neighbors in a different group, as captured in (10) and (11). Strength of attraction and repulsion toward local averages is summarized by the attractor functions and the associated parameters. The parameter $\delta$ captures edge persistence. For $\delta>0$, the probability of an edge at time $t$ will be increased when one exists already at time $t-1$.

The extended model is a natural extension of the original model. If there are no nodes entering or leaving the networks, the extended model is identical to the original model of Zhu et al. (2023). When the sets of nodes change over time, the model classifies nodes into two categories and models the two types of nodes separately:

1. For actors who are “retained” from time $t-1$ to $t$, the manner in which network connectivity influences the temporal evolution of latent positions remains consistent with the original model: the evolution of latent positions for the node $i$ from $t-1$ to $t$ is modeled by the normal transition distribution, the mean vector of which is a linear combination of the latent position of itself at time $t-1$ and the mean of both its neighbors in the same group and its neighbors in a different group.

2. For an actor that is present at time $t$ but is absent at time $t-1$, we cannot make an educated guess of its current position at time $t$ because of its absence at time $t-1$. For this reason we choose to model the prior distribution of its latent position as a normal distribution, the mean vector of which is the mean of all members in the same group at $t-1$, which we denote by $\overline{z}^{\pi(i)}_{t-1}$.

3 SGD-based Inference Method

We develop a two-stage algorithm to make inferences based on the posterior $\pi(\theta,Z_{1:T}|Y_{1:T})$, where $\theta=(\alpha,\delta,\gamma^{w},\gamma^{b})$. At the first stage we use SGD to compute a point estimate. At the second stage, we estimate marginal posterior standard deviations by a novel approach to quadratic approximation of the log-posterior density, which is also based on SGD.

3.1 Point Estimate

The posterior distribution of latent positions and parameters is

$$\pi(Z_{1:T},\theta\,|\,Y_{1:T})$$ (12)

$$\propto P(Y_{1:T},Z_{1:T}\,|\,\theta)\pi(\theta)$$

$$\propto\left(\prod_{t=2}^{T}P(Z_{t}\,|\,Z_{t-1},Y_{t-1})P(Y_{t}\,|\,Z_{t},Y_{t-1})\right)P(Y_{1}\,|\,Z_{1})P(Z_{1})\pi(\theta)$$

$$=\left(\prod_{t=2}^{T}\prod_{i=1}^{N_{t}}\left(P(Z_{t,i}\,|\,Z_{t-1},Y_{t-1})\prod_{j:j\neq i}P(Y_{t,ij}\,|\,Z_{t,i},Z_{t,j},Y_{t-1,ij})\right)\right)\cdot$$

$$\hskip 289.07999ptP(Y_{1}\,|\,Z_{1})P(Z_{1})\pi(\theta).$$

The log posterior distribution (i.e., taking the log of (12)), therefore, can be written as sums of simpler functions. To optimize the posterior function with SGD, at each step, we randomly sample different terms from these sums. The key steps are:

1. Initialize $Z_{1:T},\theta$.

2. Randomly sample indices of different terms from the different summands of the log-posterior distribution function. Compute the randomly sampled summand functions of the log-posterior distribution function.

3. Update $Z_{1:T},\theta$ based on stochastic gradient descent using the output from step 2.

Repeat steps 2-3 until convergence criterion is met. In practice, we use gradient descent when the sample size is small and it is computationally feasible. In our implementation, we used gradient descent with momentum (when the sample size was small) or stochastic gradient descent with momentum (when the sample size was large), see Polyak (1964).

Input: Network time-series $Y_{1:T}$. Current values $\theta^{(k)}$, $Z_{1:T}^{(k)}$ for parameters and latent positions.
      $\lambda$: step size for $Z_{1:T}$, $\lambda$: step size for $\theta$
      Output: $\theta^{(k+1)},Z_{1:T}^{(k+1)}$

1:Set $\theta$, $Z_{1:T}$ to be $\theta^{(k)}$, $Z_{1:T}^{(k)}$.

2:Uniformly sample the terms (factors) from the posterior distribution:

$$\prod_{t=2}^{T}\prod_{i=1}^{N}P(Z_{t,i}\,|\,Z_{t-1},Y_{t-1})\prod_{j:j\neq i}P(Y_{t,ij}\,|\,Z_{t,i},Z_{t,j},Y_{t-1,ij})P(Y_{1}\,|\,Z_{1})P(Z_{1})\pi(\theta)$$ (13)

Denote the sampled terms by $p_{i}$. Define $\pi_{SGD}(\theta,Z_{1:T}|Y_{1:T})$ to be $\prod_{i}p_{i}$

3:Take a SGD step (see also Remark 3.2)

$$Z_{1:T}^{(k+1)}= Z_{1:T}^{(k)}-\lambda\left(\frac{\partial\log\pi_{SGD}(\theta^{(k)},Z_{1:T}^{(k)}|Y_{1:T})}{\partial Z_{1:T}}\right)$$

$$\theta^{(k+1)}= \theta^{(k)}-\lambda\left(\frac{\partial\log\pi_{SGD}(\theta^{(k)},Z_{1:T}^{(k)}|Y_{1:T})}{\partial\theta}\right)$$ (14)

Algorithm 1 SGDstep

Formally, Algorithm 1 is the main building block for the SGD-based point estimation method, which we present as pseudo-code in Algorithm 2.

Input: Network time-series $Y_{1:T}$. Initial values $\theta^{init}$, $Z_{1:T}^{init}$ for parameters and latent positions.
      M: maximum number of iterations, $\lambda$: step size for $Z_{1:T}$, $\lambda$: step size for $\theta$
      Output: $\theta^{(k)},Z_{1:T}^{(k)}$

1:Set initial values $\theta^{(0)}$, $Z_{1:T}^{(0)}$ to be $\theta^{init}$, $Z_{1:T}^{init}$.

2:for $k\leftarrow 0$ to $M$ do

3:     Take a SGD step to optimize the posterior distribution: $(\theta^{(k+1)},Z_{1:T}^{(k+1)})$ = SGDstep($\theta^{(k)},Z_{1:T}^{(k)}$)

4:     Stop if stop** criteria is met

5:end for

Algorithm 2 SGD-based Point estimation Method

Remark 3.1.

In our work using gradient descent, we encountered a recurring challenge. For the majority of time points, the algorithm performs as expected, closely matching the true underlying values in simulation. However, there were a couple of time points at which an anomalous sign inversion occurs in one dimension of the latent position. In particular, if $z_{t,i}=[z_{t,i}^{1},z_{t,i}^{2},\ldots,z_{t,i}^{p}]$ was the true value, it was estimated to be $\tilde{z}_{t,i}=[-z_{t,i}^{1},z_{t,i}^{2},\ldots,z_{t,i}^{p}]$, even though all the estimates for other times match the truth.

This problem is rooted in the complex landscape of our problem, where the model sometimes settles in local modes due to its non-convex nature. The issue with local modes is similar to the issue of label-switching found in the dynamic stochastic block model literature and the issue may not be solved without an extra assumption e.g. that most of the nodes do not change group across two different time steps (Matias and Miele, 2017).

The resolution to this issue that we found to work well is to start with a CLSNA model with latent variables living in a higher dimensional space from the one we intend on having, that is, we aim to fit a model with $Z\in\mathbb{R}^{p}$ but we intentionally choose $Z\in\mathbb{R}^{q}$ with $q>p$ . Then, we use the first $p$ principal components of the vector of the latent position variables for $Z_{1:T}$. This method was the most effective of several alternatives attempted to remedy local modes. Intuitively, one can think of this approach as creating a path between modes thus allowing SGD to travel between modes in the higher dimension. In both the simulated and real data examples, we initially set $p$ to 2 and chose $q$ as 3. We begin by fitting a model with $Z$ in $\mathbb{R}^{3}$, followed by applying principal component analysis (PCA) to the resulting latent positions, thereby compressing $Z$ into $\mathbb{R}^{2}$ while maximizing information preservation. This reduced form is then used as the initial value for fitting a model with $Z$ in $\mathbb{R}^{2}$. This approach, albeit doubling the cost of fitting the model and performing PCA, provides more accurate and faster convergence towards the true values, as it effectively navigates through potential local optima that could hinder convergence in a single-stage fitting process.

Remark 3.2.

Motivated by the work in Hinton (2012), at the initial stage of Algorithm 1 in Eq.(14) we use the sign of the gradient of the global parameters (+1/-1), i.e., we replace (14) by

$$\theta^{(k+1)}=\theta^{(k)}-\lambda\hskip 2.84544pt\text{sign}\left(\frac{\partial\log\pi_{SGD}(\theta^{(k)},Z_{1:T}^{(k)}|Y_{1:T})}{\partial\theta}\right).$$

This is due to the fact that the magnitude and variance of the related gradient terms is much larger than the magnitude and variance of the gradients of the latent positions. This leads to more stability. As we approach the conclusion of the training phase, we switch to Eq.(14) for better accuracy.

3.2 Variance Estimation

Under an MCMC framework, both point estimates for $\theta$ and corresponding uncertainties can be estimated using samples drawn from the posterior. However, our proposed SGD approach above only gives a point estimate for $\theta$, without quantifying uncertainty.

We propose a novel approach to estimate the posterior variance for parameters of interest. It is a method for approximate Bayesian inference based on Laplace’s method. Laplace’s approximation, when applied in scenarios like state-space models and Gaussian Markov Random Fields, leverages the sparsity of the precision matrix as detailed by Rue et al. (2009). However, in more general contexts, such as CLSNA, this assumption of sparsity is not always valid. In response to this, our proposed approach builds upon Laplace’s foundational principles but adapts to the challenges of non-sparse structures. Specifically, it transforms the variance estimation problem into an optimization problem that can also be solved with SGD.

In this section, we present the novel variance estimation algorithm that we propose.

3.2.1 Some Useful Results of Multivariate Normal Distribution

Our variance estimation method exploits the properties of the conditional distribution of a multivariate Gaussian distribution. Let $x$ follow a multivariate normal distribution $x\sim\mathcal{N}(\mu,\Sigma)$, and hence the conditional distribution of a subset vector $x_{1}$, given its complement vector $x_{2}$, is also a multivariate normal distribution $x_{1}|x_{2}\sim\mathcal{N}(\mu_{1|2},\Sigma_{1|2})$ with the conditional mean and covariance given by $\mu_{1|2}=\mu_{1}+\Sigma_{12}\Sigma_{22}^{-1}(x_{2}-\mu_{2})$ and $\Sigma_{1|2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$, respectively. The block-wise mean and covariance matrices are defined as $\mu=\left[\begin{smallmatrix}\mu_{1}\\ \mu_{2}\end{smallmatrix}\right]$ and $\Sigma=\left[\begin{smallmatrix}\Sigma_{11}&\Sigma_{12}\\ \Sigma_{21}&\Sigma_{22}\end{smallmatrix}\right]$.

More specifically, the covariance of the conditional distribution $\text{Cov}(x_{1}|x_{2})$ is a constant as a function of $x_{2}$, indicating that the shape of the conditional distribution is independent of the specific value of $x_{2}$. Additionally, writing $p(x_{1}|x_{2})$ as the p.d.f of the conditional distribution of $x_{1}|x_{2}$, the mean $\mu_{1|2}=\mu_{1}+\Sigma_{12}\Sigma_{22}^{-1}(x_{2}-\mu_{2})$ is the expectation, median, and mode of the distribution. The maximum of the p.d.f. function $p(x_{1}=\mu_{1|2}|x_{2})$ is a constant as a function of $x_{2}$.

Therefore, we can recover the shape of the marginal distribution of $x_{2}$ from the joint distribution by varying $x_{2}$ and following the line $x_{1}=\mu_{1|2}$.

Theorem 3.1.

Given the multivariate Gaussian distribution setting presented above, write $p(x_{1},x_{2})$ as the p.d.f of the joint distribution of $x_{1},x_{2}$. $p(x_{2})$ as the p.d.f of the marginal distribution of $x_{2}$. Then, we have $p(x_{2})\propto p(x_{1}=\mu_{1|2},x_{2})$.

The proof of the theorem is provided in the supplementary file. In particular, the marginal distribution is proportional to the joint distribution evaluated on the curve of the conditional mean of $x_{1}$ given $x_{2}$. Theorem 7.1 provides an expression for the shape of the marginal distribution from the joint distribution, when the joint distribution is a multivariate normal distribution.

Next, we proceed with an important result demonstrating that for a normal distribution the ideas of Theorem 7.1 yield a formula for the variance of $x_{2}$ that we can then be turned into a practical SGD-based algorithm for uncertainty quantification, see Subsection 3.2.2. In particular, recall that in a univariate normal distribution, two key parameters - the mean and variance - describe the distribution fully. By considering two distinct points from the distribution, we can deduce these parameters. In Figure 2 we aim to visualize this idea, which then leads us to Corollary 7.1.1 containing the expression for the variance.

Figure 2: Variance Estimation in Bivariate Normal Distribution

Corollary 3.1.1.

Consider the multivariate Gaussian distribution setting presented above, and let $x_{2}$ be scalar. Then, for some fixed $\tilde{x}_{2}\neq\mu_{2}$,

$$\text{Var}(x_{2})=\frac{1}{2}\frac{(\mu_{2}-\tilde{x}_{2})^{2}}{\log p(x_{1}=\mu_{1},x_{2}=\mu_{2})-\log p(x_{1}=\mu_{1|2},x_{2}=\tilde{x}_{2})}$$ (15)

The proof of the corollary is provided in the supplementary file.

Beyond just the variance represented by the diagonal elements of the covariance matrix, there’s also a formula for the off-diagonal elements. This allows us to reconstruct a single column of the covariance matrix.

Corollary 3.1.2.

Consider the multivariate Gaussian distribution setting presented above. Then, for some fixed $\tilde{x}_{2}\neq\mu_{2}$,

$$\Sigma_{12}=\frac{\Sigma_{22}}{(\tilde{x}_{2}-\mu_{2})}(\mu_{1|2}-\mu_{1}),$$

where $\mu_{1|2}$ denotes the conditional mean of $x_{1}$ given $x_{2}=\tilde{x}_{2}$.

The methodologies established for a generic normal distribution $p(x_{1},x_{2})$ are now adapted to estimate variance in the posterior distribution $\pi(\cdot|Y)$, effectively applying the same theoretical concepts to the Bayesian analysis under a Laplace approximation.

3.2.2 The Variance Estimation Algorithm

Corollary 7.1.1 and Corollary 3.1.2 are the primary components in our proposed SGD-based method for variance estimation of a single parameter, as detailed in Algorithm 3. The method involves two stages. Initially, we compute a point estimate using an SGD parameter estimation method. In the second stage, with this point estimate, we will redo the optimization with the constraint of fixing the parameter to be variance-estimated, adjusting it by a certain $\eta$ perturbation size. Thanks to Corollary 7.1.1 and Corollary 3.1.2, we can then estimate the variance/covariance of this parameter by observing the changes in the posterior function and the changes in the other parameters resulting from the perturbation.

Input: Network time-series $Y_{1:T}$. The mode of $\log\pi(Z,\theta|Y)$, $\theta^{*}$ denoting the mode for all hyperparameters $(\alpha,\delta,\gamma^{w},\gamma^{b})$, $Z_{1:T}^{*}$ for latent positions.
      M: maximum number of iterations, $\lambda$: step size for $Z_{1:T}$, $\lambda$: step size for $\theta$, $\eta_{\alpha}$: perturbation size for $\alpha$       Output: $\widehat{\text{Var}(\alpha|Y)}$(the posterior variance of $\alpha$) (where $\alpha$ is the intercept term in Eq. 2)

1:Set initial values $\theta^{(0)}$, $Z_{1:T}^{(0)}$ to be $\theta^{*}$, $Z_{1:T}^{*}$.

2:for $k\leftarrow 0$ to $M$ do

3:     At the onset of each iteration, the parameter of interest is anchored at a position that’s a fixed perturbation, denoted as $\eta_{\alpha}$, from its modal value. Specifically, we set $\alpha^{(k)}=\alpha^{*}+\eta_{\alpha}$. This action positions the tuple $(\alpha^{(k)},\delta^{(k)},\gamma^{w,(k)},\gamma^{b,(k)})$ at $(\alpha^{*}+\eta_{\alpha},\delta^{(k)},\gamma^{w,(k)},\gamma^{b,(k)})$.

4:     Take a SGD step to optimize the posterior distribution: $(\theta^{(k+1)},Z_{1:T}^{(k+1)})$ = SGDstep($\theta^{(k)},Z_{1:T}^{(k)}$)

5:     Stop if stop** criteria is met and let $k$ be the corresponding iteration.

6:end for

7:

$$\widehat{\text{Var}(\alpha|Y)}=\frac{1}{2}\frac{\eta_{\alpha}^{2}}{\pi(\theta^{(0)},Z^{(0)}_{1:T}|Y_{1:T})-\pi(\theta^{(k+1)},Z^{(k+1)}_{1:T}|Y_{1:T})}$$

8:

$$\widehat{\text{Cov}(\alpha,\theta_{i}|Y)}=\frac{\widehat{\text{Var}(\alpha|Y)}}{\eta_{\alpha}}(\theta_{i}^{(k+1)}-\theta_{i}^{(0)})$$

9:

$$\widehat{\text{Cov}(\alpha,Z_{1:T}|Y)}=\frac{\widehat{\text{Var}(\alpha|Y)}}{\eta_{\alpha}}(Z_{1:T}^{(k+1)}-Z_{1:T}^{(0)})$$

$\text{Cov}(\alpha,Z_{1:T}|Y)$ denote the covariance vector between the 1-dimensional random variable $\alpha$ and the vector random variable $Z_{1:T}$.

Algorithm 3 SGD-based Var/Cov Estimation Method for a single parameter

Algorithm 3 details an SGD-based approach for estimating the variance of a single parameter, in this instance for $\alpha$. To extend this estimation to all parameters of interest, we repeatedly apply Algorithm 3 individually to each parameter. Since we are able to recover the covariance matrix, we are also able to estimate variance of functions of parameters. As an illustration, let us look for example into $\text{Var}(\gamma^{w}-\gamma^{b})$. This estimation is achieved through a sequential application of Algorithm 3 to each component parameter. Initially, the algorithm is applied to $\gamma^{w}$, yielding estimates $\widehat{\text{Var}(\gamma^{w}|Y)}$ and $\widehat{\text{Cov}(\gamma^{w},\gamma^{b}|Y)}$. Following this, a similar application to $\gamma^{b}$ provides $\widehat{\text{Var}(\gamma^{b}|Y)}$ and an alternative estimate of $\widehat{\text{Cov}(\gamma^{w},\gamma^{b}|Y)}$. The variance of the composite parameter $\gamma^{w}-\gamma^{b}$ is then approximated using the formula $\widehat{\text{Var}(\gamma^{w}|Y)}+\widehat{\text{Var}(\gamma^{b}|Y)}-2\widehat{\text{Cov}(\gamma^{w},\gamma^{b}|Y)}$. Given that the function is not exactly quadratic, discrepancies between the two covariance estimates may arise. To mitigate this, one can proceed with averaging these estimates for a more robust estimation.

4 Simulation studies

We simulate two networks, one with $n=100$ actors and one with $n=1000$ actors. For each one of them, we simulate two parameter settings, one for positive attraction or ‘flocking’ ($\gamma^{b}\geq 0$) and the other for repulsion ($\gamma^{b}\leq 0$) or ‘polarization’ between two groups over time. For each setting we simulate $20$ data sets with the number of time points $T=10$.

	$\alpha=1$		$\delta=2$		$\gamma^{w}=0.25$		$\gamma^{b}=0.5$
	$\hat{\alpha}$	$\widehat{\text{Var}(\alpha)}$	$\hat{\delta}$	$\widehat{\text{Var}(\delta)}$	$\hat{\gamma^{w}}$	$\widehat{\text{Var}(\gamma^{w})}$	$\hat{\gamma^{b}}$	$\widehat{\text{Var}(\gamma^{b})}$
n=100	0.817 (0.029)	0.025 (0.001)	1.934 (0.023)	0.024 (0.001)	0.202 (0.135)	0.125 (0.026)	0.489 (0.14)	0.117 (0.011)
n=1000	0.972 (0.003)	0.003 ($<$0.001)	1.992 (0.003)	0.002 ($<$0.001)	0.25 (0.028)	0.03 (0.001)	0.498 (0.030)	0.028 (0.001)
	$\alpha=1$		$\delta=3$		$\gamma^{w}=0.45$		$\gamma^{b}=-0.5$
	$\hat{\alpha}$	$\widehat{\text{Var}(\alpha)}$	$\hat{\delta}$	$\widehat{\text{Var}(\delta)}$	$\hat{\gamma^{w}}$	$\widehat{\text{Var}(\gamma^{w})}$	$\hat{\gamma^{b}}$	$\widehat{\text{Var}(\gamma^{b})}$
n=100	0.825 (0.043)	0.038(0.001)	2.868 (0.045)	0.038 (0.001)	0.302 (0.041)	0.035 (0.003)	-0.54 (0.035)	0.03 (0.004)
n=1000	0.971 (0.003)	0.004 ($<0.001$)	2.976 (0.003)	0.004 ( $<0.001$)	0.433 (0.018)	0.015 (0.004)	-0.509 (0.017)	0.014(0.004)

Table 1: Posterior-based mean (empirical standard deviation) for point estimation (Algorithm 2) and variance estimation (Algorithm 3) in flocking and polarization settings, based on $n=100$ and $n=1000$ nodes, with $T=10$ time points.

We want to compare the mean of the point estimates over $20$ simulations with the truth and we compare the mean of the estimated variances with the standard deviation of the point estimates over $20$ simulations.

With the number of nodes $n=100$, Table 1 shows that the point estimates are reasonably accurate compared with the truth. In the flocking setting, all estimates are slightly biased to be smaller than the truth. In the polarization setting, all estimates except for repulsion between groups ($\gamma^{b}$) are slightly biased to be smaller than the truth. The variance estimates are reasonably accurate and are generally slightly smaller than the true standard deviation (See Table 1). The empirical standard deviations are obtained by 20 MCMC trials in each parameter setting. (See Table 1, where the empirical standard deviations are shown in parenthesis).

Table 1 shows that when the number of nodes is $n=1000$, the point estimates are more accurate than when the number of nodes $n=100$, both in the flocking setting and the polarization setting, and similarly for the variance estimates.

We notice that the estimation of the standard deviation slightly underestimates the true standard deviation. One reason is that the posterior function can have more than one local minimum. This class of variation estimation methods is able to approximate the contribution of one local minimum to the total variance. Note that multiple local minima contribute to total variance, but in our experiments this did not result into serious issues.

We also compare the computational efficiency of the proposed SGD algorithm with MCMC algorithm. When both run on the same CPU resources, the MCMC algorithm took about an hour to obtain 50,000 samples when the number of nodes $n=100$ while the proposed SGD-based algorithm only took fewer than 5 minutes. In the X data analysis in the following section, on the reduced data set with 200 nodes, the MCMC took about four hours (CPU) while the proposed SGD-based algorithm only took fewer than 10 minutes to run(GPU). In addition, the proposed SGD-based algorithm took about 20 minutes when $n=1000$ as well (GPU).

5 X Data Analysis

5.1 Exploratory Analysis

The X Congressional Hashtag Networks dataset in Zhu et al. (2023) was based on 843,907 tweets from 796 US Congresspersons’ accounts from 2010-2020. Yearly binary networks were built from the tweets. Nodes represent members and edges show whether two members used common hashtags more than the year’s average. The nodes in the yearly networks vary as Congress members’ X participation changes due to reelections, late joins, or early departures see Figure 1. Figure 3 shows temporal evolution of edge density within the Democratic and Republican parties individually, alongside the inter-party edge density, as well as the edge density across the entire network. Similar trends were observed in Zhu et al. (2023): connections increased for the first four years for all types of edges. From 2015 to 2020, edge density within the Democratic party continues to rise, while inter-party edge density decays slowly and edge density within the Republican party drops sharply.

Figure 3: Temporal evolution of edge density within the Democratic and Republican parties, the inter-party edge density and the edge density among all members

Figure 1 shows numbers of re-elected and newly elected Democratic congressman and Republican congressman in the X network. We can see an upward trend in the number of X users in both parties. The number of Democratic X users grows more steadily with fewer fluctuation compared with the Republican X users. Notably, Republicans outnumber Democrats in the X network for 9 out of 11 years.

5.2 Implementation Details

We use gradient descent with momentum and choose different learning hyperparameters for the latent position parameters and the global parameters. With gradient descent, we use all the terms in the log posterior function instead of taking a sampled posterior function. We stop running gradient descent when the parameter updates drop below a threshold of $\epsilon=10^{-4}$, a condition we check every 100 gradient steps. Priors for $\alpha$ and $\delta$ were chosen to be $\mathcal{N}(0,100)$ to keep it flat and uninformative. We chose the priors for $\gamma^{w}$ and $\gamma^{b}$ to be $\mathcal{N}(0.5,100)$ and $\mathcal{N}(-0.5,100)$ to reflect the prior belief of polarization, however these are also quite uninformative given the large variance. We fix $\tau^{2}$ at 10, $\sigma^{2}$ at 1 and $\phi^{2}$ at 10.

5.3 Results

Figure 4: The point estimates of the latent positions with the extend model. Blue-Democrats, Red-Republicans. At each time, two parties consistently occupy different halves of the space. The Democrats were flocking during the whole time. Conversely, the Republicans initially displayed a similar flocking behavior, but gradually began to disperse around year 2016.

	$\hat{\alpha}$	$\hat{\delta}$	$\hat{\gamma}_{1}^{w}$	$\hat{\gamma}_{2}^{w}$	$\hat{\gamma}^{b}$
Mean	3.2	1.08	0.34	-0.11	-0.22
SD	0.011	0.010	0.017	0.013	0.0085

Table 2: 1-Democrats, 2-Republicans.The point estimates and the posterior standard deviation of the global parameters in the extended model fitted to the full X data set spanning from 2010 to 2020. The between-group attraction is -0.22, indicating that there is polarization between the two parties. The within group coefficient is 0.34 for Democrats, and -0.11 for Republicans, indicating that the Democrats were flocking, while the Republicans were polarizing within their own group.

We fit our model with time-invariant parameters to the full X data set from year 2010 to 2020. The point estimates and the posterior standard deviation of model parameters are provided in Table 2. Point estimates were produced using the SGD methodology of Section 3.1 and standard deviation estimates were produced using the methodology of Section 3.2. In the supplementary file, we present two diagnostic tests demonstrating that the underlying assumptions of the variance estimation method proposed in Section 3.2 are largely accurate for this dataset, i.e., the diagnostic tests do not show evidence of violating the underlying assumptions of the variance estimation method.

The point estimates of persistence parameter is 1.08, implying that there is a higher likelihood for it to also appear at time $t$ if an edge is present at time $t-1$. The between-group attraction is -0.22, indicating that there is polarization between the two parties. The within group coefficient is 0.34 for Democrats, and -0.11 for Republicans, indicating that the Democrats were flocking, while the Republicans were polarizing within their own group. Importantly, we note that in the original model in Zhu et al. (2023), only members that are present at all times are kept. The parameter estimation for attractions within group indicated that while both parties have moved away from one another, they generally flocked to their own. By incorporating all members into the extended model, we effectively eliminate the potential for selection bias, ensuring a more accurate and comprehensive analysis of the data. As a result of this approach, we have uncovered that the force within the Republican group is negative, in contrast to Zhu et al. (2023).

Figure 4 shows the point estimates of latent positions for each member of Congress in the X hashtag networks. At each time, two parties consistently occupy different halves of the space. The Democrats were flocking during the whole time. Conversely, the Republicans initially displayed a similar flocking behavior, but gradually began to disperse after year 2017.

Figure 5: The trajectory of the mean of the latent positions of the members in each party. The lighter color are results obtained from the reduced data set and the darker color are results obtained from the full data set. In both cases, the average position of the two parties gradually and consistently approached each other from 2011 to 2016 and after 2016, both of them took a U-turn and moved apart.

Figure 5 shows the trajectory of the mean of the latent positions of the members in each party. The lighter color represents results obtained from the reduced data set used in Zhu et al. (2023)’s original analysis. While the darker color represents results obtained with our model and inference techniques from the full data set. In both cases, the average position of the two parties gradually and consistently approached each other from 2011 to 2016. But after 2016, both of them took a U-turn and moved apart. However, we see that the analysis based on the full data set suggests that the Democrats were in fact noticeably closer to the Republicans just before 2016 than suggested by the analysis based on the reduced data.

Figure 6: Comparison of attraction parameters from MCMC method on the reduced data set and the SGD-based method on the reduced and full data set. The MCMC method and the SGD-based method yield very similar point estimates and variance estimates for the attraction parameters on the reduced data set. On the full data set, the model concludes that there is a large attractive force within the Democratic Party. By contrast, we find that within the Republican Party the force is repulsive.

Figure 6 compares attraction parameters with that found in Zhu et al. (2023). The MCMC and the SGD-based method yield close point estimates and variance estimates for the attraction parameters on the reduced data set. The point estimates obtained from the MCMC method fall within the 95% confidence interval of the SGD-based method. On the full data set, the model concludes that there is a large attractive force within the Democratic Party. By contrast, we find that within the Republican Party the force is repulsive.

Figure 7: Evolution of posterior means and 95% CI for Democrats and Republicans, and between-group attraction/repulsion in X Congressional Hashtag Networks, the values of which at each time period in the horizontal axis are obtained by fitting the model using separately parameterized networks in the corresponding time periods.

To analyze change in attraction, repulsion, we fit a series of models that allow a change-point to vary from 2011 to 2019. The resulting fitted models with different change-points were compared based on Bayesian Information Criterion (BIC). The one with the lowest BIC value was selected. We identified 2012 as the year of change-point (See supplementary file for more details on this). Figure 7 illustrates the evolution of within-group attraction and repulsion for Democrats and Republicans, and between-group attraction and repulsion. The within-Republican coefficient is positive in the first time period and is negative in the second time periods from 2012 to 2020. This suggests polarization within Republican members started to rise within the period 2012-2020. The within-Democrat coefficient is positive in both time periods, although its magnitude decreases a bit in the second time period from 2012 to 2020. The between-group coefficient (orange bars) is not statistically significant at the 95% confidence level from 2010 to 2012 and is negative in the second time periods from 2012 to 2020.

6 Discussion

The extended CLSNA model proposed in this paper together with the SGD-based inference method, successfully addressed the challenge of scalability to large networks with nodes entering and leaving the network. In particular, (a) the introduction of the SGD parameter estimation method, (b) the development of the novel variance estimation approach for approximate Bayesian inference which transforms variance estimation into an optimization problem solvable with SGD, and, (c) the extension of the model to allow dynamic node participation, have significantly improved the CLSNA model’s utility for analyzing dynamic networks. When applied to longitudinal social networks on X, the model mitigates selection bias and reveals previously concealed negative force within the Republican party.

The SGD parameter estimation method and the variance estimation method are general approaches to approximate Bayesian inference. Therefore, they can be combined with other strategies to make inference faster. For example, they can be used with case-control log-likelihood approximation as in Raftery et al. (2012). This can be achieved by using stratified sampling to draw the edge probability terms during the stochastic sampling process.

In future work we plan to incorporate observed explanatory variable terms into the edge probability. For instance, a linear predictor based on explanatory variables can be integrated into the edge probability function, see Hoff (2007). In this way, we can more effectively separate the latent effect from the fixed effect and accurately capture the repulsive and attractive force among the different actors.

Supplementary Information for ‘Stochastic gradient descent-based inference for dynamic network models with attractors’

Hancong Pan

Department of Mathematics and Statistics, Boston University

and

Xiao**g Zhu

Department of Mathematics and Statistics, Boston University

and

Cantay Caliskan

Goergen Institute for Data Science, University of Rochester

and

Dino P. Christenson

Department of Political Science, Washington University in St. Louis

and

Konstantinos Spiliopoulos

Department of Mathematics and Statistics, Boston University

and

Dylan Walker

Argyros School of Business and Economics, Chapman University

and

Eric D. Kolaczyk

Department of Mathematics and Statistics, McGill University

E-mail: [email protected]

This supplementary document contains two sections. In Section 7 we present the proofs of the theoretical results that appear in the main body of the paper. In Section 8 we present some additional statistical tests and diagnostics to back up the validity of the numerical results reported in the main body of the paper.

7 Proofs

Theorem 7.1.

Given the multivariate Gaussian distribution setting presented above, write $p(x_{1},x_{2})$ as the p.d.f of the joint distribution of $x_{1},x_{2}$. $p(x_{2})$ as the p.d.f of the marginal distribution of $x_{2}$. Then, we have $p(x_{2})\propto p(x_{1}=\mu_{1|2},x_{2})$.

Proof.

Let us explicitly state the conditional mean and covariance matrix for our given multivariate Gaussian distribution. The conditional mean of $x_{1}$ given $x_{2}$, denoted $\mu_{1|2}$, is:

$$\mu_{1|2}=\mu_{1}+\Sigma_{12}\Sigma_{22}^{-1}(x_{2}-\mu_{2}).$$

Additionally, the conditional covariance matrix, denoted $\Sigma_{1|2}$, is described as:

$$\Sigma_{1|2}=\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}.$$

The joint distribution, $p(x_{1},x_{2})$, for the multivariate Gaussian is:

$$p(x_{1},x_{2})=\frac{1}{(2\pi)^{\frac{k}{2}}|\Sigma|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)\right).$$

The conditional distributions $p(x_{1}|x_{2})$ is:

$$p(x_{1}|x_{2})=\frac{1}{(2\pi)^{\frac{n}{2}}|\Sigma_{1|2}|^{\frac{1}{2}}}\exp\left(-\frac{1}{2}(x_{1}-\mu_{1|2})^{T}\Sigma_{1|2}^{-1}(x_{1}-\mu_{1|2})\right).$$

By the definition of conditional probabilities, we have:

$$p(x_{2})=\frac{p(x_{1},x_{2})}{p(x_{1}|x_{2})}.$$

Inserting $x_{1}=\mu_{1|2}$ into this relation, we get:

$$p(x_{2})=\frac{p(\mu_{1|2},x_{2})}{p(\mu_{1|2}|x_{2})}.$$

Since $p(\mu_{1|2}|x_{2})=\frac{1}{(2\pi)^{\frac{n}{2}}|\Sigma_{1|2}|^{\frac{1}{2}}}$ and $\Sigma_{1|2}$ is constant as a function of $x_{2}$, we get that

$$p(x_{2})=Cp(\mu_{1|2},x_{2}).$$

where $C=(2\pi)^{\frac{n}{2}}|\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}|^{\frac{1}{2}}$, a constant as a function of $x_{2}$. ∎

Corollary 7.1.1.

Consider the multivariate Gaussian distribution setting presented above, and let $x_{2}$ be scalar. Then, for some fixed $\tilde{x}_{2}\neq\mu_{2}$.

$$\text{Var}(x_{2})=\frac{1}{2}\frac{(\mu_{2}-\tilde{x}_{2})^{2}}{\log p(x_{1}=\mu_{1},x_{2}=\mu_{2})-\log p(x_{1}=\mu_{1|2},x_{2}=\tilde{x}_{2})}$$ (16)

Proof.

Recall that the p.d.f of a univariate Gaussian distribution $x_{2}\sim\mathcal{N}(\mu_{2},\text{Var}(x_{2}))$ is given by:

$$p(x_{2})=\frac{1}{\sqrt{2\pi\text{Var}(x_{2})}}\exp\left(-\frac{(x_{2}-\mu_{2})^{2}}{2\text{Var}(x_{2})}\right).$$ (17)

Taking the logarithm, we obtain:

$$\log p(x_{2})=-\frac{1}{2}\log(2\pi\text{Var}(x_{2}))-\frac{(x_{2}-\mu_{2})^{2}}{2\text{Var}(x_{2})}.$$ (18)

From Theorem 7.1, we deduce that $\log p(x_{2})$ and $\log p(x_{1}=\mu_{1|2},x_{2})$ differ by a constant since $p(x_{2})\propto p(x_{1}=\mu_{1|2},x_{2})$. Considering two distinct points, $x_{2}=\mu_{2}$ and $x_{2}=\tilde{x}_{2}$:

For $x_{2}=\mu_{2}$:

$$\log p(x_{2}=\mu_{2})=-\frac{1}{2}\log(2\pi\text{Var}(x_{2})).$$ (19)

For $x_{2}=\tilde{x}_{2}$:

$$\log p(x_{2}=\tilde{x}_{2})=-\frac{1}{2}\log(2\pi\text{Var}(x_{2}))-\frac{(\tilde{x}_{2}-\mu_{2})^{2}}{2\text{Var}(x_{2})}.$$ (20)

Define the difference in these logarithmic probabilities as:

$$\Delta=\log p(x_{1}=\mu_{1},x_{2}=\mu_{2})-\log p(x_{1}=\mu_{1|2},x_{2}=\tilde{x}_{2}).$$ (21)

Using the above, we get:

$$\Delta=\frac{(\mu_{2}-\tilde{x}_{2})^{2}}{2\text{Var}(x_{2})}.$$ (22)

Rearranging gives:

$$\text{Var}(x_{2})=\frac{(\mu_{2}-\tilde{x}_{2})^{2}}{2(\log p(x_{1}=\mu_{1},x_{2}=\mu_{2})-\log p(x_{1}=\mu_{1|2},x_{2}=\tilde{x}_{2}))},$$ (23)

completing the proof of the corollary. ∎

8 Supplementary Results in X Data Analysis

8.1 BIC values

In our analysis, we utilized the Bayesian Information Criterion (BIC) to determine the best change-point among the competing models. For the X platform dataset, we evaluated the BIC values for potential single change-points from 2012 onwards, as presented in Table below. The model suggesting a change-point in 2012 yielded the lowest BIC value.

	2012	2013	2014	2015	2016	2017	2018	2019
BIC	220138	220163	220201	220189	220212	220203	220234	220224

Table 3: BIC values for competing models with different change-point for the X platform data.

8.2 Diagnostics

Note that the variance estimation algorithm is similar to a quadratic approximation to the log-posterior density. We performed diagnostics to evaluate the assumption that the log likelihood function can be approximated by a quadratic function in a small neighborhood centered around the mode.

8.2.1 Normality test

We rewrite equation (20) as follows:

$$\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)=\max_{\theta_{1}=\theta_{1}^{*}}\log\pi(Z,\theta|Y)-\frac{1}{2}\frac{\eta^{2}}{\widehat{\text{Var}(\theta_{1}|Y)}},$$ (24)

and if our assumptions are correct, then $\widehat{\text{Var}(\theta_{1}|Y)}$ should be a constant no matter the choice of $\eta$. We vary $\eta$ to obtain a plot of $\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)$ as a function of $\eta$ to test whether this assumption is valid. If this assumption holds, $\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)$ should be approximately a quadratic function as a function of $\eta$.

We choose $\theta_{1}$ to be $\gamma^{w}_{R}$, and set $\eta$ to be -0.03, -0.02, -0.01, 0.01, 0.02, 0.03. In Figure 8 we plot $\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)$ as a function of $\eta$ (left) and $\eta^{2}$ (right). We can see that $\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)$ is approximately quadratic as a function of $\eta$ and linear as a function of $\eta^{2}$. The diagnostic plot does not show evidence of violating the underlying assumptions of the variance estimation method.

Figure 8: We choose $\theta_{1}$ to be $\gamma^{w}_{R}$, and set $\eta$ to be -0.03, -0.02, -0.01, 0.01, 0.02, 0.03. We plot $\max_{\theta_{1}=\theta_{1}^{*}-\eta}\log\pi(Z,\theta|Y)$ as a function of $\eta$ (left) and $\eta^{2}$ (right).

8.2.2 Linearity test

Figure 9: We choose $\theta_{1}$ to be $\gamma^{w}_{R}$, and set $\eta$ to be -0.03, -0.02, -0.01, 0.01, 0.02, 0.03. We calculate $\frac{\mu(X_{j}|X_{1}=\mu_{1}+\eta)-\mu_{j}}{\eta}$ for each latent position parameters under each $\eta$. The latent position parameters are ordered by the mean of its slopes and plot the mean maximum and minimum of its slopes (left) and the slopes of 95% of the latent position parameters after removing the tail 5% that have a very large or very small slopes.

From Theorem 7.1, we see that the conditional mean of $X_{2:n}|X_{1}$ is linear as a function of $X_{1}$ under the Gaussian assumption, i.e.,

$$\mu(X_{2:n}|X_{1}=\mu_{1}+\eta)=\mu_{2:n}+\Sigma_{21}\Sigma_{11}^{-1}\eta.$$ (25)

We vary $\eta$ to obtain the slope $\frac{\mu(X_{j}|X_{1}=\mu_{1}+\eta)-\mu_{j}}{\eta}$ for each $X_{j}$ with different $\eta$ to test whether this assumption is valid. If this assumption holds, $\frac{\mu(X_{j}|X_{1}=\mu_{1}+\eta)-\mu_{j}}{\eta}$ should be a constant regardless the choice of $\eta$ for each $X_{j}$.

We use the results of the previous test, set $\theta_{1}$ to be $\gamma^{w}_{R}$, and $\eta$ to take values in the set $\{-0.03,-0.02,-0.01,0.01,0.02,0.03\}$. We calculate $\frac{\mu(X_{j}|X_{1}=\mu_{1}+\eta)-\mu_{j}}{\eta}$ for each latent position parameters and each $\eta$. In Figure 9 we order the latent position parameters by the mean of its slopes and plot the mean maximum and minimum of its slopes (left) and the slopes of 95% of the latent position parameters after removing the tail 5% that have a very large or very small slopes. We can see that the slopes, or the correlated changes for each latent position parameters when we change $\gamma^{w}_{R}$ are close under different choice of $\eta$. Therefore the diagnostic plot does not show substantial evidence of violating the underlying assumptions of the variance estimation method.