Analyzing Disparity and Temporal Progression of Internet Quality through Crowdsourced Measurements with Bias-Correction

Ookla’s Speedtest²²2http://speedtest.net (data provided through Ookla’s Speedtest Intelligence^®) possesses over 16k measurement servers worldwide [13] and allows users to assess the quality of their Internet connection using either a web-based portal or native mobile application [11]. For each Speedtest, a nearby test server is selected, and (potentially multiple) TCP connections are used to calculate the throughput of the path. Ookla’s Speedtest Intelligence dataset contains individual Speedtest measurements that include QoS metrics (up/down throughput, latency, packet loss, jitter), as well as meta-features such as ISP, device type, and access type. Ookla provides performance data aggregated over time and space to the public [12]. However, our Data Usage Agreement (DUA) with Ookla provides us access to nearly 1 million individual Speedtest measurements from four major metropolitan cities in the U.S., which we use for this study. Each of these cities has a population in the range of 370—650k.

We use the data obtained through our DUA to investigate the download speed data from two different device types—iOS and Android. We focus on these two device types due to the richness of metadata and geographical information associated with each data point, as well as due to the widespread popularity and usage of these platforms. For demonstration purposes, we use two representative cities for our detailed analysis. Due to our DUA, we maintain the confidentiality of these cities and only refer to them as City A and City B. We also provide numerical analysis of two additional cities, referred to as City C and City D, in Appendix B. Since precise geographic location is crucial for our study, we ensure each measurement includes latitude/longitude data recorded at the time the Speedtest is conducted; truncated GPS coordinates are reported for users who allow location sharing on their equipment. We utilize the geographic location of each Speedtest to associate the test with the relevant census block group. Table 1 shows summary statistics for the data used in this paper. We observe that in every listed city, iOS devices outnumber Android devices in our dataset, yielding almost double the number of Speedtest data points from iOS compared to Android.

To study demographic trends with performance data, we leverage data from the U.S. Census Bureau. In particular, we select the 2019 Census demographic information due to the inaccuracy of data since the COVID-19 pandemic [8]. For City A and B, we obtain population data for each census block group as well as the following demographic attributes: income, age, sex, education level, internet penetration rate, and ethnicity distribution. Income and age are represented by the median household income and median age, respectively. Sex information is characterized by the proportion of males within the total population of each census block group. The level of education is quantified as the proportion of individuals with a bachelor’s degree or higher among the population aged 25 and older. The internet penetration rate is the percentage of households with an internet subscription out of the total number of households of each census block group. Ethnic distribution is summarized by the proportion of white, black, Asian, and Hispanic populations within the total population of each census block group.

The demographic information from the U.S. Census Bureau is only available at the level of the census block group. Therefore, we build an integrated dataset with a spatial granular structure, where speed measurements identified in each census block group are correlated with the demographic profiles for that census block group based on U.S. Census Bureau data.

Each Ookla Speedtest measurement is associated with a certain census block group based on the geographic location of the test. By comparing the distribution of the sample sizes across census block groups with their population distribution, we can test whether the sample sizes are homogeneous with the population across regions. As described in Section 2, the population of each census block group is based on data from the U.S. Census Bureau. We plot the cumulative distributions and probability mass functions between the sample and population at the level of census block group in Fig. 1 for City A and B, where the census block groups are ordered according to their population sizes. We observe that there is a recognizable discrepancy between the distribution of sample sizes from both Android and iOS in Ookla Speedtests and the population at each census block group. The comparison suggests that there is a regional sampling bias in the Speedtest measurements, as certain census block groups contain disproportionately larger samples than others.

We utilize the chi-squared test, one of the most frequently used tests for homogeneity of the samples, to quantify whether the discrepancy between the population and sample size distributions across census block groups is significant. The two assumptions of the test are summarized as follows:

Assumption 1 approximately holds in the data set, but it can be affected by internet traffic and availability. For instance, an internet outage event may affect more than one census block group, making speed tests not independent. Second, certain groups of internet users may be more likely to utilize Speedtest; the relationship between internet speed and demographic profiles of the census block groups will be studied in Section 4.1.

We collect the sample of Ookla Speedtests from $k$ census block groups. For the $i$-th census block group, define $n_{i}$ and $N_{i}$ as the sizes of its sample and population, respectively, for $i=1,\cdots,k$. Furthermore, let $n$ and $N$ respectively be the total number of samples and the population among all the regions, i.e., $n=\sum_{i=1}^{k}n_{i}$ and $N=\sum_{i=1}^{k}N_{i}$. When the samples are drawn according to the population distribution, i.e., the null distribution, the expected number of samples in the $i$-th region is $\frac{nN_{i}}{N}$. Then, the $\chi^{2}$ test statistic is defined by:

Under the null hypothesis, where there is no regional sampling bias across different census block groups, $W\sim\chi^{2}_{k-1}$, i.e., the test statistic $W$ is distributed as a chi-squared distribution with $k-1$ degrees of freedom.

We find that the disparity observed in Fig. 1 is in line with the results from the statistical testing as shown in Table 2. We observe that the p-values from both iOS and Android devices in City A and B are significantly low, suggesting substantial inhomogeneity in the sample sizes relative to the population across regions.

Because we found significant evidence of the Speedtest distribution being disproportionate across regions, we introduce two ways to correct the regional sampling bias. By comparing the estimation of the cumulative distribution function from the original and bias-corrected samples, we can assess whether this sampling bias affects the estimation of the distribution of internet speed.

To delineate the empirical CDF of internet speed, let $y_{ij}$ represent the internet speed measurement from the $j$-th unit of the $i$-th census block group for $j=1,\cdots,n_{i}$ and $i=1,\cdots,k$. Since our estimation covers $k$ different census block groups, we define $y$ as a simple random sample of internet speed from a collection of these $k$ census block groups. Based on Assumptions 1 and 2, the simple random sample $y$ can be represented by a mixture of $k$ random variables $y_{1},y_{2},y_{3},\cdots,y_{k}$, which denote the internet speed measurement from the 1st, 2nd, 3rd, $\cdots,$ and $k$-th census block group, respectively. Without any regional sampling bias, a simple random sample $y$ of internet speed can be expressed as

where $I(\cdot)$ refers to an indicator function, $z_{i}$ only takes either 0 or 1 for $i=1,\cdots,k$, $N=\sum_{i=1}^{k}N_{i}$ indicates the total population over all $k$ census block groups, and $\sum_{i=1}^{k}z_{i}=1$. A simple random sample is critical for inferring the population, and sampling bias can lead to misleading conclusions, such as those in 2016 U.S. presidential election [29].

For a given internet speed $x$, we are interested in the cumulative probability of internet speed for the entire city:

Similarly, we define $F_{i}(\cdot)$ to be the cumulative distribution at the $i$-th census block group, for $i=1,...,k$. The cumulative probability in (3) is a weighted sum of the regional cumulative probability by the corresponding population, as shown in Lemma 1.

While the impact of regional sampling bias over different census block groups on the estimation of the CDF is not as pronounced as the regional bias itself, this impact depends on whether the samples within the census block groups are representative and independent, as outlined in Assumptions 1-2. These assumptions may not strictly hold in practice. For instance, if a certain demographic group is over-sampled, the samples may not be representative in each census block group, violating Assumption 2. Thus, it is crucial to discover potential sampling bias among other sub-groups. We collect the demographic variable data for each census block group for the following variables: income, age, gender, educational level, internet penetration rate, and ethnicity. The variables that characterize the demographic profile of the $i$-th census block group, for each $i=1,\cdots,k$, are summarized in Table 3.

We begin with a multiple linear regression model that relates internet speed $y_{ij}$, the Speedtest $j$ at census block group $i$, to the demographic variable $\mathbf{x}_{i}$:

where $\mathbf{x}_{i}=(1,x_{1i},x_{2i},\cdots,x_{pi})^{T}$ denotes a $(p+1)$-dimensional vector of predictors (such as income, age, level of education, etc.) with $p=9$, $\boldsymbol{\beta}=(\beta_{0},\beta_{1},\cdots,\beta_{p})^{T}\in\mathbb{R}^{(p+1)}$ is a vector of linear coefficients with $p=9$, and $\epsilon_{ij}\sim\mathcal{N}(0,\sigma^{2})$ denotes a Gaussian white noise with variance $\sigma^{2}$, for $j=1,\cdots,n_{i},$ and $i=1,\cdots,k$. Note that the observations from the $i^{th}$ census block group, denoted as $y_{i1},\cdots,y_{in_{i}}$, share the same predictor vector $\mathbf{x}_{i}$ for $i=1,\cdots,k$. It follows that the model (12) is equivalent to:

where $\mathbf{y}=(y_{11},\cdots,y_{1n_{1}},\cdots,y_{k1},\cdots,y_{kn_{k}})^{T}\in\mathbb{R}^{n}$ with $n=\sum_{i=1}^{k}n_{i}$, $\mathbf{V}=[\mathbf{x}_{1}\mathbf{1}_{n_{1}}^{T},\mathbf{x}_{2}\mathbf{1}_{n_{2}}^{T},\cdots,\mathbf{x}_{k}\mathbf{1}_{n_{k}}^{T}]^{T}\in\mathbb{R}^{n\times(p+1)}$, $\mathbf{1}_{d}$ denotes a $d$-dimensional column vector with all elements being 1, and
$\boldsymbol{\epsilon}=(\epsilon_{11},\cdots,\epsilon_{1n_{1}},\cdots,\epsilon_{k1},\cdots,\epsilon_{kn_{k}})^{T}\sim\mathcal{MN}(\mathbf{0},\sigma^{2}\mathbf{I}_{n})$ with $\mathbf{I}_{n}$ being an $n$-dimensional identity matrix.

Due to the two different spatial granular levels in model (12), an aggregated version of model (12) can be suggested to equalize the granular levels without affecting the estimation of $\boldsymbol{\beta}$ in model (12).

where $\bar{y}_{i}=\frac{1}{n_{i}}\sum_{j=1}^{n_{i}}y_{ij}$, and $\epsilon_{i}$ independently follows $\mathcal{N}(0,\sigma^{2}/n_{i})$ for $i=1,\cdots,k$. Model (14) can be written as a matrix form:

where $\bar{\mathbf{y}}=(\bar{y}_{1},\cdots,\bar{y}_{k})^{T}\in\mathbb{R}^{k}$, $\mathbf{X}=[\mathbf{x}_{1},\cdots,\mathbf{x}_{k}]^{T}\in\mathbb{R}^{k\times(p+1)}$, $\boldsymbol{\epsilon}_{\text{agg}}=(\epsilon_{1},\cdots,\epsilon_{k})^{T}\sim\mathcal{MN}(\mathbf{0},\sigma^{2}\mathbf{W}^{-2})$, and $\mathbf{W}$ is a $k\times k$ diagonal matrix with diagonal entries being $\sqrt{n_{1}},\cdots,\sqrt{n_{k}}$. We have the following lemma that connects the individual level model (12) and the aggregated level model (14).

Lemma 3 is derived in Appendix A.5. Equation (16) in Lemma 3 means that the difference between the log-likelihood of model (12) and (14) does not depend on the linear coefficient $\boldsymbol{\beta}$, and only depends on the variance $\sigma^{2}$. Thus the sufficient statistics of the linear coefficients [19] is $(\bar{\mathbf{y}},\sigma^{2})$ with $\bar{\mathbf{y}}=(\bar{y}_{1},\cdots,\bar{y}_{k})^{T}$ being the aggregated data. Note the maximum likelihood estimator of linear coefficients does not depend on $\sigma^{2}$, whereas the uncertainty of the estimation depends on the noise variance.

In practice, however, the variance parameter $\sigma^{2}$ of the noise is also unknown. One may be tempted to use the aggregated model (15) to estimate $\sigma^{2}$. The lemma below indicates that the estimation of the variance of the noise by individual-level data is more efficient than that by the aggregated data.

Thus, we adopt the formulation of model (12) to assess the linear relationship between internet download speed (Mbps) and demographic features. We apply the regression approaches for both the original samples and the re-sampled data studied in Section 3.2.2. Similarly, for the re-sampled data, we construct the linear regression as follows:

where $\boldsymbol{\beta}^{r}=(\beta_{0}^{r},\beta_{1}^{r},\cdots,\beta_{p}^{r})^{T}\in\mathbb{R}^{(p+1)}$, and $\epsilon_{ij}^{r}$ independently follows $\mathcal{N}(0,\sigma_{r}^{2})$ for $j=1,\cdots,n_{i}^{*}$, and $i=1,\cdots,k$.

Instead of directly comparing the full models presented in (12) and (17), we employ a backward model selection procedure to select demographic variables that significantly impact measured internet speeds based on the Akaike Information Criterion (AIC)  [17]. Let $\mathcal{M}_{m}$ denote a model with index $m$, and define $d_{m}$ as the dimension of model $\mathcal{M}_{m}$. The AIC for $\mathcal{M}_{m}$ is defined as follows:

where $\hat{l}_{n,m}$ represents the maximum log-likelihood of model $\mathcal{M}_{m}$ given $n$ observed data. Commencing with a full model featuring $p$ predictors, we consider $p$ distinct submodels created by eliminating one variable at a time. For each submodel, we compute the corresponding AIC using (18). If any submodel exhibits a smaller AIC compared to the existing model, we select that model for the subsequent stage and repeat the same process. If no such submodel is found to have a small AIC, we conclude with the existing model as the final selected model.

Fig. 5 provides estimated linear coefficients from the regression analysis conducted on both the original and re-sampled data after the backward elimination technique using AIC. It presents estimates of coefficients and their corresponding 95% confidence intervals for variables, whereas the variables dropped from the backward selection procedure are not shown. From the graphs, we can make multiple observations.

First, there is a significant negative correlation between measured internet speed and age in City A, City C, and City D shown in Fig 12 in the Appendix. This means internet speed is greater in census block groups with a younger population on average, given other estimated variables. City B is an exception, as the effect of age does not seem to be clear.

Second, regions with a larger percentage of Hispanic population generally have lower measured internet speed for both cities and device types shown in Fig. 5. For the two cities shown in Fig.12 in Appendix B.2, the effect of the percentage of the Hispanic population is not as clear as the two cities shown in Fig. 5; however, after examining the pairwise correlation plot between the covariates in Fig. 14-15, we find that the percentage of the Hispanic population is negatively correlated with the percentage of the bachelor’s degree and availability of the internet. Thus the effect of Hispanic percentage in population can be partly offset by these two effects. For instance, the coefficient of bachelor’s degree in the re-sampled data is significantly larger than zero in part (a) of Fig. 12, which implicitly suggests that the regions with higher Hispanic percentage may have comparatively lower internet speed, as these regions tend to have a lower percentage of residents with bachelor’s degrees.

Third, we find that the regression coefficients for median income are positive for the original data of both device types in cities A and B, meaning that the regions with higher income tend to have faster-measured internet speed. This can likely be attributed to the availability of faster internet plans, as well as the higher purchasing power of local residents; prior work found that the median income of the census block groups play a critical role in determining whether a region gets a fiber deployment and consequently faster internet speeds [30]. The coefficients of median income in the re-sampled data are positive for City B but negative for City A, as shown in Fig. 5. In both cities, the linear coefficients of income in the re-sampled data are smaller than the ones from the original data. To further explore the difference, we find that the pairwise correlation between income and bachelor’s degree is strongly positive in both cities, as shown in Fig. 6. The estimated linear coefficients of the bachelor’s degree in the re-sampled data are larger than those in the original data for both devices and cities. As the bachelor’s degree and income have strongly positive correlation, the larger coefficient of the bachelor’s degree explains the positive impact on the internet speed, which makes the coefficients of the income smaller in the re-sampled data. Note that an estimated coefficient represents the conditional effects of a covariate given all other covariates in multiple linear regression. The effect of a covariate typically depends on the effects of other variables as multicollinearity of the covariates is common in practice [28].

The regression analysis of Speedtest data from the two cities shows that measured internet speed critically depends on demographic profiles of regions, such as the income, education level and ethnic composition. Future work should study the reasons behind these associations, such as the availability of faster internet plans, and their cost per bit, e.g. carriage value [30].

We first analyze the linear trends of internet speed. Let $y(t_{i})$ denote the measured internet speed at time point $t_{i}$ for $i=1,\cdots,n$; i.e., we have $n$ distinct time points and each $t_{i}$ corresponds to a positive real number indicating the time lapse from the starting date measured by days. The linear regression model of internet speed with an intercept and time as a covariate is given by: