Instance-Optimal Private Density Estimation in the Wasserstein Distance

We start by describing the rate we obtain for distributions $P$ over $\mathbb{R}$.In order to state the rate, we will use $q_{\alpha}$ to represent the $\alpha$-quantile of the distribution $P$ and use $P|_{a,b}$ to define a certain restricted distribution described below. The rate consists of three terms and roughly looks as follows— we suppress logarithmic factors in $n$.

The first term is $\mathbb{E}[\mathcal{W}(P,\hat{P}_{n})]$, the expected Wasserstein distance between the true distribution and the empirical distribution over $n$ samples, and is the non-private term. The remaining two terms represent the cost of privacy- the first is a specific interquantile distance, roughly $\frac{1}{\varepsilon n}(q_{1-\frac{1}{\varepsilon n}}-q_{\frac{1}{\varepsilon n}})$, and the second can be thought of as capturing the weight of the tails- represented by the Wasserstein distance between $P$ and a ‘restricted’ version of $P$ with its tails chopped off (i.e. the cumulative distribution function is $0$ below $q_{1/\varepsilon n}$ and $1$ above $q_{1-1/\varepsilon n}$ and identical to $P$ otherwise). Observe that all $3$ of the terms above are smaller for distributions with small support or greater concentration, and hence the rate adapts to the hardness of the distributions.

Upper Bounds: The upper bound involves estimating roughly $\varepsilon n$ equally spaced quantiles of the empirical distribution differentially privately (using a known private CDF estimation algorithm), and placing roughly $1/\varepsilon n$ mass at each of the estimated quantile points. For the analysis, the intuition for each of the terms is as follows: since we only have access to the empirical distribution, the non-private term $\mathbb{E}[\mathcal{W}(P,\hat{P}_{n})]$ comes from that. Next, if the quantile estimates are good, then the pointwise CDF differences between the empirical distribution and the estimated distribution are at most $1/\varepsilon n$ (due to the discretization), and so we will pay $1/\varepsilon n$ multiplied by the interquantile distance of the empirical distribution. This aligns with the accuracy of state-of-the-art DP quantile estimation algorithms. Finally, since the distribution is restricted to the estimated quantiles, the distribution is $0$ before the first estimated quantile and $1$ above the last estimated quantile and so we pay the Wasserstein distance between the empirical distribution and a restricted version of the empirical distribution. Some care needs to be taken while reasoning about expectation versus high probability (for various terms), and in relating population quantities to empirical quantities (which we do using various concentration inequalities). Details can be found in Section 6.2.

Lower Bounds: We prove that the private and non-private terms are lower bounds separately. Both proofs follow the same framework. The idea is that given knowledge of two distributions $P$ and $Q$, we can use a (private) Wasserstein estimation algorithm to construct a hypothesis test distinguishing $P$ from $Q$. If the (private) estimate for $P$ and $Q$ with $n$ samples gives error smaller than $\frac{1}{2}\mathcal{W}(P,Q)$, we can use this to distinguish $P$ from $Q$. This would give a contradiction if $P$ and $Q$ are (privately) indistinguishable with $n$ samples. Hence, this would give a lower bound of $\frac{1}{2}\mathcal{W}(P,Q)$ on the error of the Wasserstein estimation algorithm on $P$ or $Q$.

Thus the task reduces to constructing a distribution $Q$ that satisfies three properties: 1) it is (privately) indistinguishable from $P$ given $n$ samples, 2) the Wasserstein distance between $P$ and $Q$ is sufficiently large, 3) $D_{\infty}(P,Q)\leq\ln 2$. The main technical work is in identifying a distribution $Q$ that satisfies these properties.

For the privacy term, we construct the distribution $Q$ by taking half the mass from the first $1/\varepsilon n$-quantile of $P$ (scaling the density function by half) and moving it to the last $1/\varepsilon n$-quantile of $P$ (scaling the density function by $3/2$). The third property is satisfied by definition, so we reason about the other two. Intuitively, since the Wasserstein distance captures how hard it is to ‘move’ $P$ to $Q$, this mass needs to move at least the interquantile distance to change $P$ to $Q$. This implies that the Wasserstein distance is at least the interquantile distance scaled by $1/\varepsilon n$, as described in the rate. Additionally, mass that is further out in the tail needs to move more, which is captured by the Wasserstein distance between the distribution $P$ and its ‘restriction’. Hence, the Wasserstein distance between $P$ and $Q$ is lower bounded by these two terms of interest. The intuition behind Property 2 is that it is hard for any $\varepsilon$-DP algorithm to pinpoint the location of an $\frac{1}{\varepsilon n}$-fraction of the points in the dataset. Overall, this shows the privacy lower bound.

The non-private lower bound requires a more careful construction of $Q$. We divide $P$ into various scales and carefully adjust them differently to obtain the desired properties. Formally, to construct $Q$ from $P$, we consider $q_{1/2}$ and all quantiles of the form $q_{1/2^{i}}$ and $q_{1-1/2^{i}}$ for $i>1$. For $1\leq i<\log n$, we add mass to $[q_{1/2^{i+1}},q_{1/2^{i}})$, by setting the density $f_{Q}$ to be $(1+\sqrt{2^{i}/n})f_{P}$ and balance out the extra mass by setting $f_{Q}$ to be $(1-\sqrt{2^{i}/n})f_{P}$ between $[q_{1-1/2^{i}},q_{1-1/2^{i+1}})$. For $i\geq\log n$ (i.e. the tail), we add mass to $[q_{1/2^{i+1}},q_{1/2^{i}})$, by setting $f_{Q}$ to be $(1+\frac{1}{2})f_{P}$ and balance out the extra mass by setting $f_{Q}$ to be $(1-\frac{1}{2})f_{P}$ between $[q_{1-1/2^{i}},q_{1-1/2^{i+1}})$.

The third property is again trivially satisfied. For the first property, observe that to ‘move’ $P$ to $Q$ the extra $\frac{1}{\sqrt{2^{i}n}}$ mass between $[q_{1/2^{i+1}},q_{1/2^{i}})$ has to ‘travel’ between $q_{1/2^{i}}$ and $q_{1-1/2^{i}}$, and so the Wasserstein distance between $P$ and $Q$ can be lower bounded by a sum of various scaled interquantile distances. We attempt to upper bound the expected Wasserstein distance between $P$ and $\hat{P}_{n}$ by a similar term. It is more intuitive to reason about this using an alternative (equivalent) formulation of Wasserstein distance as the area between the CDF curves of $P$ and $Q$. The intuition is that the expected pointwise CDF difference between $P$ and $\hat{P}_{n}$ in the interval $[q_{1/2^{i+1}}$, $q_{1/2^{i}})$ would be roughly $\frac{1}{\sqrt{2^{i}n}}$ (by properties of a Binomial) and hence the contribution of this interval to the area would be roughly $\frac{1}{\sqrt{2^{i}n}}\left(q_{1/2^{i}}-q_{1/2^{i+1}}\right)$ and similarly for the corresponding interval $[q_{1-1/2^{i}}$, $q_{1/2^{i+1}})$. Hence, the expected Wasserstein distance would be a sum of these scaled quantile interval distances. We formalize this intuition using a result of Bobkov and Ledoux [BL19] that characterizes the expected Wasserstein distance between $P$ and $\hat{P}_{n}$ as an integral of a function of the CDF of $P$. We now have a bound in terms of the sum of scaled quantile interval distances, but we want to bound it by a sum of scaled interquantile distances. We can telescope the sum to indeed bound it by a sum of scaled interquantile distances. This establishes that $\mathcal{W}(P,Q)\geq\mathbb{E}[\mathcal{W}(P,\hat{P}_{n}]$. Next, we show that $P$ is indistinguishable from $Q$ by analyzing the KL divergence between $P$ and $Q$. The main idea is that high density intervals are modified by a small multiplicative factor of roughly $1+\frac{1}{\sqrt{n}}$, but low density intervals (with mass less than $1/n$) are modified by a constant multiplicative factor, so overall the contribution of each interval to the KL divergence is sufficiently small. This establishes indistinguishability with $n$ samples. For formal details we refer the reader to Section 6.1.

Since the main technical challenge of proving Theorem 1.3 is proving the equivalent result for distributions on HST metric spaces, we focus on that problem in this section. Standard results on low distortion embeddings of metric spaces into HST metric spaces can be used to translate the HST result to $[0,1]^{2}$ and to general metric spaces $X$ with $\log|X|$ overhead.

HSTs can be defined with any geometric scaling but we will only need a factor of 2 in this work. HSTs may also have arbitrary degree. A HST defines a metric on its leaf nodes by defining the distance between any two leaf nodes to be the weight of the minimum weight path between them.

HST metric spaces are particularly well-behaved when working with the Wasserstein distance since the Wasserstein distance on a HST has a simple closed form. A distribution $P$ on the the underlying metric space in a HST induces a function $\mathfrak{G}_{P}$ on the nodes of the tree where the value of a node $\nu$ is given by the weight in $P$ of the leaf nodes in the subtree rooted at $\nu$. For every level $\ell\in[0pt]$ of the tree, let $P_{\ell}$ be the distribution induced on the nodes at level $\ell$ where the probability of node $\nu$ is $\mathfrak{G}_{P}(\nu)$. Thus $P_{\ell}$ is a discrete distribution on a domain of size $N_{\ell}$, where $N_{\ell}$ is the number of nodes in level $\ell$ of the tree.

We will call a node $\nu$ $\alpha$-active under the distribution $P$ if $\mathfrak{G}_{P}(\nu)\geq\alpha$. Let $\gamma_{{P}}\left({\alpha}\right)$ be the set of $\alpha$-active nodes under $P$ and $\gamma_{{P_{\ell}}}\left({\alpha}\right)$ be the set of $\alpha$-active nodes at level $\ell$. Then there exists an algorithm $\mathcal{A}$ such that given a distribution $P$, $\varepsilon>0$, and $n\in\mathbb{N}$,

where the max is over all the levels of the tree and $\kappa=\Theta(\frac{\log(n)}{\varepsilon n})$. Further, this bound matches (up to logarithmic factors) the lower bound $\min_{\varepsilon\text{-DP }\mathcal{A}^{\prime}}\sup_{P^{\prime}:D_{\infty}(P,P^{\prime})\leq\ln 2}\operatorname*{\mathbb{E}}[\mathcal{W}(\mathcal{A}^{\prime}(\hat{P^{\prime}}_{n^{\prime}}),P^{\prime})]$ where $n^{\prime}\approx\frac{n}{\operatorname{polylog}n}$. The error rate $\mathcal{R}_{\mathcal{A},n}$ does indeed adapt to easy instances as we expected. The error decomposes into three components. The first component is the non-private sampling error; the error that would occur even if privacy was not required. The second component indicates that we can not privately estimate the value of nodes that have probability less than $\approx 1/(\varepsilon n)$. The third component is the error due to privacy on the active nodes. If $P$ is highly concentrated then we expect most nodes to either be $\kappa$-active or have weight 0, so the first two terms in $\mathcal{R}_{\mathcal{N},n,\varepsilon}(P)$ are small. There should also be few active nodes, making the last term also small. Conversely, if $P$ has a large region of low density then we expect a large number of inactive nodes, as well as non-zero inactive nodes that are at higher levels of the tree and hence contribute more to the final term. Thus, in distributions with high dispersion we expect the right hand side to be large.

Upper Bounds: As in the one-dimensional setting, we want to restrict to only privately estimating the density at a small number ($\approx\varepsilon n$) of points. While we could try to mimic the one-dimensional solution by privately estimating a solution to the $\varepsilon n$-median problem, it’s not clear how to prove such an approach is instance-optimal. It turns out that a simpler solution more amenable to analysis will suffice. Our algorithm has two stages; first we attempt to find the set of $\kappa$-active nodes, then we estimate the weight of these active nodes. Since these nodes have weight greater than $\frac{\log(n)}{\varepsilon n}$, we can privately estimate them to within constant multiplicative error. Any nodes that are not detected as active, are initially ascribed a weight of 0. The error due to not estimating the non-active nodes is absorbed into the third error term. The final step is to project the noisy density function into the space of distributions on the underlying metric space. The error of the upper bound algorithm is summed over all levels of the tree, although since the depth of the tree is logarithmic in the size of the metric space, this is within a logarithmic factor of the maximum over the levels.

Lower Bound: We first observe that in order to estimate the distribution well in Wasserstein distance, an algorithm must estimate each level of the tree well in TV distance. This is derived from Lemma 1.5. This allows us to reduce to the problem of lower bounding the error of density estimation of discrete distributions in TV distance. The main tool we use is a differentially private version of Assouad’s method. Similar to how the technique in the previous section allowed us to relate lower bounding estimation rates to simple hypothesis testing, Assouad’s lemma allows us to relate lower bounding estimation rates to multiple hypothesis testing. Note that unlike the technique in the previous section, Assouad’s lemma allows us to prove lower bounds on the expected error, rather than lower bounds on high probability error bounds. It involves constructing nets of distributions in $\mathcal{N}(P)$ that are pairwise far in the relevant metric of interest (which for us in the TV distance) but the multiple hypothesis testing problem between the distributions is sufficiently hard. For proving the third term belongs in the lower bound, the standard statement of DP Assouad’s lemma [ASZ21] suffices, where one builds a set of distributions indexed by a hypercube. For the first and second terms, we need to slightly generalise the statement to allow for sets of distributions indexed by a product of hypercubes. We use the approximate DP version of DP Assouad’s so while our upper bounds are for pure differential privacy, our lower bounds hold for both pure and approximate differential privacy.

Let us start with the third term. Suppose the number of active nodes is even (a small tweak is made if there is an odd number of active nodes). We pair up the active nodes and index each pair by a coordinate of the hypercube. For each corner of the hypercube, $(u^{0},u^{1},\cdots,u^{k})\in\{\pm 1\}^{k}$, for each coordinate $j\in[k]$, if $u^{j}=+1$, we move $\tilde{O}(\kappa)$ mass from one node in the $j$th pair to the other node. If $u^{j}=-1$ then we leave the $j$th pair of nodes alone. Since each active node has mass $>\kappa$, it’s clear that each resulting distribution belongs in $\mathcal{N}(P)$. We can also show that these distributions form a sufficiently hard multiple hypothesis testing problem. By DP Assouad’s (Lemma 5.8), this allows us to lower bound the estimation error by $\Omega(k\kappa)$, which is within $\tilde{\Omega}$ of the third term when the number of active nodes is $\geq 2$. We treat the case where there is a single active node separately.

For the second term, we want to pair up the inactive nodes in a similar manner and move half their mass from one node to the other. However, since we want to remain within $\mathcal{N}(P)$, we can’t pair any two inactive nodes together. Thus, we divide the inactive nodes into scales, where nodes within a certain scale all have weight within a multiplicative factor of two. We then pair up nodes within each scale and have a different hypercube for each scale. Again, it’s clear that these distributions are all in $\mathcal{N}(P)$ and we can show that these distributions form a sufficiently hard multiple hypothesis testing problem. The proof for the first term follows similarly.