Sequential Manipulation against Rank Aggregation: Theory and Algorithm

Abstract

Rank aggregation with pairwise comparisons is widely encountered in sociology, politics, economics, psychology, sports, etc. Given the enormous social impact and the consequent incentives, the potential adversary has a strong motivation to manipulate the ranking list. However, the ideal attack opportunity and the excessive adversarial capability cause the existing methods to be impractical. To fully explore the potential risks, we leverage an online attack on the vulnerable data collection process. Since it is independent of rank aggregation and lacks effective protection mechanisms, we disrupt the data collection process by fabricating pairwise comparisons without knowledge of the future data or the true distribution. From the game-theoretic perspective, the confrontation scenario between the online manipulator and the ranker who takes control of the original data source is formulated as a distributionally robust game that deals with the uncertainty of knowledge. Then we demonstrate that the equilibrium in the above game is potentially favorable to the adversary by analyzing the vulnerability of the sampling algorithms such as Bernoulli and reservoir methods. According to the above theoretical analysis, different sequential manipulation policies are proposed under a Bayesian decision framework and a large class of parametric pairwise comparison models. For attackers with complete knowledge, we establish the asymptotic optimality of the proposed policies. To increase the success rate of the sequential manipulation with incomplete knowledge, a distributionally robust estimator, which replaces the maximum likelihood estimation in a saddle point problem, provides a conservative data generation solution. Finally, the corroborating empirical evidence shows that the proposed method manipulates the results of rank aggregation methods in a sequential manner.

1 Introduction

Rank aggregation has wide-ranging applications in social choice theory [2], psychology [46], economics [45], statistic [29], bioinformatic [4], and other fields. In pursuit of large benefits, the potential attackers have strong motivations to manipulate the ranking aggregation algorithms which are utilized in high-stakes scenarios, e.g. elections [6], sports competitions [31], and recommendations [43]. A profit-seeking adversary will try his/her best to designate the ranking list and fulfill his/her demands. In addition to statistical [17] and computational [50] properties, the integrity issue of ranking results becomes a new direction in the study of rank aggregation algorithms.

(a) offline attack against rank aggregation

(b) online attack against rank aggregation

Figure 1: Overview of the offline and online adversarial settings. (a) In the offline confrontation scenario, the adversary observes the whole comparison graph on Oct. 17 and he/her obtains the attack strategy which needs to flip the comparisons which have occurred on Sep. 25 and Oct. 2. However, no one can return to the past and change what has happened. Moreover, bypassing the defense mechanisms of the rank aggregation to modify the completed comparison graph is really a challenging task. (b) Different form the offline attack methods, we consider the sequential manipulation strategies which has no knowledge of all future observed pairwise comparisons. The proposed online attack method inserts malicious into the data stream before the construction of comparison graph.

The pioneer in conducting security-related research on rank aggregation is [37]. [37] develops a strong threat model for perturbing the aggregated results. The adversary has complete knowledge of the initial truthful data and corresponding feedback of victims. He/her can corrupt the original data by inserting, deleting, or flip** any pairwise comparisons with limitations on quantity of modification. [37] also considers the adversary with incomplete knowledge, who lacks the preference score generated by the victims. The attack strategies are solved by maximizing the objective functions of the victims with global modification on the weights of comparison graph. Their results show that the rank aggregation algorithms are vulnerable to these attackers. Concurrent to [37], [32] and [1] restrict the modification scope and degree of weights towards specific families of comparison graphs, then provide the recovery guarantees for the ground truth ranking with the proposed procedures. It is noteworthy that these weaker threat models could not be translated into any defense mechanism against the unregulated attackers. Furthermore, [38] poses the manipulation problem against rank aggregation algorithms. The purposeful attackers are not satisfied with simply perturbing the ranking list, but with designating it. The attack behavior with a target ranking list is a fixed point belonging to the composition of the adversary and the victim from the perspective of the dynamical system. The manipulation strategies equal to the conditions that the weights of comparison graph should satisfy when the victims obtain the target ranking list. From the above analysis, we conclude that the existing methods study the security issue of rank aggregation in an “offline” adversarial scenario [51, 52, 53]. In general, the attackers from the existing methods try to modify the pairwise comparisons that have already been collected. These offline attacks must occur after the construction of the comparison graph and before the victims aggregate its results. The rank aggregation algorithms would wait for the adversary to complete his/her malicious actions and unconditionally accept the modifications to the data before they can begin their own jobs. Opportunity for such attacks affords the adversary some privileges. There exists an implicit assumption that the adversary is capable of changing the existed data in the possession of the victims. However, the data held by the rank aggregation algorithms is often immutable in practice. In sports competitions, the final ranking is only produced when all the races have been completed. Theoretically, the existing methods could perturb or manipulate the ranking lists of all teams or players. But no one can travel to the past and change the outcome of a match that has already finished. Once a vote has been cast at the polling station, the ballot will not be changed by any third party. In the partial confrontation scenarios, the existing methods assume that they have completely bypassed the constraints of time and space. Therefore, these offline methods fail to profile the capability boundary of the potential attackers and illuminate the underlying risks of ranking aggregation algorithms.

To address the above challenges, we need a new online paradigm for manipulating rank aggregation algorithms. In terms of attack opportunities, attackers need to seek more chances for archiving his/her goal and bypass the time and space constraints. The whole process of obtaining a ranking list can be divided into two parts: online data collection and offline data aggregation. Compared to offline aggregation, online collection is much more vulnerable. As a distributed and asynchronous process, data collection can’t be done in a controlled environment and is therefore independent of rank aggregation. The defense mechanisms of aggregation often fail to protect online data collection. In addition, data collection always takes a long time and the attacker has sufficient chances to execute his/her actions. Consequently, disrupting the data collection process by online falsifying pairwise comparisons is more sophisticated than offline changing the collected data. Having determined the attack opportunity, it is necessary to identify the attacker’s capabilities during data collection. During the collection process, a data source generates many pairwise comparisons waiting to be sampled. Once a comparison is sampled, it is used to construct the comparison graph and cannot be modified. If the attacker performed malicious actions during data collection, all he/she could do was mimic the behavior of normal data. The adversary could construct an adversarial data source which generates specific pairwise comparisons and insert them into the data stream. Since the cost of authenticating data sources is much greater than the cost of fabricating a pairwise comparison with malicious intention, an attacker can effectively bypass the victim’s defenses. To the best of our knowledge, manipulating aggregation results by fabricating the data source and continuously injecting malicious pairwise comparisons into the data stream is a new formulation for attacking against rank aggregation algorithms, which is still under-explored.

The core of this paper is to make the above analysis rigorous by establishing a principle framework for sequential manipulating rank aggregation algorithms. The main methodology and theoretical contributions are summarized as follows.

• Under a distributionally robust game theoretic framework, we construct the confrontation model between the online manipulator and the ranker who is bound to the original data sources. We then prove the existence of distributionally robust Nash equilibrium in such a game, which guarantees the possibility of sequential manipulation. This adversarial game describes the goal, knowledge and capability of the attacker, with particular emphasis on the uncertainty that all players must deal with.

• We characterize the data collection process as a sampling algorithm and focus on two of the most basic and well-known sampling algorithms: Bernoulli sampling and reservoir sampling. Our theoretical analysis shows that the sampled results could be representative with respect to the mixture of the original and adversarial data sources. Such results suggest that the actions of adversary could resist the effects of randomness in the original data source and the data collection.

• Different sequential manipulation policies are proposed under a Bayesian decision framework and a large class of parametric pairwise comparison models. The underlying Bayes risk consists of the expected Kendall’s tau distance and the expected relative generation cost. We then derive the asymptotic optimality of the proposed policies with complete knowledge.

• To increase the success rate of the sequential manipulation with incomplete knowledge, we empower the generation rule against uncertainty. A distributionally robust estimator replaces the maximum likelihood estimation in a saddle point optimization problem. Then the corresponding conservative generation rule is obtained by mirror descent algorithm.

The rest of the paper is organized as follows. In Section 2, we introduce the basic concept of rank aggregation and two representative algorithms as HodgeRank [29] and RankCentrality [41]. Section 3 establishes the general framework for sequential manipulating rank aggregation algorithms. We present the details of manipulation strategies and the theoretical results in Section 4. Section 5 illustrates the simulated and real-world data results, followed by concluding remarks in Section 6. Technical proofs are provided in the supplementary material.

2 Preliminary

We begin with a formal description of the parametric model for binary comparisons, a.k.a Bradley-Terry-Luce (BTL) model [12]. Then we revisit the comparison graph and the Laplacian matrix which are essential for the ranking algorithms tailored to the BTL model. Two popular approaches which rank the items based on appropriate estimation of the latent preference scores, named HodgeRank [29] and Rank Centrality [41], are chosen as the victims to motivate our target attack strategies. In the remainder of this paper, we will use positive integers to indicate alternatives and voters. Let $\boldsymbol{V}$ be the set $[n]=\{1,\ \dots,\ n\}$ which denotes a set of alternatives to be ranked. $\boldsymbol{U}=\{\boldsymbol{u}_{1},\ \dots,\ \boldsymbol{u}_{m}\}$ denotes a set of voters. We will adopt the following notation from combinatorics:

$$\begin{bmatrix}\boldsymbol{V}\\ l\end{bmatrix}:=\text{set of all}\ l\ \text{elements subset of}\ \boldsymbol{V}.$$

In particular,

$$\begin{bmatrix}\boldsymbol{V}\\ 2\end{bmatrix} := \ \ \text{set of all unordered pairs of elements of}\ \boldsymbol{V}$$

$$:= \left\{[i,\ j]\ \Big{|}\ \forall\ i,\ j\in\boldsymbol{V},\ i\neq j\right\}.$$

Moreover, for any $i,j\in\boldsymbol{V},\ i\neq j$, we write $i\succ j$ to mean that alternative $i$ is preferred over alternative $j$. Such a comparison could be converted into an ordered pair $(i,\ j)$. The set of ordered pair will be denoted as

$$\boldsymbol{V}\times\boldsymbol{V}:=\left\{(i,\ j)\ \Big{|}\ i\succ j,\ \forall\ i,\ j\in\boldsymbol{V},\ i\neq j\right\}.$$

Ordered and unordered pairs will be delimited by parentheses $(i,j)$ and braces $\{i,j\}$ respectively. If we wish to emphasize the preference judgment from a particular voter $\boldsymbol{u}\in\boldsymbol{U}$, we will write $i\succ_{\boldsymbol{u}}j$.

2.1 Parametric Model and Pairwise Comparisons

Given a collection $\boldsymbol{V}$ of $n$ alternatives, the parametric model of pairwise comparisons assumes that each $i\in\boldsymbol{V}$ has a certain numeric quality score $\theta^{*}_{i}$. Suppose that $\boldsymbol{\theta}^{*}\in\mathbb{R}^{n}$

$$\boldsymbol{\theta}^{*}=\left[\theta^{*}_{1},\ \dots,\ \theta^{*}_{n}\right]^{\top}$$ (1)

comprises the underlying preference scores assigned to each of the $n$ items. Without loss of generality, $\boldsymbol{\theta}^{*}$ could be positive as

$$\theta^{*}_{i}>0,\ \forall\ i\in[n].$$

Specifically, a comparison of any pair $\{i,j\}\in\begin{bmatrix}\boldsymbol{V}\\ 2\end{bmatrix}$ is generated via the comparing between the corresponding scores $\theta^{*}_{i}$ and $\theta^{*}_{j}$ (in the presence of noise) by the BTL model. Let $y^{*}_{ij}$ denote the outcome of the comparison of the pair $i$ and $j$ based on $\boldsymbol{\theta}^{*}$, such that $y^{*}_{ij}=1$ if $i$ is preferred over $j$ and $y^{*}_{ij}=-1$ otherwise. Then, according to the BTL model,

$$y^{*}_{ij}=\left\{\begin{array}[]{rl}1,&\text{with probability}\ \theta^{*}_{i}/(\theta^{*}_{i}+\theta^{*}_{j}),\\[5.0pt] -1,&\text{otherwise}.\end{array}\right.$$ (2)

Since the BTL model is invariant under the scaling of the scores, the latent preference score is not unique. Indeed, under the BTL model, a score vector $\boldsymbol{\theta}^{*}\in\mathbb{R}^{n}_{+}$ is the equivalence class

$$\boldsymbol{\Theta}^{*}=\left\{\boldsymbol{\theta}\ \Big{|}\ \text{there exists}\ \alpha>0\ \text{such that}\ \boldsymbol{\theta}=\alpha\boldsymbol{\theta}^{*}\right\}.$$

The outcome of a comparison depends on the equivalence class $\boldsymbol{\Theta}^{*}$.

2.2 Comparison Graph and Combinatorial Laplacian

A graph structure, named comparison graph, arises naturally from pairwise comparisons as follows. Let $\boldsymbol{\mathcal{G}}=(\boldsymbol{V},\boldsymbol{E})$ stand for a comparison graph, where the vertex set $\boldsymbol{V}=[n]$ represents the $n$ candidates. In our problem setting, we pay attention to the complete graph setting: the directed edge set $\boldsymbol{E}=\boldsymbol{V}\times\boldsymbol{V}$ and $N:=|\boldsymbol{E}|=n(n-1)$. One can further associate weights $\boldsymbol{w}^{*}$ on $\boldsymbol{E}$ as voters $\boldsymbol{U}$ would have rated, i.e. assigned cardinal scores or given an ordinal ordering to, the complete set of the alternatives $\boldsymbol{V}$. But no matter how incomplete the rated portion is, one may always convert such judgments into pairwise rankings that have no missing values as follows. For each voter $\boldsymbol{u}\in\boldsymbol{U}$, the pairwise ranking matrix is a skew-symmetric matrix $\boldsymbol{Y}^{\boldsymbol{u}}=\{y^{\boldsymbol{u}}_{ij}\}\in\{-1,0,1\}^{n\times n}$ as

$$y^{\boldsymbol{u}}_{ij}=-y^{\boldsymbol{u}}_{ji},\ \forall\ (i,\ j)\in\boldsymbol{E},\ \forall\ \boldsymbol{u}\in\boldsymbol{U},$$ (3)

where

$$y^{\boldsymbol{u}}_{ij}=\left\{\begin{array}[]{rl}1,&\text{if}\ i\succ_{\boldsymbol{u}}j,\\[5.0pt] -1,&\text{if}\ j\succ_{\boldsymbol{u}}i,\\[5.0pt] 0,&\text{otherwise}.\end{array}\right.$$ (4)

Furthermore, we associate weight with each directed edge as $\boldsymbol{w}^{*}=[w^{*}_{12},w^{*}_{13},\dots,w^{*}_{n,n-1}]^{\top}\in\mathbb{Z}_{+}$

$$w^{*}_{ij}:=\underset{\boldsymbol{u}\in\boldsymbol{U}}{\sum}\ \mathbb{I}[y^{\boldsymbol{u}}_{ij}>0]+\mathbb{I}[y^{\boldsymbol{u}}_{ji}<0],$$ (5)

where $\mathbb{I}[\cdot]$ is the Iverson bracket. Consequently, we can represent any pairwise ranking data as a comparison graph $\boldsymbol{\mathcal{G}}$ with edge weights $\boldsymbol{w}^{*}$.

Given a graph $\boldsymbol{\mathcal{G}}$ and weights $\boldsymbol{w}^{*}$, it is common to consider the weight matrix $\boldsymbol{W}^{*}$ with $w^{*}_{ij}$ as matrix elements, as well as the diagonal degree matrix $\boldsymbol{D}^{*}=\textbf{diag}(d^{*}_{1},\dots,d^{*}_{n})$ given by $d^{*}_{i}=\sum_{j\in\boldsymbol{V}}w^{*}_{ij}$, which represents the volume taken by each node in the graph $\boldsymbol{\mathcal{G}}$. The combinatorial Laplacian $\mathcal{L}_{0}$ is defined as

$$\mathcal{L}_{0}=\boldsymbol{D}^{*}-\boldsymbol{W}^{*}.$$ (6)

In both solving process and the theoretical analysis, the combinatorial Laplacian $\mathcal{L}_{0}$ plays a vital role in the popular approaches based on the parametric model.

Remark 1. In this paper, we select HodgeRank [29] and RankCentrality [41] to verify that online manipulation behavior is a potentially significant threat to rank aggregation methods. This is due to the following considerations. First, these two representative methods that have received much recent attention have been well studied by [29, 17, 15] and their theoretical properties guarantee the promising recovery performance. The successful manipulation will be in stark contrast to the original aggregated results. Second, the variants of HodgeRank and RankCentrality are hot topics of the literature [36, 14, 33, 11]. The online attack method proposed in this paper has a large potential victimization. Third, the destructive results of these two estimators for the famous Bradley–Terry–Luce (BTL) model will prompt researchers to focus on the security issue of rank aggregation in the high-stakes applications.

Remark 2. When exists a purposeful adversary, the collected pairwise comparisons would be a mixture of the data which supports the original ranking list and the fabricated data by the adversary. To manipulate the aggregated results, the attacker will predict the ranker’s behavior with incomplete information and fabricate the suitable pairwise comparisons. Therefore, we need a mathematical tool to formulate the ranker’s and the adversary’s behaviors, which has been extensively modeled as a two-player, non-cooperative game in the adversarial learning[20]. Specifically, the confrontation scenario between the online manipulator and the ranker who takes control of the original data source is formulated as a distributionally robust game that deals with the uncertainty of knowledge. The ranker’s set of actions corresponds to selecting pairwise comparisons and minimizing the difference between the aggregation result and the original ranking list. Meanwhile, the adversary’s set of actions corresponds to generate pairwise comparisons and minimizing the difference between the aggregation result and the desired ranking list. For two players, the upcoming data is the uncertain knowledge.

Remark 3. Although the offline [38] and online attackers have the same goal, different behavioral patterns result in the two having different knowledge and capabilities. Specifically, let $T_{0}$ be the stop** time of data collection, the offline attacker has full/partial knowledge of the comparison graph weight $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(T_{0})$. Then the offline manipulator has the ability to modify $\boldsymbol{w}(T_{0})$ in its entirety, increasing or decreasing the values of $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(T_{0})$ at arbitrary position. Meanwhile, the online manipulator of this paper sequentially obtains his/her knowledge but knows nothing about the forthcoming pairwise comparisons. More importantly, the online attacker will execute his/her strategy based on the knowledge $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(t)$ at each time step $t$ instead of waiting for the moment $T_{0}$. Thus, the greatest limitation on the ability of the online attacker is that he/she can only insert fabricated pairwise comparisons. The online attack paradigm could bypass the existing defense mechanisms of rank aggregation algorithms and break the barrier of time. We provide an example in Fig. 1. It is noteworthy that utilizing the offline method at each time step can’t achieve a similar result as the online method, since the offline method does not guarantee that the collected data keep unchanged.

Remark 4. In order to accomplish an effective online attack without modifying the collected data, the adversary will generate the most destructive data to inject based on the current partial information and stop when the ambiguity of ranking list falls below a certain level. This paper develops a general framework against the parametric models of rank aggregation, especially the BTL model. The proposed adversarial generation process, corresponding to the third core contribution, can designate the leading candidate of the aggregated ranking lists by HodgeRank and RankCentrality. In addition, the offline attack methods [37, 38] cannot yield the available attack results in the online manipulation setting of this paper.

3 General Framework

In this section, we systematically introduce the general framework for sequential manipulating against pairwise ranking algorithms. To mathematically characterize the successive interaction between the manipulator and the victims, we perform threat modeling to profile the attacker’s goal, knowledge and capability in Sec. 3.1 and dissect the online adversarial behavior. Then we develop the game-theoretic formulation between the online adversary and the offline rank aggregation procedure in Sec. 3.2 with particular emphasis on the uncertainty that the online manipulator must deal with. Such a game with fundamental uncertainty about future data and the opponent’s strategies and the settings of [37, 38] are significantly different. Meanwhile the existence of the distributionally robust Nash equilibrium is also established.

3.1 Threat Model of Online Adversary

Here we present the threat model of the manipulator to specify his/her goal, knowledge and capability with online behavioral pattern. The threat model helps to establish the online interactions between the purposeful attacker and the rank aggregation with pairwise comparisons.

The Goal of Online Adversary. Inducing the threatened rank aggregation approaches to produce the designated ranking is the goal of a manipulator. On the one hand, the adversary cannot interact directly with the threatened rank aggregation procedure due to the inevitable defense mechanisms. On the other hand, the collection of pairwise comparisons is an online process which is independent of the subsequent rank aggregation method. It often takes place in open environments and lacks adequate supervision. If the attacker could interfere with the data collection procedure, he/she has a high possibility of bypassing defense mechanisms and accomplishing manipulation. The data collection procedure is always treated as a random sampling process. All possible pairwise comparisons consist of the data stream. A random sampling algorithm will receive and choose the data which constructs the comparison graph. To archive manipulation, the adversary proactively disguises the crafted malicious data as part of the data stream. Then these malicious data could be adopted by sampling algorithms and used to construct a comparison graph. After sampling, the ranker produces the aggregated result based on the comparison graph. These sequential actions of the adversary will induce the ranker to produce a designated ranking result. If the ranking list meets the demand of adversary, we will say that the adversary has executed a successful manipulation.

We denote $\boldsymbol{\mathcal{A}}$ and $\boldsymbol{\mathcal{R}}$ be the adversary and the ranker respectively. Let $\boldsymbol{C}=\{c_{1},c_{2},\dots\}$ be a sequence of recurring pairwise comparisons involving at most $n$ candidates. The perturbed sequence by $\boldsymbol{\mathcal{A}}$ is $\boldsymbol{C}^{\prime}$. The sequence of pairwise comparisons will be transferred into the comparison graphs as (5). Suppose that $\boldsymbol{\mathcal{G}}(\boldsymbol{C})$ is the comparison graph constructed by $\boldsymbol{C}$. The relative ranking scores $\boldsymbol{\theta}$ and $\boldsymbol{\theta}^{\prime}$ are produced by $\boldsymbol{\mathcal{R}}$ with $\boldsymbol{\mathcal{G}}(\boldsymbol{C})$ and $\boldsymbol{\mathcal{G}}(\boldsymbol{C}^{\prime})$ accordingly. The non-adversarial rank aggregation can be portrayed as

$$\boldsymbol{\mathcal{R}}(\boldsymbol{\mathcal{G}}(\boldsymbol{C}))=\boldsymbol{\theta}.\$$ (7)

Then the rank aggregation result under online manipulation strategies would be

$$\boldsymbol{\mathcal{R}}(\boldsymbol{\mathcal{G}}(\boldsymbol{C}^{\prime}))=\boldsymbol{\theta}^{\prime}.$$ (8)

Although $\boldsymbol{\mathcal{A}}$ is able to achieve multiple objectives with the help of $\boldsymbol{\theta}^{\prime}$, designating the winner will be the most desired achievement of $\boldsymbol{\mathcal{A}}$. Therefore, we consider the following scenario: after the action of $\boldsymbol{\mathcal{R}}$, it holds that

$$\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}:=\left\{\boldsymbol{\theta}\in\mathbb{R}^{n}\ \Big{|}\ \underset{i\in[n],\ i\neq i_{0}}{\textbf{max}}\theta_{i}\leq\theta_{i_{0}}\right\},$$ (9)

where $i_{0}$ is the winner candidate desired by $\boldsymbol{\mathcal{A}}$. Then we will say that $\boldsymbol{\mathcal{A}}$ has a successful online manipulating strategy against $\boldsymbol{\mathcal{R}}$ by substituting $\boldsymbol{\mathcal{G}}(\boldsymbol{C})$ with $\boldsymbol{\mathcal{G}}(\boldsymbol{C}^{\prime})$ through sequential behavior. It is noteworthy that the goal in this paper implicitly requires sequential/online attack behavior, while [38] needs the help of offline manipulation strategy. The differences between online and offline strategies are shown in the following parts.

The Knowledge of Online Adversary. Let

$$\boldsymbol{C}(T)=\{c_{1},c_{2},\dots,c_{T}\}$$ (10)

be a sub-sequence of $\boldsymbol{C}$ with its first $T$ pairwise comparisons. Without loss of generality, the number of pairwise comparisons in $\boldsymbol{C}$ will be increased by $1$ at each step as

$$\boldsymbol{C}(T)=[\boldsymbol{C}(T-1),c_{T}],\ T\in\mathbb{N}.$$ (11)

As a consequence, the knowledge of $\boldsymbol{\mathcal{A}}$ at $T$ step, denoted $\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}(T)=[\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T-1)},\varphi(c_{T})]$, contains two parts:

• a subset of $\boldsymbol{C}(T-1)$ as

  $$\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T-1)}\subseteq\boldsymbol{C}{(T-1)},$$ (12)

• and the state of $c_{T}$:

  $$\varphi(c_{T})=\begin{cases}c_{T},&\ \text{if}\ \boldsymbol{\mathcal{A}}\ \text{obtains}\ c_{T},\\ \varnothing,&\text{otherwise,}\end{cases}$$ (13)

  where $\varnothing$ indicates that no pairwise comparisons will enter the sequence.

Based on the completeness of $\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}$, we consider the following two adversarial scenarios:

1. If it holds that

   $$\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}=\boldsymbol{C}{(T)},\ \forall\ T\in\mathbb{N},$$ (14)

   that is $\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T-1)}=\boldsymbol{C}{(T-1)}$ and $\varphi(c_{T})=c_{T}$, $\forall\ T\in\mathbb{N}$, we say that ${\boldsymbol{\mathcal{A}}}$ has the complete knowledge.

2. If there exists a time step $T$ such that

   $$\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}\subset\boldsymbol{C}{(T)},$$ (15)

   we say that $\boldsymbol{\mathcal{A}}$ has incomplete knowledge. Limited by time and cost, the incomplete state will be held throughout the whole adversarial operation.

Special attention needs to be paid to the fact that the online manipulator in this paper lacks prior information of subsequent data $\boldsymbol{C}/\boldsymbol{C}^{(T)}$ at $T$ step. The offline manipulator of [37, 38], on the other hand, doesn’t need the prior information but requires the length of $\boldsymbol{C}$ to no longer grow, i.e. there exist a step $T_{0}$ such that $|\boldsymbol{C}|=T_{0}$. Consequently, the offline adversary in [37, 38] is a special case of the online manipulator, who is the online manipulator at the step that all pairwise comparisons have been collected. Such a distinction will affect the abilities of the offline and online attackers.

The Capability of Online Adversary. The above goal and knowledge empower the online attacker with completely divergent capabilities from those of the offline attacker. The online manipulator $\boldsymbol{\mathcal{A}}$ is able to insert arbitrary pairwise comparisons into the data stream. Then the perturbed data will replace to produce the comparison graph for rank aggregation. More specifically, the fabricated pairwise comparisons with the knowledge $\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}$ is

$$\boldsymbol{c}^{\prime}{(\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)})}=[c^{\prime}(1),\dots,c^{\prime}(a_{T})],$$ (16)

where $a_{T}$ is the maximum number of possible insertions at $T$ step. This sequence (16) reflects $\boldsymbol{\mathcal{A}}$’s capability. It is noticed that $\boldsymbol{\mathcal{A}}$ is unable to change the pairwise comparisons in $\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}\subseteq\boldsymbol{C}{(T)}$. However, in addition to insertion, the offline attackers of [37, 38] could delete or flip a pairwise comparison $c_{t}\in\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}(T)\subseteq\boldsymbol{C}(T),\forall\ t\leq T$ even through $c_{t}$ is generated in the past or protected by the defense mechanisms. Therefore, the online attacker is more restricted than its offline counterpart. The observed sequence of pairwise comparisons for $\boldsymbol{\mathcal{R}}$ at $T$ step is

$$\boldsymbol{C}^{\prime}(T)=(\boldsymbol{C}^{\prime}(T-1),\boldsymbol{C}(T)/\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)},\boldsymbol{c}^{\prime}{(\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)})}).$$ (17)

(17) is a mixture of the collected data $\boldsymbol{C}^{\prime}(T-1)$, the data inaccessible to attackers $\boldsymbol{C}(T)/\boldsymbol{C}_{\boldsymbol{\mathcal{A}}}{(T)}$, and the fabricated pairwise comparisons (16).

3.2 Distributionally Robust Game between the Ranker and the Online Adversary

With the above threat modelling, we can further understand the adversarial scenario from a game-theoretic perspective. When there exists $\boldsymbol{\mathcal{A}}$, the pairwise comparisons for $\boldsymbol{\mathcal{R}}$ come form two sources: the original data stream $\boldsymbol{C}$ and the fraud data $\boldsymbol{C}^{\prime}/\boldsymbol{C}$. Due to the extreme difficulty of identifying the possible sources of pairwise comparisons, $\boldsymbol{\mathcal{R}}$ is only able to aggregate $\boldsymbol{C}^{\prime}$ and obtain a ranking list $\boldsymbol{\theta}^{\prime}$, which is different with the result $\boldsymbol{\theta}$ form $\boldsymbol{C}$. However, the existence of normal data stream $\boldsymbol{C}$ will alleviate the impact of $\boldsymbol{\mathcal{A}}$ on $\boldsymbol{\mathcal{R}}$ and try to keep $\boldsymbol{\theta}^{\prime}$ away from $\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$. With the help of defense and protection mechanisms, we believe that $\boldsymbol{\mathcal{R}}$ will select pairwise comparisons that will preserve the original result $\boldsymbol{\theta}$. At the same time, $\boldsymbol{\mathcal{A}}$ needs to induce $\boldsymbol{\mathcal{R}}$ with the interference of $\boldsymbol{C}$ and make $\boldsymbol{C}^{\prime}$ sufficient to support $\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$. As a consequence, this adversarial scenario is a game between $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ who choose pairwise comparisons to produce the desired ranking results.

To establish the adversarial game of the online adversary and the ranker, we first transfer the sequence of pairwise comparison at $T$ step $\boldsymbol{C}(T)=\{c_{1},\dots,c_{t},\dots,c_{T}\}$ into a comparison graph $\boldsymbol{\mathcal{G}}(\boldsymbol{C}(T))=\{\boldsymbol{V},\boldsymbol{E},\boldsymbol{w}(T)\}$. The vertex set $\boldsymbol{V}$ is the set of all candidates as $\boldsymbol{V}=[n]$ and the edge set $\boldsymbol{E}$ contains all directed edges between any pair of candidates as

$$\boldsymbol{E}=\{i\rightarrow j|i,\ j\in[n],\ i\neq j\}.$$ (18)

Here $\boldsymbol{w}(T)=\{w_{1,2}(T),w_{1,3}(T),\dots,w_{n,n-1}(T)\}$ is the weights of $\boldsymbol{E}$ in $\boldsymbol{C}(T)$ as

$$w_{i,j}(T)=\sum_{t=1}^{T}\ \mathbbm{I}[c_{t}=(i,j)]$$ (19)

where $\mathbbm{I}[\cdot]$ is the Iverson bracket. The weight $\boldsymbol{w}{(T)}$ represents how often a pairwise comparison occurs in $\boldsymbol{C}(T)$. Furthermore, $\boldsymbol{w}(T)$ can be treated as a random variable defined on the probability space $(\boldsymbol{\mathfrak{C}},\boldsymbol{\mathcal{E}},\mathbb{P})$:

$$\boldsymbol{w}(T)\sim\mathbb{P},$$ (20)

where $\boldsymbol{\mathfrak{C}}$ is the sample space of all possible pairwise comparisons involving $n$ candidates

$$\boldsymbol{\mathfrak{C}}=\{(i,j)\ |\ i,j\in[n],\ i\neq j\},$$ (21)

$\boldsymbol{\mathcal{E}}$ is the event space of all sequences with length $T$ and $\mathbb{P}$ is a probability function. Consequently, the data sequence $\boldsymbol{C}(T)$ is associated with data distribution $\mathbb{P}$ which describes the occurrence frequency of different pairwise comparisons.

A notable characteristic of the adversarial game in this paper is that the decision-making processes of $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ involve uncertainty: the true distribution of the mixed sequence (the observed weight) is unknown to both $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ during the whole procedure. All players bear the risk due to the uncertainty. Then the resulting Nash equilibrium may be different from the equilibrium with the true distribution. The uncertainty drives $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ to adopt conservative strategies. To be more specific, we consider the following Nash equilibrium problem: at any time step, each player needs to make decisions prior to the realization of uncertainty by minimizing their expected dis-utility with the most pessimistic situation:

$$\underset{\boldsymbol{a}_{r}\in\mathbb{Z}^{N}_{+}}{\ \ \textbf{{min}}\phantom{g}}\ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{sup}}\phantom{g}}\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}_{r},\boldsymbol{a}_{-r},\boldsymbol{\pi}_{r},\boldsymbol{w})\Big{]},$$ (22)

where $r$ represents the $r^{\text{th}}$ player in the game. Such a distributionally robust game (DRG) models the interaction between the ranker and adversary. Each player in this game holds a continuous dis-utility function as

$$f_{r}:\mathbb{Z}^{N}_{+}\times\mathbb{Z}^{N}_{+}\times\mathbb{N}^{n}\times\mathbb{R}^{N}\rightarrow\mathbb{R},$$ (23)

$N$ is the cardinality of $\boldsymbol{\mathcal{C}}$ as $N:=|\boldsymbol{\mathcal{C}}|=n(n-1)$. The action of $r$, $\boldsymbol{a}_{r}=\{a^{r}_{1,2},\dots,a^{r}_{n,n-1}\}\in\mathbb{Z}^{N}_{+}$, indicates the number of pairwise comparisons selecting by $r$, and $\boldsymbol{a}_{-r}$ represent the actions of $r$’s opponents. $\boldsymbol{\pi}_{r}$ is the desired ranking list of player $r$. Here the “maximum” operation w.r.t $\mathbb{P}$ means the player $r$ decides his/her optimal strategy on the worst expected value of $f_{r}$ from the set of distributions $\boldsymbol{\mathcal{P}}_{r}$ which is constructed from the partially observed information $\boldsymbol{w}_{r}$. (22) is known as the distributionally robust game [35] in the literature. The solution of (22) is named as distributionally robust Nash equilibrium. If any ambiguity set $\boldsymbol{\mathcal{P}}_{r}$ only contains a single distribution, (22) collapses to a stochastic game problem [13]. It is noticed that the ranker $\boldsymbol{\mathcal{R}}$ is often unaware of the existence of $\boldsymbol{\mathcal{A}}$ and the corresponding action turns out to be

$$\underset{\boldsymbol{a}_{r}}{\textbf{{min}}}\ \ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}_{0}}\Big{[}f_{r}(\boldsymbol{a}_{r},\boldsymbol{w},\boldsymbol{\pi}_{r})\Big{]},$$ (24)

which means the ranker will focus on the original ranking list and choose the data with some sampling methods. Meanwhile the player of ${\boldsymbol{\mathcal{A}}}$ still needs to consider the most pessimistic situation as (22).

Definition 1 (Distributionally Robust Nash Equilibrium).

A tuple $(\boldsymbol{a}^{*}_{1},\dots,\boldsymbol{a}^{*}_{R})$ is a distributionally robust Nash equilibrium (DRNE) if

$$\boldsymbol{a}^{*}_{r}\in\underset{\boldsymbol{a}_{r}}{\textbf{{arg~{}min}}}\ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\textbf{{sup}}}\ \ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}_{r},\boldsymbol{a}^{*}_{-r},\boldsymbol{\pi}_{r},\boldsymbol{w})\Big{]},$$ (25)

where

$$\boldsymbol{a}^{*}_{-r}=\sum_{s\neq r}\boldsymbol{a}^{*}_{s}.$$ (26)

This definition shows that the DRNE is a solution of the corresponding DRG (22). Here we consider the case of $R=2$, say that the game between $\boldsymbol{\mathcal{R}}$ and ${\boldsymbol{\mathcal{A}}}$. In what follows, we investigate the existence of DRNE by the following theorem. The detailed proof can be found in the supplementary materials.

Theorem 1.

There exists a DRNE (25) if the following states hold for any $r=1,\dots,R$.

1. Given $(\boldsymbol{a}_{-r},\boldsymbol{\pi}_{r},\boldsymbol{w})$, $f_{r}(\cdot,\boldsymbol{a}_{-r},\boldsymbol{\pi}_{r},\boldsymbol{w})$ is convex over $\mathbb{Z}^{N}_{+}$.

2. $\mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\big{[}f_{r}(\boldsymbol{a}_{r},\boldsymbol{a}_{-r},\boldsymbol{\pi}_{r},\boldsymbol{w})\big{]}$ has finite values for any $(\boldsymbol{a}_{r},\boldsymbol{a}_{-r})$, $\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}$ and $\boldsymbol{\pi}_{r}$ is a permutation of $[n]$.

3. $\boldsymbol{\mathcal{P}}_{r}$ has weakly compactness.

This result tells us that there exists at least one stable state for both $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ with the above conditions holding. The next key step toward executing manipulation is to identify an equilibrium state that is favorable to $\boldsymbol{\mathcal{A}}$ or not. When an equilibrium state is favorable to $\boldsymbol{\mathcal{A}}$, the perturbed data $\boldsymbol{C}^{\prime}(T)$ could lead $\boldsymbol{\mathcal{R}}$ generating $\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$.

3.3 Successful Opportunity to Sequential Manipulation

Without prior knowledge of the original data distribution, analyzing the equilibrium of the proposed distributionally robust game is really challenging. Here we try to dissect the outcome after the adversarial game directly. At the end of the distributionally robust game between $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$, we have a sequence of pairwise comparisons (17) for construction of the comparison graph. To simulate the competitive results of (25), we treat (17) as a stochastic process, the output of a sampling method $\boldsymbol{\mathcal{S}}$. The random nature of the sampling process replaces the uncertainty of the distributionally robust game. In addition, utilizing $\boldsymbol{\mathcal{S}}$ to analyze $\boldsymbol{C}^{\prime}(T)$ makes our subsequent discussion more pertinent to actual confrontation scenarios. Here we show that the two classic sampling methods will become an accomplice of the online manipulator, who help to generate the stable sequences favoring to $\boldsymbol{\mathcal{A}}$’s goal.

In the non-adversarial scenario, an $(\epsilon,\delta)$-representative sampling method always suffices to take only a small number of random samples $\boldsymbol{C}$ in order to represent the data source $\boldsymbol{\mathcal{C}}$ truthfully [48]. Even with the aimless attacker[8] who still adopts the original data source $\boldsymbol{\mathcal{C}}$ as the source of perturbation, Bernoulli and the reservoir sampling methods could lead $\boldsymbol{\mathcal{R}}$ to generate the same ranking result. To be specific, the perturbed sequence $\boldsymbol{C}^{\prime}_{1}$ and the original sequence $\boldsymbol{C}$ would produce different comparison graph weights $\boldsymbol{w}_{\boldsymbol{C}^{\prime}_{1}}$ and $\boldsymbol{w}_{\boldsymbol{C}}$. However, they still obey the same distribution as

$$\boldsymbol{w}_{\boldsymbol{C}},\ \boldsymbol{w}_{\boldsymbol{C}^{\prime}_{1}}\sim\mathbb{P}_{0},$$ (27)

where $\mathbb{P}_{0}$ is the distribution of original comparison graph weights. If the probability distribution of comparison graph’s weight $\boldsymbol{w}$ is $\mathbb{P}_{0}$, it holds that

$$\boldsymbol{\mathcal{R}}(\boldsymbol{w}_{\boldsymbol{C}})=\boldsymbol{\mathcal{R}}(\boldsymbol{w}_{\boldsymbol{C}^{\prime}})=\boldsymbol{\theta}_{0}.$$ (28)

Consequently, $\boldsymbol{\mathcal{R}}$ generates the original ranking list $\boldsymbol{\pi}_{\boldsymbol{\theta}_{0}}$ even if the aimless attacker $\boldsymbol{\mathcal{A}}$ exists. This so-called “adversarial robustness”[8] is on side of the sampling methods.

Unfortunately, the attacker will be sophisticated in the real confrontation scenario. He/she could construct new data sources instead of simply using $\boldsymbol{\mathcal{C}}$. For example, given any $\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$ like (9), $\boldsymbol{\mathcal{A}}$ could generate the pairwise comparison through the BTL model: the larger the $\theta_{i}^{\prime}/\theta_{j}^{\prime}$, the higher the probability of generating pairwise comparison $i\succ j$. Such actions construct the adversarial data source $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$ whose underlying distribution is $\mathbb{P}_{\boldsymbol{\mathcal{A}}}$ which is distinct from the $\mathbb{P}_{0}$ as $\boldsymbol{\mathcal{R}}$ will produce the manipulated score:

$$\boldsymbol{\mathcal{R}}(\boldsymbol{w})=\boldsymbol{\theta}^{\prime},\ \boldsymbol{w}\sim\mathbb{P}_{\boldsymbol{\mathcal{A}}}.$$ (29)

The distributionally robust game between $\boldsymbol{\mathcal{C}}$ and $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$ (22) creates the mixed data source $\boldsymbol{\mathcal{C}}^{\prime}$. Once the underlying distribution of $\boldsymbol{\mathcal{C}}^{\prime}$ is consistent with $\mathbb{P}_{\boldsymbol{\mathcal{A}}}$, it holds that

$$\boldsymbol{w}_{\boldsymbol{C}^{\prime}_{2}}\sim\mathbb{P}_{\boldsymbol{\mathcal{A}}},$$ (30)

where $\boldsymbol{C}^{\prime}_{2}$ is the perturbed sequence form $\boldsymbol{\mathcal{C}}^{\prime}$. There is no doubt that the sampled data $\boldsymbol{C}^{\prime}_{2}$ will lead $\boldsymbol{\mathcal{R}}$ to obtain $\boldsymbol{\pi}_{\boldsymbol{\theta}^{\prime}}$ as (29). We call such an attacker the “purposeful adversary”. Then the $(\epsilon,\delta)$-representative sampling algorithms would fall into a trap. From the perspective of the purposeful adversary, the original “representativeness” turns into the “vulnerability”. This “vulnerability” is the other side of the sampling methods like Bernoulli and reservoir.

We introduce some definitions which will help to establish the vulnerability results of Bernoulli and reservoir sampling methods. The data stream from $\boldsymbol{\mathcal{C}}^{\prime}$ is a mixture of two data streams from $\boldsymbol{\mathcal{C}}$ and $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$. The mixed data source $\boldsymbol{\mathcal{C}}^{\prime}$ satisfies

$$\boldsymbol{\mathcal{C}}^{\prime}\subseteq\boldsymbol{\mathcal{C}}\cup\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}},\ \boldsymbol{\mathcal{C}}^{\prime}\cap\boldsymbol{\mathcal{C}}\neq\varnothing,\ \boldsymbol{\mathcal{C}}^{\prime}\cap\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}\neq\varnothing.$$ (31)

Here we consider the following two types of mixtures, which correspond to different behaviors of players in the distributionally robust game. In fact, the dynamic stream comes from the distributionally robust game whose players execute (22) and the static stream corresponds to players who executes (24).

Definition 2 (Static stream).

Let $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ be a sequence from $\boldsymbol{\mathcal{C}}^{\prime}$ which is a mixture of two data streams from $\boldsymbol{\mathcal{C}}$ and $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$. For any $c_{t}\in\boldsymbol{\mathcal{C}}^{\prime}\cap\boldsymbol{\mathcal{C}}$, if the generation of $c_{t}$ is independent with $\{c_{1},\dots,c_{t-1}\}$, we call $\boldsymbol{C}$ from $\boldsymbol{\mathcal{C}}^{\prime}$ is a static stream.

Definition 3 (Dynamic stream).

Let $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ be a sequence from $\boldsymbol{\mathcal{C}}^{\prime}$ which is a mixture of two data streams from $\boldsymbol{\mathcal{C}}$ and $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$. For any $c_{t}\in\boldsymbol{\mathcal{C}}^{\prime}\cap\boldsymbol{\mathcal{C}}$ and $c_{t}\notin\boldsymbol{\mathcal{C}}^{\prime}\cap\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$, if the generation of $c_{t}$ is dependent with $\{c_{1},\dots,c_{t-1}\}$, we call $\boldsymbol{C}$ from $\boldsymbol{\mathcal{C}}^{\prime}$ is a dynamic stream.

The concept of $\epsilon$-approximation measures the similarity between two sequences from the same data source.

  Input : the number of turns $T$, the sampling parameter $\varrho$, the true ranking list $\boldsymbol{\pi}_{0}$, the target ranking list $\boldsymbol{\pi}_{\boldsymbol{\mathcal{A}}}$, the dis-utility functions of the original and adversarial data sources $f$, $f_{\boldsymbol{\mathcal{A}}}$, the stop** time $S_{0}$ and the unweighted complete comparison graph $\boldsymbol{\mathcal{G}}=\{\boldsymbol{E},\boldsymbol{V}\}$.

1

2Initialization: let the stream, the sampled sequence and the comparison graph weights be empty:

$$\boldsymbol{C}=\varnothing,\ \boldsymbol{C}^{\prime}=\varnothing,\ \boldsymbol{w}(0)=\boldsymbol{0}.$$

3for $t=1$ to $T$ do

4       Action of the ranker $\boldsymbol{\mathcal{R}}$:

$$\boldsymbol{a}\in\underset{\phantom{\boldsymbol{\mathcal{P}}}\boldsymbol{a}\phantom{\boldsymbol{\mathcal{P}}}}{\textbf{{argmin}}}\underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}(\boldsymbol{w}(t-1))}{\textbf{{max\phantom{g}}}}\ \ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\big{[}f(\boldsymbol{a},\boldsymbol{w},\boldsymbol{\pi}_{0})\big{]}.$$

5      Update the data stream $\boldsymbol{C}$:

$$\boldsymbol{C}\leftarrow\boldsymbol{a}.$$

6      Action of sampling method $\boldsymbol{\mathcal{S}}$:

$$\boldsymbol{C}^{\prime}\leftarrow\boldsymbol{\mathcal{S}}(\boldsymbol{C},\varrho).$$

7      Update the weights:

$$\boldsymbol{w}(t)\leftarrow\boldsymbol{C}^{\prime}.$$

8      Let $s=1$ and update the knowledge:

$$\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(s)=\text{mask}(\boldsymbol{w}(t)).$$

9      while $s<S_{0}$ and $\boldsymbol{w}{(s)}\neq\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}{(S_{0})}$ do

10             Action of the online manipulator $\boldsymbol{\mathcal{A}}$:

$$\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \boldsymbol{a}_{\boldsymbol{\mathcal{A}}}(s)\in$$

$$\underset{\phantom{\boldsymbol{\mathcal{P}}}\boldsymbol{a}\phantom{\boldsymbol{\mathcal{P}}}}{\textbf{{argmin}}}\ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}(\boldsymbol{w}_{\boldsymbol{\mathcal{A}}(s)})}{\textbf{{max\phantom{g}}}}\ \ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\big{[}f_{\boldsymbol{\mathcal{A}}}(\boldsymbol{a},\boldsymbol{w},\boldsymbol{\pi}_{\boldsymbol{\mathcal{A}}})\big{]}.$$

11            Update the data stream $\boldsymbol{C}$:

$$\boldsymbol{C}\leftarrow\boldsymbol{a}_{\boldsymbol{\mathcal{A}}}(s).$$

12            Action of sampling method $\boldsymbol{\mathcal{S}}$:

$$\boldsymbol{C}^{\prime}\leftarrow\boldsymbol{\mathcal{S}}(\boldsymbol{C},\varrho).$$

13            Let $s\leftarrow s+1$ and Update the knowledge:

$$\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(s)\leftarrow\boldsymbol{C}^{\prime}.$$

14       end while

15      

16      Update the weights:

$$\boldsymbol{w}{(t)}\leftarrow\boldsymbol{C}^{\prime}.$$

17 end for

Output : $\boldsymbol{\mathcal{G}}^{(T)}=\{\boldsymbol{E},\boldsymbol{V},\boldsymbol{w}^{(T)}\}$.

Algorithm 1 Online Interaction between $\boldsymbol{\mathcal{A}}$ and $\boldsymbol{\mathcal{R}}$

Definition 4 ($\epsilon$-approximation).

A sequence $\boldsymbol{C}_{1}$ is an $\epsilon$-approximation of sequence $\boldsymbol{C}_{0}$ with respect to the data source $\boldsymbol{\mathcal{C}}$, if there exists an $\epsilon\in(0,1)$ such that

$$|d_{\boldsymbol{\mathcal{C}}}(\boldsymbol{C}_{0})-d_{\boldsymbol{\mathcal{C}}}(\boldsymbol{C}_{1})|\leq\epsilon,$$ (32)

where $\boldsymbol{\mathcal{C}}$ is a data source, $d_{\boldsymbol{\mathcal{C}}}(\boldsymbol{C})$ is the density of $\boldsymbol{\mathcal{C}}$ in the sequence $\boldsymbol{C}$, the fraction of pairwise comparisons in $\boldsymbol{\mathcal{C}}$ that are also in $\boldsymbol{C}$:

$$d_{\boldsymbol{\mathcal{C}}}(\boldsymbol{C})=\mathbb{P}\big{(}c\in\boldsymbol{\mathcal{C}}\ \big{|}\ c\in\boldsymbol{C}\big{)}.$$ (33)

This definition give us a similarity metric between two sequences with the density function. It is noteworthy that the lengths of $\boldsymbol{C}_{0}$ and $\boldsymbol{C}_{1}$ could be different, where the length of $\boldsymbol{C}_{0}$ could be infinite and $\boldsymbol{C}_{1}$ only has a limited number of elements. To portray the data source and analyze the vulnerability of sampling methods, we adopt the $(\epsilon,\delta)$-representativeness to quantify the quality of a sampling method w.r.t a data source.

Definition 5 ($(\epsilon,\delta)$-representativeness).

A sampling method is called $(\epsilon,\delta)$-representative if the sampled sequence of $\boldsymbol{C}_{1}$ is an $\epsilon$-approximation of the whole stream $\boldsymbol{C}_{0}$ with respect to $\boldsymbol{\mathcal{C}}$, with probability at least $1-\delta$.

The following theoretical results show that as long as the sampling parameters satisfy certain conditions, $\boldsymbol{\mathcal{S}}$ must be $(\epsilon,\delta)$-representative with respect to $\boldsymbol{\mathcal{C}}^{\prime}$.

Theorem 2.

Let $\boldsymbol{\mathcal{C}}^{\prime}$ be a mixture of $\boldsymbol{\mathcal{C}}$ and $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$ satisfying (31).

1. For any static stream $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ from $\boldsymbol{\mathcal{C}}^{\prime}$,

   • if the parameter of Bernoulli sampling method satisfies

     $$\varrho\geq c\cdot\frac{\textbf{{log}}\ n(n-1)+\textbf{{ln}}(1/\delta)}{\epsilon^{2}T},$$ (34)

     where $T$ is the number of sampling, the output of Bernoulli sampling is $(\epsilon,\delta)$-representative with respect to $\boldsymbol{\mathcal{C}}^{\prime}$.

   • if the parameter of reservoir sampling method satisfies

     $$\varrho\geq c\cdot\frac{\textbf{{log}}\ n(n-1)+\textbf{{ln}}(1/\delta)}{\epsilon^{2}},$$ (35)

     the output of reservoir sampling is $(\epsilon,\delta)$-representative with respect to $\boldsymbol{\mathcal{C}}^{\prime}$.

2. For any dynamic stream $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ from $\boldsymbol{\mathcal{C}}^{\prime}$,

   • if the parameter of Bernoulli sampling method satisfies

     $$\varrho\geq 10\cdot\frac{\displaystyle\textbf{{ln}}\ |\boldsymbol{\mathcal{C}}^{\prime}|+\textbf{{ln}}(4/\delta)}{\epsilon^{2}T},$$ (36)

     where $T$ is the number of sampling, the output of Bernoulli sampling is $(\epsilon,\delta)$-representative with respect to $\boldsymbol{\mathcal{C}}^{\prime}$.

   • if the parameter of reservoir sampling method satisfies

     $$\varrho\geq 10\cdot\frac{\displaystyle\textbf{{ln}}\ |\boldsymbol{\mathcal{C}}^{\prime}|+\textbf{{ln}}(2/\delta)}{\epsilon^{2}},$$ (37)

     the output of reservoir sampling is $(\epsilon,\delta)$-representative with respect to $\boldsymbol{\mathcal{C}}^{\prime}$.

Theorem 2 indicates that the mixed data source $\boldsymbol{\mathcal{C}}^{\prime}$ could be a DRNE which will be a favor to $\boldsymbol{\mathcal{A}}$. When the underlying distribution of $\boldsymbol{\mathcal{C}}^{\prime}$ is consistent with $\boldsymbol{\mathcal{C}}_{\boldsymbol{\mathcal{A}}}$, the Bernoulli and reservoir sampling methods always select the data that consisted with $\boldsymbol{\pi}(\boldsymbol{\theta}^{\prime})$ with high probability. The detailed proof can be found in the supplementary materials. Now we formally define the online adversarial interaction between $\boldsymbol{\mathcal{R}}$ and $\boldsymbol{\mathcal{A}}$ discussed in this paper.

• The behavior of $\boldsymbol{\mathcal{R}}$ relies on the original data source $\boldsymbol{\mathcal{C}}$ whose distribution $\mathbb{P}_{0}$ would lead $\boldsymbol{\mathcal{R}}$ to generate $\boldsymbol{\theta}$. The pairwise comparisons from $\boldsymbol{\mathcal{C}}$ would be contrary to the attacker’s goal as $\boldsymbol{\theta}\notin\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$. The way that $\boldsymbol{\mathcal{C}}$ generates data can be active or passive, corresponding to dynamic and static streams, respectively. In fact, $\boldsymbol{\mathcal{C}}$ often takes a passive approach like (24) and its dis-utility $f$ is dependent of the sampler $\boldsymbol{\mathcal{S}}$. When $\boldsymbol{\mathcal{C}}$ is active, we consider that the defense and protection mechanisms exist. It means that $\boldsymbol{\mathcal{C}}$ will help the ranker $\boldsymbol{\mathcal{R}}$ to take actions like (22) and maintain $\boldsymbol{\pi}_{\boldsymbol{\theta}_{0}}$ as much as possible.

• In the proposed adversarial game, the online manipulator $\boldsymbol{\mathcal{A}}$ will try his/her best to creat data source $\boldsymbol{\mathcal{C}^{\prime}}$ whose distribution $\mathbb{P}_{\boldsymbol{\mathcal{A}}}$ would induce $\boldsymbol{\mathcal{R}}$ to obtain $\boldsymbol{\theta}^{\prime}$. The sequential strategy that $\boldsymbol{\mathcal{A}}$ employs along the way is dynamic. Based on the current partial information $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(t)$, $\boldsymbol{\mathcal{A}}$ would like to choose the most helpful comparisons by $f_{\boldsymbol{\mathcal{A}}}$, which reduce the divergence between the potential aggregated result and the target ranking. $\boldsymbol{\mathcal{A}}$ needs to be aware of the existence of original ranking data which is always an obstruction of archiving his/her goal. Therefore, $\boldsymbol{\mathcal{A}}$ needs to take the most pessimistic action as (22).

All data sources are sampled with the same sampler $\boldsymbol{\mathcal{S}}$. The proposed online adversarial interaction is summarized in Algorithm 1. The ranker $\boldsymbol{\mathcal{R}}$ will obtain the aggregated result with the output $\boldsymbol{\mathcal{G}}(T)=\{\boldsymbol{E},\boldsymbol{V},\boldsymbol{w}(T)\}$. The remaining question of executing manipulation is how to construct the data source $\boldsymbol{\mathcal{C}}^{\prime}$ which owns $\mathbb{P}_{\boldsymbol{\mathcal{A}}}$ as the underlying distribution. We provide the details of dynamic attack strategy, say the adversarial pairwise comparison generation process, in the next section.

4 Adversarial Generation Process

In Section 4.1, we propose two adversarial policies for adversary $\boldsymbol{\mathcal{A}}$ with complete knowledge and present the asymptotic optimality of these policies. Then, we provide the efficient optimization algorithm for incomplete information in Section 4.2.

4.1 Sequential Generation with Complete Knowledge

The strategies of $\boldsymbol{\mathcal{A}}$ try to maximize the consistency between full order with his/her goal $\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\mathcal{A}}}$ in the online adversarial game. $\boldsymbol{\mathcal{A}}$ should choose the most destructive comparisons to inject based on the current partial information and stop when the ambiguity of ranking list falls below a certain level. The actions of adversary consist of two components: an adaptive generation rule and a stop** time. For the adaptive rule, we adopt probabilistic rules which contain the deterministic rules as the special cases. Let $\lambda_{i,j}$ denote the probability of generating $(i,j)$ and $\boldsymbol{\lambda}=[\lambda_{1,2},\dots,\lambda_{n,n-1}]\in\boldsymbol{\Delta}$ be the categorical distribution, where

$$\boldsymbol{\Delta}=\left\{\ \boldsymbol{\lambda}\ \Bigg{|}\ \underset{(i,j)}{\sum}\ \lambda_{i,j}=1,\lambda_{i,j}\geq 0\right\}$$ (38)

is a probability simplex over $n(n-1)$ pairs. In each turn, the distributionally robust Nash equilibrium (25) (line $7$ in Algorithm 1) decides $c_{\boldsymbol{\mathcal{A}}}$ according to $\boldsymbol{\lambda}$, which depends on the goal $\boldsymbol{\theta}^{\prime}$ and the knowledge $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}$. The generative rules in the online adversarial game constitute the following set:

$$\boldsymbol{\Lambda}=\Big{\{}\boldsymbol{\lambda}^{(s)}\ \Big{|}\ \boldsymbol{\lambda}^{(s)}\in\boldsymbol{\Delta},\ s=1,\ 2,\dots\Big{\}}.$$ (39)

There is no doubt that the longer the stop** time [26], the higher the possibility of achieving the manipulation. However, the adversary can’t insert without limitations. A large amount of $\{c_{\boldsymbol{\mathcal{A}}}\}$ will alert the ranker $\boldsymbol{\mathcal{R}}$ thus lose the opportunity to attack. Consequently, we measure the quality of sequential manipulation via the generation cost and the ranking consistency. The risk associated with the stop** time $S$ is defined as

$$\mathfrak{R}(S)=\chi\cdot S.$$ (40)

Here the constant $\chi>0$ indicates the relative cost of inserting one $c_{\boldsymbol{\mathcal{A}}}$ into $\boldsymbol{C}$ (line 10 in Algorithm 1). The choice of $\chi$ is associated with the difficulty of attack against the specific ranking system.

On the other hand, we adopt Kendall-$\tau$ distance to measure the risk of inconsistency: given a full ranking list $\boldsymbol{\pi}(\boldsymbol{\Lambda},S)$ from the victim $\boldsymbol{\mathcal{R}}$ with $(\boldsymbol{\Lambda},S)$, we convert $\boldsymbol{\pi}(\boldsymbol{\Lambda},S)$ to the binary decisions set $\boldsymbol{R}_{\boldsymbol{\Lambda},S}$ over pairs

$$\boldsymbol{R}(\boldsymbol{\Lambda},S)=\Big{\{}r_{i,j}\in\{0,1\}\ \Big{|}\ i,j\in[n],\ i\neq j\Big{\}}$$ (41)

where

$$r_{i,j}=\left\{\begin{array}[]{cl}1,&i\succ_{\boldsymbol{\pi}(\boldsymbol{\Lambda},S)}j,\\ 0,&\text{otherwise},\\ \end{array}\right.$$ (42)

and $i\succ_{\boldsymbol{\pi}}j$ means that $i$ is located before $j$ in $\boldsymbol{\pi}$. The risk of inconsistency between $\boldsymbol{\pi}(\boldsymbol{\Lambda},S)$ and the target ranking induced by $\boldsymbol{\theta}^{\prime}=[\theta^{\prime}_{1},\dots,\theta^{\prime}_{n}]$ is defined by

$$\ \ \mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))$$ (43)

$$= \ \ \underset{(i,j)}{\sum}\ \mathbbm{I}[\theta^{\prime}_{i}<\theta^{\prime}_{j}]r_{i,j}+\mathbbm{I}[\theta^{\prime}_{i}>\theta^{\prime}_{j}](1-r_{i,j}).$$

In this paper, we consider the ranking algorithms tailored to the BTL model, say HodgeRank and RankCentrality. The ranking decision of these two victims is locating the candidates based on appropriate estimates of the latent preference scores in the full ranking list. Consequently, our proposed generation policy will depend on the maximum likelihood estimation (MLE) of BTL model. Given the ranker $\boldsymbol{\mathcal{R}}$ under attack, we analyze the combination of (40) and (43) under the Bayesian decision framework, in which the manipulated preference score of the victim is assumed to be random and follows a prior distribution $\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})$ which is specified by the adversary. The Bayesian risk associated with the victim $\boldsymbol{\mathcal{R}}$ is defined as

$$\mathfrak{R}(\boldsymbol{\Lambda},S)=\mathbb{E}[\mathfrak{R}(S)+\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))],$$ (44)

where the expectation $\mathbb{E}[\cdot]$ is taken with respect to the adaptive generation rule $\boldsymbol{\Lambda}$ and the stop** time $S$. The adversary $\boldsymbol{\mathcal{A}}$ hopes to execute the optimal policy $(\boldsymbol{\Lambda}^{*},S^{*})$ which will lead to the minimal risk $\mathfrak{R}^{*}$

$$\mathfrak{R}^{*}=\underset{\boldsymbol{\Lambda},S}{\textbf{{inf}}}\ \mathfrak{R}(\boldsymbol{\Lambda},S).$$ (45)

For any given cost $\chi$, the value of $\mathfrak{R}^{*}$ represents the effect of manipulation: a small $\mathfrak{R}^{*}$ indicates $\boldsymbol{\mathcal{A}}$ would be close to his/her purpose and vice versa. However, obtaining the analytical form of $(\boldsymbol{\Lambda}^{*},S^{*})$ is typically infeasible. We turn to the asymptotic optimality [18] which is the other well-known evaluation of sequential decision. A policy $(\boldsymbol{\Lambda},S)$ for $\boldsymbol{\mathcal{R}}$ is said to be asymptotically optimal if

$$\underset{\chi\rightarrow 0}{\textbf{{inf}}}\ \frac{\mathfrak{R}(\boldsymbol{\Lambda},S)}{\mathfrak{R}^{*}}=1.$$ (46)

By the above definition, we know that the asymptotically optimal policy could work when the relative cost $\chi$ converges to $0$. Although $\chi$ cannot be ignored, the relative cost is negligible compared to the huge profit from a successful manipulation. The adversary $\boldsymbol{\mathcal{A}}$ can do whatever it takes to manipulate the ranking results. Therefore, the asymptotically optimal policy is still important for $\boldsymbol{\mathcal{A}}$. Now we pay attention to the inner loop of Algorithm 1 (from line 8 to 13). Suppose the log-likelihood function of the BTL model with a comparison graph $\boldsymbol{\mathcal{G}}=\{\boldsymbol{V},\boldsymbol{E},\boldsymbol{w}_{\mathcal{A}}(S)\}$ is

$$L\bigg{(}\boldsymbol{\theta},\boldsymbol{w}_{\mathcal{A}}(S)\bigg{)}=\sum_{(i,j)}w_{i,j}(S)\cdot\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}),$$ (47)

where $g_{i,j}(\boldsymbol{\theta})$ is the probability mass function of $i\succ j$ with $\boldsymbol{\theta}$ and $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}(S)$ represent the complete knowledge. The corresponding MLE with adversarial goal $\rho_{\boldsymbol{\theta}^{\prime}}$ is

$$\boldsymbol{\hat{\theta}}_{S}=\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{arg min}}}\ -L\bigg{(}\boldsymbol{\theta},\boldsymbol{w}_{\mathcal{A}}(S)\bigg{)},$$ (48)

where $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ is the support of the prior probability density function $\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})$.

Stop** Time. Based on the generalized likelihood ratio statistic [22], we leverage two types of stop** time to decide the number of inserted pairwise comparisons with the complete knowledge:

$$S_{1} = \ \ \textbf{{inf}}\left\{\ S>0\ \ \Bigg{|}\ \ \sum_{(i,j)}e^{-|\Delta_{i,j}L_{S}|}\leq e^{-z_{\alpha}(\chi)}\right\}$$ (49)

$$S_{2} = \ \ \textbf{{inf}}\left\{\ S>0\ \ \Bigg{|}\ \ \ \underset{(i,j)}{\textbf{{min}}}\ \ |\Delta_{i,j}L_{S}|\ \geq\ z_{\alpha}(\chi)\ \right\},$$

where $z_{\alpha}(\cdot)$ is a monotone function with $\alpha\in(0,1)$

$$z_{\alpha}(\chi)=|\textbf{{log}}(\chi)|\cdot\big{(}1+|\textbf{{log}}(\chi)|^{-\alpha}\big{)}.$$ (50)

Here $\Delta_{i,j}L_{S}$ measures the difference between $\theta_{i}\geq\theta_{j}$ and $\theta_{i}\leq\theta_{j}$ in (47):

$$\Delta_{i,j}L_{S} = \ \ \underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{min}}\ }-L\bigg{(}\boldsymbol{\theta},\boldsymbol{w}_{\mathcal{A}}(S)\bigg{)}$$ (51)

$$\ \ -\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{min}}\ }-L\bigg{(}\boldsymbol{\theta},\boldsymbol{w}_{\mathcal{A}}(S)\bigg{)},$$

where

$$\boldsymbol{\Theta}_{i,j}=\big{\{}\boldsymbol{\ \theta}\in\mathbb{R}^{n}_{+}\ |\ \theta_{i}\geq\theta_{j}\ \big{\}}\cap\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}).$$ (52)

Generally speaking, the proposed criteria (49) will stop the generation process when the likelihood (47) can decide $\theta_{i}\geq\theta_{j}$ or vice versa.

Generation Rule. Next we discuss the probabilistic generation rule for sequential manipulation. Inspired by the existing sequential design for rank aggregation [16], selecting the desired $\boldsymbol{\lambda}^{(S)}$ equals to maximize the consistency between $\boldsymbol{\hat{\theta}}_{S}$ and the goal $\boldsymbol{\theta}^{\prime}$. Such consistency could be measured by the minimum of the mutual information between $g_{i,j}(\boldsymbol{\hat{\theta}}_{S})$ and any other $g_{i,j}(\boldsymbol{\tilde{\theta}})$ when $(i,j)$ is generated according to $\boldsymbol{\lambda}^{(S)}$:

$$\underset{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{min}}} \ \ \sum_{(i,j)}\lambda^{(S)}_{i,j}\cdot g_{i,j}(\boldsymbol{\hat{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\hat{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})},$$ (53)

subject to $$\ \ \boldsymbol{\pi}(\boldsymbol{\hat{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}}).$$

It is noteworthy that (53) also minimizes the drift of log-likelihood ratio statistics between two distributions of pairwise comparisons specified by $\boldsymbol{\hat{\theta}}_{S}$ and $\boldsymbol{\tilde{\theta}}$ under the BTL model and the probabilistic generation $\boldsymbol{\lambda}^{(S)}$. The smaller the minimum value of (53) corresponding to the given $\boldsymbol{\lambda}^{(S)}$, the higher the consistency between $\boldsymbol{\hat{\theta}}_{S}$ and the goal $\boldsymbol{\theta}^{\prime}$. Then we select a generation rule $\boldsymbol{\lambda}^{(S)}$ to maximize the consistency measured by (53):

$$\underset{\ \boldsymbol{\lambda}\in\boldsymbol{\Delta}\phantom{\tilde{1}}}{\textbf{{max}}}\ \underset{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{min}}} \ \ \sum_{(i,j)}\lambda_{i,j}\cdot g_{i,j}(\boldsymbol{\hat{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\hat{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})},$$ (54)

subject to $$\ \ \boldsymbol{\pi}(\boldsymbol{\hat{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}}).$$

We discuss the detailed optimization approach for solving (54) in the following part. With the balance between the exploration and exploitation for the generation procedure controlled by (49) and (54), we provide the asymptotic optimality guarantee of the proposed policy (49) and (54) in the supplementary materials.

4.2 Robust Optimization with Incomplete Knowledge

By the adversarial policy (49) and (54) with complete knowledge, the adversary $\boldsymbol{\mathcal{A}}$ could insert pairwise comparisons to manipulate the rank aggregation results in the sequential way. However, complete knowledge assumption could be not realistic in the actual confrontation scenarios. To dissect the vulnerability as much as possible, we develop a distributionally robust formulation against the uncertainty of knowledge.

Notice that the log-likelihood function $L$ (47) is a scale-free function w.r.t the weights of a comparison graph. The MLE (48) would be invariant when we map $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}$ into a probabilistic simplex and replace the discrete variable $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}$ with a continuous variable $\boldsymbol{p}=[p_{1,2},\dots,p_{n,n-1}]\in\mathbb{R}^{n(n-1)}_{+}$:

$$\boldsymbol{p}=\frac{1}{M}\cdot\boldsymbol{w}_{\boldsymbol{\mathcal{A}}},\ \boldsymbol{p}^{\top}\boldsymbol{1}=1$$ (55)

where $\boldsymbol{1}$ is a n-dimension vector whose elements are $1$ and $M$ is the total number of observed pairwise comparisons by $\boldsymbol{\mathcal{A}}$:

$$M=\sum_{(i,j)}w_{i,j}.$$ (56)

In fact, $\boldsymbol{p}$ is drawn from a distribution $\mathbb{P}$:

$$\mathbb{P}=\frac{1}{n(n-1)}\ \underset{(i,j)}{\sum}\ \delta(p_{i,j}).$$ (57)

where $\delta(p_{i,j})$ is the Dirac measure concentrated at $p_{i,j}$. What is more, we can portray the difference between $\boldsymbol{w}_{\boldsymbol{\mathcal{S}}}{(t)}$ and $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}$ (line $7$ in Algorithm 1) by the distance between distributions. Such treatments introduce an uncertainty set of $\mathbb{P}$ which contains the probability distributions around $\mathbb{P}$:

$$\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})=\left\{\ \mathbb{Q}\ \Big{|}\ \mathcal{W}_{1}(\mathbb{P},\ \mathbb{Q})\leq\gamma\ \right\},$$ (58)

where $\mathcal{W}_{1}(\cdot,\cdot)$ is the $1$-Wasserstein distance [24] as the discrepancy measure. The definition of $1$-Wasserstein distance and related properties can be found in the supplementary materials. With the help of $\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})$, we execute a conservative strategy to estimate the parameter of BTL model with incomplete knowledge. Instead of $\boldsymbol{p}$, we choose the other random variable $\boldsymbol{q}$ as the weight in (47). The distribution of $\boldsymbol{q}$ belongs to $\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})$ and $\boldsymbol{q}$ conducts the worst expected value of $L$. Such a conservative strategy can alleviate the uncertainty generated by incomplete knowledge in the sequential decisions of manipulation policy. Then the relative ranking score with incomplete is estimated by solving the following distributionally robust optimization (DRO) problem:

$$\underset{\ \boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\ \textbf{{max\phantom{p}}}}\ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right],$$ (59)

where $L(\boldsymbol{\theta},\boldsymbol{q})$ replaces the incomplete knowledge $\boldsymbol{w}_{\boldsymbol{\mathcal{A}}}$ with the random variable $\boldsymbol{q}\sim\mathbb{Q}$ in (47). The supreme operation w.r.t. $\mathbb{Q}$ means that the estimation of the latent preference score is based on the worst expected value of $L$ from the set of distributions $\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})$.

Next, we specify the formulation of $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$. Without a lost of generality, we assume the estimated and the desired scores belong to a probability simplex. Given the desired relative ranking score $\boldsymbol{\theta}^{\prime}$, we hope that the estimation from (59) is in a neighborhood of $\boldsymbol{\theta}^{\prime}$, namely, the distance $d:\mathbb{R}^{n}\times\mathbb{R}^{n}\rightarrow\mathbb{R}_{+}$ between the estimation from (59) and $\boldsymbol{\theta}_{\mathcal{A}}$ would be sufficiently small:

$$\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})=\left\{\ \boldsymbol{\theta}\in\mathbb{R}^{n}_{+}\ \Big{|}\ \|\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\|_{2}^{2}\leq\beta,\ \ \boldsymbol{\theta}^{\top}\boldsymbol{1}=1\ \right\}.$$ (60)

Theorem 3.

Suppose that $\boldsymbol{p}$ is drawn from the empirical distribution $\mathbb{P}$ (57) and $\boldsymbol{q}$ is drawn from $\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})$ (58). If the distance between $p_{i,j}$ and $q_{i,j}$ is chosen as

$$d(p_{ij},q_{ij})=\big{|}\ p_{ij}-q_{ij}\ \big{|}.$$ (61)

Then, the DRO problem (59) has an equivalent form:

$$\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ h(\boldsymbol{\theta}),$$ (62)

where

$$h(\boldsymbol{\theta})=\sqrt{\gamma\ }\underset{(i,j)}{\sum}\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta})+\underset{(i,j)}{\sum}p_{i,j}\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}).$$ (63)

Moreover, if the comparison model is BTL model, we have

$$h(\boldsymbol{\theta}) = \ \ \sqrt{\gamma\ }\cdot\underset{(i,j)}{\sum}\textbf{{log}}(1+\textbf{{exp}}(\theta_{j}-\theta_{i}))$$ (64)

$$\phantom{\gamma\underset{(i,j)}{\sum}}+\underset{(i,j)}{\sum}\ p_{i,j}\ \textbf{{log}}(1+\textbf{{exp}}(\theta_{j}-\theta_{i})).$$

From the above theoretical results, we conduct the adversarial policy for the incomplete knowledge. The stop** time (49) turns to be

$$S^{\prime}_{1} = \ \ \textbf{{inf}}\left\{\ S>0\ \ \Bigg{|}\ \ \sum_{(i,j)}e^{-|\Delta^{\prime}_{i,j}L_{S}|}\leq e^{-z_{\alpha}(\chi)}\right\}$$ (65)

$$S^{\prime}_{2} = \ \ \textbf{{inf}}\left\{\ S>0\ \ \Bigg{|}\ \ \ \underset{(i,j)}{\textbf{{min}}}\ \ |\Delta^{\prime}_{i,j}L_{S}|\ \geq\ z_{\alpha}(\chi)\ \right\},$$

where

$$\Delta^{\prime}_{i,j}L_{S} = \ \ \underset{\ \boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\ \textbf{{max\phantom{p}}}}\ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right]$$ (66)

$$\ \ -\underset{\ \boldsymbol{\theta}\in\boldsymbol{\Theta}_{j,i}}{\ \textbf{{max\phantom{p}}}}\ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right]$$

$$= \ \ \underset{\ \boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\ \textbf{{max\phantom{p}}}}h(\boldsymbol{\theta})-\underset{\ \boldsymbol{\theta}\in\boldsymbol{\Theta}_{j,i}}{\ \textbf{{max\phantom{p}}}}h(\boldsymbol{\theta}).$$

Now we discuss the generation rule with the robust estimation by (62). Let $\boldsymbol{\bar{\theta}}_{S}$ be a solution of (62) with stop** time (65). We obtain the generation rule with incomplete knowledge by replacing $\boldsymbol{\hat{\theta}}_{S}$ with $\boldsymbol{\bar{\theta}}_{S}$ in (54):

$$\underset{\ \boldsymbol{\lambda}\in\boldsymbol{\Delta}\phantom{\tilde{1}}}{\textbf{{max}}}\ \underset{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{min}}} \ \ \sum_{(i,j)}\lambda_{i,j}\cdot g_{i,j}(\boldsymbol{\bar{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\bar{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})},$$ (67)

subject to $$\ \ \boldsymbol{\pi}(\boldsymbol{\bar{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}}).$$

Consider the inner problem

$$\ \underset{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{min}}} \ \ \sum_{(i,j)}\lambda_{i,j}\cdot g_{i,j}(\boldsymbol{\bar{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\bar{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})},$$ (68)

subject to $$\ \ \boldsymbol{\pi}(\boldsymbol{\bar{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}}),$$

we know that the objective function is smooth and convex w.r.t $\boldsymbol{\tilde{\theta}}$ for the BTL model $g_{i,j}(\cdot)$. Moreover, the flexible set

$$\left\{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\ \Big{|}\ \boldsymbol{\pi}(\boldsymbol{\bar{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\right\}$$ (69)

could be re-written as a union of convex sets which contain at most $2*n$ linear equalities and inequalities like

$$\left\{\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\ \Big{|}\ \tilde{\theta}_{i}<\bar{\theta}_{\pi_{\bar{\theta}}(i^{\prime}-1)},\tilde{\theta}_{i}=\bar{\theta}_{\pi_{\bar{\theta}}(i^{\prime})}\right\},$$ (70)

where $\boldsymbol{\tilde{\theta}}=[\tilde{\theta}_{1},\dots,\tilde{\theta}_{n}]$, and $\bar{\theta}_{\pi_{\bar{\theta}}(i^{\prime})}$ indicates the preference score of candidates $\pi_{\bar{\theta}}(i)$ whose position in $\boldsymbol{\pi}_{\boldsymbol{\bar{\theta}}}$ is $i^{\prime}$. Therefore, the inner problem is the convex problem which can be solved efficiently by the standard numerical solvers. Then we analyze the outer problem:

$$\underset{\boldsymbol{\lambda}\in\boldsymbol{\Delta}}{\textbf{{min}}}\ F(\boldsymbol{\lambda}),\ \ F(\boldsymbol{\lambda})=\underset{\begin{matrix}\scriptstyle\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\\ \scriptstyle\boldsymbol{\pi}(\boldsymbol{\bar{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\end{matrix}}{\textbf{{max}}}\phi(\boldsymbol{\lambda},\boldsymbol{\tilde{\theta}})$$ (71)

where

$$\phi(\boldsymbol{\lambda},\boldsymbol{\tilde{\theta}})=-\sum_{(i,j)}\lambda_{i,j}\cdot g_{i,j}(\boldsymbol{\bar{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\bar{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})}.$$ (72)

It is noteworthy that $\phi(\boldsymbol{\lambda},\boldsymbol{\tilde{\theta}})$ is a continuous and bounded function. Furthermore, $\phi(\boldsymbol{\lambda},\boldsymbol{\tilde{\theta}})$ is convex w.r.t $\boldsymbol{\lambda}$ for any $\boldsymbol{\tilde{\theta}}$ and $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ is a convex set. By Danskin Theorem [9], $F(\boldsymbol{\lambda})$ is a convex function w.r.t $\boldsymbol{\lambda}$ and the min-max optimization problem (67) can be solved efficiently using the mirror descent algorithm [7]. The corresponding solution process is summarized as Algorithm 2. We elaborate the steps of Algorithm 2 in the supplementary materials.

  Input : the probability mass function $g$, the incomplete knowledge $\boldsymbol{p}$, the uncertainty radius $\gamma$.

1

2Obtain the robust estimation based on the partial observation by solving (62):

$$\boldsymbol{\theta}=\textbf{{RobustEstimation}}(g,\boldsymbol{p},\gamma).$$

3Solve the generation rule $\boldsymbol{\lambda}$ via the min-max problem (67):

$$\boldsymbol{\lambda}=\textbf{{MirrorDescent}}(g,\boldsymbol{\theta}).$$

4Select pairwise comparison $c$ according to the categorical distribution $\boldsymbol{\lambda}$.

Output : a pairwise comparison $c$.

Algorithm 2 Adversarial Generation

5 Experiments

In this section, three examples are exhibited with both simulated and real-world data to illustrate the validity of the proposed online attack strategy against the Bernoulli method for rank aggregation like HodgeRank [29] and RankCentrality [41]. The first example is with simulated data while the latter two exploit real-world datasets involved in election and crowdsourcing.

5.1 General Setting

We treat the online manipulation against the rank aggregation as the interaction between the original and adversarial data source like Algorithm 1. In each turn of this game, the original and adversarial data sources generate the pairwise comparisons separately. The generation process of the original data source is always a black box for the adversary and the action of the original data source (line $3$ of Algorithm 1) will be replaced by a random procedure. Based on the analysis in Sec. 3.3, the sampling method could be $(\epsilon,\delta)-$ representative w.r.t the mixed data source $\boldsymbol{\mathcal{C}^{\prime}}$ when the sampling parameters are sufficiently large. In most cases, the sampling methods for rank aggregation satisfy these conditions. Even if the sampling methods reject the samples generated by the adversary, he/she can still try repeatedly until such samples affect the final aggregation result. As a consequence, we don’t consider the effects of sampling methods (Line $5$ and $11$ of Algorithm 1) in the experimental studies. It is noteworthy that the attacker only has incomplete knowledge in the adversarial game, say that he/she only observes partial weight of the comparison graph (Line $7$ of Algorithm 1). The actions of the attacker is the adversarial generation process with the target ranking list $\boldsymbol{\pi}^{\prime}$, which is discussed in Sec. 4 summarized as Algorithm 2. In each turn, the adversarial generation process will insert $S_{0}$ pairwise comparisons into the mixed data stream, where $S_{0}$ is the stoo** time. We establish the asymptotic optimality of the proposed stop** time $S_{1},S_{2}$ (49) under the complete knowledge condition. With the incomplete knowledge, we set the stop** time $S_{0}$ empirically. At the end of the adversarial game, we finish the collection of pairwise comparisons and obtain the weighted comparison graph $\boldsymbol{\mathcal{G}}^{(T)}$ (the output of Algorithm 1). Then the rank aggregation methods leverage $\boldsymbol{\mathcal{G}}^{(T)}$ to create the ranking list $\boldsymbol{\pi}^{\prime\prime}$. We evaluate the similarity between $\boldsymbol{\pi}^{\prime}$ and $\boldsymbol{\pi}^{\prime\prime}$. The more the two orders are similar, the more the manipulation is successful.

(a) R. Rank of HodgeRank

(b) K. $\tau$ of HodgeRank

(c) R. Rank of RankCentrality

(d) K. $\tau$ of RankCentrality

Figure 2: Comparative results of different sequential manipulation methods against HodgeRank and RankCentrality on simulated data. The box plot illustrates the results of $50$ trials with different data sequences which will make HodgeRank and RankCentrality generate $\boldsymbol{\pi}_{0}=(10,9,8,7,6,5,4,3,2,1)$. The target list of the adversary is $\boldsymbol{\pi}^{\prime}=(8,9,10,7,5,6,4,3,2,1)$. The proposed method provides a stable manipulation in the form of sequential action. All metrics of the proposed method will be $1$ with rare outliers. Meanwhile the three competitors fail to manipulate HodgeRank and RankCentrality with sequential actions. The ‘Greedy’ perturbation only focuses on the top-$1$ candidate but can’t guarantee the designation of a winner. The result of ‘Straightforward’ strategy is inferior to the proposed method when the number of actions is the same.

(a) R. Rank of HodgeRank

(b) K. $\tau$ of HodgeRank

(c) R. Rank of RankCentrality

(d) K. $\tau$ of RankCentrality

Figure 3: Change of evaluation metrics on simulated data for different sequential manipulation methods against HodgeRank and RankCentrality. The horizontal axis lists the turns of game. When the interaction proceeds, the proposed method is able to generate malicious pairwise comparisons with incomplete knowledge and manipulate the victim, whose aggregated results are consistent with the attacker’s target.

5.2 Evaluation Metrics

Here we adopt Reciprocal rank and Kendall ${\tau}$ coefficient for evaluating the correlation between the target ranking list $\boldsymbol{\pi}^{\prime}$ and the final aggregated result $\boldsymbol{\pi}^{\prime\prime}$. These measurements can be divided into two categories. The reciprocal rank metric reflects whether the first candidates of two ranking lists are the same. The Kendall $\tau$ coefficient considers the consistency of every pairwise comparison in the two ranking lists.

Reciprocal Rank (R. Rank). The reciprocal rank is a statistic measure for evaluating any process that produces an order list of possible responses to a series of queries, ordered by the probability of correctness or the ranking scores. The reciprocal rank of a ranking list is the multiplicative inverse of the leading object’s position in the new order list. The R. Rank between $\boldsymbol{\pi}^{\prime}$ and $\boldsymbol{\pi}^{\prime\prime}$ is defined as

$$\textbf{{{R. Rank}}}(\boldsymbol{\pi}^{\prime},\boldsymbol{\pi}^{\prime\prime})=\frac{1}{\ (\boldsymbol{\pi}^{\prime\prime})^{-1}\big{[}\boldsymbol{\pi}^{\prime}(1)\big{]}\ },$$ (73)

where $\boldsymbol{\pi}(i)$ refers to the item index which lies in the $i$-th position of ranking list $\boldsymbol{\pi}$, and $\boldsymbol{\pi}^{-1}[i]$ indicates the position of ranking list $\boldsymbol{\pi}$ which belongs to the item $i$. If it holds that $\boldsymbol{\pi}^{\prime}(1)=\boldsymbol{\pi}^{\prime\prime}(1)$, we have $\textbf{{R.r}}(\boldsymbol{\pi}^{\prime},\boldsymbol{\pi}^{\prime\prime})$ archives its maximum value $1$. Lager reciprocal rank value indicates a better manipulation result.

Kendall $\boldsymbol{\tau}$ Coefficient (K. $\boldsymbol{\tau}$). The Kendall rank correlation coefficient evaluates the degree of similarity between two ranking lists given the same objects. This coefficient depends upon the number of inversions of pairs of objects which would be needed to transform one rank order into the other. The definition of Kendall $\tau$ coefficient is the normalization of (43). Lager Kendall-$\tau$ value indicates a better purposeful attack result. If $d_{\tau}(\boldsymbol{\pi}^{\prime},\ \boldsymbol{\pi}^{\prime\prime})=1$, we have $\boldsymbol{\pi}^{\prime}=\boldsymbol{\pi}^{\prime\prime}$.

(a) HodgeRank

(b) RankCentrality

Figure 4: The victims’ aggregation results of different manipulation methods on simulated data. The original ranking list is $\boldsymbol{\pi}_{0}=[10,9,8,7,6,5,4,3,2,1]$ and the target one is $\boldsymbol{\pi}^{\prime}=[8,9,10,7,5,6,4,3,2,1]$ (refer to Target in the figure). The dark dots represent the candidates with large ID and vice versa. If a candidate is not in the same position in both ranking lists, there will exist an intersection and the width of the line represent the degree of inconsistent influence on the aggregation results. We mark the inconsistent dots with red circles. When there exist multiple intersections and the key locations (top-$3$) are marked by red circles, the attacker has failed to achieve his/her goal. (a) The victim is HodgeRank. The proposed method accomplishes the manipulation of the complete ranking list (no intersection or red circle). (b) The victim is RankCentrality.

5.3 Competitors

To the best of our knowledge, the proposed method is the first overture to online manipulation strategies against the pairwise ranking algorithms. We compare the proposed methods with the following three competitors: the random strategy (referred to as ‘Random’), the greedy strategy (referred to as ‘Greedy’) and the straightforward strategy (referred to as ‘Straight’).

Random perturbation involves a random data source which generates any pairwise comparisons with the same probability. This method does not rely on any information of the desired ranking $\boldsymbol{\pi}^{\prime}$. We conjecture that the purposeless behavior of the random perturbation would not sculpt the desired results out of the mixed data stream $\boldsymbol{C}^{\prime}$. However, this strategy is still the evidence to prove the necessity of sophisticated attacker in the online manipulation against the rank aggregation methods.

Greedy manipulation generates the mixed data stream $\boldsymbol{C}^{\prime}$ in a greedy way with the help of the target ranking list $\boldsymbol{\pi}^{\prime}$. Specifically, this method only insert $\boldsymbol{\pi}^{\prime}(1)\succ\boldsymbol{\pi}^{\prime}(j),j=2,\dots,n$ with the same probability. The greedy manipulation could designate the leading position of the aggregated list. Compared with the proposed method, the greedy method lack the manipulation ability of full ranking list.

Straightforward strategy implements the adversarial data source with the so-called “Matthew Principle”: increasing the number of the pairwise comparisons which are consistent with the desired ranking list. There exist $n(n-1)/2$ pairwise comparisons which are consistent with $\boldsymbol{\pi}^{\prime}$ and they have the equal chance to insert into the mixed data stream $\boldsymbol{C}^{\prime}$. Obviously, this strategy could archive the goal with sufficient turns in the adversarial game between two data source. Compared with the proposed method, the straightforward method will waste some opportunities and need more actions to archive the goal.

5.4 Simulated Study

Description. We validate the proposed sequential manipulation strategies against HodgeRank and RankCentrality on simulated data. We generate the data stream as follows. First, we build a complete graph $\boldsymbol{\mathcal{G}}=(\boldsymbol{V},\boldsymbol{E})$ where $\boldsymbol{V}=[n]$. Then the latent preference score is assigned to each candidate/vertex of $\boldsymbol{V}$ and the true ranking is obtained by these scores. Setting $n=10$ and the true ranking is $\boldsymbol{\pi}_{0}=(10,9,8,7,6,5,4,3,2,1)$. Next, we randomly sample $744$ pairwise comparisons from $\boldsymbol{V}\times\boldsymbol{V}$ based on their preference score. Notice that the samples could contain the comparisons which are inconsistent with the true ranking. We regard these samples from the original data source (line 3 of Algorithm 1) and construct $50$ sequences with different orders. Each sequence will be a trail for the adversary. The goal of adversary is to make HodgeRank and RankCentrality produce $\boldsymbol{\pi}^{\prime}=(8,9,10,7,5,6,4,3,2,1)$. The number of turns in the adversarial game ($T=75$ in Algorithm 1) is $10$ percent of the length of the complete sequence. In each turn, the sample from the original data source has an $80\%$ chance of being observed by the adversary (line $7$ of Algorithm 1). If his/her knowledge is not updated, the attacker will not take any action and wait for another sample from the original data source. Moreover, the attackers could insert $S_{0}=5$ pairwise comparisons to construct the comparison graph in each turn (line $8$ of Algorithm 1).

(a) Random v.s. HodgeRank

(b) Straight v.s. HodgeRank

(c) Greedy v.s. HodgeRank

(d) Ours v.s. HodgeRank

(e) Random v.s. RankCentrality

(f) Straight v.s. RankCentrality

(g) Greedy v.s. RankCentrality

(h) Our v.s. RankCentrality

Figure 5: Data distribution generated by different methods on simulated data. The vertical axis lists the number of rounds in the adversarial games and the horizontal axis displays all possible pairwise comparisons. For the same victim, all results are based on the same observed data. The original ranking list is $\boldsymbol{\pi}_{0}=[10,9,8,7,6,5,4,3,2,1]$ and the target one is $\boldsymbol{\pi}^{\prime}=[8,9,10,7,5,6,4,3,2,1]$.

Comparative Results. The results of each method against HodgeRank on simulated data are reported in Fig. 2 (a)-(b). Our method exhibits higher success rate due to the parsimonious mechanism and consistently outperforms all the competitors by a significant margin. Concerning the performances of the three competitors, we can easily find that:

• Due to the existence of non-modifiable data, the blind attack strategies cannot even interfere with the aggregation results of victim with limited actions. The random perturbation (‘Random’) can’t boost the position of candidate $8$ in all aggregated results. Although the mean of Kendall $\tau$ coefficient with $50$ trials is $0.77$, the degree of consistency between Random and $\boldsymbol{\pi}_{0}$ remains higher than that between Random and $\boldsymbol{\pi}^{\prime}$. These arguments could be justified by the visualization of ranking lists in Figure 4.

• The greedy manipulation (‘Greedy’) can have an impact on the winner of the final ranking lists. However, this method failed to consistently manipulate the aggregation results over a specified number of actions. The interquartile range of reciprocal rank is a large interval and the median is $0.5$ when the adversary executes Greedy against HodgeRank. As all the insertions of Greedy are consistent with the target list, the Kendall $\tau$ coefficient of Greedy is higher than that of Random. This phenomenon does not imply that Greedy had complete control over the victim’s result as its generation mechanism only guarantees the desired winner could beat the other candidates.

• In principle, the straightforward strategy (‘Straight’) has potential to manipulate the complete aggregation results of victim. The efficiency of the method is a concern as it ignores the existing data. We suspect that the method will only work under the conditions of the so-called “flooding attack”, where $T$ and $S_{0}$ will be sufficiently large.

We report the results of each method against RankCentrality on simulated data in Fig. 2 (c)-(d). The best performance and the median of our method consistently surpass all the competitors. The Kendall $\tau$ coefficient of the proposed method is not $1$. We speculate that this phenomenon comes from the challenge posed by controlling the spectral structure of comparison graph. However, the proposed method is the only one which is able to designate the winner of the aggregation results. Furthermore, we show the specific behavior of the proposed method in every turn of the adversarial game in Fig. 3. Despite the existence of unknown data, all metrics of the proposed method grow and eventually remain stable. This phenomenon implies that the proposed method could obtain an equilibrium state which favors to the adversary even if the details of the victims are not involved. The ranking lists of different methods are shown in Fig. 4. We illustrate every pair of the manipulation result and the target ranking list. All results are based on the same observed data sequence. The proposed method could designate the winner of the aggregation result ($8$ is the top-$1$ candidate in our results) and keep a high correlation with the target ranking list (there only exist two intersections in our result).

TABLE I: Numeric results of different attack methods on crowdsourcing data. The best results are highlighted with bold text.

Methods	HodgeRank						RankCentrality
	$(20,29,8,13)$		$(8,29,20,13)$		$(13,29,20,8)$		$((20,29,8,13)$		$(8,29,20,13)$		$(13,29,20,8)$
	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$
Random	0.50	0.37	0.50	0.35	0.20	0.25	0.33	0.39	0.33	0.33	0.25	0.14
Straight	0.50	0.64	0.50	0.59	0.25	0.52	0.50	0.44	0.33	0.35	0.25	0.23
Greedy	1.00	0.52	1.00	0.49	0.50	0.38	1.00	0.55	0.33	0.55	0.25	0.24
Ours	1.00	0.55	1.00	0.54	1.00	0.44	1.00	0.67	1.00	0.55	1.00	0.36

TABLE II: Numeric results of different attack methods on Dublin election data. The best results are highlighted with bold text.

Methods	HodgeRank						RankCentrality
	$(2,4,13,5)$		$(13,4,2,5)$		$(5,4,2,13)$		$(2,4,13,5)$		$(13,4,2,5)$		$(5,4,2,13)$
	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$	R. Rank	K. $\tau$
Random	0.50	0.93	0.50	0.91	0.25	0.58	0.50	0.93	0.33	0.91	0.25	0.58
Straight	0.50	0.97	0.50	0.93	0.25	0.58	0.50	0.93	0.33	0.91	0.25	0.58
Greedy	1.00	0.98	1.00	0.96	0.50	0.87	1.00	0.93	0.50	0.91	0.33	0.49
Ours	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00

The data distributions generated by different methods are illustrated in Fig. 5. With the help of the data distribution, we can better understand the reasons why the proposed method can achieve sequential manipulation with the interference of the original data source. The victim of the first row is HodgeRank and the second one is RankCentrality. Here the horizontal axis lists all possible pairwise comparisons. The vertical axis list $5$ representative turns in the adversarial games. We index them as follows: No.$(i-1)*9+1$ to $i*9$ are the comparisons $\{(i,j)\ |\ j\in[10],j\neq i\}$. The proposed method makes efficient manipulation with specific purposes. We observe that the bins which represent the desired winner defeating the other candidates are higher after the attack procedure, especially No. $72$ ($8\succ 10$) and No. $71$ ($8\succ 9$). Such behavior ensures that the aggregation results of the victims are resistant to the original data source. The ‘Greedy’ method also increases the number of No. $64$ to No. $72$. However, it is not sufficient to guarantee that ‘Greedy’ could manipulate the victims’ ranking lists. The ‘Straight’ method disperses its power and fails to promote the position of candidate $8$. The ‘Random’ method uniformly generates all kinds of pairwise comparisons but it does not help to achieve the goal.

5.5 Crowdsourcing

Description. $30$ images from the human age dataset FGNET are annotated by a group of volunteer users on a crowdsourcing platform. The ground-truth age ranking is known to us. The annotator is presented with two images and given a binary choice of which one is older. Totally, we obtain $8,017$ pairwise comparisons from $94$ annotators. The top-$4$ candidates of the true ranking is $(29,20,8,13)$. The goals of adversary are to make HodgeRank and RankCentrality produce $(20,29,8,13)$, $(8,29,20,13)$ and $(13,29,20,8)$ as the top-$4$ candidates. The rest part of the whole ranking list remains unchanged. The number of turns in the adversarial game is $5\%$ of the length of the complete sequence. In each turn, the sample from the original data source has a $90\%$ chance of being observed by the adversary. If his/her knowledge is not updated, the attacker will not take any action and wait for another sample from the original data source. Moreover, the attackers could insert $S_{0}=10$ pairwise comparisons to construct the comparison graph in each turn.

Comparative Results. It is worth mentioning that this real-world data has a high percentage of outliers (about $20\%$ of all comparisons conflict with the correct age ranking). The proposed methods against HodgeRank and RankCentrality still show promise manipulation as Table I. It is more challenging to change $(29,20,8,13)$ to $(8,29,20,13)$ than to $(20,29,8,13)$. Consequently, the values of Kendall $\tau$ coefficient will decrease when the difficulty of the manipulation increases.

5.6 Election

Description. The Dublin election data set contains a complete record of votes for elections held in county Meath, Dublin, Ireland in 2002. This set contains $64,081$ votes over $14$ candidates. These votes could be a complete or partial list of the candidate set. The ground-truth ranking of $14$ candidates is based on their obtained first preference votes. The five candidates who receive the most first preference votes will be the winner of the election. The top-$4$ of is $\boldsymbol{\pi}_{0}=(4,2,13,5)$. Then these votes are converted into the pairwise comparisons. The total number of the comparisons is $652,817$. The goals of adversary are to make HodgeRank and RankCentrality produce $(2,4,13,5)$, $(13,4,2,5)$ and $(5,4,2,13)$ as the top-$4$ candidates. The number of turns in the adversarial game is $1\%$ of the length of the complete sequence. In each turn, the sample from the original data source has an $80\%$ chance of being observed by the adversary. If his/her knowledge is not updated, the attacker will not take any action and wait for another sample from the original data source. Moreover the attackers could insert $S_{0}=5$ pairwise comparisons to construct the comparison graph in each turn.

Comparative Results. It is worth noting that the election result is not obtained by pairwise ranking aggregation. However, the ordered list aggregated from induced comparisons still shows a positive correlation with the actual election result. Once the attackers generate a successful manipulation strategy against the ballots collection process, this attack plan could be adopted to manipulate the election in the real world. Consequently, the proposed sequential strategy is still able to manipulate the election results. The aggregation results of HodgeRank and RankCentrality are still manipulated by the proposed method, see Table II.

6 Conclusion

In this paper, we establish the first study of sequential manipulation in the context of ranking aggregation with pairwise comparisons to the best of our knowledge. We find that the data collection process is the Achilles’ heel of the rank aggregation. The sequential attack problem is formulated as a distributionally robust game between two players, the online manipulator and the ranker who possesses the original data ‘source’. Furthermore, we introduce the sampling algorithms to analyze the properties of the underlying distributionally Nash equilibrium. Like the two sides of a coin, we prove that the representation ability of sampling methods could turn into the vulnerability when the mixed data source supports the goal of an adversary. With the help of Bayesian decision theory, we develop the manipulation policy with complete knowledge, which achieves the asymptotic optimality. Then a distributionally robust generation rule is proposed to resist the uncertainty of the observed sequence. Our empirical studies show that the proposed sequential manipulation methods could achieve the attacker’s goal in the sense that the leading candidate of the aggregated ranking list is the designated one by the adversary.

Ke Ma is an associate professor with the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences (UCAS), Bei**g, China. He received the B.S. degree in mathematics from Tian** University in 2009, M.E. degree in software engineering from Beihang University (BUAA) in 2013, and the Ph.D. degree in computer science from the Key Laboratory of Information Security (SKLOIS), Institute of Information Engineering (IIE), Chinese Academy of Sciences (CAS), in 2019. His research interests include rank aggregation and algorithmic game theory.

Qianqian Xu received the B.S. degree in computer science from China University of Mining and Technology in 2007 and the Ph.D. degree in computer science from University of Chinese Academy of Sciences in 2013. She is currently a Professor with the Institute of Computing Technology, Chinese Academy of Sciences, Bei**g, China. Her research interests include statistical machine learning, with applications in multimedia and computer vision. She has authored or coauthored 70+ academic papers in prestigious international journals and conferences (including T-PAMI, IJCV, T-IP, NeurIPS, ICML, CVPR, AAAI, etc). Moreover, she serves as an associate editor of IEEE Transactions on Circuits and Systems for Video Technology, IEEE Transactions on Multimedia, and ACM Transactions on Multimedia Computing, Communications, and Applications.

**shan Zeng received the Ph.D. degree in applied mathematics from Xi’an Jiaotong University, Xi’an, China, in 2015. He is currently a professor with the School of Computer and Information Engineering, Jiangxi Normal University, Nanchang, China. His research interests include non-convex optimization and machine learning.

Wei Liu is currently a Distinguished Scientist of Tencent and the Director of Ads Multimedia AI at Tencent Data Platform. Prior to that, he has been a research staff member of IBM T. J. Watson Research Center, USA. Dr. Liu has long been devoted to fundamental research and technology development in core fields of AI, including deep learning, machine learning, computer vision, pattern recognition, information retrieval, big data, etc. To date, he has published extensively in these fields with more than 270 peer-reviewed technical papers, and also issued 23 US patents. He currently serves on the editorial boards of IEEE TPAMI, TNNLS, IEEE Intelligent Systems, and Transactions on Machine Learning Research. He is an Area Chair of top-tier computer science and AI conferences, e.g., NeurIPS, ICML, IEEE CVPR, IEEE ICCV, IJCAI, and AAAI. Dr. Liu is a Fellow of the IEEE, IAPR, AAIA, IMA, RSA, and BCS, and an Elected Member of the ISI.

Xiaochun Cao is a Professor of School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University. He received the B.E. and M.E. degrees both in computer science from Beihang University (BUAA), China, and the Ph.D. degree in computer science from the University of Central Florida, USA, with his dissertation nominated for the university level Outstanding Dissertation Award. After graduation, he spent about three years at ObjectVideo Inc. as a Research Scientist. From 2008 to 2012, he was a professor at Tian** University. Before joining SYSU, he was a professor at Institute of Information Engineering, Chinese Academy of Sciences. He has authored and coauthored over 200 journal and conference papers. In 2004 and 2010, he was the recipients of the Piero Zamperoni best student paper award at the International Conference on Pattern Recognition. He is on the editorial boards of IEEE Trans. on Image Processing and IEEE Trans. on Multimedia, and was on the editorial board of IEEE Trans. on Circuits and Systems for Video Technology.

Yingfei Sun received the Ph.D. degree in applied mathematics from the Bei**g Institute of Technology, in 1999. He is currently a Full Professor with the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences. His current research interests include machine learning and pattern recognition.

Qingming Huang is a chair professor in the University of Chinese Academy of Sciences and an adjunct research professor in the Institute of Computing Technology, Chinese Academy of Sciences. He graduated with a Bachelor degree in Computer Science in 1988 and Ph.D. degree in Computer Engineering in 1994, both from Harbin Institute of Technology, China. His research areas include multimedia computing, image processing, computer vision and pattern recognition. He has authored or coauthored more than 400 academic papers in prestigious international journals and top-level international conferences. He was the associate editor of IEEE Trans. on CSVT and Acta Automatica Sinica, and the reviewer of various international journals including IEEE Trans. on PAMI, IEEE Trans. on Image Processing, IEEE Trans. on Multimedia, etc. He is a Fellow of IEEE and has served as general chair, program chair, track chair and TPC member for various conferences, including ACM Multimedia, CVPR, ICCV, ICME, ICMR, PCM, BigMM, PSIVT, etc.

[sections] [sections]l1

The first part is about two ranking algorithms tailored to the BTL model, say Hodgerank [29] and the spectral ranking algorithm [41].

The second part is the proof details of Theorem 1, which states the existence of a distributionally robust Nash equilibrium. This result tells us that there exists at least one stable state for both ranker and attacker. To prove the existence of the distributionally robust Nash equilibrium, we need a proposition from [35], which shows that the distributionally robust Nash equilibrium is a global minimizer of the reformulation of (22). This reformulation is also well known for the deterministic Nash equilibrium problem [44]. Based on this proposition, we show the existence results of DRNE for the adversarial game. This result is an extension of the famous Kakutani’s fixed point theorem [30].

The third part is the proof details of Theorem 2. We need some important lemmas. Lemma 1 is a martingale concentration inequality which can deal with the case that the maximum value $M$ of $|x_{t+1}-x_{t}|$ is large, but the maximum is rarely attained (making the variance much smaller than $M^{2}$) [23, 39, 19]. The following two lemmas assert that for any given subset $\boldsymbol{S}_{\mathcal{A}}$ of the universe $\boldsymbol{\mathcal{C}}$, the fraction of elements from $\boldsymbol{S}_{\mathcal{A}}$ within the sample typically does not differ by much from the corresponding fraction among the whole stream. Lemma 2 corresponds to the Bernoulli sampling. Lemma 3 corresponds to the reservoir sampling. Theorem 2 shows the vulnerability of the Bernoulli and reservoir sampling methods. When the sampling parameters of these two methods fulfill the conditions, the data for the ranker will be $(\epsilon,\delta)$-representative with respect to the sequence fabricated by the adversary. The adversary are able to obtain the desired ranking list with these pairwise comparisons.

The forth part proves the asymptotic optimality of the proposed adversarial policy (49) and (54) with complete knowledge. First we show some important assumptions. Considering Assumption 1-5, Theorem 4 establishes a lower bound on the minimal Bayesian risk with the help of Lemma 4-7. Theorem 5 provides the asymptotic upper bounds for the expected Kendall tau of the proposed manipulation policy with complete information with the help of Lemma 8 and 9. Theorem 6 shows the asymptotic optimality of the expected stop** time of the proposed manipulation policy with the help of Lemma 10 and 11. These three theorems are sufficient to show the asymptotic optimality of the proposed stop** time (49) and the generation rule solved by (54) with complete knowledge as the identifiability of BTL model by adopting the MLE to obtain the preference score.

The fifth part gives us the solution of (59), which gives birth to the stop** time (65) and generation rule (67) for adversary with incomplete knowledge. Proposition 2 shows that the strong duality ensures that the inner supremum of (59) admits a reformulation which is a simple, univariate optimization problem. Theorem 3 gives the equivalent form of (59), which can be solved efficiently using the mirror descent algorithm. The detailed solving process is Algorithm 3, 4 and 5.

HodgeRank and RankCentrality

HodgeRank

The Hodgerank method discussed in [29] consists in finding the relative ranking score by solving the following least-squares problem:

$$\underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{minimize}}}\ \ \frac{\ 1\ }{\ 2\ }\underset{(i,j)\in\boldsymbol{E}}{\sum}\ w^{*}_{ij}\big{(}y_{ij}-\theta_{j}+\theta_{i}\big{)}^{2}$$ (74)

where $\boldsymbol{y}=[y_{12},y_{13},\dots,y_{n,n-1}]^{\top}$ represents the directions of edges. As $\boldsymbol{\mathcal{G}}$ is a complete graph, we set $y_{ij}=1$ which indicates a direct edge from node $i$ to $j$. Based on combinational Hodge theory [29], the minimal norm solution of (74) is simply given as

$$\boldsymbol{\bar{\theta}}=-\mathcal{L}^{\dagger}_{0}\cdot\textbf{div}(\boldsymbol{y}),$$ (75)

where $\mathcal{L}^{\dagger}_{0}$ is the Moore-Penrose pseudo-inverse of $\mathcal{L}_{0}$, and the divergence operator div is defined as

$$[\textbf{div}(\boldsymbol{y})](i)=\underset{j:(i,j)\in\boldsymbol{E}}{\sum}\ w^{*}_{ij}y_{ij},\ \forall\ i\in[n].$$ (76)

Rank Centrality

The spectral ranking algorithm, or RankCentrality [41], is motivated by the connection between the pairwise comparisons $\boldsymbol{w}^{*}$ and a random walk over a directed graph $\boldsymbol{\mathcal{G}}$. The spectral method constructs a random walk on $\boldsymbol{\mathcal{G}}$ where at each time, the random walk is likely to go from vertex $i$ to vertex $j$ if items $i$ and $j$ were ever compared; and if so, the likelihood of going from $i$ to $j$ depends on how often $i$ lost to $j$. That is, the random walk is more likely to move to a neighbor who is more probable to “wins”. How frequently this walk visits a particular node in the long run, or equivalently the stationary distribution, is the score of the corresponding item. Thus, effectively this algorithm captures the preference of the given item versus all the others, not just immediate neighbors: the global effect induced by the transitivity of comparisons is captured through the stationary distribution.

A random walk can be represented by a time-independent transition matrix $\boldsymbol{P}=\{P_{i,j}\}_{1\leq i,j\leq n}\in\mathbb{R}^{n\times n}_{+}$, where $P_{i,j}=\mathbb{P}(X_{t+1}=j|X_{t}=i)$ and $X_{t}$ represents the state of the process (arriving node) at time $t$. By definition, the entries of a transition matrix are nonnegative and satisfy

$$P_{i,j}+P_{j,i}=1,\ \ \forall\ i,\ j\in[n],\ i\neq j.$$ (77)

One way to define a valid transition matrix $\boldsymbol{P}$ of a random walk on $\boldsymbol{\mathcal{G}}$ is to scale all the edge weights by the maximum out-degree of a node, noted as $d_{\text{max}}$. This re-scaling ensures that each row-sum is at most one. Finally, to ensure that each row-sum is exactly one, the spectral method adds a self-loop to each node of $\boldsymbol{V}$. Concretely, the transition matrix $\boldsymbol{P}^{*}$ is converted from the pairwise comparison data $\boldsymbol{w}^{*}$ in such a way that

$$P^{*}_{i,j}=\begin{dcases}\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ 0,&\text{if}\ i\neq j,\ w^{*}_{ij}+w^{*}_{ji}=0,\\[3.5pt] \ \ \ \ \ \ \ \ \ \ \ \ \frac{1}{d_{\text{max}}}\ \frac{w^{*}_{ij}}{w^{*}_{ji}+w^{*}_{ij}},&\text{if}\ i\neq j,\ w^{*}_{ij}+w^{*}_{ji}\neq 0,\\[2.5pt] 1-\frac{1}{d_{\text{max}}}\underset{k\neq i}{\sum}\ \frac{w^{*}_{ik}}{w^{*}_{ik}+w^{*}_{ki}},&\text{otherwise}.\end{dcases}$$ (78)

Rank centrality estimates the probability distribution obtained by applying matrix $\boldsymbol{P}^{*}$ repeatedly starting from any initial condition. Precisely, let $\theta_{t}(i)=\mathbb{P}(X_{t}=i)$ denote the distribution of the random walk at time $t$ with $\boldsymbol{\theta}_{0}=\{\theta_{0}(i)\}\in\mathbb{R}^{n}_{+}$ as an arbitrary starting distribution on $[n]$. Then the random walk holds

$$\boldsymbol{\theta}^{\top}_{t+1}=\boldsymbol{\theta}^{\top}_{t}\boldsymbol{P}^{*}.$$ (79)

wwwwww One expects the stationary distribution of the sample version $\boldsymbol{P}^{*}$ to form a good estimate of true relative ranking score, provided the sample size is sufficiently large. When the transition matrix has a unique left eigenvector $\boldsymbol{\theta}^{*}$ related to the largest eigenvalue, then starting from any initial distribution $\boldsymbol{\theta}_{0}$, the limiting distribution $\boldsymbol{\theta}_{t+1}$ is unique. This stationary distribution $\underset{t\rightarrow\infty}{\lim}\boldsymbol{\theta}_{t}$ is the top left eigenvector of $\boldsymbol{P}^{*}$ as

$$\underset{t\rightarrow\infty}{\lim}\boldsymbol{\theta}_{t}=\boldsymbol{\bar{\theta}}\ \ \text{and}\ \ \boldsymbol{\bar{\theta}}^{\top}=\boldsymbol{\bar{\theta}}^{\top}\boldsymbol{P}^{*},$$ (80)

which only involves a simple eigenvector computation.

Proof of Theorem 1

We clarify the definition of weakly compactness of $\boldsymbol{\mathcal{P}}$.

Definition 6.

A set of probability distribution (measures) $\boldsymbol{\mathcal{P}}$ is said to be weakly compact if every sequence $\{\mathbb{P}_{N}\}\subset\boldsymbol{\mathcal{P}}$ contains a sub-sequence $\{\mathbb{P}_{N_{0}}\}$ and a point $\mathbb{P}_{0}$ such that $\{\mathbb{P}_{N_{0}}\}$ converges to $\mathbb{P}_{0}$ weakly.

To prove the existence of the distributionally robust Nash equilibrium, we need the following proposition [35], which shows that the distributionally robust Nash equilibrium is a global minimizer of the reformulation of (22). This reformulation is also well known for the deterministic Nash equilibrium problem [44].

Proposition 1.

Let

$$\phi(\boldsymbol{A}^{\prime},\boldsymbol{A})=\sum_{r=1}^{R}\underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{max}}\phantom{g}}\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}^{\prime}_{r},\boldsymbol{a}_{-r},\boldsymbol{w})\Big{]},$$ (81)

where $\boldsymbol{A}^{\prime}=[\boldsymbol{a}^{\prime}_{1},\dots,\boldsymbol{a}^{\prime}_{R}]$. Under the conditions of Theorem 1, $\boldsymbol{A}^{*}$ is a distributionally robust Nash equilibrium as (25) if and only if

$$\boldsymbol{A}^{*}\in\underset{\boldsymbol{A}^{\prime}}{\textbf{{arg min}}}\ \phi(\boldsymbol{A}^{\prime},\boldsymbol{A}^{*}).$$ (82)

Proof.

1. The ‘if’ part. If $\boldsymbol{A}^{*}$ is not an equilibrium state, there exists at least one $\boldsymbol{a}^{\prime}_{r_{0}}$ which satisfies

   $$\ \ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r_{0}}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r_{0}}(\boldsymbol{a}^{\prime}_{r_{0}},\boldsymbol{a}^{*}_{-{r_{0}}},\boldsymbol{w})\Big{]}$$ (83)

   $$< \ \ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r_{0}}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r_{0}}(\boldsymbol{a}^{*}_{r_{0}},\boldsymbol{a}^{*}_{-{r_{0}}},\boldsymbol{w})\Big{]}.$$

   Let $\boldsymbol{A}^{\prime\prime}$ be

   $$\boldsymbol{A}^{\prime\prime}=[\boldsymbol{a}^{*}_{1},\dots,\boldsymbol{a}^{*}_{r_{0}-1},\boldsymbol{a}^{\prime}_{r_{0}},\boldsymbol{a}^{*}_{r_{0}+1},\dots,\boldsymbol{a}^{*}_{R}]$$ (84)

   and we could conduct the following contradiction by (83):

   $$\phi(\boldsymbol{A}^{\prime\prime},\boldsymbol{A}^{*})<\phi(\boldsymbol{A}^{*},\boldsymbol{A}^{*}).$$ (85)

2. The ‘only if’ part. By the definition of distributionally robust Nash equilibrium, for any $\boldsymbol{A}^{\prime}$, it holds that

   $$\ \ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}^{\prime}_{r},\boldsymbol{a}^{*}_{-{r}},\boldsymbol{w})\Big{]}$$ (86)

   $$\geq \ \ \underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}^{*}_{r},\boldsymbol{a}^{*}_{-{r}},\boldsymbol{w})\Big{]},\ \ \forall\ r\in[R].$$

   Summing the two sides of the above inequality, we have

   $$\phi(\boldsymbol{A}^{\prime},\boldsymbol{A}^{*}) = \ \ \sum_{r=1}^{R}\underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}^{\prime}_{r},\boldsymbol{a}^{*}_{-{r}},\boldsymbol{w})\Big{]}$$ (87)

   $$\geq \ \ \sum_{r=1}^{R}\underset{\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r}}{\ \textbf{{max}}\phantom{g}\ }\ \mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}\Big{[}f_{r}(\boldsymbol{a}^{*}_{r},\boldsymbol{a}^{*}_{-{r}},\boldsymbol{w})\Big{]}$$

   $$= \ \ \phi(\boldsymbol{A}^{*},\boldsymbol{A}^{*})$$

   which implies $\boldsymbol{A}^{*}$ is a global optimal solution as (82).

∎

Based on the Proposition 1, we show the existence results of DRNE for the adversarial game. This results is an extension of the famous Kakutani’s fixed point theorem [30].

See 1

Proof.

We know that the ‘supremum’ operator will preserve the convexity. Moreover, with the weakly compactness of $\boldsymbol{\mathcal{P}}_{r}$, the ‘supremum’ operator also preserve the continuity. As a consequence, for any given $\boldsymbol{a}_{-r}$, $\mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}[f_{r}(\cdot,\boldsymbol{a}_{-r},\boldsymbol{w})]$ is continuous and convex for every $\mathbb{P}\in\boldsymbol{\mathcal{P}}_{r},r=1,\dots,R$. By the definition of $\phi$ (81), for any given $\boldsymbol{A}$, $\phi(\boldsymbol{A}^{\prime},\boldsymbol{A})$ is continuous and convex w.r.t. any $\boldsymbol{A}^{\prime}$. Besides, the existence of an optimal solution to

$$\underset{\boldsymbol{A}^{\prime}}{\textbf{{min}}}\ \phi(\boldsymbol{A}^{\prime},\boldsymbol{A})$$ (88)

with given $\boldsymbol{a}$ is guaranteed by assuming that $\mathbb{E}_{\boldsymbol{w}\sim\mathbb{P}}[f_{r}(\cdot,\boldsymbol{a}_{-r},\boldsymbol{w})]$ only has finite values.

The remaining part is to show the existence of $\boldsymbol{A}^{*}$ which satisfies

$$\boldsymbol{A}^{*}\in\underset{\boldsymbol{A}^{\prime}}{\textbf{{arg min}}}\ \phi(\boldsymbol{A}^{\prime},\boldsymbol{A}^{*}).$$ (89)

Let $\Gamma(\boldsymbol{A})$ be the solution set of $\textbf{{min}}\ \phi(\boldsymbol{A}^{\prime},\boldsymbol{A})$ with given $\boldsymbol{A}$. By the convexity of $\phi(\cdot,\boldsymbol{A})$, $\Gamma(\boldsymbol{A})$ is a convex set. $\Gamma(\boldsymbol{A})$ is also a closed set: for any $\{\boldsymbol{A}_{t}\}\rightarrow\boldsymbol{\bar{A}}$ as $t\rightarrow\infty$ and $\boldsymbol{A}^{\prime}_{t}\in\Gamma(\boldsymbol{A}_{t})$ with $\{\boldsymbol{A}^{\prime}_{t}\}\rightarrow\boldsymbol{\bar{A}}^{\prime}$, it holds that

$$\boldsymbol{\bar{A}}^{\prime}\in\Gamma(\boldsymbol{\bar{A}}).$$ (90)

By [5], $\Gamma(\boldsymbol{A})$ is upper semi-continuous on $\mathbb{R}^{N\times R}$. With the well-known Kakutani’s fixed point theorem [30], there exists $\boldsymbol{A}^{*}$ such that

$$\boldsymbol{A}^{*}\in\underset{\boldsymbol{A}^{\prime}}{\textbf{{arg min}}}\ \phi(\boldsymbol{A}^{\prime},\boldsymbol{A}^{*}).$$ (91)

With Proposition 1, we know $\boldsymbol{A}^{*}$ is a distributionally robust Nash equilibrium. ∎

Proof of Theorem 2

In this section, we prove the main technical result for Bernoulli sampling. First we need the follow well-known results of martingale inequalities.

Definition 7.

A martingale is a sequence $\boldsymbol{X}=(x_{1},\dots,x_{T})$ of random variables with finite means, such that

$$\mathbb{E}\big{[}x_{t+1}|x_{1},\dots,x_{t}\big{]}=x_{t},\ \ \forall\ t\in[T].$$ (92)

The most basic and well-known concentration result of martingale is the Azuma’s (or Hoeffding’s) inequality, which asserts that martingales with bounded differences $|x_{t+1}-x_{t}|$ are well-concentrated around their mean. However, we need the other concentration inequality which can deal with the case that the maximum value $M$ of $|x_{t+1}-x_{t}|$ is large, but the maximum is rarely attained (making the variance much smaller than $M^{2}$) [23, 39, 19]. The martingales that we investigate in this paper depict this behavior.

Lemma 1.

Let $\boldsymbol{X}=(x_{1},\dots,x_{T})$ be a martingale and the variance of $x_{t+1}$ given $x_{1},\dots,x_{t}$ be bounded

$$\textbf{{Var}}(x_{t+1}|x_{1},\dots,x_{t})\leq\sigma^{2}_{t+1},\ \ \forall\ t\in[T],$$ (93)

where $\sigma_{t}\geq 0$. If there exist a constant $M$ such that

$$|x_{t+1}-x_{t}|\leq M,$$ (94)

we have

$$\mathbb{P}\big{(}\boldsymbol{X}-\mathbb{E}[\boldsymbol{X}]\geq\lambda\big{)}\leq\textbf{{exp}}\left(-\frac{\lambda^{2}}{\sum_{t=1}^{T}\sigma^{2}_{t}+\frac{\lambda M}{3}}\right),$$ (95)

where $\mathbb{E}[\boldsymbol{X}]$ is defined as

$$\mathbb{E}[\boldsymbol{X}]=\sum_{t=1}^{T}\mathbb{E}[x_{t}].$$ (96)

Particularly,

$$\mathbb{P}\Big{(}\big{|}\boldsymbol{X}-\mathbb{E}[\boldsymbol{X}]\big{|}\geq\lambda\Big{)}\leq 2\cdot\textbf{{exp}}\left(-\frac{\lambda^{2}}{\sum_{t=1}^{T}\sigma^{2}_{t}+\frac{\lambda M}{3}}\right).$$ (97)

The following lemmas assert that for any given subset $\boldsymbol{S}_{\mathcal{A}}$ of the universe $\boldsymbol{\mathcal{C}}$, the fraction of elements from $\boldsymbol{S}_{\mathcal{A}}$ within the sample typically does not differ by much from the corresponding fraction among the whole stream. Lemma 2 corresponds to the Bernoulli sampling. Lemma 3 corresponds to the reservoir sampling.

Lemma 2.

For any dynamic stream $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ from $\boldsymbol{\mathcal{C}}^{\prime}$, if the parameter of Bernoulli method $\varrho$ holds that

$$\varrho\geq 10\cdot\frac{\textbf{{ln}}(4/\delta)}{\epsilon^{2}T},$$ (98)

we have

$$\mathbb{P}(|d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C})-d_{\boldsymbol{\mathcal{C}}^{\prime}}(\boldsymbol{C}^{\prime})|\geq\epsilon)\leq\delta,$$ (99)

where $\boldsymbol{C}^{\prime}$ is a sequence which is sampled from $\boldsymbol{C}$ by the Bernoulli method.

Proof.

At any given time step $t\in[T]$ along the sampling process, let $\boldsymbol{C}_{t}=(c_{1},\dots,c_{t})$ be the sequence of pairwise comparisons submitted to the Bernoulli method until time $t$, and $\boldsymbol{C}_{t}^{\prime}\subseteq\boldsymbol{C}_{t}$ be a sub-sequence of the sampled pairwise comparisons from $\boldsymbol{C}_{t}$. Note that $\boldsymbol{C}_{T}=\boldsymbol{C}$ and $\boldsymbol{C}^{\prime}_{T}=\boldsymbol{C}^{\prime}$, and hence, to prove the lemma, we need to show that

$$|d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{T})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{T}^{\prime})|\leq\epsilon.$$

Given a $\boldsymbol{\mathcal{C}^{\prime}}$, we define the random variables

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \frac{\ t}{T}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t})=\frac{|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}|}{T},$$ (100)

$$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \frac{|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}^{\prime}_{t}|}{\varrho T},$$

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}),$$

where the intersection between a set $\boldsymbol{\mathcal{C}^{\prime}}$ and a sequence $\boldsymbol{C}_{t}$ is the sub-sequence of $\boldsymbol{C}_{t}$ consisting of all pairwise comparisons (repetitions are allowed) that also belong to $\boldsymbol{\mathcal{C}^{\prime}}$. Next we show that $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{T}({\boldsymbol{\mathcal{C}^{\prime}}}))$ is a martingale. Suppose that the $\boldsymbol{C}^{\prime}_{t-1}$ is fixed and thence the values of $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ are fixed. Now a new pairwise comparison $c_{t}$ is ready to submit. Note that $c_{t}$ may be either the original data or the comparison perturbed by the adversary.

If $c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}$, we have

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),$$ (101)

$$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),$$

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),$$

and it holds that

$$\mathbb{E}\big{[}\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}\ \big{]}=Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$$ (102)

When $c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}$, we have

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+\frac{1}{T},$$ (103)

$$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \begin{cases}\displaystyle B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),&\ \text{if}\ c_{t}\ \text{is not sampled},\\[5.0pt] \displaystyle B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+\frac{1}{\varrho T},&\text{otherwise.}\end{cases}$$

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \begin{cases}\displaystyle Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})-\frac{1}{T},&\ \text{if}\ c_{t}\ \text{is not sampled},\\[7.5pt] \displaystyle Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+\frac{1}{\varrho T}-\frac{1}{T},&\ \text{otherwise.}\end{cases}$$

Recall that each pairwise comparison is sampled by the Bernoulli method, independently, with probability $\varrho$. Therefore, we have that

$$\ \ \mathbb{E}\big{[}\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}\ \big{]}$$ (104)

$$= \ \ Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+\varrho\cdot\left(\frac{1}{\varrho T}-\frac{1}{T}\right)+(1-\varrho)\cdot\left(-\frac{1}{T}\right)$$

$$= \ \ Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}).$$

Combine the two cases, we know that $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{T}({\boldsymbol{\mathcal{C}^{\prime}}}))$ is a martingale.

Next we will show that the variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ conditioned on $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ is bounded by $1/\varrho T^{2}$. If $c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}$ and with simple calculation from (101) and (102), the variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ given $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ equals to zeros as

$$\textbf{{Var}}(\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}\ )=0.$$ (105)

When $c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}$, we have

$$\ \ \textbf{{Var}}(\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}\ )$$ (106)

$$= \ \ (1-\varrho)\cdot\left(\frac{1}{T}\right)^{2}+\varrho\cdot\left(\frac{1}{\varrho T}-\frac{1}{T}\right)^{2}$$

$$= \ \ \frac{1}{T^{2}}\cdot\left(\frac{1}{\varrho}-1\right)\leq\frac{1}{\varrho T^{2}}.$$

Combine the two cases, we know that the variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ conditioned on $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ is bounded by $1/\varrho T^{2}$.

It always holds that

$$|Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})|\leq\textbf{{max}}\left\{\frac{1}{T},\frac{1}{\varrho T}-\frac{1}{T}\right\}\leq\frac{1}{\varrho T}.$$ (107)

At last, we complete the proof of this lemma by proving the following two inequalities for any $\varrho$ satisfying the condition (98).

$$\mathbb{P}\left(\ \big{|}A_{T}(\boldsymbol{\mathcal{C}^{\prime}})\ -B_{T}(\boldsymbol{\mathcal{C}^{\prime}})\big{|}\geq\frac{\epsilon}{2}\ \right)\leq\frac{\delta}{2}$$ (108a)

$$\mathbb{P}\left(\ \big{|}B_{T}(\boldsymbol{\mathcal{C}^{\prime}})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{T}^{\prime})\big{|}\geq\frac{\epsilon}{2}\ \right)\leq\frac{\delta}{2}$$ (108b)

We can choose $\lambda=\epsilon/2$, $\sigma^{2}_{t}=1/\varrho T^{2}$ and $M=1/\varrho T$ and apply Lemma 1 on $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{T}({\boldsymbol{\mathcal{C}^{\prime}}}))$. As $Z_{0}(\boldsymbol{\mathcal{C}^{\prime}})=0$, we have

$$|A_{T}(\boldsymbol{\mathcal{C}^{\prime}})-B_{T}(\boldsymbol{\mathcal{C}^{\prime}})|=|Z_{T}(\boldsymbol{\mathcal{C}^{\prime}})-Z_{0}(\boldsymbol{\mathcal{C}^{\prime}})|,$$ (109)

and (108a) will holds that

$$\ \ \mathbb{P}\left(\ \big{|}A_{T}(\boldsymbol{\mathcal{C}^{\prime}})-B_{T}(\boldsymbol{\mathcal{C}^{\prime}})\big{|}\geq\frac{\epsilon}{2}\ \right)$$ (110)

$$\leq \ \ 2\cdot\textbf{{exp}}\left(-\frac{(\epsilon/2)^{2}}{2T\cdot(1/\varrho T^{2})+\epsilon/(6\varrho T)}\right)$$

$$< \ \ 2\cdot\textbf{{exp}}\left(-\frac{\epsilon^{2}\varrho T}{9}\right).$$

When

$$\varrho\geq 9\cdot\frac{\textbf{{ln}}(\delta/4)}{\epsilon^{2}T},$$ (111)

the (110) is further bounded by $\delta/2$.

To prove (108b), we observe that

$$B_{T}(\boldsymbol{\mathcal{C}^{\prime}})=d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{T})\cdot\frac{|\boldsymbol{C}^{\prime}_{T}|}{\varrho T},$$ (112)

and each pairwise comparison is selected by Bernoulli method with probability $\varrho$, independently of other pairwise comparison. The size of $\boldsymbol{C}^{\prime}_{t}$ equals to the expectation of binaomial distribution $\textbf{{Binaomial}}(T,\varrho)$, regardless of the adversary’s strategy. Applying the Chernoff inequality with $\delta=\epsilon/2$, we have

$$\ \ \mathbb{P}\left(\big{|}|\boldsymbol{C}^{\prime}_{T}|-\varrho T\big{|}\geq\frac{\epsilon\varrho T}{2}\right)$$ (113)

$$\leq \ \ 2\cdot\textbf{{exp}}\left(-\frac{(\epsilon/2)^{2}\varrho T}{2+\epsilon/3}\right)$$

$$< \ \ 2\cdot\textbf{{exp}}\left(-\frac{\epsilon^{2}\varrho T}{10}\right).$$

When

$$\varrho\geq 10\cdot\frac{\textbf{{ln}}(\delta/4)}{\epsilon^{2}T},$$ (114)

the above inequality is further bounded by $\delta/2$. Conditioning on this event ($||\boldsymbol{C}^{\prime}_{t}|-\varrho T|\geq\epsilon\varrho T/2$), we have

$$\ \ \big{|}d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{T})-B_{T}(\boldsymbol{\mathcal{C}^{\prime}})\big{|}$$ (115)

$$= \ \ \left|1-\frac{|\boldsymbol{C}^{\prime}_{T}|}{\varrho T}\right|\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{T})$$

$$\leq \ \ \left|1-\frac{|\boldsymbol{C}^{\prime}_{T}|}{\varrho T}\right|\leq\frac{\epsilon}{2},$$

where the first inequality follows the fact $d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{T})$ is always bounded by $1$, and the second inequality follows form the condition $||\boldsymbol{C}^{\prime}_{t}|-\varrho T|\geq\epsilon\varrho T/2$. We complete the proof of (108b).

Indeed, taking a union bound over (108a) and (108b), applying the triangle inequality and observing that $A_{T}(\boldsymbol{\mathcal{C}^{\prime}})=d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{T})$, we obtain the desired conclusion (99). ∎

Lemma 3.

For any dynamic stream $\boldsymbol{C}=\{c_{t}\}_{t=1}^{\infty}$ from $\boldsymbol{\mathcal{C}}^{\prime}$, if the parameter of reservoir method $\varrho$ holds that

$$\varrho\geq 2\cdot\frac{\textbf{{ln}}(2/\delta)}{\epsilon^{2}},$$ (116)

we have

$$\mathbb{P}(|d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime})|\geq\epsilon)\leq\delta,$$ (117)

where $\boldsymbol{C}^{\prime}$ is a sequence which is sampled from $\boldsymbol{C}$ by the reservoir method.

Proof.

Generally speaking, the proof of this lemma goes along the same lines as Lemma 2, except that adopting the other martingale. Specifically, we define

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ t\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t})=|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}|,$$ (118)

$$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ t\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime}_{t})=\frac{t}{\varrho}|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}^{\prime}_{t}|,$$

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}),$$

for $t\in(\varrho,T]$. When $t<\varrho$, we define

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}|.$$ (119)

The next step is similar to Lemma 2. We first show that $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{T}({\boldsymbol{\mathcal{C}^{\prime}}}))$ is a martingale. Notice that $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{T}({\boldsymbol{\mathcal{C}^{\prime}}}))$ is obviously a martingale for $t\leq\varrho$. When $t>\varrho$, the $\boldsymbol{C}^{\prime}_{t-1}$ is fixed and thence the values of $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ are fixed. Let $c_{t}$ be the next pairwise comparison for the reservoir sampling method, which could be either from the original data source or the adversarial data source controlled by the adversary. It is easy to check that

$$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=\left\{\begin{matrix}A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),&c_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}},\\ A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+1.&c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}.\end{matrix}\right.$$ (120)

For $B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$, we consider the three factors:

• is $c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}$ or not?

• is $c_{t}$ sampled or not?

• conditioning on $c_{t}$ being sampled, does it replace an element $r_{t}$ from ${\boldsymbol{\mathcal{C}^{\prime}}}$ in the sample, or $r_{t}$ not in ${\boldsymbol{\mathcal{C}^{\prime}}}$?

• When $c_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}$ is either not sampled, or sampled but with $r_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}$, the pairwise comparisons from $\boldsymbol{\mathcal{C}^{\prime}}$ are neither added nor removed into the cache of the reservoir method. Consequently, we have

  $$\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}=\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t-1}.$$ (121)

  By the definition of $B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$

  $$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \frac{t}{\varrho}\cdot|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}|$$ (122)

  $$= \ \ \frac{t-1}{\varrho}\cdot|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t-1}|+\frac{1}{\varrho}\cdot|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t-1}|$$

  $$= \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1}),$$

  where the third equality stands since the sampled pairwise comparisons of $\boldsymbol{C}_{t-1}$ is $|\boldsymbol{C}_{t-1}|=\varrho$ when $t>\varrho$. Therefore conditioned on $c_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}$, the expectation of $B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ is

  $$\ \ \mathbb{E}[B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})|c_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}]$$ (123)

  $$= \ \ \left(1-\frac{\varrho}{t}d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)\cdot\Big{(}B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\Big{)}+\frac{\varrho}{t}d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\left(B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})-\frac{t}{\varrho}\right)$$

  $$= \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}).$$

  Moreover, $A_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ when $c_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}$ and we deduce that

  $$\mathbb{E}\big{[}\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}\ \big{]}=Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}).$$ (124)

• Now $c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}$. When $c_{t}$ is neither added nor removed into the cache of the reservoir method, we have $|\boldsymbol{C}_{t}|=|\boldsymbol{C}_{t-1}|$ and $B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$. If $c_{t}$ into the cache and the replaced element $r_{t}\notin{\boldsymbol{\mathcal{C}^{\prime}}}$, which has probability $(\varrho/t)\cdot(1-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1}))$, we have

  $$|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}| = \ \ |\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t-1}|+1$$ (125)

  $$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \frac{t}{\varrho}\cdot|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t}|$$

  $$= \ \ \frac{t}{\varrho}\cdot|\boldsymbol{\mathcal{C}^{\prime}}\cap\boldsymbol{C}_{t-1}|+\frac{t}{\varrho}$$

  $$= \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})+\frac{t}{\varrho}.$$

  Then the expectation of $B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ conditioned on $c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}$ is

  $$\ \ \mathbb{E}[B_{t}({\boldsymbol{\mathcal{C}^{\prime}}})|c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}]$$ (126)

  $$= \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})+\left(\frac{\varrho}{t}\cdot(1-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)\cdot\frac{t}{\varrho}$$

  $$= \ \ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+1.$$

  Furthermore, with the definition of $A_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ when $c_{t}\in{\boldsymbol{\mathcal{C}^{\prime}}}$, we know that

  $$\mathbb{E}\big{[}\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}\ \big{]}=Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}).$$ (127)

  The analysis of the above two cases implies that $(Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}}))$ is indeed a martingale.

  The second part of proof is to obtain the bounds on the difference $|Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})|$ and the variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ given $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$. With the above analysis, we know that

  $$A_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) = \ \ \left\{\begin{matrix}A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),&c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}},\\[5.0pt] A_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+1,&c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}.\end{matrix}\right.$$ (128)

  $$B_{t}({\boldsymbol{\mathcal{C}^{\prime}}}) \in \ \ \left\{\begin{matrix}\displaystyle\left[B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+1\right],&c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}},\\[5.0pt] \displaystyle\left[B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ B_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})+1+\frac{t}{\varrho}\right],&c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}.\end{matrix}\right.$$

  By the definition of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$, we conclude that

  $$|Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})|\leq\frac{t}{\varrho}.$$ (129)

We next bound the variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ conditioned on $Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}})$ and $d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$. When $c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}}$, with probability $(\varrho/t)\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$, it holds that

$$\mathbb{E}[Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})]-Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=\frac{t}{\varrho}-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1}).$$ (130)

Otherwise, with probability $1-(\varrho/t)\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$, we have

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-\mathbb{E}[Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})]=d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1}).$$ (131)

Therefore,

$$\ \ \textbf{{Var}}(\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\notin\boldsymbol{\mathcal{C}^{\prime}},d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\ )$$ (132)

$$= \ \ \frac{\varrho}{t}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\cdot\left(\frac{t}{\varrho}-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)^{2}+\left(1-\frac{\varrho}{t}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)\cdot d^{2}_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$$

$$= \ \ \frac{t}{\varrho}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})-d^{2}_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$$

$$\leq \ \ \frac{t}{\varrho}.$$

When $c_{t}\in\boldsymbol{\mathcal{C}^{\prime}}$, with probability $(\varrho/t)\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$, it holds that

$$Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})-\mathbb{E}[Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})]=\frac{t}{\varrho}+d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})-1.$$ (133)

Otherwise, with probability $1-(\varrho/t)\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$, we have

$$\mathbb{E}[Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})]-Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})=1-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1}).$$ (134)

Thus,

$$\ \ \textbf{{Var}}(\ Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})\ \big{|}\ Z_{0}({\boldsymbol{\mathcal{C}^{\prime}}}),\dots,Z_{t-1}({\boldsymbol{\mathcal{C}^{\prime}}}),\ c_{t}\in\boldsymbol{\mathcal{C}^{\prime}},d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\ )$$ (135)

$$= \ \ \frac{\varrho}{t}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\cdot\left(\frac{t}{\varrho}-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)^{2}+\left(1-\frac{\varrho}{t}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})\right)\cdot d^{2}_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$$

$$= \ \ \frac{t}{\varrho}\cdot d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})-d^{2}_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$$

$$\leq \ \ \frac{t}{\varrho}.$$

(132) and (135) indicate that the conditional variance of $Z_{t}({\boldsymbol{\mathcal{C}^{\prime}}})$ is bounded by $t/\varrho$. Moreover, the bound remains intact when we remove the condition on $d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}_{t-1})$.

Now we come to the conclusion of the whole lemma. Observe that

$$\ \ \mathbb{P}(|d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime})|\geq\epsilon)$$ (136)

$$= \ \ \mathbb{P}(|B_{T}(\boldsymbol{\mathcal{C}^{\prime}})-A_{T}(\boldsymbol{\mathcal{C}^{\prime}})|\geq\epsilon\cdot T)$$

$$= \ \ \mathbb{P}(|Z_{T}(\boldsymbol{\mathcal{C}^{\prime}})-Z_{0}(\boldsymbol{\mathcal{C}^{\prime}})|\geq\epsilon\cdot T).$$

Then we apply Lemma 1 on the martingale $Z(\boldsymbol{\mathcal{C}^{\prime}})=(Z_{0}(\boldsymbol{\mathcal{C}^{\prime}}),\dots,Z_{T}(\boldsymbol{\mathcal{C}^{\prime}}))$ with $\lambda=\epsilon T$, $\sigma^{2}_{t}=t/\varrho$ for $t\geq\varrho$ and $\sigma^{2}_{t}=0$ for $t\geq\varrho$, and $M=T/\varrho$

$$\ \ \mathbb{P}(|Z_{T}(\boldsymbol{\mathcal{C}^{\prime}})-Z_{0}(\boldsymbol{\mathcal{C}^{\prime}})|\geq\epsilon\cdot T)$$ (137)

$$\leq \ \ 2\textbf{{exp}}\left(-\frac{\displaystyle\lambda^{2}}{\displaystyle 2\sum_{t=1}^{T}\sigma^{2}_{t}+\frac{\lambda M}{3}}\right)$$

$$= \ \ 2\textbf{{exp}}\left(-\frac{\epsilon^{2}T^{2}}{\displaystyle 2\sum_{t=1}^{T}\frac{t}{\varrho}+\frac{\epsilon T^{2}}{3\varrho}}\right)$$

$$= \ \ 2\textbf{{exp}}\left(-\frac{\epsilon^{2}T^{2}\varrho}{\displaystyle T(T+1)+\frac{\epsilon T^{2}}{3}}\right)$$

$$\leq \ \ 2\textbf{{exp}}\left(-\frac{\epsilon^{2}T^{2}\varrho}{2T^{2}}\right)=2\textbf{{exp}}\left(-\frac{\epsilon^{2}\varrho}{2}\right).$$

Therefore, it suffices to hold

$$\varrho\geq\frac{2}{\epsilon^{2}}\textbf{{ln}}\ \left(\frac{2}{\delta}\right)$$ (138)

and get the desired result. ∎

See 2

Proof.

For $i)$, the results of static case have been discussed by [47, 34, 48]. Notice that $\textbf{{log}}\ n(n-1)$ is the VC-dimension of $\boldsymbol{\mathcal{C}^{\prime}}$. For $ii)$, we start with the Bernoulli sampling method. For any dynamic stream $\boldsymbol{C}$ form $\boldsymbol{\mathcal{C}^{\prime}}$, we apply the first part of Lemma 2 with $\epsilon$ and $|\boldsymbol{\mathcal{C}^{\prime}}|$

$$P\Big{(}\big{|}d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime})\big{|}\geq\epsilon\Big{)}\leq\frac{\delta}{|\boldsymbol{\mathcal{C}^{\prime}}|},$$ (139)

where $\boldsymbol{C}^{\prime}$ is the sampled sequence by the Bernoulli sampling method. In the event

$$\big{|}d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C})-d_{\boldsymbol{\mathcal{C}^{\prime}}}(\boldsymbol{C}^{\prime})\big{|}\leq\epsilon,\ \ \forall\ \boldsymbol{C}\in\boldsymbol{\mathcal{C}^{\prime}},$$ (140)

by definition we know that $\boldsymbol{C}^{\prime}$ is an $\epsilon$-approximation of $\boldsymbol{C}$. Taking a union bound over all $\boldsymbol{C^{\prime}}\in\boldsymbol{\mathcal{C}^{\prime}}$, we conclude that the probability of this event not to hold is bounded by

$$\frac{\delta}{|\boldsymbol{\mathcal{C}^{\prime}}|}\cdot|\boldsymbol{\mathcal{C}^{\prime}}|=\delta,$$ (141)

meaning that Bernoulli method with $\varrho$ as above is $(\epsilon,\delta)$-representative.

The proof for reservoir method is identical, except that we apple Lemma 3. ∎

Asymptotic Optimality of the Proposed Policy with Complete Knowledge

We discuss the asymptotic optimality of the proposed stoo** time and generation rule with the complete knowledge. The assumption for the theoretical analysis is described. First there exist some regularity conditions on the prior distribution $\rho_{\boldsymbol{\theta}^{\prime}}$. Without loss of generality, we could set $\theta^{\prime}_{1}=0$ and the unknown model parameter satisfies $\boldsymbol{\theta}=[\theta_{2},\dots,\theta_{n}]\in\mathbb{R}^{n-1}$. The following assumptions have been applied in the sequential design for rank aggregation [16]. It is noteworthy that the following assumptions are mainly for the theoretical analysis. The proposed sequential manipulation method does not depend on the condition of $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ or the probability mass function $g$.

Assumption 1.

The support

$$\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})=\overline{\left\{\ \boldsymbol{\theta}\in\mathbb{R}^{n-1}\ \Bigg{|}\ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})>0\ \right\}}$$ (142)

is a compact set, where $\overline{\{\cdot\}}$ denotes the closeure of $\{\cdot\}$. Besides, for any full ranking list $\boldsymbol{\pi}_{0}$ which does not include the candidates belong to $\theta^{\prime}_{1}$,

$$\left\{\left\{\ \boldsymbol{\theta}\in\mathbb{R}^{n-1}\ \Bigg{|}\ \boldsymbol{\pi}(\boldsymbol{\theta})=\boldsymbol{\pi}_{0}\right\}\bigcap\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\right\}^{\circ}\neq\varnothing,$$ (143)

where $\{\cdot\}^{\circ}$ represents the interior of $\{\cdot\}$.

This assumption assigns the bounded support $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ to the prior distribution $\rho_{\boldsymbol{\theta}^{\prime}}$. The second part tell us that the support $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ contains a non-empty interior for every full ranking list.

Assumption 2.

For all $s>0$, there exists a constant $\delta>0$ such that

$$\mathcal{L}\left(\boldsymbol{B}(\boldsymbol{\theta},\epsilon)\cap\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\right)\geq\textbf{{min}}\{\delta\epsilon^{n-1},1\},$$ (144)

where $\boldsymbol{B}(\boldsymbol{\theta},\epsilon)$ denotes the open ball centered at $\boldsymbol{\theta}$ with radius $\epsilon$, and $\mathcal{L}(\cdot)$ denotes the Lebesgue measure.

This assumption keeps the support $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$ be non-singular.

Assumption 3.

The log probability mass function $\textbf{{log}}\ g_{ij}(\boldsymbol{\theta})$ is uniform continuous differentiable w.r.t. $\boldsymbol{\theta}$ for all pairwise comparisons $(i,j)$, that is

$$\underset{\begin{matrix}\scriptstyle\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle(i,j)\in\boldsymbol{\mathfrak{C}}\end{matrix}}{\textbf{{sup}}}\ \left\|\nabla_{\boldsymbol{\theta}}\ \textbf{{log}}\ g_{ij}(\boldsymbol{\theta})\right\|<\infty.$$ (145)

This assumption needs the smoothness of the likelihood function. The BTl model satisfies this assumption.

Assumption 4.

The probability mass function $g$ satisfies:

$$\underset{\begin{matrix}\scriptstyle\boldsymbol{\theta},\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\end{matrix}}{\textbf{{min}}}\ \underset{(i,j)}{\textbf{{max}}}\ \ g_{i,j}(\boldsymbol{\theta})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\theta})}{g_{i,j}(\boldsymbol{\tilde{\theta}})}>0.$$ (146)

This assumption requires the distinguishability between any pair of candidates in $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$, e.g. there does not exist tie between any pair of candidates. Assumption 4 is a standard assumption in sequential hypothesis testing, which is known as the “indifference zone” assumption. The “indifference zone” condition tell us that the null and alternative hypotheses are separated in the sense that the Kullback-Leibler divergence between the two hypotheses is positive. Here the assumption excludes the case that the true preference score is in between the two hypotheses ($i\succ j$ and $j\succ i$). Furthermore, it means that selecting $i\succ j$ or $j\succ i$ (the null and alternative hypothesis) will be different.

Assumption 5.

The prior distribution $\rho_{\boldsymbol{\theta}^{\prime}}$ satisfies

$$\underset{\boldsymbol{\theta}\in\{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\}^{\circ}}{\textbf{{inf}}}\ \ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})>0,$$ (147)

$$\underset{\ \ \boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\ \ }{\textbf{{sup}}}\ \ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})<0.$$

This assumption requires that the density function of prior distribution $\rho_{\boldsymbol{\theta}^{\prime}}$ is positive over $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$.

Considering the Assumption 1-5, the following theorem establishes a lower bound on the minimal Bayesian risk (45)

$$\mathfrak{R}^{*}=\underset{\boldsymbol{\Lambda},S}{\textbf{{inf}}}\ \mathfrak{R}(\boldsymbol{\Lambda},S).$$ (45)

Theorem 4.

If the Assumption 1-5 hold, we have

$$\underset{\chi\rightarrow 0}{\textbf{{lim inf}}}\ \ \frac{\mathfrak{R}^{*}}{\chi\mathbb{E}\left[\tau_{\chi}(\Theta)\right]}\geq 1,$$ (148)

where

$$\tau_{\chi}(\boldsymbol{\theta})=|\textbf{{log}}\ \chi|\cdot\left(\underset{\boldsymbol{\lambda}\in\boldsymbol{\Delta}\phantom{\boldsymbol{\tilde{\theta}}}}{\textbf{{max\phantom{f}}}}\underset{\begin{matrix}\scriptstyle\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\end{matrix}}{\textbf{{min}}}\ \ \sum_{(i,j)}\lambda_{i,j}g_{i,j}(\boldsymbol{\theta})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\theta})}{g_{i,j}(\boldsymbol{\tilde{\theta}})}\right)^{-1},$$ (149)

and

$$\mathbb{E}\left[\tau_{\chi}(\Theta)\right]=\int_{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\tau_{\chi}(\boldsymbol{\theta})\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})d\boldsymbol{\theta}.$$ (150)

To avoid confusion, we write $\Theta$ when $\boldsymbol{\theta}$ is viewed as a random variable.

Proof.

For any manipulation policy $(\boldsymbol{\Lambda},S)$ and a prior probability density function $\rho_{\boldsymbol{\theta}^{\prime}}$, there only exist two cases:

• $$\boldsymbol{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))]\geq\chi|\textbf{{log}}\ \chi|^{2}.$$ (151)

• $$\boldsymbol{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))]<\chi|\textbf{{log}}\ \chi|^{2}.$$ (152)

For the first case, the Bayesian risk

$$\mathfrak{R}(\boldsymbol{\Lambda},S)=\mathbb{E}[\mathfrak{R}(S)+\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))]$$ (44)

satisfies

$$\mathfrak{R}(\boldsymbol{\Lambda},S)\geq\chi|\textbf{{log}}\ \chi|^{2}\geq(1+o(1))\chi\mathbb{E}\left[\tau_{\chi}(\Theta)\right].$$ (153)

For the second case, it is easy to see

$$\mathfrak{R}(\boldsymbol{\Lambda},S)\geq\mathbb{E}[\mathfrak{R}(S)]=\chi\mathbb{E}[S].$$ (154)

Therefore, proving the results equals to show that

$$\underset{\chi\rightarrow 0}{\textbf{{lim inf}}}\ \ \frac{\chi\mathbb{E}[S]}{\chi\mathbb{E}\left[\tau_{\chi}(\Theta)\right]}\geq 1.$$ (155)

In other words, for any $\delta>0$, there exists a $\chi_{0}>0$ such taht when $\chi<\chi_{0}$,

$$\mathbb{E}[S]\geq(1-\delta)\mathbb{E}\left[\tau_{\chi}(\Theta)\right].$$ (156)

For each $\delta>0$, we set

$$\tau_{\chi,\delta}(\boldsymbol{\theta})=\left(1-\frac{2}{3}\delta\right)\tau_{\chi}(\boldsymbol{\theta}),$$ (157)

then

$$\mathbb{E}[S] \geq \ \ \ \mathbb{E}[S|S>\tau_{\chi,\delta}(\Theta)]$$ (158)

$$\geq \ \ \ \int_{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})\tau_{\chi,\delta}(\boldsymbol{\theta})\mathbb{P}(S>\tau_{\chi,\delta}(\Theta)|\Theta=\boldsymbol{\theta})d\boldsymbol{\theta}$$

$$= \ \ \mathbb{E}\left[\tau_{\chi,\delta}(\Theta)\right]-\int_{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})\tau_{\chi,\delta}(\boldsymbol{\theta})\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta)|\Theta=\boldsymbol{\theta})d\boldsymbol{\theta}$$

$$\geq \ \ \mathbb{E}\left[\tau_{\chi,\delta}({\Theta})\right]-\tau^{\textbf{{max}}}_{\chi,\delta}\cdot\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta)),$$

where $\tau^{\textbf{{max}}}_{\chi,\delta}$ is defined as

$$\tau^{\textbf{{max}}}_{\chi,\delta}=\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ \tau_{\chi,\delta}(\boldsymbol{\theta}).$$ (159)

According to Assumption 4, we know that

$$\tau^{\textbf{{max}}}_{\chi,\delta}=O(|\textbf{{log}}\ \chi|)=O(\mathbb{E}\left[\tau_{\chi}({\Theta})\right]).$$ (160)

Furthermore, to prove (156), it is sufficient to show

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta))=o(1).$$ (161)

The next step is to establish an upper bound for $\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta))$. Given a full ranking list $\boldsymbol{\pi}_{0}$, we write

$$\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}=\{\boldsymbol{\theta}|\boldsymbol{\pi}(\boldsymbol{\theta})=\boldsymbol{\pi}_{0}\}$$ (162)

as all preference score which will generate the full ranking $\boldsymbol{\pi}_{0}$. Then

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta)) = \ \ \sum_{\boldsymbol{\pi}_{0}}\ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$$ (163)

$$= \ \ O(1)\cdot\underset{\boldsymbol{\pi}_{0}}{\textbf{{max}}}\ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}).$$

Then we conduct an upper bound of $\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$ for any $\boldsymbol{\pi}_{0}$. Define an event $\boldsymbol{E}_{\boldsymbol{\pi}_{0}}$:

$$\boldsymbol{E}_{\boldsymbol{\pi}_{0}}=\left\{\frac{\mathbb{P}(\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}|\mathcal{F}_{S})}{\underset{(i,j):\boldsymbol{\Theta}_{i,j}\cap\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}=\varnothing}{\textbf{{max}}}\ \mathbb{P}(\Theta\in\boldsymbol{\Theta}_{i,j}|\mathcal{F}_{S})}>\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\right\},$$ (164)

where $\mathcal{F}_{S}=\sigma(c_{1},\dots,c_{S})$ denotes the $\sigma$-algebra generated by $c_{1},\dots,c_{S}$, $\epsilon>0$ is a constant and the definition of $\boldsymbol{\Theta}_{i,j}$ is

$$\boldsymbol{\Theta}_{i,j}=\big{\{}\boldsymbol{\ \theta}\in\mathbb{R}^{n}_{+}\ |\ \theta_{i}\geq\theta_{j}\ \big{\}}\cap\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}).$$ (52)

For each $\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$, we have

$$\ \ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$$ (165)

$$= \ \ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}_{\boldsymbol{\pi}_{0}})+\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}^{c}_{\boldsymbol{\pi}_{0}})$$

where $\boldsymbol{E}^{c}_{\boldsymbol{\pi}_{0}}$ is the complement of $\boldsymbol{E}_{\boldsymbol{\pi}_{0}}$. Then $\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$ can be further bounded by

$$\ \ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})$$ (166)

$$\leq \ \ \mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}_{\boldsymbol{\pi}_{0}})+\mathbb{P}(\boldsymbol{\theta}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}^{c}_{\boldsymbol{\pi}_{0}}).$$

The first term $\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}_{\boldsymbol{\pi}_{0}})$ equals to

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}_{\boldsymbol{\pi}_{0}})=\int_{\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}}\mathbb{P}(S\leq\tau_{\chi,\delta}(\boldsymbol{\theta}),\boldsymbol{E}_{\boldsymbol{\pi}_{0}}|\Theta=\boldsymbol{\theta})\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})d\boldsymbol{\theta}.$$ (167)

Notice that the intersection of $\{S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})\}$ and $\boldsymbol{E}_{\boldsymbol{\pi}_{0}}$ will be

$$\{S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})\}\cap\boldsymbol{E}_{\boldsymbol{\pi}_{0}}\subset\left\{\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\frac{\mathbb{P}(\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}|\mathcal{F}_{S})}{\underset{(i,j):\boldsymbol{\Theta}_{i,j}\cap\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}=\varnothing}{\textbf{{max}}}\ \mathbb{P}(\Theta\in\boldsymbol{\Theta}_{i,j}|\mathcal{F}_{S})}>\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\right\}.$$ (168)

Then

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\boldsymbol{\theta}),\boldsymbol{E}_{\boldsymbol{\pi}_{0}}|\Theta=\boldsymbol{\theta})\leq\mathbb{P}\left(\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\frac{\mathbb{P}(\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}|\mathcal{F}_{S})}{\underset{(i,j):\boldsymbol{\Theta}_{i,j}\cap\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}=\varnothing}{\textbf{{max}}}\ \mathbb{P}(\Theta\in\boldsymbol{\Theta}_{i,j}|\mathcal{F}_{S})}>\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\Bigg{|}\Theta=\boldsymbol{\theta}\right).$$ (169)

The next step is to transform the right-side of the inequality to a probability by a martingale parameterized by $\boldsymbol{\theta}$. The corresponding results are represented by the Lemma 3 in [16].

Lemma 4.

Suppose that $\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})$ is a martingale w.r.t the filtration $\{\mathcal{F}_{S}:S\geq 1\}$ and probability measure $\mathbb{P}(\cdot|\Theta=\boldsymbol{\theta})$ for any $\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}$,

$$\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})=\ell_{S}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S}(\boldsymbol{\theta}^{*}_{S})-\sum_{s=1}^{S}\sum_{(i,j)}\lambda_{i,j}^{(s)}g_{i,j}(\boldsymbol{\theta})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\theta})}{g_{i,j}(\boldsymbol{\theta}^{*}_{S})}+\sum_{s=1}^{S}\sum_{(i,j)}\lambda_{i,j}^{(s)}g_{i,j}(\boldsymbol{\theta})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\theta})}{g_{i,j}(\boldsymbol{\theta}^{\prime\prime})},$$ (170)

where

$$\ell_{S}(\boldsymbol{\theta})=\textbf{{log}}\ \prod_{s=1}^{S}g_{ij}(\boldsymbol{\theta}),$$ (171)

and

$$\boldsymbol{\theta}^{*}_{S}\in\underset{\begin{matrix}\scriptstyle\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\end{matrix}}{\textbf{{arg\ min}}}\ \sum_{s=1}^{S}\sum_{(i,j)}\lambda^{(s)}_{i,j}g_{i,j}(\boldsymbol{\theta})\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\theta})}{g_{i,j}(\tilde{\boldsymbol{\theta}})}.$$ (172)

There exists a $\chi_{0}>0$ such that when $\chi$ satisfies $0<\chi<\chi_{0}$, it holds that

$$\ \ \mathbb{P}\left(\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\frac{\mathbb{P}(\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}|\mathcal{F}_{S})}{\underset{(i,j):\boldsymbol{\Theta}_{i,j}\cap\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}=\varnothing}{\textbf{{max}}}\ \mathbb{P}(\Theta\in\boldsymbol{\Theta}_{i,j}|\mathcal{F}_{S})}>\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$ (173)

$$\leq \ \ \mathbb{P}\left(\underset{\begin{matrix}\scriptstyle 1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta}),\\ \scriptstyle\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}\end{matrix}}{\textbf{{max}}}\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})\geq\frac{\delta}{2}|\textbf{{log}}\ \chi|\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right).$$

With Lemma 4, we know that establishing the upper bound of (169) equals to find an upper bound of the right hand side of (173). It is noteworthy that the right hand side of (173) is a stochastic process, indexed by $\boldsymbol{\theta}^{\prime\prime}$ and $S$, going beyond a certain level ($\delta|\textbf{{log}}\ \chi|/2$).

To handle the level-crossing probabilities, we introduce the Azuma-Hoeffding inequality [3, 28] and derive the level-crossing probability by aggregating marginal tail bounds of a random field by Lemma 5 and 6.

Lemma 5 (Azuma-Hoeffding Inequality).

Let $\mathcal{M}_{S}$ be a martingale w.r.t. the filtration $\{\mathcal{F}_{S}:S\geq 1\}$ and $\Delta\mathcal{M}_{S}=\mathcal{M}_{S}-\mathcal{M}_{S-1}$. Suppose that $\Delta\mathcal{M}_{S}\in[a_{S},b_{S}]$ where $a_{S}$ and $b_{S}$ are deterministic constants, for each $S>0$, we have

$$\mathbb{P}\left(\underset{1\leq s\leq S}{\textbf{{max}}}\ \mathcal{M}_{s}\geq S_{0}\right)\leq\textbf{{exp}}\left(-\frac{2S_{0}^{2}}{\sum_{s=1}^{S}(b_{s}-a_{s})^{2}}\right).$$ (174)

Lemma 6.

Let $\{\zeta(\boldsymbol{\theta}):\boldsymbol{\theta}\in\boldsymbol{\Theta}\}$ be a random filed over a compact set $\boldsymbol{\Theta}\subset\mathbb{R}^{n}$ that satisfies Assumption 2, where $\zeta(\cdot)$ has a continuous sample path almost surely under a probability measure $\mathbb{P}$. Moreover, $\zeta(\cdot)$ has a Lipschitz-continuous sample path in the sense that there exists a constant $\kappa$ such that for all $\boldsymbol{\theta},\boldsymbol{\theta}^{\prime}\in\boldsymbol{\Theta}$,

$$\big{|}\zeta(\boldsymbol{\theta})-\zeta(\boldsymbol{\theta}^{\prime})\big{|}\leq\kappa\|\boldsymbol{\theta}-\boldsymbol{\theta}^{\prime}\|$$ (175)

almost surely under $\mathbb{P}$. We define

$$\beta(\boldsymbol{\theta},b)=\mathbb{P}\big{(}\zeta(\boldsymbol{\theta})\geq b\big{)},$$ (176)

and for all $\gamma>0$, it holds that

$$\mathbb{P}\left(\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}}{\textbf{{max}}}\ \zeta(\boldsymbol{\theta})\geq b\right)\leq\frac{\kappa^{n-1}_{L}}{\gamma^{n-1}\delta_{b}}\times\int_{\boldsymbol{\Theta}}\beta(\boldsymbol{\theta},b-\gamma)d\boldsymbol{\theta},$$ (177)

where $\delta_{b}$ is the constant in Assumption 2.

With Lemma 5, we set

$$\mathbb{P} = \ \ \mathbb{P}(\cdot|\Theta=\boldsymbol{\theta}),$$ (178)

$$S = \ \ \tau_{\chi,\delta}(\boldsymbol{\theta}),$$

$$S_{0} = \ \ \frac{\delta}{2}|\textbf{{log}}\ \chi|-1,$$

$$\mathcal{M}_{S} = \ \ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime}),$$

$$a_{S}\ \ =\ \ b_{S} = \ \ 2\ \underset{\begin{matrix}\scriptstyle\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle(i,j)\in\mathfrak{C}\end{matrix}}{\textbf{{max}}}|\textbf{{log}}\ g_{i,j}(\boldsymbol{\theta})|,$$

and for each $\boldsymbol{\theta}^{\prime\prime}$, we have

$$\mathbb{P}\left(\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})\geq\frac{\delta}{2}|\textbf{{log}}\ \chi|-1\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)\leq\textbf{{exp}}\left(-\frac{2\left(\frac{\delta}{2}|\textbf{{log}}\ \chi|-1\right)^{2}}{\tau_{\chi,\delta}(\boldsymbol{\theta})\cdot a_{1}^{2}}\right).$$ (179)

According to Assumption 1 and 3, $a_{1}<\infty$ stands and

$$\mathbb{P}\left(\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})\geq\frac{\delta}{2}|\textbf{{log}}\ \chi|-1\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)\leq\textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right),$$ (180)

where $O(\cdot)$ is the infinitesimal of the same order. Notice that

$$\ \ \underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})-\underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime\prime})$$ (181)

$$\leq \ \ \underset{1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta})}{\textbf{{max}}}\ |\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})-\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime\prime})|$$

$$\leq \ \ \tau_{\chi,\delta}(\boldsymbol{\theta})\cdot\kappa_{0}\cdot\|\boldsymbol{\theta}^{\prime\prime}-\boldsymbol{\theta}^{\prime\prime\prime}\|,\ \ \ \forall\ \ \boldsymbol{\theta}^{\prime\prime},\boldsymbol{\theta}^{\prime\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},$$

where

$$\kappa_{0}=4\ \underset{\begin{matrix}\scriptstyle\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}}),\\ \scriptstyle(i,j)\in\mathfrak{C}\end{matrix}}{\textbf{{max}}}\ |\nabla\textbf{{log}}\ g_{ij}(\boldsymbol{\theta})|<\infty$$ (182)

is the Lipschitz constant of $\mathcal{M}_{1}(\boldsymbol{\theta}^{\prime\prime})$. Consequently, $\mathcal{M}_{S}(\boldsymbol{\theta}^{\prime})$ is a Lipschitz-continuous random field w.r.t. $\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}$. Then adding the constraint $\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}$ into (180) and adopting Lemma 6 give

$$\ \ \mathbb{P}\left(\underset{\begin{matrix}\scriptstyle 1\leq S\leq\tau_{\chi,\delta}(\boldsymbol{\theta}),\\ \scriptstyle\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}}\end{matrix}}{\textbf{{max}}}\ \mathcal{M}_{S}(\boldsymbol{\theta}^{\prime\prime})\geq\frac{\delta}{2}|\textbf{{log}}\ \chi|-1\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$ (183)

$$\leq \ \ \textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right)\cdot\mathcal{L}(\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})\cdot\frac{\tau_{\chi,\delta}(\boldsymbol{\theta})^{n-1}\kappa^{n-1}_{0}}{\delta_{b}}$$

$$= \ \ \textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right)\cdot O(|\textbf{{log}}\ \chi|^{n-1}).$$

Combine (167), (169), (173) and (183), we have

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}_{\boldsymbol{\pi}_{0}})\leq\textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right)\cdot O(|\textbf{{log}}\ \chi|^{n-1}).$$ (184)

we have established a upper bound for first term in the right-hand side of (166). To bound the second term, we adopt the following lemma [16].

Lemma 7.

For every full ranking involving $n$ candidates $\boldsymbol{\pi}_{0}$ generated by $(\boldsymbol{\Lambda},S)$, the corresponding risk is

$$\mathfrak{R}_{\Theta}(\boldsymbol{R}(\boldsymbol{\Lambda},S))=\underset{(i,j)\in\boldsymbol{\mathfrak{C}}}{\sum}\mathbbm{I}[\theta^{\prime}_{i}<\theta^{\prime}_{j}]r_{i,j}+\mathbbm{I}[\theta^{\prime}_{i}>\theta^{\prime}_{j}](1-r_{i,j}).$$ (43)

If

$$\mathbb{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))]\leq\epsilon,$$ (185)

we have

$$\mathbb{P}(\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}},\boldsymbol{E}^{c}_{\boldsymbol{\pi}_{0}})\leq\left(1+\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\right)\epsilon.$$ (186)

Consequently, (166) will be bounded by

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta),\Theta\in\boldsymbol{\Theta}_{\boldsymbol{\pi}_{0}})\leq\textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right)\cdot O(|\textbf{{log}}\ \chi|^{n-1})+\left(1+\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\right)\epsilon.$$ (187)

Back to (163), it holds that

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta)\leq O(1)\times\left\{\textbf{{exp}}\left(-\Omega(\delta^{2}|\textbf{{log}}\ \chi|)\right)\cdot O(|\textbf{{log}}\ \chi|^{n-1})+\left(1+\frac{\chi^{\frac{\delta}{10}}}{\epsilon}\right)\epsilon\right\}.$$ (188)

Therefore, when $\chi\rightarrow 0$

$$\mathbb{P}(S\leq\tau_{\chi,\delta}(\Theta)=o(1).$$ (189)

We finish the proof. ∎

By the definition of asymptotic optimality of manipulation policy $(\boldsymbol{\Lambda},S)$:

$$\underset{\chi\rightarrow 0}{\textbf{{inf}}}\ \frac{\mathfrak{R}(\boldsymbol{\Lambda},S)}{\mathfrak{R}^{*}}=1,$$ (46)

we know that $(\boldsymbol{\Lambda},S)$ is asymptotic optimal when $\chi\rightarrow 0$ if

$$\mathfrak{R}(\boldsymbol{\Lambda},S)=(1+o(1))\cdot\mathfrak{R}^{*}.$$ (190)

Theorem 4 tells us that

$$\mathfrak{R}(\boldsymbol{\Lambda},S)=(1+o(1))\cdot\mathfrak{R}^{*}=(1+o(1))\chi\mathbb{E}_{\boldsymbol{\theta}\sim\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\left[\tau_{\chi}(\boldsymbol{\theta})\right]$$ (191)

as $\chi\rightarrow 0$ is sufficient to show the asymptotic optimality of $(\boldsymbol{\Lambda},S)$. To prove the asymptotic optimality with complete knowledge of the proposed stop** time (49) and the generation rule solved by (54), we require the identifiability of model to adopt the MLE and obtain the preference score. It is worth noting that the BTL model could satisfy this requirement.

The following theorem provides the asymptotic upper bounds for the expected Kendall tau of the proposed manipulation policy with complete information.

Theorem 5.

Consider the proposed stop** time (49) and the generation rule solved by (54) with complete knowledge, when the exploration probability $p\in(0,1)$ will be

$$p\propto|\textbf{{log}}\chi|^{-\frac{1}{2}+\delta_{0}}$$ (192)

for some $\delta_{0}\in\big{(}0,\frac{1}{2}\big{)}$. Then we have

$$\mathbb{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S))]=O(\chi).$$ (193)

Proof.

We first discuss the second stop** time $S_{2}$. By the generation rule solved through (54), the expected Kendall tau at $S_{2}$ is

$$\mathbb{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S_{2}))] = \ \ \mathbb{E}\underset{(i,j)\in\boldsymbol{\mathfrak{C}}}{\sum}\mathbbm{I}[\theta^{\prime}_{i}<\theta^{\prime}_{j}]r_{i,j}$$ (194)

$$= \ \ \int_{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\underset{(i,j):\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\sum}\mathbb{P}\left\{\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}_{1})>\underset{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}_{2})\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})d\boldsymbol{\theta}$$

$$= \ \ \int_{\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\sum}\mathbb{P}\left\{\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}_{2})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})d\boldsymbol{\theta}$$

where $\boldsymbol{R}(\boldsymbol{\Lambda},S)$, $r_{i,j}$, $\boldsymbol{\Theta}_{i,j}$, $z_{\alpha}(\chi)$, $\ell_{S}(\boldsymbol{\theta})$ are defined as (41), (42), (52), (50) and (171) correspondingly. The above equation can be further bounded by

$$\ \ \mathbb{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S_{2}))]$$ (195)

$$\leq \ \ \frac{n(n-1)}{2}\cdot\mathcal{L}\big{(}\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\big{)}\times\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})$$

$$\ \ \times\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \underset{(i,j):\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{max\phantom{p}}}}\ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}.$$

Here the inequality holds as

$$\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ell_{S_{2}}(\boldsymbol{\theta}_{1})\geq\ell_{S_{2}}(\boldsymbol{\theta}_{2}),\ \ \text{if}\ \ \boldsymbol{\theta}_{2}\notin\boldsymbol{\Theta}_{i,j}$$ (196)

and

$$\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})<\infty$$ (197)

by Assumption (5). The last part of (195) can be decomposed as

$$\ \ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}$$ (198)

$$\leq \ \ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \text{and}\ S_{2}\leq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}+\mathbb{P}\left\{S_{2}\geq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}.$$

For the first term of the above right-hand side, we introduce $S_{2}\wedge\tau=\textbf{{min}}(S_{2},\tau)$ and write

$$\ \ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \text{and}\ S_{2}\leq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}$$ (199)

$$\leq \ \ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}\wedge\tau}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}\wedge\tau}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}$$

Let $\eta(\boldsymbol{\theta}^{\prime\prime})$ be a random field

$$\eta(\boldsymbol{\theta}^{\prime\prime})=\ell_{S_{2}\wedge\tau}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}\wedge\tau}(\boldsymbol{\theta}),\ \ \forall\ \ \boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}.$$ (200)

The marginal tail probability of $\eta(\boldsymbol{\theta}^{\prime\prime})$ can be obtained by the following lemma [16].

Lemma 8.

For all $\boldsymbol{\theta}^{\prime\prime}\neq\boldsymbol{\theta}$ and constant $L>0$, we have

$$\mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S\wedge\tau}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S\wedge\tau}(\boldsymbol{\theta})>L\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\leq\textbf{{exp}}(-L).$$ (201)

We can take $L=z_{\alpha}(\chi)-1$ and obtain

$$\mathbb{P}\left\{\eta(\boldsymbol{\theta}^{\prime\prime})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\leq\textbf{{exp}}(-z_{\alpha}(\chi)+1).$$ (202)

Moreover, $\eta(\boldsymbol{\theta}^{\prime\prime})$ is a Lipschitz-continuous function as

$$\eta(\boldsymbol{\theta}^{\prime\prime})-\eta(\boldsymbol{\theta}^{\prime\prime\prime})\leq|\ell_{S_{2}\wedge\tau}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}\wedge\tau}(\boldsymbol{\theta}^{\prime\prime\prime})|\leq\tau\cdot\kappa_{0}\|\boldsymbol{\theta}^{\prime\prime}-\boldsymbol{\theta}^{\prime\prime\prime}\|.$$ (203)

Combining Lemma 6 and 7, we arrive at

$$\mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \eta(\boldsymbol{\theta}^{\prime\prime})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\leq O\Big{(}\tau^{n-1}\textbf{{exp}}\big{(}-z_{\alpha}(\chi)\big{)}\Big{)}.$$ (204)

To bound $\mathbb{P}\left\{S_{2}\geq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}$, we need the following lemma.

Lemma 9.

When $\tau$ satisfies

$$\tau=\Omega(|\textbf{{log}}\ \chi|^{3}),$$ (205)

we have

$$\mathbb{P}\left\{S_{2}\geq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\leq\chi^{2}.$$ (206)

For the stop** time $S_{1}$, we have the similar result:

$$\mathbb{P}\left\{S_{2}\geq\tau\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}\leq\chi^{2},\ \ \text{if}\ \tau=\Omega(|\textbf{{log}}\ \chi|^{3}).$$ (207)

Combining (198) with (204) and Lemma 8, we have

$$\ \ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{2}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{2}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}$$ (208)

$$\leq \ \ O(\chi^{2})+O\Big{(}\tau^{n-1}\textbf{{exp}}\big{(}-z_{\alpha}(\chi)\big{)}\Big{)}$$

$$= \ \ O(\chi^{2})+O\Big{(}\textbf{{exp}}(-|\textbf{{log}}\ \chi|-|\textbf{{log}}\ \chi|^{1-\alpha}+(n-1)\textbf{{log}}\ \tau)\Big{)}$$

$$= \ \ O(\chi^{2})+O\Big{(}\chi\textbf{{exp}}(-|\textbf{{log}}\ \chi|^{1-\alpha}+3(n-1)\textbf{{log}}\ |\textbf{{log}}\ \chi|)\Big{)}$$

$$= \ \ o(\chi)$$

and finish the analysis of $S_{2}$.

For $S_{1}$, we have

$$\ \ \underset{(i,j):1\leq i<j\leq n}{\textbf{{max}}}\ \textbf{{exp}}\left\{\underset{}{\textbf{{min}}}\left(\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{2}),\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{2})\right)\right\}$$ (209)

$$\leq \ \ \underset{(i,j):1\leq i<j\leq n}{\sum}\textbf{{exp}}\left\{\underset{}{\textbf{{min}}}\left(\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{2}),\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{2})\right)\right\}$$

$$\leq \ \ \textbf{{exp}}(-z_{\alpha}(\chi)).$$

We take logarithm for the both sides of the inequality

$$\underset{}{\textbf{{min}}}\left(\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{2})-\underset{}{\textbf{{min}}}\left(\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1}),\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}_{1})\right)\right)\geq z_{\alpha}(\chi).$$ (210)

Then the expected Kendall tau at $S_{1}$ will be bounded by

$$\ \ \mathbb{E}[\mathfrak{R}(\boldsymbol{R}(\boldsymbol{\Lambda},S_{1}))]$$ (211)

$$\leq \ \ \frac{n(n-1)}{2}\cdot\mathcal{L}\big{(}\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\big{)}\times\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \rho_{\boldsymbol{\theta}^{\prime}}(\boldsymbol{\theta})$$

$$\ \ \times\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{sup}}}\ \underset{(i,j):\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{max\phantom{p}}}}\ \mathbb{P}\left\{\underset{\boldsymbol{\theta}^{\prime\prime}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S_{1}}(\boldsymbol{\theta}^{\prime\prime})-\ell_{S_{1}}(\boldsymbol{\theta})>z_{\alpha}(\chi)\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right\}.$$

The rest of the proof is similar to that for $S_{2}$. ∎

Now we arrive at the asymptotic optimality of the expected stop** time of the proposed manipulation policy.

Theorem 6.

Consider the proposed stop** time (49) and the generation rule solved by (54) with complete knowledge, when the exploration probability $p\in(0,1)$ will be

$$p\propto|\textbf{{log}}\chi|^{-\frac{1}{2}+\delta_{0}}$$ (212)

for some $\delta_{0}\in\big{(}0,\frac{1}{2}\big{)}$. Then we have

$$\underset{\chi\rightarrow 0}{\textbf{{lim sup}}}\ \frac{\mathbb{E}[S]}{\mathbb{E}\left[\tau_{\chi}(\Theta)\right]}\leq 1.$$ (213)

Proof.

First we establish a upper bound for the expectation of a stop** time $S$, the numerator $\mathbb{E}[S]$.

$$\mathbb{E}[S] = \ \ \sum_{m=0}^{\infty}\mathbb{E}\left[S\ \big{|}\ m(1+\delta)\tau_{\chi}(\Theta)\leq S<(m+1)(1+\delta)\tau_{\chi}(\Theta)\right]$$ (214)

$$\leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+\sum_{m=1}^{\infty}\mathbb{E}\left[S\ \big{|}\ m(1+\delta)\tau_{\chi}(\Theta)\leq S<(m+1)(1+\delta)\tau_{\chi}(\Theta)\right]$$

$$\leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+(1+\delta)\cdot\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ \tau_{\chi}(\boldsymbol{\theta})\cdot$$

$$\ \ \phantom{(1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]}\sum_{m=1}^{\infty}(m+1)\mathbb{P}(m(1+\delta)\tau_{\chi}(\Theta)\leq S<(m+1)(1+\delta)\tau_{\chi}(\Theta))$$

$$\leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+(1+\delta)\cdot\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ \tau_{\chi}(\boldsymbol{\theta})\cdot$$

$$\ \ \phantom{(1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]}\sum_{m=1}^{\infty}(m+1)\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S<(m+1)(1+\delta)\tau_{\chi}(\Theta)\ \Big{|}\ \Theta=\boldsymbol{\theta}\right).$$

We start with stop** time $S_{2}$. For $m\geq 1$,

$$\ \ \mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S_{2}<(m+1)(1+\delta)\tau_{\chi}(\Theta)\ \Big{|}\ \Theta=\boldsymbol{\theta}\right)$$ (215)

$$\leq \ \ \mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S_{2}<(m+1)(1+\delta)\tau_{\chi}(\Theta),\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1]\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\leq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$

$$\ \ +\mathbb{P}\left(\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1]\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\geq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right),$$

where $\boldsymbol{\hat{\theta}}_{S}$ is the MLE

$$\boldsymbol{\hat{\theta}}_{S}=\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{arg sup}}}\ L\bigg{(}\boldsymbol{\theta},\boldsymbol{w}^{(S)}_{\mathcal{A}}\bigg{)},$$ (48)

and the $\delta_{1}$ and $\delta_{2}$ is two constants related to exploration probability $p\propto|\textbf{{log}}\chi|^{-\frac{1}{2}+\delta_{0}}$:

$$\delta_{1}=\frac{\delta_{0}}{8},\ \ \delta_{2}=|\textbf{{log}}\ \chi|^{-\frac{\delta_{0}}{2}}.$$ (216)

The following lemma show a upper bound of the second term in the preceding display.

Lemma 10.

Suppose $\boldsymbol{\lambda}^{(S)}=\{\lambda^{(S)}_{i,j}\}$ is a generation rule at $S$ step and $\{\epsilon_{\boldsymbol{\lambda},T_{1},T_{2}}\}$, $\{\delta_{T_{1},T_{2}}\}$ is two sequences of real numbers such that

$$\underset{\begin{matrix}\scriptstyle T_{1}\leq S\leq T_{2},\\ \scriptstyle(i,j)\in\mathfrak{C}\end{matrix}}{\textbf{{min}}}\ \lambda^{(S)}_{i,j}\geq\epsilon_{\boldsymbol{\lambda},T_{1},T_{2}},$$ (217)

$$\underset{T_{1}\rightarrow\infty}{\textbf{{lim}}}\ T_{1}\cdot\epsilon_{\boldsymbol{\lambda},T_{1},T_{2}}\cdot\delta^{2}_{T_{1},T_{2}}=\infty.$$

Then, it holds that

$$\mathbb{P}\left(\underset{T_{1}\leq S\leq T_{2}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\geq\delta_{T_{1},T_{2}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)\leq\textbf{{exp}}\Big{(}-\Omega\big{(}T_{1}\epsilon^{2}_{\boldsymbol{\lambda},T_{1},T_{2}}\delta^{4}_{T_{1},T_{2}}\big{)}\Big{)}\times O\big{(}T_{2}^{n}\big{)}.$$ (218)

When

$$T_{1}=m(1+\delta)\delta_{2}\tau_{\chi}(\boldsymbol{\theta}),\ \ T_{2}=m(1+\delta)\tau_{\chi}(\boldsymbol{\theta}),\ \ \epsilon_{\boldsymbol{\lambda},T_{1},T_{2}}=\Omega\Big{(}|\textbf{{log}}\chi|^{-\frac{1}{2}+\delta_{0}}\Big{)},\ \ \delta^{2}_{T_{1},T_{2}}=|\textbf{{log}}\chi|^{-\delta_{1}},$$ (219)

we have

$$\ \ \mathbb{P}\left(\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1)\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\geq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$ (220)

$$\leq \ \ \textbf{{exp}}\left(-\Omega\Big{(}m(1+\delta)\delta_{2}\tau_{\chi}(\boldsymbol{\theta})|\textbf{{log}}\chi|^{-4\delta_{1}}|\textbf{{log}}\chi|^{-1+2\delta_{0}}\Big{)}\right)\times O\Big{(}m^{n-1}|\textbf{{log}}\chi|^{n-1}\Big{)}$$

$$= \ \ \textbf{{exp}}\left(-\Omega\Big{(}m|\textbf{{log}}\chi|^{2\delta_{0}-4\delta_{1}}\delta_{2}\Big{)}\right)\times O\Big{(}m^{n-1}|\textbf{{log}}\chi|^{n-1}\Big{)}$$

$$= \ \ \textbf{{exp}}\left(-\Omega\Big{(}m|\textbf{{log}}\chi|^{\delta_{0}}\Big{)}\right)\times O\Big{(}m^{n-1}|\textbf{{log}}\chi|^{n-1}\Big{)}.$$

Now we analyze the first term on the right-hand side of (215). For $m\geq 1$, $S_{2}>m(1+\delta)\tau_{\chi}(\boldsymbol{\theta})$ implies that there exists a pairwise comparison $(i,j)$ such that

$$\left|\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\right|\leq z_{\alpha}(\chi),$$ (221)

where $S=m(1+\delta)\tau_{\chi}(\boldsymbol{\theta})$. Without loss of generality, let $\boldsymbol{\theta}\in\boldsymbol{\Theta}_{i,j}$ and $S_{2}>m(1+\delta)\tau_{\chi}(\boldsymbol{\theta})$ further shows that

$$\ell_{S}(\boldsymbol{\theta})-\underset{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\leq z_{\alpha}(\chi).$$ (222)

Consequently, the first term on the right-hand side of (215) will be bounded

$$\ \ \mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S_{2}<(m+1)(1+\delta)\tau_{\chi}(\Theta),\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1)\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\leq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$ (223)

$$\leq \ \ \mathbb{P}\left(\ell_{S}(\boldsymbol{\theta})-\underset{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\leq z_{\alpha}(\chi),\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1)\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\leq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right).$$

The preceding display can be bounded by the following lemma.

Lemma 11.

Suppose that the generation rule $\boldsymbol{\lambda}^{*}(\boldsymbol{\hat{\theta}}_{S})$ is solved by (54):

$$\boldsymbol{\lambda}^{*}(\boldsymbol{\hat{\theta}}_{S})\in\underset{\ \boldsymbol{\lambda}\in\boldsymbol{\Delta}\phantom{\tilde{1}}}{\textbf{{arg\ max}}}\ \underset{\begin{matrix}\scriptstyle\boldsymbol{\tilde{\theta}}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})\\ \scriptstyle\boldsymbol{\pi}(\boldsymbol{\hat{\theta}}_{S})\neq\boldsymbol{\pi}(\boldsymbol{\tilde{\theta}})\end{matrix}}{\textbf{{min\phantom{g}}}}\ \sum_{(i,j)}\lambda_{i,j}\cdot g_{i,j}(\boldsymbol{\hat{\theta}}_{S})\cdot\textbf{{log}}\frac{g_{i,j}(\boldsymbol{\hat{\theta}}_{S})}{g_{i,j}(\boldsymbol{\tilde{\theta}})}.$$ (54)

If $\boldsymbol{\lambda}^{*}(\boldsymbol{\hat{\theta}}_{S})$ is adopted with probability $1-o(1)$ uniformly for $S\in[m(1+\delta)\delta_{2}\tau_{\chi}(\boldsymbol{\theta}),m(1+\delta)\tau_{\chi}(\boldsymbol{\theta})]$, we have

$$\ \ \mathbb{P}\left(\ell_{S}(\boldsymbol{\theta})-\underset{\boldsymbol{\theta}_{2}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\leq z_{\alpha}(\chi),\underset{\begin{matrix}\scriptstyle S/(1+\delta)\tau_{\chi}(\Theta)\\ \scriptstyle\in[\delta_{2}m,m+1)\end{matrix}}{\textbf{{max}}}\ \|\boldsymbol{\hat{\theta}}_{S}-\boldsymbol{\theta}\|\leq\ |\textbf{{log}}\ \chi|^{-\delta_{1}}\ \Bigg{|}\ \Theta=\boldsymbol{\theta}\right)$$ (224)

$$\leq \ \ \textbf{{exp}}\Big{(}-\Omega(m|\textbf{{log}}\ \chi|)\Big{)}\times O\Big{(}|\textbf{{log}}\ \chi|^{n-1}m^{n-1}\Big{)},$$

where $S=m(1+\delta)\tau_{\chi}(\boldsymbol{\theta})$.

Combining the results of Lemma 9 and Lemma 10, (215) will be bounded as

$$\ \ \mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S_{2}<(m+1)(1+\delta)\tau_{\chi}(\Theta)\ \Big{|}\ \Theta=\boldsymbol{\theta}\right)$$ (225)

$$\leq \ \ \left(\textbf{{exp}}\Big{(}-\Omega(m|\textbf{{log}}\ \chi|)\Big{)}+\textbf{{exp}}\Big{(}-\Omega(m|\textbf{{log}}\ \chi|^{\delta_{0}})\Big{)}\right)\times O\Big{(}|\textbf{{log}}\ \chi|^{n-1}m^{n-1}\Big{)}.$$

Aggregating the preceding display with (214), the expectation of the stop** time $S_{2}$ will be bounded

$$\mathbb{E}[S_{2}] \leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+(1+\delta)\cdot\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ \tau_{\chi}(\boldsymbol{\theta})\cdot$$ (226)

$$\ \ \phantom{(1+\delta)}\sum_{m=1}^{\infty}(m+1)\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\mathbb{P}\left(m(1+\delta)\tau_{\chi}(\Theta)\leq S<(m+1)(1+\delta)\tau_{\chi}(\Theta)\ \Big{|}\ \Theta=\boldsymbol{\theta}\right)$$

$$\leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+O(|\textbf{{log}}\ \chi|)\times$$

$$\ \ \phantom{(1+\delta)}\sum_{m=1}^{\infty}(m+1)\left(\textbf{{exp}}\Big{(}-\Omega(m|\textbf{{log}}\ \chi|)\Big{)}+\textbf{{exp}}\Big{(}-\Omega(m|\textbf{{log}}\ \chi|^{\delta_{0}})\Big{)}\right)\times O\Big{(}|\textbf{{log}}\ \chi|^{n-1}m^{n-1}\Big{)}$$

$$\leq \ \ (1+\delta)\mathbb{E}[\tau_{\chi}(\Theta)]+o(|\textbf{{log}}\ \chi|).$$

The preceding display is the desired conclusion for the asymptotic optimality of $S_{2}$.

Next we proceed to the case of $S_{1}$. Notice that the event $S_{1}>S$ implies that

$$\sum_{(i,j)}\textbf{{exp}}\left(\textbf{{min}}\left(\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\boldsymbol{\theta}^{\prime})}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2}),\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\boldsymbol{\theta}^{\prime})}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\right)\right)>\textbf{{exp}}(-z_{\alpha}(\chi))$$ (227)

which further conducts

$$\ \ n(n-1)\cdot\underset{(i,j)}{\textbf{{max}}}\ \textbf{{exp}}\left(\textbf{{min}}\left(\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\boldsymbol{\theta}^{\prime})}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2}),\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\textbf{{Supp}}(\boldsymbol{\theta}^{\prime})}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\right)\right)$$ (228)

$$> \ \ \textbf{{exp}}(-z_{\alpha}(\chi)).$$

It means that there exist a pairwise comparison $(i,j)$ such that

$$\left|\underset{\boldsymbol{\theta}_{1}\in\boldsymbol{\Theta}_{i,j}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{1})-\underset{\boldsymbol{\theta}_{2}\in\in\boldsymbol{\Theta}_{j,i}}{\textbf{{sup}}}\ \ell_{S}(\boldsymbol{\theta}_{2})\right|\leq z_{\alpha}(\chi)+\textbf{{log}}\ n(n-1).$$ (229)

Then the analysis process of $S_{1}$ is similar to $S_{2}$ by replacing $z_{\alpha}(\chi)$ with $z_{\alpha}(\chi)+\textbf{{log}}\ n(n-1)$. ∎

Proof of Theorem 3

The following proposition [10] shows the strong duality for the Wasserstein DRO problem that we investigate in this paper:

$$\underset{\ \boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\ \textbf{{max\phantom{p}}}}\ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right],$$ (59)

This strong duality result ensures that the inner supremum admits a reformulation which is a simple, univariate optimization problem. Note that there exists the other strong duality result of Wasserstein DRO [25].

Proposition 2.

Let $d:\mathbb{R}\times\mathbb{R}\rightarrow[0,\infty]$ be a lower semi-continuous cost function satisfying $d(p,\ q)=0$ whenever $p=q$. For $\lambda\geq 0$ and loss function $\ell$

$$\ell(\boldsymbol{\theta},p_{i,j})=p_{i,j}\cdot\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}),$$ (230)

where $f_{(i,j)}(\boldsymbol{\theta})$ is the probabilistic mass function of pairwise comparison $(i,j)$, we define

$$\psi_{\lambda,\ell}(\boldsymbol{\theta},q_{i,j}):=\underset{q_{i,j}\in\mathbb{R}_{+}}{\textbf{{sup}}}\ \Big{\{}\ \ell(\boldsymbol{\theta},q_{i,j})-\lambda\cdot d(p_{i,j},q_{i,j})\ \Big{\}}.$$ (231)

Then it holds that

$$\underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\big{[}L\big{(}\boldsymbol{\theta},\boldsymbol{q}\big{)}\big{]}=\underset{\lambda\geq 0}{\ \textbf{{min\phantom{p}}}}\left\{\ \lambda\gamma+\sum_{(i,j)}\psi_{\lambda,\ell}(\boldsymbol{\theta},q_{i,j})\ \right\}$$ (232)

With the strong duality, we have the following result.

See 3

Proof.

Let $\Delta_{ij}=q_{ij}-p_{ij}$ where $\boldsymbol{q}=(q_{1,2},\dots,q_{n,n-1})\in\mathbb{R}^{N}_{+}$. We define $\psi_{\lambda,\ell}(\boldsymbol{\theta})$ as

$$\psi_{\lambda,\ell}(\boldsymbol{\theta}) = \ \ \underset{\boldsymbol{q}}{\textbf{{sup}}}\ \underset{(i,j)}{\sum}\ \Big{\{}\ell(\boldsymbol{\theta},\ q_{ij})-\lambda\big{[}d(p_{i,j},\ q_{i,j})\big{]}^{2}\Big{\}}$$ (233)

$$= \ \ \underset{\boldsymbol{q}}{\textbf{{sup}}}\ \underset{(i,j)}{\sum}\ \left\{q_{ij}\cdot\textbf{{log}}~{}g_{(i,j)}(\boldsymbol{\theta})-\lambda\big{|}p_{ij}-q_{ij}\big{|}^{2}\right\}$$

$$= \ \ \underset{(i,j)}{\sum}\ \underset{\Delta_{ij}\in\mathbb{R}}{\textbf{{sup}}}\ \Big{(}\Delta_{ij}b_{ij}-\lambda\Delta_{ij}^{2}+p_{ij}b_{ij}\Big{)},$$

where

$$b_{i,j}=\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}),$$ (234)

and the third equality holds due to $\psi_{\lambda,\ell}(\boldsymbol{\theta})$ is a decomposable function. Expanding (233), we can simplify $\psi_{\lambda,\ell}(\boldsymbol{\theta})$ as below:

$$\psi_{\lambda,\ell}(\boldsymbol{\theta}) = \ \ \big{\langle}\boldsymbol{p},\boldsymbol{b}\big{\rangle}+\underset{(i,j)}{\sum}\ \underset{\Delta_{ij}\in\mathbb{R}}{\textbf{{sup}}}\ \big{(}\Delta_{ij}b_{ij}-\lambda\Delta_{ij}^{2}\big{)}$$ (235)

$$= \ \ \left\{\begin{array}[]{lc}\displaystyle\langle\boldsymbol{p},\ \boldsymbol{b}\rangle+\frac{1}{4\lambda}\|\boldsymbol{b}\|^{2}_{2},&\ \text{if }\ \lambda>0,\\[10.0pt] \infty,&\ \text{if }\ \lambda=0.\end{array}\right.$$

Next, we investigate the duality of (59) with Proposition 2. As $\psi_{\lambda,\ell}(\boldsymbol{\theta},\ q_{i,j})=\infty$ when $\lambda=0$, the dual formulation of the supremum in (59) would be

$$\ \ \ \ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right]$$ (236)

$$=\ \ \underset{\lambda\geq 0}{\textbf{{min}}}\ \ \Bigg{\{}\lambda\gamma+\psi_{\lambda,\ell}(\boldsymbol{\theta})\Bigg{\}}$$

$$=\ \ \underset{\lambda>0}{\textbf{{min}}}\ \ \Bigg{\{}\lambda\gamma+\langle\boldsymbol{p},\ \boldsymbol{b}\rangle+\frac{1}{4\lambda}\|\boldsymbol{b}\|^{2}_{2}\Bigg{\}}.$$

By the definition of $\boldsymbol{b}$, we know that

$$L(\boldsymbol{\theta},\ \boldsymbol{p})=\big{\langle}\boldsymbol{p},\ \boldsymbol{b}\big{\rangle}$$ (237)

Moreover, notice that the right hand side of (236) is a convex function which approaches infinity when $\lambda\rightarrow\infty$, the global optimal of it can be obtained uniquely via the first order optimality condition as

$$\frac{\partial}{\partial\lambda}\Bigg{\{}\ \lambda\gamma+\langle\boldsymbol{p},\ \boldsymbol{b}\rangle+\frac{1}{4\lambda}\|\boldsymbol{b}\|^{2}_{2}\ \Bigg{\}}=0,$$ (238)

and the optimal dual variable is

$$\lambda^{*}_{\gamma}=\frac{\|\boldsymbol{b}\|_{2}}{2\sqrt{\gamma}}.$$ (239)

Substituting $\lambda^{*}_{\gamma}$ and $\boldsymbol{b}$ into (236), we have

$$\ \ \underset{\mathbb{Q}\in\boldsymbol{\mathfrak{U}}^{\gamma}(\mathbb{P})}{\textbf{{sup}}}\ \mathbb{E}_{\boldsymbol{q}\sim\mathbb{Q}}\left[L(\boldsymbol{\theta},\boldsymbol{q})\right]$$ (240)

$$= \ \ \sqrt{\gamma}\cdot\|\boldsymbol{b}\|_{2}+\langle\boldsymbol{p},\ \boldsymbol{b}\rangle$$

$$= \ \ \sqrt{\ \gamma\underset{(i,j)}{\sum}[\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta})]^{2}}+\underset{(i,j)}{\sum}p_{i,j}\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}).$$

When we sepicify the probability mass function of $(i,j)$ as

$$g_{i,j}(\boldsymbol{\theta})=\frac{e^{\theta_{i}}}{e^{\theta_{i}}+e^{\theta_{j}}},$$

we can obtain the formulation of $h$ for the BTL model. ∎

Details of Algorithm 3 and 4

  Input : the probability mass function $g$, the support set $\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})$, the incomplete knowledge $\boldsymbol{S}$ and the solution accuracy $\epsilon$.

1 Initialization:

$$\mu_{\text{min}}=0,$$

$$\mu_{\text{max}}=\mu_{\infty}=\textbf{{max}}\left\{m\cdot\|\boldsymbol{z}\|_{\infty},\sqrt{\frac{m}{2\beta}}\cdot\|\boldsymbol{z}\|_{2}\right\},$$

where $\boldsymbol{z}=[z_{1,2},\dots,z_{n,n-1}]$

$$z_{ij}=\underset{\boldsymbol{\theta}\in\textbf{{Supp}}(\rho_{\boldsymbol{\theta}^{\prime}})}{\textbf{{max}}}\ -\textbf{{log}}~{}g_{i,j}(\boldsymbol{\theta}).$$

2while $|\mu_{\text{max}}-\mu_{\text{min}}|>\epsilon\cdot\mu_{\infty}$ do

3      

$$\mu = \ \ \ \ \frac{1}{2}(\mu_{\text{min}}+\mu_{\text{max}}),$$

$$\boldsymbol{\theta}(\mu) = \ \ \ \ \textbf{{SimplexProjection}}(\boldsymbol{\theta}^{(0)},\mu),$$

$$\nabla\mathcal{H}(\mu) = \ \ \ \ \frac{1}{2}\|\boldsymbol{\theta}(\mu)-\boldsymbol{\theta}_{\mathcal{A}}\|^{2}_{2}-\beta.$$

4      if $\nabla\mathcal{H}(\mu)>0$ then

5            

$$\mu_{\text{min}}=\mu,$$

6      else

7            

$$\mu_{\text{max}}=\mu.$$

8       end if

9      

10 end while

11

12Update

$$\mu=\frac{1}{2}\big{(}\mu_{\text{min}}+\mu_{\text{max}}\big{)}.$$

13Solve the distributionally robust estimation

$$\boldsymbol{\hat{\theta}}=\textbf{{SimplexProjection}}(\boldsymbol{\theta}^{(0)},\mu)$$

Output : the distributionally robust estimation $\boldsymbol{\hat{\theta}}$.

Algorithm 3 Robust Estimation

  Input : the partial dual problem (241)

1 Initialization: the initial step size $\eta_{0}$, the maximum iteration number $T_{1}$, $\boldsymbol{U}=[n]$, $s=0$, $v=0$.

2for $t=0$ to $T_{1}-1$ do

3      

$$\boldsymbol{\theta}^{(t+1)}=\boldsymbol{\theta}^{(t)}-\eta_{t}\nabla\mathcal{L}(\boldsymbol{\theta}^{(t)},\mu).$$

4 end for

5

6while $\boldsymbol{U}\neq\varnothing$ do

7       Pick $k\in\boldsymbol{U}$ at random and separate $\boldsymbol{U}$ as

$$\boldsymbol{G}=\{\ j\in\boldsymbol{U}\ |\ \theta^{(T_{1})}_{j}\geq\theta^{(T_{1})}_{k}\ \},$$

$$\boldsymbol{L}=\{\ j\in\boldsymbol{U}\ |\ \theta^{(T_{1})}_{j}<\theta^{(T_{1})}_{k}\ \}.$$

8      Set

$$\Delta v=|\boldsymbol{G}|,\ \Delta s=\sum_{j\in\boldsymbol{G}}\hat{\theta}^{(T_{1})}_{j}.$$

9      if $(s+\Delta s)-(v+\Delta v)<1$ then

10            

$$s\leftarrow s+\Delta s,\ v\leftarrow v+\Delta v,\ \boldsymbol{U}\leftarrow\boldsymbol{L}$$

11      else

12            

$$\boldsymbol{U}\leftarrow\boldsymbol{G}/\{k\}.$$

13       end if

14      

15 end while

16

17

$$\boldsymbol{\hat{\theta}}^{(T_{1})}=\big{[}\boldsymbol{\theta}^{(T_{1})}-\gamma\cdot\boldsymbol{1}\big{]}_{+},$$

where

$$\gamma=\frac{s-1}{v}.$$

Output : $\boldsymbol{\hat{\theta}}^{(T_{1})}$.

Algorithm 4 $\textbf{{SimplexProjection}}(\boldsymbol{\theta}^{(0)},\mu)$

Definition 8 ($p$-Wasserstein distance).

Let $p\in[1,\infty]$. The $p$-Wasserstein distance between distributions $\mathbb{P},\ \mathbb{Q}\in\mathcal{P}(\boldsymbol{\Omega})$ is defined as

$$\mathcal{W}_{p}\ (\mathbb{P},\ \mathbb{Q})=\left\{\begin{array}[]{ll}\Bigg{(}\underset{\gamma\in\Gamma(\mathbb{P},\ \mathbb{Q})}{\textbf{{min}}}{\int}_{\boldsymbol{\Omega}\times\boldsymbol{\Omega}}\big{[}d(\boldsymbol{p},\ \boldsymbol{q})\big{]}^{p}\gamma\big{(}\mathrm{d}\boldsymbol{p},\ \mathrm{d}\boldsymbol{q}\big{)}\Bigg{)}^{\frac{1}{p}},&p<\infty\\[20.0pt] \underset{\gamma\in\Gamma(\mathbb{P},\ \mathbb{Q})}{\textbf{{inf}}}\ \underset{\vphantom{\gamma\in\Gamma(\mathbb{P},\ \mathbb{Q})}\boldsymbol{\Omega}\times\boldsymbol{\Omega}}{\gamma\textnormal{-}\textbf{{ess sup}}}\ d\big{(}\boldsymbol{p},\ \boldsymbol{q}\big{)},&p=\infty,\end{array}\right.$$

where $\Gamma(\mathbb{P},\ \mathbb{Q})$ denotes the set of all Borel probability distributions on $\boldsymbol{\Omega}\times\boldsymbol{\Omega}$ with marginal distributions $\mathbb{P}$ and $\mathbb{Q}$, $d:\boldsymbol{\Omega}\times\boldsymbol{\Omega}\rightarrow\mathbb{R}_{+}$ is a nonnegative function, and $\gamma\textnormal{-}\textbf{{ess sup}}$ expresses the essential supremum of $d(\cdot,\ \cdot)$ with respect to the measure $\gamma$.

The Wasserstein distance arises in the problem of optimal transport [40, 49]: for any coupling $\gamma\in\Gamma(\mathbb{P},\ \mathbb{Q})$, the conditional distribution $\gamma_{\boldsymbol{w}|\boldsymbol{w}^{\prime}}$ can be viewed as a randomized overhead for ‘transporting’ a unit quantity of some material from a random location $\boldsymbol{w}\sim\mathbb{P}$ to another location $\boldsymbol{w}^{\prime}\sim\mathbb{Q}$. If the cost of transportation from $\boldsymbol{w}\in\boldsymbol{\Omega}$ to $\boldsymbol{w}^{\prime}\in\boldsymbol{\Omega}$ is given by $[d(\boldsymbol{w},\boldsymbol{w}^{\prime})]^{p}$, $\mathcal{W}_{p}\ (\mathbb{P},\ \mathbb{Q})$ will be the minimum expected transport cost [42].

Introducing a dual variable $\mu$ for the constraint $1/2\|\boldsymbol{\theta}-\boldsymbol{\theta}_{\mathcal{A}}\|_{2}^{2}\leq\beta$, we solve the original optimization problem (62) by maximizing its dual problem. Furthermore, the strong duality stands for (62) as the Slater condition is satisfied by $\boldsymbol{\theta}_{\mathcal{A}}$. Then the standard min-max swap will be performed as

$$\underset{\mu\geq 0}{\ \ \textbf{{max}}\phantom{f}}\mathcal{H}(\mu) := \ \ \underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{inf}}}\ \Big{\{}\ \mathcal{L}_{1}(\boldsymbol{\theta},\mu)\ \Big{|}\ \boldsymbol{\theta}\succeq 0,\ \boldsymbol{1}^{\top}\boldsymbol{\theta}=1\ \Big{\}}$$ (241)

$$= \ \ \underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{inf}}}\Big{\{}\frac{\mu}{2}\big{\|}\boldsymbol{\theta}-\boldsymbol{\theta}_{\mathcal{A}}\big{\|}^{2}_{2}-\mu\beta+h(\boldsymbol{\theta})\ \Big{|}\ \boldsymbol{\theta}\succeq 0,\boldsymbol{1}^{\top}\boldsymbol{\theta}=1\Big{\}}.$$

By Corollary 4.4.5 of Chapter VI in [27], we know that

$$\nabla\mathcal{H}(\mu) = \ \ \nabla_{\mu}\ \mathcal{L}_{1}(\boldsymbol{\theta}(\mu),\mu)$$ (242)

$$= \ \ \nabla_{\mu}\ \Bigg{\{}\frac{\mu}{2}\big{\|}\boldsymbol{\theta}(\mu)-\boldsymbol{\theta}_{\mathcal{A}}\big{\|}^{2}_{2}-\mu\beta+h(\boldsymbol{\theta}(\mu))\Bigg{\}}$$

$$= \ \ \frac{1}{2}\big{\|}\boldsymbol{\theta}(\mu)-\boldsymbol{\theta}_{\mathcal{A}}\big{\|}^{2}_{2}-\beta.$$

If $\nabla\mathcal{H}(\mu)$ can be solved efficiently, it is possible to binary search with $\log(1/\varepsilon)$ iterations and the accuracy $\varepsilon$ for the $\mu^{*}$ as $\|\nabla\mathcal{H}(\mu^{*})\|\leq\varepsilon$.

Given $\mu^{*}$ and denote the probabilistic simplex as

$$\Delta=\{\ \boldsymbol{\theta}\in\mathbb{R}^{n}\ |\ \boldsymbol{\theta}\succeq 0,\ \boldsymbol{1}^{\top}\boldsymbol{\theta}=1\ \},$$ (243)

the corresponding $\boldsymbol{\theta}(\lambda^{*})$ can be obtained by the projected sub-gradient descent which minimizes the objective $\mathcal{L}$ with the following sequence $\{\boldsymbol{\theta}^{(t)}\}_{t=1}^{T}$,

$$\boldsymbol{\theta}^{(t+1)}=\textbf{Proj}_{\Delta}\big{(}\boldsymbol{\theta}^{(t)}-\eta_{t}\nabla\mathcal{L}(\boldsymbol{\theta}^{(t)},\mu^{*})\big{)},$$ (244)

where $\nabla\mathcal{L}(\boldsymbol{\theta}^{(t)},\mu^{*})$ is the (sub)gradient of $\mathcal{L}(\boldsymbol{\theta},\mu^{*})$ at $\boldsymbol{\theta}^{(t)}$, $\eta_{t}$ is the positive step size and $\textbf{Proj}_{\Delta}(\boldsymbol{\varphi})$ is the Euclidean projection of $\boldsymbol{\varphi}$ onto $\Delta$, i.e. the solution of

$$\underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{min}}}\ \frac{1}{2}\|\boldsymbol{\varphi}-\boldsymbol{\theta}\|^{2}_{2},\ \textit{s.t.}\ \boldsymbol{\theta}\succeq 0,\ \boldsymbol{1}^{\top}\boldsymbol{\theta}=1.$$ (245)

The projection $\boldsymbol{\theta}^{(t+1)}$ has the form

$$\boldsymbol{\theta}^{(t+1)} = \ \ [\boldsymbol{\varphi}^{(t+1)}-\gamma\boldsymbol{1}]_{+}$$ (246)

where $[\cdot]_{+}=\textbf{{max}}\{\cdot,0\}$ and $\gamma$ holds

$$\gamma=\frac{1}{\rho}\left(\sum_{i=1}^{\rho}\varphi^{(t+1)}_{i}-1\right),$$ (247)

$$\rho=\textbf{{max}}\left\{j\in[n]\ \left|\ \psi_{j}-\frac{1}{j}\left(\sum_{r=1}^{j}\psi_{r}-1\right)>0\right.\right\},$$ (248)

where $\boldsymbol{\psi}$ is the sorted version of $\boldsymbol{\varphi}^{(t+1)}$ with descent order.

Details of Algorithm 5

  Input : The MLE estimator $\boldsymbol{\hat{\theta}}$ and the total number of iterations $L$.

1

2Initialization: A starting point $\boldsymbol{\lambda}^{0}$ and a constant $c_{0}>0$.

3for $l=1$ to $L$ do

4       Solve the maximizing sub-problem

$$\boldsymbol{\theta}(\boldsymbol{\lambda}^{l-1})\in\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}:r(\boldsymbol{\theta})\neq r(\boldsymbol{\hat{\theta}})}{\textbf{{arg max}}}\ -g(\boldsymbol{\theta};\boldsymbol{\lambda}^{l-1},\boldsymbol{\hat{\theta}}),$$

where

$$g(\boldsymbol{\theta};\boldsymbol{\lambda}^{l-1},\boldsymbol{\hat{\theta}})=\underset{(i,j)\in\mathcal{A}}{\sum}\ \lambda^{l-1}_{i,j}D^{i,j}(\boldsymbol{\hat{\theta}}\|\boldsymbol{\theta}).$$

5      Calculate the sub-gradient $d(\boldsymbol{\lambda}^{l-1})$

$$d(\boldsymbol{\lambda}^{l-1})=\big{(}d(\boldsymbol{\lambda}^{l-1})_{1,2},\dots,d(\boldsymbol{\lambda}^{l-1})_{n,n-1}\big{)},$$

where

$$d(\boldsymbol{\lambda}^{l-1})_{i,j}=-D^{i,j}(\boldsymbol{\hat{\theta}}\|\boldsymbol{\theta}(\boldsymbol{\lambda}^{l-1})).$$

6      Solve the minimizing sub-problem

$$\boldsymbol{\lambda}^{l}\in\underset{\boldsymbol{\lambda}\in\Delta}{\textbf{{arg min}}}\ \eta_{l}\langle d(\boldsymbol{\lambda}^{l-1}),\boldsymbol{\lambda}\rangle+D(\boldsymbol{\lambda}\|\boldsymbol{\lambda}^{l-1}),$$

where $\eta_{l}=c_{0}/\sqrt{~{}l~{}}$ and

$$D(\boldsymbol{\lambda}\|\boldsymbol{\lambda}^{l-1})=\underset{(i,j)\in\mathcal{A}}{\sum}\lambda_{i,j}\log\frac{\lambda_{i,j}}{\lambda^{l-1}_{i,j}}$$

7 end for

8The categorical distribution is obtained through averaging the sequence $\{\boldsymbol{\lambda}^{1},\dots,\boldsymbol{\lambda}^{L}\}$ like

$$\boldsymbol{\hat{\lambda}}=\frac{1}{L}\sum_{l=1}^{L}\boldsymbol{\lambda}^{l}.$$

Output : The categorical distribution $\boldsymbol{\hat{\lambda}}$.

Algorithm 5 $\textbf{{MirrorDescent}}(\boldsymbol{S},f,\boldsymbol{\Theta}^{\beta}_{\mathcal{A}},\boldsymbol{\theta}^{(m)})$

$$\boldsymbol{\hat{\lambda}}^{(m)}=\underset{\phantom{\boldsymbol{\hat{\theta}}^{(m)}}\boldsymbol{\lambda}\in\Delta\phantom{\boldsymbol{\hat{\theta}}^{(m)}}}{\textbf{{arg max}}}\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}:\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\hat{\theta}}^{(m)})}{\ \textbf{{min}\phantom{g}}}\ g(\boldsymbol{\theta},\boldsymbol{\lambda};\boldsymbol{\hat{\theta}}^{(m)}),$$

where $\boldsymbol{\lambda}=(\lambda_{1,2},\dots,\lambda_{n,n-1})$,

$$g(\boldsymbol{\theta},\boldsymbol{\lambda};\boldsymbol{\hat{\theta}}^{(m)})=\underset{(i,j)\in\mathcal{A}}{\sum}\ \lambda_{i,j}D^{i,j}(\boldsymbol{\hat{\theta}}^{(m)}\|\boldsymbol{\theta}),$$

and $D^{i,j}(\boldsymbol{\hat{\theta}}\|\boldsymbol{\theta})$ is the Kullback-Leibler (KL) divergence from $f(\boldsymbol{\theta};(i,j),y)$ to $f(\boldsymbol{\hat{\theta}};(i,j),y)$ as

$$D^{i,j}(\boldsymbol{\hat{\theta}}\|\boldsymbol{\theta})=\underset{y\in\{-1,1\}}{\sum}f(\boldsymbol{\hat{\theta}};(i,j),y)\log\frac{f(\boldsymbol{\hat{\theta}};(i,j),y)}{f(\boldsymbol{\theta};(i,j),y)}.$$

$$\boldsymbol{\theta}(\boldsymbol{\lambda}^{(l-1)})\in\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}:r(\boldsymbol{\theta})\neq r(\boldsymbol{\hat{\theta}})}{\textbf{{arg max}}}\ -g(\boldsymbol{\theta};\boldsymbol{\lambda}^{(l-1)},\boldsymbol{\hat{\theta}}),$$

$$\boldsymbol{\theta}(\boldsymbol{\lambda}^{(l-1)})\in\underset{\boldsymbol{\theta}\in\boldsymbol{\Theta}^{\beta}_{\mathcal{A}}:\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\hat{\theta}})}{\textbf{{arg min}}}\ g(\boldsymbol{\theta};\boldsymbol{\lambda}^{(l-1)},\boldsymbol{\hat{\theta}}),$$

$$\ \ \underset{\delta\geq 0}{\textbf{{max}}}\ \ \mathcal{G}(\delta)$$

$$:= \ \ \underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{inf}}}\ \Big{\{}\ \mathcal{L}_{2}(\boldsymbol{\theta},\delta)\ \Big{|}\ \boldsymbol{\theta}\succeq 0,\ \boldsymbol{1}^{\top}\boldsymbol{\theta}=1,\boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\hat{\theta}})\ \Big{\}}$$

$$= \ \underset{\boldsymbol{\theta}\in\mathbb{R}^{n}}{\textbf{{inf}}}\ \left\{\ \begin{matrix}\displaystyle\frac{\delta}{2}\big{\|}\boldsymbol{\theta}-\boldsymbol{\theta}_{\mathcal{A}}\big{\|}^{2}_{2}-\delta\beta\\[5.0pt] +\ g(\boldsymbol{\theta};\boldsymbol{\lambda}^{(l-1)},\boldsymbol{\hat{\theta}})\end{matrix}\ \left|\ \begin{matrix}\boldsymbol{\theta}\succeq 0,\boldsymbol{1}^{\top}\boldsymbol{\theta}=1,\\[10.0pt] \boldsymbol{\pi}(\boldsymbol{\theta})\neq\boldsymbol{\pi}(\boldsymbol{\hat{\theta}})\end{matrix}\right.\right\}$$

Balance the choice probability of the selection rule

$$\lambda^{(m)}_{i,j}=p\cdot\frac{2}{n(n-1)}+(1-p)\hat{\lambda}^{(m)}_{i,j},\ i,j\in[n],\ i\neq j.$$

is chosen such that $\sum_{i}\theta^{(t+1)}_{i}=1$. With Lemma 2 of [21], we know that $\gamma$ plays the same role as the unique index $i$ which

$$\sum_{j=1}^{i}\Big{(}\varphi^{(t+1)}_{j}-\varphi^{(t+1)}_{i}\Big{)}<1$$ (249)

$$\sum_{j=1}^{i+1}\Big{(}\varphi^{(t+1)}_{j}-\varphi^{(t+1)}_{i+1}\Big{)}\geq 1,$$

or $i=n$ if there does not exist any index satisfies (249).

The projection $\boldsymbol{\theta}(\lambda)=[\theta_{i}(\lambda),\dots,\theta_{n}(\lambda)]$ has the form

$$\theta_{i}(\lambda)=\left[\tilde{\theta}_{i}-\frac{1}{\lambda}\frac{\partial F(\boldsymbol{\theta})}{\partial\theta_{i}}\right]$$ (250)

finding the the Euclidean projection of vector $\boldsymbol{v}(\lambda)=[v_{1},\dots,v_{n}]\in\mathbb{R}^{n}$

$$v_{i}=$$ (251)

onto the probabilistic simplex. Then the projection $\boldsymbol{\theta}(\lambda)$ has the form $\theta_{i}(\lambda)=(v_{i}-\eta)_{+}$ for some $\eta\in\mathbb{R}$ where $\eta$ is chosen such that $\boldsymbol{\theta}(\lambda)$ will satisfy the probabilistic unit sphere constraint. It is equivalent to find the unique index $i$ such that

$$\sum_{j=1}^{i}(v_{j}-v_{i})<1,\ \text{and}\ \sum_{j=1}^{i+1}(v_{j}-v_{i+1})\geq 1,$$ (252)

and $i=n$ if there does not exist such an index. By (252), we know that

$$v_{j}-\eta\left\{\begin{array}[]{ll}\geq 0,&j\leq i,\\ \leq 0,&j\geq i\end{array}\right.$$ (253)