Stability of Weighted Majority Voting under Estimated Weights

Abstract.

Weighted Majority Voting (WMV) is a well-known decision making rule. The weights of sources are determined by the probabilities that sources provide accurate information (trustworthiness). However, in reality, the trustworthiness is usually not a known quantity to the decision maker – they have to rely on an estimate called trust. An algorithm that computes trust is called unbiased when it has the property that it does not systematically overestimate or underestimate the trustworthiness. To formally analyze the uncertainty to the decision process brought by such unbiased trust values, we introduce and analyze two important properties of WMV: Stability of Correctness and Stability of Optimality. Stability of Correctness measures the difference between the decision accuracy that the decision maker believes he can achieve and the accuracy he actually achieves. We prove Stability of Correctness absolutely holds for WMV – the difference is $0$. Stability of Optimality measures the difference between the actual accuracy of decisions made using trust values, and those made using trustworthiness values. We find a relatively tight upper bound on the Stability of Optimality, meaning that, although using (unbiased) trust values is suboptimal compared to using the true trustworthiness values, the difference is small. Meanwhile, a counter-intuitive observation is that while distributions of trustworthiness influence the Stability of Optimality, the number of sources barely influences it. We also provide an overview of how sensitive decision accuracy is to the changes in trust and trustworthiness.

1. Introduction

Crowd wisdom has been playing a fundamental role in hel** make better decisions in many scenarios, e.g., hiring workers for labeling tasks in crowdsourcing Luo (2023), aggregating classifiers for prediction in ensemble learning Kotary et al. (2023), asking for opinions of reliability in online rating systems Carbo and Molina (2023); Ge et al. (2023), etc. Decisions are derived based on aggregating the information or feedback from a collection of sources, the quality of which can be variable. It can be inaccurate due to lack of expertise, mistakes or malice, e.g., low-quality labeling for machine learning, fake ratings introduced by sellers to promote their reputation, etc.

Among the aggregation mechanisms, Weighted Majority Voting has long been a popular one. Basically, each source supports an option and is assigned a weight. WMV chooses the feedback option that is supported by sources with the maximal total weight. WMV has been seeing its use in a variety of domains ranging from voting Nitzan and Paroush (1982), crowdsourcing Dawid and Skene (1979), classification Littlestone and Warmuth (1994) to trust systems Yu et al. (2004) and even distributed systems Tong and Kain (1991). In different contexts, the weight of a source can mean differently. For instance, in determining a collective choice that is widely acceptable to individuals with diverse preference Sen (1977), WMV is used for preference aggregation and the weight means the importance of an individual. We are more interested in the scenarios where there is a notion of correctness (or accuracy) of decisions, and the weight of a source depends on how trustworthy it is in providing the feedback that corresponds to the correct decision, denoted as trustworthiness, which is usually modelled as a probability value. The examples include the aggregation of the crowd-sourced labels Li and Yu (2014), the crowd-sensed navigation data James (2020), or the outputs of the multiple classifiers Manino et al. (2019b), etc.

While WMV is proven to be optimal when source trustworthiness is given Nitzan and Paroush (1982), in practice, decision makers have to resort to an estimation or a belief (denoted as trust) that may not equal to the actual values of trustworthiness. Deviation in estimation may decrease decision accuracy. There exists plenty of effort to improve the estimation of source trustworthiness by learning from historical data (e.g., direct observations or indirect evidences), with a principle that more data increases the confidence in the estimation Berend and Kontorovich (2013); Wu and Yang (2016). Several approaches even treat the belief about source trustworthiness as its actual values Mazzetto et al. (2021); Maystre et al. (2021). However, no algorithm always produces perfect trust values. It is worth studying how the quality of the trust values impacts the decision quality of WMV. Is WMV able to maintain a tolerant level of decision incorrectness with the inaccuracy in the estimation bounded, meaning having certain levels of stability w.r.t the inaccurate estimation?

In this paper, we propose a formal analysis of the stability properties of WMV. Firstly, we study how sensitive the decision accuracy is to the changes in source trust and trustworthiness, with both the arguments taking fixed values. We find that unsurprisingly, decision accuracy decreases with the increasing deviation from trust to trustworthiness, and a sufficiently small deviation barely influences the accuracy. Besides, compared with overestimating, underestimating trustworthiness is usually less harmful to the decision accuracy. Secondly, we study the influence of trust and trustworthiness in a statistical way. Considering that a decision maker may sometimes overestimate source trustworthiness while sometimes underestimate it, the expectation remains correct – unbiased estimation. We define two types of stability based on such unbiased estimation: Stability of Correctness and Stability of Optimality. Stability of Correctness reasons whether the decision accuracy a decision maker believes he achieves (i.e., the accuracy he computes with trust) equals what he actually achieves (i.e., the accuracy computed with trustworthiness). We prove that whatever distribution source trustworthiness follows, as long as the estimation is unbiased, a decision maker gets the accuracy as if the trustworthiness is known – absolute stability. This means that the shape and variance of trustworthiness are irrelevant to the Stability of Correctness.

Stability of Optimality reasons whether the decisions made based on unbiased trust values are as good as those made based on trustworthiness. Considering trustworthiness is usually unknown, Stability of Optimality measures the gap between the practical situation where the decision maker decides with trust, and where (magically) he has access to the actual trustworthiness. We prove that Stability of Optimality does not hold for WMV, but the degradation in decision accuracy caused by the incorrect but (averagely) unbiased trust is relatively tightly bounded. That is, decision accuracy with unbiased trust will not be too far off the theoretically determined value. Moreover, unlike Stability of Correctness, the distribution of trustworthiness influences the upper bound of that accuracy gap, and also determines how well the accuracy can be in the ideal situation, namely where trustworthiness is given. Last but not least, while it may usually be perceived that more sources improve accuracy, we observe counterintuitively that source number influences little on the accuracy gap.

The rest of this paper is organized as follows. In Section 2 the related work is presented. In Section 3 we introduce a formal framework to study WMV decision rule. In Section 4 we present how trust and trustworthiness influence the decision accuracy of WMV. In Section 5 we analyze the two types of stability. The numerical analysis is also performed where needed to demonstrate theories.

2. Related Work

The Weighted Majority Voting rule has been studied in several domains, e.g., decision theory, voting theory, management science, and receiving various applications. We focus on the scenarios where the weight of a source or “voter” is determined by how trustworthy it is in suggesting the correct decision. Some approaches utilizing WMV assume source trustworthiness is given Nitzan and Paroush (1982); Berend and Kontorovich (2015), although in practice it is usually unknown. Plenty of work focuses on modeling and learning source trustworthiness from observation and interaction history Zeynalvand et al. (2021); Wu et al. (2023); Ge et al. (2023). Some researchers model trust as a probability value. To get trust, they either rely on frequency estimation by counting the times of making the right decisions Berend and Kontorovich (2013), or solving an optimization problem based on their models by minimizing the decision error rate Rekatsinas et al. (2017) or maximizing the likelihood Dong et al. (2015); Manino et al. (2019a); Meir et al. (2023). Moreover, model-checking-based methods are also applied in quantifying the probability of trust on individual agents, representing the agent’s own beliefs Drawel et al. (2020); Bentahar et al. (2022); Drawel et al. (2022); Telang et al. (2023). Besides, trust also can be modeled as a random variable. Bayesian models have also been proposed and applied to this problem by Raykar et al. (2010); Sardana et al. (2018); Guo (2023), combining the prior knowledge and the observations to infer the trustworthiness. Expectation Maximization-based methods are also proposed to estimate source trustworthiness and the correct decision at the same time, via iterative updating Dawid and Skene (1979); Zhang et al. (2016).

Such learned trust is sometimes treated as an estimation of the source trustworthiness with the deviation considered Gao et al. (2016); Wu et al. (2021a), while sometimes treated equivalently as trustworthiness, namely as the probability of a source suggesting correctly and is further used to evaluate decision accuracy  Littlestone and Warmuth (1994); Guan et al. (2018); Martín-Morató and Mesaros (2023). However, trust essentially represents the belief of a decision maker about the source quality, which may deviate from the actual probability. And he may not gain the claimed decision accuracy based on trust.

Besides the efforts in modeling and learning trustworthiness, there exists work that theoretically analyzes how trustworthiness and trust would influence decision accuracy, which is most relevant to ours. Given trustworthiness, the decision accuracy of WMV is analyzed in Berend and Kontorovich (2015) without considering the learning process of trustworthiness. On the other hand, some other work takes the learning process into consideration. To measure the estimation quality, the decision accuracy bounds for learning algorithms have been proposed through PAC techniques in Lacasse et al. (2006); Germain et al. (2015); Wu et al. (2021b). Considering trust is derived from finite samples, some researchers then study precise characterizations of the relationship between the decision accuracy and the sample size in Gao et al. (2016). More recently, tighter bounds for decision accuracy under arbitrary estimation are provided, ignoring particular assumptions for trustworthiness  Manino et al. (2019b). Unfortunately, none of them have analyzed the relationship between the estimate error and the decision accuracy of WMV in a quantitative way.

3. Preliminaries

In this section, we outline a formal framework to support our study of the stability of Weighted Majority Voting decision rule. Note that the capital letters represent random variables, and the lower cases represent non-random variables. The bold letters represent a vector of multiple variables, and the non-bold letters represent single variables.

Consider a decision-making scenario, a decision maker is faced with multiple possible decisions $\mathcal{O}=\{o_{1},\dots,o_{K}\}$ and only one of them is correct. The random variable $O$ determines which of the options is actually correct, e.g., $O{=}o_{1}$ if $o_{1}$ is the correct decision. The decision maker receives feedback from a set of sources, $\mathcal{S}=\{s_{1},..,s_{n}\}$. The random variable $F_{i}$, with $f_{i}$ denoting an outcome, represents the feedback of source $s_{i}$. The feedback may or may not correspond to the correct decision. For WMV, we assume a one-to-one correspondence between the feedback that suggests the correct decision and the correct decision itself, and denote $F_{i}=O$ iff $f_{i}$ suggests correctly, $F_{i}\in\mathcal{O}$. Feedback of all the sources is represented by random variable $\bm{F}:\bm{F}=(F_{1},\dots,F_{n})$, with $\bm{f}{:}\bm{f}{=}(f_{1},\dots,f_{n})$ denoting an outcome, and its sample space is defined as $\mathcal{F}{:}\bm{f}{\in}\mathcal{F}$. A decision mechanism is a function: $\mathcal{D}:\mathcal{F}{\to}\mathcal{O}$. The quantity that the decision maker wants to maximize is the probability of making the correct decisions (which we shorthand as decision accuracy or decision correctness throughout the paper): $\mathbb{P}(\mathcal{D}(\bm{F})=O)$.

we define $\Upsilon_{i}$ as a $\{-1,1\}$-indicator random variable of whether source $s_{i}$ suggests the correct decision and $\upsilon_{i}$ as one of its outcome: $\Upsilon_{i}{=}1$ if $F_{i}{=}O$ and $\Upsilon_{i}{=}-1$ if $F_{i}{\neq}O$. For the indicator variables of all the sources i.e., $\bm{\Upsilon}:\bm{\Upsilon}=(\Upsilon_{1},\dots,\Upsilon_{n})$, one of its samples is an indicator vector i.e., $\bm{\upsilon}{=}(\upsilon_{1},\dots,\upsilon_{n}),\bm{\upsilon}\in\mathcal{T}$. $\mathcal{T}$ denote the sample space of $\bm{\Upsilon}$. Let $-\bm{\upsilon}{=}(-\upsilon_{1},\dots,-\upsilon_{n})$ denote the opposite indicator vector of $\bm{\upsilon}$ where source indicators are flipped. The set of all the possible feedback under $\bm{\upsilon}$ is denoted as $\mathcal{F}_{\bm{\upsilon}}$.

We use the following running example in this section to demonstrate the relevant concepts.

Example 1.

There are three sources $\mathcal{S}{=}\{s_{1},s_{2},s_{3}\}$. If $\mathcal{O}=\{A,B\}$, $O{=}B$ and the indicator vector $\bm{\upsilon}=(1,1,-1)$, then $\bm{f}=(B,B,A)$ and $\mathcal{F}_{\bm{\upsilon}}=\{(B,B,A),(A,A,B)\}$.

When decision “correctness” is a concern, Weighted Majority Voting usually considers how probable each source suggests the correct decision. For source $i$, let $\mathbb{P}(F_{i}{=}O)=p_{i}$ and $\bm{p}=(p_{1},\dots,p_{n})$. Hence $\mathbb{P}(\Upsilon_{i}=1)=p_{i}$. We refer to $p_{i}$ as the trustworthiness of source $s_{i}$. In practice, the trustworthiness of a source is usually unknown to a decision maker. And an estimation is used, denoted as $\hat{p}_{i}$, with $\hat{\bm{p}}=(\hat{p}_{1},\dots\hat{p}_{n})$. We call the value $\hat{p}_{i}$ trust, which represents the subjective estimation or belief of the decision maker regarding how probable a source suggests correctly. There exist multiple ways to compute $\hat{p}_{i}$, e.g., counting the frequency of making correct decisions, or Bayesian learning methods based on prior interaction data. In the literature, the trustworthiness of a source can have different meanings, e.g. honesty of an agent in a rating system Muller et al. (2020), competency of a voter Condorcet (1785), reliability of a worker in crowdsourcing Dawid and Skene (1979), correctness of a sensor in crowdsensing Moslem et al. (2012), etc. Whatever the meanings, $p_{i}$ represents an intrinsic quality or the fact that how probable the source reports correctly, while $\hat{p}_{i}$ represents how the decision maker thinks of or estimates that probability Walter (2008). We assume sources independently provide feedback, hence $\mathbb{P}(\bm{\Upsilon}=\bm{\upsilon})=\prod_{i:\upsilon_{i}{=}1}p_{i}\cdot\prod_{i:\upsilon_{i}{=}-1}(1{-}p_{i})$.

In Example 1, suppose $\bm{p}=(0.6,0.6,0.7)$, the estimation of $\bm{p}$ by a decision maker may be inaccurate: $\hat{\bm{p}}=(0.6,0.7,0.8)$.

Below, we introduce the Weighted Majority Voting (WMV) decision scheme. It can be treated as an extension of the more commonly known Majority Voting decision scheme. The difference is that Majority Voting treats sources without distinguishing, while WMV assigns sources different weights. The weight of a source is usually determined by how trustworthy its feedback is. Formally:

Definition 0 (Weighted Majority Voting $\mathcal{D}_{W}$).

Given a set of $n$ sources $\mathcal{S}$, their trustworthiness $\bm{p}$ and independent feedback $\bm{f}$, $\mathcal{D}_{W}$ makes decisions via the function Muller et al. (2020); Nitzan and Paroush (1982):

$$\mathcal{D}_{W}(\bm{f})=\text{argmax}_{o\in\mathcal{O}}\left(\sum_{i:f_{i}=o}{w_{i}}\right)$$ (1)

where $f_{i}\in\mathcal{O}$, $w_{i}=\log(\nicefrac{{p_{i}}}{{1-p_{i}}})$ with $p_{i}\geq 0.5$.

To give an instance, consider Example 1, suppose $\bm{p}=(0.6,0.6,0.9)$, $\mathcal{O}=(A,B)$, $O=B$ and $\bm{\upsilon}=(1,1,-1)$, then $\bm{f}=(B,B,A)$. $w_{1}\approx 0.18,w_{2}\approx 0.18,w_{3}\approx 0.60$. since $w_{1}+w_{2}<w_{3}$, $\mathcal{D}_{W}(\bm{f})=A$.

Here $p_{i}\geq 0.5$ and the log weight function are well-known for classical WMV in the literature Nitzan and Paroush (1982); Grofman et al. (1983), where trust and trustworthiness are not distinguished. The assumption $p_{i}\geq 0.5$ means that sources with $p_{i}<0.5$ are ignored. For a source with $p_{i}<0.5$, a decision maker may assign negative weight to its feedback. Or he can just simply reverse the vote of the source (e.g., replacing the reported option A with C). But if either the operation is realized by the malicious sources, they can push the decision to a wrong one by reporting correctly, purposely reducing the chance of correct option being selected. Therefore, it is in the interest of the decision maker to ignore such sources.

It has been shown in the literature that the decision accuracy of WMV is determined by the indicator vectors where it always decides correctly (an example is where all sources report correctly). For such indicator vectors, whether a decision is correct is not influenced by the feedback of the sources that suggest incorrectly. To give an opposite example, consider Example 1. Suppose $\mathcal{O}=\{A,B,C\}$, $\bm{p}=(0.70,0.65,0.65)$ and $\bm{\upsilon}=(1,-1,-1)$ (only $s_{1}$ reports correctly). We get $(w_{1},w_{2},w_{3})\approx(0.37,0.27,0.27)$. Both the feedback $\bm{f}=(A,B,C)$ and $\bm{f}^{\prime}=(A,C,C)$ are possible under $\bm{\upsilon}$ (both belong to $\mathcal{F}_{\bm{\upsilon}}$. However, WMV decides correctly by choosing $A$ under $\bm{f}$ and decides incorrectly by choosing $C$ under $\bm{f}^{\prime}$. Whether WMV decides correctly is influenced by what incorrect feedback is. Given the same $\bm{p}$, for $\bm{\upsilon}^{\prime}=(1,1,-1)$, it can be seen that whatever $s_{3}$ reports, WMV can always decides correctly by trusting $s_{1},s_{2}$.

Let $\mathbb{D}_{W}(\bm{p})$ denote the set of all the indicator vectors where WMV always decides correctly when using $\bm{p}$, namely $\mathbb{D}_{W}(\bm{p})=\{\bm{\upsilon}|\mathcal{D}_{W}(\bm{f})=O,\bm{f}{\in}\mathcal{F}_{\bm{\upsilon}}\}$. It has been proven that $\mathbb{D}_{W}(\bm{p})=\{\bm{\upsilon}|\mathbb{P}(\bm{\upsilon})\geq\mathbb{P}(-\bm{\upsilon})\}$ and the decision accuracy of $\mathcal{D}_{W}$ is (Refer to  Nitzan and Paroush (1982); Berend and Kontorovich (2015); Muller et al. (2020)):

$$\mathbb{P}(\mathcal{D}_{W}(\bm{F})=O) =\!\!\sum_{\bm{\upsilon}:\bm{\upsilon}\in\mathbb{D}_{W}(\bm{p})}\mathbb{P}(\bm{\upsilon})$$ (2)

$$=\!\!\sum_{\bm{\upsilon}:\mathbb{P}(\bm{\upsilon})\geq\mathbb{P}(-\bm{\upsilon})}\!\!\left(\prod_{i:\upsilon_{i}{=}1}\!\!p_{i}\cdot\!\!\!\prod_{i:\upsilon_{i}{=}-1}\!\!(1{-}p_{i})\right)$$

Equation 2 indicates that the accuracy of WMV is determined by the probabilities of indicator vectors, which depend on the trustworthiness values of the sources.

In Example 1, if $\bm{p}=(0.6,0.6,0.9)$, then $\mathbb{D}_{W}(\bm{p})=\{(1,1,1),\\ (-1,1,1),(1,-1,1),(-1,-1,1)\}$ and the decision accuracy is 0.9 (i.e., a.l.a source $s_{3}$ reports correctly).

WMV has been proved to be optimal when trustworthiness $\bm{p}$ and the log weight function are used for decision making and the sources are independent in providing feedback Nitzan and Paroush (1982).

In practice, when trustworthiness is unknown, the weight assigned to each source depends on the trust, that is, $w_{i}=\log(\nicefrac{{\hat{p}_{i}}}{{1-\hat{p}_{i}}})$. Besides, the decision maker computes the probabilities of indicator vectors with trust values, which we use the subscript $\mathbb{P}_{\hat{\bm{p}}}$ to distinguish from their actual probabilities: $\mathbb{P}_{\hat{\bm{p}}}(\bm{\Upsilon}=\bm{\upsilon})=\prod_{i:\upsilon_{i}{=}1}\hat{p}_{i}\cdot\prod_{i:\upsilon_{i}{=}-1}(1{-}\hat{p}_{i})$. With $\mathbb{P}(\bm{\upsilon})$ replaced by $\mathbb{P}_{\hat{\bm{p}}}(\bm{\upsilon})$, the decisions would always be correct for those indicator vectors which the decision maker thinks are more probable than their opposite, namely $\mathbb{D}_{W}(\hat{\bm{p}})=\{\bm{\upsilon}|\mathbb{P}_{\hat{\bm{p}}}(\bm{\upsilon})\geq\mathbb{P}_{\hat{\bm{p}}}(-\bm{\upsilon})\}$. As a result, $\mathbb{D}_{W}(\hat{\bm{p}})$ and $\mathbb{D}_{W}(\bm{p})$ may be different. In Example 1, if $\bm{p}=(0.6,0.6,0.9)$ and $\hat{\bm{p}}=(0.8,0.6,0.8)$, then $(-1,-1,1)\in\mathbb{D}_{W}(\bm{p})$ while its opposite indicator vector $(1,1,-1)\in\mathbb{D}_{W}(\hat{\bm{p}})$. This may result in different decision accuracy. We introduce $\omega(\hat{\bm{p}},\bm{p})$ to distinguish:

$$\mathbb{P}(\mathcal{D}_{W}(\bm{F})=O) \triangleq\omega(\hat{\bm{p}},\bm{p})=\sum_{\bm{\upsilon}\in\mathbb{D}_{W}(\hat{\bm{p}})}\mathbb{P}(\bm{\upsilon})$$ (3)

The first parameter of the function $\omega()$ represents the value used for decision making, and the second parameter represents the value used to compute the probability of deciding correctly. For $\omega(\hat{\bm{p}},\bm{p})$, decisions are made using trust values $\hat{\bm{p}}$, while the decision accuracy that the decision maker actually obtains still depends on source trustworthiness, which challenges the optimality of WMV.

Generally, both the parameters of $\omega()$ can be either trust or trustworthiness, and we assume that the parameter (either trust or trustworthiness) used for decision-making is at least 0.5. Trust values are, by definition, known to the decision-maker. Therefore, it’s reasonable to apply the assumption for trust, meaning ignoring sources with trust below 0.5. For trustworthiness, we assume it is at least 0.5 only when it is used to decide (e.g., in Section 4.1), and otherwise, its value ranges from $(0,1)$ (e.g., in Section 4.2,  4.3 and  5).

Depending on what we equip the parameters with, trust or trustworthiness, we will obtain different meanings for decision accuracy as follows. The quantity $\omega(\bm{p},\bm{p})$ denotes the “ideal” decision accuracy, where the decision maker knows and uses the trustworthiness values to decide and compute. The quantity $\omega(\hat{\bm{p}},\bm{p})$ denotes the ”practical” decision accuracy, where the decision maker decides with the trust values $\hat{\bm{p}}$, but the accuracy he actually achieves depends on trustworthiness. The quantity $\omega(\hat{\bm{p}},\hat{\bm{p}})$ denotes the “perceived” decision accuracy that the decision maker thinks he can obtain (decides and computes accuracy with trust), while the actual accuracy may not equal $\omega(\hat{\bm{p}},\hat{\bm{p}})$.

4. Parameter Sensitivity

In this section, we analyze how changes in the values of trust and trustworthiness influence the decision accuracy or the correctness of WMV. There are several ways: 1) how the decision accuracy changes when the trustworthiness and trust change simultaneously; 2) how the decision accuracy changes with trustworthiness when trust remains constant; 3) how the decision accuracy changes with trust when trustworthiness remains constant. If the changes show relatively little effect on the correctness, then we can say that WMV is not very sensitive to the parameters. Sensitivity relates to stability, the analysis in this section provides several important insights for the analysis in the next section.

We will also take numerical analysis based on the setting in the following running Example 2 to further illustrate the theories.

Example 2.

There are four sources $\mathcal{S}{=}\{s_{1},s_{2},s_{3},s_{4}\}$ and their trustworthiness values are $\bm{p}=(0.8,0.75,0.7,0.6)$ respectively.

4.1. Direct Sensitivity Analysis

Here, we analyze the case where the parameters used for making decisions and that for computing accuracy are equal. There are two different rationales for doing this, but the mathematics is identical for both. First, consider that the decision maker is given the actual trustworthiness values to make decisions. Second, consider analyzing the sensitivity of the beliefs of the decision maker. Assume the decision maker only knows trust values, and uses them to compute their belief about how probable a decision is correct.

For simplicity, we use trustworthiness everywhere, but the analysis remains unchanged when using trust instead (simply put a hat on all $p$’s and $\bm{p}$’s). Observe that if trustworthiness of only $1$ source varies, then decision accuracy would appear to be a piecewise linear non-decreasing convex function. In Figure 1(a), we depict Example 2, with each plot representing a source trustworthiness variable.

Lemma 0.

Let $f(p_{i})=\omega(\bm{p},\bm{p})$, where $p_{j}$ is constant for $j\neq i$. The function $f(p_{i})$ is a piecewise linear non-decreasing convex function.

Sketch of Proof.

The computation for correctness of a decision can be characterized as $p_{i}\cdot x+(1-p_{i})\cdot y$, where $x-y\geq 0$ and this coefficient increases with $p_{i}$ increasing. Each decision based on corresponding $\mathbb{D}_{W}(\bm{p})$ represents a non-decreasing line. ∎

Generally, if a source is more probable to be trustworthy, the decision will be better and improved faster. Figure 1(a) illustrates Lemma 2. The accuracy of WMV is determined by comparing the $\mathbb{P}(\bm{\upsilon})$ and $\mathbb{P}(-\bm{\upsilon})$ for all the indicator vectors. The relation between $\mathbb{P}(\bm{\upsilon})$ and $\mathbb{P}(-\bm{\upsilon})$ either remains or changes, depending on the value of $p_{i}$ and how much it changes. Intuitively, this results in the piece-wise characteristics of $f(p_{i})$. Moreover, we can vary trustworthiness values of multiple (or even all) sources. If we vary $k$ trustworthiness, then we get an $k$-dimensional piecewise surface. In Figure 1(b), we depict our running example with $p_{1}$ and $p_{2}$ on the two axes, and $p_{3}$ and $p_{4}$ remaining constant. The surface appears like a collection of intersecting planes, but in fact, the graph consists of surfaces described by a polynomial, rather than a linear one.

(a) One Source $p_{i}$

(b) Two sources $p_{1}$,$p_{2}$

Figure 1. Sensitivity of $\mathcal{D}_{W}$ to $\bm{p}$ when $\bm{p}=\hat{\bm{p}}$

In Lemma 2, we assume that trustworthiness of all the sources remains constant and independent. However, sources can collude to influence decisions, or one can update the trust values of multiple sources at a time. For such situations, we assume the trustworthiness values of multiple sources are consistently equal, meaning they are not independent.

Lemma 0.

In the special case of the identical sources, let $f(p)=\omega(\bm{p},\bm{p})$, where $p_{1}=\dots=p_{m}=p$, $m\leq n$ and $p_{j}$ are constant, $j>m$. The function $f(p)$ is a piecewise non-decreasing function, and it is concave (linear or strictly concave) in each segment.

Sketch of Proof.

The computation for correctness can be characterized as a summation: $\omega(\bm{p},\bm{p})=\sum_{i}g_{i}(p)$. For each $g_{i}(p)$, it meets piecewise non-decreasing property, and concave property in each segment. Thus, $\omega(\bm{p},\bm{p})$ also holds. ∎

Note that in Lemma 3, if $m{=}n$, meaning all the sources are identical, WMV becomes the classical Majority Voting decision rule and $f(p)$ becomes a concave monotonically increasing function  Boland (1989).

Figure 2(a) shows how the decision accuracy $\omega(\bm{p},\bm{p})$ changes with varying $p$ and $m$ values where there are originally $2$ sources with the same $rest\_{}p=0.7$, then $m$ identical sources join with their $p\geq 0.5$ and $n=m+2$. Given $m$ value, $f(p)$ increases piecewisely with $p$, and specifically in each segment, it is concavely increasing. Given $p$ value, $f(p)$ increases monotonically with $m$. In Figure 2(b), we fix $n=10,m=6$ and vary the trustworthiness of the $m$ identical sources $p$ and the rest sources $rest\_{}p$. This figure illustrates that even the rest sources are in minority, but the higher $rest\_{}p$ is, the more insensitive the decision to $p$ is. Besides, it demonstrates that when the trustworthiness of multiple sources updates in a particular way, the variation characteristic of the decision accuracy may be captured and described.

(a) $rest\_p=0.7$

(b) $n=10$

Figure 2. Accuracy of $\mathcal{D}_{W}$ with $m$ sources being identical

4.2. Trustworthiness Sensitivity Analysis

Next, we analyze the cases where trustworthiness and trust are not identical. The decisions are made based on the trust values $\hat{\bm{p}}$, while the probability of each indicator vector is determined by $\bm{p}$. The probability of deciding correctly is $\omega(\hat{\bm{p}},\bm{p})$. In this section, trustworthiness varies with trust value fixed. Recall Equation 3, this means that decisions remain unchanged for given feedback (as the set $\mathbb{D}_{W}(\hat{\bm{p}})$ remain unchanged), while decision accuracy $\omega(\hat{\bm{p}},\bm{p})$ may change with trustworthiness.

If only one parameter $p_{i}$ varies in $\omega(\hat{\bm{p}},\bm{p})$, then the resulting decision accuracy is a non-decreasing function, which follows trivially from the proof of Lemma 2. In fact, the line corresponds to one of the line segments from the piece-wise linear graph from the previous section as depicted in Figure 3(a). Besides, we can have multiple variables as before. The surface obtained is non-decreasing and polynomial. The surface corresponds to one of the fragments from the graph discussed in the previous section. A 2d example is depicted in Figure 3(b). The result shows that the decision accuracy has a unique continuous differentiable function, rather than a piecewise function with different functions in different segments.

(a) One Source $p_{i}$

(b) Two sources $p_{1}$,$p_{2}$

Figure 3. Sensitivity of $\bm{p}$ with $\hat{\bm{p}}$ fixed

4.3. Trust Sensitivity Analysis

Alternatively, we can take trust to be variable and trustworthiness to be fixed. Different from before, the actual probability of each indicator vector ($\mathbb{P}(\bm{\upsilon})$) now remains unchanged, but the decisions may change (as $\mathbb{P}_{\hat{\bm{p}}}(\bm{\upsilon})$ and accordingly $\mathbb{D}_{W}(\hat{\bm{p}})$ may change). Then we can analyze what happens if a trust value used for decision making moves away from the actual trustworthiness in either direction.

First, consider the case where we vary the trust value of only one source in $\omega(\hat{\bm{p}},\bm{p})$. We get a uni-modal discontinuous staircase function, which is non-decreasing when $\hat{p}_{i}<p_{i}$ and non-increasing when $\hat{p}_{i}>p_{i}$. Figure 4(a) depicts Example 2 with one variable. Second, when trust values of multiple sources are variable, the resulting surface consists of flat fragments at different heights, with an increasing height with proximity to the point $\hat{\bm{p}}=\bm{p}$. Figure 4(b) depicts Example 2 with $\hat{p}_{1}$ and $\hat{p}_{2}$ being the variables. Generally:

Lemma 0.

Let $f(\hat{\bm{p}})=\omega(\hat{\bm{p}},\bm{p})$, where the trustworthiness $\bm{p}$ is constant. The function $f(\hat{\bm{p}})$ is a discontinuous staircase function consisting of flat plateaus. Decision accuracy reaches the maximum at the plateau containing the point $\hat{\bm{p}}=\bm{p}$.

Sketch of Proof.

The probability that a decision is correct depends on $\bm{p}$, which is constant. Changing $\hat{\bm{p}}$ does not affect the probability that a decision is correct, until it reaches a point where it changes the actual decision away from the optimum. Then, there is a discontinuous step to a new platform. ∎

An insight is that the nearby points are more likely to be on the same plateau. In other words, there is an area of trust values around the trustworthiness values, meaning small estimation deviation may be unlikely to affect the accuracy. However, it is possible that a certain trustworthiness $\bm{p}$ is exactly at a border (or corner) of a plateau, meaning that even a tiny difference between trustworthiness and trust can lead to a staircase difference in correctness. The positive news is that the plateaus directly bordering the one containing $\bm{p}$ are still more often correct than the ones further away.

Besides, while both underestimation and overestimation cause wrong judgment on $\mathbb{P}(\bm{\upsilon})$ vs. $\mathbb{P}(-\bm{\upsilon})$, the numerical (Figure 4) results imply that overestimation perhaps results in the worse accuracy degradation compared with underestimation. Our intuition is that, if there is a high $p$-valued source, then that source tends to have a lot of sway on the vote, so any inaccuracies will be noticeable, whereas a low $p$-valued source tends to only matter in cases where the vote is tight, and thus any inaccuracies tend to matter less. From a micro perspective, the underlying reason might be that overestimation of a trustworthiness value makes the difference $\left|\mathbb{P}(\bm{\upsilon})-\mathbb{P}(-\bm{\upsilon})\right|$ also overestimated, while for underestimation, the difference would be underestimated. Therefore, when the estimation error is typically inevitable, it is better to underestimate trustworthiness.

(a) One Source $\hat{p}_{i}$

(b) Two sources $\hat{p}_{1}$,$\hat{p}_{2}$

Figure 4. Sensitivity of $\hat{\bm{p}}$ with $\bm{p}$ fixed

5. Stability

The results of the Parameter Sensitivity Section are unsurprising. Increasing trustworthiness typically increases correctness, and cannot decrease correctness. Hence, if a source is believed to decide correctly with probability $\hat{p}$ , while its trustworthiness $p<\hat{p}$, then the actual correctness achieved is lower than what the decision maker believes: $\omega(\hat{\bm{p}},\bm{p}){<}\omega(\hat{\bm{p}},\hat{\bm{p}})$. Vice versa when $p>\hat{p}$. Our suspicion is that these two effects cancel each other out, if the algorithm that establishes trust is not biased towards overly trusting or being suspicious on average. We call this property Stability of Correctness, and prove it absolutely holds for WMV.

A better procedure to obtain trust returns values closer to the trustworthiness values, with little variance, meaning what is believed about the sources is close to the ground truth. The quality of the procedure does not affect the Stability of Correctness at all when it is unbiased, which may initially seem counter-intuitive. However, another property captures the idea that even when it’s unbiased, poor trust values still result in worse performance of WMV, Stability of Optimality. We prove Stability of Optimality does not hold absolutely, but that drop in the performance is bounded.

Beforehand, we need to formally define what we mean by an algorithm or procedure to establish trust values, and by it being unbiased (on average).

5.1. Parameter Distributions

We introduce random variables for our parameters. For trustworthiness: $P_{i}$ is a random variable with outcome $p_{i}\in[0,1]$, and $\bm{P}$ is a joint random variable with outcome $\bm{p}=(p_{1},\dots,p_{n})$. Similarly, for trust: $\hat{P}_{i}$ is a random variable with outcome $\hat{p}_{i}\in[0,1]$, and $\bm{\hat{P}}$ is a joint random variable with outcome $\hat{\bm{p}}=(\hat{p}_{1},\dots,\hat{p}_{n})$. The uncertainty of source trustworthiness may be due to lack of behavior consistency, or experience, so the sources can not provide stable-quality feedback. On the other hand, inadequate interaction with sources or inaccurate modeling by decision maker may incur uncertain trust estimation.

Weighted Majority Voting requires a weight for each source which is determined by $\hat{p}_{i}$ (the outcome of $\hat{P}_{i}$). Practical usage of WMV, therefore, must have some algorithms to arrive at values for $\hat{\bm{p}}$. Depending on the quality of the algorithm, there is a degree of correlation between trust and trustworthiness: $\bm{\hat{P}}$ and $\bm{P}$. We consider the procedure to get the trust values $\hat{\bm{p}}$ as unbiased when the expectation of trustworthiness equals the trust value: $\mathbb{E}(\bm{P})=\hat{\bm{p}}$. Hence, if an unbiased trust value $\hat{p}_{i}$ is $0.7$, then the trustworthiness $P_{i}$ can sometimes be greater or smaller than $0.7$. Note that this is a reasonable assumption for various machine learning-based procedures or Bayesian learning in particular. In reality, we cannot guarantee that any machine learning method is completely free of such bias, but the unbiased case is interesting to study, and we expect any residual bias to be fairly small, if the algorithm is configured using sufficient empirical data.

We extend our definition of $\omega$ to accept random variables as parameters. In that case, the output of $\omega$ is a distribution over accuracy. The expectation of such decision accuracy is:

$$\mathbb{E}(\omega(\bm{\hat{P}},\bm{P}))=\sum_{\hat{\bm{p}},\bm{p}}\mathbb{P}(\bm{\hat{P}}=\hat{\bm{p}},\bm{P}=\bm{p})\omega(\hat{\bm{p}},\bm{p})$$ (4)

Besides, there can be an ideal situation where ”magically” the decision maker knows the actual trustworthiness variable (i.e., $\bm{P}$), and can use it to make decisions. The expected probability of making correct decision is,

$$\mathbb{E}(\omega(\bm{P},\bm{P}))=\sum_{\bm{p}}\mathbb{P}(\bm{P}=\bm{p})\omega(\bm{p},\bm{p})$$ (5)

5.2. Stability of Correctness

In this section, we do not care about what the distribution of $\bm{P}$ actually looks like, as long as $\mathbb{E}(\bm{P})=\hat{\bm{p}}$, meaning the trust values used for decision making are unbiased. The main result is that in this case, the decision accuracy that $\mathcal{D}_{W}$ is believed to achieve by the decision maker, equals the probability that the decision is actually correct. This is an important positive result, that supports the idea of using WMV in practice. Decision makers are not delusional about the correctness of their decisions. Formally, we define the property of Stability of Correctness (SoC) as:

Theorem 5.

Stability of Correctness (SoC): For WMV, if $\hat{\bm{p}}=\mathbb{E}(\bm{P})$, then $\mathbb{E}(\omega(\hat{\bm{p}},\bm{P}))-\omega(\hat{\bm{p}},\hat{\bm{p}})=0$.

Sketch of Proof.

It follows from the fact in Section 4.2 that the indicator vector set $\mathbb{D}_{W}(\hat{\bm{p}})$ where decisions are supposed to be correct remains unchanged, when $\hat{\bm{p}}$ is unchanged. Also consider the fact that each $P_{i}$ is independently distributed. ∎

We show the results of two Monte Carlo simulations with $100,000$ runs over Example 2 to demonstrate the effect of distribution variance on the expected correctness of an unbiased estimate. In Figure 5(a), we depict $\omega(\hat{\bm{p}},\hat{\bm{p}})$ and $\mathbb{E}(\omega(\hat{\bm{p}},\bm{P}))$, where trustworthiness $\bm{P}$ is a Beta distribution with expected value $\hat{\bm{p}}$ equal to trust (unbiased) and a variance set by the $x$-axis. This figure shows the variance of the trustworthiness $\bm{P}$ has no effect on the correctness on average, which confirms our theorem. In contrast, in Figure 5(b), we depict $\omega(\bm{p},\bm{p})$ and $\mathbb{E}(\omega(\bm{\hat{P}},\bm{p}))$, letting the trust be the quantity being a random variable, distributed around trustworthiness with increasing variance. Unsurprisingly, this figure shows that a more divergent trust distribution leads to lower average correctness $\mathbb{E}(\omega(\bm{\hat{P}},\bm{p}))$ since the trust is more likely to be far away from trustworthiness and results in accuracy degradation. Furthermore, $\mathbb{E}(\omega(\bm{\hat{P}},\bm{p}))$ can never exceed $\omega(\bm{p},\bm{p})$, in line with the conclusion of section 4.3. In the next section, we will study this case further.

(a) $\hat{\bm{p}}$ fixed, $\bm{P}$ probabilistic

(b) $\bm{p}$ fixed, $\bm{\hat{P}}$ probabilistic

Figure 5. Effect of variance on Stability of Correctness

5.3. Stability of Optimality

Although for Stability of Correctness the shape (and variance) of the trustworthiness distribution was irrelevant, intuitively a distribution with less variance should still be better for the decision maker. We introduce another stability property in this section to capture this idea: Stability of Optimality (SoO). Formally, it means whether decisions made with the trustworthiness variables revealed are as good as those made only with the trust values available. We formally capture this gap with the definition below:

$$SoO(\bm{P})=\mathbb{E}(\omega(\bm{P},\bm{P}))-\mathbb{E}(\omega(\hat{\bm{p}},\bm{P}))$$ (6)

In other words, it also measures compared with using trust to decide, how much the decision accuracy can be improved, if trustworthiness values are available. Note that an equivalent (via Theorem 5) formulation is: $\mathbb{E}(\omega(\bm{P},\bm{P}))-\omega(\hat{\bm{p}},\hat{\bm{p}})$, when $\hat{\bm{p}}=\mathbb{E}(\bm{P})$.

To analyze Stability of Optimality formally, we introduce some definitions. Assume the trustworthiness $p_{i}$ is bounded in some range, e.g. $\forall i,a_{i}\leq p_{i}\leq b_{i}$. Denote the value space of $\bm{p}$ as Hypercube $\mathbb{H}$, $\bm{p}\in\mathbb{H}$. The set of vertexes of the Hypercube is denoted as Vertex Space $\mathbb{Q}$, where each vertex $\bm{q}\in\mathbb{Q}$ and $q_{i}$ is either $a_{i}$ or $b_{i}$. Defined in the hypercube, the distribution of $\bm{P}$ with expectation $\hat{\bm{p}}$ can be arbitrary. We name an extreme distribution for random variables $\bm{P}$ in the vertex space $\mathbb{Q}$ of the hypercube, where $\mathbb{P}(P_{i}=a_{i})=\frac{b_{i}-\hat{p}_{i}}{b_{i}-a_{i}}$, $\mathbb{P}(P_{i}=b_{i})=\frac{\hat{p}_{i}-a_{i}}{b_{i}-a_{i}}$.

When trustworthiness is revealed for decision-making, we observe that a high variance in trustworthiness is good for accuracy, especially the extreme distribution. That is, when a source is more trustworthy than the average, increasing its weight enhances overall decision accuracy. Conversely, when a source is less trustworthy than the average, it can degrade decision quality to some extent, but the impact is mitigated by reducing the weight of this source. In other words, it’s better to have a $50\%$ chance for a source with $P=0.9$ and $50\%$ for $P=0.5$, than a source with $p=0.7$. Formally,

Lemma 0.

Take random variables $\bm{P}$ defined in a Hypercube with $\mathbb{E}(\bm{P})=\hat{\bm{p}}$. The correctness of $\mathbb{E}(\omega(\bm{P},\bm{P}))$ is bounded by the extreme distribution:

$$\mathbb{E}(\omega(\bm{P},\bm{P}))\leq\sum_{\bm{q}\in\mathbb{Q}}\left(\omega(\bm{q},\bm{q})\prod_{i=1}^{n}\mathbb{P}(P_{i}=q_{i})\right)$$ (7)

Sketch of Proof.

Per Lemma 2, $\omega(\bm{p},\bm{p})$ is convex in one dimension, and the extreme distribution maximizes $\mathbb{E}(\omega(\bm{P},\bm{P}))$ in that dimension. The Lemma follows by independence of the trustworthiness variables. ∎

Lemma 6 demonstrates that the decision accuracy is bounded (not always 100%), even in the ideal situation where trustworthiness is given, and it is determined by the distribution of trustworthiness. This is intuitive as more trustworthy sources should lead to better decisions. Further, if trustworthiness is a constant rather than a random variable, Lemma 6 still holds. That is:

Corollary 0.

For any point $\bm{p}$ in the Hypercube, $\omega(\bm{p},\bm{p})$ is bounded by a linear combination of the correctness of the vertexes of the hypercube.

$$\omega(\bm{p},\bm{p})\!\leq\frac{1}{\prod_{i=1}^{n}(b_{i}-a_{i})}\!\!\sum_{\bm{q}\in\mathbb{Q}}\!\omega(\bm{q},\bm{q})\!\!\!\prod_{i:q_{i}=a_{i}}\!\!\!(b_{i}-p_{i})\!\!\!\prod_{i:q_{i}=b_{i}}\!\!\!(p_{i}-a_{i})$$ (8)

Proof.

Let $\mathbb{P}(\bm{P}=\bm{p})=1$ in Lemma 6. ∎

(a) $\hat{p}$ fixed, $P$ probabilistic

(b) Example of Beta Distribution

Figure 6. Effect of Variance on Stability of Optimality

Stability of Optimality does not strictly hold, as the gap (Equation 6) is typically non-zero. We prove upper bounds on the gap, which goes to $0$ as the distribution of trustworthiness becomes tighter. Let $\delta$ quantify the size of the support of the distribution.

Theorem 8.

Stability of Optimality: If $\hat{\bm{p}}=\mathbb{E}(\bm{P})$ and all $P_{i}$ have support $[\hat{p}_{i}-\delta_{i},\hat{p}_{i}+\delta_{i}]$, then

$$SoO(\bm{P})\leq\!\left(1-\omega(\hat{\bm{p}},\hat{\bm{p}})\right)\!\cdot\!\left(1-\prod_{i=1}^{n}\left(1-\frac{1}{2}\cdot\frac{\delta_{i}}{1-\hat{p}_{i}}\right)\right)$$ (9)

A weaker but more intuitive bound is also derived using the Bernoulli Inequality,

$$SoO(\bm{P})\leq\frac{1-\omega(\hat{\bm{p}},\hat{\bm{p}})}{2}\sum_{i=1}^{n}\frac{\delta_{i}}{1-\hat{p}_{i}}$$ (10)

Sketch of Proof.

Via Lemma 6, we know the extreme distribution that maximizes $\mathbb{E}(\omega(\bm{P},\bm{P}))$, relying on the correctness of the vertexes. Via Corollary 7, the upper bounds for the correctness of vertexes can be obtained, only relying on $\omega(\hat{\bm{p}},\hat{\bm{p}})$. With some algebra, both bounds (9) and (10) can be obtained. ∎

While there is a gap between making decisions based on unbiased trust and based on trustworthiness, Theorem 8 proves that this gap is bounded by a relatively small threshold, implying that the unbiased trust would not reduce the decision quality too much. The upper bound is influenced by the distribution of trustworthiness, and converges towards zero with that variance reducing.

To illustrate the effect of distribution variance on $SoO(\bm{P})$, we provide a Monte Carlo simulation with $100,000$ runs over Example 2. In Figure 6(a), we measure $\mathbb{E}(\omega(\bm{P},\bm{P}))$ and $\mathbb{E}(\omega(\hat{\bm{p}},\bm{P}))$, where $\hat{\bm{p}}$ is constant, and $\bm{P}$ follows Beta distribution with increasing variance. It presents that the larger the variance is, the larger $SoO(\bm{P})$ is, which validates the result of Lemma 6. To put the quantity of the variance in context, we provide examples of trustworthiness being Beta distributions with a certain variance in Figure 6(b).

(a) Effect of $\delta$

(b) Effect of $n$

(c) Effect of single $\hat{p}_{1}$

(d) Effect of identical $\hat{\bm{p}}$

Figure 7. Parameter Analysis on Stability of Optimality

In Figure 7, we provide a parameter analysis with numerical experiments to demonstrate how they influence $SoO(\bm{P})$ and the bounds. Example 2 is used in Figures 7(a) and 7(c); Figures 7(b) and 7(d) needed some adaptation, where the sources have identical $\hat{p}=0.7$. And $\delta=0.05$ is the default for all the sources.

Figure 7(a) represents that with $\delta$ decreasing, $SoO(\bm{P})$ and its bounds also decrease to zero. There is a linear bound on the effect of $\delta$ (via the weaker bound). This implies that the uncertainty level of sources plays a significant role in determining the Stability of Optimality. In Figure 7(b), with the number of identical sources $n$ increasing ($\hat{p}=0.7$), $SoO(\bm{P})$ and the bounds change little, which implies source number perhaps influences little on the accuracy gap $SoO(\bm{P})$ (i.e., the gap in accuracy between decision making using unbiased trust and using trustworthiness). This means that the number of sources may barely influence how valuable it is to know the sources’ combined trustworthiness.

In Figure 7(c), only $\hat{p}_{1}$ is variable and it shows that $SoO(\bm{P})$ always remains low level and it is a piecewise function with local maximization. As studied in Section 4.1, it becomes evident that the local maximization results from the piece-wise nature of the trustworthiness effect on decision accuracy. In Figure 7(d), where all $\hat{p}$ are equal and increase to $1$ simultaneously, $SoO(\bm{P})$ almost decreases to $0$. It makes sense because with the trustworthiness increasing, the decision accuracy increases more slowly due to the concavity of WMV with identical sources (See Lemma 3).

To conclude, the gap of Stability of Optimality $SoO(\bm{P})$ is somewhat sensitive to parameter $\delta$, which depicts the range and variance of sources trustworthiness, but not sensitive to the number of sources and other parameters. Overall, the optimality of WMV has a high degree of stability, meaning $SoO(\bm{P})$ tends to be close to $0$.

6. Conclusion and Future Work

The common dependence on an estimate or trust of source trustworthiness brings out the need to analyze whether WMV is stable, meaning having tolerant decision inaccuracy with the difference between trust and trustworthiness bounded.

We first analyze how sensitive WMV is to the changes in trust and trustworthiness. We find that small deviation between trust and trustworthiness does not affect accuracy, and also underestimation usually harms less than overestimation. We then introduced two statistical properties of WMV, Stability of Correctness and Stability of Optimality. Assuming that on average the estimation procedure has no bias towards over or underestimating, we proved that Stability of Correctness holds absolutely, regardless of which estimation procedure is used or how well it estimates. This guarantees that relying on an unbiased estimate of source trustworthiness is safe, which is also common in practice. However, the amount of inefficiency introduced by relying on an estimate instead of the trustworthiness itself is limited, as we prove a linear bound on Stability of Optimality. The proposed formal framework and the two types of stability properties can be generalized to analyze other types of decision mechanisms or scenarios (e.g., where sources are dependent).

For future work, beyond the bounded assumption, it’s valuable to explore a more precise characterization of the impact of the trustworthiness distribution on SoO in the unbiased setting. Besides, it is also worth studying the stability of WMV in a more general case, namely when trust is a biased estimate of trustworthiness. Some researchers have found that although some sources are assigned weights, they have no influence on the decision result Allouche et al. (2021); Bowen (2009). In other words, we may distribute more estimate error on such sources.

This work was supported by National Natural Science Foundation of China (NSFC) under Grant 62106223 and (NSFC) Grant 62293511.