Verifying the Selected Completely at Random Assumption in Positive-Unlabeled Learning
Abstract
The goal of positive-unlabeled (PU) learning is to train a binary classifier on the basis of training data containing positive and unlabeled instances, where unlabeled observations can belong either to the positive class or to the negative class. Modeling PU data requires certain assumptions on the labeling mechanism that describes which positive observations are assigned a label. The simplest assumption, considered in early works, is SCAR (Selected Completely at Random Assumption), according to which the propensity score function, defined as the probability of assigning a label to a positive observation, is constant. On the other hand, a much more realistic assumption is SAR (Selected at Random), which states that the propensity function solely depends on the observed feature vector. SCAR-based algorithms are much simpler and computationally much faster compared to SAR-based algorithms, which usually require challenging estimation of the propensity score. In this work, we propose a relatively simple and computationally fast test that can be used to determine whether the observed data meet the SCAR assumption. Our test is based on generating artificial labels conforming to the SCAR case, which in turn allows to mimic the distribution of the test statistic under the null hypothesis of SCAR. We justify our method theoretically. In experiments, we demonstrate that the test successfully detects various deviations from SCAR scenario and at the same time it is possible to effectively control the type I error. The proposed test can be recommended as a pre-processing step to decide which final PU algorithm to choose in cases when nature of labeling mechanism is not known.
1 Introduction
Learning from positive-unlabeled data (PU learning) is an active research topic that has attracted great deal of interest in the machine learning community in recent years [1, 2, 3]. The goal of PU learning is to train a binary classifier on the basis of training data containing positive and unlabeled instances, where unlabeled observations can belong either to the positive or to the negative class. The problem is motivated by many practical applications. A representative example is detection of illegal or harmful content in social networks. Some profiles are reported as containing such content (positive cases). However, profiles not reported as illegal may also contain content that violates the law, but this has not been verified. Another example is reporting side effects of taking medications. The lack of a reported side effect does not mean that it did not occur. Therefore, it is reasonable to treat reported cases as positive and unreported cases as unlabeled. PU data appear naturally in the classification of texts and images [4], anomaly detection [5, 6], survey research [7] and in many bioinformatics applications [8].
The simplest approach in PU learning (called naive or biased method) is to treat all unlabeled observations as negative and use standard binary classifiers. However, this method may lead to a significantly biased posterior probability estimate for the true class variable and consequently to poor classification accuracy, especially if the unlabeled set contains relatively many positive cases. Therefore, most authors approach modeling PU data by imposing certain assumptions on the labeling mechanism that describes which positive observations are labeled.
The simplest assumption is SCAR (Selected Completely at Random Assumption), according to which the propensity score function, i.e. the probability of a labeling a positive observation, is constant [2, 1, 9, 10, 11]. Under the SCAR assumption, a possible approach is to estimate label frequency [12, 13, 14, 15] and then use it to scale the posterior probabilities obtained from the naive method or, alternatively, optimize weighted empirical risk function with weights depending on the label frequency [1, 16]. Generally, SCAR based algorithms are relatively simple and computationally fast. However, the SCAR assumption is not met in many practical situations [3]. For example, among people experiencing drug side effects, the likelihood of reporting may depend on age or socioeconomic factors.
A much more realistic assumption is SAR (Selected at Random), which states that the propensity score function depends solely on the observed feature vector [17, 18, 19, 3, 20, 21]. Figure 1 shows the difference between SCAR and SAR assumptions for artificially generated two-dimensional data. However SAR based algorithms are usually computationally more expensive as they require challenging estimation of the propensity score. An exception is the situation when we consider assumptions that are special cases of SAR, such as Probabilistic Gap Assumption [18], invariance of order assumption [22] or impose some additional assumptions such as knowledge of prior probability of positive class [23]. Most existing SAR algorithms are based on the alternating fitting of two models: one is related to the posterior probability of the true class variable, and the other is related to the propensity score [17, 19, 20]. For example, SAR-EM [17] and LBE [19] are based on an EM-type algorithm, whereas TM [20] relies on iterative approximation of a set of positive observations and using this set to estimate the propensity score.
These approaches require many iterations, each of which includes training the classifier. Moreover, importantly, applying one of these methods, when in reality propensity score is constant, leads to a loss of efficiency with respect to SCAR-designed approaches. Table 1 contains a comparison of representative methods based on SCAR and SAR, in a situation where the true propensity score is constant and equal to . As the SCAR method, we used the popular estimator of labeling frequency called TiCE [14] and then scaled the posterior probabilities obtained from the naive model by . As the SAR method, we used the LBE algorithm [19] mentioned above. The SCAR-based method has higher classification accuracy for most considered datasets and, importantly, significantly shorter training time. Therefore, verifying the SCAR assumption becomes an important task that, to our knowledge, has not been discussed in the literature.
SCAR method | SAR method | |||
---|---|---|---|---|
Dataset | Accuracy | Time | Accuracy | Time |
Breast | 0.969 0.017 | 1.62 0.02 | 0.954 0.011 | 18.4 0.80 |
Wdbc | 0.930 0.022 | 2.94 0.04 | 0.930 0.019 | 19.6 0.80 |
Banknote | 0.983 0.004 | 1.49 0.01 | 0.987 0.012 | 20.8 0.40 |
Segment | 0.972 0.004 | 8.69 0.15 | 0.990 0.006 | 24.8 1.91 |
CIFAR10∗ | 0.810 0.009 | 5.36 0.11 | 0.718 0.031 | 25.8 0.74 |
USPS∗ | 0.726 0.022 | 5.47 0.21 | 0.712 0.017 | 26.4 0.80 |
Fashion∗ | 0.824 0.014 | 5.28 0.15 | 0.816 0.021 | 25.4 0.49 |
∗ Randomly chosen subsamples of 5000 images were considered.
In this work, we propose a relatively simple and computationally fast test that can be used to determine whether the observed data meet the SCAR assumption. The proposed procedure consists of two steps. In the first step, our goal is to determine the set of positive observations. In the second step, we generate artificial labels conforming to the SCAR situation, which in turn allows us to mimic the distribution of the test statistic under the null hypothesis of SCAR. The idea of the method is based on the property that the SCAR assumption is equivalent to the equality of the distribution of the feature vector for positive observations and the distribution for labeled observations. This leads to the selection of 4 different test statistics that measure the divergence between the above distributions. In experiments, we demonstrate that the test successfully detects various SAR schemes and at the same time it is possible to effectively control type I error (observed significance level) for most considered datasets. This is supported by theoretical results which show that (i) the proposed test is indeed consistent and that (ii) the essential part of the proposal, namely selection of positive elements among unlabeled ones satisfies probabilistic guarantees in an idealized scenario. The proposed test can be recommended as a pre-processing step to decide which final PU algorithm to choose.
2 Background
2.1 Positive-unlabeled learning
In PU learning, each observation can be described by the triple , where is feature vector, is true class variable ( denotes positive class), which is not observed directly and is class label indicator, describing whether the instance is labeled and thus positive () or unlabeled . The unlabeled instance can be either positive or negative. In PU learning it is assumed that negative examples cannot be labeled, i.e., . The fraction of positive observations that are assigned a label is determined by the labeling frequency . In this work, we adopt a single-training sample scenario [1] assuming that iid random vectors for are generated from some unknown distribution . The PU training data is as we do not observe . The goal is to train a classifier that predicts for some new instance using the incompletely labeled training set only. Note that training the naive classifier which treats as the class variable, we can estimate , whereas our goal is to estimate . Table 2 contains the most important notations used in the paper.
Notation | Meaning |
---|---|
number of instances | |
number of features | |
feature vector | |
unobserved true class variable | |
label indicator | |
PU training data | |
class prior | |
labeling frequency | |
positive set (unobserved) | |
labeled set (observed) | |
unlabeled set (observed) | |
posterior probability of | |
posterior probability of | |
propensity score function | |
labeling frequency |
2.2 SCAR and SAR assumptions
Learning from PU data is challenging task and certain assumptions are required to make inference from PU data possible. The assumptions concern the labeling mechanism describing which positive observations are assigned a label. Specifically, the labeling mechanism assigns a probability , called propensity score, of being labeled to each positive example. A high value of propensity score indicates that a positive observation described by vector will be assigned a label with a high probability. The two assumptions which we want to check can be expressed in terms of the propensity score function.
Assumption 1 (Selected Completely at Random).
Propensity score is constant .
Clearly, SCAR is unlikely to hold in many situations. Therefore, many works consider a more general and less restrictive assumption called SAR.
Assumption 2 (Selected at Random).
Propensity score is non-constant function depending solely on the observed features . 111Typically, SAR refers to the situation where e(x) can be any function with values in , which also includes the case of a constant function (SCAR). In this work, it is more convenient to assume that SAR refers to non-constant propensity score.
Importantly, various different labeling mechanisms fall in SAR category, including cases where the labeling mechanism depends on a single feature or on many of them simultaneously. More generally, the labeling mechanism depends on variables that are not observed in our data and then SAR is not met. However, in such a situation, modeling of PU data becomes impossible unless all active predictors become available. Therefore, SAR can be treated as the most general assumption made in PU learning.
The SCAR assumption can be characterized by the following property [1, 2]; for completeness, we provide proof in the supplement 222https://github.com/teisseyrep/SCAR/tree/main. It states that SCAR is equivalent to
(1) |
which means that the feature distribution for labeled observations matches the feature distribution in the positive class. Property (1) will be used to construct a proposed testing procedure.
3 Verifying the SCAR assumption in PU learning
3.1 Null and alternative hypotheses
Our goal is to verify the SCAR assumption based on PU data and determine which mechanism corresponds to how our PU data was generated: SCAR or SAR. We use a statistical hypothesis testing framework. In view of property (1), the null and alternative hypotheses can be written as
The test statistic should measure how close the empirical distributions and corresponding to the true distributions and distributions are. A small value of the test statistic should indicate , while large values of the statistic should lead to its rejection. In Section 3.3, we present possible test statistics that can be used to measure the divergence between these two distributions. However, even with a defined test statistic, we face two challenges. First, distribution cannot be directly estimated because we do not observe . Second, we need to know the distribution of under to determine which values of are typical under and consequently be able to control for the type I error. The above two issues are addressed in section 3.1.
Finally, it is worth noting that one can make two errors: type I error (reject when it is true) and type II error (not reject when is true). The above errors are not symmetric. Rejection of suggests applying SAR-based algorithms, which are usually more demanding computationally but which are also valid in SCAR situations. On the other hand, SCAR algorithms are unable to estimate non-constant propensity functions and thus may fail in some situations related to SAR. Therefore, a type II error can potentially have more serious negative consequences.
3.2 Testing procedure
The testing procedure consists of two steps. In step (1) our goal is to approximate the positive set , and in step (2) it is to generate the distribution of the test statistic under (null distribution). In the following, we assume that the class prior is known, although in practice it is usually replaced by an estimated value. The assumption is commonly adopted in PU inference [24, 25, 26]. We note in passing that for estimation of under SAR, estimation of posterior probability seems unavoidable, this however requires assumptions in its turn to ensure identifiability.
Step (1) involves training a naive model in which is treated as a class variable. This allows us to estimate , and then sort the unlabeled observations in descending order: , where . The positive set is estimated as the sum of the labeled set and the set of unlabeled observations with the highest estimated posterior probabilities , i.e., , where and . Estimator is simply a fraction of labeled exampels in training data. The rationale for this method of estimating is explained in Lemma 1 in Section 4. Note that contains approximately observations, which corresponds to the expected number of observations in . We also define variable iff , which approximates the true class indicator .
In step (2), we generate the artificial label indicator which mimics a true label indicator , but corresponds to a SCAR situation. Specifically, for each we generate from Bernoulli distribution with success probability and we set , for . The above step is repeated for and in each loop we compute . Finally, based on the values , we can estimate the distribution of the statistic under . Figure 2 visualizes steps (1) and (2) for SAR dataset. The higher the value of parameter , the better the approximation of the distribution under , but at the same time the greater the computational cost.
The last step is to calculate the p-value , where is the value of test statistic for the observed label indicator . A small p-value indicates that takes unusually large values compared to the values corresponding to , which should lead to the rejection of . Formally, we reject , when , where is user-specified significance level. The whole procedure is described by Algorithm 1.
3.3 Test statistics
Algorithm 1 is generic and allows any statistic to be used. However, for the algorithm to work effectively, the statistics should meet two requirements. Firstly, it should describe the deviation from , in the sense that its theoretical value should be for and take positive values for . Second, it should be computationally fast because we have to compute it times. In this chapter, we present 4 possible statistics that meet the above conditions.
Let us denote by and the probability distributions corresponding to and in Algorithm 1. A natural way to measure how different the two distributions are is to use the Kullback-Leibner (KL) divergence [27, 28]. Despite the desirable properties, calculating KL is computationally demanding for multidimensional distributions. However, under certain assumptions, computations can be simplified. For example, assuming a Gaussian distribution of features, we obtain:
(2) |
where is a difference between the means and for the and distributions, respectively, whereas and are the corresponding covariance matrices. In the experiments, we consider two variants of (2): the first one is based on the assumption of independence of variables (we simply denote it as KL) and the second one in which we estimate the covariance matrices (denoted as KLCOV).
In addition, we consider Kolmogorov-Smirnov (KS) statistic defined as
(3) |
where and are marginal distributions corresponding to the multivariate distributions and and is standard Kolmogorov-Smirnov statistic for one-dimensional probability distributions.
Finally, we also consider a classifier-based statistic. Since we want to decide how much the distributions and differ from each other, we define an auxiliary class variable such that if and if . Then we train simple Naive Bayes classifier using training data . Other classifiers can also be used as long as their training time is acceptable. We measure the quality of the classifier using and define test statistic as . If the distributions coincide, then and the value of the statistic will be around . On the other hand, if the distributions are well separated, then and the value of the statistic will be around . In experiments, we refer to this method as NB AUC.
4 Theoretical justifications
We first show that Algorithm 1 allows to control the type I error (the probability of rejecting when SCAR is met) in an idealized situation when the set coincides with the positive set . Then we provide some justification for the choice of . In order to address the first problem note that probability of rejecting can be written in terms of p-value as . The following Theorem indicates that the probability does not exceed , provided that is true.
Theorem 1.
Assume that SCAR assumption is met and the algorithm is based on in place of . Then distribution of is super-uniform i.e.
Proof.
Let us denote by , and , where samples corresponding to distributions , and , respectively. Test statistics, considered in Algorithm 1 can be written as functions of the samples, i.e.,
Under SCAR, contains conditionally independent observations given , generated from , whence they are distributionally equal to observations from . The random variables are exchangeable, i.e. their joint distribution does not change when their positions are randomly ordered. Exchangeability implies that p-value is uniformly distributed on which implies that , where is integer part of . ∎
In order to check the soundness of the choice of sample as a substitute of all positive observations, we consider the idealized scenario in which is known and is an iid sequence from . Thus, with a slight abuse of previous notion, correspond to observed unlabeled observations, whereas corresponding are not observed and is deterministic sequence corresponding to expected number of unlabeled observations . We consider and denote by th order statistic in this sequence starting from the largest one i.e.
We will consider top values . We disregard ties assuming in the following that is continuous random variable. This corresponds in the algorithm to considering top values of and adding them to labeled observations. The above approach is justified by the following Lemma which shows that ordering observations with respect to is equivalent to ordering with respect to conditional probability
Lemma 1.
Assume SCAR assumption is hold. Then for any observations and
Proof.
Under SCAR, we have and thus ordering wrt to can be replaced by ordering wrt . From Bayes Theorem,
is increasing function of , which proves the assertion. ∎
We will establish a bound on the probability that for the lowest chosen observation the corresponding value of . To this end define as the concomitant value of i.e.
We define the following function
Discussion of the properties of can be found in the supplement. The crucial observation is that the following equality holds
and thus
(4) |
Let denote cdf of where is distributed according to i.e. . We have the following result.
Theorem 2.
Assume that is strictly increasing function.
-
(i)
Let be a sequence such that , where . Moreover, has continuous density . Then we have for
(5) -
(ii)
For we have
-
(iii)
for any we have
Proof.
Part (i) follows from Theorem 2.2 (b) in [29]) for there ( denotes the order of the moment in the cited paper) and application of Schwarz inequality after noting that can be represented as th order statistic from the sequence , where is iid sample from the uniform distribution. Note that is valid in view of monotonicity of . Proofs of (ii) and (iii) are given in the supplement.
∎
Note that the magnitude of appearing in (5) is inherently related to separability of and . In order to see that recall again that in view of conditional independence of and given under SCAR we have that and . Thus if and are well separated is close to 1 for all positive unlabeled observations, which constitute fraction of all unlabeled ones. Thus for we have that . Moreover, it follows from part (i) that, provided the following condition holds
(6) |
i.e. stochastically dominates -uniformly distributed random variable [30], that we have that
and analogous result, with replaced by , holds for probability of two adjacent concomitants. Interestingly, we can also have more general and simpler result provided (6) is valid. Note that now the result concerns concomitants corresponding to to top order statistics.
Theorem 3.
Assume that condition (6) holds for . Then we have
(7) |
The proof of this result is given in the supplemental material. Note that, e.g. for , and , we need to choose additional observations from unlabeled observations. In this case the probability bound in (7) is . However, for large and small , the bound may become weak. In order to obtain better guarantees, one may choose smaller number of top order statistics than , focusing on most likely positive observations among unlabeled ones in the modified algorithm.
5 Experiments
We analyze the effectiveness of the proposed testing procedure and compare the performance of 4 statistics: KL, KLCOV, KS and NB AUC. As evaluation measures, we consider: type I error (probability of rejecting when is true), which should not exceed assumed significance level and power of the test (probability of rejecting when is true). In particular, we aim to answer the following research questions. (1) Do the tests control for a type I error? (2) Which of the proposed statistics has the greatest power ? (3) How does the power depend on various factors such as: the sample size, dependence between features or the discrepancy between data distribution and distribution? In the experiments we set , . Moreover, Random Forest classifier [31] was used as a base learner to estimate . To estimate the probability of rejecting we repeated experiments times.
5.1 Datasets
In experiments we used 4 popular tabular datasets (Breast Cancer, Wdbc, Banknote and Segment) [32] and 3 image datasets (CIFAR 10, USPS and Fashion) [33]. Details about preprocessing the datasets are described in the supplement; Table 1 in the supplement contains summary statistics. Moreover, we used two artificial datasets (Art1 and Art2), which are obtained as follows. In Art1, we first generate from the Bernoulli distribution with a success probability of . Then we generate feature vectors from the distributions and , where . In Art2, the feature vectors are generated from the distributions and , where . In Art1 we assume independence of features, while in Art2, the features are dependent and additionally covariance matrices in the positive and negative classes are different.
5.2 Labeling strategies
Given a dataset with binary class variable , we artificially generate PU data using various labeling strategies. All negative observations are assigned to unlabeled subset. From the positive observations we randomly select those that will be labeled with probability , whereas the remaining observations are assigned to unlabeled set. The following strategies are considered.
-
S0.
Propensity score is constant .
-
S1.
Propensity score , where is a value of the first feature, for -th observation and .
-
S2.
Propensity score .
-
S3.
Propensity score .
Strategy S0 is used to analyze type I error for the methods. Strategies S2 and S3 were already used in papers on instance-based PU learning [19, 20]. Parameter vector is obtained from logistic regression model fitted on the fully labeled data, i.e., assuming the knowledge of . Parameter controls how far we are from the null hypothesis . Note that the value , corresponds to SCAR, i.e. the propensity function is constant. The value of corresponds to the SAR situation and by increasing , we move away from . Moreover, parameter controls how much affects the propensity score. Parameter is determined to control the value of labeling frequency . Value of is calculated for the previously found parameter and fixed . We report the results for S1-S2 and , the results for as well as for S3 are given in the supplement.
5.3 Discussion
5.3.1 Controlling type I error
Figure 3 (top left and bottom left panels) indicates that, in the case of artificial data, all methods control for type I error when the features are independent (Art1). In the case of dependencies (Art2), KL, KS and NB AUC work correctly, i.e., they do not exceed the assumed significance level . For KLCOV, the probability of rejecting exhibits undesirable increase with the sample size. KS and NB AUC control for the type I error for all 7 real data sets (Table 3). KL and KLCOV exceed for 2 and 3 datasets, respectively. In particular, KLCOV always rejects for Banknote data, which is due to the lack of robustness to the inaccurate estimation of the positive set . Indeed, when assuming knowledge of set , the type I error does not exceed for this method. In summary, the KL and NB AUC perform conservatively for all data sets and they should be recommended if controlling the type I error is our important objective. Conclusions remain similar for (Tables 2, 3 in the supplement).
5.3.2 Power of the tests
As expected, for artificial datasets, the power of the tests increases when the number of observations increases (Figure 3). KL method has the largest power, followed by KLCOV. Importantly, however, KLCOV does not control for type I error for Art2, so analyzing the power may be misleading for this method. KS and NB AUC converge more slowly to , but this is the price for effectively controlling the type I error. The power also increases when the parameter is increased, which is natural because a larger indicates more significant deviation from (Figure 3). For both artificial datasets and , the power for all methods approaches for relatively small sample size , whereas for , we need significantly more observations to achieve this level of power.
For real datasets, we also see that the power increases as the parameter increases (Figure 4 and Table 4). Figure 4 shows that the KS method usually achieves the highest power. The exception is the Wdbc dataset and S2, for which KL and KLCOV have the highest power (Figure 4), but for these methods the type I error is significantly exceeded, so they should not be taken into account in the comparison for this particular dataset. Among methods that properly control type 1 error (KS and NB AUC), KS achieves greater power for most datasets, labeling schemes, and parameter values (Table 4). For example, for S1, KS is the winner 12 out of 14 times, for S2 times. Conclusions remain similar for an for S3 (Tables 4-6 in the supplement).
We also examined the robustness of the testing procedure to the class prior estimation error. The results in Table 7 (supplement) indicate that overestimation of has a greater negative impact than underestimation. For overestimated we observe that very often type I error exceeds the level, which is due to the fact that in this case, set contains, in addition to true positive observations, too many negative observations.
Dataset | KL | KLCOV | KS | NB AUC |
---|---|---|---|---|
Breast-w | 0.01 0.01 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
Wdbc | 0.18 0.04 | 0.2 0.04 | 0.0 0.0 | 0.02 0.01 |
Banknote | 0.03 0.02 | 1.0 0.0 | 0.04 0.02 | 0.0 0.0 |
Segment | 0.12 0.03 | 0.15 0.04 | 0.04 0.02 | 0.01 0.01 |
CIFAR10 | 0.01 0.01 | 0.03 0.02 | 0.05 0.02 | 0.0 0.0 |
USPS | 0.02 0.01 | 0.05 0.03 | 0.02 0.01 | 0.0 0.0 |
Fashion | 0.01 0.01 | 0.03 0.02 | 0.02 0.01 | 0.0 0.0 |
labeling scheme S1 | |||||
---|---|---|---|---|---|
Dataset | KL | KLCOV | KS | NB AUC | |
Breast | 1 | 0.15 0.04 | 0.02 0.01 | 0.24 0.04 | 0.02 0.01 |
2 | 0.79 0.04 | 0.17 0.04 | 0.74 0.04 | 0.54 0.05 | |
Wdbc | 1 | 0.79 0.04 | 0.3 0.05 | 0.91 0.03 | 0.8 0.04 |
2 | 1.0 0.0 | 0.66 0.05 | 1.0 0.0 | 1.0 0.0 | |
Banknote | 1 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 |
2 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | |
Segment | 1 | 0.15 0.04 | 0.19 0.04 | 0.4 0.05 | 0.27 0.04 |
2 | 0.18 0.04 | 0.26 0.04 | 0.95 0.02 | 0.83 0.04 | |
CIFAR10 | 1 | 0.07 0.03 | 0.0 0.0 | 0.41 0.05 | 0.0 0.0 |
2 | 0.19 0.04 | 0.01 0.01 | 0.57 0.05 | 0.0 0.0 | |
USPS | 1 | 0.96 0.02 | 0.92 0.03 | 0.87 0.03 | 0.92 0.03 |
2 | 0.96 0.02 | 0.9 0.03 | 0.94 0.02 | 0.91 0.03 | |
Fashion | 1 | 0.99 0.01 | 0.95 0.02 | 0.99 0.01 | 1.0 0.0 |
2 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | |
labeling scheme S2 | |||||
Dataset | KL | KLCOV | KS | NB AUC | |
Breast | 1 | 0.95 0.02 | 0.37 0.05 | 0.95 0.02 | 0.89 0.03 |
2 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | 1.0 0.0 | |
Wdbc | 1 | 0.59 0.05 | 0.52 0.05 | 0.2 0.04 | 0.15 0.04 |
2 | 0.89 0.03 | 0.66 0.05 | 0.79 0.04 | 0.65 0.05 | |
Banknote | 1 | 0.8 0.04 | 1.0 0.0 | 0.46 0.05 | 1.0 0.0 |
2 | 1.0 0.0 | 1.0 0.0 | 0.99 0.01 | 1.0 0.0 | |
Segment | 1 | 0.06 0.02 | 0.13 0.03 | 0.06 0.02 | 0.21 0.04 |
2 | 0.12 0.03 | 0.16 0.04 | 0.3 0.05 | 0.79 0.04 | |
CIFAR10 | 1 | 0.29 0.05 | 0.2 0.04 | 0.05 0.02 | 0.08 0.03 |
2 | 0.9 0.03 | 0.88 0.03 | 0.83 0.04 | 0.79 0.04 | |
USPS | 1 | 0.48 0.05 | 0.48 0.05 | 0.44 0.05 | 0.37 0.05 |
2 | 1.0 0.0 | 1.0 0.0 | 0.99 0.01 | 0.98 0.01 | |
Fashion | 1 | 0.53 0.05 | 0.41 0.05 | 0.65 0.05 | 0.26 0.04 |
2 | 0.99 0.01 | 0.98 0.01 | 1.0 0.0 | 1.0 0.0 |
6 Conclusions
Using the proposed method, it is possible to decide whether PU data correspond the SCAR or SAR assumption, contolling type I error. The method is of significant practical importance, because it allows to choose between using more computationally expensive SAR algorithms or simpler methods based on SCAR. In many real applications, the impact of features on propensity score may be negligible and then SCAR algorithms are clearly a better choice. Theoretical results justify the method of estimating the set of positive observations and show that if it is estimated correctly, controlling the type I error is actually possible. In future work, other positive set estimation methods can be considered that do not assume knowledge of class prior and are based, for example, on FDR control. The proposed procedure is generic and can be combined with any classifier and test statistic that meets the general conditions. Among the investigated test statistics, we recommend using the Kolmogorov-Smirnov statistic, which properly controls for type I error while still having high power.
References
- [1] J. Bekker and J. Davis. Learning from positive and unlabeled data: a survey. Machine Learning, 109:719–760, 2020.
- [2] C. Elkan and K. Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’08, pages 213–220, 2008.
- [3] C. Gong, M. I. Zulfiqar, C. Zhang, S. Mahmood, and J. Yang. A recent survey on instance-dependent positive and unlabeled learning. Fundamental Research, 2022.
- [4] F. Chiaroni, M-C. Rahal, N. Hueber, and F. Dufaux. Learning with a generative adversarial network from a positive unlabeled dataset for image classification. In Proceedings of the 25th IEEE International Conference on Image Processing, ICIP’18, pages 1–6, 2018.
- [5] Y. Luo, S. Cheng, C. Liu, , and F. Jiang. Pu-learning in payload-based web anomaly detection. In Proceedings of the Third Conference on Security of Smart Cities, industrial Control Systems and Communications, SSIC’2018, pages 1–5, 2018.
- [6] L. Perini, V. Vercruyssen, and J. Davis. Learning from positive and unlabeled multi-instance bags in anomaly detection. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’23, page 1897–1906, 2023.
- [7] K. Sechidis, M. Sperrin, E. S. Petherick, M. Luján, and G. Brown. Dealing with under-reported variables: An information theoretic solution. International Journal of Approximate Reasoning, 85:159 – 177, 2017.
- [8] F. Li, S. Dong, A. Leier, M. Han, X. Guo, J. Xu, X. Wang, S. Pan, C. Jia, Y. Zhang, G. Webb, L. J. M. Coin, C. Li, and J. Song. Positive-unlabeled learning in bioinformatics and computational biology: a brief review. Briefings in Bioinformatics, 23(1), 2021.
- [9] H. Chen, F. Liu, Y. Wang, L. Zhao, and H. Wu. A variational approach for learning from positive and unlabeled data. In Proceedings of the International Conference on Neural Information Processing Systems, NIPS’20, pages 14844–14854, 2020.
- [10] Y. Zhao, Q. Xu, Y. Jiang, P. Wen, and Q. Huang. Dist-pu: Positive-unlabeled learning from a label distribution perspective. In Proceedings of the Conference on Computer Vision and Pattern Recognition, CVPR’22, pages 14461–14470, 2022.
- [11] Changchun Li, Ximing Li, Lei Feng, and Jihong Ouyang. Who is your right mixup partner in positive and unlabeled learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
- [12] H. Ramaswamy, C. Scott, and A. Tewari. Mixture proportion estimation via kernel embeddings of distributions. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pages 2052–2060, 2016.
- [13] S. Jain, M. White, and P. Radivojac. Estimating the class prior and posterior from noisy positives and unlabeled data. In Proceedings of the 30th International Conference on Neural Information Processing Systems, page 2693–2701, 2016.
- [14] J. Bekker and J. Davis. Estimating the class prior in positive and unlabeled data through decision tree induction. In Proceedings of the 32th AAAI Conference on Artificial Intelligence, pages 1–8, 2018.
- [15] M. Łazecka, J. Mielniczuk, and P. Teisseyre. Estimating the class prior for positive and unlabelled data via logistic regression. Advances in Data Analysis and Classification, 15:1039–1068, 2021.
- [16] P. Teisseyre, J. Mielniczuk, and M. Łazecka. Different strategies of fitting logistic regression for positive and unlabelled data. In Proceedings of Intrernational Conference on Computational Science, ICCS’20, pages 1–14, 2020.
- [17] J. Bekker, P. Robberechts, and J. Davis. Beyond the Selected Completely At Random Assumption for Learning from Positive and Unlabeled Data. In Proceedings of the 2019 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML’19, pages 71–85, 2019.
- [18] W. Gerych, T. Hartvigsen, L. Buquicchio, E. Agu, and E. Rundensteiner. Recovering the propensity score from biased positive unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI’22, pages 6694–6702, 2022.
- [19] C. Gong, Q. Wang, T. Liu, B. Han, J. You, J. Yang, and D. Tao. Instance-dependent positive and unlabeled learning with labeling bias estimation. IEEE Trans Pattern Anal Mach Intell, pages 1–16, 2021.
- [20] K. Furmańczyk, J. Mielniczuk, W. Rejchel, and P. Teisseyre. Double logistic regression approach to biased positive-unlabeled data. In Proceedings of the European Conference on Artificial Intelligence, ECAI’23, pages 764–771, 2023.
- [21] V. Verreet, L. De Raedt, and J. Bekker. Modeling PU learning using probabilistic logic programming. Machine Learning, pages 1–22, 2023.
- [22] M. Kato, T. Teshima, and J. Honda. Learning from positive and unlabeled data with a selection bias. In Proceedings of the 7th International Conference on Learning Representations, pages 1–12, 2019.
- [23] B. Na, H. Kim, K. Song, W. Joo, Y-Y. Kim, and I. Moon. Deep generative positive-unlabeled learning under selection bias. In Proceedings of CIKM’20, CIKM ’20, pages 1155––1164, New York, NY, USA, 2020. ACM.
- [24] G. Ward, T. Hastie, S. Barry, J. Elith, and J. Leathwick. Presence-only data and the EM algorithm. Biometrics, 65:554–563, 2009.
- [25] Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, pages 703–711. Curran Associates, Inc., 2014.
- [26] Ryuichi Kiryo, Gang Niu, Marthinus C. du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. In Proceedings of the NIPS’17, NIPS’17, pages 1674––1684, Red Hook, NY, USA, 2017. Curran Associates Inc.
- [27] T. M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, 2006.
- [28] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 531–540, 2018.
- [29] P. Bickel. Some contributions to theory of order statistics. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pages 575–591, 1967.
- [30] P. Bickel and K. Doksum. Mathematical Statistics. CRC, London, 2015.
- [31] L. Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.
- [32] Kelly Markelle, Longjohn Rachel, and Nottingham Kolby. UCI Machine Learning Repository, 2023.
- [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, NIPS’19, pages 8024–8035, 2019.