-
A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
Authors:
Tiago Salvador,
Kilian Fatras,
Ioannis Mitliagkas,
Adam Oberman
Abstract:
Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training…
▽ More
Unsupervised Domain Adaptation (UDA) aims at classifying unlabeled target images leveraging source labeled ones. In this work, we consider the Partial Domain Adaptation (PDA) variant, where we have extra source classes not present in the target domain. Most successful algorithms use model selection strategies that rely on target labels to find the best hyper-parameters and/or models along training. However, these strategies violate the main assumption in PDA: only unlabeled target domain samples are available. Moreover, there are also inconsistencies in the experimental settings - architecture, hyper-parameter tuning, number of runs - yielding unfair comparisons. The main goal of this work is to provide a realistic evaluation of PDA methods with the different model selection strategies under a consistent evaluation protocol. We evaluate 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. Our two main findings are: (i) without target labels for model selection, the accuracy of the methods decreases up to 30 percentage points; (ii) only one method and model selection pair performs well on both datasets. Experiments were performed with our PyTorch framework, BenchmarkPDA, which we open source.
△ Less
Submitted 3 October, 2022;
originally announced October 2022.
-
Frustratingly Easy Uncertainty Estimation for Distribution Shift
Authors:
Tiago Salvador,
Vikram Voleti,
Alexander Iannantuono,
Adam Oberman
Abstract:
Distribution shift is an important concern in deep image classification, produced either by corruption of the source images, or a complete change, with the solution involving domain adaptation. While the primary goal is to improve accuracy under distribution shift, an important secondary goal is uncertainty estimation: evaluating the probability that the prediction of a model is correct. While imp…
▽ More
Distribution shift is an important concern in deep image classification, produced either by corruption of the source images, or a complete change, with the solution involving domain adaptation. While the primary goal is to improve accuracy under distribution shift, an important secondary goal is uncertainty estimation: evaluating the probability that the prediction of a model is correct. While improving accuracy is hard, uncertainty estimation turns out to be frustratingly easy. Prior works have appended uncertainty estimation into the model and training paradigm in various ways. Instead, we show that we can estimate uncertainty by simply exposing the original model to corrupted images, and performing simple statistical calibration on the image outputs. Our frustratingly easy methods demonstrate superior performance on a wide range of distribution shifts as well as on unsupervised domain adaptation tasks, measured through extensive experimentation.
△ Less
Submitted 17 October, 2021; v1 submitted 7 June, 2021;
originally announced June 2021.
-
FairCal: Fairness Calibration for Face Verification
Authors:
Tiago Salvador,
Stephanie Cairns,
Vikram Voleti,
Noah Marshall,
Adam Oberman
Abstract:
Despite being widely used, face recognition models suffer from bias: the probability of a false positive (incorrect face match) strongly depends on sensitive attributes such as the ethnicity of the face. As a result, these models can disproportionately and negatively impact minority groups, particularly when used by law enforcement. The majority of bias reduction methods have several drawbacks: th…
▽ More
Despite being widely used, face recognition models suffer from bias: the probability of a false positive (incorrect face match) strongly depends on sensitive attributes such as the ethnicity of the face. As a result, these models can disproportionately and negatively impact minority groups, particularly when used by law enforcement. The majority of bias reduction methods have several drawbacks: they use an end-to-end retraining approach, may not be feasible due to privacy issues, and often reduce accuracy. An alternative approach is post-processing methods that build fairer decision classifiers using the features of pre-trained models, thus avoiding the cost of retraining. However, they still have drawbacks: they reduce accuracy (AGENDA, PASS, FTC), or require retuning for different false positive rates (FSN). In this work, we introduce the Fairness Calibration (FairCal) method, a post-training approach that simultaneously: (i) increases model accuracy (improving the state-of-the-art), (ii) produces fairly-calibrated probabilities, (iii) significantly reduces the gap in the false positive rates, (iv) does not require knowledge of the sensitive attribute, and (v) does not require retraining, training an additional model, or retuning. We apply it to the task of Face Verification, and obtain state-of-the-art results with all the above advantages.
△ Less
Submitted 30 March, 2022; v1 submitted 7 June, 2021;
originally announced June 2021.
-
A Partial Differential Equation Obstacle Problem for the Level Set Approach to Visibility
Authors:
Adam Oberman,
Tiago Salvador
Abstract:
In this article we consider the problem of finding the visibility set from a given point when the obstacles are represented as the level set of a given function. Although the visibility set can be computed efficiently by ray tracing, there are advantages to using a level set representation for the obstacles, and to characterizing the solution using a Partial Differential Equation (PDE). A nonlocal…
▽ More
In this article we consider the problem of finding the visibility set from a given point when the obstacles are represented as the level set of a given function. Although the visibility set can be computed efficiently by ray tracing, there are advantages to using a level set representation for the obstacles, and to characterizing the solution using a Partial Differential Equation (PDE). A nonlocal PDE formulation was proposed in Tsai et. al. (Journal of Computational Physics 199(1):260-290, 2004): in this article we propose a simpler PDE formulation, involving a nonlinear obstacle problem. We present a simple numerical scheme and show its convergence using the framework of Barles and Souganidis. Numerical examples in both two and three dimensions are presented.
△ Less
Submitted 1 August, 2019;
originally announced August 2019.
-
The Role of Surface Tension and Mobility Model in Simulations of Grain Growth
Authors:
Tiago Salvador,
Selim Esedoglu
Abstract:
We explore the effects of surface tension and mobility models in simulations of grain growth using threshold dynamics algorithms that allow performing large scale simulations, while naturally capturing the Herring angle condition at junctions and automatically handling topological transitions. The results indicate that in two dimensions, the different surface tension / mobility models considered d…
▽ More
We explore the effects of surface tension and mobility models in simulations of grain growth using threshold dynamics algorithms that allow performing large scale simulations, while naturally capturing the Herring angle condition at junctions and automatically handling topological transitions. The results indicate that in two dimensions, the different surface tension / mobility models considered do not play a significant role in the stationary grain size distribution. However, in three dimensions, there is a substantial difference between the distributions obtained from the same three models, depending on whether the reduced mobilities are isotropic or anisotropic. Additional results show that in three dimensions, the misorientation distribution function of a grain network with random orientation texture returns to the close vicinity of the Mackenzie distribution even if started very far from it.
△ Less
Submitted 26 July, 2019;
originally announced July 2019.
-
Calibrated Top-1 Uncertainty estimates for classification by score based models
Authors:
Adam M. Oberman,
Chris Finlay,
Alexander Iannantuono,
Tiago Salvador
Abstract:
While the accuracy of modern deep learning models has significantly improved in recent years, the ability of these models to generate uncertainty estimates has not progressed to the same degree. Uncertainty methods are designed to provide an estimate of class probabilities when predicting class assignment.
While there are a number of proposed methods for estimating uncertainty, they all suffer f…
▽ More
While the accuracy of modern deep learning models has significantly improved in recent years, the ability of these models to generate uncertainty estimates has not progressed to the same degree. Uncertainty methods are designed to provide an estimate of class probabilities when predicting class assignment.
While there are a number of proposed methods for estimating uncertainty, they all suffer from a lack of calibration: predicted probabilities can be off from empirical ones by a few percent or more. By restricting the scope of our predictions to only the probability of Top-1 error, we can decrease the calibration error of existing methods to less than one percent. As a result, the scores of the methods also improve significantly over benchmarks.
△ Less
Submitted 16 June, 2020; v1 submitted 21 March, 2019;
originally announced March 2019.
-
A simplified threshold dynamics algorithm for isotropic surface energies
Authors:
Tiago Salvador,
Selim Esedoglu
Abstract:
We present a simplified version of the threshold dynamics algorithm given in the work of Esedoglu and Otto (2015). The new version still allows specifying N-choose-2 possibly distinct surface tensions and N-choose-2 possibly distinct mobilities for a network with N phases, but achieves this level of generality without the use of retardation functions. Instead, it employs linear combinations of Gau…
▽ More
We present a simplified version of the threshold dynamics algorithm given in the work of Esedoglu and Otto (2015). The new version still allows specifying N-choose-2 possibly distinct surface tensions and N-choose-2 possibly distinct mobilities for a network with N phases, but achieves this level of generality without the use of retardation functions. Instead, it employs linear combinations of Gaussians in the convolution step of the algorithm. Convolutions with only two distinct Gaussians is enough for the entire network, maintaining the efficiency of the original thresholding scheme. We discuss stability and convergence of the new algorithm, including some counterexamples in which convergence fails. The apparently convergent cases include unequal surface tensions given by the Read \& Shockley model and its three dimensional extensions, along with equal mobilities, that are a very common choice in computational materials science.
△ Less
Submitted 17 July, 2018;
originally announced July 2018.
-
Higher-order Adaptive Finite Difference Methods for Fully Nonlinear Elliptic Equations
Authors:
Brittany D. Froese,
Tiago Salvador
Abstract:
We introduce generalised finite difference methods for solving fully nonlinear elliptic partial differential equations. Methods are based on piecewise Cartesian meshes augmented by additional points along the boundary. This allows for adaptive meshes and complicated geometries, while still ensuring consistency, monotonicity, and convergence. We describe an algorithm for efficiently computing the n…
▽ More
We introduce generalised finite difference methods for solving fully nonlinear elliptic partial differential equations. Methods are based on piecewise Cartesian meshes augmented by additional points along the boundary. This allows for adaptive meshes and complicated geometries, while still ensuring consistency, monotonicity, and convergence. We describe an algorithm for efficiently computing the non-traditional finite difference stencils. We also present a strategy for computing formally higher-order convergent methods. Computational examples demonstrate the efficiency, accuracy, and flexibility of the methods.
△ Less
Submitted 23 June, 2017;
originally announced June 2017.
-
Numerical methods for motion of level sets by affine curvature
Authors:
Adam M. Oberman,
Tiago Salvador
Abstract:
We study numerical methods for the nonlinear partial differential equation that governs the motion of level sets by affine curvature. We show that standard finite difference schemes are nonlinearly unstable. We build convergent finite difference schemes, using the theory of viscosity solutions. We demonstrate that our approximate solutions capture the affine invariance and morphological properties…
▽ More
We study numerical methods for the nonlinear partial differential equation that governs the motion of level sets by affine curvature. We show that standard finite difference schemes are nonlinearly unstable. We build convergent finite difference schemes, using the theory of viscosity solutions. We demonstrate that our approximate solutions capture the affine invariance and morphological properties of the evolution. Numerical experiments demonstrate the accuracy and stability of the discretization.
△ Less
Submitted 28 October, 2016; v1 submitted 27 October, 2016;
originally announced October 2016.
-
Numerical Methods for the 2-Hessian Elliptic Partial Differential Equation
Authors:
Brittany D. Froese,
Adam M. Oberman,
Tiago Salvador
Abstract:
The elliptic 2-Hessian equation is a fully nonlinear partial differential equation (PDE) that is related to intrinsic curvature for three dimensional manifolds. We introduce two numerical methods for this PDE: the first is provably convergent to the viscosity solution, and the second is more accurate, and convergent in practice but lacks a proof. The PDE is elliptic on a restricted set of function…
▽ More
The elliptic 2-Hessian equation is a fully nonlinear partial differential equation (PDE) that is related to intrinsic curvature for three dimensional manifolds. We introduce two numerical methods for this PDE: the first is provably convergent to the viscosity solution, and the second is more accurate, and convergent in practice but lacks a proof. The PDE is elliptic on a restricted set of functions: a convexity type constraint is needed for the ellipticity of the PDE operator. Solutions with both discretizations are obtained using Newton's method. Computational results are presented on a number of exact solutions which range in regularity from smooth to nondifferentiable and in shape from convex to non convex.
△ Less
Submitted 10 February, 2016; v1 submitted 17 February, 2015;
originally announced February 2015.
-
Filtered schemes for Hamilton-Jacobi equations: a simple construction of convergent accurate difference schemes
Authors:
Adam M. Oberman,
Tiago Salvador
Abstract:
We build a simple and general class of finite difference schemes for first order Hamilton-Jacobi (HJ) Partial Differential Equations. These filtered schemes are convergent to the unique viscosity solution of the equation. The schemes are accurate: we implement second, third and fourth order accurate schemes in one dimension and second order accurate schemes in two dimensions, indicating how to bui…
▽ More
We build a simple and general class of finite difference schemes for first order Hamilton-Jacobi (HJ) Partial Differential Equations. These filtered schemes are convergent to the unique viscosity solution of the equation. The schemes are accurate: we implement second, third and fourth order accurate schemes in one dimension and second order accurate schemes in two dimensions, indicating how to build higher order ones. They are also explicit, which means they can be solved using the fast swee** method or the fast marching method.The accuracy of the method is validated with computational results for the eikonal equation in one and two dimensions, using filtered schemes made from standard centered differences, higher order upwinding and ENO interpolation.
△ Less
Submitted 12 November, 2014;
originally announced November 2014.
