-
Superhuman Fairness
Authors:
Omid Memarrast,
Linh Vu,
Brian Ziebart
Abstract:
The fairness of machine learning-based decisions has become an increasingly important focus in the design of supervised machine learning methods. Most fairness approaches optimize a specified trade-off between performance measure(s) (e.g., accuracy, log loss, or AUC) and fairness metric(s) (e.g., demographic parity, equalized odds). This begs the question: are the right performance-fairness trade-…
▽ More
The fairness of machine learning-based decisions has become an increasingly important focus in the design of supervised machine learning methods. Most fairness approaches optimize a specified trade-off between performance measure(s) (e.g., accuracy, log loss, or AUC) and fairness metric(s) (e.g., demographic parity, equalized odds). This begs the question: are the right performance-fairness trade-offs being specified? We instead re-cast fair machine learning as an imitation learning task by introducing superhuman fairness, which seeks to simultaneously outperform human decisions on multiple predictive performance and fairness measures. We demonstrate the benefits of this approach given suboptimal decisions.
△ Less
Submitted 31 January, 2023;
originally announced January 2023.
-
Fairness for Robust Learning to Rank
Authors:
Omid Memarrast,
Ashkan Rezaei,
Rizal Fathony,
Brian Ziebart
Abstract:
While conventional ranking systems focus solely on maximizing the utility of the ranked items to users, fairness-aware ranking systems additionally try to balance the exposure for different protected attributes such as gender or race. To achieve this type of group fairness for ranking, we derive a new ranking system based on the first principles of distributional robustness. We formulate a minimax…
▽ More
While conventional ranking systems focus solely on maximizing the utility of the ranked items to users, fairness-aware ranking systems additionally try to balance the exposure for different protected attributes such as gender or race. To achieve this type of group fairness for ranking, we derive a new ranking system based on the first principles of distributional robustness. We formulate a minimax game between a player choosing a distribution over rankings to maximize utility while satisfying fairness constraints against an adversary seeking to minimize utility while matching statistics of the training data. We show that our approach provides better utility for highly fair rankings than existing baseline methods.
△ Less
Submitted 12 December, 2021;
originally announced December 2021.
-
ParsiNLU: A Suite of Language Understanding Challenges for Persian
Authors:
Daniel Khashabi,
Arman Cohan,
Siamak Shakeri,
Pedram Hosseini,
Pouya Pezeshkpour,
Malihe Alikhani,
Moin Aminnaseri,
Marzieh Bitaab,
Faeze Brahman,
Sarik Ghazarian,
Mozhdeh Gheini,
Arman Kabiri,
Rabeeh Karimi Mahabadi,
Omid Memarrast,
Ahmadreza Mosallanezhad,
Erfan Noury,
Shahab Raji,
Mohammad Sadegh Rasooli,
Sepideh Sadeghi,
Erfan Sadeqi Azer,
Niloofar Safi Samghabadi,
Mahsa Shafaei,
Saber Sheybani,
Ali Tazarv,
Yadollah Yaghoobzadeh
Abstract:
Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this rich language. The availability of high-quality evaluat…
▽ More
Despite the progress made in recent years in addressing natural language understanding (NLU) challenges, the majority of this progress remains to be concentrated on resource-rich languages like English. This work focuses on Persian language, one of the widely spoken languages in the world, and yet there are few NLU datasets available for this rich language. The availability of high-quality evaluation datasets is a necessity for reliable assessment of the progress on different NLU tasks and domains. We introduce ParsiNLU, the first benchmark in Persian language that includes a range of high-level tasks -- Reading Comprehension, Textual Entailment, etc. These datasets are collected in a multitude of ways, often involving manual annotations by native speakers. This results in over 14.5$k$ new instances across 6 distinct NLU tasks. Besides, we present the first results on state-of-the-art monolingual and multi-lingual pre-trained language-models on this benchmark and compare them with human performance, which provides valuable insights into our ability to tackle natural language understanding challenges in Persian. We hope ParsiNLU fosters further research and advances in Persian language understanding.
△ Less
Submitted 13 July, 2021; v1 submitted 11 December, 2020;
originally announced December 2020.
-
Robust Fairness under Covariate Shift
Authors:
Ashkan Rezaei,
Anqi Liu,
Omid Memarrast,
Brian Ziebart
Abstract:
Making predictions that are fair with regard to protected group membership (race, gender, age, etc.) has become an important requirement for classification algorithms. Existing techniques derive a fair model from sampled labeled data relying on the assumption that training and testing data are identically and independently drawn (iid) from the same distribution. In practice, distribution shift can…
▽ More
Making predictions that are fair with regard to protected group membership (race, gender, age, etc.) has become an important requirement for classification algorithms. Existing techniques derive a fair model from sampled labeled data relying on the assumption that training and testing data are identically and independently drawn (iid) from the same distribution. In practice, distribution shift can and does occur between training and testing datasets as the characteristics of individuals interacting with the machine learning system change. We investigate fairness under covariate shift, a relaxation of the iid assumption in which the inputs or covariates change while the conditional label distribution remains the same. We seek fair decisions under these assumptions on target data with unknown labels. We propose an approach that obtains the predictor that is robust to the worst-case in terms of target performance while satisfying target fairness requirements and matching statistical properties of the source data. We demonstrate the benefits of our approach on benchmark prediction tasks.
△ Less
Submitted 6 February, 2021; v1 submitted 11 October, 2020;
originally announced October 2020.
-
Fairness for Robust Log Loss Classification
Authors:
Ashkan Rezaei,
Rizal Fathony,
Omid Memarrast,
Brian Ziebart
Abstract:
Develo** classification methods with high accuracy that also avoid unfair treatment of different groups has become increasingly important for data-driven decision making in social applications. Many existing methods enforce fairness constraints on a selected classifier (e.g., logistic regression) by directly forming constrained optimizations. We instead re-derive a new classifier from the first…
▽ More
Develo** classification methods with high accuracy that also avoid unfair treatment of different groups has become increasingly important for data-driven decision making in social applications. Many existing methods enforce fairness constraints on a selected classifier (e.g., logistic regression) by directly forming constrained optimizations. We instead re-derive a new classifier from the first principles of distributional robustness that incorporates fairness criteria into a worst-case logarithmic loss minimization. This construction takes the form of a minimax game and produces a parametric exponential family conditional distribution that resembles truncated logistic regression. We present the theoretical benefits of our approach in terms of its convexity and asymptotic convergence. We then demonstrate the practical advantages of our approach on three benchmark fairness datasets.
△ Less
Submitted 14 October, 2020; v1 submitted 9 March, 2019;
originally announced March 2019.
