Fairness Auditing with Multi-Agent Collaboration
Authors:
Martijn de Vos,
Akash Dhasade,
Jade Garcia Bourrée,
Anne-Marie Kermarrec,
Erwan Le Merrer,
Benoit Rottembourg,
Gilles Tredan
Abstract:
Existing work in fairness auditing assumes that each audit is performed independently. In this paper, we consider multiple agents working together, each auditing the same platform for different tasks. Agents have two levers: their collaboration strategy, with or without coordination beforehand, and their strategy for sampling appropriate data points. We theoretically compare the interplay of these…
▽ More
Existing work in fairness auditing assumes that each audit is performed independently. In this paper, we consider multiple agents working together, each auditing the same platform for different tasks. Agents have two levers: their collaboration strategy, with or without coordination beforehand, and their strategy for sampling appropriate data points. We theoretically compare the interplay of these levers. Our main findings are that (i) collaboration is generally beneficial for accurate audits, (ii) basic sampling methods often prove to be effective, and (iii) counter-intuitively, extensive coordination on queries often deteriorates audits accuracy as the number of agents increases. Experiments on three large datasets confirm our theoretical results. Our findings motivate collaboration during fairness audits of platforms that use ML models for decision-making.
△ Less
Submitted 26 April, 2024; v1 submitted 13 February, 2024;
originally announced February 2024.
On the relevance of APIs facing fairwashed audits
Authors:
Jade Garcia Bourrée,
Erwan Le Merrer,
Gilles Tredan,
Benoît Rottembourg
Abstract:
Recent legislation required AI platforms to provide APIs for regulators to assess their compliance with the law. Research has nevertheless shown that platforms can manipulate their API answers through fairwashing. Facing this threat for reliable auditing, this paper studies the benefits of the joint use of platform scra** and of APIs. In this setup, we elaborate on the use of scra** to detect…
▽ More
Recent legislation required AI platforms to provide APIs for regulators to assess their compliance with the law. Research has nevertheless shown that platforms can manipulate their API answers through fairwashing. Facing this threat for reliable auditing, this paper studies the benefits of the joint use of platform scra** and of APIs. In this setup, we elaborate on the use of scra** to detect manipulated answers: since fairwashing only manipulates API answers, exploiting scraps may reveal a manipulation. To abstract the wide range of specific API-scrap situations, we introduce a notion of proxy that captures the consistency an auditor might expect between both data sources. If the regulator has a good proxy of the consistency, then she can easily detect manipulation and even bypass the API to conduct her audit. On the other hand, without a good proxy, relying on the API is necessary, and the auditor cannot defend against fairwashing.
We then simulate practical scenarios in which the auditor may mostly rely on the API to conveniently conduct the audit task, while maintaining her chances to detect a potential manipulation. To highlight the tension between the audit task and the API fairwashing detection task, we identify Pareto-optimal strategies in a practical audit scenario.
We believe this research sets the stage for reliable audits in practical and manipulation-prone setups.
△ Less
Submitted 23 May, 2023;
originally announced May 2023.
