-
Can I trust my fake data -- A comprehensive quality assessment framework for synthetic tabular data in healthcare
Authors:
Vibeke Binz Vallevik,
Aleksandar Babic,
Serena Elizabeth Marshall,
Severin Elvatun,
Helga Brøgger,
Sharmini Alagaratnam,
Bjørn Edwin,
Narasimha Raghavan Veeraragavan,
Anne Kjersti Befring,
Jan Franz Nygård
Abstract:
Ensuring safe adoption of AI tools in healthcare hinges on access to sufficient data for training, testing and validation. In response to privacy concerns and regulatory requirements, using synthetic data has been suggested. Synthetic data is created by training a generator on real data to produce a dataset with similar statistical properties. Competing metrics with differing taxonomies for qualit…
▽ More
Ensuring safe adoption of AI tools in healthcare hinges on access to sufficient data for training, testing and validation. In response to privacy concerns and regulatory requirements, using synthetic data has been suggested. Synthetic data is created by training a generator on real data to produce a dataset with similar statistical properties. Competing metrics with differing taxonomies for quality evaluation have been suggested, resulting in a complex landscape. Optimising quality entails balancing considerations that make the data fit for use, yet relevant dimensions are left out of existing frameworks. We performed a comprehensive literature review on the use of quality evaluation metrics on SD within the scope of tabular healthcare data and SD made using deep generative methods. Based on this and the collective team experiences, we developed a conceptual framework for quality assurance. The applicability was benchmarked against a practical case from the Dutch National Cancer Registry. We present a conceptual framework for quality assurance of SD for AI applications in healthcare that aligns diverging taxonomies, expands on common quality dimensions to include the dimensions of Fairness and Carbon footprint, and proposes stages necessary to support real-life applications. Building trust in synthetic data by increasing transparency and reducing the safety risk will accelerate the development and uptake of trustworthy AI tools for the benefit of patients. Despite the growing emphasis on algorithmic fairness and carbon footprint, these metrics were scarce in the literature review. The overwhelming focus was on statistical similarity using distance metrics while sequential logic detection was scarce. A consensus-backed framework that includes all relevant quality dimensions can provide assurance for safe and responsible real-life applications of SD.
△ Less
Submitted 24 January, 2024;
originally announced January 2024.
-
Flip** the Script on Criminal Justice Risk Assessment: An actuarial model for assessing the risk the federal sentencing system poses to defendants
Authors:
Mikaela Meyer,
Aaron Horowitz,
Erica Marshall,
Kristian Lum
Abstract:
In the criminal justice system, algorithmic risk assessment instruments are used to predict the risk a defendant poses to society; examples include the risk of recidivating or the risk of failing to appear at future court dates. However, defendants are also at risk of harm from the criminal justice system. To date, there exists no risk assessment instrument that considers the risk the system poses…
▽ More
In the criminal justice system, algorithmic risk assessment instruments are used to predict the risk a defendant poses to society; examples include the risk of recidivating or the risk of failing to appear at future court dates. However, defendants are also at risk of harm from the criminal justice system. To date, there exists no risk assessment instrument that considers the risk the system poses to the individual. We develop a risk assessment instrument that "flips the script." Using data about U.S. federal sentencing decisions, we build a risk assessment instrument that predicts the likelihood an individual will receive an especially lengthy sentence given factors that should be legally irrelevant to the sentencing decision. To do this, we develop a two-stage modeling approach. Our first-stage model is used to determine which sentences were "especially lengthy." We then use a second-stage model to predict the defendant's risk of receiving a sentence that is flagged as especially lengthy given factors that should be legally irrelevant. The factors that should be legally irrelevant include, for example, race, court location, and other socio-demographic information about the defendant. Our instrument achieves comparable predictive accuracy to risk assessment instruments used in pretrial and parole contexts. We discuss the limitations of our modeling approach and use the opportunity to highlight how traditional risk assessment instruments in various criminal justice settings also suffer from many of the same limitations and embedded value systems of their creators.
△ Less
Submitted 13 July, 2022; v1 submitted 26 May, 2022;
originally announced May 2022.
-
The Characterization of planar, 4-connected, K_{2,5}-minor-free graphs
Authors:
Emily Abernethy Marshall,
Liana Yepremyan,
Zach Gaslowitz
Abstract:
We show that every planar, 4-connected, K2;5-minor- free graph is the square of a cycle of even length at least six.
We show that every planar, 4-connected, K2;5-minor- free graph is the square of a cycle of even length at least six.
△ Less
Submitted 20 August, 2015; v1 submitted 24 July, 2015;
originally announced July 2015.
