-
Capturing Requirements for a Data Annotation Tool for Intensive Care: Experimental User-Centered Design Study
Authors:
Marceli Wac,
Raul Santos-Rodriguez,
Chris McWilliams,
Christopher Bourdeaux
Abstract:
Intensive care units (ICUs) are complex and data-rich environments. Data routinely collected in the ICUs provides tremendous opportunities for machine learning, but their use comes with significant challenges. Complex problems may require additional input from humans which can be provided through a process of data annotation. Annotation is a complex, time-consuming process that requires domain exp…
▽ More
Intensive care units (ICUs) are complex and data-rich environments. Data routinely collected in the ICUs provides tremendous opportunities for machine learning, but their use comes with significant challenges. Complex problems may require additional input from humans which can be provided through a process of data annotation. Annotation is a complex, time-consuming process that requires domain expertise and technical proficiency. Existing data annotation tools fail to provide an effective solution to this problem. In this study, we investigated clinicians' approach to the annotation task. We focused on establishing the characteristics of the annotation process in the context of clinical data and identifying differences in the annotation workflow between different staff roles. The overall goal was to elicit requirements for a software tool that could facilitate an effective and time-efficient data annotation. We conducted an experiment involving clinicians from the ICUs annotating printed sheets of data. The participants were observed during the task and their actions were analysed in the context of Norman's Interaction Cycle to establish the requirements for the digital tool. The annotation process followed a constant loop of annotation and evaluation, during which participants incrementally analysed and annotated the data. No distinguishable differences were identified between how different staff roles annotate data. We observed preferences towards different methods for applying annotation which varied between different participants and admissions. We established 11 requirements for the digital data annotation tool for the healthcare setting. We conducted a manual data annotation activity to establish the requirements for a digital data annotation tool, characterised the clinicians' approach to annotation and elicited 11 key requirements for effective data annotation software.
△ Less
Submitted 4 June, 2024; v1 submitted 28 September, 2023;
originally announced September 2023.
-
TraCE: Trajectory Counterfactual Explanation Scores
Authors:
Jeffrey N. Clark,
Edward A. Small,
Nawid Keshtmand,
Michelle W. L. Wan,
Elena Fillola Mayoral,
Enrico Werner,
Christopher P. Bourdeaux,
Raul Santos-Rodriguez
Abstract:
Counterfactual explanations, and their associated algorithmic recourse, are typically leveraged to understand, explain, and potentially alter a prediction coming from a black-box classifier. In this paper, we propose to extend the use of counterfactuals to evaluate progress in sequential decision making tasks. To this end, we introduce a model-agnostic modular framework, TraCE (Trajectory Counterf…
▽ More
Counterfactual explanations, and their associated algorithmic recourse, are typically leveraged to understand, explain, and potentially alter a prediction coming from a black-box classifier. In this paper, we propose to extend the use of counterfactuals to evaluate progress in sequential decision making tasks. To this end, we introduce a model-agnostic modular framework, TraCE (Trajectory Counterfactual Explanation) scores, which is able to distill and condense progress in highly complex scenarios into a single value. We demonstrate TraCE's utility across domains by showcasing its main properties in two case studies spanning healthcare and climate change.
△ Less
Submitted 26 January, 2024; v1 submitted 27 September, 2023;
originally announced September 2023.
-
Strategies for engaging clinical participants in the co-design of software for healthcare domains
Authors:
Marceli Wac,
Raul Santos-Rodriguez,
Chris McWilliams,
Christopher Bourdeaux
Abstract:
Co-design is an effective method for designing software, but implementing it within the clinical setting comes with a set of unique challenges. This makes recruitment and engagement of participants difficult, which has been demonstrated in our study. Our work focused on designing and evaluating a data annotation tool, however, different types of interventions had to be carried out due to poor enga…
▽ More
Co-design is an effective method for designing software, but implementing it within the clinical setting comes with a set of unique challenges. This makes recruitment and engagement of participants difficult, which has been demonstrated in our study. Our work focused on designing and evaluating a data annotation tool, however, different types of interventions had to be carried out due to poor engagement with the study. We evaluated the effectiveness and feasibility of each of these strategies, their applicability to different stages of co-design research and discussed the barriers to participation present among participants from a clinical background.
△ Less
Submitted 31 August, 2023;
originally announced August 2023.
-
Identification, explanation and clinical evaluation of hospital patient subtypes
Authors:
Enrico Werner,
Jeffrey N. Clark,
Ranjeet S. Bhamber,
Michael Ambler,
Christopher P. Bourdeaux,
Alexander Hepburn,
Christopher J. McWilliams,
Raul Santos-Rodriguez
Abstract:
We present a pipeline in which unsupervised machine learning techniques are used to automatically identify subtypes of hospital patients admitted between 2017 and 2021 in a large UK teaching hospital. With the use of state-of-the-art explainability techniques, the identified subtypes are interpreted and assigned clinical meaning. In parallel, clinicians assessed intra-cluster similarities and inte…
▽ More
We present a pipeline in which unsupervised machine learning techniques are used to automatically identify subtypes of hospital patients admitted between 2017 and 2021 in a large UK teaching hospital. With the use of state-of-the-art explainability techniques, the identified subtypes are interpreted and assigned clinical meaning. In parallel, clinicians assessed intra-cluster similarities and inter-cluster differences of the identified patient subtypes within the context of their clinical knowledge. By confronting the outputs of both automatic and clinician-based explanations, we aim to highlight the mutual benefit of combining machine learning techniques with clinical expertise.
△ Less
Submitted 19 January, 2023;
originally announced January 2023.