-
Safe AI for health and beyond -- Monitoring to transform a health service
Authors:
Mahed Abroshan,
Michael Burkhart,
Oscar Giles,
Sam Greenbury,
Zoe Kourtzi,
Jack Roberts,
Mihaela van der Schaar,
Jannetta S Steyn,
Alan Wilson,
May Yong
Abstract:
Machine learning techniques are effective for building predictive models because they identify patterns in large datasets. Development of a model for complex real-life problems often stop at the point of publication, proof of concept or when made accessible through some mode of deployment. However, a model in the medical domain risks becoming obsolete as patient demographics, systems and clinical…
▽ More
Machine learning techniques are effective for building predictive models because they identify patterns in large datasets. Development of a model for complex real-life problems often stop at the point of publication, proof of concept or when made accessible through some mode of deployment. However, a model in the medical domain risks becoming obsolete as patient demographics, systems and clinical practices change. The maintenance and monitoring of predictive model performance post-publication is crucial to enable their safe and effective long-term use. We will assess the infrastructure required to monitor the outputs of a machine learning algorithm, and present two scenarios with examples of monitoring and updates of models, firstly on a breast cancer prognosis model trained on public longitudinal data, and secondly on a neurodegenerative stratification algorithm that is currently being developed and tested in clinic.
△ Less
Submitted 6 June, 2023; v1 submitted 2 March, 2023;
originally announced March 2023.
-
Faking feature importance: A cautionary tale on the use of differentially-private synthetic data
Authors:
Oscar Giles,
Kasra Hosseini,
Grigorios Mingas,
Oliver Strickson,
Louise Bowler,
Camila Rangel Smith,
Harrison Wilde,
Jen Ning Lim,
Bilal Mateen,
Kasun Amarasinghe,
Rayid Ghani,
Alison Heppenstall,
Nik Lomax,
Nick Malleson,
Martin O'Reilly,
Sebastian Vollmerteke
Abstract:
Synthetic datasets are often presented as a silver-bullet solution to the problem of privacy-preserving data publishing. However, for many applications, synthetic data has been shown to have limited utility when used to train predictive models. One promising potential application of these data is in the exploratory phase of the machine learning workflow, which involves understanding, engineering a…
▽ More
Synthetic datasets are often presented as a silver-bullet solution to the problem of privacy-preserving data publishing. However, for many applications, synthetic data has been shown to have limited utility when used to train predictive models. One promising potential application of these data is in the exploratory phase of the machine learning workflow, which involves understanding, engineering and selecting features. This phase often involves considerable time, and depends on the availability of data. There would be substantial value in synthetic data that permitted these steps to be carried out while, for example, data access was being negotiated, or with fewer information governance restrictions. This paper presents an empirical analysis of the agreement between the feature importance obtained from raw and from synthetic data, on a range of artificially generated and real-world datasets (where feature importance represents how useful each feature is when predicting a the outcome). We employ two differentially-private methods to produce synthetic data, and apply various utility measures to quantify the agreement in feature importance as this varies with the level of privacy. Our results indicate that synthetic data can sometimes preserve several representations of the ranking of feature importance in simple settings but their performance is not consistent and depends upon a number of factors. Particular caution should be exercised in more nuanced real-world settings, where synthetic data can lead to differences in ranked feature importance that could alter key modelling decisions. This work has important implications for develo** synthetic versions of highly sensitive data sets in fields such as finance and healthcare.
△ Less
Submitted 2 March, 2022;
originally announced March 2022.
-
Near Real-Time Social Distance Estimation in London
Authors:
James Walsh,
Oluwafunmilola Kesa,
Andrew Wang,
Mihai Ilas,
Patrick O'Hara,
Oscar Giles,
Neil Dhir,
Mark Girolami,
Theodoros Damoulas
Abstract:
During the COVID-19 pandemic, policy makers at the Greater London Authority, the regional governance body of London, UK, are reliant upon prompt and accurate data sources. Large well-defined heterogeneous compositions of activity throughout the city are sometimes difficult to acquire, yet are a necessity in order to learn 'busyness' and consequently make safe policy decisions. One component of our…
▽ More
During the COVID-19 pandemic, policy makers at the Greater London Authority, the regional governance body of London, UK, are reliant upon prompt and accurate data sources. Large well-defined heterogeneous compositions of activity throughout the city are sometimes difficult to acquire, yet are a necessity in order to learn 'busyness' and consequently make safe policy decisions. One component of our project within this space is to utilise existing infrastructure to estimate social distancing adherence by the general public. Our method enables near immediate sampling and contextualisation of activity and physical distancing on the streets of London via live traffic camera feeds. We introduce a framework for inspecting and improving upon existing methods, whilst also describing its active deployment on over 900 real-time feeds.
△ Less
Submitted 14 August, 2022; v1 submitted 7 December, 2020;
originally announced December 2020.
-
An Expectation-Based Network Scan Statistic for a COVID-19 Early Warning System
Authors:
Chance Haycock,
Edward Thorpe-Woods,
James Walsh,
Patrick O'Hara,
Oscar Giles,
Neil Dhir,
Theodoros Damoulas
Abstract:
One of the Greater London Authority's (GLA) response to the COVID-19 pandemic brings together multiple large-scale and heterogeneous datasets capturing mobility, transportation and traffic activity over the city of London to better understand 'busyness' and enable targeted interventions and effective policy-making. As part of Project Odysseus we describe an early-warning system and introduce an ex…
▽ More
One of the Greater London Authority's (GLA) response to the COVID-19 pandemic brings together multiple large-scale and heterogeneous datasets capturing mobility, transportation and traffic activity over the city of London to better understand 'busyness' and enable targeted interventions and effective policy-making. As part of Project Odysseus we describe an early-warning system and introduce an expectation-based scan statistic for networks to help the GLA and Transport for London, understand the extent to which populations are following government COVID-19 guidelines. We explicitly treat the case of geographically fixed time-series data located on a (road) network and primarily focus on monitoring the dynamics across large regions of the capital. Additionally, we also focus on the detection and reporting of significant spatio-temporal regions. Our approach is extending the Network Based Scan Statistic (NBSS) by making it expectation-based (EBP) and by using stochastic processes for time-series forecasting, which enables us to quantify metric uncertainty in both the EBP and NBSS frameworks. We introduce a variant of the metric used in the EBP model which focuses on identifying space-time regions in which activity is quieter than expected.
△ Less
Submitted 8 December, 2020;
originally announced December 2020.
-
Modelling visual-vestibular integration and behavioural adaptation in the driving simulator
Authors:
Gustav Markkula,
Richard Romano,
Rachel Waldram,
Oscar Giles,
Callum Mole,
Richard Wilkie
Abstract:
It is well established that not only vision but also other sensory modalities affect drivers' control of their vehicles, and that drivers adapt over time to persistent changes in sensory cues (for example in driving simulators), but the mechanisms underlying these behavioural phenomena are poorly understood. Here, we consider the existing literature on how driver steering in slalom tasks is affect…
▽ More
It is well established that not only vision but also other sensory modalities affect drivers' control of their vehicles, and that drivers adapt over time to persistent changes in sensory cues (for example in driving simulators), but the mechanisms underlying these behavioural phenomena are poorly understood. Here, we consider the existing literature on how driver steering in slalom tasks is affected by the down-scaling of vestibular cues, and propose a driver model that can explain the empirically observed effects, namely: decreased task performance and increased steering effort during initial exposure, followed by a partial reversal of these effects as task exposure is prolonged. Unexpectedly, the model also reproduced another empirical finding: a local optimum for motion down-scaling, where path-tracking is better than when one-to-one motion cues are available. Overall, the results imply that: (1) drivers make direct use of vestibular information as part of determining appropriate steering, and (2) motion down-scaling causes a yaw rate underestimation phenomenon, where drivers behave as if the simulated vehicle is rotating more slowly than it is. However, (3) in the slalom task, a certain degree of such yaw rate underestimation is beneficial to path tracking performance. Furthermore, (4) behavioural adaptation, as empirically observed in slalom tasks, may occur due to (a) down-weighting of vestibular cues, and/or (b) increased sensitivity to control errors, in determining when to adjust steering and by how much, but (c) seemingly not in the form of a full compensatory rescaling of the received vestibular input. The analyses presented here provide new insights and hypotheses about simulator driving, and the developed models can be used to support research on multisensory integration and behavioural adaptation in both driving and other task domains.
△ Less
Submitted 7 November, 2018; v1 submitted 29 October, 2018;
originally announced October 2018.
