PhilHumans: Benchmarking Machine Learning for Personal Health

Abstract

The use of machine learning in Healthcare has the potential to improve patient outcomes as well as broaden the reach and affordability of Healthcare. The history of other application areas indicates that strong benchmarks are essential for the development of intelligent systems. We present Personal Health Interfaces Leveraging HUman-MAchine Natural interactions (PhilHumans), a holistic suite of benchmarks for machine learning across different Healthcare settings - talk therapy, diet coaching, emergency care, intensive care, obstetric sonography - as well as different learning settings, such as action anticipation, timeseries modeling, insight mining, language modeling, computer vision, reinforcement learning and program synthesis

1 Introduction

Understaffing has been consistently identified as the major challenge facing Healthcare today ashleyy.metcalfHospitalUnitUnderstaffing2016 ; SurveyShowsHidden1993 ; UnderstaffingSignificantIssue2012 ; campbell_universal_2013 ; hudsonUnderstaffing2015 ; mercerMessageEditorinChief2008 ; r.stanleyUnderstaffedOverwhelmed2010 ; munnUnderstaffingWardsCompromising2017 ; thelancetHealthcareSystemStaffing2018 . Automation tools that make use of Machine Learning (also known as Healthcare 4.0 tortorellaHealthcareTrendsChallenges2020 ) have been consistently identified as crucial for reducing the workload of Healthcare professionals and improving the quality of care agrawalMachineLearningHealthcare2020 ; deviDesignImplementationAdvanced2022 ; g.kumarSurveyMachineLearning2016 ; ganguliMachineLearningPursuit2020 ; maityMachineLearningImproved2017 ; mitraMachineLearningHealthcare2021 ; pianykhImprovingHealthcareOperations2020 ; xhaferraRoleMachineLearning2022 . In turn, the shortage of standard benchmarks has been consistently identified as a central roadblock for machine learning in Healthcare Crown2015Potential ; David2020Evaluating ; guSupervisedLearningPervasive2023 ; harutyunyanMultitaskLearningBenchmarking2019 ; Kathrin2022Benchmark ; liventsevEffectivePatientSimulators2021 ; mcdermottReproducibilityMachineLearning2021 ; purushothamBenchmarkingDeepLearning2018 ; S2017Benchmark .

Whether it’s ImageNet dengImagenetLargescaleHierarchical2009 in Computer Vision or GLUE wangGLUEMultitaskBenchmark2018 in natural language processing, benchmarks are a core research tool in mature applications of machine learning, enabling quantitative analysis of learning methodologies to guide and orient their development. Machine learning for Healthcare, an emergent field with unique challenges in availability of research datasets Anshik2021Handling ; Gilbert2015market ; Pahwa2021Big ; Yazhini2019State lacks an accepted benchmarking standard: recent literature reviews palMachineLearningHealthcare2023 ; tortorellaHealthcareTrendsChallenges2020 of the field cover a variety of studies that each use their own (often non-public) benchmark.

To address this, we propose a suite of tasks that each include a dataset and an evaluation procedure and cover a wide variety of both healthcare settings (therapy and coaching, emergency care, intensive care, sonography) and machine learning challenges (conversational agents, computer vision, time series prediction, reinforcement learning). We also provide a preliminary evaluation of widely used machine learning approaches on these tasks.

2 Benchmarks

2.1 Tabular perspective

Our first two benchmarks evaluate methods for machine learning on low-dimensional tabular data. This perspective restricts the types of data made available to learning algorithms, excluding for example, radiological data. At the same time, it is anything but contrived: many real-world tasks, such as risk assessment for negative outcomes in intensive care based on the patient’s vital signs, are within the tabular domain reynaEarlyPredictionSepsis2020 .

MIMIC-IV-Ext-SEQ

Reinforcement Learning in Healthcare is typically concerned with narrow self-contained tasks such as sepsis prediction or anesthesia control. However, previous research has demonstrated the potential of generalist models reedGeneralistAgent2022 (the prime example being Large Language Models brown2020:language ) to outperform task-specific approaches due to their capability for implicit transfer learning. To enable training of foundation models for Healthcare as well as leverage the capabilities of state of the art Transformer architectures, we propose the paradigm of Healthcare as Sequence Modeling, in which interaction between the patient and the healthcare provider is represented as an event stream and tasks like diagnosis and treatment selection are modeled as prediction of future events in the stream. To explore this paradigm experimentally we develop MIMIC-IV-Ext-SEQ liventsevIntensiveCareOne2024 , a soon-to-be-published sequence modeling benchmark derived by translating heterogeneous clinical records from MIMIC-IV johnsonMIMICIVFreelyAccessible2023 dataset into a uniform event stream format. Information from different sources in the MIMIC-IV clinical database is represented as an offline reinforcement learning trajectory of interactions between the doctor and the patient where every is a word from a 87899-word interaction vocabulary.

Auto-ALS

Figure 1: Virtu-ALS

Virtu-ALS is an emergency care simulator designed to train students and junior healthcare professionals in following the ABCDE Assessment Protocol thimInitialAssessmentTreatment2012 for emergency care, although its application as a reinforcement learning benchmark was anticipated and accounted for by the authors briskAIEnhanceInteractive2018 . For a benchmark task, we add an Application Programming Interface to Virtu-ALS and propose Auto-ALS liventsevEffectivePatientSimulators2021 : a Gymnasium environment for training automated emergency care agents with Reinforcement Learning. This is achieved by attaching an event listener to Virtu-ALS that registers all observable events that can occur in the simulator in response to the user’s actions.

At each step of the episode the agent receives an observation vector of size 33+7+7=47 that represents various events that happened in the environment prior to the moment when ‘observation‘ is received. It implements an event encoding strategy described in liventsevReinforcementLearningMessage2021 .

The first 33 elements indicate whether one of 33 types of events has occured and how long ago it happened. For i=1,2,…,33:

$$o_{i}=\exp(t_{i}-t)$$ (1)

i.e. the inverse exponent of how much time has passed since this event has last occured that can be interpreted as relevance of this event at the moment. If the event has never occurred,

$$t-t_{i}=\infty\Rightarrow\exp(t_{i}-t)=0$$ (2)

The 33 events are as follows:

ResponseVerbal, ResponseGroan, ResponseNone, AirwayClear, AirwayVomit, AirwayBlood, AirwayTongue, BreathingNone,
BreathingSnoring, BreathingSeeSaw, BreathingEqualChestExpansion, BreathingBibasalCrepitations, BreathingWheeze,
BreathingCoarseCrepitationsAtBase, BreathingPneumothoraxSymptoms, VentilationResistance, RadialPulsePalpable, RadialPulseNonPalpable, HeartSoundsMuffled, HeartSoundsNormal, AVPU_A, AVPU_U, AVPU_V, PupilsPinpoint, PupilsNormal, ExposureRash,
ExposurePeripherallyShutdown, ExposureStainedUnderwear, HeartRhythm0, HeartRhythm1, HeartRhythm2, HeartRhythm3, HeartRhythm4

The next 7 components use the same time encoding $o_{i}=\exp(t_{i}-t)$ for vital signs measurement, i.e. how recently the last measurement has occured:

MeasuredHeartRate, MeasuredRespRate, MeasuredCapillaryGlucose, MeasuredTemperature, MeasuredMAP, MeasuredSats, MeasuredResps

The last 7 components contain the measurements themselves.

After receiving the observation vector, the agent has to output an action: an integer no less than 0 and no more than 48. The 49 actions are:

DoNothing, CheckSignsOfLife, CheckRhythm, ExamineAirway, ExamineBreathing, ExamineCirculation, ExamineDisability, ExamineExposure, ExamineResponse, GiveAdenosine, GiveAdrenaline, GiveAmiodarone, GiveAtropine, GiveMidazolam, UseVenflonIVCatheter, GiveFluids, ViewMonitor, StartChestCompression, OpenAirwayDrawer, OpenBreathingDrawer, OpenCirculationDrawer, OpenDrugsDrawer, BagDuringCPR, ResumeCPR, UseMonitorPads, UseSatsProbe, UseAline, UseBloodPressureCuff, AttachDefibPads, UseBagValveMask,
UseNonRebreatherMask, UseYankeurSucionCatheter, UseGuedelAirway, TakeBloodForArtherialBloodGas, TakeRoutineBloods,
PerformAirwayManoeuvres, PerformHeadTiltChinLift, PerformJawThrust, TakeBloodPressure, TurnOnDefibrillator, DefibrillatorCharge, DefibrillatorCurrentUp, DefibrillatorCurrentDown, DefibrillatorPace, DefibrillatorPacePause, DefibrillatorRateUp, DefibrillatorRateDown, DefibrillatorSync, Finish

Note, in particular, the ‘Examine‘ actions. These actions, just like ‘DoNothing‘ are guaranteed to have no effect on the patient state. However, some events will not trigger unless the agent goes looking for them. To check the blood pressure, one needs to attach the blood pressure cuff to the patient and look at the monitor. Hence, the ‘MeasuredMAP‘ event will only trigger after you ‘BPCuffOn‘ and ‘ExamineMonitor‘. Examination skills are crucial for patient resusciation thimInitialAssessmentTreatment2012 - the simulation would be woefully inadequate if the Examinations were just provided automatically.

The episode ends when the agent outputs selects the Finish action. The episode is considered successful if the patient has been stabilized (has a clear airway, oxygen saturation of at least 88%, a respiratory rate of at least 8 breaths per minute and a mean arterial pressure of at least 60mmHg).

2.2 Vision perspective

Human-Robot interaction

Robots that can autonomously navigate and interact with humans hold immense potential in various assistive scenarios. Consider a robot assisting an elderly individual in their home, hel** with daily activities like cooking, medication reminders, or encouraging physical activity. However, to fulfill these tasks effectively, robots must first locate and approach humans appropriately. Our research addresses this crucial challenge in embodied visual navigation, where an agent utilizes visual input to navigate complex 3D environments, avoid obstacles, and reach desired destinations. An example of such a complex task is shown in Fig 2.

Figure 2: (left) Robot responding to a human call by exploring the environment with the help of multiple global goals (1-6) to locate and reach the human. (center) Robot’s observations upon reaching each global goal (1-6) during its exploration of the environment. (right) Robot’s final observation upon successfully reaching the human at an appropriate angle, depending on the human’s activity, to initiate a conversation.

Over the past decade, significant advancements in this field have been driven by the availability of photorealistic 3D scene datasets proceeding5 ; proceeding6 ; proceeding7 and fast navigation simulators proceeding4 ; proceeding7 ; proceeding8 . Our work builds on these advancements but takes a systematic approach to investigate the task of locating and navigating to a human for initiating interactions. We introduce a pipeline that leverages global and local goal policies to efficiently find and reach humans in complex environments with a focus on assistive tasks and enhancing human-robot interactions enhancement. To achieve this we utilize the Habitat AI simulator proceeding4 and the Gibson dataset proceeding7 for this task conducting a comparison with different baseline methods. Our aim is to evaluate how well the robot performs in finding and navigating toward humans who are engaged in activities such as cooking, eating, being, on a call, or watching TV. In each scenario, the robot needs to explore the environment in order to find the human and approach them safely. In our study, we present a well-defined problem formulation and establish a set of baseline methods that integrate human detection and point-goal navigation to address this challenge effectively.

PH-Ego

Figure 3: The example sequence of frames from PH-Ego dataset capturing coffee preparation and adding sugar actions.

The egocentric action anticipation task consists in predicting future (unobserved) actions performed by a camera-wearer based on input video from a wearable first-person view camera. We are focusing on applying egocentric action anticipation methods to the personal health domain, i.e., utilizing them for the analysis of dietary and hygienic activities routine and for the prevention of undesirable behavior (such as tasting food before washing hands or adding sugar). We collect a dataset PH-Ego, consisting of egocentric videos, capturing specific activities related to food preparation; fine-tune existing action anticipation models on this dataset and analyse effects caused by domain shift.

The dataset was collected from nine healthy volunteers residing in Eindhoven, Netherlands, during the summer of 2022. To maintain privacy and consistency, participants were instructed to remove any hand accessories and wear latex gloves during the video recording process.

The data collection was facilitated using a GoPro Hero 7 Black camera, mounted on the head of each participant. The videos were captured at a frame rate of 50 fps and a resolution of 1920x1280 pixels, with each video having an average duration of 6.5 minutes. In total, 67 action segments were recorded.

For the purposes of model training and validation, the videos of seven participants were utilized in the training dataset, and the videos of the remaining two participants were reserved for model validation. An illustrative example of a sequence of frames, showcasing the actions of making coffee and adding sugar, is depicted in Figure 3.

Imagym

For the last image-based task, we utilize a dataset of fetal ultrosound images to propose a novel data-driven patient simulator for decision support and automation in the field of obstetric ultrasonography obstetrics-sonography .

The goal of this simulator is to accurately model the job of an ultrasound sonographer in the context of a patient undergoing a pregnancy in a way compatible with modern Reinforcement Learning methods liDeepReinforcementLearning2017 to pave the way for autonomous or semi-autonomous ultrasound sonography autonomous-ultrasound-review . The job in question entails moving the ultrasound probe along the patient’s body in order to aquire an image of the fetus that satisfies the guidelines for fetal-screening isoug-guidelines , most importantly, the fact that the fetus’ stomach and umbilical vein are on the image while their heart is not.

(a) Starting point: the probe is located at the exact average of available positions. Heart, stomach and umbilical vein are unseen.

(b) Endpoint: stomach (green) and umbilical vein (blue) are present, heart is absent.

Figure 4: Two examples of the agent’s observation at different positions of the probe

The simulator is based on a dataset of 3D volumes representing fetal abdominal ultrasound scans (though be easily adapted to other medical imaging scenarios as long as a dataset of volumes and relevant organ annotations is available). Each scan is a function defined over a cuboid $B$ of size $\langle x_{\text{max}},y_{\text{max}},z_{\text{max}}\rangle$ (in millimeters).

$$I(x,y,z):B\rightarrow[0;1]$$ (3)

Each scan is accompanied by three mask images $M_{\text{heart}}$, $M_{\text{stomach}}$ and $M_{\text{uv}}$ over the same domain, indicating which parts of the scan are considered to be the heart, the stomach and the umbilical vein respectively.

$$M(x,y,z):B\rightarrow\{0,1\}$$ (4)

The state, in this case, is the current position of the proble defined by 2 3-vectors, the position of the probe $s_{\text{loc}}\in\mathcal{S}_{\text{loc}}$ and its direction $s_{\text{dir}}\in R^{3}$.

The simulator has 2 modes for the space of possible locations $\mathcal{S}_{\text{loc}}$: free and realistic movement. In free movement mode, $\mathcal{S}_{\text{loc}}=B$, thus the agent can place the probe anywhere within the bounds of the image, even inside the patient. In realistic movement mode, $\mathcal{S}_{\text{loc}}\subset B$ representing the surface of the patient’s body available to the agent.

The action space $\mathcal{A}=R^{7}$, where any $a\in\mathcal{A}$ can be decomposed as

$$a=(a_{x},a_{y},a_{z},a_{\text{roll}},a_{\text{pitch}},a_{\text{yaw}},a_{\text{end}})$$ (5)

The first 3 components modify the $s_{\text{loc}}$, the next 3 modify $s_{\text{dir}}$ and, finally if $a_{\text{end}}>0$, the episode terminates and the current image is considered final. Note that in realistic movement mode, blind application of $(a_{x},a_{y},a_{z})$ can lead to the probe being placed outside of the allowed domain of $\mathcal{S}_{\text{loc}}$. In this case, a legal location will be chosen, in a manner that minimizes the distance between requested and real probe position:

$$s_{\text{loc}}\leftarrow\min_{s\in\mathcal{S}_{\text{loc}}}\left\{\lVert s-(s_{\text{loc}}+(a_{x},a_{y},a_{z}))\rVert\right\}$$ (6)

The agent’s observation is obtained by projecting $I(x,y,z)$ onto a plane defined by the current position of the probe $s_{\text{loc}}$ and its direction $s_{\text{dir}}$, simulating the image that the sonographer would see on their screen. It is, theoretically, a function $I_{\text{proj}}(x_{\text{proj}},y_{\text{proj}})$, however, since it is customary in POMDP framework for observations $o$ to be matrices, we replace it with a matrix of evaluations of $I_{\text{proj}}(x_{\text{proj}},y_{\text{proj}})$ on an arbitrary (hyperparametrized) coordinate grid $\{(x_{\text{proj}},y_{\text{proj}})\}$.

The quality metric of the image is defined in accordance with ISOUG guidelines isoug-guidelines to be measured by the surface area of the heart, the stomach and the umbilical vein on the projection, normalized by the volume of these organs.

$$Q(I_{\text{proj}})=\frac{S_{\text{stomach}}(I_{\text{proj}})}{V_{\text{stomach}}(I)}+\frac{S_{\text{heart}}(I_{\text{proj}})}{V_{\text{heart}}(I)}+\frac{S_{\text{uv}}(I_{\text{proj}})}{V_{\text{uv}}(I)}$$ (7)

The reward assigned to the agent then is the difference in quality between the current image and the one obtained at the previous iteration, so that probe movements that improve the image are rewarded positively and all rewards obtained during the episode sum up to the quality of the chosen image.

2.3 Natural language perspective

Many chronic health conditions such as diabetes, obesity, substance abuse, and sleep disorders, can be helped by a lifestyle change. Passive health self-management services with a watch and a tracking app do not seem to be very effective for lifestyle change finkelstein_effectiveness_2016 ; wise_activity_2016 . Health counseling, on the other hand, is known to work rubak_motivational_2005 ; hofmann_efficacy_2012 . However, health counseling is expensive and do not scale well in a world with a growing deficit of healthcare workers. One solution for personal health self-management is automated counseling, AC. After 50 years of research, see, e.g., bickmore_health_2006 , the recent progress in dialog system technologies is finally making AC a realistic option. AC has also recently been shown to be effective in a controlled trial fitzpatrick_delivering_2017 .

Figure 5: Challenges in conversations technologies and applications for health self-management.

We believe that collaborative care management with a conversational agent will be central for future health self-management. The key element is the engaging dialogue, which makes the user reflect own lifestyle choices and barriers and find opportunities and motivation for a change miller_toward_2009 . This requires advances in sensing technology, model-based cognitive interaction technologies, and embodiments. Figure 5 lists enabling technologies that we believe are central for conversational interfaces in health self-management, but where also breakthroughs are needed.

Users may have a very different lifestyle and requirements, which requires highly adaptive solutions. In NLU, NLG, and dialog management zora ; fred there is a clear progression towards deep E2E solutions which provide flexibility over conventional rule-based systems constantin_end–end_2018 ; gehrmann_end–end_2018 ; madotto_mem2seq:_2018 . However, in healthcare, it is very important to be able to combine data-driven learning with validated clinical knowledge RiboniBCJH16 , which is a challenge for many deep-learning solutions. The system must understand what the words of the user mean in the desired clinical and therapeutic setting DBLP:conf/ijcai/ChaouaRCHH18 ; ChaouaRCHH182 , and how the output is expected to influence the user, and the goals of the counseling.

In order to lead a meaningful conversation about the lifestyle of the user, the system has to have a good understanding of it. This requires data-driven techniques for modeling of the lifestyle harma_probabilistic_2016 ; van_wissen_optimization_2016 , and situational intelligence, which can put the conversation in the right context geriatric . In particular, wearable sensors and egocentric vision systems Kanade2012 , together with cameras integrated into the dialog agent, seem to be necessary for human-level situational intelligence. Wearable computing has a long history Betancourt2015 ; Mann97 , it has become of more interest in the last years with the advancement of both hardware and software technologies Furnari2017a . The main applications of FPV systems in the context of assistive Computer Vision Leo2017 is related to memory augmentation Damen2015 and life logging Furnari2018b ; Gurrin2014 ; Ortis2017 . Recent studies have also considered the FPV paradigm to recognize important objects observed by user Lee2015 and to understand or to anticipate the actions and activities performed by the user Damen2018EPIC ; Furnari2018a ; Ryoo2016 ; Singh2016 or the next object to which the user is going to interact with Furnari2017c . A robotic embodiment for the conversational agent is another way to extend the capabilities of the agent. Robotic embodiments have proven to attract more interest and engagement from patients rather than other technologies. Different studies mann2015people ; feingold2018differences reported that both old and young adults preferred to interact with embodied robot over the non-embodied computer screen, and thus carebots may offer benefits over smartphones or tablets in delivering healthcare.

In recent years, Large Language Models (LLM) have successfully imparted close to human-like abilities for several NLP downstream tasks devlin-etal-2019-bert ; li2019fine ; rogers2020primer ; budzianowski2019hello ; radford2018improving . However, there are two significant constraints: a) available training data size sdaih23 ; kumar2023data and b) the complex domain-specific context jeopardizing LLM’s optimal and reliable performance DBLP:conf/iui/DessiHKRR20 ; 9286431 . At first, it may seem that ”available training data size’ should not be a problem at all due to the vast amount of data humanity produces. However, in reality, usable data in several domains, including healthcare, is scarce and not publicly available to train deep neural network models sufficiently.

A potential solution to these limitations is to equip the LMs with domain. While the available research works use Knowledge Graphs Embeddings (KGEs) wang2017knowledge to inject domain knowledge bordes2013translating ; wang2014knowledge ; lin2015learning ; schlichtkrull2018modeling ; trouillon2016complex ; sun2019rotate ; nickel2016holographic ; shang2019end ; nguyen2018novel ; xie2016representation . KGE has proven to be helpful in incorporating world knowledge, but it is still debatable if the KGEs sufficiently capture KGs semantics jain2021embeddings . The work in 9866735 provides a Knowledge-based LLM to use the Resource Description Framework (RDF) triples directly at the input level. The proposed model K-LM works at the crossroads of Generative Pretrained Transformer (GPT-2) and Bidirectional Encoder Representations from Transformers (BERT). K-LM uses a novel pipeline to select, categorize, and filter the RDF triples and introduces heuristic methods to inject domain-specific Knowledge into Knowledge LM.

Diet coaching

NHS UK defines malnutrition as the situation caused by either undernutrition or overnutrition nhs-malnutrition . Nowadays, malnutrition has become a major public health issue diet-stats-worldwide , with profound consequences in society. This problem is particularly acute in UK bma-diet ; ultraprocessed-food-survey-eu where 63% of the UK adult population are overweight, and 27% are obese. Undernutrition is estimated to affect three million people in UK bda-malnutrition , especially among elders and socially isolated individuals. To this regard, obesity is a known cause of multiple illnesses including Type 2 diabetes, infertility, cancer and cardiovascular diseases obesity-act-scot ; undernutrition leads to unplanned weight loss, reduced energy, infections and more nhs-malnutrition . The economic impact of unhealthy eating is notable: every year, obesity-related diseases put £6bn of additional costs on NHS, impacting wider society for about £27bn bma-diet . These costs are expected to reach £9.7bn and £49.9bn by 2050 respectively. Undernutrition is estimated to cost £19bn in England alone bda-malnutrition .

Taken together, the effects of malnutrition on people’s health and on countries’ economies shed light on the need for innovative technologies to encourage healthy eating. We contribute to this by presenting a series of tailored and understandable communication technologies for diet coaching, with a focus on the use of Natural Language Generation (NLG), that is the set of technologies capable of producing text from structured data representation reiter-dale-2000 ; puzikov2018e2e . In details, we implement and evaluate both static and dynamic NLG systems to deliver personalised diet reports.

AnnoMI

Motivational interviewing (MI, mi ) is an important counselling style aimed at evoking a client’s own motivation to adopt a certain positive behaviour change, such as smoking cessation and alcohol use reduction. As a result, MI is widely used in healthcare-related scenarios mi-in-healthcare .

Natural language processing (NLP) for MI has seen considerable progress in recent years (e.g., text-based-behaviour-coding-forecasting ), but it has been substantially limited by the lack of available MI dialogue data and annotations, which is mostly attributable to the concerns and regulations w.r.t. client confidentiality and data sharing.

To address this challenge, we present AnnoMI (annomi-icassp ; annomi-journal ), the first publicly available expert-annotated dataset of MI dialogues. We sourced the conversations of AnnoMI from professionally produced therapy demonstration videos, where each video is a one-to-one dialogue between a therapist and a client. Those conversations cover a wide range of topics.

Notably, AnnoMI is unambiguously legal and well-transcribed, as we 1) gained explicit consent from the video owners for the creation and release of the dataset; 2) opted for a professional transcription service to obtain high-quality transcripts.

AnnoMI also comes with utterance-level annotations from professional therapists. Two major annotated attributes are therapist (main) behaviour for therapist utterances and client talk type for client utterances: therapist (main) behaviour indicates the (main) action/behaviour displayed in a therapist utterance, while client talk type shows whether the client is leaning towards, showing resistance to, or not showing any particular incliniation w.r.t. the behaviour change goal.

Insight mining

Insights from personal health data help the population make better lifestyle choices leading to better health and well-being. The Cambridge Dictionary defines an insight as ”a clear, deep, and sometimes sudden understanding of a complicated problem or situation, or the ability to have such an understanding”. In the domain of data science, it is an overloaded term law2020characterizing used in many perspectives such as findings obtained from data saraiya2005insight or the outcome of evaluating data findings with domain knowledge sacha2014knowledge . In line with the earlier perspective, we define insight as statements that help us understand patterns or behaviors observable from data. Behaviors that stand out, for example, ”You sleep less on Mondays than other days”, are worth communicating to the user with the hope of a remedy. Therefore, we introduced the insight generator framework susaiyah2020towards that automatically generates such insights and picks the most useful ones to show to the user. The generation of such insights requires us to define an abstract structure of such insights that we refer to as an insight schema susaiyah2022aberdeen . For example, ”You measurement comparison time_period1 time_period2” is an insight-schema that defines all insights that compare a measurement such as sleep duration between two time periods such as Mondays and other days. The framework iteratively checks through the data looking for insights matching the schema and generates insights candidates when a match is found. The candidates are later ranked based on various factors such as significance, accuracy, user preference, and reliability susaiyah2021neural ; susaiyah2022aberdeen ; susaiyah2022RL ; ecai-paper . Finally, the top-ranking insights are recommended to the user.

3 Evaluations

3.1 MIMIC-IV-Ext-SEQ

We train a baseline model that consists of a two-layer multilayer perceptron (MLP) with 1000 hidden layer size RELU agarapDeepLearningUsing2018 activation function and batch normalization after each layer. As input, all events from the first day are used and one-hot encoded in a 87899-dimensional vector. As prediction target, all events from the second day are used and encoded via their clustering map**, e.g. c10, c100, c1000, c10000, into a vector of the corresponding dimension. As objective function we used binary cross entropy. The last layer contains a sigmoid function which transforms the output to probabilities for each vector element. A threshold is set at 0.5 to decide if an event occurs or not. All models were trained with batch size 512 for 3 epochs.

We test our baseline on the second day event prediction task and summarize the results for different clusterings in table 1. It can be seen that the more classes are in the clustering, the harder the prediction task becomes. As noted above, accuracy is a deceptive metric in this scenario.

clustering	recall	accuracy	precision	F1
c10	0.903	0.840	0.790	0.827
c100	0.568	0.885	0.713	0.632
c1000	0.500	0.976	0.710	0.586
c10000	0.509	0.996	0.703	0.589

Table 1: Evaluation results of 2 x 1000 MLP for first day - second day prediction, entire dataset

For many patients it is the case that they are in the hospital for only one day. For these patients, the model should predict no event on the second day. However, evaluation of the models showed that this is almost never the case. However, one can argue that it is more important to get problematic patients correct than the ones who leave the hospital after one day. Therefore, we evaluated the model also only on patients which are in the hospital for at least 2 days. Since the data is roughly ordered according to length of stay, we achieved this by skip** the first 100k samples in the training and the first 1k samples in the test data. The models shows improved performance here, as can be seen in Table 2.

clustering	recall	accuracy	precision	F1
c10	0.942	0.854	0.850	0.890
c100	0.633	0.878	0.765	0.692
c1000	0.548	0.974	0.770	0.640
c10000	0.539	0.995	0.771	0.634

Table 2: Evaluation results of 2 x 1000 MLP for first day - second day prediction, skip** first 100k train / 1k test samples

The same setup but with 3 hidden layers and 5000 units each shows improved performance as can be seen in table 3. Bigger models were tested as well, but showed no further improvement.

configuration	recall	accuracy	precision	F1
1	0.505	0.996	0.727	0.595
2	0.544	0.9959	0.784	0.642

Table 3: All evaluations with c10000; 1: Evaluation results of 3 x 5000 MLP for first day - second day prediction, entire dataset; 2: Evaluation results of 3 x 5000 MLP for first day - second day prediction, skip first 100k train / 1k test samples

Replacing ones in the one-hot encoding with the corresponding events’ intensities impaired the models’ performance, likely because it introduces a false equivalency between a zero-intensity event and lack of an event, i.e. ”average blood pressure” is different from ”no blood pressure measurement”.

3.2 Auto-ALS

Two experimental emergency care agents were train with Reinforcement Learning in bhalwankarNeuroSymbolicReinforcementLearning : one used a common Proximal Policy Optimization (PPO) algorithm and no prior knowledge of emergency care, another implemented several common sense constraints on the agent in Linear Temporal Logic and trained a Graph Neural Network that took those constraints into account. See table 4 for details of the agents.

	PPO	GNNprog w/rules
Env. steps per update	2048	2048
Number of epochs	4	4
Minibatch Size	256	1024
Discount factor ()	0.99	0.99
Learning rate	0.0001	0.0001
GAE-	0.95	0.95
Entropy coefficient	0.01	0.01
Value loss coefficient	0.5	0.5
Gradient Clip**	0.5	0.5
PPO Clip** ()	0.2	0.2

Table 4: Hyperparameters of evaluated agents

The results are presented in table 5

Group Statistics
	$\mathrm{v}26$	$\mathrm{~{}N}$	Mean	Std. Deviation	Std. Error Mean
average discounted return	1	781	0.85	1.02	0.04
	2	781	-4.04	7.03	0.25

Table 5: Group Statistics w.r.t to average discounted returns on LTL vs No-LTL model outputs in AutoALS Env

3.3 Human-Robot interaction

We evaluate the robot’s performance in locating and navigating to humans engaged in various activities, such as cooking, eating, on a call, and watching TV. We validate the feasibility of this task and the effectiveness of our proposed baselines using the Habitat AI simulator proceeding4 and the Gibson modified dataset, which consists of twenty complex 3D environments. Since the Gibson dataset lacks human models, we have created human models in poses aligned with different activities (eating, cooking, watching TV, and on a call). These human models were strategically placed in various locations within the Gibson environments, such as the kitchen, TV lounge, bedroom, and other relevant areas.

We run $2000$ evaluation episodes, with each scene containing $100$ episodes. We consider the two variants of the task: V1 aims to reach the human from any angle, whereas in V2 episode success depends on the orientation difference between the robot and the human at the end of the episode. Table 6 provides Success weighted by Path Length (SPL) and success rate (SR) for both V1 and V2. Our proposed approach achieves $93\%$ SR and $57\%$ SPL under $V1$. However, in $V2(\theta=30^{\circ})$, our approach only achieved $26\%$ SR and $14\%$ SPL. This suggests that V2 of the task is much more challenging and more research is still needed. Overall, the considered baselines achieve promising performance in locating and reaching humans in previously unseen environments. However, further research is required in this exciting field. We believe that our proposed approach serves as a first step towards building autonomous task-oriented assistive robots for home use cases.

Table 6: Average SPL and SR on the Gibson validation dataset.

Task	SPL ($\uparrow$)	SR ($\uparrow$)
V1 (any angle)	$0.57$	$0.93$
V2 ($\theta\leq 60^{\circ}$)	$0.25$	$0.44$
V2 ($\theta\leq 30^{\circ}$)	$0.14$	$0.26$

3.4 PH-Ego

The task of trimmed action anticipation involves predicting the class $y$ of an impending action, initiating at time $\tau_{s}$, by analyzing a trimmed video segment that precedes the action. The final observed timestamp is defined as $\tau_{s}-\tau_{a}$, where $\tau_{a}$ represents the anticipation time, typically established at 1 second damen2022rescaling .

In contrast, the untrimmed action anticipation task operates without reliance on pre-annotated action annotations, executing anticipations and their subsequent evaluations at intervals of every $\alpha$ seconds, irrespective of whether an action is forthcoming or not rodin2022untrimmed . This untrimmed approach holds significant relevance in the context of personal assistant settings, where the system necessitates continuous monitoring of activities to facilitate timely alerts or assistance. However, this method also carries an elevated risk of generating false-positive predictions.

In the study rodin2022untrimmed , we propose the utilization of the mean Average Precision (mAP) as an evaluation metric for untrimmed action anticipation. Drawing inspiration from object detection benchmarks, the research suggests treating untrimmed anticipation as a future action detection problem. This approach encourages the diversification of models, factoring in false-positive predictions to enhance the robustness and accuracy of the anticipatory models in identifying and forecasting future actions. However, for a smaller problem, where the time-to-action predictions do not play important roles, we propose to use simpler classification metrics rodin2023egocentric .

3.5 AnnoMI

Based on the publicly available AnnoMI dataset that we created, we explore two natural language understanding (NLU) tasks with real-world applicability: 1) current-turn therapist/client behaviour prediction (prediction task, annomi-journal ); and 2) next-turn therapist behaviour forecasting (forecasting task, mi-action-forecasting ). The code and data of our experiments are publicly available.

The prediction task aims to predict the (main) behaviour of each therapist utterance and the talk type of each client utterance, given the utterance itself as the input. An accurate prediction model would be readily usable for automatically annotating transcripts of MI sessions, which would considerably save resources on laborious human annotation and accelerate insight generation from recorded sessions.

Overall, we found BERT with adapters houlsby-adapters  — a parameter-efficient version of BERT devlinBERTPretrainingDeep2019  — to be the best-performing model, with performance higher for therapist behaviours than for client talk types. Encouragingly, for therapist behaviour prediction, the model also proved to maintain a high level of performance when tested on utterances from new dialogue topics to which it had not been exposed during training, which is meaningful for real-world deployment.

The forecasting task aims to forecast which (main) behaviour a good therapist would show in their next-turn response to the client, given the history of the ongoing dialogue where the last turn comes from the client. An accurate forecasting model would be able to assist a (junior) therapist in real time in deciding what strategy to choose for responding to the client.

For this task, we explored a variety of NLP modelling choices with the RoBERTa roberta model, but the performance was suboptimal. We believe that this reflects the considerable flexibility in how a good therapist could respond to a client, as there are often multiple “optimal behaviours/actions” to display in the next turn. Therefore, future work could model this task in a probabilistic setup to accommodate this flexibility.

The information retrieval task aims to filter therapeutical content by estimating quality automatically 10.1007/978-3-031-37249-0_10 . First, we applied state-of-the-art information retrieval models to evaluate their applicability in the psychological domain for ranking therapy sessions by estimated quality. Then, given the sensitive psychological information associated with each therapy session, we analyzed the potential risk of unfair outcomes across therapy topics, i.e., mental issues, under a common fairness definition. Our experimental results show that the employed ranking models are reliable for systematically ranking high-quality content above low-quality one, while unfair outcomes across topics are model-dependent and associated low-quality content distribution.

3.6 Insight mining

Insights are subjective and evaluating them requires human participants. Therefore, Evaluating Insight Generation systems is time-consuming and expensive due to recruitment, compensation, sample size, etc. Therefore, we examined the possibility of using Reinforcement Learning (RL) to build and Evaluate Insight mining systems susaiyah2022RL . Here, we first developed a simulator that generates a human lifestyle that assumes performing a certain task at a given time of the day, for example, waking up, having breakfast, communicating to work, etc. We then set up an RL environment that iteratively trains an agent to recommend useful insights to the simulated user. The insights modify the lifestyle of the user and this leads to different insights and vice versa. The cost function is determined by the lifestyle parameter that is being improved. In our experiments, we used the Pittsburg sleep quality index (PSQI) as a reward function. Experiments show that the above approach was able to recommend insights that significantly quickened the adaptation of a behavior change towards a better lifestyle in the simulated environment compared to baseline techniques.

4 Discussion

Our benchmark suite demonstrates the state of the art across different tasks of Machine Learning in Healthcare. All problems discussed are, however, open and PhilHumans is meant to be an evaluation tools for future methodological research.

5 Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 812882.