Showing 1–4 of 4 results for author: Demasi, O

Towards Emotional Support Dialog Systems

Authors: Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, Minlie Huang

Abstract: Emotional support is a crucial ability for many conversation scenarios, including social interactions, mental health support, and customer service chats. Following reasonable procedures and using various support skills can help to effectively provide support. However, due to the lack of a well-designed task and corpora of effective emotional support conversations, research on building emotional su… ▽ More Emotional support is a crucial ability for many conversation scenarios, including social interactions, mental health support, and customer service chats. Following reasonable procedures and using various support skills can help to effectively provide support. However, due to the lack of a well-designed task and corpora of effective emotional support conversations, research on building emotional support into dialog systems remains untouched. In this paper, we define the Emotional Support Conversation (ESC) task and propose an ESC Framework, which is grounded on the Hel** Skills Theory. We construct an Emotion Support Conversation dataset (ESConv) with rich annotation (especially support strategy) in a help-seeker and supporter mode. To ensure a corpus of high-quality conversations that provide examples of effective emotional support, we take extensive effort to design training tutorials for supporters and several mechanisms for quality control during data collection. Finally, we evaluate state-of-the-art dialog models with respect to the ability to provide emotional support. Our results show the importance of support strategies in providing effective emotional support and the utility of ESConv in training more emotional support systems. △ Less

Submitted 2 June, 2021; originally announced June 2021.

Comments: Accepted to ACL 2021 (Long Paper)

An example of how false conclusions could be made with personalized health tracking and suggestions for avoiding similar situations

Authors: Orianna DeMasi, Benjamin Recht

Abstract: Personalizing interventions and treatments is a necessity for optimal medical care. Recent advances in computing, such as personal electronic devices, have made it easier than ever to collect and utilize vast amounts of personal data on individuals. This data could support personalized medicine; however, there are pitfalls that must be avoided. We discuss an example, longitudinal medical tracking,… ▽ More Personalizing interventions and treatments is a necessity for optimal medical care. Recent advances in computing, such as personal electronic devices, have made it easier than ever to collect and utilize vast amounts of personal data on individuals. This data could support personalized medicine; however, there are pitfalls that must be avoided. We discuss an example, longitudinal medical tracking, in which traditional methods of evaluating machine learning algorithms fail and present the opportunity for false conclusions. We then pose three suggestions for avoiding such opportunities for misleading results in medical applications, where reliability is essential. △ Less

Submitted 15 November, 2017; originally announced November 2017.

Comments: Presented at the Data For Good Exchange 2017

Meaningless comparisons lead to false optimism in medical machine learning

Authors: Orianna DeMasi, Konrad Kording, Benjamin Recht

Abstract: A new trend in medicine is the use of algorithms to analyze big datasets, e.g. using everything your phone measures about you for diagnostics or monitoring. However, these algorithms are commonly compared against weak baselines, which may contribute to excessive optimism. To assess how well an algorithm works, scientists typically ask how well its output correlates with medically assigned scores.… ▽ More A new trend in medicine is the use of algorithms to analyze big datasets, e.g. using everything your phone measures about you for diagnostics or monitoring. However, these algorithms are commonly compared against weak baselines, which may contribute to excessive optimism. To assess how well an algorithm works, scientists typically ask how well its output correlates with medically assigned scores. Here we perform a meta-analysis to quantify how the literature evaluates their algorithms for monitoring mental wellbeing. We find that the bulk of the literature ($\sim$77%) uses meaningless comparisons that ignore patient baseline state. For example, having an algorithm that uses phone data to diagnose mood disorders would be useful. However, it is possible to over 80% of the variance of some mood measures in the population by simply guessing that each patient has their own average mood - the patient-specific baseline. Thus, an algorithm that just predicts that our mood is like it usually is can explain the majority of variance, but is, obviously, entirely useless. Comparing to the wrong (population) baseline has a massive effect on the perceived quality of algorithms and produces baseless optimism in the field. To solve this problem we propose "user lift" that reduces these systematic errors in the evaluation of personalized medical monitoring. △ Less

Submitted 19 July, 2017; originally announced July 2017.

Dimension Reduction Using Rule Ensemble Machine Learning Methods: A Numerical Study of Three Ensemble Methods

Authors: Orianna DeMasi, Juan Meza, David H. Bailey

Abstract: Ensemble methods for supervised machine learning have become popular due to their ability to accurately predict class labels with groups of simple, lightweight "base learners." While ensembles offer computationally efficient models that have good predictive capability they tend to be large and offer little insight into the patterns or structure in a dataset. We consider an ensemble technique that… ▽ More Ensemble methods for supervised machine learning have become popular due to their ability to accurately predict class labels with groups of simple, lightweight "base learners." While ensembles offer computationally efficient models that have good predictive capability they tend to be large and offer little insight into the patterns or structure in a dataset. We consider an ensemble technique that returns a model of ranked rules. The model accurately predicts class labels and has the advantage of indicating which parameter constraints are most useful for predicting those labels. An example of the rule ensemble method successfully ranking rules and selecting attributes is given with a dataset containing images of potential supernovas where the number of necessary features is reduced from 39 to 21. We also compare the rule ensemble method on a set of multi-class problems with boosting and bagging, which are two well known ensemble techniques that use decision trees as base learners, but do not have a rule ranking scheme. △ Less

Submitted 30 August, 2011; originally announced August 2011.