-
Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation
Authors:
Francesco Moramarco,
Alex Papadopoulos Korfiatis,
Mark Perera,
Damir Juric,
Jack Flann,
Ehud Reiter,
Anya Belz,
Aleksandar Savkov
Abstract:
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 cl…
▽ More
In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.
△ Less
Submitted 1 April, 2022;
originally announced April 2022.
-
Towards more patient friendly clinical notes through language models and ontologies
Authors:
Francesco Moramarco,
Damir Juric,
Aleksandar Savkov,
Jack Flann,
Maria Lehl,
Kristian Boda,
Tessa Grafen,
Vitalii Zhelezniak,
Sunir Gohil,
Alex Papadopoulos Korfiatis,
Nils Hammerla
Abstract:
Clinical notes are an efficient way to record patient information but are notoriously hard to decipher for non-experts. Automatically simplifying medical text can empower patients with valuable information about their health, while saving clinicians time. We present a novel approach to automated simplification of medical text based on word frequencies and language modelling, grounded on medical on…
▽ More
Clinical notes are an efficient way to record patient information but are notoriously hard to decipher for non-experts. Automatically simplifying medical text can empower patients with valuable information about their health, while saving clinicians time. We present a novel approach to automated simplification of medical text based on word frequencies and language modelling, grounded on medical ontologies enriched with layman terms. We release a new dataset of pairs of publicly available medical sentences and a version of them simplified by clinicians. Also, we define a novel text simplification metric and evaluation framework, which we use to conduct a large-scale human evaluation of our method against the state of the art. Our method based on a language model trained on medical forum data generates simpler sentences while preserving both grammar and the original meaning, surpassing the current state of the art.
△ Less
Submitted 23 December, 2021;
originally announced December 2021.
-
Towards objectively evaluating the quality of generated medical summaries
Authors:
Francesco Moramarco,
Damir Juric,
Aleksandar Savkov,
Ehud Reiter
Abstract:
We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.
We propose a method for evaluating the quality of generated text by asking evaluators to count facts, and computing precision, recall, f-score, and accuracy from the raw counts. We believe this approach leads to a more objective and easier to reproduce evaluation. We apply this to the task of medical report summarisation, where measuring objective quality and accuracy is of paramount importance.
△ Less
Submitted 9 April, 2021;
originally announced April 2021.
-
Can Embeddings Adequately Represent Medical Terminology? New Large-Scale Medical Term Similarity Datasets Have the Answer!
Authors:
Claudia Schulz,
Damir Juric
Abstract:
A large number of embeddings trained on medical data have emerged, but it remains unclear how well they represent medical terminology, in particular whether the close relationship of semantically similar medical terms is encoded in these embeddings. To date, only small datasets for testing medical term similarity are available, not allowing to draw conclusions about the generalisability of embeddi…
▽ More
A large number of embeddings trained on medical data have emerged, but it remains unclear how well they represent medical terminology, in particular whether the close relationship of semantically similar medical terms is encoded in these embeddings. To date, only small datasets for testing medical term similarity are available, not allowing to draw conclusions about the generalisability of embeddings to the enormous amount of medical terms used by doctors. We present multiple automatically created large-scale medical term similarity datasets and confirm their high quality in an annotation study with doctors. We evaluate state-of-the-art word and contextual embeddings on our new datasets, comparing multiple vector similarity metrics and word vector aggregation techniques. Our results show that current embeddings are limited in their ability to adequately encode medical terms. The novel datasets thus form a challenging new benchmark for the development of medical embeddings able to accurately represent the whole medical terminology.
△ Less
Submitted 24 March, 2020;
originally announced March 2020.
-
An Interface-Tracking Technique for Multiphase Flow with Soluble Surfactant
Authors:
Seungwon Shin,
Jalel Chergui,
Damir Juric,
Lyes Kahouadji,
Omar K. Matar,
Richard V. Craster
Abstract:
We adapt and extend a formulation for soluble surfactant transport in multiphase flows recently presented by Muradoglu & Tryggvason (JCP 274 (2014) 737-757) to the context of the Level Contour Reconstruction Method (Shin et al. IJNMF 60 (2009) 753-778) which is a hybrid method that combines the advantages of the Front-tracking and Level Set methods. Particularly close attention is paid to the form…
▽ More
We adapt and extend a formulation for soluble surfactant transport in multiphase flows recently presented by Muradoglu & Tryggvason (JCP 274 (2014) 737-757) to the context of the Level Contour Reconstruction Method (Shin et al. IJNMF 60 (2009) 753-778) which is a hybrid method that combines the advantages of the Front-tracking and Level Set methods. Particularly close attention is paid to the formulation and numerical implementation of the surface gradients of surfactant concentration and surface tension. Various benchmark tests are performed to demonstrate the accuracy of different elements of the algorithm. To verify surfactant mass conservation, values for surfactant diffusion along the interface are compared with the exact solution for the problem of uniform expansion of a sphere. The numerical implementation of the discontinuous boundary condition for the source term in the bulk concentration is compared with the approximate solution. Surface tension forces are tested for Marangoni drop translation. Our numerical results for drop deformation in simple shear are compared with experiments and results from previous simulations. All benchmarking tests compare well with existing data thus providing confidence that our adapted LCRM formulation for surfactant advection and diffusion is accurate and effective in three-dimensional multiphase flows. We also demonstrate that this approach applies easily to massively parallel simulations.
△ Less
Submitted 8 February, 2017;
originally announced February 2017.
-
A Solver for Massively Parallel Direct Numerical Simulation of Three-Dimensional Multiphase Flows
Authors:
S. Shin,
J. Chergui,
D. Juric
Abstract:
We present a new solver for massively parallel simulations of fully three-dimensional multiphase flows. The solver runs on a variety of computer architectures from laptops to supercomputers and on 65536 threads or more (limited only by the availability to us of more threads). The code is wholly written by the authors in Fortran 2003 and uses a domain decomposition strategy for parallelization with…
▽ More
We present a new solver for massively parallel simulations of fully three-dimensional multiphase flows. The solver runs on a variety of computer architectures from laptops to supercomputers and on 65536 threads or more (limited only by the availability to us of more threads). The code is wholly written by the authors in Fortran 2003 and uses a domain decomposition strategy for parallelization with MPI. The fluid interface solver is based on a parallel implementation of the LCRM hybrid Front Tracking/Level Set method designed to handle highly deforming interfaces with complex topology changes. We discuss the implementation of this interface method and its particular suitability to distributed processing where all operations are carried out locally on distributed subdomains. We have developed parallel GMRES and Multigrid iterative solvers suited to the linear systems arising from the implicit solution of the fluid velocities and pressure in the presence of strong density and viscosity discontinuities across fluid phases. Particular attention is drawn to the details and performance of the parallel Multigrid solver. The code includes modules for flow interaction with immersed solid objects, contact line dynamics, species and thermal transport with phase change. Here, however, we focus on the simulation of the canonical problem of drop splash onto a liquid film and report on the parallel performance of the code on varying numbers of threads. The 3D simulations were run on mesh resolutions up to $1024^3$ with results at the higher resolutions showing the fine details and features of droplet ejection, crown formation and rim instability observed under similar experimental conditions. Keywords:
△ Less
Submitted 30 October, 2014;
originally announced October 2014.
-
Flexible Visual Quality Inspection in Discrete Manufacturing
Authors:
Tomislav Petković,
Darko Jurić,
Sven Lončarić
Abstract:
Most visual quality inspections in discrete manufacturing are composed of length, surface, angle or intensity measurements. Those are implemented as end-user configurable inspection tools that should not require an image processing expert to set up. Currently available software solutions providing such capability use a flowchart based programming environment, but do not fully address an inspection…
▽ More
Most visual quality inspections in discrete manufacturing are composed of length, surface, angle or intensity measurements. Those are implemented as end-user configurable inspection tools that should not require an image processing expert to set up. Currently available software solutions providing such capability use a flowchart based programming environment, but do not fully address an inspection flowchart robustness and can require a redefinition of the flowchart if a small variation is introduced. In this paper we propose an acquire-register-analyze image processing pattern designed for discrete manufacturing that aims to increase the robustness of the inspection flowchart by consistently addressing variations in product position, orientation and size. A proposed pattern is transparent to the end-user and simplifies the flowchart. We describe a developed software solution that is a practical implementation of the proposed pattern. We give an example of its real-life use in industrial production of electric components.
△ Less
Submitted 1 October, 2013;
originally announced October 2013.
