-
LLMs Accelerate Annotation for Medical Information Extraction
Authors:
Akshay Goel,
Almog Gueta,
Omry Gilon,
Chang Liu,
Sofia Erell,
Lan Huong Nguyen,
Xiaohong Hao,
Bolous Jaber,
Shashir Reddy,
Rupesh Kartha,
Jean Steiner,
Itay Laish,
Amir Feder
Abstract:
The unstructured nature of clinical notes within electronic health records often conceals vital patient-related information, making it challenging to access or interpret. To uncover this hidden information, specialized Natural Language Processing (NLP) models are required. However, training these models necessitates large amounts of labeled data, a process that is both time-consuming and costly wh…
▽ More
The unstructured nature of clinical notes within electronic health records often conceals vital patient-related information, making it challenging to access or interpret. To uncover this hidden information, specialized Natural Language Processing (NLP) models are required. However, training these models necessitates large amounts of labeled data, a process that is both time-consuming and costly when relying solely on human experts for annotation. In this paper, we propose an approach that combines Large Language Models (LLMs) with human expertise to create an efficient method for generating ground truth labels for medical text annotation. By utilizing LLMs in conjunction with human annotators, we significantly reduce the human annotation burden, enabling the rapid creation of labeled datasets. We rigorously evaluate our method on a medical information extraction task, demonstrating that our approach not only substantially cuts down on human intervention but also maintains high accuracy. The results highlight the potential of using LLMs to improve the utilization of unstructured clinical data, allowing for the swift deployment of tailored NLP solutions in healthcare.
△ Less
Submitted 4 December, 2023;
originally announced December 2023.
-
Surfacing Biases in Large Language Models using Contrastive Input Decoding
Authors:
Gal Yona,
Or Honovich,
Itay Laish,
Roee Aharoni
Abstract:
Ensuring that large language models (LMs) are fair, robust and useful requires an understanding of how different modifications to their inputs impact the model's behaviour. In the context of open-text generation tasks, however, such an evaluation is not trivial. For example, when introducing a model with an input text and a perturbed, "contrastive" version of it, meaningful differences in the next…
▽ More
Ensuring that large language models (LMs) are fair, robust and useful requires an understanding of how different modifications to their inputs impact the model's behaviour. In the context of open-text generation tasks, however, such an evaluation is not trivial. For example, when introducing a model with an input text and a perturbed, "contrastive" version of it, meaningful differences in the next-token predictions may not be revealed with standard decoding strategies. With this motivation in mind, we propose Contrastive Input Decoding (CID): a decoding algorithm to generate text given two inputs, where the generated text is likely given one input but unlikely given the other. In this way, the contrastive generations can highlight potentially subtle differences in how the LM output differs for the two inputs in a simple and interpretable manner. We use CID to highlight context-specific biases that are hard to detect with standard decoding strategies and quantify the effect of different input perturbations.
△ Less
Submitted 12 May, 2023;
originally announced May 2023.
-
Useful Confidence Measures: Beyond the Max Score
Authors:
Gal Yona,
Amir Feder,
Itay Laish
Abstract:
An important component in deploying machine learning (ML) in safety-critic applications is having a reliable measure of confidence in the ML model's predictions. For a classifier $f$ producing a probability vector $f(x)$ over the candidate classes, the confidence is typically taken to be $\max_i f(x)_i$. This approach is potentially limited, as it disregards the rest of the probability vector. In…
▽ More
An important component in deploying machine learning (ML) in safety-critic applications is having a reliable measure of confidence in the ML model's predictions. For a classifier $f$ producing a probability vector $f(x)$ over the candidate classes, the confidence is typically taken to be $\max_i f(x)_i$. This approach is potentially limited, as it disregards the rest of the probability vector. In this work, we derive several confidence measures that depend on information beyond the maximum score, such as margin-based and entropy-based measures, and empirically evaluate their usefulness, focusing on NLP tasks with distribution shifts and Transformer-based models. We show that when models are evaluated on the out-of-distribution data ``out of the box'', using only the maximum score to inform the confidence measure is highly suboptimal. In the post-processing regime (where the scores of $f$ can be improved using additional in-distribution held-out data), this remains true, albeit less significant. Overall, our results suggest that entropy-based confidence is a surprisingly useful measure.
△ Less
Submitted 25 October, 2022;
originally announced October 2022.
-
Audio De-identification: A New Entity Recognition Task
Authors:
Ido Cohn,
Itay Laish,
Genady Beryozkin,
Gang Li,
Izhak Shafran,
Idan Szpektor,
Tzvika Hartman,
Avinatan Hassidim,
Yossi Matias
Abstract:
Named Entity Recognition (NER) has been mostly studied in the context of written text. Specifically, NER is an important step in de-identification (de-ID) of medical records, many of which are recorded conversations between a patient and a doctor. In such recordings, audio spans with personal information should be redacted, similar to the redaction of sensitive character spans in de-ID for written…
▽ More
Named Entity Recognition (NER) has been mostly studied in the context of written text. Specifically, NER is an important step in de-identification (de-ID) of medical records, many of which are recorded conversations between a patient and a doctor. In such recordings, audio spans with personal information should be redacted, similar to the redaction of sensitive character spans in de-ID for written text. The application of NER in the context of audio de-identification has yet to be fully investigated. To this end, we define the task of audio de-ID, in which audio spans with entity mentions should be detected. We then present our pipeline for this task, which involves Automatic Speech Recognition (ASR), NER on the transcript text, and text-to-audio alignment. Finally, we introduce a novel metric for audio de-ID and a new evaluation benchmark consisting of a large labeled segment of the Switchboard and Fisher audio datasets and detail our pipeline's results on it.
△ Less
Submitted 5 May, 2019; v1 submitted 17 March, 2019;
originally announced March 2019.
-
Efficient Dynamic Approximate Distance Oracles for Vertex-Labeled Planar Graphs
Authors:
Itay Laish,
Shay Mozes
Abstract:
Let $G$ be a graph where each vertex is associated with a label. A Vertex-Labeled Approximate Distance Oracle is a data structure that, given a vertex $v$ and a label $λ$, returns a $(1+\varepsilon)$-approximation of the distance from $v$ to the closest vertex with label $λ$ in $G$. Such an oracle is dynamic if it also supports label changes. In this paper we present three different dynamic approx…
▽ More
Let $G$ be a graph where each vertex is associated with a label. A Vertex-Labeled Approximate Distance Oracle is a data structure that, given a vertex $v$ and a label $λ$, returns a $(1+\varepsilon)$-approximation of the distance from $v$ to the closest vertex with label $λ$ in $G$. Such an oracle is dynamic if it also supports label changes. In this paper we present three different dynamic approximate vertex-labeled distance oracles for planar graphs, all with polylogarithmic query and update times, and nearly linear space requirements.
△ Less
Submitted 27 August, 2017; v1 submitted 8 July, 2017;
originally announced July 2017.
