-
CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering
Authors:
Nirmalie Wiratunga,
Ramitha Abeyratne,
Lasal Jayawardena,
Kyle Martin,
Stewart Massie,
Ikechukwu Nkisi-Orji,
Ruvan Weerasinghe,
Anne Liret,
Bruno Fleisch
Abstract:
Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of…
▽ More
Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR's case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
-
TRIALSCOPE: A Unifying Causal Framework for Scaling Real-World Evidence Generation with Biomedical Language Models
Authors:
Javier González,
Cliff Wong,
Zelalem Gero,
Jass Bagga,
Risa Ueno,
Isabel Chien,
Eduard Oravkin,
Emre Kiciman,
Aditya Nori,
Roshanthi Weerasinghe,
Rom S. Leidner,
Brian Piening,
Tristan Naumann,
Carlo Bifulco,
Hoifung Poon
Abstract:
The rapid digitization of real-world data offers an unprecedented opportunity for optimizing healthcare delivery and accelerating biomedical discovery. In practice, however, such data is most abundantly available in unstructured forms, such as clinical notes in electronic medical records (EMRs), and it is generally plagued by confounders. In this paper, we present TRIALSCOPE, a unifying framework…
▽ More
The rapid digitization of real-world data offers an unprecedented opportunity for optimizing healthcare delivery and accelerating biomedical discovery. In practice, however, such data is most abundantly available in unstructured forms, such as clinical notes in electronic medical records (EMRs), and it is generally plagued by confounders. In this paper, we present TRIALSCOPE, a unifying framework for distilling real-world evidence from population-level observational data. TRIALSCOPE leverages biomedical language models to structure clinical text at scale, employs advanced probabilistic modeling for denoising and imputation, and incorporates state-of-the-art causal inference techniques to combat common confounders. Using clinical trial specification as generic representation, TRIALSCOPE provides a turn-key solution to generate and reason with clinical hypotheses using observational data. In extensive experiments and analyses on a large-scale real-world dataset with over one million cancer patients from a large US healthcare network, we show that TRIALSCOPE can produce high-quality structuring of real-world data and generates comparable results to marquee cancer trials. In addition to facilitating in-silicon clinical trial design and optimization, TRIALSCOPE may be used to empower synthetic controls, pragmatic trials, post-market surveillance, as well as support fine-grained patient-like-me reasoning in precision diagnosis and treatment.
△ Less
Submitted 6 November, 2023; v1 submitted 2 November, 2023;
originally announced November 2023.
-
Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology
Authors:
Cliff Wong,
Sheng Zhang,
Yu Gu,
Christine Moung,
Jacob Abel,
Naoto Usuyama,
Roshanthi Weerasinghe,
Brian Piening,
Tristan Naumann,
Carlo Bifulco,
Hoifung Poon
Abstract:
Clinical trial matching is a key process in health delivery and discovery. In practice, it is plagued by overwhelming unstructured data and unscalable manual processing. In this paper, we conduct a systematic study on scaling clinical trial matching using large language models (LLMs), with oncology as the focus area. Our study is grounded in a clinical trial matching system currently in test deplo…
▽ More
Clinical trial matching is a key process in health delivery and discovery. In practice, it is plagued by overwhelming unstructured data and unscalable manual processing. In this paper, we conduct a systematic study on scaling clinical trial matching using large language models (LLMs), with oncology as the focus area. Our study is grounded in a clinical trial matching system currently in test deployment at a large U.S. health network. Initial findings are promising: out of box, cutting-edge LLMs, such as GPT-4, can already structure elaborate eligibility criteria of clinical trials and extract complex matching logic (e.g., nested AND/OR/NOT). While still far from perfect, LLMs substantially outperform prior strong baselines and may serve as a preliminary solution to help triage patient-trial candidates with humans in the loop. Our study also reveals a few significant growth areas for applying LLMs to end-to-end clinical trial matching, such as context limitation and accuracy, especially in structuring patient information from longitudinal medical records.
△ Less
Submitted 18 August, 2023; v1 submitted 4 August, 2023;
originally announced August 2023.
-
Automated use case diagram generator using NLP and ML
Authors:
Rukshan Piyumadu Dias,
C. S. L. Vidanapathirana,
Rukshala Weerasinghe,
Asitha Manupiya,
R. M. S. J. Bandara,
Y. P. H. W. Ranasinghe
Abstract:
This paper presents a novel approach to generate a use case diagram by analyzing the given user story using NLP and ML. Use case diagrams play a major role in the designing phase of the SDLC. This proves the fact that automating the use case diagram designing process would save a lot of time and effort. Numerous manual and semi-automated tools have been developed previously. This paper also discus…
▽ More
This paper presents a novel approach to generate a use case diagram by analyzing the given user story using NLP and ML. Use case diagrams play a major role in the designing phase of the SDLC. This proves the fact that automating the use case diagram designing process would save a lot of time and effort. Numerous manual and semi-automated tools have been developed previously. This paper also discusses the need for use case diagrams and problems faced during designing that. This paper is an attempt to solve those issues by generating the use case diagram in a fully automatic manner.
△ Less
Submitted 12 June, 2023;
originally announced June 2023.
-
Towards Structuring Real-World Data at Scale: Deep Learning for Extracting Key Oncology Information from Clinical Text with Patient-Level Supervision
Authors:
Sam Preston,
Mu Wei,
Rajesh Rao,
Robert Tinn,
Naoto Usuyama,
Michael Lucas,
Roshanthi Weerasinghe,
Soohee Lee,
Brian Piening,
Paul Tittel,
Naveen Valluri,
Tristan Naumann,
Carlo Bifulco,
Hoifung Poon
Abstract:
Objective: The majority of detailed patient information in real-world data (RWD) is only consistently available in free-text clinical documents. Manual curation is expensive and time-consuming. Develo** natural language processing (NLP) methods for structuring RWD is thus essential for scaling real-world evidence generation.
Materials and Methods: Traditional rule-based systems are vulnerable…
▽ More
Objective: The majority of detailed patient information in real-world data (RWD) is only consistently available in free-text clinical documents. Manual curation is expensive and time-consuming. Develo** natural language processing (NLP) methods for structuring RWD is thus essential for scaling real-world evidence generation.
Materials and Methods: Traditional rule-based systems are vulnerable to the prevalent linguistic variations and ambiguities in clinical text, and prior applications of machine-learning methods typically require sentence-level or report-level labeled examples that are hard to produce at scale. We propose leveraging patient-level supervision from medical registries, which are often readily available and capture key patient information, for general RWD applications. To combat the lack of sentence-level or report-level annotations, we explore advanced deep-learning methods by combining domain-specific pretraining, recurrent neural networks, and hierarchical attention.
Results: We conduct an extensive study on 135,107 patients from the cancer registry of a large integrated delivery network (IDN) comprising healthcare systems in five western US states. Our deep learning methods attain test AUROC of 94-99% for key tumor attributes and comparable performance on held-out data from separate health systems and states.
Discussion and Conclusion: Ablation results demonstrate clear superiority of these advanced deep-learning methods over prior approaches. Error analysis shows that our NLP system sometimes even corrects errors in registrar labels. We also conduct a preliminary investigation in accelerating registry curation and general RWD structuring via assisted curation for over 1.2 million cancer patients in this healthcare network.
△ Less
Submitted 19 March, 2022;
originally announced March 2022.
