MedAide: Leveraging Large Language Models for On-Premise Medical Assistance on Edge Devices

Abstract.

Large language models (LLMs) are revolutionizing various domains with their remarkable natural language processing (NLP) abilities. However, deploying LLMs in resource-constrained edge computing and embedded systems presents significant challenges. Another challenge lies in delivering medical assistance in remote areas with limited healthcare facilities and infrastructure. To address this, we introduce MedAide, an on-premise healthcare chatbot. It leverages tiny-LLMs integrated with LangChain, providing efficient edge-based preliminary medical diagnostics and support. MedAide employs model optimizations for minimal memory footprint and latency on embedded edge devices without server infrastructure. The training process is optimized using low-rank adaptation (LoRA). Additionally, the model is trained on diverse medical datasets, employing reinforcement learning from human feedback (RLHF) to enhance its domain-specific capabilities. The system is implemented on various consumer GPUs and Nvidia Jetson development board. MedAide achieves 77% accuracy in medical consultations and scores 56 in USMLE benchmark, enabling an energy-efficient healthcare assistance platform that alleviates privacy concerns due to edge-based deployment, thereby empowering the community.

1. Introduction

In recent years, large language models (LLMs) (Zhao2023, ) have emerged as a groundbreaking technology with the potential to revolutionize multiple domains, and are exceedingly gaining research interest. State-of-the-art LLMs, such as GPT-3 (Brown2020, ) and its successor GPT-4 (Bubeck2023, ) have revolutionized NLP tasks. Other LLMs are used for various generative tasks like robotics and smart homes/cities. These auto-regressive models learn the statistical patterns and semantic structures of human language. With their vast knowledge base, LLMs can generate coherent and contextually relevant text and understand complex queries. This prowess makes them a valuable resource for a wide range of applications including medicine and autonomous cognitive systems (10.3389/frai.2023.1169595, )(jeblick2022chatgpt, ) and serve as a driving force behind our research.

Language and interaction are fundamental elements in medicine, enabling healthcare professionals to gather comprehensive insights (Franz2018, ), develo** a deeper understanding of the individual’s health status. Navigating the landscape of medical applications, there is a compelling need for remote assistance as highlighted in Figure 1. A World Bank and WHO report (who, ) states that half of the world’s population lacks essential health services, with healthcare costs pushing 100 million into extreme poverty annually. This underscores the urgent need for universal health coverage.

Figure 1. Mortality rate for 137 countries due to inadequate healthcare facilities (Kruk2018MortalityDT, ) where MedAide could be effective.

Moreover, the deployment of LLMs on resource-constrained devices while minimizing latency presents a significant challenge, see Figure 2. State-of-the-art LLMs (Brown2020, ) (Bubeck2023, ) (yin2023survey, ) heavily rely on server-based deployment, limiting accessibility, and require significant power consumption and computational resources (Zhao2023, ).

Figure 2. The analysis of embedded devices from NVIDIA, Intel, and Google reveals a significant gap between the capabilities of embedded boards and the requirements of recent LLMs due to escalating memory demands and computational requirements.

Figure 3. Open-LLM Leader-board benchmark (Open-LLM-Leaderboard-Report-2023, ) competing various state-of-the-art LLMs across diverse benchmarks, encompassing TruthfulQA (lin2022truthfulqa, ), MMLU (hendrycks2021measuring, ), ARC (clark2018think, ), and HellaSwag (zellers2019hellaswag, ) for a comprehensive evaluation.

The selection of LLM for edge deployment depends on network parameters, performance metrics, open-source availability, and hardware constraints. We utilized the open-LLM leader-board (Open-LLM-Leaderboard-Report-2023, ), which serves as a valuable platform for comprehensive analysis of LLMs. An analysis was conducted to identify suitable LLMs, and among them, LLaMa2-7B(touvron2023llama, ) emerged as the standout choice owing to its exceptional performance across benchmarks. Moreover, Bloom-560M model (BigScience, ) and OPT-125M (Zhang2022, ) models were selected based on fewer parameter facilitating deployment on resource-constrained embedded devices, open-source availability, and highest TruthfulQA score(lin2022truthfulqa, ) for their parameter range illustrated in Figure. 3.

We conducted additional analysis to benchmark our models against those with significantly larger parameter counts, which are specialized for medical datasets. Our analysis reveals that OPT-125M and Bloom-560M, with a small memory footprint, exhibit modest accuracy scores of 27.6%, 29.5%. These models are particularly attractive due to their relatively lower parameter counts, facilitating easier deployment, thereby achieving low-latency performance. Respectively, LLaMa2-7B achieves 51.9% accuracy on these tests, reflecting its efficacy in medical applications. Figure 4 showcasing their potential for deployment in medical applications.

Figure 4. A comprehensive comparison between selected LLMs (OPT, LLaMa2, and Bloom) and state-of-the-art LLMs (Galactica, Gopher, Chinchilla, and Flan-Palm) to evaluate the performance of these models on different domains, shedding light on their feasibility for medical assistance.

LangChain (langchain, ) plays a crucial role in our model deployment tool-chain through collection of medical databases, optimizing the results through Facebook AI Similarity Search (FAISS) (johnson2019billion, ). LangChain efficiently searches a database, to accelerate the retrieval of medical prescription. This streamlined process enhances the overall USMLE score by 2.5 %, facilitating more effective medical knowledge retrieval and utilization.

To enhance the domain-specific understanding of our models, we curated a custom dataset detailed in Table 1, from various sources including online medical forums, publicly available biomedical databases, and synthesized real-world clinical case studies enabling our models to grasp medical terminologies effectively. By training our models on this specialized dataset, we enrich their knowledge representation and enhance their performance in medical consultation scenarios. Moreover, we devise an experimental validation setup for our models primarily focusing on the doctor-patient interaction aspect which is lacking in most of the available datasets. In this evaluation, chatGPT4 reviews interactive dialogues for authenticity, response effectiveness, and overall performance, ranking MedAide (LLaMa-2) with a 77% accuracy in comparative analysis.

Figure 5. MedAide System overview with the input system requirements and system processes to generate the outputs.

To address the above-discussed research challenges, we present our novel MedAide system with the following contributions:

• MedAide system (overview in Figure 5) enables on-premise healthcare diagnosis by learning from domain specific dataset, by automatically collecting, refining and updating samples.

• We investigated MedAide system with a backbone of three prominent LLM architectures that have demonstrated outstanding performance on the Open-LLM leader-board (Open-LLM-Leaderboard-Report-2023, ). Our analysis is centered around benchmarking their performance, with a particular emphasis on their effectiveness in the TruthfulQA task, by mitigating false answers learned from imitating human texts.

• Our proposed MedAide system leverages LLMs with optimizations, enabling seamless deployment on devices such as Nvidia Jetson or consumer-grade GPUs, by identifying hardware constrains through a rigorous model selection criteria for a specific edge device.

• Towards a practical solution, our system integrates LangChain (langchain, ) to construct toolchains for effective searching of medical databases for accurate prescription and consultation and medical recommendations.

Figure 6. An Overview of Our MedAide: Our system integrates quality requirements and hardware constraints to guide its operation. We employ domain-specific training on a diverse medical dataset to fine-tune LLMs, followed by model quantization. We establish a model selection criteria, ensuring efficient deployment, by integrating LangChain empowered by FAISS.

Paper Organization: Section 2 discusses state-of-the-art. Section 3 presents the MedAide system in further detail. Section 4 describes the evaluation framework, and results are presented in Section 5. We conclude in Section 6.

2. Background and Related Work

LLMs are typically based on transformer architectures (Vaswani2017, ) (Wolf2019TransformersSN, ), which have become the de facto standard for NLP tasks. Transformers leverage attention mechanisms to proficiently capture contextual and long-range dependencies within textual data (Peters2018DeepCW, ), while understanding the chain of discussion. LLMs have made significant contributions in various specialized domains. In healthcare, LLMs have been used for medical diagnosis, clinical decision support, and biomedical text mining. Based on a recent study by Dave et al. (2023) (10.3389/frai.2023.1169595, ), it is evident that ChatGPT and even GPT-4 demonstrate comparatively lower performance in vertical domains, particularly in the field of medicine. This can be attributed, in part, to the potential insufficiency of medical knowledge among annotators. Consequently, there are substantial prospects for further investigation and enhancement within this domain.

Efforts to overcome this limitation and enhance performance in medical domains hold significant promise for advancing the capabilities of language models in healthcare. ChatDoctor (li2023chatdoctor, ), an advanced language model based on the LLaMa model (Touvron2023, ), is specifically designed for medical assistance. It simulates doctor-patient conversations, enabling patients to receive accurate diagnoses, personalized medical advice, and appropriate treatment options. Luo et al. (Luo2022, ) presented BioGPT, a domain-specific generative transformer language model trained on extensive biomedical literature offering generation capabilities tailored specifically for the biomedical domain and achieves superior performance across various biomedical datasets. DrBERT (labrak2023drbert, ), a specialized pre-trained language models (PLMs) for the French medical domain, trained on BERT (Devlin2018, ) introduced a novel approach by leveraging both left and right context during pre-training, resulting in state-of-the-art performance across various medical tasks. HuatuoGPT (huatuogpt-2023, ), a distilled language model for medical consultation, trained using ChatGPT synthesised dialogue and real-world data from doctors, employing RLAIF, achieves state-of-the-art performance in medical consultation. These model have excessive server dependence for deployment, due to high parameter count, constraining the availability on GPUs. Moreover, these LLMs often face a challenge known as ”hallucination,” (hallucination, ) where they generate plausible but factually incorrect or nonsensical information. This occurs due to their reliance on patterns in the data they were trained on, without an intrinsic understanding of truth. LangChain integrated with LLMs addresses this issue by structuring interactions with the model in a way that mitigates hallucinations.

DoctorGLM (xiong2023doctorglm, ), trained using ChatGLM-6B (du2022glm, ) sets a benchmark on Chinese datasets; however, our work outperforms it with a notable 21% increase in overall accuracy. Han et al. (Han2023, ) proposed MedAlpaca addressed the need for open-source models that prioritize patient privacy by develo** specialized dataset for medical applications, highlighting the use-case in medical education. However, MedAide outperforms this work by a considerable margin, as demonstrated in our experimental evaluation section.

3. MedAide: An LLM-based System for medical assistance

In this section we will describe the MedAide system in detail, along with the dataset generation, refinement and training workflow. We elaborate on LLM quantization and model selection along with LangChain integration (see an overview in Figure 6).

3.1. User Requirements

User requirements, pivotal in designing medical applications, dictate both the user interface design and model selection criteria, ensuring alignment with healthcare professionals’ and patients’ needs. Our analysis highlighted key demands: an intuitive user interface, accurate medical results, secure authentication, and adherence to data privacy laws like HIPAA. Accordingly, we crafted a user interface facilitating secure login and local data storage, overseen by authorized personnel. Additionally, our model selection caters to real-time processing and accuracy modes, tailoring the application to meet these critical user expectations effectively.

3.2. Hardware Constraints

There is a trade-off between ensuring high-quality performance and a set of hardware constraints for deployment in consumer-grade GPUs and embedded devices as highlighted in Figure 2. These constraints ensure that LLMs remain within the computational and memory capabilities of the target platforms. The metrics such as storage overhead, based on network parameters, and floating-point operations, quantifying the execution time of the LLMs on GPUs are employed. Given the resource constraints, the trade-off is considering model size for hardware specifications.

3.3. Dataset Construction & Refinement

To prepare our dataset, we conducted an extensive survey of online forums and discussions where individuals post information about their symptoms and seek professional medical advice. We identified several notable forums for data collection, including WebMD, AskDocs, HealthcareMagic, eHealthForum, Icliniq, and HealthTap offering a diverse range of discussions and medical queries and potential to provide data without introducing inherent biases. Additionally, we gathered data from several medical databases for medicine, and clinical records. These online resources provided a significant contribution towards our model training.

In some datasets, particularly those obtained from platforms like Kaggle, the original format was in csv, featuring columns such as “Disease,” “Symptom,” “Reason,” “Tests And Procedures,” and “Common Medications.” To convert this csv data into a question-answer format, we performed data augmentation techniques using a high-fidelity pre-trained LLaMA-70B model. Each row of the csv file was transformed into a common English question using the formulation: “What are the symptoms, reasons, tests and procedures, and common medications for Panic disorder?” Each row was then converted into a common English question-answer pair in the format:

• The “instruction” key contained the original question.

• The “output” key contained the doctor’s response.

• An ”input” field for prompt type, and previous discussion.

In some cases, we encountered datasets collected in the Chinese language. To address this challenge, we utilized Google Translate APIs to translate the dataset into English. After compiling the dataset, we conducted an exploratory data refinement phase to filter out incomplete questions, deleted questions, and personal details ensuring the quality and reliability of the dataset. The final dataset comprised approximately 400k instruction-output pairs derived from the mentioned sources.

Table 1. Dataset Collection from Different Sources

Source	# Samples	URL
HealthcareMagic	112,641	www.healthcaremagic.com/
WebMD	88,207	www.webmd.com/
AskDocs	24,256	www.askdocs.com
iCliniq	4,651	www.icliniq.com/
HealthTap	3,647	www.healthtap.com/
ehealth	1,710	medicalforums.omeka.net
Huato-26M	85,000	https://doi.org/10.48550
MedQuAD	47,457	github.com/abachaa/MedQuAD
MedMCQa	25,679	https://doi.org/10.48550
MedQSum	1000	http://doi.org/10.18653
Medical Cases	4363	www.github.com/adahealth

3.4. Domain Specific Training Methodology

Training LLMs such as LLaMa 2-7B (Touvron2023, ) and Bloom-560M (BigScience, ) on consumer devices is challenging due to resource limitations. We used Low Rank Adaptation (LoRA) (hu2022lora, ) to approximate high-dimensional datasets in lower-dimensional spaces, preserving key features. This technique reduced trainable parameters and GPU memory usage significantly. Bloom-560M, optimized for computational efficiency and accuracy, includes self-attention and feed-forward networks. Similarly, we trained OPT-125M, a model with 12 layers and attention heads, suitable for various hardware configurations used by end-users. Additionally, our approach incorporated Reinforcement Learning from Human Feedback (RLHF) to train a reward model. This model played a pivotal role in refining the training process by providing valuable, human-centric insights. We utilized this reward model to iteratively train a Proximal Policy Optimization (PPO) model, leveraging policy-based rewards. This integration of RLHF allowed for a more nuanced and effective training process, as the PPO model dynamically adjusted its learning strategy based on human feedback. This feedback loop ensured that the model’s outputs not only adhered to technical accuracy but also aligned closely with practical, real-world applicability and user expectations. By embedding human feedback directly into the training cycle, we achieved a more robust and contextually aware LLM, capable of addressing the nuanced demands of medical applications with greater precision. Our implementation followed a structured three-step process as highlighted in Figure 7.

Figure 7. Model Training Workflow.

3.5. Model Selection and Optimization

Our model selection and optimization process for the LLaMA 2-7B models includes quantization in Q4, Q8, and F16 formats to ensure efficient deployment. Q4 quantization reduces precision to 4 bits, offering smaller model sizes but potentially less accuracy, and is incompatible with Nvidia Jetson boards. Q8 quantization, with 8-bit precision, balances performance and accuracy, while F16 quantization uses 16-bit floating-point representation for complex tasks with reduced size and computational demands. After quantization, the models are calibrated using weight clip**, adjusting weight values to minimize precision loss and maintain performance.

The model selection criteria is designed to ensure efficient deployment on the target hardware. We begin by assessing the compatibility of models with the target hardware, particularly noting that NVIDIA boards do not support Q4 models. This ensures that the chosen model is viable for the intended device. We evaluate metrics such as memory usage, inference speed, and the balance between model accuracy and resource consumption. This evaluation is key to identifying the most efficient model within the hardware’s operational constraints. The selection is further refined based on user-defined application requirements, such as the need for real-time processing or a focus on accuracy. This step ensures the model’s alignment with the specific functional demands of the application. Finally, we conduct a comparative analysis of various LLMs, weighing the trade-offs each presents. This comprehensive analysis covers the model’s feasibility on the device and its conformity to user requirements regarding processing speed and accuracy, ensuring a well-rounded selection process.

In our comprehensive dataset collection, we leveraged medical databases, e-books, and documents for LangChain (langchain, ) integration. To enhance processing efficiency, we segmented this extensive medical knowledge into 1000-character blocks with a 50-character overlap. Leveraging the robust capabilities of Hugging Face Instruct, we generated embeddings for each block, storing them in our local infrastructure with the support of the FAISS library (johnson2019billion, ). Notably, we optimized performance by utilizing the GPU-accelerated version of FAISS, resulting in a remarkable speed boost—achieving search speeds 5-10 x faster than the CPU counterpart.

In response to a user query, we employ FAISS to identify the two closest neighbors using its similarity search function. These data chunks are then presented as contextual input to our prompt chain, in conjunction with the selected LLMs. We significantly alleviate the common issue of hallucination in LLMs. This approach ensures that the medical guidance provided is not only precise but also grounded in reliable and verified medical content, enhancing the overall safety and dependability of the advice offered.

4. Evaluation Setup

The evaluation setup employs a wide range of software and hardware resources for thorough testing and benchmarking. The software environment integrates various tools such as Python 3.10, Cuda 11.8, PyTorch, Lightning, and other essential libraries, along with FAISS for similarity search. Hardware-wise, the setup features diverse Nvidia GPUs, ranging from high-end models like RTX 6000 Ada to embedded solutions like Jetson AGX Xavier, facilitating a comprehensive evaluation of model performance across different computational environments, as indicated in Figure 8.

The methodology for evaluation involves deploying models on these hardware platforms to assess key metrics such as accuracy, runtime, and benchmark scores. The USMLE and ChatGPT-4 Scores serve as evaluation metrics described later in Section 5.2, providing a standardized measure of model performance.

Figure 8. Experimental evaluation setup deployed on Python 3.10 and tested on various testing platforms, and evaluation based on metrics and resources.

5. Results

5.1. Quantitative Analysis of Models

In our research, we performed a quantitative analysis of MedAide models using the USMLE QA dataset, a standard for evaluating LLMs in the medical domiain, depicted in Figure 9, also comparing models’ performance and GPU memory consumption with SOTA LLMs, highlighting the efficiency. Figure 10 complements this by illustrating latency differences across a range of devices, crucial for practical healthcare applications.

Figure 9. Performance and Memory comparison for different LLM configurations and their respective USMLE scores

Figure 10. Latency comparison on GPUs architectures, providing a comprehensive overview of model performance across devices.

5.2. Qualitative Analysis of Models

This model comparison evaluates the performance of different models by examining their responses to 100 single-turn medical questions. Specifically, the comparison focuses on assessing the ratio of model performance for various categories. The evaluation was conducted using chatGPT4 for reviewing and analyzing the models’ outputs, since it scores significantly higher on majority of the benchmarks, considering it as a baseline for performance. The aim of this comparison is to gain insights into how the models perform across various categories and to identify any variations or differences in their effectiveness in health care assessment. Figure 11 presents these findings, and our analysis reveals that our LLaMA 2-7B+FAISS model outperforms MedAlpaca and DoctorGLM by 21% and 13%, respectively. This significant margin underscores the efficacy of the LLaMA 2-7B+FAISS model in the evaluated scenarios.

Figure 11. Model perf. ratio for each category evaluated on 100 single-turn questions, reviewed by ChatGPT4.

6. Conclusion

This paper presented MedAide, an on-premise medical assistant powered by LLMs. MedAide utilizes specialized medical dataset to provide accurate and reliable healthcare support, including answering medical queries, offering personalized recommendations, and aiding in diagnostics. The results demonstrated the effectiveness of MedAide in various medical domains, showcasing its ability to comprehend complex medical queries, generate informative responses, and assist in clinical decision-making. The results showcase its potential to improve medical workflows, diagnostics, and patient care. Integrating MedAide into real-world healthcare settings holds great promise in enhancing the delivery of healthcare services.