-
An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
Authors:
Sulaiman Khan,
Md. Rafiul Biswas,
Alina Murad,
Hazrat Ali,
Zubair Shah
Abstract:
Recent developments in multimodal large language models (MLLMs) have spurred significant interest in their potential applications across various medical imaging domains. On the one hand, there is a temptation to use these generative models to synthesize realistic-looking medical image data, while on the other hand, the ability to identify synthetic image data in a pool of data is also significantl…
▽ More
Recent developments in multimodal large language models (MLLMs) have spurred significant interest in their potential applications across various medical imaging domains. On the one hand, there is a temptation to use these generative models to synthesize realistic-looking medical image data, while on the other hand, the ability to identify synthetic image data in a pool of data is also significantly important. In this study, we explore the potential of the Gemini (\textit{gemini-1.0-pro-vision-latest}) and GPT-4V (gpt-4-vision-preview) models for medical image analysis using two modalities of medical image data. Utilizing synthetic and real imaging data, both Gemini AI and GPT-4V are first used to classify real versus synthetic images, followed by an interpretation and analysis of the input images. Experimental results demonstrate that both Gemini and GPT-4 could perform some interpretation of the input images. In this specific experiment, Gemini was able to perform slightly better than the GPT-4V on the classification task. In contrast, responses associated with GPT-4V were mostly generic in nature. Our early investigation presented in this work provides insights into the potential of MLLMs to assist with the classification and interpretation of retinal fundoscopy and lung X-ray images. We also identify key limitations associated with the early investigation study on MLLMs for specialized tasks in medical image analysis.
△ Less
Submitted 2 June, 2024;
originally announced June 2024.
-
Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
Authors:
Ashhadul Islam,
Md. Rafiul Biswas,
Wajdi Zaghouani,
Samir Brahim Belhaouari,
Zubair Shah
Abstract:
$ $The synergy of language and vision models has given rise to Large Language and Vision Assistant models (LLVAs), designed to engage users in rich conversational experiences intertwined with image-based queries. These comprehensive multimodal models seamlessly integrate vision encoders with Large Language Models (LLMs), expanding their applications in general-purpose language and visual comprehen…
▽ More
$ $The synergy of language and vision models has given rise to Large Language and Vision Assistant models (LLVAs), designed to engage users in rich conversational experiences intertwined with image-based queries. These comprehensive multimodal models seamlessly integrate vision encoders with Large Language Models (LLMs), expanding their applications in general-purpose language and visual comprehension. The advent of Large Multimodal Models (LMMs) heralds a new era in Artificial Intelligence (AI) assistance, extending the horizons of AI utilization. This paper takes a unique perspective on LMMs, exploring their efficacy in performing image classification tasks using tailored prompts designed for specific datasets. We also investigate the LLVAs zero-shot learning capabilities. Our study includes a benchmarking analysis across four diverse datasets: MNIST, Cats Vs. Dogs, Hymnoptera (Ants Vs. Bees), and an unconventional dataset comprising Pox Vs. Non-Pox skin images. The results of our experiments demonstrate the model's remarkable performance, achieving classification accuracies of 85\%, 100\%, 77\%, and 79\% for the respective datasets without any fine-tuning. To bolster our analysis, we assess the model's performance post fine-tuning for specific tasks. In one instance, fine-tuning is conducted over a dataset comprising images of faces of children with and without autism. Prior to fine-tuning, the model demonstrated a test accuracy of 55\%, which significantly improved to 83\% post fine-tuning. These results, coupled with our prior findings, underscore the transformative potential of LLVAs and their versatile applications in real-world scenarios.
△ Less
Submitted 29 December, 2023;
originally announced January 2024.
-
Potentials of ChatGPT for Annotating Vaccine Related Tweets
Authors:
Md. Rafiul Biswas,
Farida Mohsen,
Zubair Shah,
Wajdi Zaghouani
Abstract:
This study evaluates ChatGPT's performance in annotating vaccine-related Arabic tweets by comparing its annotations with human annotations. A dataset of 2,100 tweets representing various factors contributing to vaccine hesitancy was examined. Two domain experts annotated the data, with a third resolving conflicts. ChatGPT was then employed to annotate the same dataset using specific prompts for ea…
▽ More
This study evaluates ChatGPT's performance in annotating vaccine-related Arabic tweets by comparing its annotations with human annotations. A dataset of 2,100 tweets representing various factors contributing to vaccine hesitancy was examined. Two domain experts annotated the data, with a third resolving conflicts. ChatGPT was then employed to annotate the same dataset using specific prompts for each factor. The ChatGPT annotations were evaluated through zero-shot, one-shot, and few-shot learning tests, with an average accuracy of 82.14%, 83.85%, and 85.57%, respectively. Precision averaged around 86%, minimizing false positives. The average recall and F1-score ranged from 0.74 to 0.80 and 0.65 to 0.93, respectively. AUC for zero-shot, one-shot, and few-shot learning was 0.79, 0.80, and 0.83. In cases of ambiguity, both human annotators and ChatGPT faced challenges. These findings suggest that ChatGPT holds promise as a tool for annotating vaccine-related tweets.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Can ChatGPT be Your Personal Medical Assistant?
Authors:
Md. Rafiul Biswas,
Ashhadul Islam,
Zubair Shah,
Wajdi Zaghouani,
Samir Brahim Belhaouari
Abstract:
The advanced large language model (LLM) ChatGPT has shown its potential in different domains and remains unbeaten due to its characteristics compared to other LLMs. This study aims to evaluate the potential of using a fine-tuned ChatGPT model as a personal medical assistant in the Arabic language. To do so, this study uses publicly available online questions and answering datasets in Arabic langua…
▽ More
The advanced large language model (LLM) ChatGPT has shown its potential in different domains and remains unbeaten due to its characteristics compared to other LLMs. This study aims to evaluate the potential of using a fine-tuned ChatGPT model as a personal medical assistant in the Arabic language. To do so, this study uses publicly available online questions and answering datasets in Arabic language. There are almost 430K questions and answers for 20 disease-specific categories. GPT-3.5-turbo model was fine-tuned with a portion of this dataset. The performance of this fine-tuned model was evaluated through automated and human evaluation. The automated evaluations include perplexity, coherence, similarity, and token count. Native Arabic speakers with medical knowledge evaluated the generated text by calculating relevance, accuracy, precision, logic, and originality. The overall result shows that ChatGPT has a bright future in medical assistance.
△ Less
Submitted 19 December, 2023;
originally announced December 2023.
-
Influences of social media usage on public attitudes and behavior towards COVID-19 vaccine in the Arab world
Authors:
Md. Rafiul Biswas,
Hazrat Ali,
Raian Ali,
Zubair Shah
Abstract:
Background: Vaccination programs are effective only when a significant percentage of people are vaccinated. However, vaccine acceptance varies among communities around the world. Social media usage is arguably one of the factors affecting public attitudes towards vaccines. Objective: This study aims to identify if the social media usages factors can be used to predict attitudes and behavior toward…
▽ More
Background: Vaccination programs are effective only when a significant percentage of people are vaccinated. However, vaccine acceptance varies among communities around the world. Social media usage is arguably one of the factors affecting public attitudes towards vaccines. Objective: This study aims to identify if the social media usages factors can be used to predict attitudes and behavior towards the COVID-19 vaccines among the people in the Arab world. Methods: An online survey was conducted in the Arab countries and 217 Arab people participated in this study. Logistic regression was applied to identify what demographics and social media usage factors predict public attitudes and behavior towards the COVID-19 vaccines. Results: Of the 217 participants, 56.22% of them were willing to accept the vaccine and 41.47% of them were hesitant. This study shows that none of the social media usages factors were significant enough to predict the actual vaccine acceptance behavior. Whereas the analysis showed few of the social media usage factors can predict public attitudes towards the COVID-19 vaccines. For example, frequent social media users were 2.85 times more likely to agree that the risk of COVID-19 is being exaggerated (OR=2.85, 95% CI=0.86-9.45, p=0.046) than infrequent social media users. Whereas participants having more trust in vaccine information shared by their contacts are less likely to agree that decision-makers have verified that vaccines are safe (OR=0.528, 95% CI= 0.276-1.012, p=0.05). Conclusion: The use of social media and information shared on it may affect public attitudes towards COVID-19 vaccines. Therefore, disseminating correct and validated information about COVID-19 and other vaccines on social media is important for increasing public trust and countering the impact of incorrect and misinformation.
△ Less
Submitted 8 June, 2022; v1 submitted 20 January, 2022;
originally announced January 2022.
-
Characterising Communities of Twitter Users Who Posted Vaccines Related Tweets by Information Sources
Authors:
Md. Rafiul Biswas
Abstract:
Objective: We formed community structure based on web page credibility and we measured the types of information for characterizing communities of tweeter users who posted about tweets related to vaccine. Methods: We performed the experiment on only Twitter data (tweets) regarding vaccine. The duration of data collection was between 17 January 2017 and 14 March 2018. We formulated cluster based on…
▽ More
Objective: We formed community structure based on web page credibility and we measured the types of information for characterizing communities of tweeter users who posted about tweets related to vaccine. Methods: We performed the experiment on only Twitter data (tweets) regarding vaccine. The duration of data collection was between 17 January 2017 and 14 March 2018. We formulated cluster based on the information on its contents and sources it resides (i.e., website domains). We only focused the topics which were related to vaccine. To detect the structure and network of community, we applied Louvain community algorithm along with Random walks called Info map method over vaccines related tweeter user. We defined the communities based on various measures derived from the information shared by Twitter users. Representations and visualizations of the communities based on these derived measures help the public health organization to make understand the possible cause of the rejecting the safety and efficacy of vaccines. Results: To analyse people perception over social media, we downloaded 6,591,566 tweets. We distinguished 1,860,662 users who were posting related to vaccine. We applied Louvain community detection algorithm along with DMM values and we found 192 communities. It also produced higher alignment values as long as the number of topics were low. With the increase of topics number, the alignment becomes lower. The total tweets were divided into two parts based on their characterizes. The first category contained 163,148 (57.16%) tweets which were based as evidence and advocacy and the second category contained 6244 (2.19%) tweets which were based on the experiences and opinion. We observed 4548 users were posting about experiential tweets about vaccine and of them 3449 users (75.84%) were posted evidence and advocacy related.
△ Less
Submitted 21 March, 2021;
originally announced March 2021.
