Can GPT-4 Help Detect Quit Va** Intentions? An Exploration of Automatic Data Annotation Approach
Authors:
Sai Krishna Revanth Vuruma,
Dezhi Wu,
Saborny Sen Gupta,
Lucas Aust,
Valerie Lookingbill,
Wyatt Bellamy,
Yang Ren,
Erin Kasson,
Li-Shiun Chen,
Patricia Cavazos-Rehg,
Dian Hu,
Ming Huang
Abstract:
In recent years, the United States has witnessed a significant surge in the popularity of va** or e-cigarette use, leading to a notable rise in cases of e-cigarette and va** use-associated lung injury (EVALI) that caused hospitalizations and fatalities during the EVALI outbreak in 2019, highlighting the urgency to comprehend va** behaviors and develop effective strategies for cessation. Due…
▽ More
In recent years, the United States has witnessed a significant surge in the popularity of va** or e-cigarette use, leading to a notable rise in cases of e-cigarette and va** use-associated lung injury (EVALI) that caused hospitalizations and fatalities during the EVALI outbreak in 2019, highlighting the urgency to comprehend va** behaviors and develop effective strategies for cessation. Due to the ubiquity of social media platforms, over 4.7 billion users worldwide use them for connectivity, communications, news, and entertainment with a significant portion of the discourse related to health, thereby establishing social media data as an invaluable organic data resource for public health research. In this study, we extracted a sample dataset from one va** sub-community on Reddit to analyze users' quit-va** intentions. Leveraging OpenAI's latest large language model GPT-4 for sentence-level quit va** intention detection, this study compares the outcomes of this model against layman and clinical expert annotations. Using different prompting strategies such as zero-shot, one-shot, few-shot and chain-of-thought prompting, we developed 8 prompts with varying levels of detail to explain the task to GPT-4 and also evaluated the performance of the strategies against each other. These preliminary findings emphasize the potential of GPT-4 in social media data analysis, especially in identifying users' subtle intentions that may elude human detection.
△ Less
Submitted 28 June, 2024;
originally announced July 2024.
Utilizing Large Language Models to Identify Reddit Users Considering Va** Cessation for Digital Interventions
Authors:
Sai Krishna Revanth Vuruma,
Dezhi Wu,
Saborny Sen Gupta,
Lucas Aust,
Valerie Lookingbill,
Caleb Henry,
Yang Ren,
Erin Kasson,
Li-Shiun Chen,
Patricia Cavazos-Rehg,
Dian Hu,
Ming Huang
Abstract:
The widespread adoption of social media platforms globally not only enhances users' connectivity and communication but also emerges as a vital channel for the dissemination of health-related information, thereby establishing social media data as an invaluable organic data resource for public health research. The surge in popularity of va** or e-cigarette use in the United States and other countr…
▽ More
The widespread adoption of social media platforms globally not only enhances users' connectivity and communication but also emerges as a vital channel for the dissemination of health-related information, thereby establishing social media data as an invaluable organic data resource for public health research. The surge in popularity of va** or e-cigarette use in the United States and other countries has caused an outbreak of e-cigarette and va** use-associated lung injury (EVALI), leading to hospitalizations and fatalities in 2019, highlighting the urgency to comprehend va** behaviors and develop effective strategies for cession. In this study, we extracted a sample dataset from one va** sub-community on Reddit to analyze users' quit va** intentions. Leveraging large language models including both the latest GPT-4 and traditional BERT-based language models for sentence-level quit-va** intention prediction tasks, this study compares the outcomes of these models against human annotations. Notably, when compared to human evaluators, GPT-4 model demonstrates superior consistency in adhering to annotation guidelines and processes, showcasing advanced capabilities to detect nuanced user quit-va** intentions that human evaluators might overlook. These preliminary findings emphasize the potential of GPT-4 in enhancing the accuracy and reliability of social media data analysis, especially in identifying subtle users' intentions that may elude human detection.
△ Less
Submitted 25 April, 2024;
originally announced April 2024.
