Showing 1–1 of 1 results for author: Fuccella, V
-
LLMs in HCI Data Work: Bridging the Gap Between Information Retrieval and Responsible Research Practices
Authors:
Neda Taghizadeh Serajeh,
Iman Mohammadi,
Vittorio Fuccella,
Mattia De Rosa
Abstract:
Efficient and accurate information extraction from scientific papers is significant in the rapidly develo** human-computer interaction research in the literature review process. Our paper introduces and analyses a new information retrieval system using state-of-the-art Large Language Models (LLMs) in combination with structured text analysis techniques to extract experimental data from HCI liter…
▽ More
Efficient and accurate information extraction from scientific papers is significant in the rapidly develo** human-computer interaction research in the literature review process. Our paper introduces and analyses a new information retrieval system using state-of-the-art Large Language Models (LLMs) in combination with structured text analysis techniques to extract experimental data from HCI literature, emphasizing key elements. Then We analyze the challenges and risks of using LLMs in the world of research. We performed a comprehensive analysis on our conducted dataset, which contained the specified information of 300 CHI 2020-2022 papers, to evaluate the performance of the two large language models, GPT-3.5 (text-davinci-003) and Llama-2-70b, paired with structured text analysis techniques. The GPT-3.5 model gains an accuracy of 58\% and a mean absolute error of 7.00. In contrast, the Llama2 model indicates an accuracy of 56\% with a mean absolute error of 7.63. The ability to answer questions was also included in the system in order to work with streamlined data. By evaluating the risks and opportunities presented by LLMs, our work contributes to the ongoing dialogue on establishing methodological validity and ethical guidelines for LLM use in HCI data work.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
