InCA: Rethinking In-Car Conversational System Assessment Leveraging Large Language Models
Authors:
Ken E. Friedl,
Abbas Goher Khan,
Soumya Ranjan Sahoo,
Md Rashad Al Hasan Rony,
Jana Germies,
Christian Süß
Abstract:
The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the…
▽ More
The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the anticipated system behavior. Within the context of the automotive industry, existing evaluation metrics prove inadequate for assessing in-car conversational question answering (ConvQA) systems. The unique demands of these systems, where answers may relate to driver or car safety and are confined within the car domain, highlight the limitations of current metrics. To address these challenges, this paper introduces a set of KPIs tailored for evaluating the performance of in-car ConvQA systems, along with datasets specifically designed for these KPIs. A preliminary and comprehensive empirical evaluation substantiates the efficacy of our proposed approach. Furthermore, we investigate the impact of employing varied personas in prompts and found that it enhances the model's capacity to simulate diverse viewpoints in assessments, mirroring how individuals with different backgrounds perceive a topic.
△ Less
Submitted 15 November, 2023; v1 submitted 13 November, 2023;
originally announced November 2023.
Conscious Commerce -- Digital Nudging and Sustainable E-commerce Purchase Decisions
Authors:
Milad Mirbabaie,
Julian Marx,
Johanna Germies
Abstract:
So-called 'fast fashion' consumption, amplified through cost-effective e-commerce, constitutes a major factor negatively impacting climate change. A recently noted strategy to motivate consumers to more sustainable decisions is digital nudging. This paper explores the capability of digital nudging in the context of green fashion e-commerce. To do so, digital default and social norm nudges are test…
▽ More
So-called 'fast fashion' consumption, amplified through cost-effective e-commerce, constitutes a major factor negatively impacting climate change. A recently noted strategy to motivate consumers to more sustainable decisions is digital nudging. This paper explores the capability of digital nudging in the context of green fashion e-commerce. To do so, digital default and social norm nudges are tested in an experimental setting of green fashion purchases. An online experiment (n=320) was conducted, simulating an online retail scenario. Results failed to show statistically significant relationships between nudging strategies and purchase decisions. However, explorative analyses show a backfiring effect for the combination of nudges and thus, reveal a hitherto neglected impact of participants' identification on the effectiveness of the digital nudging strategies. Consequently, this study contributes to digital nudging literature and informs practice with new insights on effective choice architectures in e-commerce.
△ Less
Submitted 17 February, 2022;
originally announced February 2022.
