Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese
Authors:
Rifki Afina Putri,
Faiz Ghifari Haznitrama,
Dea Adhista,
Alice Oh
Abstract:
Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in gener…
▽ More
Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ~4.5K questions per language (~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.
△ Less
Submitted 16 April, 2024; v1 submitted 27 February, 2024;
originally announced February 2024.
NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages
Authors:
Samuel Cahyawijaya,
Holy Lovenia,
Fajri Koto,
Dea Adhista,
Emmanuel Dave,
Sarah Oktavianti,
Salsabil Maulana Akbar,
Jhonson Lee,
Nuur Shadieq,
Tjeng Wawan Cenggoro,
Hanung Wahyuning Linuwih,
Bryan Wilie,
Galih Pradipta Muridan,
Genta Indra Winata,
David Moeljadi,
Alham Fikri Aji,
Ayu Purwarianti,
Pascale Fung
Abstract:
Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on develo** labeled and unlabeled corpora for these languages through online scra** and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resul…
▽ More
Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on develo** labeled and unlabeled corpora for these languages through online scra** and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resulting corpora, including a lack of lexical diversity and cultural relevance to local communities. To address this gap, we conduct a case study on Indonesian local languages. We compare the effectiveness of online scra**, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content. In addition, we present the \datasetname{} benchmark, encompassing 12 underrepresented and extremely low-resource languages spoken by millions of individuals in Indonesia. Our empirical experiment results using existing multilingual large language models conclude the need to extend these models to more underrepresented languages. We release the NusaWrites dataset at https://github.com/IndoNLP/nusa-writes.
△ Less
Submitted 19 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
