-
Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages
Authors:
Samuel Cahyawijaya,
Holy Lovenia,
Fajri Koto,
Rifki Afina Putri,
Emmanuel Dave,
Jhonson Lee,
Nuur Shadieq,
Wawan Cenggoro,
Salsabil Maulana Akbar,
Muhammad Ihza Mahendra,
Dea Annisayanti Putri,
Bryan Wilie,
Genta Indra Winata,
Alham Fikri Aji,
Ayu Purwarianti,
Pascale Fung
Abstract:
Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder…
▽ More
Large language models (LLMs) show remarkable human-like capability in various domains and languages. However, a notable quality gap arises in low-resource languages, e.g., Indonesian indigenous languages, rendering them ineffective and inefficient in such linguistic contexts. To bridge this quality gap, we introduce Cendol, a collection of Indonesian LLMs encompassing both decoder-only and encoder-decoder architectures across a range of model sizes. We highlight Cendol's effectiveness across a diverse array of tasks, attaining 20% improvement, and demonstrate its capability to generalize to unseen tasks and indigenous languages of Indonesia. Furthermore, Cendol models showcase improved human favorability despite their limitations in capturing indigenous knowledge and cultural values in Indonesia. In addition, we discuss the shortcomings of parameter-efficient tunings, such as LoRA, for language adaptation. Alternatively, we propose the usage of vocabulary adaptation to enhance efficiency. Lastly, we evaluate the safety of Cendol and showcase that safety in pre-training in one language such as English is transferable to low-resource languages, such as Indonesian, even without RLHF and safety fine-tuning.
△ Less
Submitted 9 April, 2024;
originally announced April 2024.
-
NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages
Authors:
Samuel Cahyawijaya,
Holy Lovenia,
Fajri Koto,
Dea Adhista,
Emmanuel Dave,
Sarah Oktavianti,
Salsabil Maulana Akbar,
Jhonson Lee,
Nuur Shadieq,
Tjeng Wawan Cenggoro,
Hanung Wahyuning Linuwih,
Bryan Wilie,
Galih Pradipta Muridan,
Genta Indra Winata,
David Moeljadi,
Alham Fikri Aji,
Ayu Purwarianti,
Pascale Fung
Abstract:
Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on develo** labeled and unlabeled corpora for these languages through online scra** and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resul…
▽ More
Democratizing access to natural language processing (NLP) technology is crucial, especially for underrepresented and extremely low-resource languages. Previous research has focused on develo** labeled and unlabeled corpora for these languages through online scra** and document translation. While these methods have proven effective and cost-efficient, we have identified limitations in the resulting corpora, including a lack of lexical diversity and cultural relevance to local communities. To address this gap, we conduct a case study on Indonesian local languages. We compare the effectiveness of online scra**, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content. In addition, we present the \datasetname{} benchmark, encompassing 12 underrepresented and extremely low-resource languages spoken by millions of individuals in Indonesia. Our empirical experiment results using existing multilingual large language models conclude the need to extend these models to more underrepresented languages. We release the NusaWrites dataset at https://github.com/IndoNLP/nusa-writes.
△ Less
Submitted 19 September, 2023; v1 submitted 19 September, 2023;
originally announced September 2023.
-
Transfer-Learning-Aware Neuro-Evolution for Diseases Detection in Chest X-Ray Images
Authors:
Albert Susanto,
Herman,
Tjeng Wawan Cenggoro,
Suharjito,
Bens Pardamean
Abstract:
The neural network needs excessive costs of time because of the complexity of architecture when trained on images. Transfer learning and fine-tuning can help improve time and cost efficiency when training a neural network. Yet, Transfer learning and fine-tuning needs a lot of experiment to try with. Therefore, a method to find the best architecture for transfer learning and fine-tuning is needed.…
▽ More
The neural network needs excessive costs of time because of the complexity of architecture when trained on images. Transfer learning and fine-tuning can help improve time and cost efficiency when training a neural network. Yet, Transfer learning and fine-tuning needs a lot of experiment to try with. Therefore, a method to find the best architecture for transfer learning and fine-tuning is needed. To overcome this problem, neuro-evolution using a genetic algorithm can be used to find the best architecture for transfer learning. To check the performance of this study, dataset ChestX-Ray 14 and DenseNet-121 as a base neural network model are used. This study used the AUC score, differences in execution time for training, and McNemar's test to the significance test. In terms of result, this study got a 5% difference in the AUC score, 3 % faster in terms of execution time, and significance in most of the disease detection. Finally, this study gives a concrete summary of how neuro-evolution transfer learning can help in terms of transfer learning and fine-tuning.
△ Less
Submitted 15 April, 2020;
originally announced April 2020.
-
Deep Learning for Imbalance Data Classification using Class Expert Generative Adversarial Network
Authors:
Fanny,
Tjeng Wawan Cenggoro
Abstract:
Without any specific way for imbalance data classification, artificial intelligence algorithm cannot recognize data from minority classes easily. In general, modifying the existing algorithm by assuming that the training data is imbalanced, is the only way to handle imbalance data. However, for a normal data handling, this way mostly produces a deficient result. In this research, we propose a clas…
▽ More
Without any specific way for imbalance data classification, artificial intelligence algorithm cannot recognize data from minority classes easily. In general, modifying the existing algorithm by assuming that the training data is imbalanced, is the only way to handle imbalance data. However, for a normal data handling, this way mostly produces a deficient result. In this research, we propose a class expert generative adversarial network (CE-GAN) as the solution for imbalance data classification. CE-GAN is a modification in deep learning algorithm architecture that does not have an assumption that the training data is imbalance data. Moreover, CE-GAN is designed to identify more detail about the character of each class before classification step. CE-GAN has been proved in this research to give a good performance for imbalance data classification.
△ Less
Submitted 12 July, 2018; v1 submitted 12 July, 2018;
originally announced July 2018.
