KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models

Pretrained LLM (Large Language Models, Shanahan (2022); Brown et al. (2020); Taylor et al. (2022); Chowdhery et al. (2022); Touvron et al. (2023) are Transformer-based language models (Vaswani et al., 2017) with an extensive parameter count, typically in the hundreds of billions or beyond, and they undergo training on vast corpora. These models are typically employed for their intended purposes following instruction tuning methodology, a technique aimed at optimizing models to effectively adhere to user-provided instructions.

Since the release of InstructGPT (Ouyang et al., 2022), instruction datasets in English have been widely developed and made publicly available across various domains such as Ouyang et al. (2022); Taori et al. (2023a). However, for Korean, which falls into the category of a relatively low-resource language, there is a notable absence of datasets created in the native language. Notably, the Korean instruction datasets currently available are either created by translating existing English datasets using DeepL¹¹1https://www.deepl.com/translator or by relying on outputs from large language models like the ChatGPT API²²2https://openai.com/blog/chatgpt, without adequately capturing the cultural nuances of the Korean language (Table 1). Hence, to enhance and advance the performance of Korean Large Language Models, it is imperative to have datasets constructed specifically in the Korean language.

We introduce KIT-19 (A Comprehensive Korean Instruction Toolkit on 19 Tasks), a universal dataset for Korean Instruction Tuning. KIT-19 is a dataset constructed for Instruction Tuning, derived from 19 different NLP datasets in the Korean language, each consisting of 5,000 examples. It follows the methodology of Longpre et al. (2023); Bach et al. (2022) which integrates existing NLP datasets into an Instruction Dataset, without relying on machine-translated outputs from other languages or utilizing LLM output as a training dataset. KIT-19, as compared to the datasets currently employed for Korean LLM modeling, effectively captures the cultural features of the Korean language and, due to the comprehensive nature of its 19 distinct datasets, contributes to the model’s generalizability.

In this paper, we transparently disclose the construction process of KIT-19 and provide detailed information about the source of the 19 datasets, respectively. To assess the quality of the KIT-19 dataset, we conduct Full Fine-tuning of our dataset on Polyglot-Ko-5.8b and Polyglot-Ko-1.3b Ko et al. (2023), which are the Korean representative Pretrained LLMs.

Finally, we evaluate Korean Large Language Models publicly available on a total of 6 benchmark datasets for assessment. The results show that the performance of the Polyglot-Ko-5.8b model trained with KIT-19 outperforms others, and we could also observe that the Polyglot-Ko-1.3b model exhibits higher performance compared to other Korean LLMs.

The contributions of our study are as follows:

• •

  We construct and release 100K Korean instruction datasets, addressing the data scarcity problem and reducing the reliance on translated or GPT-generated instructions for Korean LLMs.

• •

  We demonstrate the efficacy of our KIT-19 by evaluating LLMs using various benchmarks, comparing those trained with KIT-19 and those without.