-
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages
Authors:
Thuat Nguyen,
Chien Van Nguyen,
Viet Dac Lai,
Hieu Man,
Nghia Trung Ngo,
Franck Dernoncourt,
Ryan A. Rossi,
Thien Huu Nguyen
Abstract:
The driving factors behind the development of large language models (LLMs) with impressive learning capabilities are their colossal model sizes and extensive training datasets. Along with the progress in natural language processing, LLMs have been frequently made accessible to the public to foster deeper investigation and applications. However, when it comes to training datasets for these LLMs, es…
▽ More
The driving factors behind the development of large language models (LLMs) with impressive learning capabilities are their colossal model sizes and extensive training datasets. Along with the progress in natural language processing, LLMs have been frequently made accessible to the public to foster deeper investigation and applications. However, when it comes to training datasets for these LLMs, especially the recent state-of-the-art models, they are often not fully disclosed. Creating training data for high-performing LLMs involves extensive cleaning and deduplication to ensure the necessary level of quality. The lack of transparency for training data has thus hampered research on attributing and addressing hallucination and bias issues in LLMs, hindering replication efforts and further advancements in the community. These challenges become even more pronounced in multilingual learning scenarios, where the available multilingual text datasets are often inadequately collected and cleaned. Consequently, there is a lack of open-source and readily usable dataset to effectively train LLMs in multiple languages. To overcome this issue, we present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages, tailored for LLM development. Our dataset undergoes meticulous cleaning and deduplication through a rigorous pipeline of multiple stages to accomplish the best quality for model training, including language identification, URL-based filtering, metric-based cleaning, document refinement, and data deduplication. CulturaX is fully released to the public in HuggingFace to facilitate research and advancements in multilingual LLMs: https://huggingface.co/datasets/uonlp/CulturaX.
△ Less
Submitted 17 September, 2023;
originally announced September 2023.
-
Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
Authors:
Viet Dac Lai,
Chien Van Nguyen,
Nghia Trung Ngo,
Thuat Nguyen,
Franck Dernoncourt,
Ryan A. Rossi,
Thien Huu Nguyen
Abstract:
A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercia…
▽ More
A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at https://github.com/nlp-uoregon/Okapi.
△ Less
Submitted 1 August, 2023; v1 submitted 29 July, 2023;
originally announced July 2023.
-
ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
Authors:
Viet Dac Lai,
Nghia Trung Ngo,
Amir Pouran Ben Veyseh,
Hieu Man,
Franck Dernoncourt,
Trung Bui,
Thien Huu Nguyen
Abstract:
Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting ap…
▽ More
Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. While this work will be an ongoing effort to include additional experiments in the future, our current paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. We also focus on the zero-shot learning setting for ChatGPT to improve reproducibility and better simulate the interactions of general users. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.
△ Less
Submitted 12 April, 2023;
originally announced April 2023.
-
Indecomposable characters on direct limit of symmetric groups with diagonal embeddings
Authors:
N. Nessonov,
N. T. S. Ngo
Abstract:
In this paper we obtain the complete description of all indecomposable characters (central positive-definite functions) of inductive limits of the symmetric groups under block diagonal embedding.
As a corollary we obtain the full classification of the isomorphism classes of these inductive limits.
In this paper we obtain the complete description of all indecomposable characters (central positive-definite functions) of inductive limits of the symmetric groups under block diagonal embedding.
As a corollary we obtain the full classification of the isomorphism classes of these inductive limits.
△ Less
Submitted 4 June, 2022;
originally announced June 2022.
-
FAMIE: A Fast Active Learning Framework for Multilingual Information Extraction
Authors:
Minh Van Nguyen,
Nghia Trung Ngo,
Bonan Min,
Thien Huu Nguyen
Abstract:
This paper presents FAMIE, a comprehensive and efficient active learning (AL) toolkit for multilingual information extraction. FAMIE is designed to address a fundamental problem in existing AL frameworks where annotators need to wait for a long time between annotation batches due to the time-consuming nature of model training and data selection at each AL iteration. This hinders the engagement, pr…
▽ More
This paper presents FAMIE, a comprehensive and efficient active learning (AL) toolkit for multilingual information extraction. FAMIE is designed to address a fundamental problem in existing AL frameworks where annotators need to wait for a long time between annotation batches due to the time-consuming nature of model training and data selection at each AL iteration. This hinders the engagement, productivity, and efficiency of annotators. Based on the idea of using a small proxy network for fast data selection, we introduce a novel knowledge distillation mechanism to synchronize the proxy network with the main large model (i.e., BERT-based) to ensure the appropriateness of the selected annotation examples for the main model. Our AL framework can support multiple languages. The experiments demonstrate the advantages of FAMIE in terms of competitive performance and time efficiency for sequence labeling with AL. We publicly release our code (\url{https://github.com/nlp-uoregon/famie}) and demo website (\url{http://nlp.uoregon.edu:9000/}). A demo video for FAMIE is provided at: \url{https://youtu.be/I2i8n_jAyrY}.
△ Less
Submitted 4 May, 2022; v1 submitted 16 February, 2022;
originally announced February 2022.
-
$\mathrm{GL}_n$-structure and principal $\mathfrak{sl}_2$-triple on the cohomology ring of complex Grassmannian
Authors:
Nhok Tkhai Shon Ngo
Abstract:
In this note we describe the cohomology ring of the Grassmannian of $k$-planes in $n$-dimensional complex vector space as an $\mathrm{GL}_n$-module. We give explicit formulas for the operators of its principal $\mathfrak{sl}_2$-triple. It is proved that one of these operators corresponds to the shifted cohomology degree operator and the second operator coincides with the Lefschetz map on cohomolog…
▽ More
In this note we describe the cohomology ring of the Grassmannian of $k$-planes in $n$-dimensional complex vector space as an $\mathrm{GL}_n$-module. We give explicit formulas for the operators of its principal $\mathfrak{sl}_2$-triple. It is proved that one of these operators corresponds to the shifted cohomology degree operator and the second operator coincides with the Lefschetz map on cohomology (as in the hard Lefschetz theorem). We check that the cohomology ring of the complex Grassmannian as a $\mathrm{GL}_n$-representation is isomorphic to the $k$-th exterior power of the standard $n$-dimensional representation.
△ Less
Submitted 16 November, 2021;
originally announced November 2021.
