-
AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages
Authors:
Odunayo Ogundepo,
Tajuddeen R. Gwadabe,
Clara E. Rivera,
Jonathan H. Clark,
Sebastian Ruder,
David Ifeoluwa Adelani,
Bonaventure F. P. Dossou,
Abdou Aziz DIOP,
Claytone Sikasote,
Gilles Hacheme,
Happy Buzaaba,
Ignatius Ezeani,
Rooweither Mabuya,
Salomey Osei,
Chris Emezue,
Albert Njoroge Kahira,
Shamsuddeen H. Muhammad,
Akintunde Oladipo,
Abraham Toluwase Owodunni,
Atnafu Lambebo Tonja,
Iyanuoluwa Shode,
Akari Asai,
Tunde Oluwaseyi Ajayi,
Clemencia Siro,
Steven Arthur
, et al. (27 additional authors not shown)
Abstract:
African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -- those that retrieve answer content from other languages while serving people in their native language -- offer a means of filling this gap. To this end, we create…
▽ More
African languages have far less in-language content available digitally, making it challenging for question answering systems to satisfy the information needs of users. Cross-lingual open-retrieval question answering (XOR QA) systems -- those that retrieve answer content from other languages while serving people in their native language -- offer a means of filling this gap. To this end, we create AfriQA, the first cross-lingual QA dataset with a focus on African languages. AfriQA includes 12,000+ XOR QA examples across 10 African languages. While previous datasets have focused primarily on languages where cross-lingual QA augments coverage from the target language, AfriQA focuses on languages where cross-lingual answer content is the only high-coverage source of answer content. Because of this, we argue that African languages are one of the most important and realistic use cases for XOR QA. Our experiments demonstrate the poor performance of automatic translation and multilingual retrieval methods. Overall, AfriQA proves challenging for state-of-the-art QA models. We hope that the dataset enables the development of more equitable QA technology.
△ Less
Submitted 11 May, 2023;
originally announced May 2023.
-
An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
Authors:
Albert Njoroge Kahira,
Truong Thao Nguyen,
Leonardo Bautista Gomez,
Ryousei Takano,
Rosa M Badia,
Mohamed Wahib
Abstract:
Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communicat…
▽ More
Deep Neural Network (DNN) frameworks use distributed training to enable faster time to convergence and alleviate memory capacity limitations when training large models and/or using high dimension inputs. With the steady increase in datasets and model sizes, model/hybrid parallelism is deemed to have an important role in the future of distributed training of DNNs. We analyze the compute, communication, and memory requirements of Convolutional Neural Networks (CNNs) to understand the trade-offs between different parallelism approaches on performance and scalability. We leverage our model-driven analysis to be the basis for an oracle utility which can help in detecting the limitations and bottlenecks of different parallelism approaches at scale. We evaluate the oracle on six parallelization strategies, with four CNN models and multiple datasets (2D and 3D), on up to 1024 GPUs. The results demonstrate that the oracle has an average accuracy of about 86.74% when compared to empirical results, and as high as 97.57% for data parallelism.
△ Less
Submitted 19 April, 2021;
originally announced April 2021.
-
A Study of Checkpointing in Large Scale Training of Deep Neural Networks
Authors:
Elvis Rojas,
Albert Njoroge Kahira,
Esteban Meneses,
Leonardo Bautista Gomez,
Rosa M Badia
Abstract:
Deep learning (DL) applications are increasingly being deployed on HPC systems, to leverage the massive parallelism and computing power of those systems for DL model training. While significant effort has been put to facilitate distributed training by DL frameworks, fault tolerance has been largely ignored. In this work, we evaluate checkpoint-restart, a common fault tolerance technique in HPC wor…
▽ More
Deep learning (DL) applications are increasingly being deployed on HPC systems, to leverage the massive parallelism and computing power of those systems for DL model training. While significant effort has been put to facilitate distributed training by DL frameworks, fault tolerance has been largely ignored. In this work, we evaluate checkpoint-restart, a common fault tolerance technique in HPC workloads. We perform experiments with three state-of-the-art DL frameworks common in HPC Chainer, PyTorch, and TensorFlow). We evaluate the computational cost of checkpointing, file formats and file sizes, the impact of scale, and deterministic checkpointing. Our evaluation shows some critical differences in checkpoint mechanisms and exposes several bottlenecks in existing checkpointing implementations. We provide discussion points that can aid users in selecting a fault-tolerant framework to use in HPC. We also provide takeaway points that framework developers can use to facilitate better checkpointing of DL workloads in HPC.
△ Less
Submitted 29 March, 2021; v1 submitted 1 December, 2020;
originally announced December 2020.
