Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
Authors:
Neha Sengupta,
Sunil Kumar Sahu,
Bokang Jia,
Satheesh Katipomu,
Haonan Li,
Fajri Koto,
William Marshall,
Gurpreet Gosal,
Cynthia Liu,
Zhiming Chen,
Osama Mohammed Afzal,
Samta Kamboj,
Onkar Pandit,
Rahul Pal,
Lalit Pradhan,
Zain Muhammad Mujahid,
Massa Baali,
Xudong Han,
Sondos Mahmoud Bsharat,
Alham Fikri Aji,
Zhiqiang Shen,
Zhengzhong Liu,
Natalia Vassilieva,
Joel Hestness,
Andy Hock
, et al. (7 additional authors not shown)
Abstract:
We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code in various programming languages. With 13 billion parameters, they demonstrate better knowledge and reasoning…
▽ More
We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric foundation and instruction-tuned open generative large language models (LLMs). The models are based on the GPT-3 decoder-only architecture and are pretrained on a mixture of Arabic and English texts, including source code in various programming languages. With 13 billion parameters, they demonstrate better knowledge and reasoning capabilities in Arabic than any existing open Arabic and multilingual models by a sizable margin, based on extensive evaluation. Moreover, the models are competitive in English compared to English-centric open models of similar size, despite being trained on much less English data. We provide a detailed description of the training, the tuning, the safety alignment, and the evaluation of the models. We release two open versions of the model -- the foundation Jais model, and an instruction-tuned Jais-chat variant -- with the aim of promoting research on Arabic LLMs. Available at https://huggingface.co/inception-mbzuai/jais-13b-chat
△ Less
Submitted 29 September, 2023; v1 submitted 30 August, 2023;
originally announced August 2023.
Probing for Bridging Inference in Transformer Language Models
Authors:
Onkar Pandit,
Yufang Hou
Abstract:
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in…
▽ More
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to language models play an important role in the inference.
△ Less
Submitted 19 April, 2021;
originally announced April 2021.
