No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
Authors:
Angéline Pouget,
Lucas Beyer,
Emanuele Bugliarello,
Xiao Wang,
Andreas Peter Steiner,
Xiaohua Zhai,
Ibrahim Alabdulmohsin
Abstract:
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this pe…
▽ More
We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for develo** VLMs that better represent global perspectives.
△ Less
Submitted 24 May, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
PaLI-X: On Scaling up a Multilingual Vision and Language Model
Authors:
Xi Chen,
Josip Djolonga,
Piotr Padlewski,
Basil Mustafa,
Soravit Changpinyo,
Jialin Wu,
Carlos Riquelme Ruiz,
Sebastian Goodman,
Xiao Wang,
Yi Tay,
Siamak Shakeri,
Mostafa Dehghani,
Daniel Salz,
Mario Lucic,
Michael Tschannen,
Arsha Nagrani,
Hexiang Hu,
Mandar Joshi,
Bo Pang,
Ceslee Montgomery,
Paulina Pietrzyk,
Marvin Ritter,
AJ Piergiovanni,
Matthias Minderer,
Filip Pavetic
, et al. (18 additional authors not shown)
Abstract:
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-sh…
▽ More
We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
