-
Fine-tuning of Geospatial Foundation Models for Aboveground Biomass Estimation
Authors:
Michal Muszynski,
Levente Klein,
Ademir Ferreira da Silva,
Anjani Prasad Atluri,
Carlos Gomes,
Daniela Szwarcman,
Gurkanwar Singh,
Kewen Gu,
Maciel Zortea,
Naomi Simumba,
Paolo Fraccaro,
Shraddha Singh,
Steve Meliksetian,
Campbell Watson,
Daiki Kimura,
Harini Srinivasan
Abstract:
Global vegetation structure map** is critical for understanding the global carbon cycle and maximizing the efficacy of nature-based carbon sequestration initiatives. Moreover, vegetation structure map** can help reduce the impacts of climate change by, for example, guiding actions to improve water security, increase biodiversity and reduce flood risk. Global satellite measurements provide an i…
▽ More
Global vegetation structure map** is critical for understanding the global carbon cycle and maximizing the efficacy of nature-based carbon sequestration initiatives. Moreover, vegetation structure map** can help reduce the impacts of climate change by, for example, guiding actions to improve water security, increase biodiversity and reduce flood risk. Global satellite measurements provide an important set of observations for monitoring and managing deforestation and degradation of existing forests, natural forest regeneration, reforestation, biodiversity restoration, and the implementation of sustainable agricultural practices. In this paper, we explore the effectiveness of fine-tuning of a geospatial foundation model to estimate above-ground biomass (AGB) using space-borne data collected across different eco-regions in Brazil. The fine-tuned model architecture consisted of a Swin-B transformer as the encoder (i.e., backbone) and a single convolutional layer for the decoder head. All results were compared to a U-Net which was trained as the baseline model Experimental results of this sparse-label prediction task demonstrate that the fine-tuned geospatial foundation model with a frozen encoder has comparable performance to a U-Net trained from scratch. This is despite the fine-tuned model having 13 times less parameters requiring optimization, which saves both time and compute resources. Further, we explore the transfer-learning capabilities of the geospatial foundation models by fine-tuning on satellite imagery with sparse labels from different eco-regions in Brazil.
△ Less
Submitted 28 June, 2024;
originally announced June 2024.
-
Harm Amplification in Text-to-Image Models
Authors:
Susan Hao,
Renee Shelby,
Yuchi Liu,
Hansa Srinivasan,
Mukul Bhutani,
Burcu Karagol Ayan,
Ryan Poplin,
Shivani Poddar,
Sarah Laszlo
Abstract:
Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input, poses a potentially greater risk than adversarial prompts, leaving…
▽ More
Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input, poses a potentially greater risk than adversarial prompts, leaving users unintentionally exposed to harms. Our paper addresses this issue by formalizing a definition for this phenomenon which we term harm amplification. We further contribute to the field by develo** a framework of methodologies to quantify harm amplification in which we consider the harm of the model output in the context of user input. We then empirically examine how to apply these different methodologies to simulate real-world deployment scenarios including a quantification of disparate impacts across genders resulting from harm amplification. Together, our work aims to offer researchers tools to comprehensively address safety challenges in T2I systems and contribute to the responsible deployment of generative AI models.
△ Less
Submitted 17 May, 2024; v1 submitted 1 February, 2024;
originally announced February 2024.
-
Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images
Authors:
Hansa Srinivasan,
Candice Schumann,
Aradhana Sinha,
David Madras,
Gbolahan Oluwafemi Olanubi,
Alex Beutel,
Susanna Ricco,
Jilin Chen
Abstract:
Capturing the diversity of people in images is challenging: recent literature tends to focus on diversifying one or two attributes, requiring expensive attribute labels or building classifiers. We introduce a diverse people image ranking method which more flexibly aligns with human notions of people diversity in a less prescriptive, label-free manner. The Perception-Aligned Text-derived Human repr…
▽ More
Capturing the diversity of people in images is challenging: recent literature tends to focus on diversifying one or two attributes, requiring expensive attribute labels or building classifiers. We introduce a diverse people image ranking method which more flexibly aligns with human notions of people diversity in a less prescriptive, label-free manner. The Perception-Aligned Text-derived Human representation Space (PATHS) aims to capture all or many relevant features of people-related diversity, and, when used as the representation space in the standard Maximal Marginal Relevance (MMR) ranking algorithm, is better able to surface a range of types of people-related diversity (e.g. disability, cultural attire). PATHS is created in two stages. First, a text-guided approach is used to extract a person-diversity representation from a pre-trained image-text model. Then this representation is fine-tuned on perception judgments from human annotators so that it captures the aspects of people-related similarity that humans find most salient. Empirical results show that the PATHS method achieves diversity better than baseline methods, according to side-by-side ratings from human annotators.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
Gemini: A Family of Highly Capable Multimodal Models
Authors:
Gemini Team,
Rohan Anil,
Sebastian Borgeaud,
Jean-Baptiste Alayrac,
Jiahui Yu,
Radu Soricut,
Johan Schalkwyk,
Andrew M. Dai,
Anja Hauth,
Katie Millican,
David Silver,
Melvin Johnson,
Ioannis Antonoglou,
Julian Schrittwieser,
Amelia Glaese,
Jilin Chen,
Emily Pitler,
Timothy Lillicrap,
Angeliki Lazaridou,
Orhan Firat,
James Molloy,
Michael Isard,
Paul R. Barham,
Tom Hennigan,
Benjamin Lee
, et al. (1325 additional authors not shown)
Abstract:
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultr…
▽ More
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family consists of Ultra, Pro, and Nano sizes, suitable for applications ranging from complex reasoning tasks to on-device memory-constrained use-cases. Evaluation on a broad range of benchmarks shows that our most-capable Gemini Ultra model advances the state of the art in 30 of 32 of these benchmarks - notably being the first model to achieve human-expert performance on the well-studied exam benchmark MMLU, and improving the state of the art in every one of the 20 multimodal benchmarks we examined. We believe that the new capabilities of the Gemini family in cross-modal reasoning and language understanding will enable a wide variety of use cases. We discuss our approach toward post-training and deploying Gemini models responsibly to users through services including Gemini, Gemini Advanced, Google AI Studio, and Cloud Vertex AI.
△ Less
Submitted 17 June, 2024; v1 submitted 18 December, 2023;
originally announced December 2023.
-
Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting
Authors:
Preethi Lahoti,
Nicholas Blumm,
Xiao Ma,
Raghavendra Kotikalapudi,
Sahitya Potluri,
Qijun Tan,
Hansa Srinivasan,
Ben Packer,
Ahmad Beirami,
Alex Beutel,
Jilin Chen
Abstract:
A crucial challenge for generative large language models (LLMs) is diversity: when a user's prompt is under-specified, models may follow implicit assumptions while generating a response, which may result in homogenization of the responses, as well as certain demographic groups being under-represented or even erased from the generated responses. In this paper, we formalize diversity of representati…
▽ More
A crucial challenge for generative large language models (LLMs) is diversity: when a user's prompt is under-specified, models may follow implicit assumptions while generating a response, which may result in homogenization of the responses, as well as certain demographic groups being under-represented or even erased from the generated responses. In this paper, we formalize diversity of representation in generative LLMs. We present evaluation datasets and propose metrics to measure diversity in generated responses along people and culture axes. We find that LLMs understand the notion of diversity, and that they can reason and critique their own responses for that goal. This finding motivated a new prompting technique called collective-critique and self-voting (CCSV) to self-improve people diversity of LLMs by tap** into its diversity reasoning capabilities, without relying on handcrafted examples or prompt tuning. Extensive empirical experiments with both human and automated evaluations show that our proposed approach is effective at improving people and culture diversity, and outperforms all baseline methods by a large margin.
△ Less
Submitted 25 October, 2023;
originally announced October 2023.
-
SOMPS-Net : Attention based social graph framework for early detection of fake health news
Authors:
Prasannakumaran D,
Harish Srinivasan,
Sowmiya Sree S,
Sri Gayathri Devi I,
Saikrishnan S,
Vineeth Vijayaraghavan
Abstract:
Fake news is fabricated information that is presented as genuine, with intention to deceive the reader. Recently, the magnitude of people relying on social media for news consumption has increased significantly. Owing to this rapid increase, the adverse effects of misinformation affect a wider audience. On account of the increased vulnerability of people to such deceptive fake news, a reliable tec…
▽ More
Fake news is fabricated information that is presented as genuine, with intention to deceive the reader. Recently, the magnitude of people relying on social media for news consumption has increased significantly. Owing to this rapid increase, the adverse effects of misinformation affect a wider audience. On account of the increased vulnerability of people to such deceptive fake news, a reliable technique to detect misinformation at its early stages is imperative. Hence, the authors propose a novel graph-based framework SOcial graph with Multi-head attention and Publisher information and news Statistics Network (SOMPS-Net) comprising of two components - Social Interaction Graph (SIG) and Publisher and News Statistics (PNS). The posited model is experimented on the HealthStory dataset and generalizes across diverse medical topics including Cancer, Alzheimer's, Obstetrics, and Nutrition. SOMPS-Net significantly outperformed other state-of-the-art graph-based models experimented on HealthStory by 17.1%. Further, experiments on early detection demonstrated that SOMPS-Net predicted fake news articles with 79% certainty within just 8 hours of its broadcast. Thus the contributions of this work lay down the foundation for capturing fake health news across multiple medical topics at its early stages.
△ Less
Submitted 22 November, 2021;
originally announced November 2021.
-
Fair treatment allocations in social networks
Authors:
James Atwood,
Hansa Srinivasan,
Yoni Halpern,
D Sculley
Abstract:
Simulations of infectious disease spread have long been used to understand how epidemics evolve and how to effectively treat them. However, comparatively little attention has been paid to understanding the fairness implications of different treatment strategies -- that is, how might such strategies distribute the expected disease burden differentially across various subgroups or communities in the…
▽ More
Simulations of infectious disease spread have long been used to understand how epidemics evolve and how to effectively treat them. However, comparatively little attention has been paid to understanding the fairness implications of different treatment strategies -- that is, how might such strategies distribute the expected disease burden differentially across various subgroups or communities in the population? In this work, we define the precision disease control problem -- the problem of optimally allocating vaccines in a social network in a step-by-step fashion -- and we use the ML Fairness Gym to simulate epidemic control and study it from both an efficiency and fairness perspective. We then present an exploratory analysis of several different environments and discuss the fairness implications of different treatment strategies.
△ Less
Submitted 1 November, 2019;
originally announced November 2019.
