-
Evaluating Very Long-Term Conversational Memory of LLM Agents
Authors:
Adyasha Maharana,
Dong-Ho Lee,
Sergey Tulyakov,
Mohit Bansal,
Francesco Barbieri,
Yuwei Fang
Abstract:
Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to gen…
▽ More
Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Debiasing Multimodal Models via Causal Information Minimization
Authors:
Vaidehi Patil,
Adyasha Maharana,
Mohit Bansal
Abstract:
Most existing debiasing methods for multimodal models, including causal intervention and inference methods, utilize approximate heuristics to represent the biases, such as shallow features from early stages of training or unimodal features for multimodal tasks like VQA, etc., which may not be accurate. In this paper, we study bias arising from confounders in a causal graph for multimodal data and…
▽ More
Most existing debiasing methods for multimodal models, including causal intervention and inference methods, utilize approximate heuristics to represent the biases, such as shallow features from early stages of training or unimodal features for multimodal tasks like VQA, etc., which may not be accurate. In this paper, we study bias arising from confounders in a causal graph for multimodal data and examine a novel approach that leverages causally-motivated information minimization to learn the confounder representations. Robust predictive features contain diverse information that helps a model generalize to out-of-distribution data. Hence, minimizing the information content of features obtained from a pretrained biased model helps learn the simplest predictive features that capture the underlying data distribution. We treat these features as confounder representations and use them via methods motivated by causal theory to remove bias from models. We find that the learned confounder representations indeed capture dataset biases, and the proposed debiasing methods improve out-of-distribution (OOD) performance on multiple multimodal datasets without sacrificing in-distribution performance. Additionally, we introduce a novel metric to quantify the sufficiency of spurious features in models' predictions that further demonstrates the effectiveness of our proposed methods. Our code is available at: https://github.com/Vaidehi99/CausalInfoMin
△ Less
Submitted 28 November, 2023;
originally announced November 2023.
-
D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning
Authors:
Adyasha Maharana,
Prateek Yadav,
Mohit Bansal
Abstract:
Analytical theories suggest that higher-quality data can lead to lower test errors in models trained on a fixed data budget. Moreover, a model can be trained on a lower compute budget without compromising performance if a dataset can be stripped of its redundancies. Coreset selection (or data pruning) seeks to select a subset of the training data so as to maximize the performance of models trained…
▽ More
Analytical theories suggest that higher-quality data can lead to lower test errors in models trained on a fixed data budget. Moreover, a model can be trained on a lower compute budget without compromising performance if a dataset can be stripped of its redundancies. Coreset selection (or data pruning) seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. There are two dominant approaches: (1) geometry-based data selection for maximizing data diversity in the coreset, and (2) functions that assign difficulty scores to samples based on training dynamics. Optimizing for data diversity leads to a coreset that is biased towards easier samples, whereas, selection by difficulty ranking omits easy samples that are necessary for the training of deep learning models. This demonstrates that data diversity and importance scores are two complementary factors that need to be jointly considered during coreset selection. We represent a dataset as an undirected graph and propose a novel pruning algorithm, D2 Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection. D2 Pruning updates the difficulty scores of each example by incorporating the difficulty of its neighboring examples in the dataset graph. Then, these updated difficulty scores direct a graph-based sampling method to select a coreset that encapsulates both diverse and difficult regions of the dataset space. We evaluate supervised and self-supervised versions of our method on various vision and language datasets. Results show that D2 Pruning improves coreset selection over previous state-of-the-art methods for up to 70% pruning rates. Additionally, we find that using D2 Pruning for filtering large multimodal datasets leads to increased diversity in the dataset and improved generalization of pretrained models.
△ Less
Submitted 11 October, 2023;
originally announced October 2023.
-
Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models
Authors:
Adyasha Maharana,
Amita Kamath,
Christopher Clark,
Mohit Bansal,
Aniruddha Kembhavi
Abstract:
As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support. Inconsistent AI models are considered brittle and untrustworthy by human users and are more challenging to incorporate into larger systems that take dependencies on their outputs. Measuring consistency between very heterogeneous tasks that migh…
▽ More
As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support. Inconsistent AI models are considered brittle and untrustworthy by human users and are more challenging to incorporate into larger systems that take dependencies on their outputs. Measuring consistency between very heterogeneous tasks that might include outputs in different modalities is challenging since it is difficult to determine if the predictions are consistent with one another. As a solution, we introduce a benchmark dataset, CocoCon, where we create contrast sets by modifying test instances for multiple tasks in small but semantically meaningful ways to change the gold label and outline metrics for measuring if a model is consistent by ranking the original and perturbed instances across tasks. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks, especially for more heterogeneous tasks. To alleviate this issue, we propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets, that improves the multi-task consistency of large unified models while retaining their original accuracy on downstream tasks.
△ Less
Submitted 21 February, 2024; v1 submitted 28 March, 2023;
originally announced March 2023.
-
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation
Authors:
Adyasha Maharana,
Darryl Hannan,
Mohit Bansal
Abstract:
Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story vi…
▽ More
Recent advances in text-to-image synthesis have led to large pretrained transformers with excellent capabilities to generate visualizations from a given text. However, these models are ill-suited for specialized tasks like story visualization, which requires an agent to produce a sequence of images given a corresponding sequence of captions, forming a narrative. Moreover, we find that the story visualization task fails to accommodate generalization to unseen plots and characters in new narratives. Hence, we first propose the task of story continuation, where the generated visual story is conditioned on a source image, allowing for better generalization to narratives with new characters. Then, we enhance or 'retro-fit' the pretrained text-to-image synthesis models with task-specific modules for (a) sequential image generation and (b) copying relevant elements from an initial frame. Then, we explore full-model finetuning, as well as prompt-based tuning for parameter-efficient adaptation, of the pre-trained model. We evaluate our approach StoryDALL-E on two existing datasets, PororoSV and FlintstonesSV, and introduce a new dataset DiDeMoSV collected from a video-captioning dataset. We also develop a model StoryGANc based on Generative Adversarial Networks (GAN) for story continuation, and compare it with the StoryDALL-E model to demonstrate the advantages of our approach. We show that our retro-fitting approach outperforms GAN-based models for story continuation and facilitates copying of visual elements from the source image, thereby improving continuity in the generated visual story. Finally, our analysis suggests that pretrained transformers struggle to comprehend narratives containing several characters. Overall, our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation.
△ Less
Submitted 13 September, 2022;
originally announced September 2022.
-
Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization
Authors:
Adyasha Maharana,
Mohit Bansal
Abstract:
While much research has been done in text-to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story). Prior work in this domain has shown that there is ample room…
▽ More
While much research has been done in text-to-image synthesis, little work has been done to explore the usage of linguistic structure of the input text. Such information is even more important for story visualization since its inputs have an explicit narrative structure that needs to be translated into an image sequence (or visual story). Prior work in this domain has shown that there is ample room for improvement in the generated image sequence in terms of visual quality, consistency and relevance. In this paper, we first explore the use of constituency parse trees using a Transformer-based recurrent architecture for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we also incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images within a dual learning setup. We show that off-the-shelf dense-captioning models trained on Visual Genome can improve the spatial structure of images from a different target domain without needing fine-tuning. We train the model end-to-end using intra-story contrastive loss (between words and image sub-regions) and show significant improvements in several metrics (and human evaluation) for multiple datasets. Finally, we provide an analysis of the linguistic and visuo-spatial information. Code and data: https://github.com/adymaharana/VLCStoryGan.
△ Less
Submitted 20 October, 2021;
originally announced October 2021.
-
Improving Generation and Evaluation of Visual Stories via Semantic Consistency
Authors:
Adyasha Maharana,
Darryl Hannan,
Mohit Bansal
Abstract:
Story visualization is an under-explored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform…
▽ More
Story visualization is an under-explored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform text-to-image synthesis models on this task. However, there is room for improvement of generated images in terms of visual quality, coherence and relevance. We present a number of improvements to prior modeling approaches, including (1) the addition of a dual learning framework that utilizes video captioning to reinforce the semantic alignment between the story and generated images, (2) a copy-transform mechanism for sequentially-consistent story visualization, and (3) MART-based transformers to model complex interactions between frames. We present ablation studies to demonstrate the effect of each of these techniques on the generative power of the model for both individual images as well as the entire narrative. Furthermore, due to the complexity and generative nature of the task, standard evaluation metrics do not accurately reflect performance. Therefore, we also provide an exploration of evaluation metrics for the model, focused on aspects of the generated frames such as the presence/quality of generated characters, the relevance to captions, and the diversity of the generated images. We also present correlation experiments of our proposed automated metrics with human evaluations. Code and data available at: https://github.com/adymaharana/StoryViz
△ Less
Submitted 20 May, 2021;
originally announced May 2021.
-
Use of Technology and Innovations in the COVID-19 Pandemic Response in Africa
Authors:
Adyasha Maharana,
Morine Amutorine,
Moinina David Sengeh,
Elaine O. Nsoesie
Abstract:
The use of technology has been ubiquitous in efforts to combat the ongoing public health crisis due to emergence and spread of the SARS-CoV-2 virus. African countries have made tremendous use of technology to disseminate information, counter the spread of COVID-19, and develop cutting-edge techniques to help with diagnosis, treatment and management of patients. The nature and outcomes of these eff…
▽ More
The use of technology has been ubiquitous in efforts to combat the ongoing public health crisis due to emergence and spread of the SARS-CoV-2 virus. African countries have made tremendous use of technology to disseminate information, counter the spread of COVID-19, and develop cutting-edge techniques to help with diagnosis, treatment and management of patients. The nature and outcomes of these efforts sometimes differ in Africa compared to other areas of the world due to its unique challenges and opportunities. Several countries have developed innovative technology-driven solutions to cater to a diverse population with varying access to technology. Much of the efforts are also earmarked by a flexible approach to problem solving, local tech entrepreneurship, and swift adoption of cutting-edge technology.
△ Less
Submitted 11 December, 2020;
originally announced December 2020.
-
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension
Authors:
Adyasha Maharana,
Mohit Bansal
Abstract:
Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation. Training with adversarially augmented dataset improves robustness against those adversarial attacks but hurts generalization of the models. In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehe…
▽ More
Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation. Training with adversarially augmented dataset improves robustness against those adversarial attacks but hurts generalization of the models. In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation, but also improving generalization to the source domain as well as new domains and languages. We first propose three new methods for generating QA adversaries, that introduce multiple points of confusion within the context, show dependence on insertion location of the distractor, and reveal the compounding effect of mixing adversarial strategies with syntactic and semantic paraphrasing methods. Next, we find that augmenting the training datasets with uniformly sampled adversaries improves robustness to the adversarial attacks but leads to decline in performance on the original unaugmented dataset. We address this issue via RL and more efficient Bayesian policy search methods for automatically learning the best augmentation policy combinations of the transformation probability for each adversary in a large search space. Using these learned policies, we show that adversarial training can lead to significant improvements in in-domain, out-of-domain, and cross-lingual (German, Russian, Turkish) generalization.
△ Less
Submitted 17 November, 2020; v1 submitted 13 April, 2020;
originally announced April 2020.
-
Using Deep Learning to Examine the Association between the Built Environment and Neighborhood Adult Obesity Prevalence
Authors:
Adyasha Maharana,
Elaine O. Nsoesie
Abstract:
More than one-third of the adult population in the United States is obese. Obesity has been linked to factors such as, genetics, diet, physical activity and the environment. However, evidence indicating associations between the built environment and obesity has varied across studies and geographical contexts. Here, we used deep learning and approximately 150,000 high resolution satellite images to…
▽ More
More than one-third of the adult population in the United States is obese. Obesity has been linked to factors such as, genetics, diet, physical activity and the environment. However, evidence indicating associations between the built environment and obesity has varied across studies and geographical contexts. Here, we used deep learning and approximately 150,000 high resolution satellite images to extract features of the built environment. We then developed linear regression models to consistently quantify the association between the extracted features and obesity prevalence at the census tract level for six cities in the United States. The extracted features of the built environment explained 72% to 90% of the variation in obesity prevalence across cities. Outof-sample predictions were considerably high with correlations greater than 80% between predicted and true obesity prevalence across all census tracts. This study supports a strong association between the built environment and obesity prevalence. Additionally, it also illustrates that features of the built environment extracted from satellite images can be useful for studying health indicators, such as obesity. Understanding the association between specific features of the built environment and obesity prevalence can lead to structural changes that could encourage physical activity and decreases in obesity prevalence.
△ Less
Submitted 2 November, 2017;
originally announced November 2017.
-
Using Deep Learning and Satellite Imagery to Quantify the Impact of the Built Environment on Neighborhood Crime Rates
Authors:
Adyasha Maharana,
Quynh C. Nguyen,
Elaine O. Nsoesie
Abstract:
The built environment has been postulated to have an impact on neighborhood crime rates, however, measures of the built environment can be subjective and differ across studies leading to varying observations on its association with crime rates. Here, we illustrate an accurate and straightforward approach to quantify the impact of the built environment on neighborhood crime rates from high-resoluti…
▽ More
The built environment has been postulated to have an impact on neighborhood crime rates, however, measures of the built environment can be subjective and differ across studies leading to varying observations on its association with crime rates. Here, we illustrate an accurate and straightforward approach to quantify the impact of the built environment on neighborhood crime rates from high-resolution satellite imagery. Using geo-referenced crime reports and satellite images for three United States cities, we demonstrate how image features consistently identified using a convolutional neural network can explain up to 82% of the variation in neighborhood crime rates. Our results suggest the built environment is a strong predictor of crime rates, and this can lead to structural interventions shown to reduce crime incidence in urban settings.
△ Less
Submitted 15 October, 2017;
originally announced October 2017.
