-
Explaining Representation Learning with Perceptual Components
Authors:
Yavuz Yarici,
Kiran Kokilepersaud,
Mohit Prabhushankar,
Ghassan AlRegib
Abstract:
Self-supervised models create representation spaces that lack clear semantic meaning. This interpretability problem of representations makes traditional explainability methods ineffective in this context. In this paper, we introduce a novel method to analyze representation spaces using three key perceptual components: color, shape, and texture. We employ selective masking of these components to ob…
▽ More
Self-supervised models create representation spaces that lack clear semantic meaning. This interpretability problem of representations makes traditional explainability methods ineffective in this context. In this paper, we introduce a novel method to analyze representation spaces using three key perceptual components: color, shape, and texture. We employ selective masking of these components to observe changes in representations, resulting in distinct importance maps for each. In scenarios, where labels are absent, these importance maps provide more intuitive explanations as they are integral to the human visual system. Our approach enhances the interpretability of the representation space, offering explanations that resonate with human visual perception. We analyze how different training objectives create distinct representation spaces using perceptual components. Additionally, we examine the representation of images across diverse image domains, providing insights into the role of these components in different contexts.
△ Less
Submitted 11 June, 2024;
originally announced June 2024.
-
Taxes Are All You Need: Integration of Taxonomical Hierarchy Relationships into the Contrastive Loss
Authors:
Kiran Kokilepersaud,
Yavuz Yarici,
Mohit Prabhushankar,
Ghassan AlRegib
Abstract:
In this work, we propose a novel supervised contrastive loss that enables the integration of taxonomic hierarchy information during the representation learning process. A supervised contrastive loss operates by enforcing that images with the same class label (positive samples) project closer to each other than images with differing class labels (negative samples). The advantage of this approach is…
▽ More
In this work, we propose a novel supervised contrastive loss that enables the integration of taxonomic hierarchy information during the representation learning process. A supervised contrastive loss operates by enforcing that images with the same class label (positive samples) project closer to each other than images with differing class labels (negative samples). The advantage of this approach is that it directly penalizes the structure of the representation space itself. This enables greater flexibility with respect to encoding semantic concepts. However, the standard supervised contrastive loss only enforces semantic structure based on the downstream task (i.e. the class label). In reality, the class label is only one level of a \emph{hierarchy of different semantic relationships known as a taxonomy}. For example, the class label is oftentimes the species of an animal, but between different classes there are higher order relationships such as all animals with wings being ``birds". We show that by explicitly accounting for these relationships with a weighting penalty in the contrastive loss we can out-perform the supervised contrastive loss. Additionally, we demonstrate the adaptability of the notion of a taxonomy by integrating our loss into medical and noise-based settings that show performance improvements by as much as 7%.
△ Less
Submitted 10 June, 2024;
originally announced June 2024.
-
FOCAL: A Cost-Aware Video Dataset for Active Learning
Authors:
Kiran Kokilepersaud,
Yash-Yee Logan,
Ryan Benkert,
Chen Zhou,
Mohit Prabhushankar,
Ghassan AlRegib,
Enrique Corona,
Kunjan Singh,
Mostafa Parchami
Abstract:
In this paper, we introduce the FOCAL (Ford-OLIVES Collaboration on Active Learning) dataset which enables the study of the impact of annotation-cost within a video active learning setting. Annotation-cost refers to the time it takes an annotator to label and quality-assure a given video sequence. A practical motivation for active learning research is to minimize annotation-cost by selectively lab…
▽ More
In this paper, we introduce the FOCAL (Ford-OLIVES Collaboration on Active Learning) dataset which enables the study of the impact of annotation-cost within a video active learning setting. Annotation-cost refers to the time it takes an annotator to label and quality-assure a given video sequence. A practical motivation for active learning research is to minimize annotation-cost by selectively labeling informative samples that will maximize performance within a given budget constraint. However, previous work in video active learning lacks real-time annotation labels for accurately assessing cost minimization and instead operates under the assumption that annotation-cost scales linearly with the amount of data to annotate. This assumption does not take into account a variety of real-world confounding factors that contribute to a nonlinear cost such as the effect of an assistive labeling tool and the variety of interactions within a scene such as occluded objects, weather, and motion of objects. FOCAL addresses this discrepancy by providing real annotation-cost labels for 126 video sequences across 69 unique city scenes with a variety of weather, lighting, and seasonal conditions. We also introduce a set of conformal active learning algorithms that take advantage of the sequential structure of video data in order to achieve a better trade-off between annotation-cost and performance while also reducing floating point operations (FLOPS) overhead by at least 77.67%. We show how these approaches better reflect how annotations on videos are done in practice through a sequence selection framework. We further demonstrate the advantage of these approaches by introducing two performance-cost metrics and show that the best conformal active learning method is cheaper than the best traditional active learning method by 113 hours.
△ Less
Submitted 17 November, 2023;
originally announced November 2023.
-
Clinical Trial Active Learning
Authors:
Zoe Fowler,
Kiran Kokilepersaud,
Mohit Prabhushankar,
Ghassan AlRegib
Abstract:
This paper presents a novel approach to active learning that takes into account the non-independent and identically distributed (non-i.i.d.) structure of a clinical trial setting. There exists two types of clinical trials: retrospective and prospective. Retrospective clinical trials analyze data after treatment has been performed; prospective clinical trials collect data as treatment is ongoing. T…
▽ More
This paper presents a novel approach to active learning that takes into account the non-independent and identically distributed (non-i.i.d.) structure of a clinical trial setting. There exists two types of clinical trials: retrospective and prospective. Retrospective clinical trials analyze data after treatment has been performed; prospective clinical trials collect data as treatment is ongoing. Typically, active learning approaches assume the dataset is i.i.d. when selecting training samples; however, in the case of clinical trials, treatment results in a dependency between the data collected at the current and past visits. Thus, we propose prospective active learning to overcome the limitations present in traditional active learning methods and apply it to disease detection in optical coherence tomography (OCT) images, where we condition on the time an image was collected to enforce the i.i.d. assumption. We compare our proposed method to the traditional active learning paradigm, which we refer to as retrospective in nature. We demonstrate that prospective active learning outperforms retrospective active learning in two different types of test settings.
△ Less
Submitted 20 July, 2023;
originally announced July 2023.
-
Clinically Labeled Contrastive Learning for OCT Biomarker Classification
Authors:
Kiran Kokilepersaud,
Stephanie Trejo Corona,
Mohit Prabhushankar,
Ghassan AlRegib,
Charles Wykoff
Abstract:
This paper presents a novel positive and negative set selection strategy for contrastive learning of medical images based on labels that can be extracted from clinical data. In the medical field, there exists a variety of labels for data that serve different purposes at different stages of a diagnostic and treatment process. Clinical labels and biomarker labels are two examples. In general, clinic…
▽ More
This paper presents a novel positive and negative set selection strategy for contrastive learning of medical images based on labels that can be extracted from clinical data. In the medical field, there exists a variety of labels for data that serve different purposes at different stages of a diagnostic and treatment process. Clinical labels and biomarker labels are two examples. In general, clinical labels are easier to obtain in larger quantities because they are regularly collected during routine clinical care, while biomarker labels require expert analysis and interpretation to obtain. Within the field of ophthalmology, previous work has shown that clinical values exhibit correlations with biomarker structures that manifest within optical coherence tomography (OCT) scans. We exploit this relationship by using the clinical data as pseudo-labels for our data without biomarker labels in order to choose positive and negative instances for training a backbone network with a supervised contrastive loss. In this way, a backbone network learns a representation space that aligns with the clinical data distribution available. Afterwards, we fine-tune the network trained in this manner with the smaller amount of biomarker labeled data with a cross-entropy loss in order to classify these key indicators of disease directly from OCT scans. We also expand on this concept by proposing a method that uses a linear combination of clinical contrastive losses. We benchmark our methods against state of the art self-supervised methods in a novel setting with biomarkers of varying granularity. We show performance improvements by as much as 5\% in total biomarker detection AUROC.
△ Less
Submitted 24 May, 2023;
originally announced May 2023.
-
Exploiting the Distortion-Semantic Interaction in Fisheye Data
Authors:
Kiran Kokilepersaud,
Mohit Prabhushankar,
Yavuz Yarici,
Ghassan AlRegib,
Armin Parchami
Abstract:
In this work, we present a methodology to shape a fisheye-specific representation space that reflects the interaction between distortion and semantic context present in this data modality. Fisheye data has the wider field of view advantage over other types of cameras, but this comes at the expense of high radial distortion. As a result, objects further from the center exhibit deformations that mak…
▽ More
In this work, we present a methodology to shape a fisheye-specific representation space that reflects the interaction between distortion and semantic context present in this data modality. Fisheye data has the wider field of view advantage over other types of cameras, but this comes at the expense of high radial distortion. As a result, objects further from the center exhibit deformations that make it difficult for a model to identify their semantic context. While previous work has attempted architectural and training augmentation changes to alleviate this effect, no work has attempted to guide the model towards learning a representation space that reflects this interaction between distortion and semantic context inherent to fisheye data. We introduce an approach to exploit this relationship by first extracting distortion class labels based on an object's distance from the center of the image. We then shape a backbone's representation space with a weighted contrastive loss that constrains objects of the same semantic class and distortion class to be close to each other within a lower dimensional embedding space. This backbone trained with both semantic and distortion information is then fine-tuned within an object detection setting to empirically evaluate the quality of the learnt representation. We show this method leads to performance improvements by as much as 1.1% mean average precision over standard object detection strategies and .6% improvement over other state of the art representation learning approaches.
△ Less
Submitted 6 May, 2023; v1 submitted 28 April, 2023;
originally announced May 2023.
-
Clinical Contrastive Learning for Biomarker Detection
Authors:
Kiran Kokilepersaud,
Mohit Prabhushankar,
Ghassan AlRegib
Abstract:
This paper presents a novel positive and negative set selection strategy for contrastive learning of medical images based on labels that can be extracted from clinical data. In the medical field, there exists a variety of labels for data that serve different purposes at different stages of a diagnostic and treatment process. Clinical labels and biomarker labels are two examples. In general, clinic…
▽ More
This paper presents a novel positive and negative set selection strategy for contrastive learning of medical images based on labels that can be extracted from clinical data. In the medical field, there exists a variety of labels for data that serve different purposes at different stages of a diagnostic and treatment process. Clinical labels and biomarker labels are two examples. In general, clinical labels are easier to obtain in larger quantities because they are regularly collected during routine clinical care, while biomarker labels require expert analysis and interpretation to obtain. Within the field of ophthalmology, previous work has shown that clinical values exhibit correlations with biomarker structures that manifest within optical coherence tomography (OCT) scans. We exploit this relationship between clinical and biomarker data to improve performance for biomarker classification. This is accomplished by leveraging the larger amount of clinical data as pseudo-labels for our data without biomarker labels in order to choose positive and negative instances for training a backbone network with a supervised contrastive loss. In this way, a backbone network learns a representation space that aligns with the clinical data distribution available. Afterwards, we fine-tune the network trained in this manner with the smaller amount of biomarker labeled data with a cross-entropy loss in order to classify these key indicators of disease directly from OCT scans. Our method is shown to outperform state of the art self-supervised methods by as much as 5% in terms of accuracy on individual biomarker detection.
△ Less
Submitted 9 November, 2022;
originally announced November 2022.
-
OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics
Authors:
Mohit Prabhushankar,
Kiran Kokilepersaud,
Yash-yee Logan,
Stephanie Trejo Corona,
Ghassan AlRegib,
Charles Wykoff
Abstract:
Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. Clinical practitioners use all available data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling…
▽ More
Clinical diagnosis of the eye is performed over multifarious data modalities including scalar clinical labels, vectorized biomarkers, two-dimensional fundus images, and three-dimensional Optical Coherence Tomography (OCT) scans. Clinical practitioners use all available data modalities for diagnosing and treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema (DME). Enabling usage of machine learning algorithms within the ophthalmic medical domain requires research into the relationships and interactions between all relevant data over a treatment period. Existing datasets are limited in that they neither provide data nor consider the explicit relationship modeling between the data modalities. In this paper, we introduce the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset that addresses the above limitation. This is the first OCT and near-IR fundus dataset that includes clinical labels, biomarker labels, disease labels, and time-series patient treatment information from associated clinical trials. The dataset consists of 1268 near-IR fundus images each with at least 49 OCT scans, and 16 biomarkers, along with 4 clinical labels and a disease diagnosis of DR or DME. In total, there are 96 eyes' data averaged over a period of at least two years with each eye treated for an average of 66 weeks and 7 injections. We benchmark the utility of OLIVES dataset for ophthalmic data as well as provide benchmarks and concrete research directions for core and emerging machine learning paradigms within medical image analysis.
△ Less
Submitted 22 September, 2022;
originally announced September 2022.
-
Volumetric Supervised Contrastive Learning for Seismic Semantic Segmentation
Authors:
Kiran Kokilepersaud,
Mohit Prabhushankar,
Ghassan AlRegib
Abstract:
In seismic interpretation, pixel-level labels of various rock structures can be time-consuming and expensive to obtain. As a result, there oftentimes exists a non-trivial quantity of unlabeled data that is left unused simply because traditional deep learning methods rely on access to fully labeled volumes. To rectify this problem, contrastive learning approaches have been proposed that use a self-…
▽ More
In seismic interpretation, pixel-level labels of various rock structures can be time-consuming and expensive to obtain. As a result, there oftentimes exists a non-trivial quantity of unlabeled data that is left unused simply because traditional deep learning methods rely on access to fully labeled volumes. To rectify this problem, contrastive learning approaches have been proposed that use a self-supervised methodology in order to learn useful representations from unlabeled data. However, traditional contrastive learning approaches are based on assumptions from the domain of natural images that do not make use of seismic context. In order to incorporate this context within contrastive learning, we propose a novel positive pair selection strategy based on the position of slices within a seismic volume. We show that the learnt representations from our method out-perform a state of the art contrastive learning methodology in a semantic segmentation task.
△ Less
Submitted 16 June, 2022;
originally announced June 2022.
-
Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification
Authors:
Y. Logan,
K. Kokilepersaud,
G. Kwon,
G. AlRegib,
C. Wykoff,
H. Yu
Abstract:
In this paper, we propose a framework that incorporates experts diagnostics and insights into the analysis of Optical Coherence Tomography (OCT) using multi-modal learning. To demonstrate the effectiveness of this approach, we create a medical diagnostic attribute dataset to improve disease classification using OCT. Although there have been successful attempts to deploy machine learning for diseas…
▽ More
In this paper, we propose a framework that incorporates experts diagnostics and insights into the analysis of Optical Coherence Tomography (OCT) using multi-modal learning. To demonstrate the effectiveness of this approach, we create a medical diagnostic attribute dataset to improve disease classification using OCT. Although there have been successful attempts to deploy machine learning for disease classification in OCT, such methodologies lack the experts insights. We argue that injecting ophthalmological assessments as another supervision in a learning framework is of great importance for the machine learning process to perform accurate and interpretable classification. We demonstrate the proposed framework through comprehensive experiments that compare the effectiveness of combining diagnostic attribute features with latent visual representations and show that they surpass the state-of-the-art approach. Finally, we analyze the proposed dual-stream architecture and provide an insight that determine the components that contribute most to classification performance.
△ Less
Submitted 20 March, 2022;
originally announced March 2022.
