-
LucidPPN: Unambiguous Prototypical Parts Network for User-centric Interpretable Computer Vision
Authors:
Mateusz Pach,
Dawid Rymarczyk,
Koryna Lewandowska,
Jacek Tabor,
Bartosz Zieliński
Abstract:
Prototypical parts networks combine the power of deep learning with the explainability of case-based reasoning to make accurate, interpretable decisions. They follow the this looks like that reasoning, representing each prototypical part with patches from training images. However, a single image patch comprises multiple visual features, such as color, shape, and texture, making it difficult for us…
▽ More
Prototypical parts networks combine the power of deep learning with the explainability of case-based reasoning to make accurate, interpretable decisions. They follow the this looks like that reasoning, representing each prototypical part with patches from training images. However, a single image patch comprises multiple visual features, such as color, shape, and texture, making it difficult for users to identify which feature is important to the model.
To reduce this ambiguity, we introduce the Lucid Prototypical Parts Network (LucidPPN), a novel prototypical parts network that separates color prototypes from other visual features. Our method employs two reasoning branches: one for non-color visual features, processing grayscale images, and another focusing solely on color information. This separation allows us to clarify whether the model's decisions are based on color, shape, or texture. Additionally, LucidPPN identifies prototypical parts corresponding to semantic parts of classified objects, making comparisons between data classes more intuitive, e.g., when two bird species might differ primarily in belly color.
Our experiments demonstrate that the two branches are complementary and together achieve results comparable to baseline methods. More importantly, LucidPPN generates less ambiguous prototypical parts, enhancing user understanding.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Token Recycling for Efficient Sequential Inference with Vision Transformers
Authors:
Jan Olszewski,
Dawid Rymarczyk,
Piotr Wójcik,
Mateusz Pach,
Bartosz Zieliński
Abstract:
Vision Transformers (ViTs) overpass Convolutional Neural Networks in processing incomplete inputs because they do not require the imputation of missing values. Therefore, ViTs are well suited for sequential decision-making, e.g. in the Active Visual Exploration problem. However, they are computationally inefficient because they perform a full forward pass each time a piece of new sequential inform…
▽ More
Vision Transformers (ViTs) overpass Convolutional Neural Networks in processing incomplete inputs because they do not require the imputation of missing values. Therefore, ViTs are well suited for sequential decision-making, e.g. in the Active Visual Exploration problem. However, they are computationally inefficient because they perform a full forward pass each time a piece of new sequential information arrives.
To reduce this computational inefficiency, we introduce the TOken REcycling (TORE) modification for the ViT inference, which can be used with any architecture. TORE divides ViT into two parts, iterator and aggregator. An iterator processes sequential information separately into midway tokens, which are cached. The aggregator processes midway tokens jointly to obtain the prediction. This way, we can reuse the results of computations made by iterator.
Except for efficient sequential inference, we propose a complementary training policy, which significantly reduces the computational burden associated with sequential decision-making while achieving state-of-the-art accuracy.
△ Less
Submitted 26 November, 2023;
originally announced November 2023.
-
Interpretability Benchmark for Evaluating Spatial Misalignment of Prototypical Parts Explanations
Authors:
Mikołaj Sacha,
Bartosz Jura,
Dawid Rymarczyk,
Łukasz Struski,
Jacek Tabor,
Bartosz Zieliński
Abstract:
Prototypical parts-based networks are becoming increasingly popular due to their faithful self-explanations. However, their similarity maps are calculated in the penultimate network layer. Therefore, the receptive field of the prototype activation region often depends on parts of the image outside this region, which can lead to misleading interpretations. We name this undesired behavior a spatial…
▽ More
Prototypical parts-based networks are becoming increasingly popular due to their faithful self-explanations. However, their similarity maps are calculated in the penultimate network layer. Therefore, the receptive field of the prototype activation region often depends on parts of the image outside this region, which can lead to misleading interpretations. We name this undesired behavior a spatial explanation misalignment and introduce an interpretability benchmark with a set of dedicated metrics for quantifying this phenomenon. In addition, we propose a method for misalignment compensation and apply it to existing state-of-the-art models. We show the expressiveness of our benchmark and the effectiveness of the proposed compensation methodology through extensive empirical studies.
△ Less
Submitted 16 August, 2023;
originally announced August 2023.
-
ProMIL: Probabilistic Multiple Instance Learning for Medical Imaging
Authors:
Łukasz Struski,
Dawid Rymarczyk,
Arkadiusz Lewicki,
Robert Sabiniewicz,
Jacek Tabor,
Bartosz Zieliński
Abstract:
Multiple Instance Learning (MIL) is a weakly-supervised problem in which one label is assigned to the whole bag of instances. An important class of MIL models is instance-based, where we first classify instances and then aggregate those predictions to obtain a bag label. The most common MIL model is when we consider a bag as positive if at least one of its instances has a positive label. However,…
▽ More
Multiple Instance Learning (MIL) is a weakly-supervised problem in which one label is assigned to the whole bag of instances. An important class of MIL models is instance-based, where we first classify instances and then aggregate those predictions to obtain a bag label. The most common MIL model is when we consider a bag as positive if at least one of its instances has a positive label. However, this reasoning does not hold in many real-life scenarios, where the positive bag label is often a consequence of a certain percentage of positive instances. To address this issue, we introduce a dedicated instance-based method called ProMIL, based on deep neural networks and Bernstein polynomial estimation. An important advantage of ProMIL is that it can automatically detect the optimal percentage level for decision-making. We show that ProMIL outperforms standard instance-based MIL in real-world medical applications. We make the code available.
△ Less
Submitted 12 March, 2024; v1 submitted 18 June, 2023;
originally announced June 2023.
-
ICICLE: Interpretable Class Incremental Continual Learning
Authors:
Dawid Rymarczyk,
Joost van de Weijer,
Bartosz Zieliński,
Bartłomiej Twardowski
Abstract:
Continual learning enables incremental learning of new tasks without forgetting those previously learned, resulting in positive knowledge transfer that can enhance performance on both new and old tasks. However, continual learning poses new challenges for interpretability, as the rationale behind model predictions may change over time, leading to interpretability concept drift. We address this pro…
▽ More
Continual learning enables incremental learning of new tasks without forgetting those previously learned, resulting in positive knowledge transfer that can enhance performance on both new and old tasks. However, continual learning poses new challenges for interpretability, as the rationale behind model predictions may change over time, leading to interpretability concept drift. We address this problem by proposing Interpretable Class-InCremental LEarning (ICICLE), an exemplar-free approach that adopts a prototypical part-based approach. It consists of three crucial novelties: interpretability regularization that distills previously learned concepts while preserving user-friendly positive reasoning; proximity-based prototype initialization strategy dedicated to the fine-grained setting; and task-recency bias compensation devoted to prototypical parts. Our experimental results demonstrate that ICICLE reduces the interpretability concept drift and outperforms the existing exemplar-free methods of common class-incremental learning when applied to concept-based models.
△ Less
Submitted 31 July, 2023; v1 submitted 14 March, 2023;
originally announced March 2023.
-
ProtoSeg: Interpretable Semantic Segmentation with Prototypical Parts
Authors:
Mikołaj Sacha,
Dawid Rymarczyk,
Łukasz Struski,
Jacek Tabor,
Bartosz Zieliński
Abstract:
We introduce ProtoSeg, a novel model for interpretable semantic image segmentation, which constructs its predictions using similar patches from the training set. To achieve accuracy comparable to baseline methods, we adapt the mechanism of prototypical parts and introduce a diversity loss function that increases the variety of prototypes within each class. We show that ProtoSeg discovers semantic…
▽ More
We introduce ProtoSeg, a novel model for interpretable semantic image segmentation, which constructs its predictions using similar patches from the training set. To achieve accuracy comparable to baseline methods, we adapt the mechanism of prototypical parts and introduce a diversity loss function that increases the variety of prototypes within each class. We show that ProtoSeg discovers semantic concepts, in contrast to standard segmentation models. Experiments conducted on Pascal VOC and Cityscapes datasets confirm the precision and transparency of the presented method.
△ Less
Submitted 28 January, 2023;
originally announced January 2023.
-
ProGReST: Prototypical Graph Regression Soft Trees for Molecular Property Prediction
Authors:
Dawid Rymarczyk,
Daniel Dobrowolski,
Tomasz Danel
Abstract:
In this work, we propose the novel Prototypical Graph Regression Self-explainable Trees (ProGReST) model, which combines prototype learning, soft decision trees, and Graph Neural Networks. In contrast to other works, our model can be used to address various challenging tasks, including compound property prediction. In ProGReST, the rationale is obtained along with prediction due to the model's bui…
▽ More
In this work, we propose the novel Prototypical Graph Regression Self-explainable Trees (ProGReST) model, which combines prototype learning, soft decision trees, and Graph Neural Networks. In contrast to other works, our model can be used to address various challenging tasks, including compound property prediction. In ProGReST, the rationale is obtained along with prediction due to the model's built-in interpretability. Additionally, we introduce a new graph prototype projection to accelerate model training. Finally, we evaluate PRoGReST on a wide range of chemical datasets for molecular property prediction and perform in-depth analysis with chemical experts to evaluate obtained interpretations. Our method achieves competitive results against state-of-the-art methods.
△ Less
Submitted 27 December, 2022; v1 submitted 7 October, 2022;
originally announced October 2022.
-
Interpretable Image Classification with Differentiable Prototypes Assignment
Authors:
Dawid Rymarczyk,
Łukasz Struski,
Michał Górszczak,
Koryna Lewandowska,
Jacek Tabor,
Bartosz Zieliński
Abstract:
We introduce ProtoPool, an interpretable image classification model with a pool of prototypes shared by the classes. The training is more straightforward than in the existing methods because it does not require the pruning stage. It is obtained by introducing a fully differentiable assignment of prototypes to particular classes. Moreover, we introduce a novel focal similarity function to focus the…
▽ More
We introduce ProtoPool, an interpretable image classification model with a pool of prototypes shared by the classes. The training is more straightforward than in the existing methods because it does not require the pruning stage. It is obtained by introducing a fully differentiable assignment of prototypes to particular classes. Moreover, we introduce a novel focal similarity function to focus the model on the rare foreground features. We show that ProtoPool obtains state-of-the-art accuracy on the CUB-200-2011 and the Stanford Cars datasets, substantially reducing the number of prototypes. We provide a theoretical analysis of the method and a user study to show that our prototypes are more distinctive than those obtained with competitive methods.
△ Less
Submitted 5 September, 2022; v1 submitted 6 December, 2021;
originally announced December 2021.
-
ProtoMIL: Multiple Instance Learning with Prototypical Parts for Whole-Slide Image Classification
Authors:
Dawid Rymarczyk,
Adam Pardyl,
Jarosław Kraus,
Aneta Kaczyńska,
Marek Skomorowski,
Bartosz Zieliński
Abstract:
Multiple Instance Learning (MIL) gains popularity in many real-life machine learning applications due to its weakly supervised nature. However, the corresponding effort on explaining MIL lags behind, and it is usually limited to presenting instances of a bag that are crucial for a particular prediction. In this paper, we fill this gap by introducing ProtoMIL, a novel self-explainable MIL method in…
▽ More
Multiple Instance Learning (MIL) gains popularity in many real-life machine learning applications due to its weakly supervised nature. However, the corresponding effort on explaining MIL lags behind, and it is usually limited to presenting instances of a bag that are crucial for a particular prediction. In this paper, we fill this gap by introducing ProtoMIL, a novel self-explainable MIL method inspired by the case-based reasoning process that operates on visual prototypes. Thanks to incorporating prototypical features into objects description, ProtoMIL unprecedentedly joins the model accuracy and fine-grained interpretability, which we present with the experiments on five recognized MIL datasets.
△ Less
Submitted 6 September, 2022; v1 submitted 24 August, 2021;
originally announced August 2021.
-
Classifying bacteria clones using attention-based deep multiple instance learning interpreted by persistence homology
Authors:
Adriana Borowa,
Dawid Rymarczyk,
Dorota Ochońska,
Monika Brzychczy-Włoch,
Bartosz Zieliński
Abstract:
In this work, we analyze if it is possible to distinguish between different clones of the same bacteria species (Klebsiella pneumoniae) based only on microscopic images. It is a challenging task, previously considered impossible due to the high clones similarity. For this purpose, we apply a multi-step algorithm with attention-based multiple instance learning. Except for obtaining accuracy at the…
▽ More
In this work, we analyze if it is possible to distinguish between different clones of the same bacteria species (Klebsiella pneumoniae) based only on microscopic images. It is a challenging task, previously considered impossible due to the high clones similarity. For this purpose, we apply a multi-step algorithm with attention-based multiple instance learning. Except for obtaining accuracy at the level of 0.9, we introduce extensive interpretability based on CellProfiler and persistence homology, increasing the understandability and trust in the model.
△ Less
Submitted 23 July, 2021; v1 submitted 2 December, 2020;
originally announced December 2020.
-
ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
Authors:
Dawid Rymarczyk,
Łukasz Struski,
Jacek Tabor,
Bartosz Zieliński
Abstract:
In this paper, we introduce ProtoPShare, a self-explained method that incorporates the paradigm of prototypical parts to explain its predictions. The main novelty of the ProtoPShare is its ability to efficiently share prototypical parts between the classes thanks to our data-dependent merge-pruning. Moreover, the prototypes are more consistent and the model is more robust to image perturbations th…
▽ More
In this paper, we introduce ProtoPShare, a self-explained method that incorporates the paradigm of prototypical parts to explain its predictions. The main novelty of the ProtoPShare is its ability to efficiently share prototypical parts between the classes thanks to our data-dependent merge-pruning. Moreover, the prototypes are more consistent and the model is more robust to image perturbations than the state of the art method ProtoPNet. We verify our findings on two datasets, the CUB-200-2011 and the Stanford Cars.
△ Less
Submitted 29 November, 2020;
originally announced November 2020.
-
Kernel Self-Attention in Deep Multiple Instance Learning
Authors:
Dawid Rymarczyk,
Adriana Borowa,
Jacek Tabor,
Bartosz Zieliński
Abstract:
Not all supervised learning problems are described by a pair of a fixed-size input tensor and a label. In some cases, especially in medical image analysis, a label corresponds to a bag of instances (e.g. image patches), and to classify such bag, aggregation of information from all of the instances is needed. There have been several attempts to create a model working with a bag of instances, howeve…
▽ More
Not all supervised learning problems are described by a pair of a fixed-size input tensor and a label. In some cases, especially in medical image analysis, a label corresponds to a bag of instances (e.g. image patches), and to classify such bag, aggregation of information from all of the instances is needed. There have been several attempts to create a model working with a bag of instances, however, they are assuming that there are no dependencies within the bag and the label is connected to at least one instance. In this work, we introduce Self-Attention Attention-based MIL Pooling (SA-AbMILP) aggregation operation to account for the dependencies between instances. We conduct several experiments on MNIST, histological, microbiological, and retinal databases to show that SA-AbMILP performs better than other models. Additionally, we investigate kernel variations of Self-Attention and their influence on the results.
△ Less
Submitted 5 March, 2021; v1 submitted 25 May, 2020;
originally announced May 2020.
-
Deep learning approach to describe and classify fungi microscopic images
Authors:
Bartosz Zieliński,
Agnieszka Sroka-Oleksiak,
Dawid Rymarczyk,
Adam Piekarczyk,
Monika Brzychczy-Włoch
Abstract:
Preliminary diagnosis of fungal infections can rely on microscopic examination. However, in many cases, it does not allow unambiguous identification of the species by microbiologist due to their visual similarity. Therefore, it is usually necessary to use additional biochemical tests. That involves additional costs and extends the identification process up to 10 days. Such a delay in the implement…
▽ More
Preliminary diagnosis of fungal infections can rely on microscopic examination. However, in many cases, it does not allow unambiguous identification of the species by microbiologist due to their visual similarity. Therefore, it is usually necessary to use additional biochemical tests. That involves additional costs and extends the identification process up to 10 days. Such a delay in the implementation of targeted therapy may be grave in consequence as the mortality rate for immunosuppressed patients is high. In this paper, we apply a machine learning approach based on deep neural networks and Fisher Vector (advanced bag-of-words method) to classify microscopic images of various fungi species. Our approach has the potential to make the last stage of biochemical identification redundant, shortening the identification process by 2-3 days, and reducing the cost of the diagnosis.
△ Less
Submitted 24 May, 2020;
originally announced May 2020.
-
Deep learning approach to description and classification of fungi microscopic images
Authors:
Bartosz Zieliński,
Agnieszka Sroka-Oleksiak,
Dawid Rymarczyk,
Adam Piekarczyk,
Monika Brzychczy-Włoch
Abstract:
Diagnosis of fungal infections can rely on microscopic examination, however, in many cases, it does not allow unambiguous identification of the species due to their visual similarity. Therefore, it is usually necessary to use additional biochemical tests. That involves additional costs and extends the identification process up to 10 days. Such a delay in the implementation of targeted treatment is…
▽ More
Diagnosis of fungal infections can rely on microscopic examination, however, in many cases, it does not allow unambiguous identification of the species due to their visual similarity. Therefore, it is usually necessary to use additional biochemical tests. That involves additional costs and extends the identification process up to 10 days. Such a delay in the implementation of targeted treatment is grave in consequences as the mortality rate for immunosuppressed patients is high. In this paper, we apply machine learning approach based on deep learning and bag-of-words to classify microscopic images of various fungi species. Our approach makes the last stage of biochemical identification redundant, shortening the identification process by 2-3 days and reducing the cost of the diagnostic examination.
△ Less
Submitted 28 January, 2020; v1 submitted 22 June, 2019;
originally announced June 2019.
