Cross-Modal Information Maximization for Medical Imaging: CMIM
Authors:
Tristan Sylvain,
Francis Dutil,
Tess Berthier,
Lisa Di Jorio,
Margaux Luck,
Devon Hjelm,
Yoshua Bengio
Abstract:
In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not…
▽ More
In hospitals, data are siloed to specific information systems that make the same information available under different modalities such as the different medical imaging exams the patient undergoes (CT scans, MRI, PET, Ultrasound, etc.) and their associated radiology reports. This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time.
In this paper, we propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality drop** at test-time, using recent advances in mutual information maximization. By maximizing cross-modal information at train time, we are able to outperform several state-of-the-art baselines in two different settings, medical image classification, and segmentation. In particular, our method is shown to have a strong impact on the inference-time performance of weaker modalities.
△ Less
Submitted 1 February, 2021; v1 submitted 20 October, 2020;
originally announced October 2020.
InfoMask: Masked Variational Latent Representation to Localize Chest Disease
Authors:
Saeid Asgari Taghanaki,
Mohammad Havaei,
Tess Berthier,
Francis Dutil,
Lisa Di Jorio,
Ghassan Hamarneh,
Yoshua Bengio
Abstract:
The scarcity of richly annotated medical images is limiting supervised deep learning based solutions to medical image analysis tasks, such as localizing discriminatory radiomic disease signatures. Therefore, it is desirable to leverage unsupervised and weakly supervised models. Most recent weakly supervised localization methods apply attention maps or region proposals in a multiple instance learni…
▽ More
The scarcity of richly annotated medical images is limiting supervised deep learning based solutions to medical image analysis tasks, such as localizing discriminatory radiomic disease signatures. Therefore, it is desirable to leverage unsupervised and weakly supervised models. Most recent weakly supervised localization methods apply attention maps or region proposals in a multiple instance learning formulation. While attention maps can be noisy, leading to erroneously highlighted regions, it is not simple to decide on an optimal window/bag size for multiple instance learning approaches. In this paper, we propose a learned spatial masking mechanism to filter out irrelevant background signals from attention maps. The proposed method minimizes mutual information between a masked variational representation and the input while maximizing the information between the masked representation and class labels. This results in more accurate localization of discriminatory regions. We tested the proposed model on the ChestX-ray8 dataset to localize pneumonia from chest X-ray images without using any pixel-level or bounding-box annotations.
△ Less
Submitted 6 June, 2019; v1 submitted 27 March, 2019;
originally announced March 2019.
