-
FRET nanoscopy enables seamless imaging of molecular assemblies with sub-nanometer resolution
Authors:
Jan-Hendrik Budde,
Nicolaas van der Voort,
Suren Felekyan,
Julian Folz,
Ralf Kühnemuth,
Paul Lauterjung,
Markus Köhler,
Andreas Schönle,
Julian Sindram,
Marius Otten,
Matthias Karg,
Christian Herrmann,
Anders Barth,
Claus A. M. Seidel
Abstract:
By circumventing the optical diffraction limit, super-resolved fluorescence microscopies enable the study of larger cellular structures and molecular assemblies. However, fluorescence nanoscopy currently lacks the spatiotemporal resolution to resolve distances on the size of individual molecules and reveal the conformational fine structure and dynamics of molecular complexes. Here we establish FRE…
▽ More
By circumventing the optical diffraction limit, super-resolved fluorescence microscopies enable the study of larger cellular structures and molecular assemblies. However, fluorescence nanoscopy currently lacks the spatiotemporal resolution to resolve distances on the size of individual molecules and reveal the conformational fine structure and dynamics of molecular complexes. Here we establish FRET nanoscopy by combining colocalization STED microscopy with multiparameter FRET spectroscopy. We simultaneously localize donor and acceptor dyes of single FRET pairs with nanometer resolution and quantitatively measure intramolecular distances with sub-nanometer precision over a large dynamic range. While FRET provides isotropic 3D distance information, colocalization measures the projected distance onto the image plane. The combined information allows us to directly determine its 3D orientation using Pythagoras's theorem. Studying two DNA model systems and the human guanylate binding protein hGBP1, we demonstrate that FRET nanoscopy unravels the interplay between their spatial organization and local molecular conformation in a complex environment.
△ Less
Submitted 21 January, 2022; v1 submitted 30 July, 2021;
originally announced August 2021.
-
Training Deep Neural Networks Without Batch Normalization
Authors:
Divya Gaur,
Joachim Folz,
Andreas Dengel
Abstract:
Training neural networks is an optimization problem, and finding a decent set of parameters through gradient descent can be a difficult task. A host of techniques has been developed to aid this process before and during the training phase. One of the most important and widely used class of method is normalization. It is generally favorable for neurons to receive inputs that are distributed with ze…
▽ More
Training neural networks is an optimization problem, and finding a decent set of parameters through gradient descent can be a difficult task. A host of techniques has been developed to aid this process before and during the training phase. One of the most important and widely used class of method is normalization. It is generally favorable for neurons to receive inputs that are distributed with zero mean and unit variance, so we use statistics about dataset to normalize them before the first layer. However, this property cannot be guaranteed for the intermediate activations inside the network. A widely used method to enforce this property inside the network is batch normalization. It was developed to combat covariate shift inside networks. Empirically it is known to work, but there is a lack of theoretical understanding about its effectiveness and potential drawbacks it might have when used in practice. This work studies batch normalization in detail, while comparing it with other methods such as weight normalization, gradient clip** and dropout. The main purpose of this work is to determine if it is possible to train networks effectively when batch normalization is removed through adaption of the training process.
△ Less
Submitted 18 August, 2020;
originally announced August 2020.
-
P $\approx$ NP, at least in Visual Question Answering
Authors:
Shailza Jolly,
Sebastian Palacio,
Joachim Folz,
Federico Raue,
Joern Hees,
Andreas Dengel
Abstract:
In recent years, progress in the Visual Question Answering (VQA) field has largely been driven by public challenges and large datasets. One of the most widely-used of these is the VQA 2.0 dataset, consisting of polar ("yes/no") and non-polar questions. Looking at the question distribution over all answers, we find that the answers "yes" and "no" account for 38 % of the questions, while the remaini…
▽ More
In recent years, progress in the Visual Question Answering (VQA) field has largely been driven by public challenges and large datasets. One of the most widely-used of these is the VQA 2.0 dataset, consisting of polar ("yes/no") and non-polar questions. Looking at the question distribution over all answers, we find that the answers "yes" and "no" account for 38 % of the questions, while the remaining 62% are spread over the more than 3000 remaining answers. While several sources of biases have already been investigated in the field, the effects of such an over-representation of polar vs. non-polar questions remain unclear. In this paper, we measure the potential confounding factors when polar and non-polar samples are used jointly to train a baseline VQA classifier, and compare it to an upper bound where the over-representation of polar questions is excluded from the training. Further, we perform cross-over experiments to analyze how well the feature spaces align. Contrary to expectations, we find no evidence of counterproductive effects in the joint training of unbalanced classes. In fact, by exploring the intermediate feature space of visual-text embeddings, we find that the feature space of polar questions already encodes sufficient structure to answer many non-polar questions. Our results indicate that the polar (P) and the non-polar (NP) feature spaces are strongly aligned, hence the expression P $\approx$ NP
△ Less
Submitted 27 March, 2020; v1 submitted 26 March, 2020;
originally announced March 2020.
-
What do Deep Networks Like to See?
Authors:
Sebastian Palacio,
Joachim Folz,
Jörn Hees,
Federico Raue,
Damian Borth,
Andreas Dengel
Abstract:
We propose a novel way to measure and understand convolutional neural networks by quantifying the amount of input signal they let in. To do this, an autoencoder (AE) was fine-tuned on gradients from a pre-trained classifier with fixed parameters. We compared the reconstructed samples from AEs that were fine-tuned on a set of image classifiers (AlexNet, VGG16, ResNet-50, and Inception~v3) and found…
▽ More
We propose a novel way to measure and understand convolutional neural networks by quantifying the amount of input signal they let in. To do this, an autoencoder (AE) was fine-tuned on gradients from a pre-trained classifier with fixed parameters. We compared the reconstructed samples from AEs that were fine-tuned on a set of image classifiers (AlexNet, VGG16, ResNet-50, and Inception~v3) and found substantial differences. The AE learns which aspects of the input space to preserve and which ones to ignore, based on the information encoded in the backpropagated gradients. Measuring the changes in accuracy when the signal of one classifier is used by a second one, a relation of total order emerges. This order depends directly on each classifier's input signal but it does not correlate with classification accuracy or network size. Further evidence of this phenomenon is provided by measuring the normalized mutual information between original images and auto-encoded reconstructions from different fine-tuned AEs. These findings break new ground in the area of neural network understanding, opening a new way to reason, debug, and interpret their results. We present four concrete examples in the literature where observations can now be explained in terms of the input signal that a model uses.
△ Less
Submitted 22 March, 2018;
originally announced March 2018.
-
Adversarial Defense based on Structure-to-Signal Autoencoders
Authors:
Joachim Folz,
Sebastian Palacio,
Joern Hees,
Damian Borth,
Andreas Dengel
Abstract:
Adversarial attack methods have demonstrated the fragility of deep neural networks. Their imperceptible perturbations are frequently able fool classifiers into potentially dangerous misclassifications. We propose a novel way to interpret adversarial perturbations in terms of the effective input signal that classifiers actually use. Based on this, we apply specially trained autoencoders, referred t…
▽ More
Adversarial attack methods have demonstrated the fragility of deep neural networks. Their imperceptible perturbations are frequently able fool classifiers into potentially dangerous misclassifications. We propose a novel way to interpret adversarial perturbations in terms of the effective input signal that classifiers actually use. Based on this, we apply specially trained autoencoders, referred to as S2SNets, as defense mechanism. They follow a two-stage training scheme: first unsupervised, followed by a fine-tuning of the decoder, using gradients from an existing classifier. S2SNets induce a shift in the distribution of gradients propagated through them, strip** them from class-dependent signal. We analyze their robustness against several white-box and gray-box scenarios on the large ImageNet dataset. Our approach reaches comparable resilience in white-box attack scenarios as other state-of-the-art defenses in gray-box scenarios. We further analyze the relationships of AlexNet, VGG 16, ResNet 50 and Inception v3 in adversarial space, and found that VGG 16 is the easiest to fool, while perturbations from ResNet 50 are the most transferable.
△ Less
Submitted 21 March, 2018;
originally announced March 2018.
-
Multi-Task Learning for Segmentation of Building Footprints with Deep Neural Networks
Authors:
Benjamin Bischke,
Patrick Helber,
Joachim Folz,
Damian Borth,
Andreas Dengel
Abstract:
The increased availability of high resolution satellite imagery allows to sense very detailed structures on the surface of our planet. Access to such information opens up new directions in the analysis of remote sensing imagery. However, at the same time this raises a set of new challenges for existing pixel-based prediction methods, such as semantic segmentation approaches. While deep neural netw…
▽ More
The increased availability of high resolution satellite imagery allows to sense very detailed structures on the surface of our planet. Access to such information opens up new directions in the analysis of remote sensing imagery. However, at the same time this raises a set of new challenges for existing pixel-based prediction methods, such as semantic segmentation approaches. While deep neural networks have achieved significant advances in the semantic segmentation of high resolution images in the past, most of the existing approaches tend to produce predictions with poor boundaries. In this paper, we address the problem of preserving semantic segmentation boundaries in high resolution satellite imagery by introducing a new cascaded multi-task loss. We evaluate our approach on Inria Aerial Image Labeling Dataset which contains large-scale and high resolution images. Our results show that we are able to outperform state-of-the-art methods by 8.3\% without any additional post-processing step.
△ Less
Submitted 18 September, 2017;
originally announced September 2017.
-
An Evolutionary Algorithm to Learn SPARQL Queries for Source-Target-Pairs: Finding Patterns for Human Associations in DBpedia
Authors:
Jörn Hees,
Rouven Bauer,
Joachim Folz,
Damian Borth,
Andreas Dengel
Abstract:
Efficient usage of the knowledge provided by the Linked Data community is often hindered by the need for domain experts to formulate the right SPARQL queries to answer questions. For new questions they have to decide which datasets are suitable and in which terminology and modelling style to phrase the SPARQL query.
In this work we present an evolutionary algorithm to help with this challenging…
▽ More
Efficient usage of the knowledge provided by the Linked Data community is often hindered by the need for domain experts to formulate the right SPARQL queries to answer questions. For new questions they have to decide which datasets are suitable and in which terminology and modelling style to phrase the SPARQL query.
In this work we present an evolutionary algorithm to help with this challenging task. Given a training list of source-target node-pair examples our algorithm can learn patterns (SPARQL queries) from a SPARQL endpoint. The learned patterns can be visualised to form the basis for further investigation, or they can be used to predict target nodes for new source nodes.
Amongst others, we apply our algorithm to a dataset of several hundred human associations (such as "circle - square") to find patterns for them in DBpedia. We show the scalability of the algorithm by running it against a SPARQL endpoint loaded with > 7.9 billion triples. Further, we use the resulting SPARQL queries to mimic human associations with a Mean Average Precision (MAP) of 39.9 % and a Recall@10 of 63.9 %.
△ Less
Submitted 13 September, 2016; v1 submitted 25 July, 2016;
originally announced July 2016.
