-
Learnable latent embeddings for joint behavioral and neural analysis
Authors:
Steffen Schneider,
** Hwa Lee,
Mackenzie Weygandt Mathis
Abstract:
Map** behavioral actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioral data increases, there is growing interest in modeling neural dynamics during adaptive behaviors to probe neural representations. In particular, neural latent embeddings can reveal underlying correlates of behavior, yet, we lack non-linear techniques that can exp…
▽ More
Map** behavioral actions to neural activity is a fundamental goal of neuroscience. As our ability to record large neural and behavioral data increases, there is growing interest in modeling neural dynamics during adaptive behaviors to probe neural representations. In particular, neural latent embeddings can reveal underlying correlates of behavior, yet, we lack non-linear techniques that can explicitly and flexibly leverage joint behavior and neural data. Here, we fill this gap with a novel method, CEBRA, that jointly uses behavioral and neural data in a hypothesis- or discovery-driven manner to produce consistent, high-performance latent spaces. We validate its accuracy and demonstrate our tool's utility for both calcium and electrophysiology datasets, across sensory and motor tasks, and in simple or complex behaviors across species. It allows for single and multi-session datasets to be leveraged for hypothesis testing or can be used label-free. Lastly, we show that CEBRA can be used for the map** of space, uncovering complex kinematic features, and rapid, high-accuracy decoding of natural movies from visual cortex.
△ Less
Submitted 5 October, 2022; v1 submitted 1 April, 2022;
originally announced April 2022.
-
SuperAnimal pretrained pose estimation models for behavioral analysis
Authors:
Shaokai Ye,
Anastasiia Filippova,
Jessy Lauer,
Steffen Schneider,
Maxime Vidal,
Tian Qiu,
Alexander Mathis,
Mackenzie Weygandt Mathis
Abstract:
Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a serie…
▽ More
Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models that can be used on over 45 species, without additional human labels. Concretely, we introduce a method to unify the keypoint space across differently labeled datasets (via our generalized data converter) and for training these diverse datasets in a manner such that they don't catastrophically forget keypoints given the unbalanced inputs (via our keypoint gradient masking and memory replay approaches). These models show excellent performance across six pose benchmarks. Then, to ensure maximal usability for end-users, we demonstrate how to fine-tune the models on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If the models are fine-tuned, we show SuperAnimal models are 10-100$\times$ more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation.
△ Less
Submitted 30 December, 2023; v1 submitted 14 March, 2022;
originally announced March 2022.
-
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
Authors:
Alexander Mathis,
Steffen Schneider,
Jessy Lauer,
Mackenzie W. Mathis
Abstract:
Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss…
▽ More
Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.
△ Less
Submitted 2 September, 2020; v1 submitted 1 September, 2020;
originally announced September 2020.
-
Random field theory-based p-values: A review of the SPM implementation
Authors:
Dirk Ostwald,
Sebastian Schneider,
Rasmus Bruckner,
Lilla Horvath
Abstract:
P-values and null-hypothesis significance testing are popular data-analytical tools in functional neuroimaging. Sparked by the analysis of resting-state fMRI data, there has been a resurgence of interest in the validity of some of the p-values evaluated with the widely used software SPM in recent years. The default parametric p-values reported in SPM are based on the application of results from ra…
▽ More
P-values and null-hypothesis significance testing are popular data-analytical tools in functional neuroimaging. Sparked by the analysis of resting-state fMRI data, there has been a resurgence of interest in the validity of some of the p-values evaluated with the widely used software SPM in recent years. The default parametric p-values reported in SPM are based on the application of results from random field theory to statistical parametric maps, a framework commonly referred to as RFT. While RFT was established two decades ago and has since been applied in a plethora of fMRI studies, there does not exist a unified documentation of the mathematical and computational underpinnings of RFT as implemented in current versions of SPM. Here, we provide such a documentation with the aim of contributing to contemporary efforts towards higher levels of computational transparency in functional neuroimaging.
△ Less
Submitted 9 August, 2021; v1 submitted 13 August, 2018;
originally announced August 2018.
-
Particle methods enable fast and simple approximation of Sobolev gradients in image segmentation
Authors:
Ivo F. Sbalzarini,
Sophie Schneider,
Janick Cardinale
Abstract:
Bio-image analysis is challenging due to inhomogeneous intensity distributions and high levels of noise in the images. Bayesian inference provides a principled way for regularizing the problem using prior knowledge. A fundamental choice is how one measures "distances" between shapes in an image. It has been shown that the straightforward geometric L2 distance is degenerate and leads to pathologica…
▽ More
Bio-image analysis is challenging due to inhomogeneous intensity distributions and high levels of noise in the images. Bayesian inference provides a principled way for regularizing the problem using prior knowledge. A fundamental choice is how one measures "distances" between shapes in an image. It has been shown that the straightforward geometric L2 distance is degenerate and leads to pathological situations. This is avoided when using Sobolev gradients, rendering the segmentation problem less ill-posed. The high computational cost and implementation overhead of Sobolev gradients, however, have hampered practical applications. We show how particle methods as applied to image segmentation allow for a simple and computationally efficient implementation of Sobolev gradients. We show that the evaluation of Sobolev gradients amounts to particle-particle interactions along the contour in an image. We extend an existing particle-based segmentation algorithm to using Sobolev gradients. Using synthetic and real-world images, we benchmark the results for both 2D and 3D images using piecewise smooth and piecewise constant region models. The present particle approximation of Sobolev gradients is 2.8 to 10 times faster than the previous reference implementation, but retains the known favorable properties of Sobolev gradients. This speedup is achieved by using local particle-particle interactions instead of solving a global Poisson equation at each iteration. The computational time per iteration is higher for Sobolev gradients than for L2 gradients. Since Sobolev gradients precondition the optimization problem, however, a smaller number of overall iterations may be necessary for the algorithm to converge, which can in some cases amortize the higher per-iteration cost.
△ Less
Submitted 2 March, 2014;
originally announced March 2014.
