Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex
Authors:
Spandan Madan,
Will Xiao,
Mingran Cao,
Hanspeter Pfister,
Margaret Livingstone,
Gabriel Kreiman
Abstract:
We characterized the generalization capabilities of DNN-based encoding models when predicting neuronal responses from the visual cortex. We collected \textit{MacaqueITBench}, a large-scale dataset of neural population responses from the macaque inferior temporal (IT) cortex to over $300,000$ images, comprising $8,233$ unique natural images presented to seven monkeys over $109$ sessions. Using \tex…
▽ More
We characterized the generalization capabilities of DNN-based encoding models when predicting neuronal responses from the visual cortex. We collected \textit{MacaqueITBench}, a large-scale dataset of neural population responses from the macaque inferior temporal (IT) cortex to over $300,000$ images, comprising $8,233$ unique natural images presented to seven monkeys over $109$ sessions. Using \textit{MacaqueITBench}, we investigated the impact of distribution shifts on models predicting neural activity by dividing the images into Out-Of-Distribution (OOD) train and test splits. The OOD splits included several different image-computable types including image contrast, hue, intensity, temperature, and saturation. Compared to the performance on in-distribution test images -- the conventional way these models have been evaluated -- models performed worse at predicting neuronal responses to out-of-distribution images, retaining as little as $20\%$ of the performance on in-distribution test images. The generalization performance under OOD shifts can be well accounted by a simple image similarity metric -- the cosine distance between image representations extracted from a pre-trained object recognition model is a strong predictor of neural predictivity under different distribution shifts. The dataset of images, neuronal firing rate recordings, and computational benchmarks are hosted publicly at: https://bit.ly/3zeutVd.
△ Less
Submitted 16 June, 2024;
originally announced June 2024.
Single-objective selective-volume illumination microscopy enables high-contrast light-field imaging
Authors:
Sara Madaan,
Kevin Keomanee-Dizon,
Matt Jones,
Chenyang Zhong,
Anna Nadtochiy,
Peter Luu,
Scott E. Fraser,
Thai V. Truong
Abstract:
The performance of light-field microscopy is improved by selectively illuminating the relevant subvolume of the specimen with a second objective lens [1-3]. Here we advance this approach to a single-objective geometry, using an oblique one-photon illumination path or two-photon illumination to accomplish selective-volume excitation. The elimination of the second orthogonally oriented objective to…
▽ More
The performance of light-field microscopy is improved by selectively illuminating the relevant subvolume of the specimen with a second objective lens [1-3]. Here we advance this approach to a single-objective geometry, using an oblique one-photon illumination path or two-photon illumination to accomplish selective-volume excitation. The elimination of the second orthogonally oriented objective to selectively excite the volume of interest simplifies specimen mounting; yet, this single-objective approach still reduces out-of-volume background, resulting in improvements in image contrast, effective resolution, and volume reconstruction quality. We validate our new approach through imaging live develo** zebrafish, demonstrating the technology's ability to capture imaging data from large volumes synchronously with high contrast, while remaining compatible with standard microscope sample mounting.
△ Less
Submitted 30 October, 2020; v1 submitted 1 October, 2020;
originally announced October 2020.
