-
CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders
Authors:
Anthony Fuller,
Koreen Millard,
James R. Green
Abstract:
A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral opti…
▽ More
A vital and rapidly growing application, remote sensing offers vast yet sparsely labeled, spatially aligned multimodal data; this makes self-supervised learning algorithms invaluable. We present CROMA: a framework that combines contrastive and reconstruction self-supervised objectives to learn rich unimodal and multimodal representations. Our method separately encodes masked-out multispectral optical and synthetic aperture radar samples -- aligned in space and time -- and performs cross-modal contrastive learning. Another encoder fuses these sensors, producing joint multimodal encodings that are used to predict the masked patches via a lightweight decoder. We show that these objectives are complementary when leveraged on spatially aligned multimodal data. We also introduce X- and 2D-ALiBi, which spatially biases our cross- and self-attention matrices. These strategies improve representations and allow our models to effectively extrapolate to images up to 17.6x larger at test-time. CROMA outperforms the current SoTA multispectral model, evaluated on: four classification benchmarks -- finetuning (avg. 1.8%), linear (avg. 2.4%) and nonlinear (avg. 1.4%) probing, kNN classification (avg. 3.5%), and K-means clustering (avg. 8.4%); and three segmentation benchmarks (avg. 6.4%). CROMA's rich, optionally multimodal representations can be widely leveraged across remote sensing applications.
△ Less
Submitted 1 November, 2023;
originally announced November 2023.
-
Probing single electrons across 300 mm spin qubit wafers
Authors:
Samuel Neyens,
Otto K. Zietz,
Thomas F. Watson,
Florian Luthi,
Aditi Nethwewala,
Hubert C. George,
Eric Henry,
Mohammad Islam,
Andrew J. Wagner,
Felix Borjans,
Elliot J. Connors,
J. Corrigan,
Matthew J. Curry,
Daniel Keith,
Roza Kotlyar,
Lester F. Lampert,
Mateusz T. Madzik,
Kent Millard,
Fahd A. Mohiyaddin,
Stefano Pellerano,
Ravi Pillarisetty,
Mick Ramsey,
Rostyslav Savytskyy,
Simon Schaal,
Guoji Zheng
, et al. (5 additional authors not shown)
Abstract:
Building a fault-tolerant quantum computer will require vast numbers of physical qubits. For qubit technologies based on solid state electronic devices, integrating millions of qubits in a single processor will require device fabrication to reach a scale comparable to that of the modern CMOS industry. Equally importantly, the scale of cryogenic device testing must keep pace to enable efficient dev…
▽ More
Building a fault-tolerant quantum computer will require vast numbers of physical qubits. For qubit technologies based on solid state electronic devices, integrating millions of qubits in a single processor will require device fabrication to reach a scale comparable to that of the modern CMOS industry. Equally importantly, the scale of cryogenic device testing must keep pace to enable efficient device screening and to improve statistical metrics like qubit yield and voltage variation. Spin qubits based on electrons in Si have shown impressive control fidelities but have historically been challenged by yield and process variation. Here we present a testing process using a cryogenic 300 mm wafer prober to collect high-volume data on the performance of hundreds of industry-manufactured spin qubit devices at 1.6 K. This testing method provides fast feedback to enable optimization of the CMOS-compatible fabrication process, leading to high yield and low process variation. Using this system, we automate measurements of the operating point of spin qubits and probe the transitions of single electrons across full wafers. We analyze the random variation in single-electron operating voltages and find that the optimized fabrication process leads to low levels of disorder at the 300 mm scale. Together these results demonstrate the advances that can be achieved through the application of CMOS industry techniques to the fabrication and measurement of spin qubit devices.
△ Less
Submitted 3 May, 2024; v1 submitted 10 July, 2023;
originally announced July 2023.
-
Transfer Learning with Pretrained Remote Sensing Transformers
Authors:
Anthony Fuller,
Koreen Millard,
James R. Green
Abstract:
Although the remote sensing (RS) community has begun to pretrain transformers (intended to be fine-tuned on RS tasks), it is unclear how these models perform under distribution shifts. Here, we pretrain a new RS transformer--called SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along with five other models) to investigate how it performs on distributions not seen during…
▽ More
Although the remote sensing (RS) community has begun to pretrain transformers (intended to be fine-tuned on RS tasks), it is unclear how these models perform under distribution shifts. Here, we pretrain a new RS transformer--called SatViT-V2--on 1.3 million satellite-derived RS images, then fine-tune it (along with five other models) to investigate how it performs on distributions not seen during training. We split an expertly labeled land cover dataset into 14 datasets based on source biome. We train each model on each biome separately and test them on all other biomes. In all, this amounts to 1638 biome transfer experiments. After fine-tuning, we find that SatViT-V2 outperforms SatViT-V1 by 3.1% on in-distribution (matching biomes) and 2.8% on out-of-distribution (mismatching biomes) data. Additionally, we find that initializing fine-tuning from the linear probed solution (i.e., leveraging LPFT [1]) improves SatViT-V2's performance by another 1.2% on in-distribution and 2.4% on out-of-distribution data. Next, we find that pretrained RS transformers are better calibrated under distribution shifts than non-pretrained models and leveraging LPFT results in further improvements in model calibration. Lastly, we find that five measures of distribution shift are moderately correlated with biome transfer performance. We share code and pretrained model weights. (https://github.com/antofuller/SatViT)
△ Less
Submitted 28 September, 2022;
originally announced September 2022.
