Showing 1–40 of 40 results for author: Lobell, D

ExPLoRA: Parameter-Efficient Extended Pre-Training to Adapt Vision Transformers under Domain Shifts

Authors: Samar Khanna, Medhanie Irgau, David B. Lobell, Stefano Ermon

Abstract: Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new… ▽ More Parameter-efficient fine-tuning (PEFT) techniques such as low-rank adaptation (LoRA) can effectively adapt large pre-trained foundation models to downstream tasks using only a small fraction (0.1%-10%) of the original trainable weights. An under-explored question of PEFT is in extending the pre-training phase without supervised labels; that is, can we adapt a pre-trained foundation model to a new domain via efficient self-supervised pre-training on this new domain? In this work, we introduce ExPLoRA, a highly effective technique to improve transfer learning of pre-trained vision transformers (ViTs) under domain shifts. Initializing a ViT with pre-trained weights on large, natural-image datasets such as from DinoV2 or MAE, ExPLoRA continues the unsupervised pre-training objective on a new domain. In this extended pre-training phase, ExPLoRA only unfreezes 1-2 pre-trained ViT blocks and all normalization layers, and then tunes all other layers with LoRA. Finally, we fine-tune the resulting model only with LoRA on this new domain for supervised learning. Our experiments demonstrate state-of-the-art results on satellite imagery, even outperforming fully pre-training and fine-tuning ViTs. Using the DinoV2 training objective, we demonstrate up to 7% improvement in linear probing top-1 accuracy on downstream tasks while using <10% of the number of parameters that are used in prior fully-tuned state-of-the art approaches. Our ablation studies confirm the efficacy of our approach over other baselines, including PEFT and simply unfreezing more transformer blocks. △ Less

Submitted 16 June, 2024; originally announced June 2024.

Large increases in public R&D investment are needed to avoid declines of US agricultural productivity

Authors: Ariel Ortiz-Bobea, Robert G. Chambers, Yurou He, David B. Lobell

Abstract: Increasing agricultural productivity is a gradual process with significant time lags between research and development (R&D) investment and the resulting gains. We estimate the response of US agricultural Total Factor Productivity (TFP) to both R&D investment and weather, and quantify the public R&D spending required to offset the emerging impacts of climate change. We find that offsetting the clim… ▽ More Increasing agricultural productivity is a gradual process with significant time lags between research and development (R&D) investment and the resulting gains. We estimate the response of US agricultural Total Factor Productivity (TFP) to both R&D investment and weather, and quantify the public R&D spending required to offset the emerging impacts of climate change. We find that offsetting the climate-induced productivity slowdown by 2050 alone requires a sustained public R&D spending growth of 5.2-7.8% per year over 2021-2050. This amounts to an additional $208-$434B investment over this period. These are substantial requirements comparable to the public R&D spending growth that followed the two World Wars. △ Less

Submitted 17 May, 2024; v1 submitted 13 May, 2024; originally announced May 2024.

Comments: Main text: 19 pages, 4 figures. Supplementary material: 47 pages, 20 figures, 13 tables

Large Language Models are Geographically Biased

Authors: Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is p… ▽ More Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $ρ$ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's $ρ$ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. △ Less

Submitted 4 February, 2024; originally announced February 2024.

DiffusionSat: A Generative Foundation Model for Satellite Imagery

Authors: Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregular… ▽ More Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregularly sampled across time -- and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently spatio-temporal, requiring conditional generation tasks not supported by traditional methods based on captions or images. In this paper, we present DiffusionSat, to date the largest generative foundation model trained on a collection of publicly available large, high-resolution remote sensing datasets. As text-based captions are sparsely available for satellite images, we incorporate the associated metadata such as geolocation as conditioning information. Our method produces realistic samples and can be used to solve multiple generative tasks including temporal generation, superresolution given multi-spectral inputs and in-painting. Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale generative foundation model for satellite imagery. The project website can be found here: https://samar-khanna.github.io/DiffusionSat/ △ Less

Submitted 25 May, 2024; v1 submitted 6 December, 2023; originally announced December 2023.

Comments: Published at ICLR 2024

Biases in estimates of air pollution impacts: the role of omitted variables and measurement errors

Authors: Dan M. Kluger, David B. Lobell, Art B. Owen

Abstract: Observational studies often use linear regression to assess the effect of ambient air pollution on outcomes of interest, such as human health indicators or crop yields. Yet pollution datasets are typically noisy and include only a subset of the potentially relevant pollutants, giving rise to both measurement error bias (MEB) and omitted variable bias (OVB). While it is well understood that these b… ▽ More Observational studies often use linear regression to assess the effect of ambient air pollution on outcomes of interest, such as human health indicators or crop yields. Yet pollution datasets are typically noisy and include only a subset of the potentially relevant pollutants, giving rise to both measurement error bias (MEB) and omitted variable bias (OVB). While it is well understood that these biases exist, less is understood about whether these biases tend to be positive or negative, even though it is sometimes falsely claimed that measurement error simply biases regression coefficient estimates towards zero. In this paper, we study the direction of these biases under the realistic assumptions that the concentrations of different types of air pollutants are positively correlated with each other and that each type of pollutant has a nonpositive association with the outcome variable. We demonstrate both theoretically and using simulations that under these two assumptions, the OVB will typically be negative and that more often than not the MEB for null pollutants or for pollutants that are perfectly measured will be negative. We also use a crop yield and air pollution dataset to show that these biases tend to be negative in the setting of our motivating application. We do this by introducing a validation scheme that does not require knowing the true coefficients. While this paper is motivated by studies assessing the effect of air pollutants on crop yields, the findings are also relevant to regression-based studies assessing the effect of air pollutants on human health outcomes. The validation scheme can also be used to empirically study OVB or MEB in other contexts. △ Less

Submitted 6 March, 2024; v1 submitted 12 October, 2023; originally announced October 2023.

Comments: Added a data-based validation using a dataset of pollutants and crop yields from the Midwestern United States

GeoLLM: Extracting Geospatial Knowledge from Large Language Models

Authors: Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, Stefano Ermon

Abstract: The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged… ▽ More The application of machine learning (ML) in a range of geospatial tasks is increasingly common but often relies on globally available covariates such as satellite imagery that can either be expensive or lack predictive power. Here we explore the question of whether the vast amounts of knowledge found in Internet language corpora, now compressed within large language models (LLMs), can be leveraged for geospatial prediction tasks. We first demonstrate that LLMs embed remarkable spatial information about locations, but naively querying LLMs using geographic coordinates alone is ineffective in predicting key indicators like population density. We then present GeoLLM, a novel method that can effectively extract geospatial knowledge from LLMs with auxiliary map data from OpenStreetMap. We demonstrate the utility of our approach across multiple tasks of central interest to the international community, including the measurement of population density and economic livelihoods. Across these tasks, our method demonstrates a 70% improvement in performance (measured using Pearson's $r^2$) relative to baselines that use nearest neighbors or use information directly from the prompt, and performance equal to or exceeding satellite-based benchmarks in the literature. With GeoLLM, we observe that GPT-3.5 outperforms Llama 2 and RoBERTa by 19% and 51% respectively, suggesting that the performance of our method scales well with the size of the model and its pretraining dataset. Our experiments reveal that LLMs are remarkably sample-efficient, rich in geospatial information, and robust across the globe. Crucially, GeoLLM shows promise in mitigating the limitations of existing geospatial covariates and complementing them well. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM △ Less

Submitted 24 February, 2024; v1 submitted 9 October, 2023; originally announced October 2023.

Comments: Accepted to ICLR 2024

HarvestNet: A Dataset for Detecting Smallholder Farming Activity Using Harvest Piles and Remote Sensing

Authors: Jonathan Xu, Amna Elmustafa, Liya Weldegebriel, Emnet Negash, Richard Lee, Chenlin Meng, Stefano Ermon, David Lobell

Abstract: Small farms contribute to a large share of the productive land in develo** countries. In regions such as sub-Saharan Africa, where 80\% of farms are small (under 2 ha in size), the task of map** smallholder cropland is an important part of tracking sustainability measures such as crop productivity. However, the visually diverse and nuanced appearance of small farms has limited the effectivenes… ▽ More Small farms contribute to a large share of the productive land in develo** countries. In regions such as sub-Saharan Africa, where 80\% of farms are small (under 2 ha in size), the task of map** smallholder cropland is an important part of tracking sustainability measures such as crop productivity. However, the visually diverse and nuanced appearance of small farms has limited the effectiveness of traditional approaches to cropland map**. Here we introduce a new approach based on the detection of harvest piles characteristic of many smallholder systems throughout the world. We present HarvestNet, a dataset for map** the presence of farms in the Ethiopian regions of Tigray and Amhara during 2020-2023, collected using expert knowledge and satellite images, totaling 7k hand-labeled images and 2k ground-collected labels. We also benchmark a set of baselines, including SOTA models in remote sensing, with our best models having around 80\% classification performance on hand labelled data and 90\% and 98\% accuracy on ground truth data for Tigray and Amhara, respectively. We also perform a visual comparison with a widely used pre-existing coverage map and show that our model detects an extra 56,621 hectares of cropland in Tigray. We conclude that remote sensing of harvest piles can contribute to more timely and accurate cropland assessments in food insecure regions. The dataset can be accessed through https://figshare.com/s/45a7b45556b90a9a11d2, while the code for the dataset and benchmarks is publicly available at https://github.com/jonxuxu/harvest-piles △ Less

Submitted 5 March, 2024; v1 submitted 23 August, 2023; originally announced August 2023.

Comments: submitted to AAAI24

Building Coverage Estimation with Low-resolution Remote Sensing Imagery

Authors: Enci Liu, Chenlin Meng, Matthew Kolodner, Eun Jee Sung, Sihang Chen, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Building coverage statistics provide crucial insights into the urbanization, infrastructure, and poverty level of a region, facilitating efforts towards alleviating poverty, building sustainable cities, and allocating infrastructure investments and public service provision. Global map** of buildings has been made more efficient with the incorporation of deep learning models into the pipeline. Ho… ▽ More Building coverage statistics provide crucial insights into the urbanization, infrastructure, and poverty level of a region, facilitating efforts towards alleviating poverty, building sustainable cities, and allocating infrastructure investments and public service provision. Global map** of buildings has been made more efficient with the incorporation of deep learning models into the pipeline. However, these models typically rely on high-resolution satellite imagery which are expensive to collect and infrequently updated. As a result, building coverage data are not updated timely especially in develo** regions where the built environment is changing quickly. In this paper, we propose a method for estimating building coverage using only publicly available low-resolution satellite imagery that is more frequently updated. We show that having a multi-node quantile regression layer greatly improves the model's spatial and temporal generalization. Our model achieves a coefficient of determination ($R^2$) as high as 0.968 on predicting building coverage in regions of different levels of development around the world. We demonstrate that the proposed model accurately predicts the building coverage from raw input images and generalizes well to unseen countries and continents, suggesting the possibility of estimating global building coverage using only low-resolution remote sensing data. △ Less

Submitted 4 January, 2023; v1 submitted 4 January, 2023; originally announced January 2023.

Annual field-scale maps of tall and short crops at the global scale using GEDI and Sentinel-2

Authors: Stefania Di Tommaso, Sherrie Wang, Vivek Vajipey, Noel Gorelick, Rob Strey, David B. Lobell

Abstract: Crop type maps are critical for tracking agricultural land use and estimating crop production. Remote sensing has proven an efficient and reliable tool for creating these maps in regions with abundant ground labels for model training, yet these labels remain difficult to obtain in many regions and years. NASA's Global Ecosystem Dynamics Investigation (GEDI) spaceborne lidar instrument, originally… ▽ More Crop type maps are critical for tracking agricultural land use and estimating crop production. Remote sensing has proven an efficient and reliable tool for creating these maps in regions with abundant ground labels for model training, yet these labels remain difficult to obtain in many regions and years. NASA's Global Ecosystem Dynamics Investigation (GEDI) spaceborne lidar instrument, originally designed for forest monitoring, has shown promise for distinguishing tall and short crops. In the current study, we leverage GEDI to develop wall-to-wall maps of short vs tall crops on a global scale at 10 m resolution for 2019-2021. Specifically, we show that (1) GEDI returns can reliably be classified into tall and short crops after removing shots with extreme view angles or topographic slope, (2) the frequency of tall crops over time can be used to identify months when tall crops are at their peak height, and (3) GEDI shots in these months can then be used to train random forest models that use Sentinel-2 time series to accurately predict short vs. tall crops. Independent reference data from around the world are then used to evaluate these GEDI-S2 maps. We find that GEDI-S2 performed nearly as well as models trained on thousands of local reference training points, with accuracies of at least 87% and often above 90% throughout the Americas, Europe, and East Asia. Systematic underestimation of tall crop area was observed in regions where crops frequently exhibit low biomass, namely Africa and South Asia, and further work is needed in these systems. Although the GEDI-S2 approach only differentiates tall from short crops, in many landscapes this distinction goes a long way toward map** the main individual crop types. The combination of GEDI and Sentinel-2 thus presents a very promising path towards global crop map** with minimal reliance on ground data. △ Less

Submitted 19 December, 2022; originally announced December 2022.

With big data come big problems: pitfalls in measuring basis risk for crop index insurance

Authors: Matthieu Stigler, Apratim Dey, Andrew Hobbs, David Lobell

Abstract: New satellite sensors will soon make it possible to estimate field-level crop yields, showing a great potential for agricultural index insurance. This paper identifies an important threat to better insurance from these new technologies: data with many fields and few years can yield downward biased estimates of basis risk, a fundamental metric in index insurance. To demonstrate this bias, we use st… ▽ More New satellite sensors will soon make it possible to estimate field-level crop yields, showing a great potential for agricultural index insurance. This paper identifies an important threat to better insurance from these new technologies: data with many fields and few years can yield downward biased estimates of basis risk, a fundamental metric in index insurance. To demonstrate this bias, we use state-of-the-art satellite-based data on agricultural yields in the US and in Kenya to estimate and simulate basis risk. We find a substantive downward bias leading to a systematic overestimation of insurance quality. In this paper, we argue that big data in crop insurance can lead to a new situation where the number of variables $N$ largely exceeds the number of observations $T$. In such a situation where $T\ll N$, conventional asymptotics break, as evidenced by the large bias we find in simulations. We show how the high-dimension, low-sample-size (HDLSS) asymptotics, together with the spiked covariance model, provide a more relevant framework for the $T\ll N$ case encountered in index insurance. More precisely, we derive the asymptotic distribution of the relative share of the first eigenvalue of the covariance matrix, a measure of systematic risk in index insurance. Our formula accurately approximates the empirical bias simulated from the satellite data, and provides a useful tool for practitioners to quantify bias in insurance quality. △ Less

Submitted 29 September, 2022; originally announced September 2022.

SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery

Authors: Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B. Lobell, Stefano Ermon

Abstract: Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Develo** similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE… ▽ More Unsupervised pre-training methods for large vision models have shown to enhance performance on downstream supervised tasks. Develo** similar techniques for satellite imagery presents significant opportunities as unlabelled data is plentiful and the inherent temporal and multi-spectral structure provides avenues to further improve existing pre-training strategies. In this paper, we present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE). To leverage temporal information, we include a temporal embedding along with independently masking image patches across time. In addition, we demonstrate that encoding multi-spectral data as groups of bands with distinct spectral positional encodings is beneficial. Our approach yields strong improvements over previous state-of-the-art techniques, both in terms of supervised learning performance on benchmark datasets (up to $\uparrow$ 7%), and transfer learning performance on downstream remote sensing tasks, including land cover classification (up to $\uparrow$ 14%) and semantic segmentation. Code and data are available on the project website: https://sustainlab-group.github.io/SatMAE/ △ Less

Submitted 15 January, 2023; v1 submitted 16 July, 2022; originally announced July 2022.

Comments: Published at NeurIPS 2022. The first two listed names contributed equally to this project

Tracking Urbanization in Develo** Regions with Remote Sensing Spatial-Temporal Super-Resolution

Authors: Yutong He, William Zhang, Chenlin Meng, Marshall Burke, David B. Lobell, Stefano Ermon

Abstract: Automated tracking of urban development in areas where construction information is not available became possible with recent advancements in machine learning and remote sensing. Unfortunately, these solutions perform best on high-resolution imagery, which is expensive to acquire and infrequently available, making it difficult to scale over long time spans and across large geographies. In this work… ▽ More Automated tracking of urban development in areas where construction information is not available became possible with recent advancements in machine learning and remote sensing. Unfortunately, these solutions perform best on high-resolution imagery, which is expensive to acquire and infrequently available, making it difficult to scale over long time spans and across large geographies. In this work, we propose a pipeline that leverages a single high-resolution image and a time series of publicly available low-resolution images to generate accurate high-resolution time series for object tracking in urban construction. Our method achieves significant improvement in comparison to baselines using single image super-resolution, and can assist in extending the accessibility and scalability of building construction tracking across the develo** world. △ Less

Submitted 4 April, 2022; originally announced April 2022.

Comments: Presented at Workshop on Machine Learning for the Develo** World (ML4D) at the 35th Conference on Neural Information Processing Systems (NeurIPS) 2021

Unlocking large-scale crop field delineation in smallholder farming systems with transfer learning and weak supervision

Authors: Sherrie Wang, Francois Waldner, David B. Lobell

Abstract: Crop field boundaries aid in map** crop types, predicting yields, and delivering field-scale analytics to farmers. Recent years have seen the successful application of deep learning to delineating field boundaries in industrial agricultural systems, but field boundary datasets remain missing in smallholder systems due to (1) small fields that require high resolution satellite imagery to delineat… ▽ More Crop field boundaries aid in map** crop types, predicting yields, and delivering field-scale analytics to farmers. Recent years have seen the successful application of deep learning to delineating field boundaries in industrial agricultural systems, but field boundary datasets remain missing in smallholder systems due to (1) small fields that require high resolution satellite imagery to delineate and (2) a lack of ground labels for model training and validation. In this work, we combine transfer learning and weak supervision to overcome these challenges, and we demonstrate the methods' success in India where we efficiently generated 10,000 new field labels. Our best model uses 1.5m resolution Airbus SPOT imagery as input, pre-trains a state-of-the-art neural network on France field boundaries, and fine-tunes on India labels to achieve a median Intersection over Union (IoU) of 0.86 in India. If using 4.8m resolution PlanetScope imagery instead, the best model achieves a median IoU of 0.72. Experiments also show that pre-training in France reduces the number of India field labels needed to achieve a given performance level by as much as $20\times$ when datasets are small. These findings suggest our method is a scalable approach for delineating crop fields in regions of the world that currently lack field boundary datasets. We publicly release the 10,000 labels and delineation model to facilitate the creation of field boundary maps and new methods by the community. △ Less

Submitted 12 January, 2022; originally announced January 2022.

Comments: Under submission

Combining randomized field experiments with observational satellite data to assess the benefits of crop rotations on yields

Authors: Dan M. Kluger, Art B. Owen, David B. Lobell

Abstract: With climate change threatening agricultural productivity and global food demand increasing, it is important to better understand which farm management practices will maximize crop yields in various climatic conditions. To assess the effectiveness of agricultural practices, researchers often turn to randomized field experiments, which are reliable for identifying causal effects but are often limit… ▽ More With climate change threatening agricultural productivity and global food demand increasing, it is important to better understand which farm management practices will maximize crop yields in various climatic conditions. To assess the effectiveness of agricultural practices, researchers often turn to randomized field experiments, which are reliable for identifying causal effects but are often limited in scope and therefore lack external validity. Recently, researchers have also leveraged large observational datasets from satellites and other sources, which can lead to conclusions biased by confounding variables or systematic measurement errors. Because experimental and observational datasets have complementary strengths, in this paper we propose a method that uses a combination of experimental and observational data in the same analysis. As a case study, we focus on the causal effect of crop rotation on corn (maize) and soy yields in the Midwestern United States. We find that, in terms of root mean squared error, our hybrid method performs 13% better than using experimental data alone and 26% better than using the observational data alone in the task of predicting the effect of rotation on corn yield at held-out experimental sites. Further, the causal estimates based on our method suggest that benefits of crop rotations on corn yield are lower in years and locations with high temperatures whereas the benefits of crop rotations on soy yield are higher in years and locations with high temperatures. In particular, we estimated that the benefit of rotation on corn yields (and soy yields) was 0.84 t/ha (0.23 t/ha) on average for the top quintile of temperatures, 1.02 t/ha (0.20 t/ha) on average for the whole dataset, and 1.18 t/ha (0.15 t/ha) on average for the bottom quintile of temperatures. △ Less

Submitted 27 December, 2021; originally announced December 2021.

MSC Class: 62P12; 62D20

Journal ref: Environmental Research Letters (2022). Volume 17, Number 4

IS-COUNT: Large-scale Object Counting from Satellite Images with Covariate-based Importance Sampling

Authors: Chenlin Meng, Enci Liu, Willie Neiswanger, Jiaming Song, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Object detection in high-resolution satellite imagery is emerging as a scalable alternative to on-the-ground survey data collection in many environmental and socioeconomic monitoring applications. However, performing object detection over large geographies can still be prohibitively expensive due to the high cost of purchasing imagery and compute. Inspired by traditional survey data collection str… ▽ More Object detection in high-resolution satellite imagery is emerging as a scalable alternative to on-the-ground survey data collection in many environmental and socioeconomic monitoring applications. However, performing object detection over large geographies can still be prohibitively expensive due to the high cost of purchasing imagery and compute. Inspired by traditional survey data collection strategies, we propose an approach to estimate object count statistics over large geographies through sampling. Given a cost budget, our method selects a small number of representative areas by sampling from a learnable proposal distribution. Using importance sampling, we are able to accurately estimate object counts after processing only a small fraction of the images compared to an exhaustive approach. We show empirically that the proposed framework achieves strong performance on estimating the number of buildings in the United States and Africa, cars in Kenya, brick kilns in Bangladesh, and swimming pools in the U.S., while requiring as few as 0.01% of satellite images compared to an exhaustive approach. △ Less

Submitted 16 December, 2021; originally announced December 2021.

Comments: AAAI 2022

Optimal index insurance and basis risk decomposition: an application to Kenya

Authors: Matthieu Stigler, David Lobell

Abstract: Index insurance is a promising tool to reduce the risk faced by farmers, but high basis risk, which arises from imperfect correlation between the index and individual farm yields, has limited its adoption to date. Basis risk arises from two fundamental sources: the intrinsic heterogeneity within an insurance zone (zonal risk), and the lack of predictive accuracy of the index (design risk). Whereas… ▽ More Index insurance is a promising tool to reduce the risk faced by farmers, but high basis risk, which arises from imperfect correlation between the index and individual farm yields, has limited its adoption to date. Basis risk arises from two fundamental sources: the intrinsic heterogeneity within an insurance zone (zonal risk), and the lack of predictive accuracy of the index (design risk). Whereas previous work has focused almost exclusively on design risk, a theoretical and empirical understanding of the role of zonal risk is still lacking. Here we investigate the relative roles of zonal and design risk, using the case of maize yields in Kenya. Our first contribution is to derive a formal decomposition of basis risk, providing a simple upper bound on the insurable basis risk that any index can reach within a given zone. Our second contribution is to provide the first large-scale empirical analysis of the extent of zonal versus design risk. To do so, we use satellite estimates of yields at 10m resolution across Kenya, and investigate the effect of using smaller zones versus using different indices. Our results show a strong local heterogeneity in yields, underscoring the challenge of implementing index insurance in smallholder systems, and the potential benefits of low-cost yield measurement approaches that can enable more local definitions of insurance zones. △ Less

Submitted 6 March, 2023; v1 submitted 16 November, 2021; originally announced November 2021.

Comments: Forthcoming in American Journal of Agricultural Economics

SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

Authors: Christopher Yeh, Chenlin Meng, Sherrie Wang, Anne Driscoll, Erik Rozi, Patrick Liu, Jihyeon Lee, Marshall Burke, David B. Lobell, Stefano Ermon

Abstract: Progress toward the United Nations Sustainable Development Goals (SDGs) has been hindered by a lack of data on key environmental and socioeconomic indicators, which historically have come from ground surveys with sparse temporal and spatial coverage. Recent advances in machine learning have made it possible to utilize abundant, frequently-updated, and globally available data, such as from satellit… ▽ More Progress toward the United Nations Sustainable Development Goals (SDGs) has been hindered by a lack of data on key environmental and socioeconomic indicators, which historically have come from ground surveys with sparse temporal and spatial coverage. Recent advances in machine learning have made it possible to utilize abundant, frequently-updated, and globally available data, such as from satellites or social media, to provide insights into progress toward SDGs. Despite promising early results, approaches to using such data for SDG measurement thus far have largely evaluated on different datasets or used inconsistent evaluation metrics, making it hard to understand whether performance is improving and where additional research would be most fruitful. Furthermore, processing satellite and ground survey data requires domain knowledge that many in the machine learning community lack. In this paper, we introduce SustainBench, a collection of 15 benchmark tasks across 7 SDGs, including tasks related to economic development, agriculture, health, education, water and sanitation, climate action, and life on land. Datasets for 11 of the 15 tasks are released publicly for the first time. Our goals for SustainBench are to (1) lower the barriers to entry for the machine learning community to contribute to measuring and achieving the SDGs; (2) provide standard benchmarks for evaluating machine learning models on tasks across a variety of SDGs; and (3) encourage the development of novel machine learning methods where improved model performance facilitates progress towards the SDGs. △ Less

Submitted 8 November, 2021; originally announced November 2021.

Comments: NeurIPS 2021 (Track on Datasets and Benchmarks)

Early- and in-season crop type map** without current-year ground truth: generating labels from historical information via a topology-based approach

Authors: Chenxi Lin, Liheng Zhong, Xiao-Peng Song, **wei Dong, David B. Lobell, Zhenong **

Abstract: Land cover classification in remote sensing is often faced with the challenge of limited ground truth. Incorporating historical information has the potential to significantly lower the expensive cost associated with collecting ground truth and, more importantly, enable early- and in-season map** that is helpful to many pre-harvest decisions. In this study, we propose a new approach that can effe… ▽ More Land cover classification in remote sensing is often faced with the challenge of limited ground truth. Incorporating historical information has the potential to significantly lower the expensive cost associated with collecting ground truth and, more importantly, enable early- and in-season map** that is helpful to many pre-harvest decisions. In this study, we propose a new approach that can effectively transfer knowledge about the topology (i.e. relative position) of different crop types in the spectral feature space (e.g. the histogram of SWIR1 vs RDEG1 bands) to generate labels, thereby support crop classification in a different year. Importantly, our approach does not attempt to transfer classification decision boundaries that are susceptible to inter-annual variations of weather and management, but relies on the more robust and shift-invariant topology information. We tested this approach for map** corn/soybeans in the US Midwest and paddy rice/corn/soybeans in Northeast China using Landsat-8 and Sentinel-2 data. Results show that our approach automatically generates high-quality labels for crops in the target year immediately after each image becomes available. Based on these generated labels from our approach, the subsequent crop type map** using a random forest classifier reach the F1 score as high as 0.887 for corn as early as the silking stage and 0.851 for soybean as early as the flowering stage and the overall accuracy of 0.873 in Iowa. In Northeast China, F1 scores of paddy rice, corn and soybeans and the overall accuracy can exceed 0.85 two and half months ahead of harvest. Overall, these results highlight unique advantages of our approach in transferring historical knowledge and maximizing the timeliness of crop maps. Our approach supports a general paradigm shift towards learning transferrable and generalizable knowledge to facilitate land cover classification. △ Less

Submitted 19 October, 2021; originally announced October 2021.

Combining GEDI and Sentinel-2 for wall-to-wall map** of tall and short crops

Authors: Stefania Di Tommaso, Sherrie Wang, David B. Lobell

Abstract: High resolution crop type maps are an important tool for improving food security, and remote sensing is increasingly used to create such maps in regions that possess ground truth labels for model training. However, these labels are absent in many regions, and models trained in other regions on typical satellite features, such as those from optical sensors, often exhibit low performance when transf… ▽ More High resolution crop type maps are an important tool for improving food security, and remote sensing is increasingly used to create such maps in regions that possess ground truth labels for model training. However, these labels are absent in many regions, and models trained in other regions on typical satellite features, such as those from optical sensors, often exhibit low performance when transferred. Here we explore the use of NASA's Global Ecosystem Dynamics Investigation (GEDI) spaceborne lidar instrument, combined with Sentinel-2 optical data, for crop type map**. Using data from three major cropped regions (in China, France, and the United States) we first demonstrate that GEDI energy profiles are capable of reliably distinguishing maize, a crop typically above 2m in height, from crops like rice and soybean that are shorter. We further show that these GEDI profiles provide much more invariant features across geographies compared to spectral and phenological features detected by passive optical sensors. GEDI is able to distinguish maize from other crops within each region with accuracies higher than 84%, and able to transfer across regions with accuracies higher than 82% compared to 64% for transfer of optical features. Finally, we show that GEDI profiles can be used to generate training labels for models based on optical imagery from Sentinel-2, thereby enabling the creation of 10m wall-to-wall maps of tall versus short crops in label-scarce regions. As maize is the second most widely grown crop in the world and often the only tall crop grown within a landscape, we conclude that GEDI offers great promise for improving global crop type maps. △ Less

Submitted 10 September, 2021; originally announced September 2021.

Two Shifts for Crop Map**: Leveraging Aggregate Crop Statistics to Improve Satellite-based Maps in New Regions

Authors: Dan M. Kluger, Sherrie Wang, David B. Lobell

Abstract: Crop type map** at the field level is critical for a variety of applications in agricultural monitoring, and satellite imagery is becoming an increasingly abundant and useful raw input from which to create crop type maps. Still, in many regions crop type map** with satellite data remains constrained by a scarcity of field-level crop labels for training supervised classification models. When tr… ▽ More Crop type map** at the field level is critical for a variety of applications in agricultural monitoring, and satellite imagery is becoming an increasingly abundant and useful raw input from which to create crop type maps. Still, in many regions crop type map** with satellite data remains constrained by a scarcity of field-level crop labels for training supervised classification models. When training data is not available in one region, classifiers trained in similar regions can be transferred, but shifts in the distribution of crop types as well as transformations of the features between regions lead to reduced classification accuracy. We present a methodology that uses aggregate-level crop statistics to correct the classifier by accounting for these two types of shifts. To adjust for shifts in the crop type composition we present a scheme for properly reweighting the posterior probabilities of each class that are output by the classifier. To adjust for shifts in features we propose a method to estimate and remove linear shifts in the mean feature vector. We demonstrate that this methodology leads to substantial improvements in overall classification accuracy when using Linear Discriminant Analysis (LDA) to map crop types in Occitanie, France and in Western Province, Kenya. When using LDA as our base classifier, we found that in France our methodology led to percent reductions in misclassifications ranging from 2.8% to 42.2% (mean = 21.9%) over eleven different training departments, and in Kenya the percent reductions in misclassification were 6.6%, 28.4%, and 42.7% for three training regions. While our methodology was statistically motivated by the LDA classifier, it can be applied to any type of classifier. As an example, we demonstrate its successful application to improve a Random Forest classifier. △ Less

Submitted 2 September, 2021; originally announced September 2021.

Comments: This is the revised version of the paper which was submitted to Remote Sensing of Environment on May 3, 2021, immediately prior to acceptance for publication

MSC Class: 62P12 (primary) 62H30 ACM Class: I.4.m; I.m; J.2

Journal ref: Remote Sensing of Environment (2021). Volume 262

Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis

Authors: Yutong He, Dingjie Wang, Nicholas Lai, William Zhang, Chenlin Meng, Marshall Burke, David B. Lobell, Stefano Ermon

Abstract: High-resolution satellite imagery has proven useful for a broad range of tasks, including measurement of global human population, local economic livelihoods, and biodiversity, among many others. Unfortunately, high-resolution imagery is both infrequently collected and expensive to purchase, making it hard to efficiently and effectively scale these downstream tasks over both time and space. We prop… ▽ More High-resolution satellite imagery has proven useful for a broad range of tasks, including measurement of global human population, local economic livelihoods, and biodiversity, among many others. Unfortunately, high-resolution imagery is both infrequently collected and expensive to purchase, making it hard to efficiently and effectively scale these downstream tasks over both time and space. We propose a new conditional pixel synthesis model that uses abundant, low-cost, low-resolution imagery to generate accurate high-resolution imagery at locations and times in which it is unavailable. We show that our model attains photo-realistic sample quality and outperforms competing baselines on a key downstream task -- object counting -- particularly in geographic locations where conditions on the ground are changing rapidly. △ Less

Submitted 4 April, 2022; v1 submitted 21 June, 2021; originally announced June 2021.

Journal ref: Advances in Neural Information Processing Systems 35 (2021) 27903-27915

On the benefits of index insurance in US agriculture: a large-scale analysis using satellite data

Authors: Matthieu Stigler, David Lobell

Abstract: Index insurance has been promoted as a promising solution for reducing agricultural risk compared to traditional farm-based insurance. By linking payouts to a regional factor instead of individual loss, index insurance reduces monitoring costs, and alleviates the problems of moral hazard and adverse selection. Despite its theoretical appeal, demand for index insurance has remained low in many deve… ▽ More Index insurance has been promoted as a promising solution for reducing agricultural risk compared to traditional farm-based insurance. By linking payouts to a regional factor instead of individual loss, index insurance reduces monitoring costs, and alleviates the problems of moral hazard and adverse selection. Despite its theoretical appeal, demand for index insurance has remained low in many develo** countries, triggering a debate on the causes of the low uptake. Surprisingly, there has been little discussion in this debate about the experience in the United States. The US is an unique case as both farm-based and index-based products have been available for more than two decades. Furthermore, the number of insurance zones is very large, allowing interesting comparisons over space. As in develo** countries, the adoption of index insurance is rather low -- less than than 5\% of insured acreage. Does this mean that we should give up on index insurance? In this paper, we investigate the low take-up of index insurance in the US leveraging a field-level dataset for corn and soybean obtained from satellite predictions. While previous studies were based either on county aggregates or on relatively small farm-level dataset, our satellite-derived data gives us a very large number of fields (close to 1.8 million) comprised within a large number of index zones (600) observed over 20 years. To evaluate the suitability of index insurance, we run a large-scale simulation comparing the benefits of both insurance schemes using a new measure of farm-equivalent risk coverage of index insurance. We make two main contributions. First, we show that in our simulations, demand for index insurance is unexpectedly high, at about 30\% to 40\% of total demand. This result is robust to relaxing several assumptions of the model and to using prospect theory instead of expected utility. △ Less

Submitted 7 November, 2021; v1 submitted 25 November, 2020; originally announced November 2020.

Geography-Aware Self-Supervised Learning

Authors: Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive a… ▽ More Contrastive learning methods have significantly narrowed the gap between supervised and unsupervised learning on computer vision tasks. In this paper, we explore their application to geo-located datasets, e.g. remote sensing, where unlabeled data is often abundant but labeled data is scarce. We first show that due to their different characteristics, a non-trivial gap persists between contrastive and supervised learning on standard benchmarks. To close the gap, we propose novel training methods that exploit the spatio-temporal structure of remote sensing data. We leverage spatially aligned images over time to construct temporal positive pairs in contrastive learning and geo-location to design pre-text tasks. Our experiments show that our proposed method closes the gap between contrastive and supervised learning on image classification, object detection and semantic segmentation for remote sensing. Moreover, we demonstrate that the proposed method can also be applied to geo-tagged ImageNet images, improving downstream performance on various tasks. Project Webpage can be found at this link geography-aware-ssl.github.io. △ Less

Submitted 8 March, 2022; v1 submitted 19 November, 2020; originally announced November 2020.

Comments: Accepted at ICCV 2021

Using satellite imagery to understand and promote sustainable development

Authors: Marshall Burke, Anne Driscoll, David B. Lobell, Stefano Ermon

Abstract: Accurate and comprehensive measurements of a range of sustainable development outcomes are fundamental inputs into both research and policy. We synthesize the growing literature that uses satellite imagery to understand these outcomes, with a focus on approaches that combine imagery with machine learning. We quantify the paucity of ground data on key human-related outcomes and the growing abundanc… ▽ More Accurate and comprehensive measurements of a range of sustainable development outcomes are fundamental inputs into both research and policy. We synthesize the growing literature that uses satellite imagery to understand these outcomes, with a focus on approaches that combine imagery with machine learning. We quantify the paucity of ground data on key human-related outcomes and the growing abundance and resolution (spatial, temporal, and spectral) of satellite imagery. We then review recent machine learning approaches to model-building in the context of scarce and noisy training data, highlighting how this noise often leads to incorrect assessment of models' predictive performance. We quantify recent model performance across multiple sustainable development domains, discuss research and policy applications, explore constraints to future progress, and highlight key research directions for the field. △ Less

Submitted 23 September, 2020; originally announced October 2020.

The Historical Impact of Anthropogenic Climate Change on Global Agricultural Productivity

Authors: Ariel Ortiz-Bobea, Toby R. Ault, Carlos M. Carrillo, Robert G. Chambers, David B. Lobell

Abstract: Agricultural research has fostered productivity growth, but the historical influence of anthropogenic climate change on that growth has not been quantified. We develop a robust econometric model of weather effects on global agricultural total factor productivity (TFP) and combine this model with counterfactual climate scenarios to evaluate impacts of past climate trends on TFP. Our baseline model… ▽ More Agricultural research has fostered productivity growth, but the historical influence of anthropogenic climate change on that growth has not been quantified. We develop a robust econometric model of weather effects on global agricultural total factor productivity (TFP) and combine this model with counterfactual climate scenarios to evaluate impacts of past climate trends on TFP. Our baseline model indicates that anthropogenic climate change has reduced global agricultural TFP by about 21% since 1961, a slowdown that is equivalent to losing the last 9 years of productivity growth. The effect is substantially more severe (a reduction of ~30-33%) in warmer regions such as Africa and Latin America and the Caribbean. We also find that global agriculture has grown more vulnerable to ongoing climate change. △ Less

Submitted 2 April, 2021; v1 submitted 20 July, 2020; originally announced July 2020.

Comments: paper (14 pages; 5 figures) and supplemental materials (24 pages; 14 figures and 4 tables)

Journal ref: Nat. Clim. Chang. 11, 306-312 (2021)

Predicting Livelihood Indicators from Community-Generated Street-Level Imagery

Authors: Jihyeon Lee, Dylan Grosz, Burak Uzkent, Sicheng Zeng, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Major decisions from governments and other large organizations rely on measurements of the populace's well-being, but making such measurements at a broad scale is expensive and thus infrequent in much of the develo** world. We propose an inexpensive, scalable, and interpretable approach to predict key livelihood indicators from public crowd-sourced street-level imagery. Such imagery can be cheap… ▽ More Major decisions from governments and other large organizations rely on measurements of the populace's well-being, but making such measurements at a broad scale is expensive and thus infrequent in much of the develo** world. We propose an inexpensive, scalable, and interpretable approach to predict key livelihood indicators from public crowd-sourced street-level imagery. Such imagery can be cheaply collected and more frequently updated compared to traditional surveying methods, while containing plausibly relevant information for a range of livelihood indicators. We propose two approaches to learn from the street-level imagery: (1) a method that creates multi-household cluster representations by detecting informative objects and (2) a graph-based approach that captures the relationships between images. By visualizing what features are important to a model and how they are used, we can help end-user organizations understand the models and offer an alternate approach for index estimation that uses cheaply obtained roadway features. By comparing our results against ground data collected in nationally-representative household surveys, we demonstrate the performance of our approach in accurately predicting indicators of poverty, population, and health and its scalability by testing in two different countries, India and Kenya. Our code is available at https://github.com/sustainlab-group/mapillarygcn. △ Less

Submitted 26 February, 2021; v1 submitted 15 June, 2020; originally announced June 2020.

Comments: Accepted to AAAI 2021. Code: https://github.com/sustainlab-group/mapillarygcn

ACM Class: I.2; I.4; K.4; E.1

Efficient Poverty Map** using Deep Reinforcement Learning

Authors: Kumar Ayush, Burak Uzkent, Kumar Tanmay, Marshall Burke, David Lobell, Stefano Ermon

Abstract: The combination of high-resolution satellite imagery and machine learning have proven useful in many sustainability-related tasks, including poverty prediction, infrastructure measurement, and forest monitoring. However, the accuracy afforded by high-resolution imagery comes at a cost, as such imagery is extremely expensive to purchase at scale. This creates a substantial hurdle to the efficient s… ▽ More The combination of high-resolution satellite imagery and machine learning have proven useful in many sustainability-related tasks, including poverty prediction, infrastructure measurement, and forest monitoring. However, the accuracy afforded by high-resolution imagery comes at a cost, as such imagery is extremely expensive to purchase at scale. This creates a substantial hurdle to the efficient scaling and widespread adoption of high-resolution-based approaches. To reduce acquisition costs while maintaining accuracy, we propose a reinforcement learning approach in which free low-resolution imagery is used to dynamically identify where to acquire costly high-resolution images, prior to performing a deep learning task on the high-resolution images. We apply this approach to the task of poverty prediction in Uganda, building on an earlier approach that used object detection to count objects and use these counts to predict poverty. Our approach exceeds previous performance benchmarks on this task while using 80% fewer high-resolution images. Our approach could have application in many sustainability domains that require high-resolution imagery. △ Less

Submitted 5 January, 2021; v1 submitted 7 June, 2020; originally announced June 2020.

Comments: Accepted at AAAI 2021

Meta-Learning for Few-Shot Land Cover Classification

Authors: Marc Rußwurm, Sherrie Wang, Marco Körner, David Lobell

Abstract: The representations of the Earth's surface vary from one geographic region to another. For instance, the appearance of urban areas differs between continents, and seasonality influences the appearance of vegetation. To capture the diversity within a single category, like as urban or vegetation, requires a large model capacity and, consequently, large datasets. In this work, we propose a different… ▽ More The representations of the Earth's surface vary from one geographic region to another. For instance, the appearance of urban areas differs between continents, and seasonality influences the appearance of vegetation. To capture the diversity within a single category, like as urban or vegetation, requires a large model capacity and, consequently, large datasets. In this work, we propose a different perspective and view this diversity as an inductive transfer learning problem where few data samples from one region allow a model to adapt to an unseen region. We evaluate the model-agnostic meta-learning (MAML) algorithm on classification and segmentation tasks using globally and regionally distributed datasets. We find that few-shot model adaptation outperforms pre-training with regular gradient descent and fine-tuning on (1) the Sen12MS dataset and (2) DeepGlobe data when the source domain and target domain differ. This indicates that model optimization with meta-learning may benefit tasks in the Earth sciences whose data show a high degree of diversity from region to region, while traditional gradient-based supervised learning remains suitable in the absence of a feature or label shift. △ Less

Submitted 28 April, 2020; originally announced April 2020.

Comments: accepted to the CVPR 2020 EarthVision Workshop

Farmland Parcel Delineation Using Spatio-temporal Convolutional Networks

Authors: Han Lin Aung, Burak Uzkent, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Farm parcel delineation provides cadastral data that is important in develo** and managing climate change policies. Specifically, farm parcel delineation informs applications in downstream governmental policies of land allocation, irrigation, fertilization, green-house gases (GHG's), etc. This data can also be useful for the agricultural insurance sector for assessing compensations following dam… ▽ More Farm parcel delineation provides cadastral data that is important in develo** and managing climate change policies. Specifically, farm parcel delineation informs applications in downstream governmental policies of land allocation, irrigation, fertilization, green-house gases (GHG's), etc. This data can also be useful for the agricultural insurance sector for assessing compensations following damages associated with extreme weather events - a growing trend related to climate change. Using satellite imaging can be a scalable and cost effective manner to perform the task of farm parcel delineation to collect this valuable data. In this paper, we break down this task using satellite imaging into two approaches: 1) Segmentation of parcel boundaries, and 2) Segmentation of parcel areas. We implemented variations of UNets, one of which takes into account temporal information, which achieved the best results on our dataset on farmland parcels in France in 2017. △ Less

Submitted 20 April, 2020; v1 submitted 11 April, 2020; originally announced April 2020.

Generating Interpretable Poverty Maps using Object Detection in Satellite Images

Authors: Kumar Ayush, Burak Uzkent, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Accurate local-level poverty measurement is an essential task for governments and humanitarian organizations to track the progress towards improving livelihoods and distribute scarce resources. Recent computer vision advances in using satellite imagery to predict poverty have shown increasing accuracy, but they do not generate features that are interpretable to policymakers, inhibiting adoption by… ▽ More Accurate local-level poverty measurement is an essential task for governments and humanitarian organizations to track the progress towards improving livelihoods and distribute scarce resources. Recent computer vision advances in using satellite imagery to predict poverty have shown increasing accuracy, but they do not generate features that are interpretable to policymakers, inhibiting adoption by practitioners. Here we demonstrate an interpretable computational framework to accurately predict poverty at a local level by applying object detectors to high resolution (30cm) satellite images. Using the weighted counts of objects as features, we achieve 0.539 Pearson's r^2 in predicting village-level poverty in Uganda, a 31% improvement over existing (and less interpretable) benchmarks. Feature importance and ablation analysis reveal intuitive relationships between object counts and poverty predictions. Our results suggest that interpretability does not have to come at the cost of performance, at least in this important domain. △ Less

Submitted 17 February, 2020; v1 submitted 4 February, 2020; originally announced February 2020.

Learning to Interpret Satellite Images in Global Scale Using Wikipedia

Authors: Burak Uzkent, Evan Sheehan, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Despite recent progress in computer vision, finegrained interpretation of satellite images remains challenging because of a lack of labeled training data. To overcome this limitation, we construct a novel dataset called WikiSatNet by pairing georeferenced Wikipedia articles with satellite imagery of their corresponding locations. We then propose two strategies to learn representations of satellite… ▽ More Despite recent progress in computer vision, finegrained interpretation of satellite images remains challenging because of a lack of labeled training data. To overcome this limitation, we construct a novel dataset called WikiSatNet by pairing georeferenced Wikipedia articles with satellite imagery of their corresponding locations. We then propose two strategies to learn representations of satellite images by predicting properties of the corresponding articles from the images. Leveraging this new multi-modal dataset, we can drastically reduce the quantity of human-annotated labels and time required for downstream tasks. On the recently released fMoW dataset, our pre-training strategies can boost the performance of a model pre-trained on ImageNet by up to 4:5% in F1 score. △ Less

Submitted 11 August, 2019; v1 submitted 7 May, 2019; originally announced May 2019.

Comments: Accepted to IJCAI 2019

Map** Missing Population in Rural India: A Deep Learning Approach with Satellite Imagery

Authors: Wenjie Hu, Jay Harshadbhai Patel, Zoe-Alanah Robert, Paul Novosad, Samuel Asher, Zhongyi Tang, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Millions of people worldwide are absent from their country's census. Accurate, current, and granular population metrics are critical to improving government allocation of resources, to measuring disease control, to responding to natural disasters, and to studying any aspect of human life in these communities. Satellite imagery can provide sufficient information to build a population map without th… ▽ More Millions of people worldwide are absent from their country's census. Accurate, current, and granular population metrics are critical to improving government allocation of resources, to measuring disease control, to responding to natural disasters, and to studying any aspect of human life in these communities. Satellite imagery can provide sufficient information to build a population map without the cost and time of a government census. We present two Convolutional Neural Network (CNN) architectures which efficiently and effectively combine satellite imagery inputs from multiple sources to accurately predict the population density of a region. In this paper, we use satellite imagery from rural villages in India and population labels from the 2011 SECC census. Our best model achieves better performance than previous papers as well as LandScan, a community standard for global population distribution. △ Less

Submitted 4 May, 2019; originally announced May 2019.

Comments: 7 pages

ACM Class: I.2.10; I.2.6; J.2; J.4

Journal ref: AAAI/ACM Conference on AI, Ethics, and Society (AIES '19), January 27-28, 2019, Honolulu, HI, USA

Predicting Economic Development using Geolocated Wikipedia Articles

Authors: Evan Sheehan, Chenlin Meng, Matthew Tan, Burak Uzkent, Neal Jean, David Lobell, Marshall Burke, Stefano Ermon

Abstract: Progress on the UN Sustainable Development Goals (SDGs) is hampered by a persistent lack of data regarding key social, environmental, and economic indicators, particularly in develo** countries. For example, data on poverty --- the first of seventeen SDGs --- is both spatially sparse and infrequently collected in Sub-Saharan Africa due to the high cost of surveys. Here we propose a novel method… ▽ More Progress on the UN Sustainable Development Goals (SDGs) is hampered by a persistent lack of data regarding key social, environmental, and economic indicators, particularly in develo** countries. For example, data on poverty --- the first of seventeen SDGs --- is both spatially sparse and infrequently collected in Sub-Saharan Africa due to the high cost of surveys. Here we propose a novel method for estimating socioeconomic indicators using open-source, geolocated textual information from Wikipedia articles. We demonstrate that modern NLP techniques can be used to predict community-level asset wealth and education outcomes using nearby geolocated Wikipedia articles. When paired with nightlights satellite imagery, our method outperforms all previously published benchmarks for this prediction task, indicating the potential of Wikipedia to inform both research in the social sciences and future policy decisions. △ Less

Submitted 11 May, 2019; v1 submitted 5 May, 2019; originally announced May 2019.

Comments: Accepted to KDD 2019

Semi-Supervised Multitask Learning on Multispectral Satellite Images Using Wasserstein Generative Adversarial Networks (GANs) for Predicting Poverty

Authors: Anthony Perez, Swetava Ganguli, Stefano Ermon, George Azzari, Marshall Burke, David Lobell

Abstract: Obtaining reliable data describing local poverty metrics at a granularity that is informative to policy-makers requires expensive and logistically difficult surveys, particularly in the develo** world. Not surprisingly, the poverty stricken regions are also the ones which have a high probability of being a war zone, have poor infrastructure and sometimes have governments that do not cooperate wi… ▽ More Obtaining reliable data describing local poverty metrics at a granularity that is informative to policy-makers requires expensive and logistically difficult surveys, particularly in the develo** world. Not surprisingly, the poverty stricken regions are also the ones which have a high probability of being a war zone, have poor infrastructure and sometimes have governments that do not cooperate with internationally funded development efforts. We train a CNN on free and publicly available daytime satellite images of the African continent from Landsat 7 to build a model for predicting local economic livelihoods. Only 5% of the satellite images can be associated with labels (which are obtained from DHS Surveys) and thus a semi-supervised approach using a GAN (similar to the approach of Salimans, et al. (2016)), albeit with a more stable-to-train flavor of GANs called the Wasserstein GAN regularized with gradient penalty(Gulrajani, et al. (2017)) is used. The method of multitask learning is employed to regularize the network and also create an end-to-end model for the prediction of multiple poverty metrics. △ Less

Submitted 25 April, 2019; v1 submitted 13 February, 2019; originally announced February 2019.

Comments: This project was recognized as the best two-person project during the Spring 2017 offering of CS 231N Convolutional Neural Networks for Visual Recognition. Second revised version corrects typographical errors and adds a few additional references

Report number: Final report of research project conducted by the authors as part of the Sustainability and Artificial Intelligence Laboratory (SAIL) at Stanford University and as part of the Spring 2017 offering of CS 231N

Learning to Interpret Satellite Images Using Wikipedia

Authors: Evan Sheehan, Burak Uzkent, Chenlin Meng, Zhongyi Tang, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Despite recent progress in computer vision, fine-grained interpretation of satellite images remains challenging because of a lack of labeled training data. To overcome this limitation, we propose using Wikipedia as a previously untapped source of rich, georeferenced textual information with global coverage. We construct a novel large-scale, multi-modal dataset by pairing geo-referenced Wikipedia a… ▽ More Despite recent progress in computer vision, fine-grained interpretation of satellite images remains challenging because of a lack of labeled training data. To overcome this limitation, we propose using Wikipedia as a previously untapped source of rich, georeferenced textual information with global coverage. We construct a novel large-scale, multi-modal dataset by pairing geo-referenced Wikipedia articles with satellite imagery of their corresponding locations. To prove the efficacy of this dataset, we focus on the African continent and train a deep network to classify images based on labels extracted from articles. We then fine-tune the model on a human annotated dataset and demonstrate that this weak form of supervision can drastically reduce the quantity of human annotated labels and time required for downstream tasks. △ Less

Submitted 19 September, 2018; originally announced September 2018.

Infrastructure Quality Assessment in Africa using Satellite Imagery and Deep Learning

Authors: Barak Oshri, Annie Hu, Peter Adelson, Xiao Chen, Pascaline Dupas, Jeremy Weinstein, Marshall Burke, David Lobell, Stefano Ermon

Abstract: The UN Sustainable Development Goals allude to the importance of infrastructure quality in three of its seventeen goals. However, monitoring infrastructure quality in develo** regions remains prohibitively expensive and impedes efforts to measure progress toward these goals. To this end, we investigate the use of widely available remote sensing data for the prediction of infrastructure quality i… ▽ More The UN Sustainable Development Goals allude to the importance of infrastructure quality in three of its seventeen goals. However, monitoring infrastructure quality in develo** regions remains prohibitively expensive and impedes efforts to measure progress toward these goals. To this end, we investigate the use of widely available remote sensing data for the prediction of infrastructure quality in Africa. We train a convolutional neural network to predict ground truth labels from the Afrobarometer Round 6 survey using Landsat 8 and Sentinel 1 satellite imagery. Our best models predict infrastructure quality with AUROC scores of 0.881 on Electricity, 0.862 on Sewerage, 0.739 on Piped Water, and 0.786 on Roads using Landsat 8. These performances are significantly better than models that leverage OpenStreetMap or nighttime light intensity on the same tasks. We also demonstrate that our trained model can accurately make predictions in an unseen country after fine-tuning on a small sample of images. Furthermore, the model can be deployed in regions with limited samples to predict infrastructure outcomes with higher performance than nearest neighbor spatial interpolation. △ Less

Submitted 3 June, 2018; originally announced June 2018.

Journal ref: KDD 2018 Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Tile2Vec: Unsupervised representation learning for spatially distributed data

Authors: Neal Jean, Sherrie Wang, Anshul Samar, George Azzari, David Lobell, Stefano Ermon

Abstract: Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distributional hypothesis from natural language -- words appearing in similar contexts tend… ▽ More Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distributional hypothesis from natural language -- words appearing in similar contexts tend to have similar meanings -- to spatially distributed data. We demonstrate empirically that Tile2Vec learns semantically meaningful representations on three datasets. Our learned representations significantly improve performance in downstream classification tasks and, similar to word vectors, visual analogies can be obtained via simple arithmetic in the latent space. △ Less

Submitted 30 May, 2018; v1 submitted 8 May, 2018; originally announced May 2018.

Comments: 8 pages, 4 figures in main text; 9 pages, 11 figures in appendix

Hierarchical Modeling of Seed Variety Yields and Decision Making for Future Planting Plans

Authors: Huaiyang Zhong, Xiaocheng Li, David Lobell, Stefano Ermon, Margaret L. Brandeau

Abstract: Eradicating hunger and malnutrition is a key development goal of the 21st century. We address the problem of optimally identifying seed varieties to reliably increase crop yield within a risk-sensitive decision-making framework. Specifically, we introduce a novel hierarchical machine learning mechanism for predicting crop yield (the yield of different seed varieties of the same crop). We integrate… ▽ More Eradicating hunger and malnutrition is a key development goal of the 21st century. We address the problem of optimally identifying seed varieties to reliably increase crop yield within a risk-sensitive decision-making framework. Specifically, we introduce a novel hierarchical machine learning mechanism for predicting crop yield (the yield of different seed varieties of the same crop). We integrate this prediction mechanism with a weather forecasting model, and propose three different approaches for decision making under uncertainty to select seed varieties for planting so as to balance yield maximization and risk.We apply our model to the problem of soybean variety selection given in the 2016 Syngenta Crop Challenge. Our prediction model achieves a median absolute error of 3.74 bushels per acre and thus provides good estimates for input into the decision models.Our decision models identify the selection of soybean varieties that appropriately balance yield and risk as a function of the farmer's risk aversion level. More generally, our models support farmers in decision making about which seed varieties to plant. △ Less

Submitted 15 November, 2017; originally announced November 2017.

Poverty Prediction with Public Landsat 7 Satellite Imagery and Machine Learning

Authors: Anthony Perez, Christopher Yeh, George Azzari, Marshall Burke, David Lobell, Stefano Ermon

Abstract: Obtaining detailed and reliable data about local economic livelihoods in develo** countries is expensive, and data are consequently scarce. Previous work has shown that it is possible to measure local-level economic livelihoods using high-resolution satellite imagery. However, such imagery is relatively expensive to acquire, often not updated frequently, and is mainly available for recent years.… ▽ More Obtaining detailed and reliable data about local economic livelihoods in develo** countries is expensive, and data are consequently scarce. Previous work has shown that it is possible to measure local-level economic livelihoods using high-resolution satellite imagery. However, such imagery is relatively expensive to acquire, often not updated frequently, and is mainly available for recent years. We train CNN models on free and publicly available multispectral daytime satellite images of the African continent from the Landsat 7 satellite, which has collected imagery with global coverage for almost two decades. We show that despite these images' lower resolution, we can achieve accuracies that exceed previous benchmarks. △ Less

Submitted 9 November, 2017; originally announced November 2017.

Comments: Presented at NIPS 2017 Workshop on Machine Learning for the Develo** World

Transfer Learning from Deep Features for Remote Sensing and Poverty Map**

Authors: Michael Xie, Neal Jean, Marshall Burke, David Lobell, Stefano Ermon

Abstract: The lack of reliable data in develo** countries is a major obstacle to sustainable development, food security, and disaster relief. Poverty data, for example, is typically scarce, sparse in coverage, and labor-intensive to obtain. Remote sensing data such as high-resolution satellite imagery, on the other hand, is becoming increasingly available and inexpensive. Unfortunately, such data is highl… ▽ More The lack of reliable data in develo** countries is a major obstacle to sustainable development, food security, and disaster relief. Poverty data, for example, is typically scarce, sparse in coverage, and labor-intensive to obtain. Remote sensing data such as high-resolution satellite imagery, on the other hand, is becoming increasingly available and inexpensive. Unfortunately, such data is highly unstructured and currently no techniques exist to automatically extract useful insights to inform policy decisions and help direct humanitarian efforts. We propose a novel machine learning approach to extract large-scale socioeconomic indicators from high-resolution satellite imagery. The main challenge is that training data is very scarce, making it difficult to apply modern techniques such as Convolutional Neural Networks (CNN). We therefore propose a transfer learning approach where nighttime light intensities are used as a data-rich proxy. We train a fully convolutional CNN model to predict nighttime lights from daytime imagery, simultaneously learning features that are useful for poverty prediction. The model learns filters identifying different terrains and man-made structures, including roads, buildings, and farmlands, without any supervision beyond nighttime lights. We demonstrate that these learned features are highly informative for poverty map**, even approaching the predictive performance of survey data collected in the field. △ Less

Submitted 27 February, 2016; v1 submitted 30 September, 2015; originally announced October 2015.

Comments: In Proc. 30th AAAI Conference on Artificial Intelligence