Showing 1–12 of 12 results for author: Bosch, M

Regional biases in image geolocation estimation: a case study with the SenseCity Africa dataset

Authors: Ximena Salgado Uribe, Martí Bosch, Jérôme Chenal

Abstract: Advances in Artificial Intelligence are challenged by the biases rooted in the datasets used to train the models. In image geolocation estimation, models are mostly trained using data from specific geographic regions, notably the Western world, and as a result, they may struggle to comprehend the complexities of underrepresented regions. To assess this issue, we apply a state-of-the-art image geol… ▽ More Advances in Artificial Intelligence are challenged by the biases rooted in the datasets used to train the models. In image geolocation estimation, models are mostly trained using data from specific geographic regions, notably the Western world, and as a result, they may struggle to comprehend the complexities of underrepresented regions. To assess this issue, we apply a state-of-the-art image geolocation estimation model (ISNs) to a crowd-sourced dataset of geolocated images from the African continent (SCA100), and then explore the regional and socioeconomic biases underlying the model's predictions. Our findings show that the ISNs model tends to over-predict image locations in high-income countries of the Western world, which is consistent with the geographic distribution of its training data, i.e., the IM2GPS3k dataset. Accordingly, when compared to the IM2GPS3k benchmark, the accuracy of the ISNs model notably decreases at all scales. Additionally, we cluster images of the SCA100 dataset based on how accurately they are predicted by the ISNs model and show the model's difficulties in correctly predicting the locations of images in low income regions, especially in Sub-Saharan Africa. Therefore, our results suggest that using IM2GPS3k as a training set and benchmark for image geolocation estimation and other computer vision models overlooks its potential application in the African context. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Comments: 13 pages, 5 figures

Scaling on-chip photonic neural processors using arbitrarily programmable wave propagation

Authors: Tatsuhiro Onodera, Martin M. Stein, Benjamin A. Ash, Mandar M. Sohoni, Melissa Bosch, Ryotatsu Yanagimoto, Marc Jankowski, Timothy P. McKenna, Tianyu Wang, Gennady Shvets, Maxim R. Shcherbakov, Logan G. Wright, Peter L. McMahon

Abstract: On-chip photonic processors for neural networks have potential benefits in both speed and energy efficiency but have not yet reached the scale at which they can outperform electronic processors. The dominant paradigm for designing on-chip photonics is to make networks of relatively bulky discrete components connected by one-dimensional waveguides. A far more compact alternative is to avoid explici… ▽ More On-chip photonic processors for neural networks have potential benefits in both speed and energy efficiency but have not yet reached the scale at which they can outperform electronic processors. The dominant paradigm for designing on-chip photonics is to make networks of relatively bulky discrete components connected by one-dimensional waveguides. A far more compact alternative is to avoid explicitly defining any components and instead sculpt the continuous substrate of the photonic processor to directly perform the computation using waves freely propagating in two dimensions. We propose and demonstrate a device whose refractive index as a function of space, $n(x,z)$, can be rapidly reprogrammed, allowing arbitrary control over the wave propagation in the device. Our device, a 2D-programmable waveguide, combines photoconductive gain with the electro-optic effect to achieve massively parallel modulation of the refractive index of a slab waveguide, with an index modulation depth of $10^{-3}$ and approximately $10^4$ programmable degrees of freedom. We used a prototype device with a functional area of $12\,\text{mm}^2$ to perform neural-network inference with up to 49-dimensional input vectors in a single pass, achieving 96% accuracy on vowel classification and 86% accuracy on $7 \times 7$-pixel MNIST handwritten-digit classification. This is a scale beyond that of previous photonic chips relying on discrete components, illustrating the benefit of the continuous-waves paradigm. In principle, with large enough chip area, the reprogrammability of the device's refractive index distribution enables the reconfigurable realization of any passive, linear photonic circuit or device. This promises the development of more compact and versatile photonic systems for a wide range of applications, including optical processing, smart sensing, spectroscopy, and optical communications. △ Less

Submitted 27 February, 2024; originally announced February 2024.

Deep Learning for Robust Motion Segmentation with Non-Static Cameras

Authors: Markus Bosch

Abstract: This work proposes a new end-to-end DCNN based approach for motion segmentation, especially for video sequences captured with such non-static cameras, called MOSNET. While other approaches focus on spatial or temporal context only, the proposed approach uses 3D convolutions as a key technology to factor in, spatio-temporal features in cohesive video frames. This is done by capturing temporal infor… ▽ More This work proposes a new end-to-end DCNN based approach for motion segmentation, especially for video sequences captured with such non-static cameras, called MOSNET. While other approaches focus on spatial or temporal context only, the proposed approach uses 3D convolutions as a key technology to factor in, spatio-temporal features in cohesive video frames. This is done by capturing temporal information in features with a low and also with a high level of abstraction. The lean network architecture with about 21k trainable parameters is mainly based on a pre-trained VGG-16 network. The MOSNET uses a new feature map fusion technique, which enables the network to focus on the appropriate level of abstraction, resolution, and the appropriate size of the receptive field regarding the input. Furthermore, the end-to-end deep learning based approach can be extended by feature based image alignment as a pre-processing step, which brings a gain in performance for some scenes. Evaluating the end-to-end deep learning based MOSNET network in a scene independent manner leads to an overall F-measure of 0.803 on the CDNet2014 dataset. A small temporal window of five input frames, without the need of any initialization is used to obtain this result. Therefore the network is able to perform well on scenes captured with non-static cameras where the image content changes significantly during the scene. In order to get robust results in scenes captured with a moving camera, feature based image alignment can implemented as pre-processing step. The MOSNET combined with pre-processing leads to an F-measure of 0.685 when cross-evaluating with a relabeled LASIESTA dataset, which underpins the capability generalise of the MOSNET. △ Less

Submitted 22 February, 2021; originally announced February 2021.

Improving Emergency Response during Hurricane Season using Computer Vision

Authors: Marc Bosch, Christian Conroy, Benjamin Ortiz, Philip Bogden

Abstract: We have developed a framework for crisis response and management that incorporates the latest technologies in computer vision (CV), inland flood prediction, damage assessment and data visualization. The framework uses data collected before, during, and after the crisis to enable rapid and informed decision making during all phases of disaster response. Our computer-vision model analyzes spaceborne… ▽ More We have developed a framework for crisis response and management that incorporates the latest technologies in computer vision (CV), inland flood prediction, damage assessment and data visualization. The framework uses data collected before, during, and after the crisis to enable rapid and informed decision making during all phases of disaster response. Our computer-vision model analyzes spaceborne and airborne imagery to detect relevant features during and after a natural disaster and creates metadata that is transformed into actionable information through web-accessible map** tools. In particular, we have designed an ensemble of models to identify features including water, roads, buildings, and vegetation from the imagery. We have investigated techniques to bootstrap and reduce dependency on large data annotation efforts by adding use of open source labels including OpenStreetMaps and adding complementary data sources including Height Above Nearest Drainage (HAND) as a side channel to the network's input to encourage it to learn other features orthogonal to visual characteristics. Modeling efforts include modification of connected U-Nets for (1) semantic segmentation, (2) flood line detection, and (3) for damage assessment. In particular for the case of damage assessment, we added a second encoder to U-Net so that it could learn pre-event and post-event image features simultaneously. Through this method, the network is able to learn the difference between the pre- and post-disaster images, and therefore more effectively classify the level of damage. We have validated our approaches using publicly available data from the National Oceanic and Atmospheric Administration (NOAA)'s Remote Sensing Division, which displays the city and street-level details as mosaic tile images as well as data released as part of the Xview2 challenge. △ Less

Submitted 8 September, 2020; v1 submitted 17 August, 2020; originally announced August 2020.

Improving Community Resiliency and Emergency Response With Artificial Intelligence

Authors: Ben Ortiz, Laura Kahn, Marc Bosch, Philip Bogden, Viveca Pavon-Harr, Onur Savas, Ian McCulloh

Abstract: New crisis response and management approaches that incorporate the latest information technologies are essential in all phases of emergency preparedness and response, including the planning, response, recovery, and assessment phases. Accurate and timely information is as crucial as is rapid and coherent coordination among the responding organizations. We are working towards a multipronged emergenc… ▽ More New crisis response and management approaches that incorporate the latest information technologies are essential in all phases of emergency preparedness and response, including the planning, response, recovery, and assessment phases. Accurate and timely information is as crucial as is rapid and coherent coordination among the responding organizations. We are working towards a multipronged emergency response tool that provide stakeholders timely access to comprehensive, relevant, and reliable information. The faster emergency personnel are able to analyze, disseminate and act on key information, the more effective and timelier their response will be and the greater the benefit to affected populations. Our tool consists of encoding multiple layers of open source geospatial data including flood risk location, road network strength, inundation maps that proxy inland flooding and computer vision semantic segmentation for estimating flooded areas and damaged infrastructure. These data layers are combined and used as input data for machine learning algorithms such as finding the best evacuation routes before, during and after an emergency or providing a list of available lodging for first responders in an impacted area for first. Even though our system could be used in a number of use cases where people are forced from one location to another, we demonstrate the feasibility of our system for the use case of Hurricane Florence in Lumberton, North Carolina. △ Less

Submitted 28 May, 2020; originally announced May 2020.

A Common Operating Picture Framework Leveraging Data Fusion and Deep Learning

Authors: Benjamin Ortiz, David Lindenbaum, Joseph Nassar, Brendan Lammers, John Wahl, Robert Mangum, Margaret Smith, Marc Bosch

Abstract: Organizations are starting to realize of the combined power of data and data-driven algorithmic models to gain insights, situational awareness, and advance their mission. A common challenge to gaining insights is connecting inherently different datasets. These datasets (e.g. geocoded features, video streams, raw text, social network data, etc.) per separate they provide very narrow answers; howeve… ▽ More Organizations are starting to realize of the combined power of data and data-driven algorithmic models to gain insights, situational awareness, and advance their mission. A common challenge to gaining insights is connecting inherently different datasets. These datasets (e.g. geocoded features, video streams, raw text, social network data, etc.) per separate they provide very narrow answers; however collectively they can provide new capabilities. In this work, we present a data fusion framework for accelerating solutions for Processing, Exploitation, and Dissemination (PED). Our platform is a collection of services that extract information from several data sources (per separate) by leveraging deep learning and other means of processing. This information is fused by a set of analytical engines that perform data correlations, searches, and other modeling operations to combine information from the disparate data sources. As a result, events of interest are detected, geolocated, logged, and presented into a common operating picture. This common operating picture allows the user to visualize in real time all the data sources, per separate and their collective cooperation. In addition, forensic activities have been implemented and made available through the framework. Users can review archived results and compare them to the most recent snapshot of the operational environment. In our first iteration we have focused on visual data (FMV, WAMI, CCTV/PTZ-Cameras, open source video, etc.) and AIS data streams (satellite and terrestrial sources). As a proof-of-concept, in our experiments we show how FMV detections can be combined with vessel tracking signals from AIS sources to confirm identity, tip-and-cue aerial reconnaissance, and monitor vessel activity in an area. △ Less

Submitted 4 June, 2020; v1 submitted 16 January, 2020; originally announced January 2020.

Contextual Sense Making by Fusing Scene Classification, Detections, and Events in Full Motion Video

Authors: Marc Bosch, Joseph Nassar, Benjamin Ortiz, Brendan Lammers, David Lindenbaum, John Wahl, Robert Mangum, Margaret Smith

Abstract: With the proliferation of imaging sensors, the volume of multi-modal imagery far exceeds the ability of human analysts to adequately consume and exploit it. Full motion video (FMV) possesses the extra challenge of containing large amounts of redundant temporal data. We aim to address the needs of human analysts to consume and exploit data given aerial FMV. We have investigated and designed a syste… ▽ More With the proliferation of imaging sensors, the volume of multi-modal imagery far exceeds the ability of human analysts to adequately consume and exploit it. Full motion video (FMV) possesses the extra challenge of containing large amounts of redundant temporal data. We aim to address the needs of human analysts to consume and exploit data given aerial FMV. We have investigated and designed a system capable of detecting events and activities of interest that deviate from the baseline patterns of observation given FMV feeds. We have divided the problem into three tasks: (1) Context awareness, (2) object cataloging, and (3) event detection. The goal of context awareness is to constraint the problem of visual search and detection in video data. A custom image classifier categorizes the scene with one or multiple labels to identify the operating context and environment. This step helps reducing the semantic search space of downstream tasks in order to increase their accuracy. The second step is object cataloging, where an ensemble of object detectors locates and labels any known objects found in the scene (people, vehicles, boats, planes, buildings, etc.). Finally, context information and detections are sent to the event detection engine to monitor for certain behaviors. A series of analytics monitor the scene by tracking object counts, and object interactions. If these object interactions are not declared to be commonly observed in the current scene, the system will report, geolocate, and log the event. Events of interest include identifying a gathering of people as a meeting and/or a crowd, alerting when there are boats on a beach unloading cargo, increased count of people entering a building, people getting in and/or out of vehicles of interest, etc. We have applied our methods on data from different sensors at different resolutions in a variety of geographical areas. △ Less

Submitted 16 January, 2020; originally announced January 2020.

Assessing Data Quality of Annotations with Krippendorff Alpha For Applications in Computer Vision

Authors: Joseph Nassar, Viveca Pavon-Harr, Marc Bosch, Ian McCulloh

Abstract: Current supervised deep learning frameworks rely on annotated data for modeling the underlying data distribution of a given task. In particular for computer vision algorithms powered by deep learning, the quality of annotated data is the most critical factor in achieving the desired algorithm performance. Data annotation is, typically, a manual process where the annotator follows guidelines and op… ▽ More Current supervised deep learning frameworks rely on annotated data for modeling the underlying data distribution of a given task. In particular for computer vision algorithms powered by deep learning, the quality of annotated data is the most critical factor in achieving the desired algorithm performance. Data annotation is, typically, a manual process where the annotator follows guidelines and operates in a best-guess manner. Labeling criteria among annotators can show discrepancies in labeling results. This may impact the algorithm inference performance. Given the popularity and widespread use of deep learning among computer vision, more and more custom datasets are needed to train neural networks to tackle different kinds of tasks. Unfortunately, there is no full understanding of the factors that affect annotated data quality, and how it translates into algorithm performance. In this paper we studied this problem for object detection and recognition.We conducted several data annotation experiments to measure inter annotator agreement and consistency, as well as how the selection of ground truth impacts the perceived algorithm performance.We propose a methodology to monitor the quality of annotations during the labeling of images and how it can be used to measure performance. We also show that neglecting to monitor the annotation process can result in significant loss in algorithm precision. Through these experiments, we observe that knowledge of the labeling process, training data, and ground truth data used for algorithm evaluation are fundamental components to accurately assess trustworthiness of an AI system. △ Less

Submitted 20 December, 2019; originally announced December 2019.

Comments: Accepted to AAAI Symposium 2019

Semantic Stereo for Incidental Satellite Images

Authors: Marc Bosch, Kevin Foster, Gordon Christie, Sean Wang, Gregory D Hager, Myron Brown

Abstract: The increasingly common use of incidental satellite images for stereo reconstruction versus rigidly tasked binocular or trinocular coincident collection is hel** to enable timely global-scale 3D map**; however, reliable stereo correspondence from multi-date image pairs remains very challenging due to seasonal appearance differences and scene change. Promising recent work suggests that semantic… ▽ More The increasingly common use of incidental satellite images for stereo reconstruction versus rigidly tasked binocular or trinocular coincident collection is hel** to enable timely global-scale 3D map**; however, reliable stereo correspondence from multi-date image pairs remains very challenging due to seasonal appearance differences and scene change. Promising recent work suggests that semantic scene segmentation can provide a robust regularizing prior for resolving ambiguities in stereo correspondence and reconstruction problems. To enable research for pairwise semantic stereo and multi-view semantic 3D reconstruction with incidental satellite images, we have established a large-scale public dataset including multi-view, multi-band satellite images and ground truth geometric and semantic labels for two large cities. To demonstrate the complementary nature of the stereo and segmentation tasks, we present lightweight public baselines adapted from recent state of the art convolutional neural network models and assess their performance. △ Less

Submitted 21 November, 2018; originally announced November 2018.

Comments: Accepted publication at WACV 2019

Sensor Adaptation for Improved Semantic Segmentation of Overhead Imagery

Authors: Marc Bosch, Gordon A. Christie, Christopher M. Gifford

Abstract: Semantic segmentation is a powerful method to facilitate visual scene understanding. Each pixel is assigned a label according to a pre-defined list of object classes and semantic entities. This becomes very useful as a means to summarize large scale overhead imagery. In this paper we present our work on semantic segmentation with applications to overhead imagery. We propose an algorithm that build… ▽ More Semantic segmentation is a powerful method to facilitate visual scene understanding. Each pixel is assigned a label according to a pre-defined list of object classes and semantic entities. This becomes very useful as a means to summarize large scale overhead imagery. In this paper we present our work on semantic segmentation with applications to overhead imagery. We propose an algorithm that builds and extends upon the DeepLab framework to be able to refine and resolve small objects (relative to the image size) such as vehicles. We have also investigated sensor adaptation as a means to augment available training data to be able to reduce some of the shortcomings of neural networks when deployed in new environments and to new sensors. We report results on several datasets and compare performance with other state-of-the-art architectures. △ Less

Submitted 20 November, 2018; originally announced November 2018.

Comments: Accepted publication at WACV 2019

Improved Image Segmentation via Cost Minimization of Multiple Hypotheses

Authors: Marc Bosch, Christopher M. Gifford, Austin G. Dress, Clare W. Lau, Jeffrey G. Skibo, Gordon A. Christie

Abstract: Image segmentation is an important component of many image understanding systems. It aims to group pixels in a spatially and perceptually coherent manner. Typically, these algorithms have a collection of parameters that control the degree of over-segmentation produced. It still remains a challenge to properly select such parameters for human-like perceptual grou**. In this work, we exploit the d… ▽ More Image segmentation is an important component of many image understanding systems. It aims to group pixels in a spatially and perceptually coherent manner. Typically, these algorithms have a collection of parameters that control the degree of over-segmentation produced. It still remains a challenge to properly select such parameters for human-like perceptual grou**. In this work, we exploit the diversity of segments produced by different choices of parameters. We scan the segmentation parameter space and generate a collection of image segmentation hypotheses (from highly over-segmented to under-segmented). These are fed into a cost minimization framework that produces the final segmentation by selecting segments that: (1) better describe the natural contours of the image, and (2) are more stable and persistent among all the segmentation hypotheses. We compare our algorithm's performance with state-of-the-art algorithms, showing that we can achieve improved results. We also show that our framework is robust to the choice of segmentation kernel that produces the initial set of hypotheses. △ Less

Submitted 31 January, 2018; originally announced February 2018.

Comments: Accepted BMVC 17

Super-Resolution for Overhead Imagery Using DenseNets and Adversarial Learning

Authors: Marc Bosch, Christopher M. Gifford, Pedro A. Rodriguez

Abstract: Recent advances in Generative Adversarial Learning allow for new modalities of image super-resolution by learning low to high resolution map**s. In this paper we present our work using Generative Adversarial Networks (GANs) with applications to overhead and satellite imagery. We have experimented with several state-of-the-art architectures. We propose a GAN-based architecture using densely conne… ▽ More Recent advances in Generative Adversarial Learning allow for new modalities of image super-resolution by learning low to high resolution map**s. In this paper we present our work using Generative Adversarial Networks (GANs) with applications to overhead and satellite imagery. We have experimented with several state-of-the-art architectures. We propose a GAN-based architecture using densely connected convolutional neural networks (DenseNets) to be able to super-resolve overhead imagery with a factor of up to 8x. We have also investigated resolution limits of these networks. We report results on several publicly available datasets, including SpaceNet data and IARPA Multi-View Stereo Challenge, and compare performance with other state-of-the-art architectures. △ Less

Submitted 28 November, 2017; originally announced November 2017.

Comments: 9 pages, 9 figures, WACV 2018 submission