-
SpY: A Context-Based Approach to Spacecraft Component Detection
Authors:
Trupti Mahendrakar,
Ryan T. White,
Madhur Tiwari
Abstract:
This paper focuses on autonomously characterizing components such as solar panels, body panels, antennas, and thrusters of an unknown resident space object (RSO) using camera feed to aid autonomous on-orbit servicing (OOS) and active debris removal. Significant research has been conducted in this area using convolutional neural networks (CNNs). While CNNs are powerful at learning patterns and perf…
▽ More
This paper focuses on autonomously characterizing components such as solar panels, body panels, antennas, and thrusters of an unknown resident space object (RSO) using camera feed to aid autonomous on-orbit servicing (OOS) and active debris removal. Significant research has been conducted in this area using convolutional neural networks (CNNs). While CNNs are powerful at learning patterns and performing object detection, they struggle with missed detections and misclassifications in environments different from the training data, making them unreliable for safety in high-stakes missions like OOS. Additionally, failures exhibited by CNNs are often easily rectifiable by humans using commonsense reasoning and contextual knowledge. Embedding such reasoning in an object detector could improve detection accuracy. To validate this hypothesis, this paper presents an end-to-end object detector called SpaceYOLOv2 (SpY), which leverages the generalizability of CNNs while incorporating contextual knowledge using traditional computer vision techniques. SpY consists of two main components: a shape detector and the SpaceYOLO classifier (SYC). The shape detector uses CNNs to detect primitive shapes of RSOs and SYC associates these shapes with contextual knowledge, such as color and texture, to classify them as spacecraft components or "unknown" if the detected shape is uncertain. SpY's modular architecture allows customizable usage of contextual knowledge to improve detection performance, or SYC as a secondary fail-safe classifier with an existing spacecraft component detector. Performance evaluations on hardware-in-the-loop images of a mock-up spacecraft demonstrate that SpY is accurate and an ensemble of SpY with YOLOv5 trained for satellite component detection improved the performance by 23.4% in recall, demonstrating enhanced safety for vision-based navigation tasks.
△ Less
Submitted 26 June, 2024;
originally announced June 2024.
-
SatSplatYOLO: 3D Gaussian Splatting-based Virtual Object Detection Ensembles for Satellite Feature Recognition
Authors:
Van Minh Nguyen,
Emma Sandidge,
Trupti Mahendrakar,
Ryan T. White
Abstract:
On-orbit servicing (OOS), inspection of spacecraft, and active debris removal (ADR). Such missions require precise rendezvous and proximity operations in the vicinity of non-cooperative, possibly unknown, resident space objects. Safety concerns with manned missions and lag times with ground-based control necessitate complete autonomy. In this article, we present an approach for map** geometries…
▽ More
On-orbit servicing (OOS), inspection of spacecraft, and active debris removal (ADR). Such missions require precise rendezvous and proximity operations in the vicinity of non-cooperative, possibly unknown, resident space objects. Safety concerns with manned missions and lag times with ground-based control necessitate complete autonomy. In this article, we present an approach for map** geometries and high-confidence detection of components of unknown, non-cooperative satellites on orbit. We implement accelerated 3D Gaussian splatting to learn a 3D representation of the satellite, render virtual views of the target, and ensemble the YOLOv5 object detector over the virtual views, resulting in reliable, accurate, and precise satellite component detections. The full pipeline capable of running on-board and stand to enable downstream machine intelligence tasks necessary for autonomous guidance, navigation, and control tasks.
△ Less
Submitted 4 June, 2024;
originally announced June 2024.
-
Physics-based material parameters extraction from perovskite experiments via Bayesian optimization
Authors:
Hualin Zhan,
Viqar Ahmad,
Azul Mayon,
Grace Tabi,
Anh Dinh Bui,
Zhuofeng Li,
Daniel Walter,
Hieu Nguyen,
Klaus Weber,
Thomas White,
Kylie Catchpole
Abstract:
The ability to extract material parameters of perovskite from quantitative experimental analysis is essential for rational design of photovoltaic and optoelectronic applications. However, the difficulty of this analysis increases significantly with the complexity of the theoretical model and the number of material parameters for perovskite. Here we use Bayesian optimization to develop an analysis…
▽ More
The ability to extract material parameters of perovskite from quantitative experimental analysis is essential for rational design of photovoltaic and optoelectronic applications. However, the difficulty of this analysis increases significantly with the complexity of the theoretical model and the number of material parameters for perovskite. Here we use Bayesian optimization to develop an analysis platform that can extract up to 8 fundamental material parameters of an organometallic perovskite semiconductor from a transient photoluminescence experiment, based on a complex full physics model that includes drift-diffusion of carriers and dynamic defect occupation. An example study of thermal degradation reveals that the carrier mobility and trap-assisted recombination coefficient are reduced noticeably, while the defect energy level remains nearly unchanged. The reduced carrier mobility can dominate the overall effect on thermal degradation of perovskite solar cells by reducing the fill factor, despite the opposite effect of the reduced trap-assisted recombination coefficient on increasing the fill factor. In future, this platform can be conveniently applied to other experiments or to combinations of experiments, accelerating materials discovery and optimization of semiconductor materials for photovoltaics and other applications.
△ Less
Submitted 29 May, 2024; v1 submitted 16 February, 2024;
originally announced February 2024.
-
Exploring higher-order neural network node interactions with total correlation
Authors:
Thomas Kerby,
Teresa White,
Kevin Moon
Abstract:
In domains such as ecological systems, collaborations, and the human brain the variables interact in complex ways. Yet accurately characterizing higher-order variable interactions (HOIs) is a difficult problem that is further exacerbated when the HOIs change across the data. To solve this problem we propose a new method called Local Correlation Explanation (CorEx) to capture HOIs at a local scale…
▽ More
In domains such as ecological systems, collaborations, and the human brain the variables interact in complex ways. Yet accurately characterizing higher-order variable interactions (HOIs) is a difficult problem that is further exacerbated when the HOIs change across the data. To solve this problem we propose a new method called Local Correlation Explanation (CorEx) to capture HOIs at a local scale by first clustering data points based on their proximity on the data manifold. We then use a multivariate version of the mutual information called the total correlation, to construct a latent factor representation of the data within each cluster to learn the local HOIs. We use Local CorEx to explore HOIs in synthetic and real world data to extract hidden insights about the data structure. Lastly, we demonstrate Local CorEx's suitability to explore and interpret the inner workings of trained neural networks.
△ Less
Submitted 6 February, 2024;
originally announced February 2024.
-
Making Informed Decisions: Supporting Cobot Integration Considering Business and Worker Preferences
Authors:
Dakota Sullivan,
Nathan Thomas White,
Andrew Schoen,
Bilge Mutlu
Abstract:
Robots are ubiquitous in small-to-large-scale manufacturers. While collaborative robots (cobots) have significant potential in these settings due to their flexibility and ease of use, proper integration is critical to realize their full potential. Specifically, cobots need to be integrated in ways that utilize their strengths, improve manufacturing performance, and facilitate use in concert with h…
▽ More
Robots are ubiquitous in small-to-large-scale manufacturers. While collaborative robots (cobots) have significant potential in these settings due to their flexibility and ease of use, proper integration is critical to realize their full potential. Specifically, cobots need to be integrated in ways that utilize their strengths, improve manufacturing performance, and facilitate use in concert with human workers. Effective integration requires careful consideration and the knowledge of roboticists, manufacturing engineers, and business administrators. We propose an approach involving the stages of planning, analysis, development, and presentation, to inform manufacturers about cobot integration within their facilities prior to the integration process. We contextualize our approach in a case study with an SME collaborator and discuss insights learned.
△ Less
Submitted 10 January, 2024;
originally announced January 2024.
-
Characterizing Satellite Geometry via Accelerated 3D Gaussian Splatting
Authors:
Van Minh Nguyen,
Emma Sandidge,
Trupti Mahendrakar,
Ryan T. White
Abstract:
The accelerating deployment of spacecraft in orbit have generated interest in on-orbit servicing (OOS), inspection of spacecraft, and active debris removal (ADR). Such missions require precise rendezvous and proximity operations in the vicinity of non-cooperative, possible unknown, resident space objects. Safety concerns with manned missions and lag times with ground-based control necessitate comp…
▽ More
The accelerating deployment of spacecraft in orbit have generated interest in on-orbit servicing (OOS), inspection of spacecraft, and active debris removal (ADR). Such missions require precise rendezvous and proximity operations in the vicinity of non-cooperative, possible unknown, resident space objects. Safety concerns with manned missions and lag times with ground-based control necessitate complete autonomy. This requires robust characterization of the target's geometry. In this article, we present an approach for map** geometries of satellites on orbit based on 3D Gaussian Splatting that can run on computing resources available on current spaceflight hardware. We demonstrate model training and 3D rendering performance on a hardware-in-the-loop satellite mock-up under several realistic lighting and motion conditions. Our model is shown to be capable of training on-board and rendering higher quality novel views of an unknown satellite nearly 2 orders of magnitude faster than previous NeRF-based algorithms. Such on-board capabilities are critical to enable downstream machine intelligence tasks necessary for autonomous guidance, navigation, and control tasks.
△ Less
Submitted 4 January, 2024;
originally announced January 2024.
-
Taking a PEEK into YOLOv5 for Satellite Component Recognition via Entropy-based Visual Explanations
Authors:
Mackenzie J. Meni,
Trupti Mahendrakar,
Olivia D. M. Raney,
Ryan T. White,
Michael L. Mayo,
Kevin Pilkiewicz
Abstract:
The escalating risk of collisions and the accumulation of space debris in Low Earth Orbit (LEO) has reached critical concern due to the ever increasing number of spacecraft. Addressing this crisis, especially in dealing with non-cooperative and unidentified space debris, is of paramount importance. This paper contributes to efforts in enabling autonomous swarms of small chaser satellites for targe…
▽ More
The escalating risk of collisions and the accumulation of space debris in Low Earth Orbit (LEO) has reached critical concern due to the ever increasing number of spacecraft. Addressing this crisis, especially in dealing with non-cooperative and unidentified space debris, is of paramount importance. This paper contributes to efforts in enabling autonomous swarms of small chaser satellites for target geometry determination and safe flight trajectory planning for proximity operations in LEO. Our research explores on-orbit use of the You Only Look Once v5 (YOLOv5) object detection model trained to detect satellite components. While this model has shown promise, its inherent lack of interpretability hinders human understanding, a critical aspect of validating algorithms for use in safety-critical missions. To analyze the decision processes, we introduce Probabilistic Explanations for Entropic Knowledge extraction (PEEK), a method that utilizes information theoretic analysis of the latent representations within the hidden layers of the model. Through both synthetic in hardware-in-the-loop experiments, PEEK illuminates the decision-making processes of the model, hel** identify its strengths, limitations and biases.
△ Less
Submitted 25 November, 2023; v1 submitted 3 November, 2023;
originally announced November 2023.
-
Entropy-based Guidance of Deep Neural Networks for Accelerated Convergence and Improved Performance
Authors:
Mackenzie J. Meni,
Ryan T. White,
Michael Mayo,
Kevin Pilkiewicz
Abstract:
Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building and training them are uncertain processes. To add structure to these efforts, we derive new mathematical results to efficiently measure the changes in entropy a…
▽ More
Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building and training them are uncertain processes. To add structure to these efforts, we derive new mathematical results to efficiently measure the changes in entropy as fully-connected and convolutional neural networks process data, and introduce entropy-based loss terms. Experiments in image compression and image classification on benchmark datasets demonstrate these losses guide neural networks to learn rich latent data representations in fewer dimensions, converge in fewer training epochs, and achieve better test metrics.
△ Less
Submitted 28 August, 2023;
originally announced August 2023.
-
On the Security Blind Spots of Software Composition Analysis
Authors:
Jens Dietrich,
Shawn Rasheed,
Alexander Jordan,
Tim White
Abstract:
Modern software heavily relies on the use of components. Those components are usually published in central repositories, and managed by build systems via dependencies. Due to issues around vulnerabilities, licenses and the propagation of bugs, the study of those dependencies is of utmost importance, and numerous software composition analysis tools have emerged for this purpose. A particular challe…
▽ More
Modern software heavily relies on the use of components. Those components are usually published in central repositories, and managed by build systems via dependencies. Due to issues around vulnerabilities, licenses and the propagation of bugs, the study of those dependencies is of utmost importance, and numerous software composition analysis tools have emerged for this purpose. A particular challenge are hidden dependencies that are the result of cloning or shading where code from a component is "inlined", and, in the case of shading, moved to different namespaces.
We present a novel approach to detect vulnerable clones in the Maven repository. Our approach is lightweight in that it does not require the creation and maintenance of a custom index. Starting with 29 vulnerabilities with assigned CVEs and proof-of-vulnerability projects, we retrieve over 53k potential vulnerable clones from Maven Central. After running our analysis on this set, we detect 727 confirmed vulnerable clones (86 if versions are aggregated) and synthesize a testable proof-of-vulnerability project for each of those. We demonstrate that existing SCA tools often miss those exposures. At the time of submission those results have led to changes to the entries for six CVEs in the GitHub Security Advisory Database (GHSA) via accepted pull requests, with more pending.
△ Less
Submitted 9 October, 2023; v1 submitted 8 June, 2023;
originally announced June 2023.
-
Challenges of movement quality using motion capture in theatre
Authors:
Georges Gagneré,
Andy Lavender,
Cédric Plessiet,
Tim White
Abstract:
We describe1 two case studies of AvatarStaging theatrical mixed reality framework combining avatars and performers acting in an artistic context. We outline a qualitative approach toward the condition for stage presence for the avatars. We describe the motion control solutions we experimented with from the perspective of building a protocol of avatar direction in a mixed reality appropriate to liv…
▽ More
We describe1 two case studies of AvatarStaging theatrical mixed reality framework combining avatars and performers acting in an artistic context. We outline a qualitative approach toward the condition for stage presence for the avatars. We describe the motion control solutions we experimented with from the perspective of building a protocol of avatar direction in a mixed reality appropriate to live performance.
△ Less
Submitted 13 March, 2023;
originally announced March 2023.
-
SpaceYOLO: A Human-Inspired Model for Real-time, On-board Spacecraft Feature Detection
Authors:
Trupti Mahendrakar,
Ryan T. White,
Markus Wilde,
Madhur Tiwari
Abstract:
The rapid proliferation of non-cooperative spacecraft and space debris in orbit has precipitated a surging demand for on-orbit servicing and space debris removal at a scale that only autonomous missions can address, but the prerequisite autonomous navigation and flightpath planning to safely capture an unknown, non-cooperative, tumbling space object is an open problem. This requires algorithms for…
▽ More
The rapid proliferation of non-cooperative spacecraft and space debris in orbit has precipitated a surging demand for on-orbit servicing and space debris removal at a scale that only autonomous missions can address, but the prerequisite autonomous navigation and flightpath planning to safely capture an unknown, non-cooperative, tumbling space object is an open problem. This requires algorithms for real-time, automated spacecraft feature recognition to pinpoint the locations of collision hazards (e.g. solar panels or antennas) and safe docking features (e.g. satellite bodies or thrusters) so safe, effective flightpaths can be planned. Prior work in this area reveals the performance of computer vision models are highly dependent on the training dataset and its coverage of scenarios visually similar to the real scenarios that occur in deployment. Hence, the algorithm may have degraded performance under certain lighting conditions even when the rendezvous maneuver conditions of the chaser to the target spacecraft are the same. This work delves into how humans perform these tasks through a survey of how aerospace engineering students experienced with spacecraft shapes and components recognize features of the three spacecraft: Landsat, Envisat, Anik, and the orbiter Mir. The survey reveals that the most common patterns in the human detection process were to consider the shape and texture of the features: antennas, solar panels, thrusters, and satellite bodies. This work introduces a novel algorithm SpaceYOLO, which fuses a state-of-the-art object detector YOLOv5 with a separate neural network based on these human-inspired decision processes exploiting shape and texture. Performance in autonomous spacecraft detection of SpaceYOLO is compared to ordinary YOLOv5 in hardware-in-the-loop experiments under different lighting and chaser maneuver conditions at the ORION Laboratory at Florida Tech.
△ Less
Submitted 1 February, 2023;
originally announced February 2023.
-
3D Reconstruction of Non-cooperative Resident Space Objects using Instant NGP-accelerated NeRF and D-NeRF
Authors:
Basilio Caruso,
Trupti Mahendrakar,
Van Minh Nguyen,
Ryan T. White,
Todd Steffen
Abstract:
The proliferation of non-cooperative resident space objects (RSOs) in orbit has spurred the demand for active space debris removal, on-orbit servicing (OOS), classification, and functionality identification of these RSOs. Recent advances in computer vision have enabled high-definition 3D modeling of objects based on a set of 2D images captured from different viewing angles. This work adapts Instan…
▽ More
The proliferation of non-cooperative resident space objects (RSOs) in orbit has spurred the demand for active space debris removal, on-orbit servicing (OOS), classification, and functionality identification of these RSOs. Recent advances in computer vision have enabled high-definition 3D modeling of objects based on a set of 2D images captured from different viewing angles. This work adapts Instant NeRF and D-NeRF, variations of the neural radiance field (NeRF) algorithm to the problem of map** RSOs in orbit for the purposes of functionality identification and assisting with OOS. The algorithms are evaluated for 3D reconstruction quality and hardware requirements using datasets of images of a spacecraft mock-up taken under two different lighting and motion conditions at the Orbital Robotic Interaction, On-Orbit Servicing and Navigation (ORION) Laboratory at Florida Institute of Technology. Instant NeRF is shown to learn high-fidelity 3D models with a computational cost that could feasibly be trained on on-board computers.
△ Less
Submitted 9 June, 2023; v1 submitted 22 January, 2023;
originally announced January 2023.
-
Autonomous Rendezvous with Non-cooperative Target Objects with Swarm Chasers and Observers
Authors:
Trupti Mahendrakar,
Steven Holmberg,
Andrew Ekblad,
Emma Conti,
Ryan T. White,
Markus Wilde,
Isaac Silver
Abstract:
Space debris is on the rise due to the increasing demand for spacecraft for com-munication, navigation, and other applications. The Space Surveillance Network (SSN) tracks over 27,000 large pieces of debris and estimates the number of small, un-trackable fragments at over 1,00,000. To control the growth of debris, the for-mation of further debris must be reduced. Some solutions include deorbiting…
▽ More
Space debris is on the rise due to the increasing demand for spacecraft for com-munication, navigation, and other applications. The Space Surveillance Network (SSN) tracks over 27,000 large pieces of debris and estimates the number of small, un-trackable fragments at over 1,00,000. To control the growth of debris, the for-mation of further debris must be reduced. Some solutions include deorbiting larger non-cooperative resident space objects (RSOs) or servicing satellites in or-bit. Both require rendezvous with RSOs, and the scale of the problem calls for autonomous missions. This paper introduces the Multipurpose Autonomous Ren-dezvous Vision-Integrated Navigation system (MARVIN) developed and tested at the ORION Facility at Florida Institution of Technology. MARVIN consists of two sub-systems: a machine vision-aided navigation system and an artificial po-tential field (APF) guidance algorithm which work together to command a swarm of chasers to safely rendezvous with the RSO. We present the MARVIN architec-ture and hardware-in-the-loop experiments demonstrating autonomous, collabo-rative swarm satellite operations successfully guiding three drones to rendezvous with a physical mockup of a non-cooperative satellite in motion.
△ Less
Submitted 22 January, 2023;
originally announced January 2023.
-
Performance Study of YOLOv5 and Faster R-CNN for Autonomous Navigation around Non-Cooperative Targets
Authors:
Trupti Mahendrakar,
Andrew Ekblad,
Nathan Fischer,
Ryan T. White,
Markus Wilde,
Brian Kish,
Isaac Silver
Abstract:
Autonomous navigation and path-planning around non-cooperative space objects is an enabling technology for on-orbit servicing and space debris removal systems. The navigation task includes the determination of target object motion, the identification of target object features suitable for gras**, and the identification of collision hazards and other keep-out zones. Given this knowledge, chaser s…
▽ More
Autonomous navigation and path-planning around non-cooperative space objects is an enabling technology for on-orbit servicing and space debris removal systems. The navigation task includes the determination of target object motion, the identification of target object features suitable for gras**, and the identification of collision hazards and other keep-out zones. Given this knowledge, chaser spacecraft can be guided towards capture locations without damaging the target object or without unduly the operations of a servicing target by covering up solar arrays or communication antennas. One way to autonomously achieve target identification, characterization and feature recognition is by use of artificial intelligence algorithms. This paper discusses how the combination of cameras and machine learning algorithms can achieve the relative navigation task. The performance of two deep learning-based object detection algorithms, Faster Region-based Convolutional Neural Networks (R-CNN) and You Only Look Once (YOLOv5), is tested using experimental data obtained in formation flight simulations in the ORION Lab at Florida Institute of Technology. The simulation scenarios vary the yaw motion of the target object, the chaser approach trajectory, and the lighting conditions in order to test the algorithms in a wide range of realistic and performance limiting situations. The data analyzed include the mean average precision metrics in order to compare the performance of the object detectors. The paper discusses the path to implementing the feature recognition algorithms and towards integrating them into the spacecraft Guidance Navigation and Control system.
△ Less
Submitted 21 January, 2023;
originally announced January 2023.
-
Resource-constrained FPGA Design for Satellite Component Feature Extraction
Authors:
Andrew Ekblad,
Trupti Mahendrakar,
Ryan T. White,
Markus Wilde,
Isaac Silver,
Brooke Wheeler
Abstract:
The effective use of computer vision and machine learning for on-orbit applications has been hampered by limited computing capabilities, and therefore limited performance. While embedded systems utilizing ARM processors have been shown to meet acceptable but low performance standards, the recent availability of larger space-grade field programmable gate arrays (FPGAs) show potential to exceed the…
▽ More
The effective use of computer vision and machine learning for on-orbit applications has been hampered by limited computing capabilities, and therefore limited performance. While embedded systems utilizing ARM processors have been shown to meet acceptable but low performance standards, the recent availability of larger space-grade field programmable gate arrays (FPGAs) show potential to exceed the performance of microcomputer systems. This work proposes use of neural network-based object detection algorithm that can be deployed on a comparably resource-constrained FPGA to automatically detect components of non-cooperative, satellites on orbit. Hardware-in-the-loop experiments were performed on the ORION Maneuver Kinematics Simulator at Florida Tech to compare the performance of the new model deployed on a small, resource-constrained FPGA to an equivalent algorithm on a microcomputer system. Results show the FPGA implementation increases the throughput and decreases latency while maintaining comparable accuracy. These findings suggest future missions should consider deploying computer vision algorithms on space-grade FPGAs.
△ Less
Submitted 21 January, 2023;
originally announced January 2023.
-
Exploring GPU Stream-Aware Message Passing using Triggered Operations
Authors:
Naveen Namashivayam,
Krishna Kandalla,
Trey White,
Nick Radcliffe,
Larry Kaplan,
Mark Pagel
Abstract:
Modern heterogeneous supercomputing systems are comprised of compute blades that offer CPUs and GPUs. On such systems, it is essential to move data efficiently between these different compute engines across a high-speed network. While current generation scientific applications and systems software stacks are GPU-aware, CPU threads are still required to orchestrate data moving communication operati…
▽ More
Modern heterogeneous supercomputing systems are comprised of compute blades that offer CPUs and GPUs. On such systems, it is essential to move data efficiently between these different compute engines across a high-speed network. While current generation scientific applications and systems software stacks are GPU-aware, CPU threads are still required to orchestrate data moving communication operations and inter-process synchronization operations.
A new GPU stream-aware MPI communication strategy called stream-triggered (ST) communication is explored to allow offloading both computation and communication control paths to the GPU. The proposed ST communication strategy is implemented on HPE Slingshot Interconnects over a new proprietary HPE Slingshot NIC (Slingshot 11) using the supported triggered operations feature. Performance of the proposed new communication strategy is evaluated using a microbenchmark kernel called Faces, based on the nearest-neighbor communication pattern in the CORAL-2 Nekbone benchmark, over a heterogeneous node architecture consisting of AMD CPUs and GPUs.
△ Less
Submitted 9 August, 2022;
originally announced August 2022.
-
SHREC 2021: Classification in cryo-electron tomograms
Authors:
Ilja Gubins,
Marten L. Chaillet,
Gijs van der Schot,
M. Cristina Trueba,
Remco C. Veltkamp,
Friedrich Förster,
Xiao Wang,
Daisuke Kihara,
Emmanuel Moebel,
Nguyen P. Nguyen,
Tommi White,
Filiz Bunyak,
Giorgos Papoulias,
Stavros Gerolymatos,
Evangelia I. Zacharaki,
Konstantinos Moustakas,
Xiangrui Zeng,
Sinuo Liu,
Min Xu,
Yaoyu Wang,
Cheng Chen,
Xuefeng Cui,
Fa Zhang
Abstract:
Cryo-electron tomography (cryo-ET) is an imaging technique that allows three-dimensional visualization of macro-molecular assemblies under near-native conditions. Cryo-ET comes with a number of challenges, mainly low signal-to-noise and inability to obtain images from all angles. Computational methods are key to analyze cryo-electron tomograms.
To promote innovation in computational methods, we…
▽ More
Cryo-electron tomography (cryo-ET) is an imaging technique that allows three-dimensional visualization of macro-molecular assemblies under near-native conditions. Cryo-ET comes with a number of challenges, mainly low signal-to-noise and inability to obtain images from all angles. Computational methods are key to analyze cryo-electron tomograms.
To promote innovation in computational methods, we generate a novel simulated dataset to benchmark different methods of localization and classification of biological macromolecules in tomograms. Our publicly available dataset contains ten tomographic reconstructions of simulated cell-like volumes. Each volume contains twelve different types of complexes, varying in size, function and structure.
In this paper, we have evaluated seven different methods of finding and classifying proteins. Seven research groups present results obtained with learning-based methods and trained on the simulated dataset, as well as a baseline template matching (TM), a traditional method widely used in cryo-ET research. We show that learning-based approaches can achieve notably better localization and classification performance than TM. We also experimentally confirm that there is a negative relationship between particle size and performance for all methods.
△ Less
Submitted 18 March, 2022;
originally announced March 2022.
-
Communicating Inferred Goals with Passive Augmented Reality and Active Haptic Feedback
Authors:
James F. Mullen Jr,
Josh Mosier,
Sounak Chakrabarti,
Anqi Chen,
Tyler White,
Dylan P. Losey
Abstract:
Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arm's motion, the robot gathers information about the human's desired task. But how does the human know what their robot has inferred? Today's approaches often focus on conveying intent: for instance, upon legible motions or gestures to indicate what the robot is p…
▽ More
Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arm's motion, the robot gathers information about the human's desired task. But how does the human know what their robot has inferred? Today's approaches often focus on conveying intent: for instance, upon legible motions or gestures to indicate what the robot is planning. However, closing the loop on robot inference requires more than just revealing the robot's current policy: the robot should also display the alternatives it thinks are likely, and prompt the human teacher when additional guidance is necessary. In this paper we propose a multimodal approach for communicating robot inference that combines both passive and active feedback. Specifically, we leverage information-rich augmented reality to passively visualize what the robot has inferred, and attention-grabbing haptic wristbands to actively prompt and direct the human's teaching. We apply our system to shared autonomy tasks where the robot must infer the human's goal in real-time. Within this context, we integrate passive and active modalities into a single algorithmic framework that determines when and which type of feedback to provide. Combining both passive and active feedback experimentally outperforms single modality baselines; during an in-person user study, we demonstrate that our integrated approach increases how efficiently humans teach the robot while simultaneously decreasing the amount of time humans spend interacting with the robot. Videos here: https://youtu.be/swq_u4iIP-g
△ Less
Submitted 3 September, 2021;
originally announced September 2021.
-
Variational voxelwise rs-fMRI representation learning: Evaluation of sex, age, and neuropsychiatric signatures
Authors:
Eloy Geenjaar,
Tonya White,
Vince Calhoun
Abstract:
We propose to apply non-linear representation learning to voxelwise rs-fMRI data. Learning the non-linear representations is done using a variational autoencoder (VAE). The VAE is trained on voxelwise rs-fMRI data and performs non-linear dimensionality reduction that retains meaningful information. The retention of information in the model's representations is evaluated using downstream age regres…
▽ More
We propose to apply non-linear representation learning to voxelwise rs-fMRI data. Learning the non-linear representations is done using a variational autoencoder (VAE). The VAE is trained on voxelwise rs-fMRI data and performs non-linear dimensionality reduction that retains meaningful information. The retention of information in the model's representations is evaluated using downstream age regression and sex classification tasks. The results on these tasks are highly encouraging and a linear regressor trained with the representations of our unsupervised model performs almost as well as a supervised neural network, trained specifically for age regression on the same dataset. The model is also evaluated with a schizophrenia diagnosis prediction task, to assess its feasibility as a dimensionality reduction method for neuropsychiatric datasets. These results highlight the potential for pre-training on a larger set of individuals who do not have mental illness, to improve the downstream neuropsychiatric task results. The pre-trained model is fine-tuned for a variable number of epochs on a schizophrenia dataset and we find that fine-tuning for 1 epoch yields the best results. This work therefore not only opens up non-linear dimensionality reduction for voxelwise rs-fMRI data but also shows that pre-training a deep learning model on voxelwise rs-fMRI datasets greatly increases performance even on smaller datasets. It also opens up the ability to look at the distribution of rs-fMRI time series in the latent space of the VAE for heterogeneous neuropsychiatric disorders like schizophrenia in future work. This can be complemented with the generative aspect of the model that allows us to reconstruct points from the model's latent space back into brain space and obtain an improved understanding of the relation that the VAE learns between subjects, timepoints, and a subject's characteristics.
△ Less
Submitted 29 August, 2021;
originally announced August 2021.
-
GPS-Denied Navigation Using SAR Images and Neural Networks
Authors:
Teresa White,
Jesse Wheeler,
Colton Lindstrom,
Randall Christensen,
Kevin R. Moon
Abstract:
Unmanned aerial vehicles (UAV) often rely on GPS for navigation. GPS signals, however, are very low in power and easily jammed or otherwise disrupted. This paper presents a method for determining the navigation errors present at the beginning of a GPS-denied period utilizing data from a synthetic aperture radar (SAR) system. This is accomplished by comparing an online-generated SAR image with a re…
▽ More
Unmanned aerial vehicles (UAV) often rely on GPS for navigation. GPS signals, however, are very low in power and easily jammed or otherwise disrupted. This paper presents a method for determining the navigation errors present at the beginning of a GPS-denied period utilizing data from a synthetic aperture radar (SAR) system. This is accomplished by comparing an online-generated SAR image with a reference image obtained a priori. The distortions relative to the reference image are learned and exploited with a convolutional neural network to recover the initial navigational errors, which can be used to recover the true flight trajectory throughout the synthetic aperture. The proposed neural network approach is able to learn to predict the initial errors on both simulated and real SAR image data.
△ Less
Submitted 22 October, 2020;
originally announced October 2020.
-
Shared Visual Abstractions
Authors:
Tom White
Abstract:
This paper presents abstract art created by neural networks and broadly recognizable across various computer vision systems. The existence of abstract forms that trigger specific labels independent of neural architecture or training set suggests convolutional neural networks build shared visual representations for the categories they understand. Computer vision classifiers encountering these drawi…
▽ More
This paper presents abstract art created by neural networks and broadly recognizable across various computer vision systems. The existence of abstract forms that trigger specific labels independent of neural architecture or training set suggests convolutional neural networks build shared visual representations for the categories they understand. Computer vision classifiers encountering these drawings often respond with strong responses for specific labels - in extreme cases stronger than all examples from the validation set. By surveying human subjects we confirm that these abstract artworks are also broadly recognizable by people, suggesting visual representations triggered by these drawings are shared across human and computer vision systems.
△ Less
Submitted 19 November, 2019;
originally announced December 2019.
-
C2P2: A Collective Cryptocurrency Up/Down Price Prediction Engine
Authors:
Chongyang Bai,
Tommy White,
Linda Xiao,
V. S. Subrahmanian,
Ziheng Zhou
Abstract:
We study the problem of predicting whether the price of the 21 most popular cryptocurrencies (according to coinmarketcap.com) will go up or down on day d, using data up to day d-1. Our C2P2 algorithm is the first algorithm to consider the fact that the price of a cryptocurrency c might depend not only on historical prices, sentiments, global stock indices, but also on the prices and predicted pric…
▽ More
We study the problem of predicting whether the price of the 21 most popular cryptocurrencies (according to coinmarketcap.com) will go up or down on day d, using data up to day d-1. Our C2P2 algorithm is the first algorithm to consider the fact that the price of a cryptocurrency c might depend not only on historical prices, sentiments, global stock indices, but also on the prices and predicted prices of other cryptocurrencies. C2P2 therefore does not predict cryptocurrency prices one coin at a time --- rather it uses similarity metrics in conjunction with collective classification to compare multiple cryptocurrency features to jointly predict the cryptocurrency prices for all 21 coins considered. We show that our C2P2 algorithm beats out a recent competing 2017 paper by margins varying from 5.1-83% and another Bitcoin-specific prediction paper from 2018 by 16%. In both cases, C2P2 is the winner on all cryptocurrencies considered. Moreover, we experimentally show that the use of similarity metrics within our C2P2 algorithm leads to a direct improvement for 20 out of 21 cryptocurrencies ranging from 0.4% to 17.8%. Without the similarity component, C2P2 still beats competitors on 20 out of 21 cryptocurrencies considered. We show that all these results are statistically significant via a Student's t-test with p<1e-5. Check our demo at https://www.cs.dartmouth.edu/dsail/demos/c2p2
△ Less
Submitted 3 June, 2019;
originally announced June 2019.
-
Heuristic algorithms for the Maximum Colorful Subtree problem
Authors:
Kai Dührkop,
Marie Anne Lataretu,
W. Timothy J. White,
Sebastian Böcker
Abstract:
In metabolomics, small molecules are structurally elucidated using tandem mass spectrometry (MS/MS); this resulted in the computational Maximum Colorful Subtree problem, which is NP-hard. Unfortunately, data from a single metabolite requires us to solve hundreds or thousands of instances of this problem; and in a single Liquid Chromatography MS/MS run, hundreds or thousands of metabolites are meas…
▽ More
In metabolomics, small molecules are structurally elucidated using tandem mass spectrometry (MS/MS); this resulted in the computational Maximum Colorful Subtree problem, which is NP-hard. Unfortunately, data from a single metabolite requires us to solve hundreds or thousands of instances of this problem; and in a single Liquid Chromatography MS/MS run, hundreds or thousands of metabolites are measured.
Here, we comprehensively evaluate the performance of several heuristic algorithms for the problem against an exact algorithm. We put particular emphasis on whether a heuristic is able to rank candidates such that the correct solution is ranked highly. We propose this "intermediate" evaluation because evaluating the approximating quality of heuristics is misleading: Even a slightly suboptimal solution can be structurally very different from the true solution. On the other hand, we cannot structurally evaluate against the ground truth, as this is unknown. We find that one particular heuristic consistently ranks the correct solution in a top position, allowing us to speed up computations about 100-fold. We also find that scores of the best heuristic solutions are very close to the optimal score; in contrast, the structure of the solutions can deviate significantly from the optimal structures.
△ Less
Submitted 13 February, 2018; v1 submitted 23 January, 2018;
originally announced January 2018.
-
Exact and heuristic algorithms for Cograph Editing
Authors:
W. Timothy J. White,
Marcus Ludwig,
Sebastian Böcker
Abstract:
We present a dynamic programming algorithm for optimally solving the Cograph Editing problem on an $n$-vertex graph that runs in $O(3^n n)$ time and uses $O(2^n)$ space. In this problem, we are given a graph $G = (V, E)$ and the task is to find a smallest possible set $F \subseteq V \times V$ of vertex pairs such that $(V, E \bigtriangleup F)$ is a cograph (or $P_4$-free graph), where…
▽ More
We present a dynamic programming algorithm for optimally solving the Cograph Editing problem on an $n$-vertex graph that runs in $O(3^n n)$ time and uses $O(2^n)$ space. In this problem, we are given a graph $G = (V, E)$ and the task is to find a smallest possible set $F \subseteq V \times V$ of vertex pairs such that $(V, E \bigtriangleup F)$ is a cograph (or $P_4$-free graph), where $\bigtriangleup$ represents the symmetric difference operator. We also describe a technique for speeding up the performance of the algorithm in practice. Additionally, we present a heuristic for solving the Cograph Editing problem which produces good results on small to medium datasets. In application it is much more important to find the ground truth, not some optimal solution. For the first time, we evaluate whether the cograph property is strict enough to recover the true graph from data to which noise has been added.
△ Less
Submitted 10 January, 2018; v1 submitted 15 November, 2017;
originally announced November 2017.
-
Generative Adversarial Networks: An Overview
Authors:
Antonia Creswell,
Tom White,
Vincent Dumoulin,
Kai Arulkumaran,
Biswa Sengupta,
Anil A Bharath
Abstract:
Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this through deriving backpropagation signals through a competitive process involving a pair of networks. The representations that can be learned by GANs may be used in a variety of applications, including image synthesis, semantic image editing, style transf…
▽ More
Generative adversarial networks (GANs) provide a way to learn deep representations without extensively annotated training data. They achieve this through deriving backpropagation signals through a competitive process involving a pair of networks. The representations that can be learned by GANs may be used in a variety of applications, including image synthesis, semantic image editing, style transfer, image super-resolution and classification. The aim of this review paper is to provide an overview of GANs for the signal processing community, drawing on familiar analogies and concepts where possible. In addition to identifying different methods for training and constructing GANs, we also point to remaining challenges in their theory and application.
△ Less
Submitted 19 October, 2017;
originally announced October 2017.
-
Sampling Generative Networks
Authors:
Tom White
Abstract:
We introduce several techniques for sampling and visualizing the latent spaces of generative models. Replacing linear interpolation with spherical linear interpolation prevents diverging from a model's prior distribution and produces sharper samples. J-Diagrams and MINE grids are introduced as visualizations of manifolds created by analogies and nearest neighbors. We demonstrate two new techniques…
▽ More
We introduce several techniques for sampling and visualizing the latent spaces of generative models. Replacing linear interpolation with spherical linear interpolation prevents diverging from a model's prior distribution and produces sharper samples. J-Diagrams and MINE grids are introduced as visualizations of manifolds created by analogies and nearest neighbors. We demonstrate two new techniques for deriving attribute vectors: bias-corrected vectors with data replication and synthetic vectors with data augmentation. Binary classification using attribute vectors is presented as a technique supporting quantitative analysis of the latent space. Most techniques are intended to be independent of model type and examples are shown on both Variational Autoencoders and Generative Adversarial Networks.
△ Less
Submitted 6 December, 2016; v1 submitted 14 September, 2016;
originally announced September 2016.
-
Apple IOS Devices for Network Administrators
Authors:
Timur Mirzoev,
Gerard Gingo,
Mike Stawchansky,
Tracy White
Abstract:
As tablet devices continue to gain market share at the expense of the traditional PC, they become a more integral part of the corporate landscape. Tablets are no longer being utilized only by sales executives for presentation purposes, or as addition to the traditional laptop. Users are attempting to perform significant amounts of their daily work on tablet devices, some even abandoning the ubiqui…
▽ More
As tablet devices continue to gain market share at the expense of the traditional PC, they become a more integral part of the corporate landscape. Tablets are no longer being utilized only by sales executives for presentation purposes, or as addition to the traditional laptop. Users are attempting to perform significant amounts of their daily work on tablet devices, some even abandoning the ubiquitous laptop or desktop entirely. Operating exclusively from a tablet device, specifically Apple IOS tablet devices creates unique challenges in a corporate environment traditionally dominated by Microsoft Windows operating systems. Interactions with file shares, presentation media, VPN, and remote access present barriers that users and helpdesk support are unfamiliar with in a relation to an iPad or iPhone. Many solutions are being offered to these challenges some of which are analyzed by this manuscript.
△ Less
Submitted 8 April, 2014;
originally announced April 2014.
-
Computational Modeling for the Activation Cycle of G-proteins by G-protein-coupled Receptors
Authors:
Yifei Bao,
Adriana Compagnoni,
Joseph Glavy,
Tommy White
Abstract:
In this paper, we survey five different computational modeling methods. For comparison, we use the activation cycle of G-proteins that regulate cellular signaling events downstream of G-protein-coupled receptors (GPCRs) as a driving example. Starting from an existing Ordinary Differential Equations (ODEs) model, we implement the G-protein cycle in the stochastic Pi-calculus using SPiM, as Petri-ne…
▽ More
In this paper, we survey five different computational modeling methods. For comparison, we use the activation cycle of G-proteins that regulate cellular signaling events downstream of G-protein-coupled receptors (GPCRs) as a driving example. Starting from an existing Ordinary Differential Equations (ODEs) model, we implement the G-protein cycle in the stochastic Pi-calculus using SPiM, as Petri-nets using Cell Illustrator, in the Kappa Language using Cellucidate, and in Bio-PEPA using the Bio-PEPA eclipse plug in. We also provide a high-level notation to abstract away from communication primitives that may be unfamiliar to the average biologist, and we show how to translate high-level programs into stochastic Pi-calculus processes and chemical reactions.
△ Less
Submitted 1 November, 2010;
originally announced November 2010.
