-
Embedding Compression for Efficient Re-Identification
Authors:
Luke McDermott
Abstract:
Real world re-identfication (ReID) algorithms aim to map new observations of an object to previously recorded instances. These systems are often constrained by quantity and size of the stored embeddings. To combat this scaling problem, we attempt to shrink the size of these vectors by using a variety of compression techniques. In this paper, we benchmark quantization-aware-training along with thre…
▽ More
Real world re-identfication (ReID) algorithms aim to map new observations of an object to previously recorded instances. These systems are often constrained by quantity and size of the stored embeddings. To combat this scaling problem, we attempt to shrink the size of these vectors by using a variety of compression techniques. In this paper, we benchmark quantization-aware-training along with three different dimension reduction methods: iterative structured pruning, slicing the embeddings at initialize, and using low rank embeddings. We find that ReID embeddings can be compressed by up to 96x with minimal drop in performance. This implies that modern re-identification paradigms do not fully leverage the high dimensional latent space, opening up further research to increase the capabilities of these systems.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Neural Architecture Codesign for Fast Bragg Peak Analysis
Authors:
Luke McDermott,
Jason Weitz,
Dmitri Demler,
Daniel Cummings,
Nhan Tran,
Javier Duarte
Abstract:
We develop an automated pipeline to streamline neural architecture codesign for fast, real-time Bragg peak analysis in high-energy diffraction microscopy. Traditional approaches, notably pseudo-Voigt fitting, demand significant computational resources, prompting interest in deep learning models for more efficient solutions. Our method employs neural architecture search and AutoML to enhance these…
▽ More
We develop an automated pipeline to streamline neural architecture codesign for fast, real-time Bragg peak analysis in high-energy diffraction microscopy. Traditional approaches, notably pseudo-Voigt fitting, demand significant computational resources, prompting interest in deep learning models for more efficient solutions. Our method employs neural architecture search and AutoML to enhance these models, including hardware costs, leading to the discovery of more hardware-efficient neural architectures. Our results match the performance, while achieving a 13$\times$ reduction in bit operations compared to the previous state-of-the-art. We show further speedup through model compression techniques such as quantization-aware-training and neural network pruning. Additionally, our hierarchical search space provides greater flexibility in optimization, which can easily extend to other tasks and domains.
△ Less
Submitted 11 December, 2023; v1 submitted 10 December, 2023;
originally announced December 2023.
-
UniCat: Crafting a Stronger Fusion Baseline for Multimodal Re-Identification
Authors:
Jennifer Crawford,
Haoli Yin,
Luke McDermott,
Daniel Cummings
Abstract:
Multimodal Re-Identification (ReID) is a popular retrieval task that aims to re-identify objects across diverse data streams, prompting many researchers to integrate multiple modalities into a unified representation. While such fusion promises a holistic view, our investigations shed light on potential pitfalls. We uncover that prevailing late-fusion techniques often produce suboptimal latent repr…
▽ More
Multimodal Re-Identification (ReID) is a popular retrieval task that aims to re-identify objects across diverse data streams, prompting many researchers to integrate multiple modalities into a unified representation. While such fusion promises a holistic view, our investigations shed light on potential pitfalls. We uncover that prevailing late-fusion techniques often produce suboptimal latent representations when compared to methods that train modalities in isolation. We argue that this effect is largely due to the inadvertent relaxation of the training objectives on individual modalities when using fusion, what others have termed modality laziness. We present a nuanced point-of-view that this relaxation can lead to certain modalities failing to fully harness available task-relevant information, and yet, offers a protective veil to noisy modalities, preventing them from overfitting to task-irrelevant data. Our findings also show that unimodal concatenation (UniCat) and other late-fusion ensembling of unimodal backbones, when paired with best-known training techniques, exceed the current state-of-the-art performance across several multimodal ReID benchmarks. By unveiling the double-edged sword of "modality laziness", we motivate future research in balancing local modality strengths with global representations.
△ Less
Submitted 28 October, 2023;
originally announced October 2023.
-
Linear Mode Connectivity in Sparse Neural Networks
Authors:
Luke McDermott,
Daniel Cummings
Abstract:
With the rise in interest of sparse neural networks, we study how neural network pruning with synthetic data leads to sparse networks with unique training properties. We find that distilled data, a synthetic summarization of the real data, paired with Iterative Magnitude Pruning (IMP) unveils a new class of sparse networks that are more stable to SGD noise on the real data, than either the dense m…
▽ More
With the rise in interest of sparse neural networks, we study how neural network pruning with synthetic data leads to sparse networks with unique training properties. We find that distilled data, a synthetic summarization of the real data, paired with Iterative Magnitude Pruning (IMP) unveils a new class of sparse networks that are more stable to SGD noise on the real data, than either the dense model, or subnetworks found with real data in IMP. That is, synthetically chosen subnetworks often train to the same minima, or exhibit linear mode connectivity. We study this through linear interpolation, loss landscape visualizations, and measuring the diagonal of the hessian. While dataset distillation as a field is still young, we find that these properties lead to synthetic subnetworks matching the performance of traditional IMP with up to 150x less training points in settings where distilled data applies.
△ Less
Submitted 28 October, 2023;
originally announced October 2023.
-
GraFT: Gradual Fusion Transformer for Multimodal Re-Identification
Authors:
Haoli Yin,
Jiayao Li,
Eva Schiller,
Luke McDermott,
Daniel Cummings
Abstract:
Object Re-Identification (ReID) is pivotal in computer vision, witnessing an escalating demand for adept multimodal representation learning. Current models, although promising, reveal scalability limitations with increasing modalities as they rely heavily on late fusion, which postpones the integration of specific modality insights. Addressing this, we introduce the \textbf{Gradual Fusion Transfor…
▽ More
Object Re-Identification (ReID) is pivotal in computer vision, witnessing an escalating demand for adept multimodal representation learning. Current models, although promising, reveal scalability limitations with increasing modalities as they rely heavily on late fusion, which postpones the integration of specific modality insights. Addressing this, we introduce the \textbf{Gradual Fusion Transformer (GraFT)} for multimodal ReID. At its core, GraFT employs learnable fusion tokens that guide self-attention across encoders, adeptly capturing both modality-specific and object-specific features. Further bolstering its efficacy, we introduce a novel training paradigm combined with an augmented triplet loss, optimizing the ReID feature embedding space. We demonstrate these enhancements through extensive ablation studies and show that GraFT consistently surpasses established multimodal ReID benchmarks. Additionally, aiming for deployment versatility, we've integrated neural network pruning into GraFT, offering a balance between model size and performance.
△ Less
Submitted 24 October, 2023;
originally announced October 2023.
-
A Generalization of Continuous Relaxation in Structured Pruning
Authors:
Brad Larson,
Bishal Upadhyaya,
Luke McDermott,
Siddha Ganju
Abstract:
Deep learning harnesses massive parallel floating-point processing to train and evaluate large neural networks. Trends indicate that deeper and larger neural networks with an increasing number of parameters achieve higher accuracy than smaller neural networks. This performance improvement, which often requires heavy compute for both training and evaluation, eventually needs to translate well to re…
▽ More
Deep learning harnesses massive parallel floating-point processing to train and evaluate large neural networks. Trends indicate that deeper and larger neural networks with an increasing number of parameters achieve higher accuracy than smaller neural networks. This performance improvement, which often requires heavy compute for both training and evaluation, eventually needs to translate well to resource-constrained hardware for practical value. Structured pruning asserts that while large networks enable us to find solutions to complex computer vision problems, a smaller, computationally efficient sub-network can be derived from the large neural network that retains model accuracy but significantly improves computational efficiency.
We generalize structured pruning with algorithms for network augmentation, pruning, sub-network collapse and removal. In addition, we demonstrate efficient and stable convergence up to 93% sparsity and 95% FLOPs reduction without loss of inference accuracy using with continuous relaxation matching or exceeding the state of the art for all structured pruning methods. The resulting CNN executes efficiently on GPU hardware without computationally expensive sparse matrix operations. We achieve this with routine automatable operations on classification and segmentation problems using CIFAR-10, ImageNet, and CityScapes datasets with the ResNet and U-NET network architectures.
△ Less
Submitted 28 August, 2023;
originally announced August 2023.
-
Distilled Pruning: Using Synthetic Data to Win the Lottery
Authors:
Luke McDermott,
Daniel Cummings
Abstract:
This work introduces a novel approach to pruning deep learning models by using distilled data. Unlike conventional strategies which primarily focus on architectural or algorithmic optimization, our method reconsiders the role of data in these scenarios. Distilled datasets capture essential patterns from larger datasets, and we demonstrate how to leverage this capability to enable a computationally…
▽ More
This work introduces a novel approach to pruning deep learning models by using distilled data. Unlike conventional strategies which primarily focus on architectural or algorithmic optimization, our method reconsiders the role of data in these scenarios. Distilled datasets capture essential patterns from larger datasets, and we demonstrate how to leverage this capability to enable a computationally efficient pruning process. Our approach can find sparse, trainable subnetworks (a.k.a. Lottery Tickets) up to 5x faster than Iterative Magnitude Pruning at comparable sparsity on CIFAR-10. The experimental results highlight the potential of using distilled data for resource-efficient neural network pruning, model compression, and neural architecture search.
△ Less
Submitted 8 August, 2023; v1 submitted 6 July, 2023;
originally announced July 2023.
-
Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA Communicating with Computers Program
Authors:
Robyn Kozierok,
John Aberdeen,
Cheryl Clark,
Christopher Garay,
Bradley Goodman,
Tonia Korves,
Lynette Hirschman,
Patricia L. McDermott,
Matthew W. Peterson
Abstract:
There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework ha…
▽ More
There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria.
We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success -- capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.
△ Less
Submitted 9 February, 2021;
originally announced February 2021.
-
Deep Echo State Networks with Uncertainty Quantification for Spatio-Temporal Forecasting
Authors:
Patrick L. McDermott,
Christopher K. Wikle
Abstract:
Long-lead forecasting for spatio-temporal systems can often entail complex nonlinear dynamics that are difficult to specify it a priori. Current statistical methodologies for modeling these processes are often highly parameterized and thus, challenging to implement from a computational perspective. One potential parsimonious solution to this problem is a method from the dynamical systems and engin…
▽ More
Long-lead forecasting for spatio-temporal systems can often entail complex nonlinear dynamics that are difficult to specify it a priori. Current statistical methodologies for modeling these processes are often highly parameterized and thus, challenging to implement from a computational perspective. One potential parsimonious solution to this problem is a method from the dynamical systems and engineering literature referred to as an echo state network (ESN). ESN models use so-called {\it reservoir computing} to efficiently compute recurrent neural network (RNN) forecasts. Moreover, so-called "deep" models have recently been shown to be successful at predicting high-dimensional complex nonlinear processes, particularly those with multiple spatial and temporal scales of variability (such as we often find in spatio-temporal environmental data). Here we introduce a deep ensemble ESN (D-EESN) model. We present two versions of this model for spatio-temporal processes that both produce forecasts and associated measures of uncertainty. The first approach utilizes a bootstrap ensemble framework and the second is developed within a hierarchical Bayesian framework (BD-EESN). This more general hierarchical Bayesian framework naturally accommodates non-Gaussian data types and multiple levels of uncertainties. The methodology is first applied to a data set simulated from a novel non-Gaussian multiscale Lorenz-96 dynamical system simulation model and then to a long-lead United States (U.S.) soil moisture forecasting application.
△ Less
Submitted 3 September, 2018; v1 submitted 27 June, 2018;
originally announced June 2018.
-
Transparent Migration of Multi-Threaded Applications on a Java Based Grid
Authors:
T. N. Ellahi,
B. Hudzia,
L. McDermott,
T. Kechadi
Abstract:
Grid computing has enabled pooling a very large number of heterogeneous resource administered by different security domains. Applications are dynamically deployed on the resources available at the time. Dynamic nature of the resources and applications requirements makes needs the grid middleware to support the ability of migrating a running application to a different resource. Especially, Grid a…
▽ More
Grid computing has enabled pooling a very large number of heterogeneous resource administered by different security domains. Applications are dynamically deployed on the resources available at the time. Dynamic nature of the resources and applications requirements makes needs the grid middleware to support the ability of migrating a running application to a different resource. Especially, Grid applications are typically long running and thus sto** them and starting them from scratch is not a feasible option. This paper presents an overview of migration support in a java based grid middleware called DGET. Migration support in DGET includes multi-threaded migration and asynchronous migration as well.
△ Less
Submitted 29 August, 2006;
originally announced August 2006.
-
A Java Based Architecture of P2P-Grid Middleware
Authors:
B. Hudzia,
T. N. Ellahi,
L. McDermott,
T. Kechadi
Abstract:
During the last decade there has been a huge interest in Grid technologies, and numerous Grid projects have been initiated with various visions of the Grid. While all these visions have the same goal of resource sharing, they differ in the functionality that a Grid supports, characterization, programming environments, etc. We present a new Grid system dedicated to dealing with data issues, calle…
▽ More
During the last decade there has been a huge interest in Grid technologies, and numerous Grid projects have been initiated with various visions of the Grid. While all these visions have the same goal of resource sharing, they differ in the functionality that a Grid supports, characterization, programming environments, etc. We present a new Grid system dedicated to dealing with data issues, called DGET (Data Grid Environment and Tools). DGET is characterized by its peerto- peer communication system and entity-based architecture, therefore, taking advantage of the main functionality of both systems; P2P and Grid. DGET is currently under development and a prototype implementing the main components is in its first phase of testing. In this paper we limit our description to the system architectural features and to the main differences with other systems. Keywords: Grid Computing, Peer to Peer, Peer to Peer Grid
△ Less
Submitted 29 August, 2006;
originally announced August 2006.
-
Entity Based Peer-to-Peer in a Data Grid Environment
Authors:
B. Hudzia,
L. McDermott,
T. N. Illahi,
M-T. Kechadi
Abstract:
During the last decade there has been a huge interest in Grid technologies, and numerous Grid projects have been initiated with various visions of the Grid. While all these visions have the same goal of resource sharing, they differ in the functionality that a Grid supports, the grid characterisation, programming environments, etc. In this paper we present a new Grid system dedicated to deal wit…
▽ More
During the last decade there has been a huge interest in Grid technologies, and numerous Grid projects have been initiated with various visions of the Grid. While all these visions have the same goal of resource sharing, they differ in the functionality that a Grid supports, the grid characterisation, programming environments, etc. In this paper we present a new Grid system dedicated to deal with data issues, called DGET (Data Grid Environment and Tools). DGET is characterized by its peer-to-peer communication system and entity-based architecture, therefore, taking advantage of the main functionality of both systems; P2P and Grid. DGET is currently under development and a prototype implementing the main components is in its first phase of testing. In this paper we limit our description to the system architectural features and to the main differences with other systems.
△ Less
Submitted 29 August, 2006;
originally announced August 2006.
