Showing 1–46 of 46 results for author: El-Khamy, M

1st Place Winner of the 2024 Pixel-level Video Understanding in the Wild (CVPR'24 PVUW) Challenge in Video Panoptic Segmentation and Best Long Video Consistency of Video Semantic Segmentation

Authors: Qingfeng Liu, Mostafa El-Khamy, Kee-Bong Song

Abstract: The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wi… ▽ More The third Pixel-level Video Understanding in the Wild (PVUW CVPR 2024) challenge aims to advance the state of art in video understanding through benchmarking Video Panoptic Segmentation (VPS) and Video Semantic Segmentation (VSS) on challenging videos and scenes introduced in the large-scale Video Panoptic Segmentation in the Wild (VIPSeg) test set and the large-scale Video Scene Parsing in the Wild (VSPW) test set, respectively. This paper details our research work that achieved the 1st place winner in the PVUW'24 VPS challenge, establishing state of art results in all metrics, including the Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). With minor fine-tuning our approach also achieved the 3rd place in the PVUW'24 VSS challenge ranked by the mIoU (mean intersection over union) metric and the first place ranked by the VC16 (16-frame video consistency) metric. Our winning solution stands on the shoulders of giant foundational vision transformer model (DINOv2 ViT-g) and proven multi-stage Decoupled Video Instance Segmentation (DVIS) frameworks for video understanding. △ Less

Submitted 8 June, 2024; originally announced June 2024.

PPG-to-ECG Signal Translation for Continuous Atrial Fibrillation Detection via Attention-based Deep State-Space Modeling

Authors: Khuong Vo, Mostafa El-Khamy, Yoo** Choi

Abstract: Photoplethysmography (PPG) is a cost-effective and non-invasive technique that utilizes optical methods to measure cardiac physiology. PPG has become increasingly popular in health monitoring and is used in various commercial and clinical wearable devices. Compared to electrocardiography (ECG), PPG does not provide substantial clinical diagnostic value, despite the strong correlation between the t… ▽ More Photoplethysmography (PPG) is a cost-effective and non-invasive technique that utilizes optical methods to measure cardiac physiology. PPG has become increasingly popular in health monitoring and is used in various commercial and clinical wearable devices. Compared to electrocardiography (ECG), PPG does not provide substantial clinical diagnostic value, despite the strong correlation between the two. Here, we propose a subject-independent attention-based deep state-space model (ADSSM) to translate PPG signals to corresponding ECG waveforms. The model is not only robust to noise but also data-efficient by incorporating probabilistic prior knowledge. To evaluate our approach, 55 subjects' data from the MIMIC-III database were used in their original form, and then modified with noise, mimicking real-world scenarios. Our approach was proven effective as evidenced by the PR-AUC of 0.986 achieved when inputting the translated ECG signals into an existing atrial fibrillation (AFib) detector. ADSSM enables the integration of ECG's extensive knowledge base and PPG's continuous measurement for early diagnosis of cardiovascular disease. △ Less

Submitted 12 June, 2024; v1 submitted 26 September, 2023; originally announced September 2023.

Comments: Accepted to 46th IEEE EMBC

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models

Authors: Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, Salman Avestimehr

Abstract: Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devi… ▽ More Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying parameter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specifically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with approximately $\sim 1\%$ density while reducing training time by up to $90\%$. △ Less

Submitted 12 August, 2023; originally announced August 2023.

Zero-Shot Learning of a Conditional Generative Adversarial Network for Data-Free Network Quantization

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: We propose a novel method for training a conditional generative adversarial network (CGAN) without the use of training data, called zero-shot learning of a CGAN (ZS-CGAN). Zero-shot learning of a conditional generator only needs a pre-trained discriminative (classification) model and does not need any training data. In particular, the conditional generator is trained to produce labeled synthetic s… ▽ More We propose a novel method for training a conditional generative adversarial network (CGAN) without the use of training data, called zero-shot learning of a CGAN (ZS-CGAN). Zero-shot learning of a conditional generator only needs a pre-trained discriminative (classification) model and does not need any training data. In particular, the conditional generator is trained to produce labeled synthetic samples whose characteristics mimic the original training data by using the statistics stored in the batch normalization layers of the pre-trained model. We show the usefulness of ZS-CGAN in data-free quantization of deep neural networks. We achieved the state-of-the-art data-free network quantization of the ResNet and MobileNet classification models trained on the ImageNet dataset. Data-free quantization using ZS-CGAN showed a minimal loss in accuracy compared to that obtained by conventional data-dependent quantization. △ Less

Submitted 25 October, 2022; originally announced October 2022.

Comments: IEEE ICIP 2021

Toward Sustainable Continual Learning: Detection and Knowledge Repurposing of Similar Tasks

Authors: Sijia Wang, Yoo** Choi, Junya Chen, Mostafa El-Khamy, Ricardo Henao

Abstract: Most existing works on continual learning (CL) focus on overcoming the catastrophic forgetting (CF) problem, with dynamic models and replay methods performing exceptionally well. However, since current works tend to assume exclusivity or dissimilarity among learning tasks, these methods require constantly accumulating task-specific knowledge in memory for each task. This results in the eventual pr… ▽ More Most existing works on continual learning (CL) focus on overcoming the catastrophic forgetting (CF) problem, with dynamic models and replay methods performing exceptionally well. However, since current works tend to assume exclusivity or dissimilarity among learning tasks, these methods require constantly accumulating task-specific knowledge in memory for each task. This results in the eventual prohibitive expansion of the knowledge repository if we consider learning from a long sequence of tasks. In this work, we introduce a paradigm where the continual learner gets a sequence of mixed similar and dissimilar tasks. We propose a new continual learning framework that uses a task similarity detection function that does not require additional learning, with which we analyze whether there is a specific task in the past that is similar to the current task. We can then reuse previous task knowledge to slow down parameter expansion, ensuring that the CL system expands the knowledge repository sublinearly to the number of learned tasks. Our experiments show that the proposed framework performs competitively on widely used computer vision benchmarks such as CIFAR10, CIFAR100, and EMNIST. △ Less

Submitted 11 October, 2022; originally announced October 2022.

Dual-Teacher Class-Incremental Learning With Data-Free Generative Replay

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our… ▽ More This paper proposes two novel knowledge transfer techniques for class-incremental learning (CIL). First, we propose data-free generative replay (DF-GR) to mitigate catastrophic forgetting in CIL by using synthetic samples from a generative model. In the conventional generative replay, the generative model is pre-trained for old data and shared in extra memory for later incremental learning. In our proposed DF-GR, we train a generative model from scratch without using any training data, based on the pre-trained classification model from the past, so we curtail the cost of sharing pre-trained generative models. Second, we introduce dual-teacher information distillation (DT-ID) for knowledge distillation from two teachers to one student. In CIL, we use DT-ID to learn new classes incrementally based on the pre-trained model for old classes and another model (pre-)trained on the new data for new classes. We implemented the proposed schemes on top of one of the state-of-the-art CIL methods and showed the performance improvement on CIFAR-100 and ImageNet datasets. △ Less

Submitted 17 June, 2021; originally announced June 2021.

Comments: CVPR 2021 Workshop on Continual Learning in Computer Vision (CLVision)

Towards Fair Federated Learning with Zero-Shot Data Augmentation

Authors: Weituo Hao, Mostafa El-Khamy, Jungwon Lee, Jianyi Zhang, Kevin J Liang, Changyou Chen, Lawrence Carin

Abstract: Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. Although it is recognized that statistical heterogeneity of the client local data yields slower global model convergence, it is less commonly recognized that it also yields a biased federated global model w… ▽ More Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. Although it is recognized that statistical heterogeneity of the client local data yields slower global model convergence, it is less commonly recognized that it also yields a biased federated global model with a high variance of accuracy across clients. In this work, we aim to provide federated learning schemes with improved fairness. To tackle this challenge, we propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server). Empirical results on a suite of datasets demonstrate the effectiveness of our methods on simultaneously improving the test accuracy and fairness. △ Less

Submitted 27 April, 2021; originally announced April 2021.

Comments: Accepted by IEEE CVPR Workshop on Fair, Data Efficient And Trusted Computer Vision

MLPerf Mobile Inference Benchmark

Authors: Vijay Janapa Reddi, David Kanter, Peter Mattson, Jared Duke, Thai Nguyen, Ramesh Chukka, Ken Shiring, Koan-Sin Tan, Mark Charlebois, William Chou, Mostafa El-Khamy, Jungwook Hong, Tom St. John, Cindy Trinh, Michael Buch, Mark Mazumder, Relia Markovic, Thomas Atta, Fatih Cakir, Masoud Charkhabi, Xiaodong Chen, Cheng-Ming Chiang, Dave Dexter, Terry Heo, Gunther Schmuelling , et al. (2 additional authors not shown)

Abstract: This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise of leading mobile-SoC vendors, ML-framework providers, and model producers. It comprises a suite of models that operate with standard data sets, quality metrics… ▽ More This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks. The benchmark draws from the expertise of leading mobile-SoC vendors, ML-framework providers, and model producers. It comprises a suite of models that operate with standard data sets, quality metrics and run rules. We describe the design and implementation of this domain-specific ML benchmark. The current benchmark version comes as a mobile app for different computer vision and natural language processing tasks. The benchmark also supports non-smartphone devices, such as laptops and mobile PCs. Benchmark results from the first two rounds reveal the overwhelming complexity of the underlying mobile ML system stack, emphasizing the need for transparency in mobile ML performance analysis. The results also show that the strides being made all through the ML stack improve performance. Within six months, offline throughput improved by 3x, while latency reduced by as much as 12x. ML is an evolving field with changing use cases, models, data sets and quality targets. MLPerf Mobile will evolve and serve as an open-source community framework to guide research and innovation for mobile AI. △ Less

Submitted 6 April, 2022; v1 submitted 3 December, 2020; originally announced December 2020.

WAFFLe: Weight Anonymized Factorization for Federated Learning

Authors: Weituo Hao, Nikhil Mehta, Kevin J Liang, Pengyu Cheng, Mostafa El-Khamy, Lawrence Carin

Abstract: In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices. In light of this need, federated learning has emerged as a popular training paradigm. However, many federated learning approaches trade transmitting data for communicating updated weight parameters for each local device. Therefore,… ▽ More In domains where data are sensitive or private, there is great value in methods that can learn in a distributed manner without the data ever leaving the local devices. In light of this need, federated learning has emerged as a popular training paradigm. However, many federated learning approaches trade transmitting data for communicating updated weight parameters for each local device. Therefore, a successful breach that would have otherwise directly compromised the data instead grants whitebox access to the local model, which opens the door to a number of attacks, including exposing the very data federated learning seeks to protect. Additionally, in distributed scenarios, individual client devices commonly exhibit high statistical heterogeneity. Many common federated approaches learn a single global model; while this may do well on average, performance degrades when the i.i.d. assumption is violated, underfitting individuals further from the mean, and raising questions of fairness. To address these issues, we propose Weight Anonymized Factorization for Federated Learning (WAFFLe), an approach that combines the Indian Buffet Process with a shared dictionary of weight factors for neural networks. Experiments on MNIST, FashionMNIST, and CIFAR-10 demonstrate WAFFLe's significant improvement to local test performance and fairness while simultaneously providing an extra layer of security. △ Less

Submitted 13 August, 2020; originally announced August 2020.

Data-Free Network Quantization With Adversarial Knowledge Distillation

Authors: Yoo** Choi, Jihwan Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are ge… ▽ More Network quantization is an essential procedure in deep learning for development of efficient fixed-point inference models on mobile or edge platforms. However, as datasets grow larger and privacy regulations become stricter, data sharing for model compression gets more difficult and restricted. In this paper, we consider data-free network quantization with synthetic data. The synthetic data are generated from a generator, while no data are used in training the generator and in quantization. To this end, we propose data-free adversarial knowledge distillation, which minimizes the maximum distance between the outputs of the teacher and the (quantized) student for any adversarial samples from a generator. To generate adversarial samples similar to the original data, we additionally propose matching statistics from the batch normalization layers for generated data and the original data in the teacher. Furthermore, we show the gain of producing diverse adversarial samples by using multiple generators and multiple students. Our experiments show the state-of-the-art data-free model compression and quantization results for (wide) residual networks and MobileNet on SVHN, CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The accuracy losses compared to using the original datasets are shown to be very minimal. △ Less

Submitted 8 May, 2020; originally announced May 2020.

Comments: CVPR 2020 Joint Workshop on Efficient Deep Learning in Computer Vision (EDLCV)

NTIRE 2020 Challenge on Real-World Image Super-Resolution: Methods and Results

Authors: Andreas Lugmayr, Martin Danelljan, Radu Timofte, Namhyuk Ahn, Dongwoon Bai, Jie Cai, Yun Cao, Junyang Chen, Kaihua Cheng, SeYoung Chun, Wei Deng, Mostafa El-Khamy, Chiu Man Ho, Xiaozhong Ji, Amin Kheradmand, Gwantae Kim, Hanseok Ko, Kanghyu Lee, Jungwon Lee, Hao Li, Ziluan Liu, Zhi-Song Liu, Shuai Liu, Yunhua Lu, Zibo Meng , et al. (21 additional authors not shown)

Abstract: This paper reviews the NTIRE 2020 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided along with a set of unpaired high-quality target images. In Track 1: Image Proc… ▽ More This paper reviews the NTIRE 2020 challenge on real world super-resolution. It focuses on the participating methods and final results. The challenge addresses the real world setting, where paired true high and low-resolution images are unavailable. For training, only one set of source input images is therefore provided along with a set of unpaired high-quality target images. In Track 1: Image Processing artifacts, the aim is to super-resolve images with synthetically generated image processing artifacts. This allows for quantitative benchmarking of the approaches \wrt a ground-truth image. In Track 2: Smartphone Images, real low-quality smart phone images have to be super-resolved. In both tracks, the ultimate goal is to achieve the best perceptual quality, evaluated using a human study. This is the second challenge on the subject, following AIM 2019, targeting to advance the state-of-the-art in super-resolution. To measure the performance we use the benchmark protocol from AIM 2019. In total 22 teams competed in the final testing phase, demonstrating new and innovative solutions to the problem. △ Less

Submitted 5 May, 2020; originally announced May 2020.

GSANet: Semantic Segmentation with Global and Selective Attention

Authors: Qingfeng Liu, Mostafa El-Khamy, Dongwoon Bai, Jungwon Lee

Abstract: This paper proposes a novel deep learning architecture for semantic segmentation. The proposed Global and Selective Attention Network (GSANet) features Atrous Spatial Pyramid Pooling (ASPP) with a novel sparsemax global attention and a novel selective attention that deploys a condensation and diffusion mechanism to aggregate the multi-scale contextual information from the extracted deep features.… ▽ More This paper proposes a novel deep learning architecture for semantic segmentation. The proposed Global and Selective Attention Network (GSANet) features Atrous Spatial Pyramid Pooling (ASPP) with a novel sparsemax global attention and a novel selective attention that deploys a condensation and diffusion mechanism to aggregate the multi-scale contextual information from the extracted deep features. A selective attention decoder is also proposed to process the GSA-ASPP outputs for optimizing the softmax volume. We are the first to benchmark the performance of semantic segmentation networks with the low-complexity feature extraction network (FXN) MobileNetEdge, that is optimized for low latency on edge devices. We show that GSANet can result in more accurate segmentation with MobileNetEdge, as well as with strong FXNs, such as Xception. GSANet improves the state-of-art semantic segmentation accuracy on both the ADE20k and the Cityscapes datasets. △ Less

Submitted 13 February, 2020; originally announced March 2020.

HyperCon: Image-To-Video Model Transfer for Video-To-Video Translation Tasks

Authors: Ryan Szeto, Mostafa El-Khamy, Jungwon Lee, Jason J. Corso

Abstract: Video-to-video translation is more difficult than image-to-image translation due to the temporal consistency problem that, if unaddressed, leads to distracting flickering effects. Although video models designed from scratch produce temporally consistent results, training them to match the vast visual knowledge captured by image models requires an intractable number of videos. To combine the benefi… ▽ More Video-to-video translation is more difficult than image-to-image translation due to the temporal consistency problem that, if unaddressed, leads to distracting flickering effects. Although video models designed from scratch produce temporally consistent results, training them to match the vast visual knowledge captured by image models requires an intractable number of videos. To combine the benefits of image and video models, we propose an image-to-video model transfer method called Hyperconsistency (HyperCon) that transforms any well-trained image model into a temporally consistent video model without fine-tuning. HyperCon works by translating a temporally interpolated video frame-wise and then aggregating over temporally localized windows on the interpolated video. It handles both masked and unmasked inputs, enabling support for even more video-to-video translation tasks than prior image-to-video model transfer techniques. We demonstrate HyperCon on video style transfer and inpainting, where it performs favorably compared to prior state-of-the-art methods without training on a single stylized or incomplete video. Our project website is available at https://ryanszeto.com/projects/hypercon . △ Less

Submitted 10 November, 2020; v1 submitted 10 December, 2019; originally announced December 2019.

Comments: Accepted to WACV 2021

End-to-End Multi-Task Denoising for the Joint Optimization of Perceptual Speech Metrics

Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: Although supervised learning based on a deep neural network has recently achieved substantial improvement on speech enhancement, the existing schemes have either of two critical issues: spectrum or metric mismatches. The spectrum mismatch is a well known issue that any spectrum modification after short-time Fourier transform (STFT), in general, cannot be fully recovered after inverse short-time Fo… ▽ More Although supervised learning based on a deep neural network has recently achieved substantial improvement on speech enhancement, the existing schemes have either of two critical issues: spectrum or metric mismatches. The spectrum mismatch is a well known issue that any spectrum modification after short-time Fourier transform (STFT), in general, cannot be fully recovered after inverse short-time Fourier transform (ISTFT). The metric mismatch is that a conventional mean square error (MSE) loss function is typically sub-optimal to maximize perceptual speech measure such as signal-to-distortion ratio (SDR), perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI). This paper presents a new end-to-end denoising framework. First, the network optimization is performed on the time-domain signals after ISTFT to avoid the spectrum mismatch. Second, three loss functions based on SDR, PESQ and STOI are proposed to minimize the metric mismatch. The experimental result showed the proposed denoising scheme significantly improved SDR, PESQ and STOI performance over the existing methods. Moreover, the proposed scheme also provided good generalization performance over generative denoising models on the perceptual speech metrics not used as a loss function during training. △ Less

Submitted 5 May, 2020; v1 submitted 23 October, 2019; originally announced October 2019.

Comments: 5 pages, submitted to Interspeech 2020. arXiv admin note: substantial text overlap with arXiv:1901.09146

T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancement

Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: Transformer neural networks (TNN) demonstrated state-of-art performance on many natural language processing (NLP) tasks, replacing recurrent neural networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in speech enhancement, whose contextual nature is different than NLP tasks, like machine translation. Self-attention is a core building block of the Transformer, which not only… ▽ More Transformer neural networks (TNN) demonstrated state-of-art performance on many natural language processing (NLP) tasks, replacing recurrent neural networks (RNNs), such as LSTMs or GRUs. However, TNNs did not perform well in speech enhancement, whose contextual nature is different than NLP tasks, like machine translation. Self-attention is a core building block of the Transformer, which not only enables parallelization of sequence computation, but also provides the constant path length between symbols that is essential to learning long-range dependencies. In this paper, we propose a Transformer with Gaussian-weighted self-attention (T-GSA), whose attention weights are attenuated according to the distance between target and context symbols. The experimental results show that the proposed T-GSA has significantly improved speech-enhancement performance, compared to the Transformer and RNNs. △ Less

Submitted 11 February, 2020; v1 submitted 13 October, 2019; originally announced October 2019.

Comments: 5 pages, Submitted to ICASSP 2020

Variable Rate Deep Image Compression With a Conditional Autoencoder

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: In this paper, we propose a novel variable-rate learned image compression framework with a conditional autoencoder. Previous learning-based image compression methods mostly require training separate networks for different compression rates so they can yield compressed images of varying quality. In contrast, we train and deploy only one variable-rate image compression network implemented with a con… ▽ More In this paper, we propose a novel variable-rate learned image compression framework with a conditional autoencoder. Previous learning-based image compression methods mostly require training separate networks for different compression rates so they can yield compressed images of varying quality. In contrast, we train and deploy only one variable-rate image compression network implemented with a conditional autoencoder. We provide two rate control parameters, i.e., the Lagrange multiplier and the quantization bin size, which are given as conditioning variables to the network. Coarse rate adaptation to a target is performed by changing the Lagrange multiplier, while the rate can be further fine-tuned by adjusting the bin size used in quantizing the encoded representation. Our experimental results show that the proposed scheme provides a better rate-distortion trade-off than the traditional variable-rate image compression codecs such as JPEG2000 and BPG. Our model also shows comparable and sometimes better performance than the state-of-the-art learned image compression models that deploy multiple networks trained for varying rates. △ Less

Submitted 10 September, 2019; originally announced September 2019.

Comments: ICCV 2019

TW-SMNet: Deep Multitask Learning of Tele-Wide Stereo Matching

Authors: Mostafa El-Khamy, Haoyu Ren, Xianzhi Du, Jungwon Lee

Abstract: In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV (WFOV) captured by a wide angle lens, and the second FOV is contained in the first FOV and is captured by a tele zoom lens. We refer to the problem of estimating the inverse depth for the union of… ▽ More In this paper, we introduce the problem of estimating the real world depth of elements in a scene captured by two cameras with different field of views, where the first field of view (FOV) is a Wide FOV (WFOV) captured by a wide angle lens, and the second FOV is contained in the first FOV and is captured by a tele zoom lens. We refer to the problem of estimating the inverse depth for the union of FOVs, while leveraging the stereo information in the overlap** FOV, as Tele-Wide Stereo Matching (TW-SM). We propose different deep learning solutions to the TW-SM problem. Since the disparity is proportional to the inverse depth, we train stereo matching disparity estimation (SMDE) networks to estimate the disparity for the union WFOV. We further propose an end-to-end deep multitask tele-wide stereo matching neural network (MT-TW-SMNet), which simultaneously learns the SMDE task for the overlapped Tele FOV and the single image inverse depth estimation (SIDE) task for the WFOV. Moreover, we design multiple methods for the fusion of the SMDE and SIDE networks. We evaluate the performance of TW-SM on the popular KITTI and SceneFlow stereo datasets, and demonstrate its practicality by synthesizing the Bokeh effect on the WFOV from a tele-wide stereo image pair. △ Less

Submitted 11 June, 2019; originally announced June 2019.

Journal ref: Multitask Deep Neural Networks for Tele-Wide Stereo Matching, IEEE Access, 2020

Deep Robust Single Image Depth Estimation Neural Network Using Scene Understanding

Authors: Haoyu Ren, Mostafa El-khamy, Jungwon Lee

Abstract: Single image depth estimation (SIDE) plays a crucial role in 3D computer vision. In this paper, we propose a two-stage robust SIDE framework that can perform blind SIDE for both indoor and outdoor scenes. At the first stage, the scene understanding module will categorize the RGB image into different depth-ranges. We introduce two different scene understanding modules based on scene classification… ▽ More Single image depth estimation (SIDE) plays a crucial role in 3D computer vision. In this paper, we propose a two-stage robust SIDE framework that can perform blind SIDE for both indoor and outdoor scenes. At the first stage, the scene understanding module will categorize the RGB image into different depth-ranges. We introduce two different scene understanding modules based on scene classification and coarse depth estimation respectively. At the second stage, SIDE networks trained by the images of specific depth-range are applied to obtain an accurate depth map. In order to improve the accuracy, we further design a multi-task encoding-decoding SIDE network DS-SIDENet based on depthwise separable convolutions. DS-SIDENet is optimized to minimize both depth classification and depth regression losses. This improves the accuracy compared to a single-task SIDE network. Experimental results demonstrate that training DS-SIDENet on an individual dataset such as NYU achieves competitive performance to the state-of-art methods with much better efficiency. Ours proposed robust SIDE framework also shows good performance for the ScanNet indoor images and KITTI outdoor images simultaneously. It achieves the top performance compared to the Robust Vision Challenge (ROB) 2018 submissions. △ Less

Submitted 7 June, 2019; originally announced June 2019.

Learning with Succinct Common Representation Based on Wyner's Common Information

Authors: J. Jon Ryu, Yoo** Choi, Young-Han Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: A new bimodal generative model is proposed for generating conditional and joint samples, accompanied with a training method with learning a succinct bottleneck representation. The proposed model, dubbed as the variational Wyner model, is designed based on two classical problems in network information theory -- distributed simulation and channel synthesis -- in which Wyner's common information aris… ▽ More A new bimodal generative model is proposed for generating conditional and joint samples, accompanied with a training method with learning a succinct bottleneck representation. The proposed model, dubbed as the variational Wyner model, is designed based on two classical problems in network information theory -- distributed simulation and channel synthesis -- in which Wyner's common information arises as the fundamental limit on the succinctness of the common representation. The model is trained by minimizing the symmetric Kullback--Leibler divergence between variational and model distributions with regularization terms for common information, reconstruction consistency, and latent space matching terms, which is carried out via an adversarial density ratio estimation technique. The utility of the proposed approach is demonstrated through experiments for joint and conditional generation with synthetic and real-world datasets, as well as a challenging zero-shot image retrieval task. △ Less

Submitted 27 July, 2022; v1 submitted 26 May, 2019; originally announced May 2019.

Comments: 20 pages, 7 figures

AMNet: Deep Atrous Multiscale Stereo Disparity Estimation Networks

Authors: Xianzhi Du, Mostafa El-Khamy, Jungwon Lee

Abstract: In this paper, a new deep learning architecture for stereo disparity estimation is proposed. The proposed atrous multiscale network (AMNet) adopts an efficient feature extractor with depthwise-separable convolutions and an extended cost volume that deploys novel stereo matching costs on the deep features. A stacked atrous multiscale network is proposed to aggregate rich multiscale contextual infor… ▽ More In this paper, a new deep learning architecture for stereo disparity estimation is proposed. The proposed atrous multiscale network (AMNet) adopts an efficient feature extractor with depthwise-separable convolutions and an extended cost volume that deploys novel stereo matching costs on the deep features. A stacked atrous multiscale network is proposed to aggregate rich multiscale contextual information from the cost volume which allows for estimating the disparity with high accuracy at multiple scales. AMNet can be further modified to be a foreground-background aware network, FBA-AMNet, which is capable of discriminating between the foreground and the background objects in the scene at multiple scales. An iterative multitask learning method is proposed to train FBA-AMNet end-to-end. The proposed disparity estimation networks, AMNet and FBA-AMNet, show accurate disparity estimates and advance the state of the art on the challenging Middlebury, KITTI 2012, KITTI 2015, and Sceneflow stereo disparity estimation benchmarks. △ Less

Submitted 19 April, 2019; originally announced April 2019.

Jointly Sparse Convolutional Neural Networks in Dual Spatial-Winograd Domains

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: We consider the optimization of deep convolutional neural networks (CNNs) such that they provide good performance while having reduced complexity if deployed on either conventional systems with spatial-domain convolution or lower-complexity systems designed for Winograd convolution. The proposed framework produces one compressed model whose convolutional filters can be made sparse either in the sp… ▽ More We consider the optimization of deep convolutional neural networks (CNNs) such that they provide good performance while having reduced complexity if deployed on either conventional systems with spatial-domain convolution or lower-complexity systems designed for Winograd convolution. The proposed framework produces one compressed model whose convolutional filters can be made sparse either in the spatial domain or in the Winograd domain. Hence, the compressed model can be deployed universally on any platform, without need for re-training on the deployed platform. To get a better compression ratio, the sparse model is compressed in the spatial domain that has a fewer number of parameters. From our experiments, we obtain $24.2\times$ and $47.7\times$ compressed models for ResNet-18 and AlexNet trained on the ImageNet dataset, while their computational cost is also reduced by $4.5\times$ and $5.1\times$, respectively. △ Less

Submitted 20 February, 2019; originally announced February 2019.

Comments: IEEE ICASSP 2019. arXiv admin note: substantial text overlap with arXiv:1805.08303

End-to-End Multi-Task Denoising for joint SDR and PESQ Optimization

Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: Supervised learning based on a deep neural network recently has achieved substantial improvement on speech enhancement. Denoising networks learn map** from noisy speech to clean one directly, or to a spectrum mask which is the ratio between clean and noisy spectra. In either case, the network is optimized by minimizing mean square error (MSE) between ground-truth labels and time-domain or spectr… ▽ More Supervised learning based on a deep neural network recently has achieved substantial improvement on speech enhancement. Denoising networks learn map** from noisy speech to clean one directly, or to a spectrum mask which is the ratio between clean and noisy spectra. In either case, the network is optimized by minimizing mean square error (MSE) between ground-truth labels and time-domain or spectrum output. However, existing schemes have either of two critical issues: spectrum and metric mismatches. The spectrum mismatch is a well known issue that any spectrum modification after short-time Fourier transform (STFT), in general, cannot be fully recovered after inverse short-time Fourier transform (ISTFT). The metric mismatch is that a conventional MSE metric is sub-optimal to maximize our target metrics, signal-to-distortion ratio (SDR) and perceptual evaluation of speech quality (PESQ). This paper presents a new end-to-end denoising framework with the goal of joint SDR and PESQ optimization. First, the network optimization is performed on the time-domain signals after ISTFT to avoid spectrum mismatch. Second, two loss functions which have improved correlations with SDR and PESQ metrics are proposed to minimize metric mismatch. The experimental result showed that the proposed denoising scheme significantly improved both SDR and PESQ performance over the existing methods. △ Less

Submitted 8 March, 2023; v1 submitted 25 January, 2019; originally announced January 2019.

DN-ResNet: Efficient Deep Residual Network for Image Denoising

Authors: Haoyu Ren, Mostafa El-Khamy, Jungwon Lee

Abstract: A deep learning approach to blind denoising of images without complete knowledge of the noise statistics is considered. We propose DN-ResNet, which is a deep convolutional neural network (CNN) consisting of several residual blocks (ResBlocks). With cascade training, DN-ResNet is more accurate and more computationally efficient than the state of art denoising networks. An edge-aware loss function i… ▽ More A deep learning approach to blind denoising of images without complete knowledge of the noise statistics is considered. We propose DN-ResNet, which is a deep convolutional neural network (CNN) consisting of several residual blocks (ResBlocks). With cascade training, DN-ResNet is more accurate and more computationally efficient than the state of art denoising networks. An edge-aware loss function is further utilized in training DN-ResNet, so that the denoising results have better perceptive quality compared to conventional loss function. Next, we introduce the depthwise separable DN-ResNet (DS-DN-ResNet) utilizing the proposed Depthwise Seperable ResBlock (DS-ResBlock) instead of standard ResBlock, which has much less computational cost. DS-DN-ResNet is incrementally evolved by replacing the ResBlocks in DN-ResNet by DS-ResBlocks stage by stage. As a result, high accuracy and good computational efficiency are achieved concurrently. Whereas previous state of art deep learning methods focused on denoising either Gaussian or Poisson corrupted images, we consider denoising images having the more practical Poisson with additive Gaussian noise as well. The results show that DN-ResNets are more efficient, robust, and perform better denoising than current state of art deep learning methods, as well as the popular variants of the BM3D algorithm, in cases of blind and non-blind denoising of images corrupted with Poisson, Gaussian or Poisson-Gaussian noise. Our network also works well for other image enhancement task such as compressed image restoration. △ Less

Submitted 15 October, 2018; originally announced October 2018.

Journal ref: Asian Conference of Computer Vision 2018

Learning Sparse Low-Precision Neural Networks With Learnable Regularization

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training. Thu… ▽ More We consider learning deep neural networks (DNNs) that consist of low-precision weights and activations for efficient inference of fixed-point operations. In training low-precision networks, gradient descent in the backward pass is performed with high-precision weights while quantized low-precision weights and activations are used in the forward pass to calculate the loss function for training. Thus, the gradient descent becomes suboptimal, and accuracy loss follows. In order to reduce the mismatch in the forward and backward passes, we utilize mean squared quantization error (MSQE) regularization. In particular, we propose using a learnable regularization coefficient with the MSQE regularizer to reinforce the convergence of high-precision weights to their quantized values. We also investigate how partial L2 regularization can be employed for weight pruning in a similar manner. Finally, combining weight pruning, quantization, and entropy coding, we establish a low-precision DNN compression pipeline. In our experiments, the proposed method yields low-precision MobileNet and ShuffleNet models on ImageNet classification with the state-of-the-art compression ratios of 7.13 and 6.79, respectively. Moreover, we examine our method for image super resolution networks to produce 8-bit low-precision models at negligible performance loss. △ Less

Submitted 23 May, 2020; v1 submitted 31 August, 2018; originally announced September 2018.

Comments: IEEE Access

Fused Deep Neural Networks for Efficient Pedestrian Detection

Authors: Xianzhi Du, Mostafa El-Khamy, Vlad I. Morariu, Jungwon Lee, Larry Davis

Abstract: In this paper, we present an efficient pedestrian detection system, designed by fusion of multiple deep neural network (DNN) systems. Pedestrian candidates are first generated by a single shot convolutional multi-box detector at different locations with various scales and aspect ratios. The candidate generator is designed to provide the majority of ground truth pedestrian annotations at the cost o… ▽ More In this paper, we present an efficient pedestrian detection system, designed by fusion of multiple deep neural network (DNN) systems. Pedestrian candidates are first generated by a single shot convolutional multi-box detector at different locations with various scales and aspect ratios. The candidate generator is designed to provide the majority of ground truth pedestrian annotations at the cost of a large number of false positives. Then, a classification system using the idea of ensemble learning is deployed to improve the detection accuracy. The classification system further classifies the generated candidates based on opinions of multiple deep verification networks and a fusion network which utilizes a novel soft-rejection fusion method to adjust the confidence in the detection results. To improve the training of the deep verification networks, a novel soft-label method is devised to assign floating point labels to the generated pedestrian candidates. A deep context aggregation semantic segmentation network also provides pixel-level classification of the scene and its results are softly fused with the detection results by the single shot detector. Our pedestrian detector compared favorably to state-of-art methods on all popular pedestrian detection datasets. For example, our fused DNN has better detection accuracy on the Caltech Pedestrian dataset than all previous state of art methods, while also being the fastest. We significantly improved the log-average miss rate on the Caltech pedestrian dataset to 7.67% and achieved the new state-of-the-art. △ Less

Submitted 1 May, 2018; originally announced May 2018.

Comments: 11 pages

Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: We consider the optimization of deep convolutional neural networks (CNNs) such that they provide good performance while having reduced complexity if deployed on either conventional systems utilizing spatial-domain convolution or lower complexity systems designed for Winograd convolution. Furthermore, we explore the universal quantization and compression of these networks. In particular, the propos… ▽ More We consider the optimization of deep convolutional neural networks (CNNs) such that they provide good performance while having reduced complexity if deployed on either conventional systems utilizing spatial-domain convolution or lower complexity systems designed for Winograd convolution. Furthermore, we explore the universal quantization and compression of these networks. In particular, the proposed framework produces one compressed model whose convolutional filters can be made sparse either in the spatial domain or in the Winograd domain. Hence, one compressed model can be deployed universally on any platform, without need for re-training on the deployed platform, and the sparsity of its convolutional filters can be exploited for further complexity reduction in either domain. To get a better compression ratio, the sparse model is compressed in the spatial domain which has a less number of parameters. From our experiments, we obtain $24.2\times$, $47.7\times$ and $35.4\times$ compressed models for ResNet-18, AlexNet and CT-SRCNN, while their computational cost is also reduced by $4.5\times$, $5.1\times$ and $23.5\times$, respectively. △ Less

Submitted 28 October, 2018; v1 submitted 21 May, 2018; originally announced May 2018.

Universal Deep Neural Network Compression

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment. Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding. In particular… ▽ More In this paper, we investigate lossy compression of deep neural networks (DNNs) by weight quantization and lossless source coding for memory-efficient deployment. Whereas the previous work addressed non-universal scalar quantization and entropy coding of DNN weights, we for the first time introduce universal DNN compression by universal vector quantization and universal source coding. In particular, we examine universal randomized lattice quantization of DNNs, which randomizes DNN weights by uniform random dithering before lattice quantization and can perform near-optimally on any source without relying on knowledge of its probability distribution. Moreover, we present a method of fine-tuning vector quantized DNNs to recover the performance loss after quantization. Our experimental results show that the proposed universal DNN compression scheme compresses the 32-layer ResNet (trained on CIFAR-10) and the AlexNet (trained on ImageNet) with compression ratios of $47.1$ and $42.5$, respectively. △ Less

Submitted 20 February, 2019; v1 submitted 6 February, 2018; originally announced February 2018.

Comments: NeurIPS 2018 Workshop on Compact Deep Neural Network Representation with Industrial Applications (CDNNRIA)

CT-SRCNN: Cascade Trained and Trimmed Deep Convolutional Neural Networks for Image Super Resolution

Authors: Haoyu Ren, Mostafa El-Khamy, Jungwon Lee

Abstract: We propose methodologies to train highly accurate and efficient deep convolutional neural networks (CNNs) for image super resolution (SR). A cascade training approach to deep learning is proposed to improve the accuracy of the neural networks while gradually increasing the number of network layers. Next, we explore how to improve the SR efficiency by making the network slimmer. Two methodologies,… ▽ More We propose methodologies to train highly accurate and efficient deep convolutional neural networks (CNNs) for image super resolution (SR). A cascade training approach to deep learning is proposed to improve the accuracy of the neural networks while gradually increasing the number of network layers. Next, we explore how to improve the SR efficiency by making the network slimmer. Two methodologies, the one-shot trimming and the cascade trimming, are proposed. With the cascade trimming, the network's size is gradually reduced layer by layer, without significant loss on its discriminative ability. Experiments on benchmark image datasets show that our proposed SR network achieves the state-of-the-art super resolution accuracy, while being more than 4 times faster compared to existing deep super resolution networks. △ Less

Submitted 10 November, 2017; originally announced November 2017.

Comments: Accepted to IEEE Winter Conf. on Applications of Computer Vision (WACV) 2018, Lake Tahoe, USA

BridgeNets: Student-Teacher Transfer Learning Based on Recursive Neural Networks and its Application to Distant Speech Recognition

Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: Despite the remarkable progress achieved on automatic speech recognition, recognizing far-field speeches mixed with various noise sources is still a challenging task. In this paper, we introduce novel student-teacher transfer learning, BridgeNet which can provide a solution to improve distant speech recognition. There are two key features in BridgeNet. First, BridgeNet extends traditional student-… ▽ More Despite the remarkable progress achieved on automatic speech recognition, recognizing far-field speeches mixed with various noise sources is still a challenging task. In this paper, we introduce novel student-teacher transfer learning, BridgeNet which can provide a solution to improve distant speech recognition. There are two key features in BridgeNet. First, BridgeNet extends traditional student-teacher frameworks by providing multiple hints from a teacher network. Hints are not limited to the soft labels from a teacher network. Teacher's intermediate feature representations can better guide a student network to learn how to denoise or dereverberate noisy input. Second, the proposed recursive architecture in the BridgeNet can iteratively improve denoising and recognition performance. The experimental results of BridgeNet showed significant improvements in tackling the distant speech recognition problem, where it achieved up to 13.24% relative WER reductions on AMI corpus compared to a baseline neural network without teacher's hints. △ Less

Submitted 21 February, 2018; v1 submitted 27 October, 2017; originally announced October 2017.

Comments: Accepted to 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2018)

Circular Buffer Rate-Matched Polar Codes

Authors: Mostafa El-Khamy, Hsien-** Lin, Jungwon Lee, Inyup Kang

Abstract: A practical rate-matching system for constructing rate-compatible polar codes is proposed. The proposed polar code circular buffer rate-matching is suitable for transmissions on communication channels that support hybrid automatic repeat request (HARQ) communications, as well as for flexible resource-element rate-matching on single transmission channels. Our proposed circular buffer rate matching… ▽ More A practical rate-matching system for constructing rate-compatible polar codes is proposed. The proposed polar code circular buffer rate-matching is suitable for transmissions on communication channels that support hybrid automatic repeat request (HARQ) communications, as well as for flexible resource-element rate-matching on single transmission channels. Our proposed circular buffer rate matching scheme also incorporates a bit-map** scheme for transmission on bit-interleaved coded modulation (BICM) channels using higher order modulations. An interleaver is derived from a puncturing order obtained with a low complexity progressive puncturing search algorithm on a base code of short length, and has the flexibility to achieve any desired rate at the desired code length, through puncturing or repetition. The rate-matching scheme is implied by a two-stage polarization, for transmission at any desired code length, code rate, and modulation order, and is shown to achieve the symmetric capacity of BICM channels. Numerical results on AWGN and fast fading channels show that the rate-matched polar codes have a competitive performance when compared to the spatially-coupled quasi-cyclic LDPC codes or LTE turbo codes, while having similar rate-dematching storage and computational complexities. △ Less

Submitted 13 February, 2017; originally announced February 2017.

Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition

Authors: Jaeyoung Kim, Mostafa El-Khamy, Jungwon Lee

Abstract: In this paper, a novel architecture for a deep recurrent neural network, residual LSTM is introduced. A plain LSTM has an internal memory cell that can learn long term dependencies of sequential data. It also provides a temporal shortcut path to avoid vanishing or exploding gradients in the temporal domain. The residual LSTM provides an additional spatial shortcut path from lower layers for effici… ▽ More In this paper, a novel architecture for a deep recurrent neural network, residual LSTM is introduced. A plain LSTM has an internal memory cell that can learn long term dependencies of sequential data. It also provides a temporal shortcut path to avoid vanishing or exploding gradients in the temporal domain. The residual LSTM provides an additional spatial shortcut path from lower layers for efficient training of deep networks with multiple LSTM layers. Compared with the previous work, highway LSTM, residual LSTM separates a spatial shortcut path with temporal one by using output layers, which can help to avoid a conflict between spatial and temporal-domain gradient flows. Furthermore, residual LSTM reuses the output projection matrix and the output gate of LSTM to control the spatial information flow instead of additional gate networks, which effectively reduces more than 10% of network parameters. An experiment for distant speech recognition on the AMI SDM corpus shows that 10-layer plain and highway LSTM networks presented 13.7% and 6.2% increase in WER over 3-layer aselines, respectively. On the contrary, 10-layer residual LSTM networks provided the lowest WER 41.0%, which corresponds to 3.3% and 2.8% WER reduction over plain and highway LSTM networks, respectively. △ Less

Submitted 5 June, 2017; v1 submitted 10 January, 2017; originally announced January 2017.

Towards the Limit of Network Quantization

Authors: Yoo** Choi, Mostafa El-Khamy, Jungwon Lee

Abstract: Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative… ▽ More Network quantization is one of network compression techniques to reduce the redundancy of deep neural networks. It reduces the number of distinct network parameter values by quantization in order to save the storage for them. In this paper, we design network quantization schemes that minimize the performance loss due to quantization given a compression ratio constraint. We analyze the quantitative relation of quantization errors to the neural network loss function and identify that the Hessian-weighted distortion measure is locally the right objective function for the optimization of network quantization. As a result, Hessian-weighted k-means clustering is proposed for clustering network parameters to quantize. When optimal variable-length binary codes, e.g., Huffman codes, are employed for further compression, we derive that the network quantization problem can be related to the entropy-constrained scalar quantization (ECSQ) problem in information theory and consequently propose two solutions of ECSQ for network quantization, i.e., uniform quantization and an iterative solution similar to Lloyd's algorithm. Finally, using the simple uniform quantization followed by Huffman coding, we show from our experiments that the compression ratios of 51.25, 22.17 and 40.65 are achievable for LeNet, 32-layer ResNet and AlexNet, respectively. △ Less

Submitted 13 November, 2017; v1 submitted 5 December, 2016; originally announced December 2016.

Comments: Published as a conference paper at ICLR 2017

Fused DNN: A deep neural network fusion approach to fast and robust pedestrian detection

Authors: Xianzhi Du, Mostafa El-Khamy, Jungwon Lee, Larry S. Davis

Abstract: We propose a deep neural network fusion architecture for fast and robust pedestrian detection. The proposed network fusion architecture allows for parallel processing of multiple networks for speed. A single shot deep convolutional network is trained as a object detector to generate all possible pedestrian candidates of different sizes and occlusions. This network outputs a large variety of pedest… ▽ More We propose a deep neural network fusion architecture for fast and robust pedestrian detection. The proposed network fusion architecture allows for parallel processing of multiple networks for speed. A single shot deep convolutional network is trained as a object detector to generate all possible pedestrian candidates of different sizes and occlusions. This network outputs a large variety of pedestrian candidates to cover the majority of ground-truth pedestrians while also introducing a large number of false positives. Next, multiple deep neural networks are used in parallel for further refinement of these pedestrian candidates. We introduce a soft-rejection based network fusion method to fuse the soft metrics from all networks together to generate the final confidence scores. Our method performs better than existing state-of-the-arts, especially when detecting small-size and occluded pedestrians. Furthermore, we propose a method for integrating pixel-wise semantic segmentation network into the network fusion architecture as a reinforcement to the pedestrian detector. The approach outperforms state-of-the-art methods on most protocols on Caltech Pedestrian dataset, with significant boosts on several protocols. It is also faster than all other methods. △ Less

Submitted 28 May, 2017; v1 submitted 11 October, 2016; originally announced October 2016.

Comments: WACV 2017

Binary Polar Codes are Optimized Codes for Bitwise Multistage Decoding

Authors: Mostafa El-Khamy, Hsien-** Lin, Jungwon Lee

Abstract: Polar codes are considered the latest major breakthrough in coding theory. Polar codes were introduced by Arıkan in 2008. In this letter, we show that the binary polar codes are the same as the optimized codes for bitwise multistage decoding (OCBM), which have been discovered before by Stolte in 2002. The equivalence between the techniques used for the constructions and decodings of both codes is… ▽ More Polar codes are considered the latest major breakthrough in coding theory. Polar codes were introduced by Arıkan in 2008. In this letter, we show that the binary polar codes are the same as the optimized codes for bitwise multistage decoding (OCBM), which have been discovered before by Stolte in 2002. The equivalence between the techniques used for the constructions and decodings of both codes is established. △ Less

Submitted 12 April, 2016; originally announced April 2016.

Comments: Accepted at Electronics Letters

Rate-Compatible Polar Codes for Wireless Channels

Authors: Mostafa El-Khamy, Hsien-** Lin, Jungwon Lee, Hessam Mahdavifar, Inyup Kang

Abstract: A design of rate-compatible polar codes suitable for HARQ communications is proposed in this paper. An important feature of the proposed design is that the puncturing order is chosen with low complexity on a base code of short length, which is then further polarized to the desired length. A practical rate-matching system that has the flexibility to choose any desired rate through puncturing or rep… ▽ More A design of rate-compatible polar codes suitable for HARQ communications is proposed in this paper. An important feature of the proposed design is that the puncturing order is chosen with low complexity on a base code of short length, which is then further polarized to the desired length. A practical rate-matching system that has the flexibility to choose any desired rate through puncturing or repetition while preserving the polarization is suggested. The proposed rate-matching system is combined with channel interleaving and a bit-map** procedure that preserves the polarization of the rate-compatible polar code family over bit-interleaved coded modulation systems. Simulation results on AWGN and fast fading channels with different modulation orders show the robustness of the proposed rate-compatible polar code in both Chase combining and incremental redundancy HARQ communications. △ Less

Submitted 31 August, 2015; originally announced August 2015.

Comments: Accepted for publication at 2015 IEEE Global Communications Conference (Globecom)

Relaxed Polar Codes

Authors: Mostafa El-Khamy, Hessam Mahdavifar, Gennady Feygin, Jungwon Lee, Inyup Kang

Abstract: Polar codes are the latest breakthrough in coding theory, as they are the first family of codes with explicit construction that provably achieve the symmetric capacity of discrete memoryless channels. Arıkan's polar encoder and successive cancellation decoder have complexities of $N \log N$, for code length $N$. Although, the complexity bound of $N \log N$ is asymptotically favorable, we report in… ▽ More Polar codes are the latest breakthrough in coding theory, as they are the first family of codes with explicit construction that provably achieve the symmetric capacity of discrete memoryless channels. Arıkan's polar encoder and successive cancellation decoder have complexities of $N \log N$, for code length $N$. Although, the complexity bound of $N \log N$ is asymptotically favorable, we report in this work methods to further reduce the encoding and decoding complexities of polar coding. The crux is to relax the polarization of certain bit-channels without performance degradation. We consider schemes for relaxing the polarization of both \emph{very good} and \emph{very bad} bit-channels, in the process of channel polarization. Relaxed polar codes are proved to preserve the capacity achieving property of polar codes. Analytical bounds on the asymptotic and finite-length complexity reduction attainable by relaxed polarization are derived. For binary erasure channels, we show that the computation complexity can be reduced by a factor of 6, while preserving the rate and error performance. We also show that relaxed polar codes can be decoded with significantly reduced latency. For AWGN channels with medium code lengths, we show that relaxed polar codes can have lower error probabilities than conventional polar codes, while having reduced encoding and decoding computation complexities. △ Less

Submitted 16 July, 2015; v1 submitted 24 January, 2015; originally announced January 2015.

Comments: Conference version,Relaxed Channel Polarization for Reduced Complexity Polar Coding, accepted for presentation at IEEE Wireless Communications and Networking Conference WCNC 2015

Journal ref: IEEE Transactions on Information Theory, 2017

Achieving the Uniform Rate Region of General Multiple Access Channels by Polar Coding

Authors: Hessam Mahdavifar, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: We consider the problem of polar coding for transmission over $m$-user multiple access channels. In the proposed scheme, all users encode their messages using a polar encoder, while a joint successive cancellation decoder is deployed at the receiver. The encoding is done separately across the users and is independent of the target achievable rate, in the sense that the encoder core is the regular… ▽ More We consider the problem of polar coding for transmission over $m$-user multiple access channels. In the proposed scheme, all users encode their messages using a polar encoder, while a joint successive cancellation decoder is deployed at the receiver. The encoding is done separately across the users and is independent of the target achievable rate, in the sense that the encoder core is the regular Arıkan's polarization matrix. For the code construction, the positions of information bits and frozen bits for each of the users are decided jointly. This is done by treating the whole polar transformation across all the $m$ users as a single polar transformation with a certain base code. We prove that the covering radius of the dominant face of the uniform rate region is upper bounded by $r = \frac{(m-1)\sqrt{m}}{L}$, where $L$ represents the length of the base code. We then prove that the proposed polar coding scheme achieves the whole uniform rate region, with small enough resolution characterized by $r$, by changing the decoding order in the joint successive cancellation decoder. The encoding and decoding complexities are $O(N \log N)$, where $N$ is the code block length, and the asymptotic block error probability of $O(2^{-N^{0.5 - ε}})$ is guaranteed. Examples of achievable rates for the case of $3$-user multiple access channel are provided. △ Less

Submitted 10 July, 2014; originally announced July 2014.

Comments: submitted to IEEE Transactions on Communications, July 9, 2014. arXiv admin note: substantial text overlap with arXiv:1307.2889

Performance Limits and Practical Decoding of Interleaved Reed-Solomon Polar Concatenated Codes

Authors: Hessam Mahdavifar, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: A scheme for concatenating the recently invented polar codes with non-binary MDS codes, as Reed-Solomon codes, is considered. By concatenating binary polar codes with interleaved Reed-Solomon codes, we prove that the proposed concatenation scheme captures the capacity-achieving property of polar codes, while having a significantly better error-decay rate. We show that for any $ε> 0$, and total fra… ▽ More A scheme for concatenating the recently invented polar codes with non-binary MDS codes, as Reed-Solomon codes, is considered. By concatenating binary polar codes with interleaved Reed-Solomon codes, we prove that the proposed concatenation scheme captures the capacity-achieving property of polar codes, while having a significantly better error-decay rate. We show that for any $ε> 0$, and total frame length $N$, the parameters of the scheme can be set such that the frame error probability is less than $2^{-N^{1-ε}}$, while the scheme is still capacity achieving. This improves upon $2^{-N^{0.5-ε}}$, the frame error probability of Arikan's polar codes. The proposed concatenated polar codes and Arikan's polar codes are also compared for transmission over channels with erasure bursts. We provide a sufficient condition on the length of erasure burst which guarantees failure of the polar decoder. On the other hand, it is shown that the parameters of the concatenated polar code can be set in such a way that the capacity-achieving properties of polar codes are preserved. We also propose decoding algorithms for concatenated polar codes, which significantly improve the error-rate performance at finite block lengths while preserving the low decoding complexity. △ Less

Submitted 5 August, 2013; originally announced August 2013.

Comments: Submitted to IEEE Transactions on Communications. arXiv admin note: substantial text overlap with arXiv:1301.7491

Achieving the Uniform Rate Region of General Multiple Access Channels by Polar Coding

Authors: Hessam Mahdavifar, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: We consider the problem of polar coding for transmission over $m$-user multiple access channels. In the proposed scheme, all users encode their messages using a polar encoder, while a multi-user successive cancellation decoder is deployed at the receiver. The encoding is done separately across the users and is independent of the target achievable rate. For the code construction, the positions of i… ▽ More We consider the problem of polar coding for transmission over $m$-user multiple access channels. In the proposed scheme, all users encode their messages using a polar encoder, while a multi-user successive cancellation decoder is deployed at the receiver. The encoding is done separately across the users and is independent of the target achievable rate. For the code construction, the positions of information bits and frozen bits for each of the users are decided jointly. This is done by treating the polar transformations across all the $m$ users as a single polar transformation with a certain \emph{polarization base}. We characterize the resolution of achievable rates on the dominant face of the uniform rate region in terms of the number of users $m$ and the length of the polarization base $L$. In particular, we prove that for any target rate on the dominant face, there exists an achievable rate, also on the dominant face, within the distance at most $\frac{(m-1)\sqrt{m}}{L}$ from the target rate. We then prove that the proposed MAC polar coding scheme achieves the whole uniform rate region with fine enough resolution by changing the decoding order in the multi-user successive cancellation decoder, as $L$ and the code block length $N$ grow large. The encoding and decoding complexities are $O(N \log N)$ and the asymptotic block error probability of $O(2^{-N^{0.5 - ε}})$ is guaranteed. Examples of achievable rates for the $3$-user multiple access channel are provided. △ Less

Submitted 28 August, 2016; v1 submitted 10 July, 2013; originally announced July 2013.

Journal ref: IEEE Transactions on Communications 64.2 (2016): 467-478

BICM Performance Improvement via Online LLR Optimization

Authors: **hong Wu, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: We consider bit interleaved coded modulation (BICM) receiver performance improvement based on the concept of generalized mutual information (GMI). Increasing achievable rates of BICM receiver with GMI maximization by proper scaling of the log likelihood ratio (LLR) is investigated. While it has been shown in the literature that look-up table based LLR scaling functions matched to each specific tra… ▽ More We consider bit interleaved coded modulation (BICM) receiver performance improvement based on the concept of generalized mutual information (GMI). Increasing achievable rates of BICM receiver with GMI maximization by proper scaling of the log likelihood ratio (LLR) is investigated. While it has been shown in the literature that look-up table based LLR scaling functions matched to each specific transmission scenario may provide close to optimal solutions, this method is difficult to adapt to time-varying channel conditions. To solve this problem, an online adaptive scaling factor searching algorithm is developed. Uniform scaling factors are applied to LLRs from different bit channels of each data frame by maximizing an approximate GMI that characterizes the transmission conditions of current data frame. Numerical analysis on effective achievable rates as well as link level simulation of realistic mobile transmission scenarios indicate that the proposed method is simple yet effective. △ Less

Submitted 18 March, 2013; originally announced March 2013.

Comments: IEEE WCNC, Apr. 2013, Shanghai

Performance of Spatially-Coupled LDPC Codes and Threshold Saturation over BICM Channels

Authors: Arvind Yedla, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: We study the performance of binary spatially-coupled low-density parity-check codes (SC-LDPC) when used with bit-interleaved coded-modulation (BICM) schemes. This paper considers the cases when transmission takes place over additive white Gaussian noise (AWGN)channels and Rayleigh fast-fading channels. The technique of upper bounding the maximum-a-posteriori (MAP) decoding performance of LDPC code… ▽ More We study the performance of binary spatially-coupled low-density parity-check codes (SC-LDPC) when used with bit-interleaved coded-modulation (BICM) schemes. This paper considers the cases when transmission takes place over additive white Gaussian noise (AWGN)channels and Rayleigh fast-fading channels. The technique of upper bounding the maximum-a-posteriori (MAP) decoding performance of LDPC codes using an area theorem is extended for BICM schemes. The upper bound is computed for both the optimal MAP demapper and the suboptimal max-log-MAP (MLM) demapper. It is observed that this bound approaches the noise threshold of BICM channels for regular LDPC codes with large degrees. The rest of the paper extends these techniques to SC-LDPC codes and the phenomenon of threshold saturation is demonstrated numerically. Based on numerical evidence, we conjecture that the belief-propagation (BP) decoding threshold of SC-LDPC codes approaches the MAP decoding threshold of the underlying LDPC ensemble on BICM channels. Numerical results also show that SC-LDPC codes approach the BICM capacity over different channels and modulation schemes. △ Less

Submitted 1 March, 2013; originally announced March 2013.

Comments: 7 pages, 6 figures

Compound Polar Codes

Authors: Hessam Mahdavifar, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: A capacity-achieving scheme based on polar codes is proposed for reliable communication over multi-channels which can be directly applied to bit-interleaved coded modulation schemes. We start by reviewing the ground-breaking work of polar codes and then discuss our proposed scheme. Instead of encoding separately across the individual underlying channels, which requires multiple encoders and decode… ▽ More A capacity-achieving scheme based on polar codes is proposed for reliable communication over multi-channels which can be directly applied to bit-interleaved coded modulation schemes. We start by reviewing the ground-breaking work of polar codes and then discuss our proposed scheme. Instead of encoding separately across the individual underlying channels, which requires multiple encoders and decoders, we take advantage of the recursive structure of polar codes to construct a unified scheme with a single encoder and decoder that can be used over the multi-channels. We prove that the scheme achieves the capacity over this multi-channel. Numerical analysis and simulation results for BICM channels at finite block lengths shows a considerable improvement in the probability of error comparing to a conventional separated scheme. △ Less

Submitted 1 February, 2013; originally announced February 2013.

On the Construction and Decoding of Concatenated Polar Codes

Authors: Hessam Mahdavifar, Mostafa El-Khamy, Jungwon Lee, Inyup Kang

Abstract: A scheme for concatenating the recently invented polar codes with interleaved block codes is considered. By concatenating binary polar codes with interleaved Reed-Solomon codes, we prove that the proposed concatenation scheme captures the capacity-achieving property of polar codes, while having a significantly better error-decay rate. We show that for any $ε> 0$, and total frame length $N$, the pa… ▽ More A scheme for concatenating the recently invented polar codes with interleaved block codes is considered. By concatenating binary polar codes with interleaved Reed-Solomon codes, we prove that the proposed concatenation scheme captures the capacity-achieving property of polar codes, while having a significantly better error-decay rate. We show that for any $ε> 0$, and total frame length $N$, the parameters of the scheme can be set such that the frame error probability is less than $2^{-N^{1-ε}}$, while the scheme is still capacity achieving. This improves upon $2^{-N^{0.5-\eps}}$, the frame error probability of Arikan's polar codes. We also propose decoding algorithms for concatenated polar codes, which significantly improve the error-rate performance at finite block lengths while preserving the low decoding complexity. △ Less

Submitted 30 January, 2013; originally announced January 2013.

On the Weight Enumerator and the Maximum Likelihood Performance of Linear Product Codes

Authors: Mostafa El-Khamy, Roberto Garello

Abstract: Product codes are widely used in data-storage, optical and wireless applications. Their analytical performance evaluation usually relies on the truncated union bound, which provides a low error rate approximation based on the minimum distance term only. In fact, the complete weight enumerator of most product codes remains unknown. In this paper, concatenated representations are introduced and ap… ▽ More Product codes are widely used in data-storage, optical and wireless applications. Their analytical performance evaluation usually relies on the truncated union bound, which provides a low error rate approximation based on the minimum distance term only. In fact, the complete weight enumerator of most product codes remains unknown. In this paper, concatenated representations are introduced and applied to compute the complete average enumerators of arbitrary product codes over a field Fq. The split weight enumerators of some important constituent codes (Hamming, Reed-Solomon) are studied and used in the analysis. The average binary weight enumerators of Reed Solomon product codes are also derived. Numerical results showing the enumerator behavior are presented. By using the complete enumerators, Poltyrev bounds on the maximum likelihood performance, holding at both high and low error rates, are finally shown and compared against truncated union bounds and simulation results. △ Less

Submitted 23 January, 2006; originally announced January 2006.

Comments: 26 pages, 10 figures, submitted to IEEE transactions in December 2005

ACM Class: E.4

Iterative Algebraic Soft-Decision List Decoding of Reed-Solomon Codes

Authors: Mostafa El-Khamy, Robert J. McEliece

Abstract: In this paper, we present an iterative soft-decision decoding algorithm for Reed-Solomon codes offering both complexity and performance advantages over previously known decoding algorithms. Our algorithm is a list decoding algorithm which combines two powerful soft decision decoding techniques which were previously regarded in the literature as competitive, namely, the Koetter-Vardy algebraic so… ▽ More In this paper, we present an iterative soft-decision decoding algorithm for Reed-Solomon codes offering both complexity and performance advantages over previously known decoding algorithms. Our algorithm is a list decoding algorithm which combines two powerful soft decision decoding techniques which were previously regarded in the literature as competitive, namely, the Koetter-Vardy algebraic soft-decision decoding algorithm and belief-propagation based on adaptive parity check matrices, recently proposed by Jiang and Narayanan. Building on the Jiang-Narayanan algorithm, we present a belief-propagation based algorithm with a significant reduction in computational complexity. We introduce the concept of using a belief-propagation based decoder to enhance the soft-input information prior to decoding with an algebraic soft-decision decoder. Our algorithm can also be viewed as an interpolation multiplicity assignment scheme for algebraic soft-decision decoding of Reed-Solomon codes. △ Less

Submitted 29 September, 2005; originally announced September 2005.

Comments: Submitted to IEEE for publication in Jan 2005

Journal ref: IEEE Journal on Selected Areas in Communications, Volume 24, Issue 3, March 2006 Page(s):481 - 490

The Partition Weight Enumerator of MDS Codes and its Applications

Authors: Mostafa El-Khamy, Robert J. McEliece

Abstract: A closed form formula of the partition weight enumerator of maximum distance separable (MDS) codes is derived for an arbitrary number of partitions. Using this result, some properties of MDS codes are discussed. The results are extended for the average binary image of MDS codes in finite fields of characteristic two. As an application, we study the multiuser error probability of Reed Solomon cod… ▽ More A closed form formula of the partition weight enumerator of maximum distance separable (MDS) codes is derived for an arbitrary number of partitions. Using this result, some properties of MDS codes are discussed. The results are extended for the average binary image of MDS codes in finite fields of characteristic two. As an application, we study the multiuser error probability of Reed Solomon codes. △ Less

Submitted 20 May, 2005; originally announced May 2005.

Comments: This is a five page conference version of the paper which was accepted by ISIT 2005. For more information, please contact the authors

ACM Class: E.4