-
DeepMPR: Enhancing Opportunistic Routing in Wireless Networks through Multi-Agent Deep Reinforcement Learning
Authors:
Saeed Kaviani,
Bo Ryu,
Ejaz Ahmed,
Deokseong Kim,
Jae Kim,
Carrie Spiker,
Blake Harnden
Abstract:
Opportunistic routing relies on the broadcast capability of wireless networks. It brings higher reliability and robustness in highly dynamic and/or severe environments such as mobile or vehicular ad-hoc networks (MANETs/VANETs). To reduce the cost of broadcast, multicast routing schemes use the connected dominating set (CDS) or multi-point relaying (MPR) set to decrease the network overhead and he…
▽ More
Opportunistic routing relies on the broadcast capability of wireless networks. It brings higher reliability and robustness in highly dynamic and/or severe environments such as mobile or vehicular ad-hoc networks (MANETs/VANETs). To reduce the cost of broadcast, multicast routing schemes use the connected dominating set (CDS) or multi-point relaying (MPR) set to decrease the network overhead and hence, their selection algorithms are critical. Common MPR selection algorithms are heuristic, rely on coordination between nodes, need high computational power for large networks, and are difficult to tune for network uncertainties. In this paper, we use multi-agent deep reinforcement learning to design a novel MPR multicast routing technique, DeepMPR, which is outperforming the OLSR MPR selection algorithm while it does not require MPR announcement messages from the neighbors. Our evaluation results demonstrate the performance gains of our trained DeepMPR multicast forwarding policy compared to other popular techniques.
△ Less
Submitted 16 June, 2023;
originally announced June 2023.
-
Open Source-based Over-The-Air 5G New Radio Sidelink Testbed
Authors:
Melissa Elkadi,
Doekseong Kim,
Ejaz Ahmed,
Moein Sadeghi,
Anh Le,
Paul Russell,
Bo Ryu
Abstract:
The focus of this paper is to demonstrate an over-the-air (OTA) 5G new radio (NR) sidelink communication prototype. 5G NR sidelink communications allow NR UEs to transfer data independently without the assistance of a base station (gNB), which enables V2X communications, including platooning, autonomous driving, sensor extension, industrial IoT, public safety communication and much more. Our desig…
▽ More
The focus of this paper is to demonstrate an over-the-air (OTA) 5G new radio (NR) sidelink communication prototype. 5G NR sidelink communications allow NR UEs to transfer data independently without the assistance of a base station (gNB), which enables V2X communications, including platooning, autonomous driving, sensor extension, industrial IoT, public safety communication and much more. Our design leverages the open-source OpenAirInterface5G (OAI) software, which operates on software-defined radios (SDRs) and can be easily extended for mesh networking. The software includes all signal processing components specified by the 3GPP 5G sidelink standards, including Low-Density Parity Check (LDPC) encoding/decoding, polar encoding/decoding, data and control multiplexing, modulation/demodulation, and orthogonal frequency-division multiplexing (OFDM) modulation/demodulation. It can be configured to operate with different bands, bandwidths, and antenna settings. The first milestone in this work was to demonstrate the completed Physical Sidelink Broadcast Channel (PSBCH) development, which conducts synchronization between a Synchronization Reference (SyncRef) UE and a nearby UE. The SyncRef UE broadcasts a sidelink synchronization signal block (S-SSB) periodically, which the nearby UE detects and uses to synchronize its timing and frequency components with the SyncRef UE. Once a connection is established, the next developmental milestone is to transmit real data (text messages) via the Physical Sidelink Shared Channel (PSSCH). Our PHY sidelink framework is tested using both an RF simulator and an OTA testbed with multiple nearby UEs. Beyond the development of synchronization and data transmission/reception in 5G sidelink, we conclude with various performance tests and validation experiments. The results of these metrics show that our simulator is comparable to the OTA testbed.
△ Less
Submitted 6 October, 2023; v1 submitted 15 June, 2023;
originally announced June 2023.
-
DeepADMR: A Deep Learning based Anomaly Detection for MANET Routing
Authors:
Alex Yahja,
Saeed Kaviani,
Bo Ryu,
Jae H. Kim,
Kevin A. Larson
Abstract:
We developed DeepADMR, a novel neural anomaly detector for the deep reinforcement learning (DRL)-based DeepCQ+ MANET routing policy. The performance of DRL-based algorithms such as DeepCQ+ is only verified within the trained and tested environments, hence their deployment in the tactical domain induces high risks. DeepADMR monitors unexpected behavior of the DeepCQ+ policy based on the temporal di…
▽ More
We developed DeepADMR, a novel neural anomaly detector for the deep reinforcement learning (DRL)-based DeepCQ+ MANET routing policy. The performance of DRL-based algorithms such as DeepCQ+ is only verified within the trained and tested environments, hence their deployment in the tactical domain induces high risks. DeepADMR monitors unexpected behavior of the DeepCQ+ policy based on the temporal difference errors (TD-errors) in real-time and detects anomaly scenarios with empirical and non-parametric cumulative-sum statistics. The DeepCQ+ design via multi-agent weight-sharing proximal policy optimization (PPO) is slightly modified to enable the real-time estimation of the TD-errors. We report the DeepADMR performance in the presence of channel disruptions, high mobility levels, and network sizes beyond the training environments, which shows its effectiveness.
△ Less
Submitted 24 January, 2023;
originally announced February 2023.
-
Deep Reinforcement Learning for System-on-Chip: Myths and Realities
Authors:
Tegg Taekyong Sung,
Bo Ryu
Abstract:
Neural schedulers based on deep reinforcement learning (DRL) have shown considerable potential for solving real-world resource allocation problems, as they have demonstrated significant performance gain in the domain of cluster computing. In this paper, we investigate the feasibility of neural schedulers for the domain of System-on-Chip (SoC) resource allocation through extensive experiments and c…
▽ More
Neural schedulers based on deep reinforcement learning (DRL) have shown considerable potential for solving real-world resource allocation problems, as they have demonstrated significant performance gain in the domain of cluster computing. In this paper, we investigate the feasibility of neural schedulers for the domain of System-on-Chip (SoC) resource allocation through extensive experiments and comparison with non-neural, heuristic schedulers. The key finding is three-fold. First, neural schedulers designed for cluster computing domain do not work well for SoC due to i) heterogeneity of SoC computing resources and ii) variable action set caused by randomness in incoming jobs. Second, our novel neural scheduler technique, Eclectic Interaction Matching (EIM), overcomes the above challenges, thus significantly improving the existing neural schedulers. Specifically, we rationalize the underlying reasons behind the performance gain by the EIM-based neural scheduler. Third, we discover that the ratio of the average processing elements (PE) switching delay and the average PE computation time significantly impacts the performance of neural SoC schedulers even with EIM. Consequently, future neural SoC scheduler design must consider this metric as well as its implementation overhead for practical utility.
△ Less
Submitted 29 July, 2022;
originally announced July 2022.
-
DeepCQ+: Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for Highly Dynamic Networks
Authors:
Saeed Kaviani,
Bo Ryu,
Ejaz Ahmed,
Kevin Larson,
Anh Le,
Alex Yahja,
Jae H. Kim
Abstract:
Highly dynamic mobile ad-hoc networks (MANETs) remain as one of the most challenging environments to develop and deploy robust, efficient, and scalable routing protocols. In this paper, we present DeepCQ+ routing protocol which, in a novel manner integrates emerging multi-agent deep reinforcement learning (MADRL) techniques into existing Q-learning-based routing protocols and their variants and ac…
▽ More
Highly dynamic mobile ad-hoc networks (MANETs) remain as one of the most challenging environments to develop and deploy robust, efficient, and scalable routing protocols. In this paper, we present DeepCQ+ routing protocol which, in a novel manner integrates emerging multi-agent deep reinforcement learning (MADRL) techniques into existing Q-learning-based routing protocols and their variants and achieves persistently higher performance across a wide range of topology and mobility configurations. While kee** the overall protocol structure of the Q-learning-based routing protocols, DeepCQ+ replaces statically configured parameterized thresholds and hand-written rules with carefully designed MADRL agents such that no configuration of such parameters is required a priori. Extensive simulation shows that DeepCQ+ yields significantly increased end-to-end throughput with lower overhead and no apparent degradation of end-to-end delays (hop counts) compared to its Q-learning based counterparts. Qualitatively, and perhaps more significantly, DeepCQ+ maintains remarkably similar performance gains under many scenarios that it was not trained for in terms of network sizes, mobility conditions, and traffic dynamics. To the best of our knowledge, this is the first successful application of the MADRL framework for the MANET routing problem that demonstrates a high degree of scalability and robustness even under environments that are outside the trained range of scenarios. This implies that our MARL-based DeepCQ+ design solution significantly improves the performance of Q-learning based CQ+ baseline approach for comparison and increases its practicality and explainability because the real-world MANET environment will likely vary outside the trained range of MANET scenarios. Additional techniques to further increase the gains in performance and scalability are discussed.
△ Less
Submitted 29 November, 2021;
originally announced November 2021.
-
SoCRATES: System-on-Chip Resource Adaptive Scheduling using Deep Reinforcement Learning
Authors:
Tegg Taekyong Sung,
Bo Ryu
Abstract:
Deep Reinforcement Learning (DRL) is being increasingly applied to the problem of resource allocation for emerging System-on-Chip (SoC) applications, and has shown remarkable promises. In this paper, we introduce SoCRATES (SoC Resource AdapTivE Scheduler), an extremely efficient DRL-based SoC scheduler which maps a wide range of hierarchical jobs to heterogeneous resources within SoC using the Ecl…
▽ More
Deep Reinforcement Learning (DRL) is being increasingly applied to the problem of resource allocation for emerging System-on-Chip (SoC) applications, and has shown remarkable promises. In this paper, we introduce SoCRATES (SoC Resource AdapTivE Scheduler), an extremely efficient DRL-based SoC scheduler which maps a wide range of hierarchical jobs to heterogeneous resources within SoC using the Eclectic Interaction Matching (EIM) technique. It is noted that the majority of SoC resource management approaches have been targeting makespan minimization with fixed number of jobs in the system. In contrast, SoCRATES aims at minimizing average latency in a steady-state condition while assigning tasks in the ready queue to heterogeneous resources (processing elements). We first show that the latency-minimization-driven SoC applications operate high-frequency job workload and distributed/parallel job execution. We then demonstrate SoCRATES successfully addresses the challenge of concurrent observations caused by the task dependency inherent in the latency minimization objective. Extensive tests show that SoCRATES outperforms other existing neural and non-neural schedulers with as high as 38% gain in latency reduction under a variety of job types and incoming rates. The resulting model is also compact in size and has very favorable energy consumption behaviors, making it highly practical for deployment in future SoC systems with built-in neural accelerator.
△ Less
Submitted 11 October, 2021; v1 submitted 28 April, 2021;
originally announced April 2021.
-
A Scalable and Reproducible System-on-Chip Simulation for Reinforcement Learning
Authors:
Tegg Taekyong Sung,
Bo Ryu
Abstract:
Deep Reinforcement Learning (DRL) underlies in a simulated environment and optimizes objective goals. By extending the conventional interaction scheme, this paper proffers gym-ds3, a scalable and reproducible open environment tailored for a high-fidelity Domain-Specific System-on-Chip (DSSoC) application. The simulation corroborates to schedule hierarchical jobs onto heterogeneous System-on-Chip (…
▽ More
Deep Reinforcement Learning (DRL) underlies in a simulated environment and optimizes objective goals. By extending the conventional interaction scheme, this paper proffers gym-ds3, a scalable and reproducible open environment tailored for a high-fidelity Domain-Specific System-on-Chip (DSSoC) application. The simulation corroborates to schedule hierarchical jobs onto heterogeneous System-on-Chip (SoC) processors and bridges the system to reinforcement learning research. We systematically analyze the representative SoC simulator and discuss the primary challenging aspects that the system (1) continuously generates indefinite jobs at a rapid injection rate, (2) optimizes complex objectives, and (3) operates in steady-state scheduling. We provide exemplary snippets and experimentally demonstrate the run-time performances on different schedulers that successfully mimic results achieved from the standard DS3 framework and real-world embedded systems.
△ Less
Submitted 27 April, 2021;
originally announced April 2021.
-
Robust and Scalable Routing with Multi-Agent Deep Reinforcement Learning for MANETs
Authors:
Saeed Kaviani,
Bo Ryu,
Ejaz Ahmed,
Kevin A. Larson,
Anh Le,
Alex Yahja,
Jae H. Kim
Abstract:
Highly dynamic mobile ad-hoc networks (MANETs) are continuing to serve as one of the most challenging environments to develop and deploy robust, efficient, and scalable routing protocols. In this paper, we present DeepCQ+ routing which, in a novel manner, integrates emerging multi-agent deep reinforcement learning (MADRL) techniques into existing Q-learning-based routing protocols and their varian…
▽ More
Highly dynamic mobile ad-hoc networks (MANETs) are continuing to serve as one of the most challenging environments to develop and deploy robust, efficient, and scalable routing protocols. In this paper, we present DeepCQ+ routing which, in a novel manner, integrates emerging multi-agent deep reinforcement learning (MADRL) techniques into existing Q-learning-based routing protocols and their variants, and achieves persistently higher performance across a wide range of MANET configurations while training only on a limited range of network parameters and conditions. Quantitatively, DeepCQ+ shows consistently higher end-to-end throughput with lower overhead compared to its Q-learning-based counterparts with the overall gain of 10-15% in its efficiency. Qualitatively and more significantly, DeepCQ+ maintains remarkably similar performance gains under many scenarios that it was not trained for in terms of network sizes, mobility conditions, and traffic dynamics. To the best of our knowledge, this is the first successful demonstration of MADRL for the MANET routing problem that achieves and maintains a high degree of scalability and robustness even in the environments that are outside the trained range of scenarios. This implies that the proposed hybrid design approach of DeepCQ+ that combines MADRL and Q-learning significantly increases its practicality and explainability because the real-world MANET environment will likely vary outside the trained range of MANET scenarios.
△ Less
Submitted 28 March, 2021; v1 submitted 8 January, 2021;
originally announced January 2021.
-
DeepSoCS: A Neural Scheduler for Heterogeneous System-on-Chip (SoC) Resource Scheduling
Authors:
Tegg Taekyong Sung,
Jeongsoo Ha,
Jeewoo Kim,
Alex Yahja,
Chae-Bong Sohn,
Bo Ryu
Abstract:
In this paper, we~present a novel scheduling solution for a class of System-on-Chip (SoC) systems where heterogeneous chip resources (DSP, FPGA, GPU, etc.) must be efficiently scheduled for continuously arriving hierarchical jobs with their tasks represented by a directed acyclic graph. Traditionally, heuristic algorithms have been widely used for many resource scheduling domains, and Heterogeneou…
▽ More
In this paper, we~present a novel scheduling solution for a class of System-on-Chip (SoC) systems where heterogeneous chip resources (DSP, FPGA, GPU, etc.) must be efficiently scheduled for continuously arriving hierarchical jobs with their tasks represented by a directed acyclic graph. Traditionally, heuristic algorithms have been widely used for many resource scheduling domains, and Heterogeneous Earliest Finish Time (HEFT) has been a dominating state-of-the-art technique across a broad range of heterogeneous resource scheduling domains over many years. Despite their long-standing popularity, HEFT-like algorithms are known to be vulnerable to a small amount of noise added to the environment. Our Deep Reinforcement Learning (DRL)-based SoC Scheduler (DeepSoCS), capable of learning the "best" task ordering under dynamic environment changes, overcomes the brittleness of rule-based schedulers such as HEFT with significantly higher performance across different types of jobs. We~describe a DeepSoCS design process using a real-time heterogeneous SoC scheduling emulator, discuss major challenges, and present two novel neural network design features that lead to outperforming HEFT: (i) hierarchical job- and task-graph embedding; and (ii) efficient use of real-time task information in the state space. Furthermore, we~introduce effective techniques to address two fundamental challenges present in our environment: delayed consequences and joint actions. Through an extensive simulation study, we~show that our DeepSoCS exhibits the significantly higher performance of job execution time than that of HEFT with a higher level of robustness under realistic noise conditions. We~conclude with a discussion of the potential improvements for our DeepSoCS neural scheduler.
△ Less
Submitted 5 June, 2020; v1 submitted 15 May, 2020;
originally announced May 2020.
-
The Demise of Single-Authored Publications in Computer Science: A Citation Network Analysis
Authors:
Brian K. Ryu
Abstract:
In this study, I analyze the DBLP bibliographic database to study role of single author publications in the computer science literature between 1940 and 2019. I examine the demographics and reception by computing the population fraction, citation statistics, and PageRank scores of single author publications over the years. Both the population fraction and reception have been continuously declining…
▽ More
In this study, I analyze the DBLP bibliographic database to study role of single author publications in the computer science literature between 1940 and 2019. I examine the demographics and reception by computing the population fraction, citation statistics, and PageRank scores of single author publications over the years. Both the population fraction and reception have been continuously declining since the 1940s. The overall decaying trend of single author publications is qualitatively consistent with those observed in other scientific disciplines, though the diminution is taking place several decades later than those in the natural sciences. Additionally, I analyze the scope and volume of single author publications, using page length and reference count as first-order approximations of the scope of publications. Although both metrics on average show positive correlations with citation count, single author papers show no significant difference in page or reference counts compared to the rest of the publications, suggesting that there exist other factors that impact the citations of single author publications.
△ Less
Submitted 2 January, 2020;
originally announced January 2020.
-
Neural Heterogeneous Scheduler
Authors:
Tegg Taekyong Sung,
Valliappa Chockalingam,
Alex Yahja,
Bo Ryu
Abstract:
Access to parallel and distributed computation has enabled researchers and developers to improve algorithms and performance in many applications. Recent research has focused on next generation special purpose systems with multiple kinds of coprocessors, known as heterogeneous system-on-chips (SoC). In this paper, we introduce a method to intelligently schedule--and learn to schedule--a stream of t…
▽ More
Access to parallel and distributed computation has enabled researchers and developers to improve algorithms and performance in many applications. Recent research has focused on next generation special purpose systems with multiple kinds of coprocessors, known as heterogeneous system-on-chips (SoC). In this paper, we introduce a method to intelligently schedule--and learn to schedule--a stream of tasks to available processing elements in such a system. We use deep reinforcement learning enabling complex sequential decision making and empirically show that our reinforcement learning system provides for a viable, better alternative to conventional scheduling heuristics with respect to minimizing execution time.
△ Less
Submitted 9 June, 2019;
originally announced June 2019.
-
Highrisk Prediction from Electronic Medical Records via Deep Attention Networks
Authors:
You ** Kim,
Yun-Geun Lee,
Jeong Whun Kim,
** Joo Park,
Borim Ryu,
Jung-Woo Ha
Abstract:
Predicting highrisk vascular diseases is a significant issue in the medical domain. Most predicting methods predict the prognosis of patients from pathological and radiological measurements, which are expensive and require much time to be analyzed. Here we propose deep attention models that predict the onset of the high risky vascular disease from symbolic medical histories sequence of hypertensio…
▽ More
Predicting highrisk vascular diseases is a significant issue in the medical domain. Most predicting methods predict the prognosis of patients from pathological and radiological measurements, which are expensive and require much time to be analyzed. Here we propose deep attention models that predict the onset of the high risky vascular disease from symbolic medical histories sequence of hypertension patients such as ICD-10 and pharmacy codes only, Medical History-based Prediction using Attention Network (MeHPAN). We demonstrate two types of attention models based on 1) bidirectional gated recurrent unit (R-MeHPAN) and 2) 1D convolutional multilayer model (C-MeHPAN). Two MeHPAN models are evaluated on approximately 50,000 hypertension patients with respect to precision, recall, f1-measure and area under the curve (AUC). Experimental results show that our MeHPAN methods outperform standard classification models. Comparing two MeHPANs, R-MeHPAN provides more better discriminative capability with respect to all metrics while C-MeHPAN presents much shorter training time with competitive accuracy.
△ Less
Submitted 30 November, 2017;
originally announced December 2017.
