-
RFRL Gym: A Reinforcement Learning Testbed for Cognitive Radio Applications
Authors:
Daniel Rosen,
Illa Rochez,
Caleb McIrvin,
Joshua Lee,
Kevin D'Alessandro,
Max Wiecek,
Nhan Hoang,
Ramzy Saffarini,
Sam Philips,
Vanessa Jones,
Will Ivey,
Zavier Harris-Smart,
Zavion Harris-Smart,
Zayden Chin,
Amos Johnson,
Alyse M. Jones,
William C. Headley
Abstract:
Radio Frequency Reinforcement Learning (RFRL) is anticipated to be a widely applicable technology in the next generation of wireless communication systems, particularly 6G and next-gen military communications. Given this, our research is focused on develo** a tool to promote the development of RFRL techniques that leverage spectrum sensing. In particular, the tool was designed to address two cog…
▽ More
Radio Frequency Reinforcement Learning (RFRL) is anticipated to be a widely applicable technology in the next generation of wireless communication systems, particularly 6G and next-gen military communications. Given this, our research is focused on develo** a tool to promote the development of RFRL techniques that leverage spectrum sensing. In particular, the tool was designed to address two cognitive radio applications, specifically dynamic spectrum access and jamming. In order to train and test reinforcement learning (RL) algorithms for these applications, a simulation environment is necessary to simulate the conditions that an agent will encounter within the Radio Frequency (RF) spectrum. In this paper, such an environment has been developed, herein referred to as the RFRL Gym. Through the RFRL Gym, users can design their own scenarios to model what an RL agent may encounter within the RF spectrum as well as experiment with different spectrum sensing techniques. Additionally, the RFRL Gym is a subclass of OpenAI gym, enabling the use of third-party ML/RL Libraries. We plan to open-source this codebase to enable other researchers to utilize the RFRL Gym to test their own scenarios and RL algorithms, ultimately leading to the advancement of RL research in the wireless communications domain. This paper describes in further detail the components of the Gym, results from example scenarios, and plans for future additions.
Index Terms-machine learning, reinforcement learning, wireless communications, dynamic spectrum access, OpenAI gym
△ Less
Submitted 20 December, 2023;
originally announced January 2024.
-
Scaled-Time-Attention Robust Edge Network
Authors:
Richard Lau,
Lihan Yao,
Todd Huster,
William Johnson,
Stephen Arleth,
Justin Wong,
Devin Ridge,
Michael Fletcher,
William C. Headley
Abstract:
This paper describes a systematic approach towards building a new family of neural networks based on a delay-loop version of a reservoir neural network. The resulting architecture, called Scaled-Time-Attention Robust Edge (STARE) network, exploits hyper dimensional space and non-multiply-and-add computation to achieve a simpler architecture, which has shallow layers, is simple to train, and is bet…
▽ More
This paper describes a systematic approach towards building a new family of neural networks based on a delay-loop version of a reservoir neural network. The resulting architecture, called Scaled-Time-Attention Robust Edge (STARE) network, exploits hyper dimensional space and non-multiply-and-add computation to achieve a simpler architecture, which has shallow layers, is simple to train, and is better suited for Edge applications, such as Internet of Things (IoT), over traditional deep neural networks. STARE incorporates new AI concepts such as Attention and Context, and is best suited for temporal feature extraction and classification. We demonstrate that STARE is applicable to a variety of applications with improved performance and lower implementation complexity. In particular, we showed a novel way of applying a dual-loop configuration to detection and identification of drone vs bird in a counter Unmanned Air Systems (UAS) detection application by exploiting both spatial (video frame) and temporal (trajectory) information. We also demonstrated that the STARE performance approaches that of a State-of-the-Art deep neural network in classifying RF modulations, and outperforms Long Short-term Memory (LSTM) in a special case of Mackey Glass time series prediction. To demonstrate hardware efficiency, we designed and developed an FPGA implementation of the STARE algorithm to demonstrate its low-power and high-throughput operations. In addition, we illustrate an efficient structure for integrating a massively parallel implementation of the STARE algorithm for ASIC implementation.
△ Less
Submitted 9 July, 2021;
originally announced July 2021.
-
Training Data Augmentation for Deep Learning Radio Frequency Systems
Authors:
William H. Clark IV,
Steven Hauser,
William C. Headley,
Alan J. Michaels
Abstract:
Applications of machine learning are subject to three major components that contribute to the final performance metrics. Within the category of neural networks, and deep learning specifically, the first two are the architecture for the model being trained and the training approach used. This work focuses on the third component, the data used during training. The primary questions that arise are ``…
▽ More
Applications of machine learning are subject to three major components that contribute to the final performance metrics. Within the category of neural networks, and deep learning specifically, the first two are the architecture for the model being trained and the training approach used. This work focuses on the third component, the data used during training. The primary questions that arise are ``what is in the data'' and ``what within the data matters?'' Looking into the Radio Frequency Machine Learning (RFML) field of Automatic Modulation Classification (AMC) as an example of a tool used for situational awareness, the use of synthetic, captured, and augmented data are examined and compared to provide insights about the quantity and quality of the available data necessary to achieve desired performance levels. There are three questions discussed within this work: (1) how useful a synthetically trained system is expected to be when deployed without considering the environment within the synthesis, (2) how can augmentation be leveraged within the RFML domain, and lastly, (3) what impact knowledge of degradations to the signal caused by the transmission channel contributes to the performance of a system. In general, the examined data types each have useful contributions to a final application, but captured data germane to the intended use case will always provide more significant information and enable the greatest performance. Despite the benefit of captured data, the difficulties and costs that arise from live collection often make the quantity of data needed to achieve peak performance impractical. This paper helps quantify the balance between real and synthetic data, offering concrete examples where training data is parametrically varied in size and source.
△ Less
Submitted 4 January, 2021; v1 submitted 30 September, 2020;
originally announced October 2020.
-
Investigating a Spectral Deception Loss Metric for Training Machine Learning-based Evasion Attacks
Authors:
Matthew DelVecchio,
Vanessa Arndorfer,
William C. Headley
Abstract:
Adversarial evasion attacks have been very successful in causing poor performance in a wide variety of machine learning applications. One such application is radio frequency spectrum sensing. While evasion attacks have proven particularly successful in this area, they have done so at the detriment of the signal's intended purpose. More specifically, for real-world applications of interest, the res…
▽ More
Adversarial evasion attacks have been very successful in causing poor performance in a wide variety of machine learning applications. One such application is radio frequency spectrum sensing. While evasion attacks have proven particularly successful in this area, they have done so at the detriment of the signal's intended purpose. More specifically, for real-world applications of interest, the resulting perturbed signal that is transmitted to evade an eavesdropper must not deviate far from the original signal, less the intended information is destroyed. Recent work by the authors and others has demonstrated an attack framework that allows for intelligent balancing between these conflicting goals of evasion and communication. However, while these methodologies consider creating adversarial signals that minimize communications degradation, they have been shown to do so at the expense of the spectral shape of the signal. This opens the adversarial signal up to defenses at the eavesdropper such as filtering, which could render the attack ineffective. To remedy this, this work introduces a new spectral deception loss metric that can be implemented during the training process to force the spectral shape to be more in-line with the original signal. As an initial proof of concept, a variety of methods are presented that provide a starting point for this proposed loss. Through performance analysis, it is shown that these techniques are effective in controlling the shape of the adversarial signal.
△ Less
Submitted 26 May, 2020;
originally announced May 2020.
-
Effects of Forward Error Correction on Communications Aware Evasion Attacks
Authors:
Matthew DelVecchio,
Bryse Flowers,
William C. Headley
Abstract:
Recent work has shown the impact of adversarial machine learning on deep neural networks (DNNs) developed for Radio Frequency Machine Learning (RFML) applications. While these attacks have been shown to be successful in disrupting the performance of an eavesdropper, they fail to fully support the primary goal of successful intended communication. To remedy this, a communications-aware attack frame…
▽ More
Recent work has shown the impact of adversarial machine learning on deep neural networks (DNNs) developed for Radio Frequency Machine Learning (RFML) applications. While these attacks have been shown to be successful in disrupting the performance of an eavesdropper, they fail to fully support the primary goal of successful intended communication. To remedy this, a communications-aware attack framework was recently developed that allows for a more effective balance between the opposing goals of evasion and intended communication through the novel use of a DNN to intelligently create the adversarial communication signal. Given the near ubiquitous usage of forward error correction (FEC) coding in the majority of deployed systems to correct errors that arise, incorporating FEC in this framework is a natural extension of this prior work and will allow for improved performance in more adverse environments. This work therefore provides contributions to the framework through improved loss functions and design considerations to incorporate inherent knowledge of the usage of FEC codes within the transmitted signal. Performance analysis shows that FEC coding improves the communications aware adversarial attack even if no explicit knowledge of the coding scheme is assumed and allows for improved performance over the prior art in balancing the opposing goals of evasion and intended communications.
△ Less
Submitted 26 May, 2020;
originally announced May 2020.
-
Deep Learning for RF Signal Classification in Unknown and Dynamic Spectrum Environments
Authors:
Yi Shi,
Kemal Davaslioglu,
Yalin E. Sagduyu,
William C. Headley,
Michael Fowler,
Gilbert Green
Abstract:
Dynamic spectrum access (DSA) benefits from detection and classification of interference sources including in-network users, out-network users, and jammers that may all coexist in a wireless network. We present a deep learning based signal (modulation) classification solution in a realistic wireless network setting, where 1) signal types may change over time; 2) some signal types may be unknown fo…
▽ More
Dynamic spectrum access (DSA) benefits from detection and classification of interference sources including in-network users, out-network users, and jammers that may all coexist in a wireless network. We present a deep learning based signal (modulation) classification solution in a realistic wireless network setting, where 1) signal types may change over time; 2) some signal types may be unknown for which there is no training data; 3) signals may be spoofed such as the smart jammers replaying other signal types; and 4) different signal types may be superimposed due to the interference from concurrent transmissions. For case 1, we apply continual learning and train a Convolutional Neural Network (CNN) using an Elastic Weight Consolidation (EWC) based loss. For case 2, we detect unknown signals via outlier detection applied to the outputs of convolutional layers using Minimum Covariance Determinant (MCD) and k-means clustering methods. For case 3, we extend the CNN structure to capture phase shifts due to radio hardware effects to identify the spoofing signal sources. For case 4, we apply blind source separation using Independent Component Analysis (ICA) to separate interfering signals. We utilize the signal classification results in a distributed scheduling protocol, where in-network (secondary) users employ signal classification scores to make channel access decisions and share the spectrum with each other while avoiding interference with out-network (primary) users and jammers. Compared with benchmark TDMA-based schemes, we show that distributed scheduling constructed upon signal classification results provides major improvements to in-network user throughput and out-network user success ratio.
△ Less
Submitted 25 September, 2019;
originally announced September 2019.
-
Evaluating Adversarial Evasion Attacks in the Context of Wireless Communications
Authors:
Bryse Flowers,
R. Michael Buehrer,
William C. Headley
Abstract:
Recent advancements in radio frequency machine learning (RFML) have demonstrated the use of raw in-phase and quadrature (IQ) samples for multiple spectrum sensing tasks. Yet, deep learning techniques have been shown, in other applications, to be vulnerable to adversarial machine learning (ML) techniques, which seek to craft small perturbations that are added to the input to cause a misclassificati…
▽ More
Recent advancements in radio frequency machine learning (RFML) have demonstrated the use of raw in-phase and quadrature (IQ) samples for multiple spectrum sensing tasks. Yet, deep learning techniques have been shown, in other applications, to be vulnerable to adversarial machine learning (ML) techniques, which seek to craft small perturbations that are added to the input to cause a misclassification. The current work differentiates the threats that adversarial ML poses to RFML systems based on where the attack is executed from: direct access to classifier input, synchronously transmitted over the air (OTA), or asynchronously transmitted from a separate device. Additionally, the current work develops a methodology for evaluating adversarial success in the context of wireless communications, where the primary metric of interest is bit error rate and not human perception, as is the case in image recognition. The methodology is demonstrated using the well known Fast Gradient Sign Method to evaluate the vulnerabilities of raw IQ based Automatic Modulation Classification and concludes RFML is vulnerable to adversarial examples, even in OTA attacks. However, RFML domain specific receiver effects, which would be encountered in an OTA attack, can present significant impairments to adversarial evasion.
△ Less
Submitted 1 March, 2019;
originally announced March 2019.
