-
LANCAR: Leveraging Language for Context-Aware Robot Locomotion in Unstructured Environments
Authors:
Chak Lam Shek,
Xiyang Wu,
Wesley A. Suttle,
Carl Busart,
Erin Zaroukian,
Dinesh Manocha,
Pratap Tokekar,
Amrit Singh Bedi
Abstract:
Navigating robots through unstructured terrains is challenging, primarily due to the dynamic environmental changes. While humans adeptly navigate such terrains by using context from their observations, creating a similar context-aware navigation system for robots is difficult. The essence of the issue lies in the acquisition and interpretation of contextual information, a task complicated by the i…
▽ More
Navigating robots through unstructured terrains is challenging, primarily due to the dynamic environmental changes. While humans adeptly navigate such terrains by using context from their observations, creating a similar context-aware navigation system for robots is difficult. The essence of the issue lies in the acquisition and interpretation of contextual information, a task complicated by the inherent ambiguity of human language. In this work, we introduce LANCAR, which addresses this issue by combining a context translator with reinforcement learning (RL) agents for context-aware locomotion. LANCAR allows robots to comprehend contextual information through Large Language Models (LLMs) sourced from human observers and convert this information into actionable contextual embeddings. These embeddings, combined with the robot's sensor data, provide a complete input for the RL agent's policy network. We provide an extensive evaluation of LANCAR under different levels of contextual ambiguity and compare with alternative methods. The experimental results showcase the superior generalizability and adaptability across different terrains. Notably, LANCAR shows at least a 7.4% increase in episodic reward over the best alternatives, highlighting its potential to enhance robotic navigation in unstructured environments. More details and experiment videos could be found in http://raaslab.org/projects/LLM_Context_Estimation/.
△ Less
Submitted 19 March, 2024; v1 submitted 30 September, 2023;
originally announced October 2023.
-
Emergent Behaviors in Multi-Agent Target Acquisition
Authors:
Piyush K. Sharma,
Erin Zaroukian,
Derrik E. Asher,
Bryson Howell
Abstract:
Only limited studies and superficial evaluations are available on agents' behaviors and roles within a Multi-Agent System (MAS). We simulate a MAS using Reinforcement Learning (RL) in a pursuit-evasion (a.k.a predator-prey pursuit) game, which shares task goals with target acquisition, and we create different adversarial scenarios by replacing RL-trained pursuers' policies with two distinct (non-R…
▽ More
Only limited studies and superficial evaluations are available on agents' behaviors and roles within a Multi-Agent System (MAS). We simulate a MAS using Reinforcement Learning (RL) in a pursuit-evasion (a.k.a predator-prey pursuit) game, which shares task goals with target acquisition, and we create different adversarial scenarios by replacing RL-trained pursuers' policies with two distinct (non-RL) analytical strategies. Using heatmaps of agents' positions (state-space variable) over time, we are able to categorize an RL-trained evader's behaviors. The novelty of our approach entails the creation of an influential feature set that reveals underlying data regularities, which allow us to classify an agent's behavior. This classification may aid in catching the (enemy) targets by enabling us to identify and predict their behaviors, and when extended to pursuers, this approach towards identifying teammates' behavior may allow agents to coordinate more effectively.
△ Less
Submitted 15 December, 2022;
originally announced December 2022.
-
Strategic Maneuver and Disruption with Reinforcement Learning Approaches for Multi-Agent Coordination
Authors:
Derrik E. Asher,
Anjon Basak,
Rolando Fernandez,
Piyush K. Sharma,
Erin G. Zaroukian,
Christopher D. Hsu,
Michael R. Dorothy,
Thomas Mahre,
Gerardo Galindo,
Luke Frerichs,
John Rogers,
John Fossaceca
Abstract:
Reinforcement learning (RL) approaches can illuminate emergent behaviors that facilitate coordination across teams of agents as part of a multi-agent system (MAS), which can provide windows of opportunity in various military tasks. Technologically advancing adversaries pose substantial risks to a friendly nation's interests and resources. Superior resources alone are not enough to defeat adversari…
▽ More
Reinforcement learning (RL) approaches can illuminate emergent behaviors that facilitate coordination across teams of agents as part of a multi-agent system (MAS), which can provide windows of opportunity in various military tasks. Technologically advancing adversaries pose substantial risks to a friendly nation's interests and resources. Superior resources alone are not enough to defeat adversaries in modern complex environments because adversaries create standoff in multiple domains against predictable military doctrine-based maneuvers. Therefore, as part of a defense strategy, friendly forces must use strategic maneuvers and disruption to gain superiority in complex multi-faceted domains such as multi-domain operations (MDO). One promising avenue for implementing strategic maneuver and disruption to gain superiority over adversaries is through coordination of MAS in future military operations. In this paper, we present overviews of prominent works in the RL domain with their strengths and weaknesses for overcoming the challenges associated with performing autonomous strategic maneuver and disruption in military contexts.
△ Less
Submitted 17 March, 2022;
originally announced March 2022.
-
Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training
Authors:
Piyush K. Sharma,
Rolando Fernandez,
Erin Zaroukian,
Michael Dorothy,
Anjon Basak,
Derrik E. Asher
Abstract:
Much work has been dedicated to the exploration of Multi-Agent Reinforcement Learning (MARL) paradigms implementing a centralized learning with decentralized execution (CLDE) approach to achieve human-like collaboration in cooperative tasks. Here, we discuss variations of centralized training and describe a recent survey of algorithmic approaches. The goal is to explore how different implementatio…
▽ More
Much work has been dedicated to the exploration of Multi-Agent Reinforcement Learning (MARL) paradigms implementing a centralized learning with decentralized execution (CLDE) approach to achieve human-like collaboration in cooperative tasks. Here, we discuss variations of centralized training and describe a recent survey of algorithmic approaches. The goal is to explore how different implementations of information sharing mechanism in centralized learning may give rise to distinct group coordinated behaviors in multi-agent systems performing cooperative tasks.
△ Less
Submitted 29 July, 2021;
originally announced July 2021.
-
Measuring collaborative emergent behavior in multi-agent reinforcement learning
Authors:
Sean L. Barton,
Nicholas R. Waytowich,
Erin Zaroukian,
Derrik E. Asher
Abstract:
Multi-agent reinforcement learning (RL) has important implications for the future of human-agent teaming. We show that improved performance with multi-agent RL is not a guarantee of the collaborative behavior thought to be important for solving multi-agent tasks. To address this, we present a novel approach for quantitatively assessing collaboration in continuous spatial tasks with multi-agent RL.…
▽ More
Multi-agent reinforcement learning (RL) has important implications for the future of human-agent teaming. We show that improved performance with multi-agent RL is not a guarantee of the collaborative behavior thought to be important for solving multi-agent tasks. To address this, we present a novel approach for quantitatively assessing collaboration in continuous spatial tasks with multi-agent RL. Such a metric is useful for measuring collaboration between computational agents and may serve as a training signal for collaboration in future RL paradigms involving humans.
△ Less
Submitted 23 July, 2018;
originally announced July 2018.
-
Adapting the Predator-Prey Game Theoretic Environment to Army Tactical Edge Scenarios with Computational Multiagent Systems
Authors:
Derrik E. Asher,
Erin Zaroukian,
Sean L. Barton
Abstract:
The historical origins of the game theoretic predator-prey pursuit problem can be traced back to Benda, et al., 1985 [1]. Their work adapted the predator-prey ecology problem into a pursuit environment which focused on the dynamics of cooperative behavior between predator agents. Modifications to the predator-prey ecology problem [2] have been implemented to understand how variations to predator […
▽ More
The historical origins of the game theoretic predator-prey pursuit problem can be traced back to Benda, et al., 1985 [1]. Their work adapted the predator-prey ecology problem into a pursuit environment which focused on the dynamics of cooperative behavior between predator agents. Modifications to the predator-prey ecology problem [2] have been implemented to understand how variations to predator [3] and prey [3-5] attributes, including communication [6], can modify dynamic interactions between entities that emerge within that environment [7-9]. Furthermore, the predator-prey pursuit environment has become a testbed for simulation experiments with computational multiagent systems [10-12]. This article extends the theoretical contributions of previous work by providing 1) additional variations to predator and prey attributes for simulated multiagent systems in the pursuit problem, and 2) military-relevant predator-prey environments simulating highly dynamic, tactical edge scenarios that Soldiers might encounter on future battlefields. Through this exploration of simulated tactical edge scenarios with computational multiagent systems, Soldiers will have a greater chance to achieve overmatch on the battlefields of tomorrow.
△ Less
Submitted 16 July, 2018;
originally announced July 2018.
-
Malware in the Future? Forecasting of Analyst Detection of Cyber Events
Authors:
Jonathan Z. Bakdash,
Steve Hutchinson,
Erin G. Zaroukian,
Laura R. Marusich,
Saravanan Thirumuruganathan,
Charmaine Sample,
Blaine Hoffman,
Gautam Das
Abstract:
There have been extensive efforts in government, academia, and industry to anticipate, forecast, and mitigate cyber attacks. A common approach is time-series forecasting of cyber attacks based on data from network telescopes, honeypots, and automated intrusion detection/prevention systems. This research has uncovered key insights such as systematicity in cyber attacks. Here, we propose an alternat…
▽ More
There have been extensive efforts in government, academia, and industry to anticipate, forecast, and mitigate cyber attacks. A common approach is time-series forecasting of cyber attacks based on data from network telescopes, honeypots, and automated intrusion detection/prevention systems. This research has uncovered key insights such as systematicity in cyber attacks. Here, we propose an alternate perspective of this problem by performing forecasting of attacks that are analyst-detected and -verified occurrences of malware. We call these instances of malware cyber event data. Specifically, our dataset was analyst-detected incidents from a large operational Computer Security Service Provider (CSSP) for the U.S. Department of Defense, which rarely relies only on automated systems. Our data set consists of weekly counts of cyber events over approximately seven years. Since all cyber events were validated by analysts, our dataset is unlikely to have false positives which are often endemic in other sources of data. Further, the higher-quality data could be used for a number for resource allocation, estimation of security resources, and the development of effective risk-management strategies. We used a Bayesian State Space Model for forecasting and found that events one week ahead could be predicted. To quantify bursts, we used a Markov model. Our findings of systematicity in analyst-detected cyber attacks are consistent with previous work using other sources. The advanced information provided by a forecast may help with threat awareness by providing a probable value and range for future cyber events one week ahead. Other potential applications for cyber event forecasting include proactive allocation of resources and capabilities for cyber defense (e.g., analyst staffing and sensor configuration) in CSSPs. Enhanced threat awareness may improve cybersecurity.
△ Less
Submitted 8 June, 2018; v1 submitted 11 July, 2017;
originally announced July 2017.
