-
Preference-Based Planning in Stochastic Environments: From Partially-Ordered Temporal Goals to Most Preferred Policies
Authors:
Hazhar Rahmani,
Abhishek N. Kulkarni,
Jie Fu
Abstract:
Human preferences are not always represented via complete linear orders: It is natural to employ partially-ordered preferences for expressing incomparable outcomes. In this work, we consider decision-making and probabilistic planning in stochastic systems modeled as Markov decision processes (MDPs), given a partially ordered preference over a set of temporally extended goals. Specifically, each te…
▽ More
Human preferences are not always represented via complete linear orders: It is natural to employ partially-ordered preferences for expressing incomparable outcomes. In this work, we consider decision-making and probabilistic planning in stochastic systems modeled as Markov decision processes (MDPs), given a partially ordered preference over a set of temporally extended goals. Specifically, each temporally extended goal is expressed using a formula in Linear Temporal Logic on Finite Traces (LTL$_f$). To plan with the partially ordered preference, we introduce order theory to map a preference over temporal goals to a preference over policies for the MDP. Accordingly, a most preferred policy under a stochastic ordering induces a stochastic nondominated probability distribution over the finite paths in the MDP. To synthesize a most preferred policy, our technical approach includes two key steps. In the first step, we develop a procedure to transform a partially ordered preference over temporal goals into a computational model, called preference automaton, which is a semi-automaton with a partial order over acceptance conditions. In the second step, we prove that finding a most preferred policy is equivalent to computing a Pareto-optimal policy in a multi-objective MDP that is constructed from the original MDP, the preference automaton, and the chosen stochastic ordering relation. Throughout the paper, we employ running examples to illustrate the proposed preference specification and solution approaches. We demonstrate the efficacy of our algorithm using these examples, providing detailed analysis, and then discuss several potential future directions.
△ Less
Submitted 26 March, 2024;
originally announced March 2024.
-
Synthesis of Opacity-Enforcing Winning Strategies Against Colluded Opponent
Authors:
Chongyang Shi,
Abhishek N. Kulkarni,
Hazhar Rahmani,
Jie Fu
Abstract:
This paper studies a language-based opacity enforcement in a two-player, zero-sum game on a graph. In this game, player 1 (P1) wins if it can achieve a secret temporal goal described by the language of a finite automaton, no matter what strategy the opponent player 2 (P2) selects. In addition, P1 aims to win while making its goal opaque to a passive observer with imperfect information. However, P2…
▽ More
This paper studies a language-based opacity enforcement in a two-player, zero-sum game on a graph. In this game, player 1 (P1) wins if it can achieve a secret temporal goal described by the language of a finite automaton, no matter what strategy the opponent player 2 (P2) selects. In addition, P1 aims to win while making its goal opaque to a passive observer with imperfect information. However, P2 colludes with the observer to reveal P1's secret whenever P2 cannot prevent P1 from achieving its goal, and therefore, opacity must be enforced against P2. We show that a winning and opacity-enforcing strategy for P1 can be computed by reducing the problem to solving a reachability game augmented with observer's belief states. Furthermore, if such a strategy does not exist, winning for P1 must entail the price of revealing his secret to the observer. We demonstrate our game-theoretic solution of opacity-enforcement control through a small illustrative example and in a robot motion planning problem.
△ Less
Submitted 3 April, 2023;
originally announced April 2023.
-
Opportunistic Qualitative Planning in Stochastic Systems with Incomplete Preferences over Reachability Objectives
Authors:
Abhishek N. Kulkarni,
Jie Fu
Abstract:
Preferences play a key role in determining what goals/constraints to satisfy when not all constraints can be satisfied simultaneously. In this paper, we study how to synthesize preference satisfying plans in stochastic systems, modeled as an MDP, given a (possibly incomplete) combinative preference model over temporally extended goals. We start by introducing new semantics to interpret preferences…
▽ More
Preferences play a key role in determining what goals/constraints to satisfy when not all constraints can be satisfied simultaneously. In this paper, we study how to synthesize preference satisfying plans in stochastic systems, modeled as an MDP, given a (possibly incomplete) combinative preference model over temporally extended goals. We start by introducing new semantics to interpret preferences over infinite plays of the stochastic system. Then, we introduce a new notion of improvement to enable comparison between two prefixes of an infinite play. Based on this, we define two solution concepts called safe and positively improving (SPI) and safe and almost-surely improving (SASI) that enforce improvements with a positive probability and with probability one, respectively. We construct a model called an improvement MDP, in which the synthesis of SPI and SASI strategies that guarantee at least one improvement reduces to computing positive and almost-sure winning strategies in an MDP. We present an algorithm to synthesize the SPI and SASI strategies that induce multiple sequential improvements. We demonstrate the proposed approach using a robot motion planning problem.
△ Less
Submitted 4 October, 2022;
originally announced October 2022.
-
Probabilistic Planning with Partially Ordered Preferences over Temporal Goals
Authors:
Hazhar Rahmani,
Abhishek N. Kulkarni,
Jie Fu
Abstract:
In this paper, we study planning in stochastic systems, modeled as Markov decision processes (MDPs), with preferences over temporally extended goals. Prior work on temporal planning with preferences assumes that the user preferences form a total order, meaning that every pair of outcomes are comparable with each other. In this work, we consider the case where the preferences over possible outcomes…
▽ More
In this paper, we study planning in stochastic systems, modeled as Markov decision processes (MDPs), with preferences over temporally extended goals. Prior work on temporal planning with preferences assumes that the user preferences form a total order, meaning that every pair of outcomes are comparable with each other. In this work, we consider the case where the preferences over possible outcomes are a partial order rather than a total order. We first introduce a variant of deterministic finite automaton, referred to as a preference DFA, for specifying the user's preferences over temporally extended goals. Based on the order theory, we translate the preference DFA to a preference relation over policies for probabilistic planning in a labeled MDP. In this treatment, a most preferred policy induces a weak-stochastic nondominated probability distribution over the finite paths in the MDP. The proposed planning algorithm hinges on the construction of a multi-objective MDP. We prove that a weak-stochastic nondominated policy given the preference specification is Pareto-optimal in the constructed multi-objective MDP, and vice versa. Throughout the paper, we employ a running example to demonstrate the proposed preference specification and solution approaches. We show the efficacy of our algorithm using the example with detailed analysis, and then discuss possible future directions.
△ Less
Submitted 7 March, 2023; v1 submitted 25 September, 2022;
originally announced September 2022.
-
Synthesizing Attack-Aware Control and Active Sensing Strategies under Reactive Sensor Attacks
Authors:
Sumukha Udupa,
Abhishek N. Kulkarni,
Shuo Han,
Nandi O. Leslie,
Charles A. Kamhoua,
Jie Fu
Abstract:
We consider the probabilistic planning problem for a defender (P1) who can jointly query the sensors and take control actions to reach a set of goal states while being aware of possible sensor attacks by an adversary (P2) who has perfect observations. To synthesize a provably-correct, attack-aware joint control and active sensing strategy for P1, we construct a stochastic game on graph with augmen…
▽ More
We consider the probabilistic planning problem for a defender (P1) who can jointly query the sensors and take control actions to reach a set of goal states while being aware of possible sensor attacks by an adversary (P2) who has perfect observations. To synthesize a provably-correct, attack-aware joint control and active sensing strategy for P1, we construct a stochastic game on graph with augmented states that include the actual game state (known only to the attacker), the belief of the defender about the game state (constructed by the attacker based on his knowledge of defender's observations). We present an algorithm to compute a belief-based, randomized strategy for P1 to ensure satisfying the reachability objective with probability one, under the worst-case sensor attack carried out by an informed P2. We prove the correctness of the algorithm and illustrate using an example.
△ Less
Submitted 29 November, 2022; v1 submitted 28 March, 2022;
originally announced April 2022.
-
Opportunistic Qualitative Planning in Stochastic Systems with Preferences over Temporal Logic Objectives
Authors:
Abhishek Ninad Kulkarni,
Jie Fu
Abstract:
Preferences play a key role in determining what goals/constraints to satisfy when not all constraints can be satisfied simultaneously. In this work, we study preference-based planning in a stochastic system modeled as a Markov decision process, subject to a possible incomplete preference over temporally extended goals. Our contributions are three folds: First, we introduce a preference language to…
▽ More
Preferences play a key role in determining what goals/constraints to satisfy when not all constraints can be satisfied simultaneously. In this work, we study preference-based planning in a stochastic system modeled as a Markov decision process, subject to a possible incomplete preference over temporally extended goals. Our contributions are three folds: First, we introduce a preference language to specify preferences over temporally extended goals. Second, we define a novel automata-theoretic model to represent the preorder induced by given preference relation. The automata representation of preferences enables us to develop a preference-based planning algorithm for stochastic systems. Finally, we show how to synthesize opportunistic strategies that achieves an outcome that improves upon the current satisfiable outcome, with positive probability or with probability one, in a stochastic system. We illustrate our solution approaches using a robot motion planning example.
△ Less
Submitted 25 March, 2022;
originally announced March 2022.
-
Synthesis of Deceptive Strategies in Reachability Games with Action Misperception (Technical Report)
Authors:
Abhishek N. Kulkarni,
Jie Fu
Abstract:
Strategic deception is an act of manipulating the opponent's perception to gain strategic advantages. In this paper, we study synthesis of deceptive winning strategies in two-player turn-based zero-sum reachability games on graphs with one-sided incomplete information of action sets. In particular, we consider the class of games in which Player 1 (P1) starts with a non-empty set of private actions…
▽ More
Strategic deception is an act of manipulating the opponent's perception to gain strategic advantages. In this paper, we study synthesis of deceptive winning strategies in two-player turn-based zero-sum reachability games on graphs with one-sided incomplete information of action sets. In particular, we consider the class of games in which Player 1 (P1) starts with a non-empty set of private actions, which she may 'reveal' to Player 2 (P2) during the course of the game. P2 is equipped with an inference mechanism using which he updates his perception of P1's action set whenever a new action is revealed. Under this information structure, the objective of P1 is to reach a set of goal states in the game graph while that of P2 is to prevent it. We address the question: how can P1 leverage her information advantages to deceive P2 into choosing actions that in turn benefit P1? To this end, we introduce a dynamic hypergame model to capture the reachability game with evolving misperception of P2. Analyzing the game qualitatively, we design algorithms to synthesize deceptive sure and almost-sure winning regions, and establish two key results: (1) under sure-winning condition, deceptive winning strategy is equivalent to the non-deceptive winning strategy - i.e. use of deception has no advantages, (2) under almost-sure winning condition, the deceptive winning strategy could be more powerful than the non-deceptive strategy. We illustrate our algorithms using a capture-the-flag game, and demonstrate the use of proposed approach to a larger class of games with temporal logic objectives.
△ Less
Submitted 23 April, 2021;
originally announced April 2021.
-
Qualitative Planning in Imperfect Information Games with Active Sensing and Reactive Sensor Attacks: Cost of Unawareness
Authors:
Abhishek N. Kulkarni,
Shuo Han,
Nandi O. Leslie,
Charles A. Kamhoua,
Jie Fu
Abstract:
We consider the probabilistic planning problem where the agent (called Player 1, or P1) can jointly plan the control actions and sensor queries in a sensor network and an attacker (called player 2, or P2) can carry out attacks on the sensors. We model such an adversarial interaction using a formal model -- a reachability game with partially controllable observation functions. The main contribution…
▽ More
We consider the probabilistic planning problem where the agent (called Player 1, or P1) can jointly plan the control actions and sensor queries in a sensor network and an attacker (called player 2, or P2) can carry out attacks on the sensors. We model such an adversarial interaction using a formal model -- a reachability game with partially controllable observation functions. The main contribution of this paper is to assess the cost of P1's unawareness: Suppose P1 misinterprets the sensor failures as probabilistic node failures due to unreliable network communication, and P2 is aware of P1's misinterpretation in addition to her partial observability. Then, from which states can P2 carry out sensor attacks to ensure, with probability one, that P1 will not be able to complete her reachability task even though, due to misinterpretation, P1 believes that she can almost-surely achieve her task. We develop an algorithm to solve the almost-sure winning sensor-attack strategy given P1's observation-based strategy. Our attack analysis could be used for attack detection in wireless communication networks and the design of provably secured attack-aware sensor allocation in decision-theoretic models for cyber-physical systems.
△ Less
Submitted 2 May, 2021; v1 submitted 31 March, 2021;
originally announced April 2021.
-
Decoy Allocation Games on Graphs with Temporal Logic Objectives
Authors:
Abhishek N. Kulkarni,
Jie Fu,
Huan Luo,
Charles A. Kamhoua,
Nandi O. Leslie
Abstract:
We study a class of games, in which the adversary (attacker) is to satisfy a complex mission specified in linear temporal logic, and the defender is to prevent the adversary from achieving its goal. A deceptive defender can allocate decoys, in addition to defense actions, to create disinformation for the attacker. Thus, we focus on the problem of jointly synthesizing a decoy placement strategy and…
▽ More
We study a class of games, in which the adversary (attacker) is to satisfy a complex mission specified in linear temporal logic, and the defender is to prevent the adversary from achieving its goal. A deceptive defender can allocate decoys, in addition to defense actions, to create disinformation for the attacker. Thus, we focus on the problem of jointly synthesizing a decoy placement strategy and a deceptive defense strategy that maximally exploits the incomplete information the attacker about the decoy locations. We introduce a model of hypergames on graphs with temporal logic objectives to capture such adversarial interactions with asymmetric information. Using the hypergame model, we analyze the effectiveness of a given decoy placement, quantified by the set of deceptive winning states where the defender can prevent the attacker from satisfying the attack objective given its incomplete information about decoy locations. Then, we investigate how to place decoys to maximize the defender's deceptive winning region. Considering the large search space for all possible decoy allocation strategies, we incorporate the idea of compositional synthesis from formal methods and show that the objective function in the class of decoy allocation problem is monotone and non-decreasing. We derive the sufficient conditions under which the objective function for the decoy allocation problem is submodular, or supermodular, respectively. We show a sub-optimal allocation can be efficiently computed by iteratively composing the solutions of hypergames with a subset of decoys and the solution of a hypergame given a single decoy. We use a running example to illustrate the proposed method.
△ Less
Submitted 2 October, 2020;
originally announced October 2020.
-
A Theory of Hypergames on Graphs for Synthesizing Dynamic Cyber Defense with Deception
Authors:
Abhishek N. Kulkarni,
Jie Fu
Abstract:
In this chapter, we present an approach using formal methods to synthesize reactive defense strategy in a cyber network, equipped with a set of decoy systems. We first generalize formal graphical security models--attack graphs--to incorporate defender's countermeasures in a game-theoretic model, called an attack-defend game on graph. This game captures the dynamic interactions between the defender…
▽ More
In this chapter, we present an approach using formal methods to synthesize reactive defense strategy in a cyber network, equipped with a set of decoy systems. We first generalize formal graphical security models--attack graphs--to incorporate defender's countermeasures in a game-theoretic model, called an attack-defend game on graph. This game captures the dynamic interactions between the defender and the attacker and their defense/attack objectives in formal logic. Then, we introduce a class of hypergames to model asymmetric information created by decoys in the attacker-defender interactions. Given qualitative security specifications in formal logic, we show that the solution concepts from hypergames and reactive synthesis in formal methods can be extended to synthesize effective dynamic defense strategy using cyber deception. The strategy takes the advantages of the misperception of the attacker to ensure security specification is satisfied, which may not be satisfiable when the information is symmetric.
△ Less
Submitted 7 August, 2020;
originally announced August 2020.
-
Dynamic Hypergames for Synthesis of Deceptive Strategies with Temporal Logic Objectives
Authors:
Lening Li,
Haoxiang Ma,
Abhishek N. Kulkarni,
Jie Fu
Abstract:
In this paper, we study the use of deception for strategic planning in adversarial environments. We model the interaction between the agent (player 1) and the adversary (player 2) as a two-player concurrent game in which the adversary has incomplete information about the agent's task specification in temporal logic. During the online interaction, the adversary can infer the agent's intention from…
▽ More
In this paper, we study the use of deception for strategic planning in adversarial environments. We model the interaction between the agent (player 1) and the adversary (player 2) as a two-player concurrent game in which the adversary has incomplete information about the agent's task specification in temporal logic. During the online interaction, the adversary can infer the agent's intention from observations and adapt its strategy so as to prevent the agent from satisfying the task. To plan against such an adaptive opponent, the agent must leverage its knowledge about the adversary's incomplete information to influence the behavior of the opponent, and thereby being deceptive. To synthesize a deceptive strategy, we introduce a class of hypergame models that capture the interaction between the agent and its adversary given asymmetric, incomplete information. A hypergame is a hierarchy of games, perceived differently by the agent and its adversary. We develop the solution concept of this class of hypergames and show that the subjectively rationalizable strategy for the agent is deceptive and maximizes the probability of satisfying the task in temporal logic. This deceptive strategy is obtained by modeling the opponent evolving perception of the interaction and integrating the opponent model into proactive planning. Following the deceptive strategy, the agent chooses actions to influence the game history as well as to manipulate the adversary's perception so that it takes actions that benefit the goal of the agent. We demonstrate the correctness of our deceptive planning algorithm using robot motion planning examples with temporal logic objectives and design a detection mechanism to notify the agent of potential errors in modeling of the adversary's behavior.
△ Less
Submitted 30 July, 2020;
originally announced July 2020.
-
Deceptive Labeling: Hypergames on Graphs for Stealthy Deception
Authors:
Abhishek N. Kulkarni,
Huan Luo,
Nandi O. Leslie,
Charles A. Kamhoua,
Jie Fu
Abstract:
With the increasing sophistication of attacks on cyber-physical systems, deception has emerged as an effective tool to improve system security and safety by obfuscating the attacker's perception. In this paper, we present a solution to the deceptive game in which a control agent is to satisfy a Boolean objective specified by a co-safe temporal logic formula in the presence of an adversary. The age…
▽ More
With the increasing sophistication of attacks on cyber-physical systems, deception has emerged as an effective tool to improve system security and safety by obfuscating the attacker's perception. In this paper, we present a solution to the deceptive game in which a control agent is to satisfy a Boolean objective specified by a co-safe temporal logic formula in the presence of an adversary. The agent intentionally introduces asymmetric information to create payoff misperception, which manifests as the misperception of the labeling function in the game model. Thus, the adversary is unable to accurately determine which logical formula is satisfied by a given outcome of the game. We introduce a model called hypergame on graph to capture the asymmetrical information with one-sided payoff misperception. Based on this model, we present the solution of such a hypergame and use the solution to synthesize stealthy deceptive strategies. Specifically, deceptive sure winning and deceptive almost-sure winning strategies are developed by reducing the hypergame to a two-player game and one-player stochastic game with reachability objectives. A running example is introduced to demonstrate the game model and the solution concept used for strategy synthesis.
△ Less
Submitted 9 June, 2020; v1 submitted 10 April, 2020;
originally announced April 2020.
-
Synthesis of Deceptive Strategies in Reachability Games with Action Misperception
Authors:
Abhishek N. Kulkarni,
Jie Fu
Abstract:
We consider a class of two-player turn-based zero-sum games on graphs with reachability objectives, known as reachability games, where the objective of Player 1 (P1) is to reach a set of goal states, and that of Player 2 (P2) is to prevent this. In particular, we consider the case where the players have asymmetric information about each other's action capabilities: P2 starts with an incomplete inf…
▽ More
We consider a class of two-player turn-based zero-sum games on graphs with reachability objectives, known as reachability games, where the objective of Player 1 (P1) is to reach a set of goal states, and that of Player 2 (P2) is to prevent this. In particular, we consider the case where the players have asymmetric information about each other's action capabilities: P2 starts with an incomplete information (misperception) about P1's action set, and updates the misperception when P1 uses an action previously unknown to P2. When P1 is made aware of P2's misperception, the key question is whether P1 can control P2's perception so as to deceive P2 into selecting actions to P1's advantage? We show that there might exist a deceptive winning strategy for P1 that ensures P1's objective is achieved with probability one from a state otherwise losing for P1, had the information being symmetric and complete. We present three key results: First, we introduce a dynamic hypergame model to capture the reachability game with evolving misperception of P2. Second, we present a fixed-point algorithm to compute the Deceptive Almost-Sure Winning (DASW) region and DASW strategy. Finally, we show that DASW strategy is at least as powerful as Almost-Sure Winning (ASW) strategy in the game in which P1 does not account for P2's misperception. We illustrate our algorithm using a robot motion planning in an adversarial environment.
△ Less
Submitted 17 February, 2020;
originally announced February 2020.
-
Secure-by-synthesis network with active deception and temporal logic specifications
Authors:
Jie Fu,
Abhishek N. Kulkarni,
Huan Luo,
Nandi O. Leslie,
Charles A. Kamhoua
Abstract:
This paper is concerned with the synthesis of strategies in network systems with active cyber deception. Active deception in a network employs decoy systems and other defenses to conduct defensive planning against the intrusion of malicious attackers who have been confirmed by sensing systems. In this setting, the defender's objective is to ensure the satisfaction of security properties specified…
▽ More
This paper is concerned with the synthesis of strategies in network systems with active cyber deception. Active deception in a network employs decoy systems and other defenses to conduct defensive planning against the intrusion of malicious attackers who have been confirmed by sensing systems. In this setting, the defender's objective is to ensure the satisfaction of security properties specified in temporal logic formulas. We formulate the problem of deceptive planning with decoy systems and other defenses as a two-player games with asymmetrical information and Boolean payoffs in temporal logic. We use level-2 hypergame with temporal logic objectives to capture the incomplete/incorrect knowledge of the attacker about the network system as a payoff misperception. The true payoff function is private information of the defender. Then, we extend the solution concepts of $omega$-regular games to analyze the attacker's rational strategy given her incomplete information. By generalizing the solution of level-2 hypergame in the normal form to extensive form, we extend the solutions of games with safe temporal logic objectives to decide whether the defender can ensure security properties to be satisfied with probability one, given any possible strategy that is perceived to be rational by the attacker. Further, we use the solution of games with co-safe (reachability) temporal logic objectives to determine whether the defender can engage the attacker, by directing the attacker to a high-fidelity honeypot. The effectiveness of the proposed synthesis methods is illustrated with synthetic network systems with honeypots.
△ Less
Submitted 17 February, 2020;
originally announced February 2020.
-
Opportunistic Synthesis in Reactive Games under Information Asymmetry
Authors:
Abhishek N. Kulkarni,
Jie Fu
Abstract:
Reactive synthesis is a class of methods to construct a provably-correct control system, referred to as a robot, with respect to a temporal logic specification in the presence of a dynamic and uncontrollable environment. This is achieved by modeling the interaction between the robot and its environment as a two-player zero-sum game. However, existing reactive synthesis methods assume both players…
▽ More
Reactive synthesis is a class of methods to construct a provably-correct control system, referred to as a robot, with respect to a temporal logic specification in the presence of a dynamic and uncontrollable environment. This is achieved by modeling the interaction between the robot and its environment as a two-player zero-sum game. However, existing reactive synthesis methods assume both players to have complete information, which is not the case in many strategic interactions. In this paper, we use a variant of hypergames to model the interaction between the robot and its environment; which has incomplete information about the specification of the robot. This model allows us to identify a subset of game states from where the robot can leverage the asymmetrical information to achieve a better outcome, which is not possible if both players have symmetrical and complete information. We then introduce a novel method of opportunistic synthesis by defining a Markov Decision Process (MDP) using the hypergame under temporal logic specifications. When the environment plays some stochastic strategy in its perceived sure-winning and sure-losing regions of the game, we show that by following the opportunistic strategy, the robot is ensured to only improve the outcome of the game - measured by satisfaction of sub-specifications - whenever an opportunity becomes available. We demonstrate the correctness and optimality of this method using a robot motion planning example in the presence of an adversary.
△ Less
Submitted 13 June, 2019;
originally announced June 2019.
