Generalized Multi-Objective Reinforcement Learning with Envelope Updates in URLLC-enabled Vehicular Networks

Following [20, 16], we update all AVs’ real-time physical location based on the Kinematics model. Suppose only front wheels can be steered, we have the following relations applied for each AV $j$, $j\in M$:

where $(x_{j},y_{j})$ is the location of AV $j$. $\psi_{j},a_{j},\beta_{j},l_{j},\delta_{j}^{\mathrm{fa}}$ are AV $j$’s heading, command for acceleration, slip angle at the gravity center, half-length of AV $j$, and front wheel angle, respectively.

As in [16, 21], each target AV $j$ follow a lane $L_{j}$ through its lateral position $y_{j}$ and heading $\psi_{L_{j}}$, for $i\in M_{1}$. Thus, target AVs follow a cascade controller of lateral position and heading, i.e.,

where $\tilde{v}_{i,y}=K_{j}^{y}\left(y_{L_{j}}-y_{j}\right),$ $K_{j}^{\psi}$ and $K_{j}^{y}$ are the control gains. $y_{L_{j}}$ is the lane $L_{j}$ lateral position. The acceleration of target AV $j$, for $j\in M_{1}$, is given as[16, 21]:

where $v_{r}$ is the desired AV speed and $K_{0}^{v}$ is the control gain.

Note that surrounding AVs will not get involved in training. Instead, the surrounding AVs select their accelerations/decelerations based on the IDM and MOBIL model, which allows them to track for a specific target speed and follow in a specific lane. For each surrounding vehicle, we compute the acceleration using the Intelligent Driver Model (IDM) [16, 22] and applied the MOBIL model for changing lanes [23, 16]. IDM is based on the idea that surrounding AVs choose their acceleration or deceleration based on their current speed and the distance to the vehicle in front. The model combines a desire to drive at a certain free-flow speed with a reaction to the traffic situation, particularly to avoid collisions. We define that AV $j$ own velocity $v_{j}$, a real gap distance $d_{j}$ to the leading AV, and the speed difference $\Delta v_{j}$ between two AVs. With IDM, the acceleration $a_{j}$ and desired minimum gap $\hat{d_{j}}$ from the vehicle ahead of AV $j$ , $j\in M_{2}$ is modeled as [16, 22]:

where

where $a_{c},b_{c},v_{0}$ are predefined parameters denoting AV’s maximum acceleration, comfortable braking deceleration, and desired velocity, respectively. $\delta_{a}$ denotes acceleration reduction factor, i.e., a higher $\delta_{a}$ reduces acceleration. $d_{0}$ and $T$ denote the minimum distance in stopped traffic and safe time to approach the front vehicle, respectively. The desired gap has a steady-state (equilibrium) term and a dynamic term that implements the intelligent braking strategy.

To maintain a high transportation efficiency, redundant brakes during lane changing for surrounding AVs are discouraged. MOBIL model focuses on minimizing overall braking induced by a lane change in AV lateral behavior. MOBIL model indicates that an AV should change lanes when one of the following conditions is met, (1) AVs can accelerate more after changing lane $\tilde{a}_{j}\geq-b_{\textbf{safe}},$ where $b_{\textbf{safe}}$ shows the maximum braking imposed on AV when the AV cuts into the adjacent lane, (2) vehicle imposes unsafe and incentives braking on their new following vehicle, i.e,

where $a_{j}$ and $\tilde{a}_{j}$ are the acceleration of AV before and after lane change. The subscript $o$ and $n$ denote the AV’s older follower and the new follower before and after the lane change, respectively. $p$ represents the politeness coefficient. $\Delta a_{\textbf{th}}$ is the acceleration gain to execute the lane change for AV. A decision of lane change adjusts the target lane $L_{j}$ of AV $j$ on its current lane segment. The actual motion planning and steering to track the newly targeted lane are then executed by the lateral controller in (11).

In the distribution of tasks pool, the interaction between tasks and agents can be defined as MDP $\mathcal{M}$, represented by a tuple, $\mathcal{M}=<\mathcal{S},\mathcal{A},r,\mathcal{P},\mathcal{\gamma}>$, where $\mathcal{S},\mathcal{A},r,$ are the set of states $s$, actions $a$, and reward $r(s,a)$, respectively. $\gamma\in[0,1)$ is the discount factor. $r(s,a)$ represents the stochastic instantaneous reward value that the agent can receive given a specific action $a\in\mathcal{A}$ has been taken in a given specific state $s\in\mathcal{S}$. $\mathcal{P}(s_{t+1}|s_{t},a_{t})$ indicates the transition probability for the agent to take an action $a_{t}\in\mathcal{A}$ on state $s_{t}\in\mathcal{S}$ to the next state $s_{t+1}\in\mathcal{S}$ in time step $t$. In RL, the agent interacts with the environment by following a trajectory $\tau=\{(s_{t},a_{t})\}_{t=0}^{\infty}$ from time stamp $t=0$ to the end of the interacting episode, and receives a total discounted reward as ${r}_{\tau}=\sum_{t=0}^{\infty}\mathcal{\gamma}\cdot r_{t}(s_{t},a_{t})$. The goal in RL is to find the map** policy $\pi$ between states and actions that maximizes the discounted total reward. In each time step $t$, the agent selects an action based on its current state to maximize the discounted total reward ${r}_{\tau}$. Finally, we can obtain a policy $\pi$ that belongs to the policy set $\Pi$. The state-action value function $Q_{\pi}(s,a)$ of the policy $\pi$ on state $s$ can be given as:

Thus, the optimal policy is given as $\pi^{*}=\max_{\pi\in\Pi}Q_{\pi}(s,a).$

The goal of MORL is to obtain policies among $M$ conflicting or competing objectives, where the relative importance (preferences) of each objective may be known or unknown to the agent. Similar to RL, MORL can be formulated by MOMDP which extends the MDP by defining a new reward space, preference space, and preference function, i.e., $\mathcal{M}=<\mathcal{S},\mathcal{A},\mathbf{r},\mathcal{P},\Omega,\iota_{0}>$, where $\mathbf{r}\in\mathbb{R}^{H}$ is a vector of reward functions corresponding to $J$ objectives, e.g., $\mathbf{r}=[r^{1},r^{2},\dots,r^{H}]$, $\Omega$ is the preference space where $\boldsymbol{\omega}\in\Omega$ is the preference vector corresponding to $H$ objectives, and $\sum_{h=1}^{H}{{\omega}}^{h}=1$. $\iota_{0}$ is the probability distribution over initial states. In MOMDPs, a policy $\pi:\mathcal{S}\to\mathcal{A}$ defines a map** from states to actions with the goal of maximizing a vector of expected rewards. Given the distribution $\iota_{0}$ and a policy $\pi$, the expected discounted return is defined as:

where $\mathbf{r}(s_{t},a_{t})$ is the immediate vector valued reward at time step $t$ for $H$ objectives. Maximizing the expected reward involves solving the following MOO problem $\max\mathbf{Q}_{\pi}=\max_{\pi}[Q_{\pi}^{1},Q_{\pi}^{2},\dots,Q_{\pi}^{H}]$.

A policy $\pi$ strictly dominates another policy $\pi^{\prime}$ if $\pi$ achieves values at least as high as $\pi^{\prime}$ in all objectives and strictly higher in at least one objective:

Furthermore, a policy $\pi$ weakly dominates another policy $\pi^{\prime}$, if $\pi$ achieves values greater than or equal to $\pi^{\prime}$ in all objectives, i.e., $\pi\geq\pi^{\prime}$, when ${V}^{h}_{\pi}\geq{V}^{h}_{\pi^{\prime}},\quad\forall h$. A policy $\pi$ is considered Pareto-optimal (or non-dominated) if it is not strictly dominated by any other policies.

Considering all returns from MOMDP, we have Pareto frontier set $\mathcal{F}^{*}:=\left\{\mathbf{\hat{r}}\mid\nexists\mathbf{\hat{r}}^{\prime}\geq\mathbf{\hat{r}}\right\}$ [24], where $\mathbf{\hat{r}}=\sum_{t=0}^{\infty}\gamma\cdot\mathbf{r}(s_{t},a_{t})$. For all possible preferences in $\Omega$, we define a convex coverage set (CCS) of the Pareto frontier which contains all returns that provide the maximum cumulative reward, i.e.,

where $(\cdot)^{T}$ denotes the transpose operator.

The state space consists of kinematics-related features, which is a $M_{1}\times F$ array that describes $F\to\{x_{j},y_{j},v_{j},\psi_{j},n_{R}^{j},n_{T}^{j}\}$ specific features of AVs. We consider $M_{1}$ target AVs and $M_{2}$ surrounding AVs. Each target AV is characterized by its (1) coordinates $(x_{j},y_{j})$, (2) forward velocity $v_{j}$, (3) heading $\psi_{j}$, and (4) $n_{R}^{j}$ and $n_{T}^{j}$ which are the number of RBSs and TBSs that makes AV achieves the desired data rate in a predefined radius from its current position, respectively. Accordingly, the aggregated state space $\mathcal{S}$ at any time step $t$ is given by:

The action space consists of self-driving action space $\mathcal{A}_{\mathrm{tran}}$ and telecommunication action space $\mathcal{A}_{\mathrm{tele}}$, which include 5 and 3 discrete actions, respectively. For each time step $t$, the AV must select driving-related action and telecommunication-related action from action space, as shown below:

Note that $\mathcal{A}_{\rm tran}=\{a_{\rm tran}^{1},\ldots,a_{\rm tran}^{5}\}$, where $a_{\rm tran}^{1},a_{\rm tran}^{2}$ and $a_{\rm tran}^{3}$ indicate that AV changes its lane to the left, maintains the same lane, and changes its lane to the right, respectively. $a_{\rm tran}^{4}$ and $a_{\rm tran}^{5}$ indicate the acceleration and deceleration of AV within the same lane. It is important to note that the acceleration and deceleration rates are dynamically determined by the model in Section II-B. With that being said, each AV selects the same actions does not imply that they will perform identical accelerations/deceleration. The communication action space is represented as $\mathcal{A}_{\rm tele}=\{a_{\rm tele}^{1},a_{\rm tele}^{2},a_{\rm tele}^{3}\}$. $a_{\rm tele}^{1}$ indicates scenarios where AV selects a BS by maximizing weighted data rate metric (defined by equation (6) in Section II-A), which encourages traffic load balancing between BSs and discourages unnecessary HOs, especially for TBSs. In $a_{\rm tele}^{2},$ the AV selects a BS with maximum $\text{WR}_{{ij}}$ by substituting $\mu=0$, if $Q_{i}\geq n_{i}$. Otherwise, AV recursively selects the next vacant best-performing BS in terms of $\text{WR}_{ij}$. In $a_{\rm tele}^{3}$, the AV chooses to connect to a BS with the maximum data rate $R_{ij}$.

The design of the associated reward is directly related to optimizing the driving policy and network selection, and is critical for accelerating the convergence of the model. Generally, the AV is given a positive reward when it receives a higher HO-aware data rate while guaranteeing safe driving. By taking any other actions that may lead to an increase in HOs, collisions, or traffic violations, the AV receives a penalty. We define the transportation reward as [16]:

where $v_{t}^{j},v_{\min}$ and $v_{\max}$ are the current longitudinal velocity for AV $j$ on time $t$, the minimum and maximum speed limits, and $\delta_{2},\delta_{3},\delta_{4}$ is the collision indicator, AV right lane indicator, on road indicator, respectively. $c_{1}$ and $c_{2}$ are the weights that adjust the value of the AV transportation reward with its collision penalty. $c_{1}$ indicates that the reward received when driving at full speed, linearly mapped to zero for lower speeds. $c_{3}$ shows that AV was rewarded for driving on the right-most lanes, and linearly mapped to zero for other lanes. $c_{4}$ is the on-road reward factor, which penalize the AV for driving off highway. It is important to note that negative rewards are not allowed since they might encourage the agent to prioritize ending an episode early by causing a collision instead of taking the risk of receiving a negative return if no satisfactory trajectory is available.

For the telecommunication side, we define the reward for AV $j$ associated with BS $i^{*}$ at time step $t$ as:

where $\text{WR}_{i^{*},j,t}$ is the achievable data rate compute by (6) and $\xi^{j}_{t}$ is the HO probability of AV $j$ computed by dividing the number of HOs accounted until the current time $t$ by the time duration of previous time slots in the episode³³3Note that $c_{1}\dots c_{5}$ are the weights to set the priority of each term. For instance, $c_{2}$ needs to be sufficiently large compared to other coefficients for collision avoidance. The highest penalty applies to vehicle collision..

Based on the instantaneous reward, we compute the accumulated rewards, which is the summation of discounted reward among all target AVs on the highway in each training episode. The expected return is defined as follows:

Our MOMDP optimal strategy for maximizing the expected reward involves the simultaneous maximization of both transportation and telecommunications objectives, i.e., $\max_{\pi}{\mathbf{Q}_{\pi}(s,a,\boldsymbol{\omega})}$.