Edge Computing Enabled Real-Time Video Analysis via Adaptive Spatial-Temporal Semantic Filtering

Abstract

This paper proposes a novel edge computing enabled real-time video analysis system for intelligent visual devices. The proposed system consists of a tracking-assisted object detection module (TAODM) and a region of interesting module (ROIM). TAODM adaptively determines the offloading decision to process each video frame locally with a tracking algorithm or to offload it to the edge server inferred by an object detection model. ROIM determines each offloading frame’s resolution and detection model configuration to ensure that the analysis results can return in time. TAODM and ROIM interact jointly to filter the repetitive spatial-temporal semantic information to maximize the processing rate while ensuring high video analysis accuracy. Unlike most existing works, this paper investigates the real-time video analysis systems where the intelligent visual device connects to the edge server through a wireless network with fluctuating network conditions. We decompose the real-time video analysis problem into the offloading decision and configurations selection sub-problems. To solve these two sub-problems, we introduce a double deep Q network (DDQN) based offloading approach and a contextual multi-armed bandit (CMAB) based adaptive configurations selection approach, respectively. A DDQN-CMAB reinforcement learning (DCRL) training framework is further developed to integrate these two approaches to improve the overall video analyzing performance. Extensive simulations are conducted to evaluate the performance of the proposed solution, and demonstrate its superiority over counterparts.

I Introduction

With the fast burgeoning of the Internet of Things (IoT), intelligent visual devices (e.g., smartphones, auto-driving vehicle, unmanned aerial vehicle) have encouraged the emergence of far-reaching innovative mobile applications, ranging from autonomous systems to extended reality[1]. These intelligent visual devices capture surroundings and generate skyrocketing amounts of video data that need to be processed in real-time. For example, self-driving applications leverage intelligent camera to accurately detect the lanes, cars, and pedestrians to avoid collisions. However, processing such massive computation-intensive and delay-sensitive video analysis tasks locally is significantly challenging due to the limited resources of intelligent visual devices[2]. Although an alternative way is to offload tasks to powerful cloud servers, it will bring unacceptable delays because they are typically deployed far away from the devices.

Fortunately, the paradigm of edge computing[3], which deploys powerful edge servers at the edge of networks, provides pervasive, reliable and fast-responsive computing services for the devices, and thereby offloading video analysis tasks to edge servers is a promising way. However, due to the limited uplink bandwidth and volatility of the celluar network, it is difficult to offload all video frames with massive sizes to edge servers in real time[4]. Besides, vast repetitive spatial-temporal semantic information among video frames, which generates large transmission delays, severely impacts real-time analyzing performance. Moreover, due to the huge amount of video traffic, users have to pay huge expense on data traffic. Above factors motivate us to design an adaptive spatial-temporal semantic filtering based video analysis offloading strategy with low bandwidth cost and high detection accuracy under the volatile network conditions.

Nevertheless, designing such a strategy is significantly challenging due to the following reasons: i) in particular, real-time video will generate massive traffic per unit of time, resulting in the visual device being unable to transmit all the images to the edge server (ES) timely, and ii) constantly growing intelligent visual devices with frequent video analysis task requests would drain the resources of edge servers fast. In addition, there are considerable differences in computing power among various intelligent visual devices, leading to the performance of the sames algorithm varying wildly.

Aiming to solve the above problems, we study a long-term semantic-filtering offloading optimization problem to maximize the processing rate, i.e., the number of frames of detection results returned in time, and the detection accuracy under the time-varying network conditions. It is observed that this problem involves two coupled intricate sub-problems, an offloading decision problem and a configurations selection problem. On this basis, we propose a double deep Q network (DDQN) based offloading decision scheme and a contextual multi-armed bandit (CMAB) based adaptive configurations selection scheme for these two sub-problems, respectively. To obtain the optimal solution, we further propose a two-layer training framework, namely DDQN-CMAB reinforcement learning (DCRL) framework, for jointly solving these two sub-problems effectively. The main contributions are summarized as follows.

• A novel edge computing enabled real-time video analysis system based on spatial-temporal semantic filtering is proposed, which realizes real-time object detection with high accuracy under volatile and fluctuating network conditions.

• A DCRL framework is designed for solving the overall semantic-filtering offloading optimization problem under unpredictable network conditions.

• Simulations are conducted to examine the performance of the proposed DCRL framework on the multi-camera pedestrian video dataset[5].

The rest of this paper is organized as follows: Section II presents the system model and problem formulation. In Section III, two specific approaches are proposed to solve the decomposed sub-problems, and a DCRL framework is proposed to integrate these two approaches. Simulation results are given in Section IV, followed by conclusions in Section V.

II System Model and Problem Formulation

II-A System Model

Figure 1: Overview of proposed real-time video analysis system.

Consider an collaborative edge-device real-time video analysis system, as illustrated in Fig. 1, which consists of a tracking-assisted object detection module (TAODM) and a region of interesting module (ROIM) deployed on a mobile intelligent visual device to process road condition video stream and a dedicated ES with various advanced object detection models deployed at the base station. Besides, a frame-based time-slotted operation framework is studied to depict time-varying uncertainties of the video analysis system, where $t\in\{1,2,\dots,T\}$ denotes the index of time slots and each time slot represents a decision interval. In the proposed edge computing enabled real-time video analysis system, an intelligent visual device with computing power $f^{device}$ connects to an ES with computing power $f^{edge}$ through the cellular network with bandwidth $b_{t}$. The intelligent visual device captures the real-time video stream and generates a real-time frame in each time slot by sampling. The real-time frame is denoted as $f_{t}$, which is assumed to be generated at beginning of time slot $t$ and will be processed by TAODM. TAODM adaptively determines the offloading mode to process the frame locally with a tracking algorithm or to offload it to the ES. If the latter, the offloaded frame would be further processed by ROIM, which determines its resolution and detection model configuration according to the information density and bandwidth for ensuring the analysis results return in time. Since the tracking algorithm requires the detection results from ES to initialize tracking targets, TAODM and ROIM are highly coupled. TAODM and ROIM interact jointly to maximize the processing rate while ensuring high video analysis accuracy.

Specifically, three decisions would be made by TAODM, namely, i) offloading decision $\alpha_{t}\in\{0,1\}$, which indicates the frame $f_{t}$ to be processed locally or to be offloaded to the ES ii) ROI decision $\beta_{t}\in\{0,1\}$ indicates whether to adopt ROI extraction or not and iii) the tracking mode $k\in\mathcal{K}$ to be adopted, where $\mathcal{K}=\{\emph{Skip, KCF, CSRT}\}$, and the Skip mode would directly reuse the detection result of the last frame if the current frame has tiny new semantic information compared to the last one, and KCF[6] and CSRT[7] are well-known trackers working well in resource-limited IoT devices. When $\alpha_{t}=0$, the frame will be processed by TAODM with tracking mode $k\in\mathcal{K}$, and we use $acc(k)$ to represent the processing accuracy of tracking mode $k$. Additionally, the processing time of tracking mode $k$ can be calculated as $l_{t}^{pro,loc}=u^{k}d_{t}/f^{device}$, where $u^{k}$ indicates the computing intensity of tracking mode $k$ and $d_{t}$ represents the data size of frame $f_{t}$. Note that, $l^{pro,loc}=0$ when $k=skip$. When $\alpha_{t}=1$, the ROI extraction decision $\beta_{t}\in\{0,1\}$ will be made according to the during time slot $t$ bandwidth and then the frame $f_{t}$ is further processed by ROIM.

In ROIM, the frame $f_{t}$ would firstly go through the process of rough pre-detection by a local lightweight deep neural network (DNN) for obtaining the bounding boxes and the amount of them, which indicate the information density $n_{t}$, with inference time $l_{t}^{dnn,loc}$. Then, if $\beta_{t}=1$, all bounding boxes will be merged into several ROI blocks with extraction time $l_{t}^{roi,loc}={u^{roi}d_{t}}/{f^{device}}$, where $u^{roi}$ represents the computing intensity of ROI . After that, only those ROI blocks are transmitted to the edge server for further processing. When $\beta_{t}=0$, the whole frame will be transmitted instead. Considering the inference requests of a frame can be served by different detection models, we use the set $\mathcal{M}=\{Yolov5x,Yolov5l,Yolov5m\}$ to denote the detection models held on the ES. YOLOv5 is the most popular object detection algorithm nowadays and provides various size models to meet different scenarios. Besides, we use a set $\mathcal{R}=\{640p,480p,320p\}$ to denote the set of offloading resolutions to accelerate the model inference and reduce the transmission delay. Thus, ROIM would make two decisions according to the information density $n_{t,i}$ and bandwidth $b_{t}$ before transmitting the frame to the edge servers for real-time performance, i.e., i) the model $m\in\mathcal{M}$ selected for object detection in the edge server and ii) offloading resolution $r\in\mathcal{R}$ for each frame or block. Then, based on the decision of offloading resolution $r\in\mathcal{R}$, frame or all blocks are resized to the offloading resolution and the size of offloading data can be obtained as $d_{t}=\sum_{i=1}^{s_{t}}{\tau b(r_{t,i})}$, where $s_{t}$ represents the number of blocks, $\tau$ is the data size of one pixel and $b(r_{t,i})$ is the pixel number of block $i$ in resolution $r$, meanwhile, transmitting a whole frame is treated as transmitting a block for simplification. Finally, the frame or all ROI blocks with the corresponding model configurations has the transmission delay $l_{t}^{trans}={d_{t}}/{b_{t}}$, and detected by the advanced detection model.

Based on above formulation, the completion time of frame $f_{t}$ can be expressed as

$$l^{total}_{t}= \alpha_{t}(\beta_{t}l_{t}^{roi,loc}+l_{t}^{dnn,loc}+l_{t}^{trans}+l_{t}^{pro,edge})$$

$$+(1-\alpha_{t})l_{t}^{pro,loc},$$ (1)

where $l_{t}^{pro,edge}={u^{m,r}d_{t}}/{f^{edge}}$ is the inference delay in the ES and $u^{m,r}$ represents the computing intensity of model $m$ in resolution $r$. The size of inference results is commonly negligible compared to that of the offloading frame, so we ignore the delay of inference results transmitting back to the intelligent visual device from the ES. Since the video frames need to be analyzed in real-time, each frame should be processed within a time duration $l^{max}$, which can be formulated as

$$l^{total}_{t}\leq l^{max},$$ (2)

where $l^{max}$ is the length of a time slot. Besides, the detection accuracy of video analysis system $acc_{t}$ can be mathematically expressed as

$$acc_{t}= \frac{\alpha_{t}}{s_{t}}\sum\nolimits_{i=1}^{s_{t}}\sum\nolimits_{m\in\mathcal{M}}\sum\nolimits_{r\in\mathcal{R}}y_{t,i}^{m}z^{r}_{t,i}acc(m,r)$$

$$+(1-\alpha_{t})\sum\nolimits_{k\in\mathcal{K}}{x_{t}^{k}acc(k)},$$ (3)

where $s_{t}$ represents the number of blocks, $x_{t}^{k}\in\{0,1\}$ indicates whether the tracking mode $k$ is selected ($x_{t}^{k}$ = 1 signifies the mode $k$ is selected, otherwise $x_{t}^{k}$ = 0). Similarly, $y_{t,i}^{m}\in\{0,1\}$ and $z_{t,i}^{r}\in\{0,1\}$ indicate whether the detection model configuration $m$ and offloading resolution $r$ are selected.

Note that, different tracking modes have different processing time. Skip mode has almost no latency, but detection accuracy degrades quickly when semantic information changes. The KCF mode is faster with lower accuracy than the CSRT mode. In addition, different configurations of detection models and offloading resolutions have different precision and inference delays. Adopting the advanced object detection model and high offloading resolution will have a better accuracy but lead to a longer inference delay and transmission delay, which damages the real-time performance. Meanwhile, selecting the entire frame can improve accuracy for that ROI extraction may miss some key information but also incurs a higher transmission delay, decreasing the real-time video analysis performance.

II-B Problem Formulation

Since the quality of the proposed video analysis system highly depends on the frame processing rate (i.e., the frame number of detection results returned in time) and detection accuracy, in this work, we aim to maximize the system utility, which is a weighted sum of these two factors. Thus, the optimization problem of video analysis can be mathematically expressed as

$$\max_{\{\alpha_{t},\beta_{t},x_{t}^{k},y_{t,i}^{m},z_{t,i}^{r}\}}~{}\frac{1}{T}\sum\nolimits_{t=1}^{T}q_{t}+\eta\frac{\sum\nolimits_{t=1}^{T}(q_{t}acc_{t})}{\sum\nolimits_{t=1}^{T}q_{t}},$$ (4)

$$\quad\text{s.t.}~{}\alpha_{t},\beta_{t}\in\{0,1\},$$ (5)

$$\qquad~{}(l^{total}_{t}-l^{max})q_{t}\leq 0,~{}q_{t}\in\{0,1\},$$ (6)

$$\qquad~{}\sum\nolimits_{k\in\mathcal{K}}x^{k}_{t}\leq 1,x^{k}_{t}\in\{0,1\},~{}k\in\mathcal{K},$$ (7)

$$\qquad~{}\sum\nolimits_{m\in\mathcal{M}}y^{m}_{t,i}\leq 1,y^{m}_{t,i}\in\{0,1\},~{}m\in\mathcal{M},$$ (8)

$$\qquad~{}\sum\nolimits_{r\in\mathcal{R}}z^{r}_{t,i}\leq 1,z^{r}_{t,i}\in\{0,1\},~{}r\in\mathcal{R},$$ (9)

$$\qquad~{}\sum\nolimits_{k\in\mathcal{K}}x^{k}_{t}=\alpha_{t},~{}k\in\mathcal{K},$$ (10)

$$\qquad~{}\sum\nolimits_{k\in\mathcal{K}}x^{k}_{t}+\sum\nolimits_{m\in\mathcal{M}}y^{m}_{t,i}=1,~{}k\in\mathcal{K},m\in\mathcal{M},$$ (11)

$$\qquad~{}\sum\nolimits_{k\in\mathcal{K}}x^{k}_{t}+\sum\nolimits_{r\in\mathcal{R}}z^{r}_{t,i}=1,~{}k\in\mathcal{K},r\in\mathcal{R},$$ (12)

where parameter $\eta>$ 0 is used to depict the weight of detection accuracy, and $q_{t}\in\{0,1\}$ denotes whether frame $f_{t}$ is processed successfully in the time slot $t$. Specifically, if $l_{t}^{total}>l^{max}$, it means that frame $f_{t}$ is not processed completely within $l^{max}$ and will be abandoned, , and $q_{t}=0$ must be established to make constraint (6) true. If $l_{t}^{total}\leq l^{max}$, $q_{t}=0$ means that the detection accuracy of frame $f_{t}$ is lower than a threshold thus is not successfully processed, while $q_{t}=1$ means that frame $f_{t}$ is really successfully processed. Constraint (10) means a tracking mode is adopted when $\alpha_{t}=1$. Otherwise, the configurations of models and resolutions will be chosen for offloading, represented by Constraint (11) and Constraint (12).

Observe that the formulated optimization problem is a mixed-integer nonlinear programming problem, which is NP-hard problem [8]. To address this problem, we partition the problem into two sub-problems, i.e., deciding the optimal offloading strategy for the frame, and selecting object detection model and offloading resolution configuration for each block. In the next section, a real-time DDQN-CMAB reinforcement learning framework is adopted to connect a DDQN-based offloading approach and a CMAB-based configuration selection approach to generate the optimal strategy for real-time video analysis.

III DDQN-CMAB Reinforcement Learning Strategy

Output: The network parameters $\theta^{*}$

1 Initialize: Randomly initialize the parameters $\theta$; initialize the target Q-network parameter $\theta^{\prime}=\theta$; reset the replay buffer $B^{d}$ of experience playback; maximum training steps $LOOP$ and $loop=0$.

2 for $loop$ $\leq$ $LOOP$ do

3       Obtain initial observation state $s_{0}$ and replay buffer $B^{d}$;

4       for $t=1$ to $T$ do

5             Observe the state $s_{t}$;

6             if random $\geq$ $\epsilon^{ddqn}$ then

7                   $a_{t}\leftarrow\operatorname*{arg\,max}_{a\in A_{t}}{Q(s_{t},a_{t};\theta_{t})}$;

8            else

9                   Randomly select $a_{t}$ from $A_{t}$;

10            Execute $a_{t}$ and receive reward $R$;

11             Store tuple $(s_{t},a_{t},R)$ into $B^{d}$;

12             Sample random batch of tuples from $B^{d}$;

13             Compute the value of Q-traget;

14             Update current Q-network parameters $\theta_{t}$ according to loss function;

15             if $t\text{ mod }C=0$ then

16                   Update target Q-network parameters $\theta^{\prime}_{t}\leftarrow\theta_{t}$;

17                  

18            $s_{t}\leftarrow s_{t+1}$, $t\leftarrow t+1$;

19            

20      $loop\leftarrow loop+1$

Algorithm 1 DDQN-based Offloading Algorithm

III-A DDQN-based Offloading Decision

In this subsection, we reformulate the first sub-problem, the offloading decision problem, into a Markov decision process (MDP), then propose a DDQN-based offloading scheme to dynamically generate the appropriate strategy to determine $\alpha_{t}$, $\beta_{t}$ and tracking mode $k$.

For each time slot $t$, the offloading strategy derived by TAODM only depends on the states consisting of hash similarity, bandwidth, tracking complexity and continuous tracking time of the last time slot $t-1$, which means that the state transition satisfies the Markov property[9][10]. Thus, the offloading decision problem can be formulated as an MDP and defined by a tuple $\{S_{t},A_{t},R_{t}\}$, which represents the state space, action space and reward space, respectively.

1) State: For TAODM, its state $s_{t}\in S_{t}$ in time slot $t$ can be represented as $s_{t}=\{h_{t},b_{t},c_{t},p_{t}\}$, where $h_{t}$ is the hash similarity between $f_{t}$ and the last processed frame $f_{t-1}$, $b_{t}$ indicates the bandwidth at time slot $t$, $c_{t}$ stands for tracking complexity that is the number of objects detected by the advanced DNN model in the last offloading frame and $p_{t}$ is the number of continuous time slots of processing frames with the same tracking mode.

2) Action: At time slot $t$, the action of TAODM is frame processing mode selection. The action space is formulated as the set of five possible decisions of offloading choices $A_{t}=\{\emph{Skip, KCF, CSRT, Offload-Full, Offload-ROI}\}$, where Offload-full indicates offloading the full frame and Offload-ROI indicates offloading ROI blocks. In pursuit of solution efficiency, we merge the ROI decision into TAODM.

3) Reward: The reward $R$ for TAODM is defined as the weighted sum of actual accuracy $acc_{t}$ and completion time $l_{t}^{total}$, which can be expressed as $R=acc_{t}+\lambda\max((l^{max}-l_{t}^{total})/l^{max},0)$, where $acc_{t}$ is the actual accuracy of frame $f_{t}$ given by environment, which is used to replace the sum of $acc(m,r)$ and $acc(k)$, and $\lambda>0$ is the parameter that controls the weight to balance completion time and detection accuracy.

For TAODM’s MDP, the deep Q network (DQN) could be applied to train the agent for discrete action space, which leverages the deep neural network to analyze states and actions to optimize the $Q$ value and has a low computation time. In the DQN training process, TAODM first observes the state $s_{t}\in\mathcal{S}_{t}$ from the environment and selects the best action $a^{*}_{t}\in\mathcal{A}_{t}$ for maximizing the action-value function, and can be expressed as $a^{*}_{t}=\operatorname*{arg\,max}_{a_{t}}{\mathbb{E}[r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1};\theta_{t})]}$, where $\theta_{t}$ is the network parameters matrix and is updated by a back propagation training process, whose loss function is $L(\theta_{t})=\mathbb{E}[(y_{t}-Q(s_{t},a_{t};\theta_{t}))^{2}]$, where $y_{t}$ is the target value for time slot $t$. However, the DQN-based algorithm may cause a large deviation in its model due to overestimating the value of Q-target, which indicates the quality of a strategy. To avoid such overestimation, we present a DDQN-based algorithm, which decouples the action selection using DQN and evaluation of Q-target using target network. In the DDQN-based algorithm, the value of Q-target is calculated by $y_{t}=r_{t}+\gamma Q\left(s_{t+1},\operatorname*{arg\,max}_{a_{t+1}}Q(s_{t+1},a_{t+1};\theta_{t});\theta^{\prime}_{t}\right)$, where $\theta_{t}^{\prime}$ is the target network parameters matrix and is updated periodically.

An overview of DDQN-based offloading approach is given in Algorithm 1. In each training step $loop$, the specific process is to first initialize hyperparameters, network parameters and replay buffer. Second, TAODM observes the state $s_{t}\in\mathcal{S}_{t}$ from the environment and select the action using $\epsilon^{DDQN}$-greedy method. Then, TAODM executes the action $a_{t}$ and calculates the reward $R$. Third, a replay buffer $B^{d}$ is adopted to store the tuple ($s_{t}$, $a_{t}$, $R$) for learning Q-network better. Finally, TAODM calculate the Q-value and loss to update network parameters matrix $\theta_{t}$, but update target network parameters matrix $\theta_{t}^{\prime}$ every $C$ times.

III-B CMAB-based Adaptive Configurations Selection

Input: Pre-trained MAB models

Output: DDQN patameters $\theta$, MAB models

1 Initialize: maximum training steps $LOOP^{\prime}$ and $loop^{\prime}=0$

2 for $loop^{\prime}\leq LOOP^{\prime}$ do

3       Offloading $f_{0}$ initialize last detection results;

4       for $t=1$ to $T$ do

5             Get new frame $f_{t}$, bandwidth $b_{t}\sim N(\rho,\sigma)$, tracking complexity $c_{t}$, continuous tracking times $p_{t}$;

6             Calculate hash similarity $h_{t}$;

7             $a_{t}\leftarrow$ TAODM($h_{t},b_{t},c_{t},p_{t}$);

8             if $a_{t}=Skip$  then

9                   Get the last detection results as results of $f_{t}$;

10            else if $a_{t}=KCF$ or $a_{t}=CSRT$ then

11                   TAODM excute tracking algorithm corresponding to $a_{t}$;

12                   Get tracking detection results as results of $f_{t}$;

13                  

14            else

15                   Detected by lightweight DNN model;

16                   if $a_{t}=$ Offload-ROI then

17                         extract ROI blocks;

18                        

19                  for each block $i$ do

20                         Get information intensity $n_{t,i}$;

21                         $g_{t,i}\leftarrow$ ROIM($n_{t,i},b_{t}$);

22                         Offload block $i$ with $g_{t,i}$ to ES;

23                        

24                  Merge detection results of blocks;

25                   Calculate MAB reward $R^{e,c}\ \forall e,c$;

26                   Update gain estimates $Q^{e,c}\ \forall e,c$;

27                  

28            Perform Algorithm 1 to update $\theta_{t}$ and $\theta_{t}^{\prime}$;

29             $t\leftarrow t+1$;

30            

31      $loop^{\prime}\leftarrow loop^{\prime}+1$;

Algorithm 2 DDQN-CMAB RL Framework

In this subsection, we reformulate the second sub-problem, adaptive configurations selection, into a contextual MAB problem and employ several multi-armed bandit models to dynamically decide the configurations $g_{t,i}=\{y_{t,i}^{m},z_{t,i}^{r}\}$, where $y^{m}_{t,i}$ represents the detection model and $z^{r}_{t,i}$ represents the offloading resolution for each block $p_{t,i}$.

Based on whether the information density $n_{t,i}$ of block $p_{t,i}$ and bandwidth $b_{t}$ at time slot $t$ are greater than the average information density $E^{i}$ and average bandwidth $E^{b}$, we maintain MABs for four different contexts: i) high information density with high bandwidth ii) low information density with high bandwidth iii) high information density with low bandwidth and iv) low information density with low bandwidth. The motivation behind these four contexts is that if the information density $n_{t,i}$ is lower than the average information density $E^{i}$, an inferior DNN model would be more likely to achieve similar performance with the advanced one because only few giant objects appear in the block. However, in high information density blocks, multiple small targets tend to appear and utilizing the advanced model will obtain better accuracy. Similarly, if the bandwidth $b_{t}$ is lower than the average bandwidth $E^{b}$, the high resolution generates high accuracy but may cause processing failure resulted from an unacceptable transmission delay.

We use the exponential moving averages method to update the average information density $E^{i}$ and average bandwidth $E^{b}$ as follows $E^{i}=\xi_{1}n_{t,i}+(1-\xi_{1})E^{i}$, $E^{b}=\xi_{2}b_{t}+(1-\xi_{2})E^{b}$, where $\xi_{1},\xi_{2}\in[0,1]$ stand for the weight factors for the most recent information density $n_{t,i}$ and bandwidth observation $b_{t}$, respectively. The above equations give higher weights to the latest information density $n_{t,i}$ and bandwidth ${b_{t}}$, allowing the model to quickly respond to recent changes.

Considering that the configurations selection strategy should be different in various contexts, we maintain four independent MAB models denoted as $MAB^{HI}_{HB}$, $MAB^{LI}_{HB}$, $MAB^{HI}_{LB}$ and $MAB^{LI}_{LB}$ to decide the configurations of detection models and offloading resolutions. We use $\mathcal{E}=\{\mathcal{HI,LI}\}\times\{\mathcal{HB,LB}\}$ to denote the context set and use $\mathcal{G}=\mathcal{M}\times\mathcal{R}$ to denote the configurations set. For each context $e\in\mathcal{E}$ and configuration $g\in\mathcal{G}$, we define MAB reward $R^{e,g}$ as $R^{e,g}=R\cdot\mathbbm{1}(e_{t}=e\wedge g_{t}=g)$, where $R$ is the same reward as that in TAODM, $\mathbbm{1}(e_{t}=e\wedge g_{t}=g)$ is the contextual indicating function. Thus, each MAB model gets the reward for its decisions allowing independent training. Besides, the reward estimate $Q^{e,g}$ is updated as follows $Q^{e,g}=Q^{e,g}+\varphi\left(R^{e,g}-Q^{e,g}\right),\forall e\in\mathcal{E},\forall g\in\mathcal{G}$, where $\varphi$ is the decay parameter. Thereby, each reward estimate $Q^{e,g}$ is updated by the corresponding reward $R^{e,g}$. For all MAB models, a $\epsilon$-greedy based method is used to take decision to train the MAB models as follows

$$g_{t,i}=\begin{cases}\operatorname*{arg\,max}_{g\in\mathcal{G}}Q^{e,g},&1-\epsilon^{cmab}\\ \text{random choice},&\epsilon^{cmab}\end{cases},\quad\forall e\in\mathcal{E},$$ (13)

where $\epsilon^{cmab}$ is the probability of choosing a random choice. Since we already have precise average estimate of information density and bandwidth after training, we set $\epsilon^{CMAB}=0$ to only use exploitation method at in the testing process.

III-C Real-time Tracking-assisted Video Analysis Framework

Figure 2: The proposed DCRL framework.

Since TAODM and ROIM are highly coupled and only when TAODM decides to offload the frame to ES, ROIM will be used to decide the configurations for each block. Based on these insights, we propose a double-layer DDQN-CMAB reinforcement learning (DCRL) framework to jointly train DDQN-based offloading agent and CMAB-based configurations selection agent.

As shown in Fig. 2, the proposed DCRL framework consists of two layers: i) in the upper layer, we assign TAODM as the first controller, which observes the hash similarity $h_{t}$, bandwidth $b_{t}$, tracking complexity $c_{t}$ and continuous tracking time $p_{t}$ from the environment, then makes the offloading decision. If TAODM decides to process the frame locally, the frame results will be obtained with a tracking mode $k$. Otherwise, the frame will be sent to the lower layer; ii) in the lower layer, ROIM observes the information density $n_{t,i}$ and bandwidth $b_{t}$, and determines the configuration $g_{t,i}$ of detection models $m$ and offloading resolution $r$ for each block.

In each training step of DCRL, we first offload the entire frame of $f_{0}$ to initialize the last detection results for tracking in the future. Then at each time slot $t$, TAODM determines $a_{t}$ by DDQN. If the $Skip$ mode is adopted, directly use the last detection results as the detection results of $f_{t}$. If the KCF or CSRT mode is adopted, TAODM executes the tracking algorithm corresponding to $a_{t}$ to get detection results. Otherwise, the frame would be processed by ROIM, where CMAB is performed to obtain the configuration $g_{t,i}$ of each block and then offload all blocks to the ES. MAB reward $R^{e,g}$ is calculated to update estimates $Q^{e,g}$ when receiving the results of all blocks. Meanwhile, TAODM receives detection results, and the rewards of DDQN are calculated for updating its DNN parameters matrix $\theta_{t}$ and target Q-network parameters matrix $\theta^{\prime}_{t}$. The detailed steps of the DCRL training framework are summarized in Algorithm 2.

IV Simulation Results

(a) Performance comparison on cumulative rewards.

(b) Performance comparison on mAP.

(c) Performance comparison on processing rate.

(d) Performance comparison on average latency.

In this section, simulations are conducted to evaluate the performance of the proposed DCRL on an open dateset, i.e., Garden2 of Multi-camera Pedestrian Video Dataset[5].To ensure fairness, 80% of each video is used for training and the remaining for testing. Furthermore, let $\tau\in\{5.6e^{-4},6.25e^{-4},6.5e^{-4}\}$ MB/Pixel, $\varepsilon^{ddqn}=0.3$, $\varepsilon^{cmab}=0.3$, $\rho=10$, $\sigma=5$, and $f^{edge}:f^{device}=2:1$.

For comparison purposes, except the proposed DCRL, the following schemes are also simulated as benchmarks. RAND-RAND (R-R): TAODM and ROIM randomly determine offloading and configurations of detection model and offloading resolution. RAND-CMAB (R-C): TAODM randomly determines offloading, while ROIM determines configurations of detection model and offloading resolution using CMAB-based adaptive configurations selection algorithm. DDQN-RAND (D-R): ROIM randomly determines configurations of detection model and offloading resolution, while TAODM determines offloading using DDQN-based offloading algorithm. FULL-HIGH (F-B): TAODM only adopts Offload-full decision and ROIM adopts the highest offloading resolution with the most accurate model configuration.

Fig. 2(a) shows the cumulative rewards with time, it can be observed the proposed DDQN-CMAB (D-C) outperforms the other benchmarks, as it filters the repeated spatial-temporal semantic information to ensure the results from ES return in time while obtaining high accuracy. In addition, F-B obtains the worst performance because offloading full frames generates a large transmission delay under fluctuating network conditions.

Fig. 2(b) illustrates the mean average precision (mAP) with different methods, we can see D-C obtains the highest accuracy except for F-B. Fig. 2(c) compares the processing rate of different methods, and it can be seen that D-C achieves the best processing rate compared with others. Specifically, DCRL improves the processing rate by up to 66.3% compared to F-B. Compare Fig. 2(b) to Fig. 2(c), it can be observed that i) F-H realizes the best mAP performance while the lowest processing rate for large transmission delays, ii) R-H achieves higher mAP with higher processing rate since the repeated spatial-temporal semantic information is filtered, where the effectiveness of ROI is fully illustrated.

Fig. 2(d) compares the average latency of different methods, and D-C achieves the lowest latency compared with others. Exploring the reasons, we find that TAODM adopts offload-ROI to minimize the adverse effects of fluctuating network conditions. When network conditions are stressful, ROIM can effectively reduce data volume to ensure that critical information is offloaded to the ES for accurate detection. Besides, dramatic network fluctuations are often temporary, and a local tracking algorithm is used to tide over this difficult period to improve accuracy when network conditions are poor.

V Conclusion

In this paper, a DCRL framework integrating a DDQN-based optimal offloading decision approach and a CMAB-based adaptive configuration selection approach is proposed to address the challenge of balancing frame processing rate and accuracy performance of edge-based real-time video analysis systems in intelligent visual devices. By decomposing the optimization problem of video analysis into two subproblems and generating an optimal strategy for offloading mode and the configurations of detection model and resolution selection, our proposed framework can effectively solve the overall optimization problem. Experimental results show that our approach outperforms counterparts in terms of ensuring a high process rate with high detection accuracy.