Fuzzy Q-learning-based Opportunistic Communication for MEC-enhanced Vehicular Crowdsensing

The fuzzification module converts the crisp values of the control inputs into fuzzy values. A fuzzy variable has values, which are defined by linguistic variables (fuzzy sets or subsets) such as low, medium, high, where each is defined by a gradually varying membership function.

The knowledge base stores IF-THEN rules provided by experts. The expert knowledge is a collection of Fuzzy membership functions and a set of Fuzzy rules having the form: IF (conditions are fulfilled) THEN (consequences are inferred). More explicitly, a Fuzzy rule $R_{i}$ with $k$-inputs and $1$-output can be represented as follows.

where $\{I_{1},\cdots,I_{k}\}$ represents the crisp inputs to the rule. $\{A_{i1},\cdots,A_{ik}\}$ and $B_{i}$ are linguistic variables. The operator $\Theta$ can be AND, OR, or NOT.

The inference engine deduces the Fuzzy control actions by employing Fuzzy implication and Fuzzy rules of inference. It calculates the membership degree ($\mu$) of the output for all linguistic variables by applying the rule set described in the Knowledge Base. For Fuzzy rules with many inputs, the output calculation depends on the operators used inside it. The calculation for each type of operator is described as follows:

The defuzzification module translates Fuzzy control values into crisp numbers, that is, it links a single point to a Fuzzy set, given that the point belongs to the support of the Fuzzy set. The most well-known defuzzification technique is the centre-of-area (COA) or centre-of-gravity (COG). For continuous membership function, the defuzzified value denoted as $x^{*}$ using COG is defined as:

where $\mu_{A}(x)$ is the output membership of the linguistic variable $A$.

For each device $D_{i}$, the state at a time slot $t$ is a quadruple consisting of the following items:

• •

  $\mu_{i}(t)$: the timing $D_{i}$ generates the last packet, i.e., the time from when the last packet is generated until the current time.

• •

  $c_{i}(t)$: the computing resource of $D_{i}$ that is remaining at time slot $t$.

• •

  $c_{\mathcal{N}}(t)$: the remaining resource of the nearest device $\mathcal{N}$, if $\mathcal{N}$ is in the communication range of $D_{i}$.

• •

  $\mathcal{N}_{i}^{R}(t)$: a binary variable indicating whether $D_{i}$ is in the communication range of a RSU.

Furthermore, the first three entries of a state are rounded as follows. $\mu_{i}(t)$ is rounded in time step units. $c_{i}(t)$ and $c_{\mathcal{N}}(t)$ are both rounded in Megabytes. In this way, we have discretized the value of the state vector. As a result, the state space is limited. A crowdsensing device can conduct one of the following actions at each time slot $t$:

1. i)

   Kee** the data in the local queue,

2. ii)

   Sending the data directly to the crowdsensing server via 4G communication channel,

3. iii)

   Sending the data to the nearest RSU, if $D_{i}$ is in the communication range of an RSU,

4. iv)

   Sending the data to the nearest device, if they are in the communication range of each other.

Because both the state and the actions obtain discrete values, the size of the Q table is fixed. As a result, we may use the common sequential searching technique to retrieve an entry in the Q table.

We denote by $\mathcal{R}_{i}(t)$ the reward received when device $D_{i}$ performs action $A_{i}(t)$. Our goal is to minimize the total amount of data transmitted by 4G while guaranteeing that the data latency does not exceed a predefined threshold $\delta$. For each type of action, the action’s goodness is reflected by different indicator. Therefore, instead of define a an unique formula for the reward function, we break it down into multiple cases as follows.

where $\Delta_{i}(t)=t-\mu_{i}(t)$ is the time elapsed from when the data is collected, $\Delta{c}(t)=c_{\mathcal{N}}(t)-c_{i}(t)$ in that $c_{\mathcal{N}}(t)$ is the remaining resource of the nearest device, and $p$ is a significantly large positive number. $\theta$ is a parameter in the range of $[0,1]$ which we call priority factor. The value of $\theta$ is determined by Fuzzy logic as will be described in Section IV-C. The rationale behind the reward function is as follows. Firstly, when an action results in a resource exhaustion of the current device, or when the data’s latency exceeds the threshold, the agent will be punished by a substantial negative reward according to Formula (6). Otherwise, the reward is calculated by Formulas from (8) to (13) depending on the action type.

The reward obtained when performing the action of sending a packet to the server is represented by Formula (8). As can be seen, this reward is inversely proportional to $c_{i}(t)$, the device’s remaining resource. It indicates that the lower $c_{i}(t)$, the bigger the reward, indicating that the action of sending to the server is encouraged. In contrast, when $c_{i}(t)$ increases, the reward for sending a packet to the server decreases. Moreover, when $c_{i}(t)$ is large enough (more than $\theta\times C^{*}_{i}$), the reward turns negative, and transmitting to the server is discouraged. It means that when the remaining resource is significant enough, the device will temporarily not need to transmit the packet to the server to save money on 4G. The reward for transmitting a packet to an RSU is represented by Equation (9). Similarly to formula (8), the smaller the remaining resource, the more strongly urged is the action of transmitting to an RSU. The reward for transmitting to an adjacent device is represented by Equation (11). This reward is positive if $\Delta c(t)$ is greater than $0$, i.e., the neighbor’s remaining resource is greater than the current device. It implies that transmitting to an adjacent device is only recommended if the neighboring device has a greater remaining resource than the current device. Finally, rewards for actions of sending to the server/RSU/other devices share a common term of $\frac{1}{1+\Delta_{i}(t)}$. It should be noted that this is inversely proportional to $\Delta_{i}(t)$. As a result, this term encourages the agent to decide on offloading tasks as quickly as feasible rather than holding the packet in local memory.

R_i(t)=&Output: the next action; updated $Q$ table.

R_i(t)=&           $C_{i}^{*}$: the maximum computing capacity; R_i(t)=           $\Delta_{i}(t)$: the elapsed time; R_i(t)=           $\delta$: the data latency threshold. R_i(t)=Output: $\theta$.

To choose the next action, we leverage the $\epsilon$-greedy policy. In particular, the $\epsilon$ steadily declines over time as follows:

where the maximum time is a predefined threshold. The details of this policy are described in Algorithm IV-B2. At each step, the agent selects a random action with a probability of $\epsilon$ from the action space (line 4 in Algorithm IV-B2) and the action with the greatest Q value with a probability of $1-\epsilon$ (line 6 in Algorithm IV-B2).

We found that simultaneously optimizing two objectives, namely minimizing the total amount of data transmitted over the 4G communication channel and guaranteeing the data latency, is challenging. Specifically, devices tend to store data until they can send it to an RSU to reduce the usage of 4G transmission. This strategy, however, may cause a large delay due to the waiting period. As a result, rather than using a fixed value for $\theta$, we propose a mechanism for adjusting $\theta$ dynamically in response to network status. We observe that the remaining resource and the amount of time the data has elapsed are two factors that influence a device’s behavior. The device may tolerate more latency when the device’s remaining resource is large or data latency is minimal; therefore, the agent should avoid transmitting immediately to the server to save on communication expenses. As a result, $\theta$ should be insignificant when the device’s remaining resource is large and the data latency is low. In contrast, when the remaining resource in the device is low or data latency is large, the device should send data to the server rather than transferring it to the nearest device to ensure the latency constraint is satisfied. Consequently, $\theta$ should be set high enough that the reward of the action sending to the server surpasses the reward of relaying to the nearest device. Furthermore, as long as the device is within an RSU’s communication range, transmission to the RSU takes precedence, i.e., the action of sending to an RSU is designed to get the highest reward of all the actions.

Motivated by the observation mentioned above, we design a Fuzzy-based algorithm for dynamically adjust the value of $\theta$. The pseudo-code of the algorithm is presented in Algorithm IV-B2.

The input of the fuzzification process is a pair consisting of the remaining resource, and the latency the data has elapsed (i.e., defined by the time from when the data is measured until the current time). The input then is mapped to three linguistic variables: low (L), medium (M) and high (H). The output of the Fuzzification is also mapped to three respective levels: low, medium and high. We use the trapezoidal Fuzzy set that could be described by the following formula:

where $x$ is the input, and $a,b,c,d$ are parameters. The values of $a,b,c,d$ are represented in Table IV and V. Fig.5 and 5 illustrate the shape of the input membership functions.

We have two input variables, namely the remaining resource, and the elapsed time, each of which is transformed into three Fuzzy sets. Therefore, we have $3^{2}=9$ Fuzzy rules in total. The rules are shown in Table VI. These rules are designed based on the observation described in Section IV-C1. Our rules have the form of “IF (the remaining resource is A) AND (the elapsed time is B) THEN ($\theta$ is C)”, where $A,B,C$ obtain the values of low, medium, and high. To ease the presentation, we denote low, medium, and high as $L$, $M$, and $H$, respectively.

As our Fuzzy rules are based on AND operator, the output membership function is defined as below.

After going through the steps above, the Fuzzy set with the highest membership degree is considered as the output variable. Finally, we utilize the CoG function in Formula (4) to calculate the crisp value of the output’s fuzzy set.

In this section, we perform grid search on $P_{keep}$, $P_{rsu}$ and $P_{sensor}$ to identify the sub-optimal combination²²2$P_{server}=1-P_{keep}-P_{rsu}-P{sensor}$. Fig. 6 illustrates the rate of dropped packets ($r_{drop}$), $\delta$-delayed packets ($r_{delay}$) and the 4G communication ratio $r_{server}$ when we varies the value from 0.1 to 0.5. The bigger circle refers to the higher rate. Through this result, we consider different configurations of FP that could have small packet dropped rate, e.g., $r_{drop}\approx 0$. We then pick up three configurations with the highest, medium and lowest 4G communication ratio $r_{server}$ to be the representations for the FP baselines (named as FP1, FP2, and FP3, respectively). The detail settings of FP are summarized in Table VIII.

We shows the breakdown of the simulation packets (in percentage) of the proposed method as well as the greedy, and the FP baseline strategies in Fig. 7. We consider the lower value to be the better performance. First of all, with the FP strategy, devices tend to hold the data in their queue until they can send it to an RSU/or the nearest device (the offload-hit as mentioned in section V-A). This strategy leads to a longer delay of a message and a higher number of dropped packets (when the queue is full). As a result, the FP baseline strategies can not avoid the packet-dropped issue in all the cases, e.g., 0.15% of $r_{drop}$ as in FP3 when we set the latency threshold to 5 time slots. In contrast, the greedy strategy does not keep the packet in the queue of a given sensor when there exist no RSUs and devices around it, yet encourage to send data directly to the cloud server. Thus, it could deliver all the packets to the server with a zero-packet dropped rate. Similar to the greedy strategy, our proposed method also achieves the zero-packet dropped rate and guarantees all the generated packets can reach the server. It is the result of flexibly constructing the priority factor $\theta$ using the Fuzzy logic approach which considers both the number of packets in the local queue and the data latency of packets.

The result in Fig. 7 also shows that our method provides a smaller number of $\delta$-delayed packet than the FP3 strategy, e.g., 1.86% and 0.93% which are $1.3\times$ and $2.94\times$ lower than those of FP3. In addition, although the $r_{delay}$ of the greedy, FP1, and FP2 strategies are lower than those of our proposed method, they require much more 4G cost than ours. Specifically, in all the experiments, our method requires the lowest communication cost ($r_{server}$), i.e., only around 60% of packets travel through the 4G network while those are around $70\%$ as in the greedy. Furthermore, the 4G communication cost, i.e., $r_{server}$ of the FP strategies are much higher than expected, i.e., around $P_{server}$, due to the offload-missed issue mentioned in Section V-A. When a device decides to send a packet to the nearest RSU/device but there exist no RSUs/devices around, or when the local queue is full, it sends the packet to server directly vía the 4G network. Such offload-missed issue increases the 4G communication cost significantly. For example, in the FP1, FP2 and FP3, more than $96\%$, $86\%$ and $73\%$ of packets use the 4G communication channel (although it is designed with a fixed possibility of 70%, 50% and 30%, respectively).

In Fig. 8, we present the total energy consumption of offloading strategies using the stacked bars. The line series shows the average energy consumption of packets that reach the crowdsensing server. Interestingly, in the case of $\delta=5$, although the energy consumption of our proposed strategy is higher than that of FP3, the average energy consumption per packet of our proposal is smaller than that of FP3. It refers to the result in Fig. 7(a) when the rate of packets transmitted to the server (1 - $r_{drop}$) of our proposed strategy is higher while the rate of the packet transmitted to the server via the 4G channel ($r_{server}$) is smaller than those of the FP1 and FP2. On the other hand, when the packets transmitted to the server of the four targeted strategies are similar, e.g., $\delta=10$, our proposal consumes the lowest energy because it uses the 4G channel least.

R_i(t)=& Figure 9: Impacts of the packet generation interval ($\lambda_{d}$). $\delta$ is fixed to 5 time slots.

R_i(t)=& Figure 10: Impact of the data latency threshold ($\delta$). $\lambda_{d}$ is fixed to 1 time slot.

In summary, the results imply that the performance/cost of the baseline FP approach is sensitive to the environment/network due to its coarse fixed configuration. Thus, it requires effort to manually figure out the best configuration when implementing this method in the real world (for a given specific environment). The greedy strategy focuses too much on the rate of news stories without balancing the cost of communication. By contrast, our proposed method learns the environment/network information to make the decision flexibly so that it can avoid both the packet-dropped issue and offload-missed issue while using a smaller communication cost.

In this evaluation, we estimate the impact of the packet generation interval, i.e., the frequency the devices collect the sensory data, on our proposed method. We set the data latency threshold to $5$ time slots while changing the packet generation interval from $1\to 5$ time slots. The result in Fig. V-B5 shows that the relative communication cost between our proposed method and the baselines does not change. There’s a trivial impact of packet generation rate on all the strategies in communication cost. Our proposed method always uses least 4G communication with a small rate of the delayed packet. Although the greedy and FP3 could achieve smaller delayed rates than ours, these two strategies scarify the communication cost. Specifically, FP3 sends 96% of packets via the 4G network, which is $1.5\times$ higher than that of our proposed method. When the packet generation interval is slow enough, instead of being dropped, a packet will be stored in a queue and, thus, the rate of delayed packets is increased. In general, this trend also appears when the relative packet size over the capacity of the local queue is too big (that leads to a smaller number of the packet can be stored in the queue until the queue is full). Interestingly, the packet generation interval does not affect the packet delayed rate of FP1, FP2 and greedy strategies where the possibility of kee** a packet in the edge devices are not too high, i.e., $P_{rsu}+P_{sensor}=0.2$ as in FP1 strategy. In contrast, because FP3 and our proposed method try to avoid to use the 4G communication channel as much as possible, the packet generation time has much higher impact on those strategies. For example, to maintain $\approx 63\%$ of 4G communication rate when the packet generation interval changed, our proposed method could not avoid the delayed packet, i.e., around $2-4\%$.

In this experiment, we explore the relationship of our proposed method’s performance with the data latency threshold, e.g., the maximum latency required by an application. We fix the packet generation interval $\lambda_{d}=1$ while changing the latency threshold $\delta$ from $5\to 25$ time slots, Fig. V-B5 shows the related results. When the latency threshold increases the $\delta$-delayed packets rate of our proposed method as well as the baseline strategies decrease. This is an expected result because a packet has a longer time to stay in the local devices. Furthermore, let us remind our strategy of the reward function in our Q-Learning method (as shown in the Formula (6)). When the data latency exceeds the threshold, there is a higher possibility of a packet being directly sent to the server using the 4G communication channel by assigning a negative reward. As the result, when the latency threshold increases, the number of packets that meets such condition and use the 4G communication channel becomes smaller. For example, the 4G communication ratio of our proposed method changes from $64.8$% to $58.7$% when the latency threshold change from 5 to 25-time slots. This trend does not appear in the greedy and FP strategies, These strategies maintain the 4G communication ratio at around 70%, 96%, 88%, and 77%. with all the configuration of the data latency threshold.

In this section, we investigate the performance of our proposed method when varying the forwarding data’s amount, e.g., the packet size from $1\to 5$ MB. It is worth noting that the packet size affects both the cost of 4G as well as the packet delayed rate because the maximum number of packets stored in the queue decrease when the packet size increases. In this evaluation, we fix the queue size of a sensor to 25 MB, i.e., the ratio of packet size to queue size varies from $1:25$ to $1:5$. First of all, we observe that there is no packet dropped, e.g., $r_{drop}=0$, in all the cases. Second, the result in Fig. V-C3 shows that the performance of the greedy and FP baselines is nearly not changed when the packet size is varied. Only our proposed method shows the decrease of ratio $\delta$-delay packet when the packet size becomes smaller because our method dynamically selects the transmission route based on the device’s remaining resource (Formula  8).

R_i(t)=& Figure 11: Impact of the packet size. $\delta$ is fixed to 10 time slots.

R_i(t)=& Figure 12: Impact of the contact rate. $\delta$ is fixed to 10 time slots.

In this section, we investigate the impacts of contact rate on the performance of the algorithms. We begin by explicitly defining contact rate mathematically. We refer to each device’s contact time as the interval at which it enters the communication area of other devices. The contact rate of a device is then calculated by dividing its contact time by its entire time on the road. Furthermore, the network’s average contact rate is defined as the average contact rate across all devices. In this section, we vary the contact rate by changing the transmission range of the devices from $10\to 120$ meter as shown in Table IX. The higher the sensor transmission range, the higher the network contact rate. We then observe the change in the 4G communication ratio, delayed packet ratio, and the packet dropped rate. The results are presented in Fig V-C4. We observe that there is no packet dropped in most of the cases except the FP3, e.g., around 0.01%. We also confirm that there is no clear correlation between the 4G communication ratio of the FP strategies and the transmission range of devices. However, when the transmission range of devices increases, the 4G communication ratio of the greedy strategy decreases significantly, e.g., from 71% to 63%. In all the configurations, our proposed method always achieves the lowest 4G communication cost. This result implies the stability of the proposed method.