Unveiling Delay Effects in Traffic Forecasting: A Perspective from Spatial-Temporal Delay Differential Equations

A large body of research has been conducted on traffic flow forecasting in recent years. Traffic flow forecasting can be viewed as a spatial-temporal forecasting task leveraging spatial-temporal data collected by various sensors to predict future traffic conditions. In recent years, deep learning methods have dominated traffic flow forecasting issues, due to their superior ability to model complex spatial-temporal correlations. The models combining graph neural networks (GNN) (Kipf and Welling, 2016; Long et al., 2021a, b) and recurrent neural networks (RNN) (Schuster and Paliwal, 1997) are the representative. Specifically, DCRNN (Li et al., 2017) views the traffic flow as a diffusion process on a directed graph and utilizes GRU to capture the temporal features. STGCN (Yu et al., 2018) utilizes graph convolution and 1D convolution to capture spatial dependencies and temporal correlations respectively. (Jiang et al., 2023) proposes a PDFormer model. While PDFormer mentions the concept of delay, its core mechanism involves utilizing attention to the historical time series, rather than explicitly utilizing delay in the propagation process. It implies that it still cannot capture the intricate delay information among graph vertices. In general, despite their achieved success, all existing works are limited to the spatial-temporal stacking structure and ignore the delay effect, which deviates from the real situation of traffic.

The neural ordinary differential equation (NODE) (Chen et al., 2018) was first proposed as a continuous version of residual neural networks (ResNet) (Wu et al., 2019). Due to its apparent suitability for dynamics-governed time-series, NODE is soon utilized in the time series analysis, especially when the input data is irregularly sampled or partially observed (De Brouwer et al., 2019; Rubanova et al., 2019; Lechner and Hasani, 2020). However, the solution of neural ODE is totally determined by the initial condition, which means later arriving data would not exert influence on the equation, this is also why neural ODE is generally applied cooperatively with RNN modules to deal with incoming data. Neural control differential equation (NCDE) (Kidger et al., 2020) solved this problem by constructing a continuous path from discrete input data, and adjusting the evolution trajectory according to the continuous control signal. Another parallel and relevant work is the Neural Delay Differential Equation (NDDE) (Zhu et al., 2021). As emphasized in (Dupont et al., 2019), the flow of NODE cannot represent the systems with the effect of time delay. The emergence of NDDE fills the blank.

Few pioneering works have been conducted in traffic forecasting with a neural differential equation framework. STGODE (Fang et al., 2021) first utilizes the neural ODE to transform graph convolution into a continuous version to acquire a larger spatial-temporal receptive field. STG-NCDE (Choi et al., 2021) adopts the neural CDE to deal with irregular-sampled time series. Despite the success have achieved, none of these works take the delay effect into consideration. In this paper, we first extend the NDDE to multi-variable conditions for spatial-temporal modeling and cooperate with NCDE to construct continuous traffic signal evolution.

In this paper, we focus on the long-term traffic flow forecasting problem. The traffic network is represented as a graph $\mathcal{G}=(V,E,A)$, where $V$ is the set of $N$ traffic nodes, $E$ is the set of edges, and $A\in\mathbb{R}^{N\times N}$ is an adjacency matrix representing the connectivity of $N$ nodes. The traffic flow is represented as a flow matrix $X\in\mathbb{R}^{T\times N}$, and $X_{tn}$ denotes the traffic flow of node $n$ at time $t$. The goal of traffic flow forecasting is to learn a map** function $f$ to predict the future $T^{\prime}$ steps traffic flow given the historical $T$ steps information, which can be formulated as follows,

Moreover, the model trained at some fixed grain may need to generate a differently-grained prediction to satisfy the complicated real-world needs, which is formulated as follows,

where $dt_{1},dt_{2},\cdots,dt_{n}$ are arbitrary positive numbers.

As shown in Fig. (1), there exist propagation delays in real-world traffic conditions, which are in contrast to the general GNN propagation process. For instance, when a traffic incident transpires in a particular area, it may necessitate several minutes to impact traffic conditions in adjacent regions. To gain more accurate depictions of the time delay, we design two delay estimators to capture the propagation delay between connected nodes.

The direct approach estimates delays among neighboring nodes by maximizing the cross-correlation (MCC) (Azaria and Hertz, 1984) as a pre-processing step. Specifically, given two time series, $x_{i}$ and $x_{j}$, where we assume that $x_{j}$ is influenced by $x_{i}$, we initially smooth them through interpolation, resulting in $\tilde{x}_{i}$ and $\tilde{x}_{j}$. Subsequently, we determine the delay, denoted as $\tau_{ij}$, by identifying the peak of their cross-correlation after shifting, as expressed by the below equation,

where $\tilde{x}_{i}^{\rightarrow{k}}$ denotes performing a $k$-step shift to $\tilde{x}_{i}$, and $corr$ is the Pearson correlation function in this paper. We estimate all the delay values in advance through pre-processing based on historical data.

The second approach involves modeling time delay as a learnable pattern. The time delay implicitly reflects external factors associated with the traffic network, including road length, road capacity, and more. Furthermore, the delay itself exhibits inherent variability, for example, longer delays often occur during morning and evening rush hours. In this approach, we assign two learnable delay parameters to each edge: one for peak hours and another for non-peak hours.

Please note that the delay value $\tau$ serves as an indicator to select a historical state in the equation (11). Generally, $\tau$ is considered non-learnable in this context because the model cannot compute the gradient of $\tau$, which is theoretically $\frac{\mathrm{d}h(t-\tau)}{\mathrm{d}\tau}$. However, thanks to the continuous modeling approach, we can indeed obtain this gradient. As demonstrated in equation (9), the derivative of $h$ with respect to $t$ is well-defined. Consequently, we can incorporate the gradient of $\tau$ in the neural network by explicitly defining the backward computation of $h$ with respect to $\tau$.

One key challenge with DDE is that, once the network parameters are fixed, the dynamic evolution becomes entirely self-contained and does not integrate incoming inputs, leading to the loss of valuable information. To address this issue, we introduce a control signal inspired by Neural CDE, offering a solution to this problem.

Following Neural CDE, we generate a continuous representation from the raw inputs through the natural cubic spline method, which ensures a minimum of two continuous differentiable properties,

where $X(t)$ is a continuous representation, $\Phi$ denotes the natural cubic spline function, and $x^{i}$ denotes the input at time $t_{i}$. Thus we have

where $\tilde{f}$ is a transformation function to match dimensions, and $X_{i}$ is the continuous representation of node $v_{i}$. The derivative of $X$, denoted as $\frac{\mathrm{d}X_{i}(t)}{\mathrm{d}t}$, signifies the trend or fluctuation in traffic flow, constantly influencing the direction of dynamic evolution.

We formulate the complete update process of hidden states as follows,

We employ another STDDE to generate the final outputs. In this approach, we consider the last stage of the hidden flow in the input process as the history function for the output process. This strategy offers two key advantages: Firstly, the hidden states remain continuous within the manifold space, ensuring unity between the input and output processes. Secondly, unlike traditional output layers that provide predictions with a fixed horizon, we can accurately predict traffic flow at various frequencies. It provides more flexibility and adaptability to different scenarios.

where $f$ is a map** function to get final outputs from hidden states, and $t^{\prime}$ is any target output point. This output approach better highlights the continuity of STDDE and fully capitalizes on its capabilities. With this model, we have the flexibility to generate predictions at any time, rather than being limited to a specific point. The continuous output module is well-suited for scenarios involving sparse traffic sensor data, especially when a higher level of precision is required during inference than in the training phase.

Finally, as our objective function in the context of traffic flow forecasting, we employ the widely-used Huber loss, which is known for its robustness in handling outliers compared to the squared error loss.

where $\hat{Y}$ is the ground truth, and $\delta$ is a hyperparameter which controls the sensitivity to outliers.

The time complexity analysis of STDDE is presented in the Appendix.

We verify the performance of STDDE on six real-world traffic datasets, namely PeMSD7 (M), PeMSD7 (L), PeMS03, PeMS04, PeMS07, and PeMS08, which are collected by Caltrans Performance Measurement System in real-time every 30 seconds (Chen et al., 2001) and aggregated into 5-min intervals, which means there are 288 time-steps for one day. More details of the datasets are listed in Table 1. We standardize the input by removing the mean and scaling to unit variance.

We select the following representative baselines as our competitors, and more details can be found in the Appendix:

• •

  Spatio-Temporal Graph Convolution Models including STGCN (Yu et al., 2018), STSGCN (Song et al., 2020a), DCRNN (Li et al., 2018), AGCRN (Bai et al., 2020), ASTGCN (Guo et al., 2019), FOGS (Rao et al., 2022). STGCN utilizes graph convolution and 1D convolution to capture spatial-temporal correlations. STSGCN utilizes multiple localized spatial-temporal subgraph modules to capture the spatial-temporal correlations directly. DCRNN integrates graph convolution into an encoder-decoder gated recurrent unit. AGCRN captures node-specific spatial and temporal correlations automatically without a pre-defined graph. ASTGCN utilizes spatial and temporal attention mechanisms to model their dynamics. DSTAGN constructs a dynamic graph instead of relying on a pre-defined static one. FOGS utilizes first-order gradients rather than specific flows, which effectively circumvent issues associated with fitting irregularly-shaped distributions.

• •

  Spatial-Temporal Graph Ordinary Differential Equation Models, including STG-ODE (Fang et al., 2021) and STG-NCDE (Choi et al., 2021). STGODE proposes an ordinary differential equation-based continuous GNN, to capture long-range spatial-temporal dependencies. STG-NCDE designs two NCDEs to capture temporal and spatial properties respectively.

• •

  Delay-aware Traffic Models only include one related work, which is PDFormer (Jiang et al., 2023). Its transformer-based mechanism involves utilizing attention to the historical time series.

We split all datasets with a ratio of 6: 2: 2 into training sets, validation sets, and test sets. One hour of historical data is used to predict traffic conditions in the next 60 minutes, i.e. $T=12$ and $T^{\prime}=12$. All experiments are conducted on the same Linux server and GPU. The dimension of hidden states is set to 64. We train our model using the Adam optimizer with a learning rate of 0.001. The batch size is 32 and the training epoch is 200. Mean absolute error (MAE), mean absolute percentage error (MAPE), and root mean squared error (RMSE) are used to measure the performance. For baselines, we use their officially reported results if accessible. If not, we run their codes based on their recommendation configurations.

We first provide conceptual experiments to evaluate the necessity of motivation and the effectiveness of the proposed delay-aware differential equation and continuous output module.

Table 2 shows the results of the proposed STDDE model and competitive baselines on traffic flow forecasting tasks in six popular used real-world datasets. We conclude with the following findings:

• •

  Our model yields the best performance regarding all the metrics for most datasets, which suggests the effectiveness of our spatial-temporal delay traffic flow forecasting.

• •

  Continuous spatial-temporal neural networks, i.e. STGODE, STG-NCDE, and STDDE, perform better than traditional GNN-based ones, such as popularly used AGCRN, STGCN, DCRNN, and DSTAGNN. It shows the direction of continuous modeling in spatial-temporal traffic flow forecasting is effective and worth gaining more attention.

• •

  The proposed STDDE and PDFormer generally perform better than other continuous spatial-temporal methods, i.e. STGODE and STG-NCDE, which indicates that capturing and utilizing historical delay-related information is necessary and of great significance.

• •

  STDDE gains better performance than PDFormer, which shows the effectiveness of explicit spatial-temporal delay-aware differential equations and continuous modeling.