Surrogate-based cross-correlation for particle image velocimetry

Abstract

This paper presents a novel surrogate-based cross-correlation (SBCC) framework to improve the correlation performance for practical particle image velocimetry (PIV). The basic idea is that an optimized surrogate filter/image, replacing one raw image, will produce a more accurate and robust correlation signal. Specifically, the surrogate image is encouraged to generate perfect Gaussian-shaped correlation map to tracking particles (PIV image pair) while producing zero responses to image noise (context images). And the problem is formularized with an objective function composed of surrogate loss and consistency loss. As a result, the closed-form solution provides an efficient multivariate operator that could consider other negative context images. Compared with the state-of-the-art baseline methods (background subtraction, robust phase correlation, etc.), our SBCC method exhibits significant performance improvement (accuracy and robustness) on the synthetic dataset and several challenging experimental PIV cases. Besides, our implementation with experimental details (https://github.com/yongleex/SBCC) is also available for interested researchers.

I Introduction

Particle Image Velocimetry (PIV) is a popular non-intrusive instrument for flow field measurement in experimental fluid dynamicsAdrian (1984); Raffel et al. (2018); Lee and Mei (2022). PIV generates quantitative vector field by analyzing the consecutive particle recordings. However, the particle images are easily deteriorated in a practical measurement due to non-uniform light illumination, light reflections, background noise sources, camera dark noise, etc Honkanen and Nobach (2005); Deen et al. (2010); Sciacchitano and Scarano (2014). Therefore, the accuracy and robustness of PIV results could be significantly decreased, recognized as uncertainty Sciacchitano (2019), peak-locking Michaelis, Neal, and Wieneke (2016) and/or outliers Wang et al. (2015); Lee, Yang, and Yin (2017a). Thus, in this work, we focus on the challenging PIV estimation problem caused by the deteriorated particle recordings.

Over the past 40 years, the mainstream velocity estimation methods — cross-correlation Willert and Gharib (1991); Scarano (2001); Wang, He, and Wang (2020); Zhu et al. (2022); Gao et al. (2021), optical flow (OF) Corpetti et al. (2006); Zhong, Yang, and Yin (2017); Lu et al. (2021) and deep neural-network (DNN) regression Lee, Yang, and Yin (2017b); Cai et al. (2019a); Lagemann et al. (2021); Cao et al. (2024)— are not particularly designed for robust PIV estimation. 1). The vanilla standard cross-correlation (SCC) computes the image similarity (dot product) as a function of the relative displacement. And SCC is not robust to image noise (such as, additive background noise, non-uniform illumination) because the noise is also correlated Eckstein and Vlachos (2009). Therefore, the generalized cross-correlation (GCC) methods improve signal-to-noise ratio of PIV cross-correlation via post-processing the correlation coefficients with different spectral filters, including phase correlation (PC) Horner and Gianino (1984), symmetric phase-only filter (SPOF) Wernet (2005), robust phase correlation (RPC) Eckstein and Vlachos (2009) to name a few. As a result, the GCC methods have achieved acceptable performance and have been extensively equipped for the majority of PIV software. 2). The optical flow Corpetti et al. (2006) methods employ a preservation principle, namely that particle image brightness attribute does not change after a movement, to estimate the particle displacement. The risk of failure rises if the brightness preservation principle breaks, which often occurs with image noise in practical measurements. Thus, replacing the brightness attribute with other robust attributes (image gradient, image phase) could be a straightforward modification Zhong, Yang, and Yin (2017). Besides, a deliberated OF model with improved regularization term also contributes to accurate PIV estimation Bao, Yang, and ** (2014); Lu et al. (2021). Meanwhile, the complex OF models often come with heavy computation cost. 3). As efficient inference methods, DNN-based regression methodsLee, Yang, and Yin (2017b); Cai et al. (2019a); Zhang and Piggott (2020); Yu et al. (2021); Lagemann et al. (2021); Yu et al. (2023) have been attracting researchers’ interest due to the powerful model capacity. However, the generalization of DNN for PIV depends on the noise type of training dataset Lagemann et al. (2022). Totally speaking, the CC, OF, DNN can be treated as bivariate operators that only take in two particle image frames, regardless of the concrete noise signal of a measurement. Herein, we focus on the CC methods employed by most practitioners Kähler et al. (2016); Liberzon et al. (2016); Xie, Wang, and Xu (2022).

A straightforward alternative to achieve robust PIV analysis is to directly improve the image quality Dellenback, Macharivilakathu, and Pierce (2000); Shavit, Lowe, and Steinbuck (2007); Lee et al. (2022); Fan et al. (2023); Zhao et al. (2024). Among different image pre-processing, the background subtraction performs well given a good reference background— concrete noise signal Mejia-Alvarez and Christensen (2013); Mendez et al. (2017); Kähler et al. (2016); Wang et al. (2020). The background image can either be recorded in the absence of seeding, or, if this is not possible, through temporal or spatial analysis from raw PIV recordings Raffel et al. (2018). The background image could be the minimum intensity image from double-frame PIV images, and works reliably for nonstational flow with severe background noise Honkanen and Nobach (2005); Deen et al. (2010). The varying background image could be extracted via a temporal Butterworth filter from a large number of raw PIV recordings Sciacchitano and Scarano (2014). A customized background can be adaptive reconstructed through proper orthogonal decomposition (POD) Mendez et al. (2017). Without extra temporal information, the spatial low-pass filter (LPF) utilizes the blur image to approximate a background, including Gaussian filter, median filter Adrian and Westerweel (2011), anisotropic diffusion Adatrao and Sciacchitano (2019), etc. Due to effectiveness, background subtraction has become an essential step of standard PIV pipeline (Fig. 1(a)). However, using one background to model the complex noise signal is still challenging.

Correlation filters have achieved competitive success in object tracking by learning a discriminative linear tracker from several image templates Bolme et al. (2010); Henriques et al. (2012, 2015). It generates an optimal filter/tracker that maximizes the convolution/tracking performance from multiple templates, as detailed in Section II. As a result of closed-form solution, correlation filter algorithm (minimum output sum of squared error, MOSSE Bolme et al. (2010)) is not only easy to implement but significant faster. The extensive experiments exhibit that the learned filter outperforms the original image templates in both robustness and accuracy.

(a) Standard PIV pipeline with background subtraction

(b) Our new PIV pipeline with SBCC

Figure 1: Different illustrative pipelines for noisy PIV images. \raisebox{-.7pt} {E}⃝: background/contexts extraction; $\ominus$: background subtraction; $\bigotimes$: cross-correlation. \raisebox{-.1pt} {$\star$}⃝: surrogate-based cross-correlation.

Our insight is that taking more negative context images (the concrete noise signal) into account will obtain a more robust tracker, and the MOSSE algorithm can be easily adapted to negative contexts with little effort. Different from object tracking video, the images of PIV pair are templates for each other Wereley and Meinhart (2001). As a result, a novel surrogate-based cross correlation (SBCC) framework is proposed by combining forward surrogate tracking with backward surrogate tracking. And SBCC—a multivariate operator— enables us to reform PIV pipeline for accurate and robust measurement, rather than pursuing a perfect clean image (Fig. 1(b)). The main contributions are:

1. Inspired by the MOSSE filter, the SBCC is a new robust PIV analysis tool that employs several negative contexts, via robust surrogates and bi-directional consistency.

2. Based on MOSSE objective and bi-directional consistency formula, a concise closed-form solution to the problem is obtained. To our surprise, a set of widely-used generalized cross-correlation methods are special cases of the closed-form solution of SBCC framework.

3. The improvement of SBCC has been extensively verified on both synthetic and real PIV images.

The rest of this paper is arranged as follows. The related works are given in Section II. Section III describes our SBCC method from the problem formulation to optimization. And Section IV demonstrates the performance on synthetic datasets and real PIV images in comparison with baseline methods. Finally, the Section V remarks the paper with several concluding comments.

II Related works

(a) SCC

(b) GCC

Figure 2: Different cross-correlation methods. The $\bigotimes$ denotes the standard cross-correlation in Fourier frequency domain (a). The generalized cross-correlation methods (b), the $\bigodot$ represents an element-wise multiplication.

II.1 Cross-correlations

The cross-correlation response $r(\mathbf{x})$ of image pair $f_{1}(\mathbf{x})$, $f_{2}(\mathbf{x})$ indicates the image displacement. Due to the Convolution Theorem, the computation of cross-correlation $r(\mathbf{x})$ becomes a fast element-wise multiplication in the Fourier frequency domain. That says, $R(\mathbf{\omega})=F_{1}(\mathbf{\omega})F^{*}_{2}(\mathbf{\omega})$, where $R(\omega),F_{1}(\omega),F_{2}(\omega)$ are the Fourier transform of $r(\mathbf{x})$, $f_{1}(\mathbf{x})$, $f_{2}(\mathbf{x})$ and the superscript $(^{*})$ denotes complex conjugation. Hereafter, we will simplify the notations ($F_{1}(\omega),F_{2}(\omega),...$) as ($F_{1},F_{2},...$) by omitting the frequency $\omega$. Due to its wide adoption in PIV estimation, this vanilla cross-correlation method will be referred to as the standard cross-correlation (SCC) method, as shown in Fig. 2 (a).

$$R_{SCC}=F_{1}F^{*}_{2}$$ (1)

To enhance the correlation signal, the generalized cross-correlation (GCC) methods amend the SCC correlation $R_{SCC}$ with different spectral filters, i.e.,

$$R_{GCC}=\psi(F_{1},F_{2})F_{1}F_{2}^{*}$$ (2)

where $\psi(\cdot)$ denotes the modification operation (PHAT filter Horner and Gianino (1984), SPOF filter Wernet (2005), RPC filter Eckstein and Vlachos (2009), etc.). Compared with the $R_{PRE}$ (image pre-processing), the $R_{GCC}$ can be viewed as a post-processing of $F_{1}F_{2}^{*}$. Several GCC instances are listed in Table. 1. The filters of these GCC methods share a special type, $\psi(F_{1},F_{2})=\phi(|F_{1}F_{2}^{*}|)$, as demonstrated in Fig. 2 (b).

Table 1: Different cross-correlation methods.

Methods	Equations	Comments
SCCRaffel et al. (2018)	$R_{SCC}=F_{1}F_{2}^{*}$	-
Pre-processing	$R_{PRE}=(HF_{1})(HF_{2})^{*}$	Filter $H$
Background subs	$R_{BGS}=(F_{1}-B)(F_{2}-B)^{*}$	Background $B$
PCHorner and Gianino (1984)	$R_{PC}=\frac{F_{1}F_{2}^{*}}{|F_{1}F_{2}^{*}|}$	GCC method
SPOFWernet (2005)	$R_{SPOF}=\frac{F_{1}F_{2}^{*}}{\sqrt{|F_{1}F_{2}^{*}|}}$	GCC method
RPCEckstein, Charonko, and Vlachos (2008); Eckstein and Vlachos (2009)	$R_{RPC}=\frac{GF_{1}F_{2}^{*}}{|F_{1}F_{2}^{*}|}$	GCC method
$\rho$-CSPC Shen and Liu (2009)	$R_{CSPC}=\frac{F_{1}F_{2}^{*}}{|F_{1}F_{2}^{*}|^{\rho}+\epsilon}$	GCC method
SBCC (ours)	$R_{SBCC}=\frac{2GF_{1}F_{2}^{*}}{F_{1}F_{1}^{*}+F_{2}F_{2}^{*}+2\mu\Sigma P_{i}P_{i}^{*}}$	Contexts $P_{i}$

II.2 Correlation filter

The correlation filter can be derived either from an objective function specifically formularized in the Fourier domain Bolme et al. (2010) or from ridge regression and circulant matrices Henriques et al. (2012). Slightly different from Bolme et al. (2010); Henriques et al. (2012), we provide our understanding of correlation filter as following. Given a set of aligned template images $T_{i},i\in\{1,2,...,n\}$, the MOSSE method Bolme et al. (2010); Henriques et al. (2012) finds a surrogate filter/tracker $S$ that produces the best tracking performance, i.e., the cross-correlation response $r_{i}(\textbf{x})=\mathcal{F}^{-1}(T_{i}S^{*})$ is encouraged to be an isotropic Gaussian-shaped response $g(\mathbf{x})$, where $\mathcal{F}^{-1}(\cdot)$ denotes inverse fourier transform. In addition, a regularization term $|S|^{2}$ is employed to avoid the over fitting and gains the stability, similar to Wiener filtering or ridge regression Henriques et al. (2012). Hence, the minimum output sum of squared error (MOSSE) objective arrives,

$$\begin{split}{J}_{MOSSE}(S)=\mathop{\Sigma}_{i=1}^{n}|G-T_{i}S^{*}|^{2}+\mu|S|^{2}\end{split}$$ (3)

where $\mu$ controls the amount of regularization, and it is recommended to be $0.1$ Bolme et al. (2010). The $n$ is the number of positive templates, and $G$ denotes the Fourier transform of a Gaussian function $g(\mathbf{x})$. A closed-form solution of MOSSE,

$$\hat{S}^{*}=\frac{\Sigma_{i}GT_{i}^{*}}{\Sigma_{i}T_{i}T_{i}^{*}+\mu}$$ (4)

The $\mu$ plays an important role to make the denominator not equal to zero. Interestingly, the numerator is the correlation between the input and the desired output and the denominator is the energy spectrum of the input.

Our observation is that the regularization term of Eq.(3), $|S|^{2}=|0-1\cdot S^{*}|^{2}$, could be regarded as a special term for a negative template (delta function, $1\Leftrightarrow\delta(\mathbf{x})$). That is to say, the MOSSE filter also expects the cross-correlation between $S$ and a negative Dirac delta function to be zero. Now, it is clear that the parameter $\mu$ controls the relative importance for this negative template response.

III Surrogate-based cross-correlation

III.1 Problem formulation

As mentioned in Section I, the standard PIV pipeline might fail due to one limited background image. We thus introduce the surrogate-based cross-correlation which utilizes multiple context background images to enhance particles correlation. Specifically, a surrogate filter/image ($S_{1}$ or $S_{2}$) is assumed to have a better cross-correlation response under a well-designed surrogate objective $J_{surr}$, which considers the tracking performance as well as robustness. Meanwhile, similar to ensemble correlation, the forward correlation response $R_{f}$ (with surrogate $S_{1}$) and backward response $R_{b}$ (with surrogate $S_{2}$) are combined via correlation consistency objective ${J}_{corr}$. Thus, the robust PIV estimation problem is formulated with two objectives, as illustrated in Fig. 3,

Figure 3: The bi-directional surrogate model, SBCC. The surrogate images $S_{1},S_{2}$ and correlation response $R$ are jointly optimized with surrogate objective ${J}_{surr}$ and correlation consistency objective ${J}_{corr}$ (blue dash arrows). The $R_{f}$ and $R_{b}$ represent the correlation with forward surrogate ($S_{1}$) and backward surrogate ($S_{2}$) respectively.

$$\hat{R},\hat{S}_{1},\hat{S}_{2}=\mathop{\arg\min}_{R,S_{1},S_{2}}{J}_{SBCC}(R,S_{1},S_{2};F_{1},F_{2})$$ (5)

with

$$\begin{split}{J}_{SBCC}&(R,S_{1},S_{2};F_{1},F_{2})\\ &=\underbrace{{J}_{surr}(S_{1};F_{1})+{J}_{surr}(S_{2};F_{2})}_{\textrm{surrogate objective}}\\ &\quad+\underbrace{{J}_{corr}(R_{f},R)+{J}_{corr}(R_{b},R)}_{\textrm{correlation consistency objective}}\end{split}$$ (6)

where $R_{f}=S_{1}F_{2}^{*},R_{b}=F_{1}S_{2}^{*}$ are the forward correlation response and backward correlation response. Note that the surrogates $S_{1},S_{2}$ are no longer the processed results of image pre-processing due to the coupled SBCC structure.

Surrogate objective. To gain robustness of surrogate filter, a well-designed surrogate objective is thus constructed, which makes use of the positive template and other negative context images. The negative samples are proved to be useful for representation learning Chen, Lee, and Soh (2021). Similar to MOSSE Bolme et al. (2010), our surrogate objective ${J}_{surr}$ (Fig. 4) is given before a detail explanation,

Figure 4: The motivation of surrogate objective is to find a surrogate $S_{1}$ that does not response to the negative context images $P_{1},P_{2},...,P_{m}$.

$$\begin{split}{J}_{surr}(S;F)&=\underbrace{|G-FS^{*}|^{2}}_{\textrm{MOSSE term}}+\underbrace{\mu\frac{1}{m}\mathop{\Sigma}_{i=1}^{m}|0-P_{i}S^{*}|^{2}}_{\textrm{negative context term}}\\ &=|G-FS^{*}|^{2}+\mu\frac{1}{m}\mathop{\Sigma}_{i=1}^{m}|P_{i}S^{*}|^{2}\\ \end{split}$$ (7)

where $P_{i},i\in\{1,2,...,m\}$ are the negative context images. The $G$ denotes the Fourier transform of Gaussian functions $g(\mathbf{x})$. The $\mu$ is also the coefficient for the negative context term. The MOSSE term is preserved to encourage a Gaussian-shaped correlation map. Reducing the filter’s response to the background/noise could decrease the number of outliers, because most outliers occur when the images have similar image background or other noisy pattern. Recall that the regularization term $|S|^{2}$ of MOSSE, it encourages the surrogate filter $S$ produce zero response to the special $\delta(\mathbf{x})$. However, the negative context term of Eq. (7) encourages the filter $S$ produce zero response to all context images. We choose the backgrounds from temporal minimum value (MIN bg) and spatial low-pass results (LPF bg) as the context images in this work. Obviously, other options are also supported. Compared to the $\delta(\mathbf{x})$, the context images are more likely to have a similar noisy pattern with PIV test images.

Correlation consistency objective. Different from object tracking, the paired images of PIV can be treated as templates to each other. Our SBCC framework (Fig. 3) models it as forward and backward correlation. To obtain a consistency result, a square error encourages a minimum distance between $R_{x}\in\{R_{b},R_{f}\}$ to the final correlation map $R$.

$$\begin{split}{J}_{corr}(R_{x},R)&=|R-R_{x}|^{2}\end{split}$$ (8)

Recall $R_{f}=S_{1}F_{2}^{*},R_{b}=F_{1}S_{2}^{*}$ are the forward correlation response and backward correlation response, and $R$ is the final cross-correlation response of SBCC.

Take the surrogate objective (Eq. (7)) and correlation consistency objective (Eq. (8)) back into the problem (Eq. (6)). The specific objective function of SBCC is arrived,

$$\begin{split}{J}_{SBCC}&(R,S_{1},S_{2};F_{1},F_{2})\\ &=|G-F_{1}S_{1}^{*}|^{2}+\mu\frac{1}{m}\mathop{\Sigma}_{i=1}^{m}|P_{i}S_{1}^{*}|^{2}\\ &\quad+|G-F_{2}S_{2}^{*}|^{2}+\mu\frac{1}{m}\mathop{\Sigma}_{i=1}^{m}|P_{i}S_{2}^{*}|^{2}\\ &\quad+|R-S_{1}F_{2}^{*}|^{2}+|R-F_{1}S_{2}^{*}|^{2}\\ \end{split}$$ (9)

This objective is the sum of several squared errors with three unknown complex variables $S_{1},S_{2},R$. The parameter $\mu$ controls the relative importance of the negative context images.

III.2 Optimization of SBCC objective

The optimization of ${J}_{SBCC}$ (Eq. 9) is almost identical to the optimization problems in Bolme et al. (2010); Henriques et al. (2012). The difference is that SBCC objective is a quadratic convex function with three complex variables. The closed-form solution thus can be found by setting the partials to zeroes,

$$\begin{split}R_{SBCC}:=\hat{R}&=\frac{2GF_{1}F_{2}^{*}}{F_{1}F_{1}^{*}+F_{2}F_{2}^{*}+2\mu Q}\\ \hat{S}_{1}&=\frac{F_{2}\hat{R}+GF_{1}}{F_{1}F_{1}^{*}+F_{2}F_{2}^{*}+\mu Q}\\ \hat{S}_{2}^{*}&=\frac{F_{1}^{*}\hat{R}+GF_{2}^{*}}{F_{1}F_{1}^{*}+F_{2}F_{2}^{*}+\mu Q}\\ \end{split}$$ (10)

where $Q=\frac{1}{m}\Sigma_{i=1}^{m}P_{i}P_{i}^{*}$ is the average Fourier power spectrum of negative context images. The cross-correlation response, $R_{SBCC}$, incorporates this $Q$ component to obtain a robustness correlation by considering the noisy background in these negative context images. The terms in $R_{SBCC}($Eq. (10)) have clear interpretation. The numerator is the cross-correlation between $F_{1}$ and $F_{2}$ with a Gaussian filter ($G$), and the denominator is the power spectrum sum of $F_{1}$, $F_{2}$, and negative context images $P_{i}$. Note that the $\hat{S}_{1}$ depends on $F_{1}$ as well as $F_{2}$.

III.3 Short discussion

Due to the clear meaning of the objective function ${J}_{SBCC}$, the solution $R_{SBCC}$ has good interpretation as well. Observing the SBCC solution, we found that a set of hand-crafted GCC methods are special cases of SBCC. That is to say, our SBCC framework provides a new perspective to understand existing GCC methods (Table. 1).

Firstly, we consider a simple but useful equation,

$$F_{1}F_{1}^{*}+F_{2}F_{2}^{*}=(|F_{1}|-|F_{2}|)^{2}+2|F_{1}F_{2}^{*}|\geq 2|F_{1}F_{2}^{*}|$$ (11)

Thus, the mathematical representation of PC, RPC, 1-CSPC become special SBCC cases with an impractical noise-free assumption ($|F_{1}|=|F_{2}|$). That says,

$$\begin{split}R_{PC}\ \ &=R_{SBCC}|_{G=1,|F_{1}|=|F_{2}|,\mu=0}\\ R_{1-CSPC}&=R_{SBCC}|_{G=1,|F_{1}|=|F_{2}|,\mu=\epsilon,P_{i}=1}\\ R_{RPC}&=R_{SBCC}|_{G\Leftrightarrow g(x),|F_{1}|=|F_{2}|,\mu=0}\\ \end{split}$$ (12)

which means that, PC and 1-CSPC encourage a Dirac delta response $(G=1)$, while RPC method expects a Gaussian-shaped response. Only 1-CSPC method has considered a special context image $\delta(\mathbf{x})$ implicitly. However, all methods employ a noise-free assumption ($|F_{1}|=|F_{2}|$).

The SPOF is beyond a brief understanding. However, the SPOF can be treated as an ensemble correlation Delnoij et al. (1999) due to the following relationship, $R_{SPOF}R_{SPOF}=R_{SCC}R_{PC}$. It might imply that multiple SBCC frameworks can provide more complex surrogate for SPOF and general $\rho-$CSPC methods. Note that none of the existing GCC methods explicitly take any negative context images into consideration.

IV Experiments

In this section, the performance of SBCC is extensively evaluated through visualizing the correlation map, analysing parameter sensitivity, conducting measurement on synthetic images and experimental PIV images. The detailed implementation and additional results are provided at the project repository, https://github.com/yongleex/SBCC.

Baseline methods. Several widely-accepted approaches are adopted to conduct a fair evaluation. And they are, standard cross-correlation (SCC) Raffel et al. (2018), symmetric phase-only filter (SPOF) Wernet (2005), robust phase correlation (RPC) Eckstein and Vlachos (2009). Regarding the background subtraction methods, we choose the minimum intensity image from double-frame PIV image Honkanen and Nobach (2005) and the spatial low-pass filter (LPF) Adrian and Westerweel (2011), resulting in SCC-MIN and SCC-LPF. To exclude the influence of other factors, single-pass cross-correlation without any post-processing is utilized for all testing methods.

Evaluation criteria. In addition to subjective visual judgement, three objective metrics are also employed to quantify the performance: 1) the root mean-square-error (RMSE) Raffel et al. (2018); Lee, Yang, and Yin (2017b); Lee and Mei (2022), 2) the average endpoint error (AEE) Lagemann et al. (2021), 3) the execution time for different image size.

$$\begin{split}RMSE&=\sqrt{\frac{1}{N}\mathop{\Sigma}_{i=1}^{N}\|\mathbf{v}_{e,i}-\mathbf{v}_{t,i}\|^{2}}\\ AEE&=\frac{1}{N}\mathop{\Sigma}_{i=1}^{N}\|\mathbf{v}_{e,i}-\mathbf{v}_{t,i}\|\end{split}$$ (13)

where $\mathbf{v}_{e,i}=(v_{x},v_{y})$ is the $i^{th}$ estimated vector out of $N$ points, while the $\mathbf{v}_{t,i}$ denotes the $i^{th}$ ground truth.

IV.1 On correlation coefficients

(a) A test image pair $(u=-5.0pixel)$ and estimated backgrounds

(b) The correlation maps

(c) The correlation coefficient at $\Delta y=0$

Figure 5: A correlation responses on synthetic particle images with background. Best viewed in color.

Fig. 5 (a) gives a test pair of synthetic PIV interrogation window (particle displacement is $-5.0pixel$ in horizontal), with a unrealistic strong additive background. Obviously, the minimum intensity image (MIN bg) recovers the still background, while the LPF (LPF bg) can not tell the background and particle image apart due to frequency aliasing.

Fig. 5 (b) and (c) provide the cross-correlation coefficients for this challenging synthetic case. The SCC, SPOF, RPC fail to obtain the correct response peak due to a lack of noise signal. Although the SCC-LPF method does not show the correct peak, the rough background also helps to increase the image similarity at the correct displacement. Without surprising, the SCC-MIN method is demonstrated with perfect correlation peak with good background estimation in this case. Compared to other methods, the SBCC has a correlation map with two distinct peaks, and correlation peak (maximum similarity) is correctly located at $(-5.0,0.0)$. Closing observing the curve around $(-5.0,0.0)$, the SBCC has a similar landscape with SCC-MIN. Thus, we can conclude that SBCC provides another feasible mechanism to perform robust cross-correlation with the help of background signals.

IV.2 On parameter sensitivity

The only parameter $\mu$ needs to be determined for our SBCC method working at the best condition. Similar to the background subtraction, it’s very difficult to obtain ideal context images. Hence, increasing $\mu$ will not always benefit the robustness or accuracy. We thus argue that there is an optimal parameter $\mu$ for the practical PIV measurement.

Figure 6: Synthetic particle images with/without different backgrounds (uniform flow).

To study the effect of different $\mu$ values, we also employ the synthetic particle images from particle image generator (PIG). Here, we use a subset of a ready dataset (1000 uniform flow) Cai et al. (2019b, a). Different from the noise-free images, a random synthetic image is added to the clean synthetic image pair, to simulate the still background. The background images are from a bubble dataset (1219 images of class 1) Bai et al. (2021); Park et al. (2021), synthetic sinusoidal and square signal, as demonstrated in Fig. 6.

(a) RMSE for a background-free image pair

(b) RMSE for an image pair with bubble background

Figure 7: Effect of the parameter $\mu$ of SBCC. ($u=0.85pixel,v=1.23pixel$)

Fig. 7 illustrates the experimental results with both background- free and bubble background image pair (top row of Fig. 6) respectively, and the parameter $\mu$ varies from 0 to 16 with interval $0.1$. In comparison with noise-free case, the RMSE of all methods are increased when additive bubble noise is added. It reflects the PIV challenge caused by the unwanted background. Interestingly, the sensitivity curve of SBCC have an optimal value that corresponds with the minimum RMSE value. The optimal value $\mu$ of complex background is larger than that of background-free situation. Meanwhile, an improvement of SBCC happens for a large range of $\mu$, i.e., range $[0,6]$ for background-free and $[1,16]$ for bubble background. Thus, setting a proper $\mu$ could be an interesting problem for future work. Anyway, we set $\mu$ to $3.0$ arbitrarily by taking all cases into consideration based on the results of this experiment. Note that, the fixed $\mu(3.0)$ is not changed for different measurement cases, and the extensive results illustrate that this value could have a universal robust performance.

IV.3 On synthetic PIV images

Table 2: Synthetic experiment on one image pair. Performance measured by RMSE. The best in Bold.

Background	Free	Bubble	Sinusoidal	Square
SCC	0.3753	0.5054	0.7930	0.4644
SCC-MIN	0.4422	0.4422	0.4422	0.4422
SCC-LPF	0.3919	0.3487	0.3934	0.3244
SPOF	0.4060	0.4530	0.4725	0.4382
RPC	0.1883	0.6597	0.3724	0.2274
SBCC	0.2195	0.2765	0.1914	0.1985

Table 3: Synthetic experiment on one image pair. Performance measured by AEE. The best in Bold.

Background	Free	Bubble	Sinusoidal	Square
SCC	0.3422	0.4557	0.7813	0.4311
SCC-MIN	0.3827	0.3827	0.3827	0.3827
SCC-LPF	0.3438	0.3213	0.3721	0.3054
SPOF	0.3768	0.4181	0.4460	0.4099
RPC	0.1619	0.4193	0.3440	0.1983
SBCC	0.2050	0.2457	0.1741	0.1810

To quantitatively compare the performance, a synthetic image pair is sampled Cai et al. (2019a) with different backgrounds (Fig. 6). Table. 2 and 3 give the RMSE and AEE values for the processed results. For the background-free scenario, vanilla SCC yields satisfactory result, while the variations (SCC-MIN, SCC-LPF and SPOF) do not show a consistent improvement in accuracy. For the cases with background, all variants have a better performance than SCC. It implies that background subtraction (SCC-MIN, SCC-LPF) and spectral filters (SPOF, RPC) are all effective to address background problem. In this experiment, the RPC has the smallest measurement error on background-free images, while SBCC achieves significant improvement for the cases with background.

Figure 8: Boxplot of the RMSE for the synthetic 1000 uniform cases with 4 types of backgrounds and 6 PIV CC estimators.

Figure 9: Boxplot of the AEE for the synthetic 1000 uniform cases with 4 types of backgrounds and 6 PIV CC estimators.

In addition to case study, the Monte Carlo simulation is widely adopted in the assessment of PIV measurement uncertainty Raffel et al. (2018); Lee, Yang, and Yin (2017b). Recall that, the 1000 synthetic particle image pairs ($256\times 256$ pixel²) are synthesized with uniform flows ground truth Cai et al. (2019b), and the backgrounds are sampled from a bubble dataset Park et al. (2021) and two artificial signals (random sinusoidal wave and square wave). The boxplots in Fig. 8 and 9 present the statistical results of 1000 cases measured by RMSE and AEE. The one-pass SCC and RPC method have acceptable measurement error (RMSE $\sim 0.2$) for background-free cases due to varying seeding density, particle diameters, illuminations in the image generator. However, they are not robust enough to backgrounds. Both SCC-MIN and SPOF have poor performance on all cases. The reasons might be the low-quality background of SCC-MIN, while peak-locking causes a significant error of SPOF. On the contrary, both SCC-LPF and SBCC performs well for all test cases. We speculate that the synthetic backgrounds might be well estimated by a LPF, resulting in the good performance of SCC-LPF. Note that, our SBCC is more accurate than SCC-LPF statistically.

IV.4 On real recorded PIV images

(a) A lab-made PIV setup with an industrial camera (HIKROBOT, MV-CA016-10UM) and multiple semiconductor lasers (532$nm$)

(b) Left: interactive flow around L-shaped plate Liberzon et al. (2016); Middle: hypersonic flow over a step model Lu (2023); Right: rotational flow with background from (a).

Figure 10: A lab-made setup (a) and 3 practical test cases (b).

Figure 11: Vectors from the interactive flow. The color background corresponds to velocity magnitude. Best viewed in color.

Figure 12: Vectors from the hypersonic flow. The color background corresponds to velocity magnitude. Best viewed in color.

Figure 13: Vectors from the rotational flow. The color background corresponds to velocity magnitude. Best viewed in color.

In addition to synthetic images, we also tested SBCC on three challenging recorded PIV image pairs (Fig. 10 (b)). The first case records an interactive flow around L-shaped plate in OpenPIV Liberzon et al. (2016). The second PIV case records a hypersonic flow ($5Ma$) over a step model Lu (2023). The third image pair is from a lab-made PIV setup (Fig. 10 (a)), which represents a liquid column rotating with a still text leaflet background (similar to PIV Challenge 2014, Case F). These flows have particular structures with large displacement ($\approx 10$ pixel), and the images have non-uniform illumination, out-of-plane effects and noise background. Thus, they are considered as the challenging test examples.

Fig. 11, 12 and 13 provide the raw vector field results, computed with different one-pass cross-correlation methods without any post-processing operation. The pseudo-color backgrounds represent the vector magnitudes. Overall, different methods output similar flow patterns, verifying that these widely-used methods do work in practice. Unfortunately, the accuracy for each method can not be exactly assessed due to unknown ground truth. Thus, we visually check the outliers for the problematic areas in white boxes to evaluate the robustness of each method. The left area of interactive flow image is under weak illumination, and thus full of random noise. The results indicate the spectral filters (SPOF, RPC and SBCC) can effectively cope with this problem, as reported in related works Eckstein and Vlachos (2009). The middle box of hypersonic flow, with strong out-of-plane movement, is full of uncorrelated particles. We argue that the MIN background could catch some content of the noise signal, resulting in the less outliers of SCC-MIN and SBCC. The complex background of rotational flow is obviously difficult to reconstruct with MIN or LPF approaches, which explains the bad performance of SCC-MIN and SCC-LPF. Besides, the spectral filter of RPC also does not work well for this case. However, combining multiple background contexts and spectral filter, our SBCC has an impressive performance with a few outliers for this strong background of advertising words. Based on the impressive results of comprehensive (synthetic and recorded) experiments, the overall effectiveness of our SBCC has been confirmed.

IV.5 On computing cost

Table 4: The running time (seconds) (averaged over 10 runs, with standard deviations).

Image size	$128\times 128$	$256\times 256$	$512\times 512$	$1014\times 1024$
SCC	$0.0022\pm 0.0002$	$0.0055\pm 0.0009$	$0.0233\pm 0.0019$	$0.0792\pm 0.0031$
SCC-MIN	$0.0020\pm 0.0005$	$0.0055\pm 0.0007$	$0.0220\pm 0.0009$	$0.0785\pm 0.0026$
SCC-LPF	$0.0070\pm 0.0127$	$0.0081\pm 0.0008$	$0.0277\pm 0.0006$	$0.0838\pm 0.0030$
SPOF	$0.0019\pm 0.0001$	$0.0071\pm 0.0004$	$0.0232\pm 0.0005$	$0.0834\pm 0.0032$
RPC	$0.0022\pm 0.0002$	$0.0067\pm 0.0002$	$0.0232\pm 0.0007$	$0.0822\pm 0.0024$
SBCC	$0.0070\pm 0.0034$	$0.0187\pm 0.0004$	$0.0576\pm 0.0034$	$0.1995\pm 0.0315$

To demonstrate the efficiency of proposed SBCC, the baseline methods and SBCC were finally tested using Python 3.8 on a 2.70GHz i5-11400H laptop computer (HP OMEN 16) with RAM 16.00GB. Tested over 10 runs, the execution results (Table. 4) with varied image size demonstrated that the SBCC has an acceptable computational cost. Recall that an extra FFT operation of contexts is needed, and it is worthwhile to cost $2\sim 3$ times computation for the accuracy improvement. For a $1024\times 1024$ case, the execution time of our SBCC is less than $0.2$ second.

V Conclusion

Inspired by correlation filter, a novel SBCC framework is proposed to enhance the cross-correlation performance by incorporating multiple negative context images. As a multivariate operator, the SBCC is the closed-form solution of a well-designed optimization problem, which is formulated with both surrogate objective and correlation consistency objective. To our surprising, this framework also provides an alternative surrogate view for a set of generalized cross-correlation methods (PC, RPC, 1-CSPC, SPOF, etc). On the correlation response, the SBCC method is more likely to achieve a desired Gaussian-shaped correlation response, encouraged by the objective function. And an arbitrary parameter $\mu(3.0)$ is fixed through parameter sensitivity analysis. Finally, the performance improvement of SBCC is verified with massive synthetic and real PIV image pairs. An interesting point is that SBCC paves a new way for robust PIV cross correlation analysis via employing negative context images. Moving forward, we plan to apply SBCC beyond PIV to other tasks including one-dimensional time series, digital image correlation, acoustic imaging, etc.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.