Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation

Gneiting et al. (2007) proposed a paradigm for the evaluation of probabilistic forecasts: "maximizing the sharpness of the predictive distributions subject to calibration". Calibration is the statistical compatibility between the forecast and the observations. Sharpness is the concentration of the forecast and is a property of the forecast itself. In other words, the paradigm aims at minimizing the uncertainty of the forecast given that the forecast is statistically consistent with the observations. Tsyplakov (2011) states that the notion of calibration in the paradigm is too vague but it holds if the definition of calibration is refined. This principle for the evaluation of probabilistic forecasts has reached a consensus in the field of probabilistic forecasting (see, e.g., Gneiting and Katzfuss 2014; Thorarinsdottir and Schuhen 2018). The paradigm proposed in Gneiting et al. (2007) is not the first mention of the link between sharpness and calibration: for example, Murphy and Winkler (1987) mentioned the relation between refinement (i.e., sharpness) and calibration.

For univariate forecasts, multiple definitions of calibration are available depending on the setting. The most used definition is probabilistic calibration and, broadly speaking, consists of computing the rank of observations among samples of the forecast and checking for uniformity with respect to observations. If the forecast is calibrated, observations should not be distinguishable from forecast samples, and thus, the distribution of their ranks should be uniform. Probabilistic calibration can be assessed by probability integral transform (PIT) histograms (Dawid, 1984) or rank histograms (Anderson, 1996; Talagrand et al., 1997) for ensemble forecasts when observations are stationary (i.e., their distribution is the same across time). The shape of the PIT or rank histogram gives information about the type of (potential) miscalibration: a triangular-shaped histogram suggests that the probabilistic forecast has a systematic bias, a $\cup$-shaped histogram suggests that the probabilistic forecast is under-dispersed and a $\cap$-shaped histogram suggests that the probabilistic forecast is over-dispersed. Moreover, probabilistic calibration implies that rank histograms should be uniform but uniformity is not sufficient. For example, rank histograms should also be uniform conditionally on different forecast scenarios (e.g., conditionally on the value of the observations available when the forecast is issued). Additionally, under certain hypotheses, calibration tools have been developed to consider real-world limitations such as serial dependence (Bröcker and Ben Bouallègue, 2020). Statistical tests have been developed to check the uniformity of rank histograms (Jolliffe and Primo, 2008). Readers interested in a more in-depth understanding of univariate forecast calibration are encouraged to consult Tsyplakov (2013, 2020).

For multivariate forecasts, a popular approach relies on a similar principle: first, multivariate forecast samples are transformed into univariate quantities using so-called pre-rank functions and then the calibration is assessed by techniques used in the univariate case (see, e.g., Gneiting et al. 2008). Pre-rank functions may be interpretable and allow targeting the calibration of specific aspects of the forecast such as the dependence structure. Readers interested in the calibration of multivariate forecasts can refer to Allen et al. (2024) for a comprehensive review of multivariate calibration.

A scoring rule $\mathrm{S}$ assigns a real-valued quantity $\mathrm{S}(F,y)$ to a forecast-observation pair $(F,y)$, where $F\in\mathcal{F}$ is a probabilistic forecast and $\bm{y}\in\mathbb{R}^{d}$ is an observation. In the negative-oriented convention, a scoring rule $\mathrm{S}$ is proper relative to the class $\mathcal{F}$ if

for all $F,G\in\mathcal{F}$, where $\mathbb{E}_{G}[\cdots]$ is the expectation with respect to $\bm{Y}\sim G$. In simple terms, a scoring rule is proper relative to a class of distribution if its expected value is minimal when the true distribution is predicted, for any distribution within the class. Forecasts minimizing the expected scoring rule are said to be efficient and the other forecasts are said to be sub-efficient. Moreover, the scoring rule $\mathrm{S}$ is strictly proper relative to the class $\mathcal{F}$ if the equality in (1) holds if and only if $F=G$. This ensures the characterization of the ideal forecast (i.e., there is a unique efficient forecast and it is the true distribution). Moreover, proper scoring rules are powerful tools as they allow the assessment of calibration and sharpness simultaneously (Winkler, 1977; Winkler et al., 1996). Sharpness can be assessed individually using the entropy associated with proper scoring rules, defined by $e_{\mathrm{S}}(F)=\mathbb{E}_{F}[\mathrm{S}(F,\bm{Y})]$. The sharper the forecast, the smaller its entropy. Strictly proper scoring rules can also be used to infer the parameters of a parametric probabilistic forecast (see, e.g., Gneiting et al. 2005; Pacchiardi et al. 2024).

Regardless of all the interesting properties of proper scoring rules, it is worth noting that they have some limitations. Proper scoring rules may have multiple efficient forecasts (i.e., associated with their minimal expected value) and, in the general setting, no guarantee is given on their relevance. Moreover, strict propriety ensures that the efficient forecast is unique and that it is the ideal forecast (i.e., the true distribution), however, no guarantee is available for forecasts within the vicinity of the minimum in the general case. This is particularly problematic since, in practice, the unavailability of the ideal distribution makes it impossible to know if the minimum expected score is achieved. In the case of calibrated forecasts, the expected scoring rule is the entropy of the forecast and the ranking of forecasts is thus linked to the information carried by the forecast (see Corollary 4, Holzmann and Eulert 2014 for the complete result). These limitations may explain the plurality of scoring rules depending on application fields.

We recall classical univariate scoring rules to explain key concepts. Some univariate scoring rules will be useful for the multivariate scoring rules construction framework proposed in Section 3. Let $\mathcal{P}(E)$ denote the class of Borel probability measures on $E$. We consider $F\in\mathcal{F}\subseteq\mathcal{P}(\mathbb{R})$ a probabilistic forecast in the form of its cumulative distribution function (cdf) and $y\in\mathbb{R}$ an observation. When the probabilistic forecast $F$ has a probability density function (pdf), it will be denoted $f$.

The simplest scoring rules can be derived from scoring functions used to assess point forecasts. The squared error (SE) is the most popular and is known through its averaged value (the mean squared error; MSE) or the square root of its average (the root mean squared error; RMSE) which has the advantage of being expressed in the same units as the observations. As a scoring rule, the SE is expressed as

where $\mu_{F}$ denotes the mean of the predicted distribution $F$. The SE solely discriminates the mean of the forecast (see Appendix A); efficient forecasts for SE are the ones matching the mean of the true distribution. The SE is proper relative to $\mathcal{P}_{2}(\mathbb{R})$, the class of Borel probability measures on $\mathbb{R}$ with a finite second moment (i.e., finite variance). Note that the SE cannot be strictly proper as the equality of mean does not imply the equality of distributions.

Another well-known scoring rule is the absolute error (AE) defined by

where $\mathrm{med}(F)$ is the median of the predicted distribution $F$. The mean absolute error (MAE), the average of the absolute error, is the most seen form of the AE and it is also expressed in the same units as the observations. Efficient forecasts are forecasts that have a median equal to the median of the true distribution. The AE is proper relative to $\mathcal{P}_{1}(\mathbb{R})$ but not strictly proper. Similarly, the quantile score (QS), also known as the pinball loss, is a scoring rule focusing on quantiles of level $\alpha$ defined by

where $0<\alpha<1$ is a probability level and $F^{-1}(\alpha)$ is the predicted quantile of level $\alpha$. The case $\alpha=0.5$ corresponds to the AE up to a factor $2$. The QS of level $\alpha$ is proper relative to $\mathcal{P}_{1}(\mathbb{R})$ but not strictly proper since efficient forecasts are ones correctly predicting the quantile of level $\alpha$ (see, e.g., Friederichs and Hense 2008).

Another summary statistic of interest is the exceedance of a threshold $t\in\mathbb{R}$. The Brier score (BS; Brier 1950) was initially introduced for binary predictions but allows also to discriminate forecasts based on the exceedance of a threshold $t$. For probabilistic forecasts, the BS is defined as

where $1-F(t)$ is the predicted probability that the threshold $t$ is exceeded. The BS is proper relative to $\mathcal{P}(\mathbb{R})$ but not strictly proper. Binary events (e.g., exceedance of thresholds) are relevant in weather forecasting as they are used, for example, in operational settings for decision-making.

All the scoring rules presented above are proper but not strictly proper since they only discriminate against specific summary statistics instead of the whole distribution. Nonetheless, they are still used as they allow forecasters to verify specific characteristics of the forecast: the mean, the median, the quantile of level $\alpha$ or the exceedance of a threshold $t$. The simplicity of these scoring rules makes them interpretable, thus making them essential verification tools.

Some univariate scoring rules contain a summary statistic: for example, the formulas of the QS (4) or the BS (5) contain the exceedance of a threshold $t$ and the quantile of level $\alpha$, respectively. They can be seen as a scoring function applied to a summary statistic. This duality can be understood through the link between scoring functions and scoring rules through consistent functionals as presented in Gneiting (2011) or Section 2.2 in Lerch et al. (2017).

Other summary statistics can be of interest depending on applications. Nonetheless, it is worth noting that mispecifications of numerous summary statistics cannot be discriminated because of their non-elicitability. Non-elicitability of a transformation implies that no proper scoring rule can be constructed such that efficient forecasts are forecasts where the transformation is equal to the one of the true distribution. For example, the variance is known to be non-elicitable; however, it is jointly elicitable with the mean (see, e.g., Brehmer 2017). Readers interested in details regarding elicitable, non-elicitable and jointly elicitable transformations may refer to Gneiting (2011), Brehmer and Strokorb (2019) and references therein.

A strictly proper scoring rule should discriminate the whole distribution and not only specific summary statistics. The continuous ranked probability score (CRPS; Matheson and Winkler 1976) is the most popular univariate scoring rule in weather forecasting applications and can be expressed by the following expressions

where $y\in\mathbb{R}$ and $X$ and $X^{\prime}$ are independent random variables following $F$, with a finite first moment. Equations (7) and (8) show that the CRPS is linked with the BS and the QS. Broadly speaking, as the QS discriminates a quantile associated with a specific level, integrating the QS across all levels discriminates the quantile function that fully characterizes univariate distributions. Similarly, integrating the BS across all thresholds discriminates the cumulative distribution function that also fully characterizes univariate distributions. The CRPS is a strictly proper scoring rule relative to $\mathcal{P}_{1}(\mathbb{R})$, the class of Borel probability measures on $\mathbb{R}$ with a finite first moment. In addition, Equation (6) indicates the CRPS values have the same units as observations. In the case of deterministic forecasts, the CRPS reduces to the absolute error, in its scoring function form (Hersbach, 2000). The use of the CRPS for ensemble forecast is straightforward using expectations as in (6). Ferro et al. (2008) and Zamo and Naveau (2017) studied estimators of the CRPS for ensemble forecasts.

In addition to scoring rules based on scoring functions, some scoring rules use the moments of the probabilistic forecast $F$. The SE (2) depends on the forecast only through its mean $\mu_{F}$. The Dawid-Sebastiani score (DSS; Dawid and Sebastiani 1999) is a scoring rule depending on the forecast $F$ only through its first two central moments. The DSS is expressed as

where $\mu_{F}$ and ${\sigma_{F}}^{2}$ are the mean and the variance of the distribution $F$. The DSS is proper relative to $\mathcal{P}_{2}(\mathbb{R})$ but not strictly proper, since efficient forecasts only need to correctly predict the first two central moments (see Appendix A). Dawid and Sebastiani (1999) proposed a more general class of proper scoring rules but the DSS, as defined in (9), can be seen as a special case of the logarithmic score (up to an additive constant), introduced further down.

Another scoring rule relying on the central moments of the probabilistic forecast $F$ up to order three is the error-spread score (ESS; Christensen et al. 2014). The ESS is defined as

where $\mu_{F}$, $\sigma_{F}^{2}$ and $\gamma_{F}$ are the mean, the variance and the skewness of the probabilistic forecast $F$. The ESS is proper relative to $\mathcal{P}_{4}(\mathbb{R})$. As for the other scoring rules only based on moments of the forecast presented above, the expected ESS compares the probabilistic forecast $F$ with the true distribution only via their four first moments (see Appendix A). Scoring rules based on central moments of higher order could be built following the process described in Christensen et al. (2014). Such scoring rules would benefit from the interpretability induced by their construction and the ease to be applied to ensemble forecasts. However, they would also inherit the limitation of being only proper.

When the probabilistic forecast $F$ has a pdf $f$, scoring rules of a different type can be defined. Let $\mathcal{L}_{\alpha}(\mathbb{R})$ denote the class of probability measures on $\mathbb{R}$ that are absolutely continuous with respect to $\mu$ (usually taken as the Lebesgue measure) and have $\mu$-density $f$ such that

The most popular scoring rule based on the pdf is the logarithmic score (also known as ignorance score; Good 1952; Roulston and Smith 2002). The logarithmic score is defined as

for $y$ such that $f(y)>0$. In its formulation, the logarithmic score is different from the scoring rules seen previously. Good (1952) proposed the logarithmic score knowing its link with the theory of information: its entropy is the Shannon entropy (Shannon, 1948) and its expectation is related to the Kullback-Leibler divergence (Kullback and Leibler, 1951) (see Appendix A). The logarithmic score is strictly proper relative to the class $\mathcal{L}_{1}(\mathbb{R})$. Moreover, inference via minimization of the expected logarithmic score is equivalent to maximum likelihood estimation (see, e.g., Dawid et al. 2015). The logarithmic score belongs to the family of local scoring rules, which are scoring rules only depending on $y$, $f(y)$ and its derivatives up to a finite order. Another local scoring rule is the Hyvärinen score (also known as the gradient scoring rule; Hyvärinen 2005) and it is defined as

for $y$ such that $f(y)>0$. The Hyvärinen score is proper relative to the subclass of $\mathcal{P}(\mathbb{R})$ such that the density $f$ exists, is twice continuously differentiable and satisfies $f^{\prime}(x)/f(x)\to 0$ as $|x|\to\infty$. It is worth noticing that the Hyvärinen score can be computed even if $f$ is only known up to a scale factor (e.g., up to a normalizing constant). This property allows circumventing the use of Monte Carlo methods or approximations of the normalizing constant when it is unavailable or hard to compute. This is a property of local proper scoring rules except for the logarithmic score (Parry et al., 2012). Readers eager to learn more about local proper scoring rules may refer to Parry et al. (2012) and Ehm and Gneiting (2012).

The logarithmic score and the Hyvärinen score do not allow $f$ to be zero. To overcome this limitation, scoring rules expressed in terms of the $L_{\alpha}$-norm have been proposed. The quadratic score is defined as

where $\lVert f\rVert^{2}_{2}=\int_{\mathbb{R}}f(y)^{2}\mathrm{d}y$. The quadratic score is strictly proper relative to the class $\mathcal{L}_{2}(\mathbb{R})$.

The pseudospherical score is defined as

with $\alpha>1$. For $\alpha=2$, it reduces to the spherical score (see, e.g., Jose 2007). The pseudospherical score is strictly proper relative to the class $\mathcal{L}_{\alpha}(\mathbb{R})$. The four scoring rules presented above have been criticized as they do not encourage a high probability in the vicinity of the observation $y$ (Gneiting and Raftery, 2007). In particular, as the logarithmic score is more sensitive to outliers, probabilistic forecasts inferred by its minimization may be overdispersive (Gneiting et al., 2005). Moreover, the pdf is not always available, for example in the case of ensemble forecasts.

Readers may refer to the various reviews of scoring rules available (see, e.g., Bröcker and Smith 2007; Gneiting and Raftery 2007; Gneiting and Katzfuss 2014; Thorarinsdottir and Schuhen 2018; Alexander et al. 2022). Formulas of the expected scoring rules presented are available in Appendix A.

Strictly proper scoring rules can be seen as more powerful than proper scoring rules. This is theoretically true when the interest is in identifying the ideal forecast (i.e., the true distribution). Regardless, in practice, scoring rules are also used to rank probabilistic forecasts and with that in mind, a given ranking of forecasts in terms of the expectation of a strictly proper scoring rule (such as the CRPS) is harder to interpret than a ranking in terms of the expectation of a proper but more interpretable scoring rule (such as the SE). The SE is known to discriminate the mean, and thus, a better rank in terms of expected SE implies a better prediction of the mean of the true distribution. Conversely, a better ranking in terms of CRPS implies a better prediction of the whole prediction, but it might not be useful as is, and other verification tools will be needed to know what caused this ranking. When forecasts are not calibrated, there seems to be a trade-off between interpretability and discriminatory power and this becomes more prominent in a multivariate setting. However, simpler interpretable tools and discriminatory-powerful tools can be used complementarily. The framework proposed in Section 3 aims at hel** the construction of interpretable proper scoring rules.

In a multivariate setting, forecasters cannot solely use univariate scoring rules as they are not able to discriminate forecasts beyond their $1$-dimensional marginals. Univariate scoring rules cannot discriminate the dependence structure between the univariate margins. Multivariate forecasts can be applied in different setups: spatial forecasts, temporal forecasts, multivariable forecasts or any combination of these categories (e.g., spatio-temporal forecasts of multiple variables). Considering weather forecasting, a spatial forecast could aim at predicting temperatures across multiple locations. A temporal forecast could be focused on predicting rainfall at multiple lead times at a given location. A multivariable forecast could predict both eastward and northward components of the wind. In the following, we consider $F\in\mathcal{F}\subseteq\mathcal{P}(\mathbb{R}^{d})$ a multivariate probabilistic forecast and $\bm{y}\in\mathbb{R}^{d}$ an observation.

Even if there is no natural ordering in the multivariate case, the notions of median and quantile can be adapted using level sets, and then scoring rules using these quantities can be constructed (see, e.g., Meng et al. 2023). Nonetheless, as the mean is well-defined, the squared error (SE) can be defined in the multivariate setting :

where $\mu_{F}$ is the mean vector of the distribution $F$. Similar to the univariate case, the SE is proper relative to $\mathcal{P}_{2}(\mathbb{R}^{d})$. Moments are well-defined in the multivariate case allowing the multivariate version of the Dawid-Sebastiani score to be defined. The Dawid-Sebastiani score (DSS) was proposed in Dawid and Sebastiani (1999) as

where $\bm{\mu}_{F}$ and $\Sigma_{F}$ are the mean vector and the covariance matrix of the distribution $F$. The DSS is proper relative to $\mathcal{P}_{2}(\mathbb{R}^{d})$ and it becomes strictly proper relative to any convex class of probability measures characterized by their first two moments (Gneiting and Raftery, 2007). The second term in the DSS is the squared Mahalanobis distance between $\bm{y}$ and $\bm{\mu}_{F}$.

To define a strictly proper scoring rule for multivariate forecast, Gneiting and Raftery (2007) proposed the energy score (ES) as a generalization of the CRPS to the multivariate case. The ES is defined by

where $\alpha\in(0,2)$ and $F\in\mathcal{P}_{\alpha}(\mathbb{R}^{d})$, the class of Borel probability measures on $\mathbb{R}^{d}$ such that the moment of order $\alpha$ is finite. The definition of the ES is related to the kernel form of the CRPS (6), to which the ES reduces for $d=1$ and $\alpha=1$. As pointed out in Gneiting and Raftery (2007), in the limiting case $\alpha=2$, the ES becomes the SE (12). The ES is strictly proper relative to $\mathcal{P}_{\alpha}(\mathbb{R}^{d})$ (Székely, 2003; Gneiting and Raftery, 2007) and is suited for ensemble forecasts (Gneiting et al., 2008). Moreover, the parameter $\alpha$ gives some flexibility: a small value of $\alpha$ can be chosen and still lead to a strictly proper scoring rule, for example, when higher-order moments are ill-defined. The discrimination ability of the ES has been studied in numerous studies (see, e.g., Pinson and Girard 2012; Pinson and Tastu 2013; Scheuerer and Hamill 2015). Pinson and Girard (2012) studied the ability of the ES to discriminate among rival sets of scenarios (i.e., forecasts) of wind power generation. In the case of bivariate Gaussian processes, Pinson and Tastu (2013) illustrated that the ES appears to be more sensitive to misspecifications of the mean rather than misspecifications of the variance or dependence structure. The lack of sensitivity to misspecifications of the dependence structure has been confirmed in Scheuerer and Hamill (2015) using multivariate Gaussian random vectors of higher dimension. Moreover, the discriminatory power of the ES deteriorates in higher dimensions (Pinson and Tastu, 2013).

To overcome the discriminatory limitation of the ES, Scheuerer and Hamill (2015) proposed the variogram score ($\mathrm{VS}$), a score targeting the verification of the dependence structure. The VS of order $p$ is defined as

where $X_{i}$ is the $i$-th component of the random vector $X$ following $F$, $w_{ij}$ are nonnegative weights and $p>0$. The variogram score capitalizes on the variogram, used in spatial statistics to access the dependence structure. The VS cannot detect an equal bias across all components. The VS of order $p$ is proper relative to the class of Borel probability measures on $\mathbb{R}^{d}$ such that the $2p$-th moments of all univariate margins are finite. The weights $w_{ij}$ can be selected to emphasize or depreciate certain pair interactions. For example, in a spatial context, it can be expected the dependence between pairs decays with the distance: choosing the weights proportional to the inverse of the distance between locations can increase the signal-to-noise ratio and improve the discriminatory power of the VS (Scheuerer and Hamill, 2015).

When the pdf $f$ of the probabilistic forecast $F$ is available, multivariate versions of the univariate scoring rules based on the pdf are available. The multivariate versions of the scoring rules have the same properties and limitations as their univariate counterpart. The logarithmic score (11) has a natural multivariate version :

for $\bm{y}$ such that $f(\bm{y})>0$. The logarithmic score is strictly proper relative to the class $\mathcal{L}_{1}(\mathbb{R}^{d})$.

The Hyvärinen score (HS; Hyvärinen 2005) was initially proposed in its multivariate form

for $\bm{y}$ such that $f(\bm{y})>0$, where $\Delta$ is the Laplace operator (i.e., the sum of the second-order partial derivatives) and $\nabla$ is the gradient operator (i.e., vector of the first-order partial derivatives). In the multivariate setting, the HS can also be computed if the predicted pdf is known up to a normalizing constant. The HS is proper relative to the subclass of $\mathcal{P}(\mathbb{R}^{d})$ such that the density $f$ exists, is twice continuously differentiable and satisfies $\lVert\nabla\log(f(x))\rVert\to 0$ as $\lVert x\rVert\to\infty$.

The quadratic score and pseudospherical score are directly suited to the multivariate setting :

where $\lVert f\rVert_{\alpha}=(\int_{\mathbb{R}^{d}}f(\bm{y})^{\alpha}\mathrm{d}\bm{y})^{1/\alpha}$. The quadratic score is strictly proper relative to the class $\mathcal{L}_{2}(\mathbb{R}^{d})$. The pseudospherical score is strictly proper relative to the class $\mathcal{L}_{\alpha}(\mathbb{R}^{d})$.

Additionally, other multivariate scoring rules have been proposed among which the marginal-copula score (Ziel and Berk, 2019) or wavelet-based scoring rules (see, e.g., Buschow et al. 2019). These scoring rules will be briefly mentioned in Section 4 in light of the proper scoring rule construction framework proposed in this article. Appendix B provides formulas for the expected multivariate scoring rules presented above.

Spatial forecasts are a very important group of multivariate forecasts as they are involved in various applications (e.g., weather or renewable energy forecasting). Spatial fields are often characterized by high dimensionality and potentially strong correlations between neighboring locations. These characteristics make the verification of spatial forecasts very demanding in terms of discriminating misspecified dependence structures, for example. In the case of spatial forecasts, it is known that traditional verification methods (e.g., gridpoint-by-gridpoint verification) may result in a double penalty. The double-penalty effect was pinned in Ebert (2008) and refers to the fact that if a forecast presents a spatial (or temporal) shift with respect to observations, the error made would be penalized twice: once where the event was observed and again where the forecast predicted it. In particular, high-resolution forecasts are more penalized than less realistic blurry forecasts. The double-penalty effect may also affect spatio-temporal forecasts in general.

In parallel with the development of scoring rules, various application-focused spatial verification methods have been developed to evaluate weather forecasts. The efforts toward improving spatial verification methods have been guided by two projects: the intercomparison project (ICP; Gilleland et al. 2009) and its second phase, called Mesoscale Verification Intercomparison over Complex Terrain (MesoVICT; Dorninger et al. 2018). These projects resulted in the comparison of spatial verification methods with a particular focus on understanding their limitations and clarifying their interpretability. Only a few links exist between the approaches studied in these projects (and the work they induced) and the proper scoring rules framework. In particular, Casati et al. (2022) noted "a lack of representation of novel spatial verification methods for ensemble prediction systems". In general, there is a clear lack of methods focusing on the spatial verification of probabilistic forecasts. Moreover, to help bridging the gap between the two communities, we would like to recall the approach of spatial verification tools in the light of the scoring rule framework introduced above.

One of the goals of the ICP was to provide insights on how to develop methods robust to the double-penalty effect. In particular, Gilleland et al. (2009) proposed a classification of spatial verification tools updated later in Dorninger et al. (2018) resulting in a five-category classification. The classes differ in the computing principle they rely on. Not all spatial verification tools mentioned in these studies can be applied to probabilistic forecasts, some of them can solely be applied to deterministic forecasts. In the following description of the classes, we try to focus on methods suited to probabilistic forecasts or at least the special case of ensemble forecasts.

Neighborhood-based methods consist of applying a smoothing filter to the forecast and observation fields to prevent the double-penalty effect. The smoothing filter can take various forms (e.g., a minimum, a maximum, a mean, or a Gaussian filter) and be applied over a given neighborhood. For example, Stein and Stoop (2022) proposed a neighborhood-based CRPS for ensemble forecasts gathering forecasts and observations made within the neighborhood of the location considered. The use of a neighborhood prevents the double-penalty effect from taking place at scales smaller than that of the neighborhood. In this general definition, neighborhood-based methods can lead to proper scoring rules, in particular, see the notion of patches in Section 4.

Scale-separation techniques denote methods for which the verification is obtained after comparing forecast and observation fields across different scales. The scale-separation process can be seen as several single-bandpass spatial filters (e.g., projection onto a base of wavelets as wavelet-based scoring rules; Buschow et al. 2019). However, in order to obtain proper scoring rules, the comparison of the scale-specific characteristics needs to be performed using a proper scoring rule. Section 4 provides a discussion on wavelet-based scoring rules and their propriety.

Object-based methods rely on the identification of objects of interest and the comparison of the objects obtained in the forecast and observation fields. Object identification is application-dependent and can take the form of objects that forecasters are familiar with (e.g., storm cells for precipitation forecasts). A well-known verification tool within this class is the structure-amplitude-location (SAL; Wernli et al. 2008) method which has been generalized to ensemble forecasts in Radanovics et al. (2018). The three components of the ensemble SAL do not lead to proper scoring rules. They rely on the mean of the forecast within scoring functions inconsistent with the mean. Thus, the ideal forecast does not minimize the expected value. Nonetheless, the three components of the SAL method could be adapted to use proper scoring rules sensitive to the misspecification of the same features.

Field-deformation techniques consist of deforming the forecasts field into the observation field (the similarity between the fields can be ensured by a metric of interest). The field of distortion associated with the morphing of the forecast field into the observation field becomes a measure of the predictive performance of the forecast (see, e.g., Han and Szunyogh 2018).

Distance measures between binary images, such as exceedance of a threshold of interest, of the forecast and observation fields. These methods are inspired by development in image processing (e.g., Baddeley’s delta measure Gilleland 2011).

These five categories are partially overlap** as it can be argued that some methods belong to multiple categories (e.g., some distance measures techniques can be seen as a mix of field-deformation and object-based). They define different principles that can be used to build verification tools that are not subject to the double-penalty effect. The reader may refer to Dorninger et al. (2018) and references therein for details on the classification and the spatial verification methods not used thereafter. The frontier between the aforementioned spatial verification methods and the proper scoring rules framework is porous with, for example, wavelet-based scoring rules belonging to both. It appears that numerous spatial verification methods seek interpretability and we believe that this is not incompatible with the use of proper scoring rules. We propose the following framework to facilitate the construction of interpretable proper scoring rules.