-
scores: A Python package for verifying and evaluating models and predictions with xarray
Authors:
Tennessee Leeuwenburg,
Nicholas Loveday,
Elizabeth E. Ebert,
Harrison Cook,
Mohammadreza Khanarmuei,
Robert J. Taggart,
Nikeeth Ramanathan,
Maree Carroll,
Stephanie Chong,
Aidan Griffiths,
John Sharples
Abstract:
`scores` is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. It supports labelled n-dimensional (multidimensional) data, which is used in many scientific fields and in machine learning. At present, `scores` primarily supports the geoscience communities; in particular, the meteorological, climatological and oce…
▽ More
`scores` is a Python package containing mathematical functions for the verification, evaluation and optimisation of forecasts, predictions or models. It supports labelled n-dimensional (multidimensional) data, which is used in many scientific fields and in machine learning. At present, `scores` primarily supports the geoscience communities; in particular, the meteorological, climatological and oceanographic communities. `scores` not only includes common scores (e.g., Mean Absolute Error), it also includes novel scores not commonly found elsewhere (e.g., FIxed Risk Multicategorical (FIRM) score, Flip-Flop Index), complex scores (e.g., threshold-weighted continuous ranked probability score), and statistical tests (such as the Diebold Mariano test). It also contains isotonic regression which is becoming an increasingly important tool in forecast verification and can be used to generate stable reliability diagrams. Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). At the time of writing, `scores` includes over 50 metrics, statistical techniques and data processing tools. All of the scores and statistical techniques in this package have undergone a thorough scientific and software review. Every score has a companion Jupyter Notebook tutorial that demonstrates its use in practice. `scores` supports `xarray` datatypes, allowing it to work with Earth system data in a range of formats including NetCDF4, HDF5, Zarr and GRIB among others. `scores` uses Dask for scaling and performance. Support for `pandas` is being introduced. The `scores` software repository can be found at https://github.com/nci/scores/
△ Less
Submitted 3 July, 2024; v1 submitted 11 June, 2024;
originally announced June 2024.
-
The Jive Verification System and its Transformative Impact on Weather Forecasting Operations
Authors:
Nicholas Loveday,
Deryn Griffiths,
Tennessee Leeuwenburg,
Robert Taggart,
Thomas C. Pagano,
George Cheng,
Kevin Plastow,
Elizabeth Ebert,
Cassandra Templeton,
Maree Carroll,
Mohammadreza Khanarmuei,
Isha Nagpal
Abstract:
Forecast verification is critical for continuous improvement in meteorological organizations. The Jive verification system was originally developed to assess the accuracy of public weather forecasts issued by the Australian Bureau of Meteorology. It started as a research project in 2015 and gradually evolved into the operational verification system that went live in 2022. The system includes daily…
▽ More
Forecast verification is critical for continuous improvement in meteorological organizations. The Jive verification system was originally developed to assess the accuracy of public weather forecasts issued by the Australian Bureau of Meteorology. It started as a research project in 2015 and gradually evolved into the operational verification system that went live in 2022. The system includes daily verification dashboards for forecasters to visualize recent forecast performance and "Evidence Targeted Automation" dashboards for exploring the performance of competing forecast systems. Additionally, there is a Jupyter Notebook server with the Jive Python library which supports research experiments, case studies, and the development of new verification metrics and tools.
This paper shows how the Jive verification project helped bring verification to the forefront at the Bureau of Meteorology, leading to more accurate, streamlined forecasts. Jive has been used to provide evidence for forecast automation decisions and has helped to understand the evolving role of meteorologists in the forecast process. It has given operational meteorologists tools for evaluating forecast processes, including identifying when and how manual interventions lead to superior predictions. The project also led to new verification science, including novel metrics that are decision-focused, including for extreme conditions. Additionally, Jive has provided the Bureau with an enterprise-wide data analysis environment and has prompted a clarification of forecast definitions.
These collective impacts have resulted in more accurate forecasts, ultimately benefiting society, and building trust with forecast users. These positive outcomes highlight the importance of meteorological organizations investing in verification science and technology.
△ Less
Submitted 29 April, 2024;
originally announced April 2024.
-
Site-specific Deterministic Temperature and Humidity Forecasts with Explainable and Reliable Machine Learning
Authors:
MengMeng Han,
Tennessee Leeuwenburg,
Brad Murphy
Abstract:
Site-specific weather forecasts are essential to accurate prediction of power demand and are consequently of great interest to energy operators. However, weather forecasts from current numerical weather prediction (NWP) models lack the fine-scale detail to capture all important characteristics of localised real-world sites. Instead they provide weather information representing a rectangular gridbo…
▽ More
Site-specific weather forecasts are essential to accurate prediction of power demand and are consequently of great interest to energy operators. However, weather forecasts from current numerical weather prediction (NWP) models lack the fine-scale detail to capture all important characteristics of localised real-world sites. Instead they provide weather information representing a rectangular gridbox (usually kilometres in size). Even after post-processing and bias correction, area-averaged information is usually not optimal for specific sites. Prior work on site optimised forecasts has focused on linear methods, weighted consensus averaging, time-series methods, and others. Recent developments in machine learning (ML) have prompted increasing interest in applying ML as a novel approach towards this problem. In this study, we investigate the feasibility of optimising forecasts at sites by adopting the popular machine learning model gradient boosting decision tree, supported by the Python version of the XGBoost package. Regression trees have been trained with historical NWP and site observations as training data, aimed at predicting temperature and dew point at multiple site locations across Australia. We developed a working ML framework, named 'Multi-SiteBoost' and initial testing results show a significant improvement compared with gridded values from bias-corrected NWP models. The improvement from XGBoost is found to be comparable with non-ML methods reported in literature. With the insights provided by SHapley Additive exPlanations (SHAP), this study also tests various approaches to understand the ML predictions and increase the reliability of the forecasts generated by ML.
△ Less
Submitted 4 April, 2024;
originally announced April 2024.
