-
Forecasting with Deep Learning: Beyond Average of Average of Average Performance
Authors:
Vitor Cerqueira,
Luis Roque,
Carlos Soares
Abstract:
Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. We hypothesize that averaging performance over all samples dilutes relevant information about the relative performance of models. Particularly, conditions in whi…
▽ More
Accurate evaluation of forecasting models is essential for ensuring reliable predictions. Current practices for evaluating and comparing forecasting models focus on summarising performance into a single score, using metrics such as SMAPE. We hypothesize that averaging performance over all samples dilutes relevant information about the relative performance of models. Particularly, conditions in which this relative performance is different than the overall accuracy. We address this limitation by proposing a novel framework for evaluating univariate time series forecasting models from multiple perspectives, such as one-step ahead forecasting versus multi-step ahead forecasting. We show the advantages of this framework by comparing a state-of-the-art deep learning approach with classical forecasting techniques. While classical methods (e.g. ARIMA) are long-standing approaches to forecasting, deep neural networks (e.g. NHITS) have recently shown state-of-the-art forecasting performance in benchmark datasets. We conducted extensive experiments that show NHITS generally performs best, but its superiority varies with forecasting conditions. For instance, concerning the forecasting horizon, NHITS only outperforms classical approaches for multi-step ahead forecasting. Another relevant insight is that, when dealing with anomalies, NHITS is outperformed by methods such as Theta. These findings highlight the importance of aspect-based model evaluation.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Boosting Mixed-Initiative Co-Creativity in Game Design: A Tutorial
Authors:
Solange Margarido,
Licínio Roque,
Penousal Machado,
Pedro Martins
Abstract:
In recent years, there has been a growing application of mixed-initiative co-creative approaches in the creation of video games. The rapid advances in the capabilities of artificial intelligence (AI) systems further propel creative collaboration between humans and computational agents. In this tutorial, we present guidelines for researchers and practitioners to develop game design tools with a hig…
▽ More
In recent years, there has been a growing application of mixed-initiative co-creative approaches in the creation of video games. The rapid advances in the capabilities of artificial intelligence (AI) systems further propel creative collaboration between humans and computational agents. In this tutorial, we present guidelines for researchers and practitioners to develop game design tools with a high degree of mixed-initiative co-creativity (MI-CCy). We begin by reviewing a selection of current works that will serve as case studies and categorize them by the type of game content they address. We introduce the MI-CCy Quantifier, a framework that can be used by researchers and developers to assess co-creative tools on their level of MI-CCy through a visual scheme of quantifiable criteria scales. We demonstrate the usage of the MI-CCy Quantifier by applying it to the selected works. This analysis enabled us to discern prevalent patterns within these tools, as well as features that contribute to a higher level of MI-CCy. We highlight current gaps in MI-CCy approaches within game design, which we propose as pivotal aspects to tackle in the development of forthcoming approaches.
△ Less
Submitted 11 January, 2024;
originally announced January 2024.
-
Performing Creativity With Computational Tools
Authors:
Daniel Lopes,
Jéssica Parente,
Pedro Silva,
Licínio Roque,
Penousal machado
Abstract:
The introduction of new tools in people's workflow has always been promotive of new creative paths. This paper discusses the impact of using computational tools in the performance of creative tasks, especially focusing on graphic design. The study was driven by a grounded theory methodology, applied to a set of semi-structured interviews, made to twelve people working in the areas of graphic desig…
▽ More
The introduction of new tools in people's workflow has always been promotive of new creative paths. This paper discusses the impact of using computational tools in the performance of creative tasks, especially focusing on graphic design. The study was driven by a grounded theory methodology, applied to a set of semi-structured interviews, made to twelve people working in the areas of graphic design, data science, computer art, music and data visualisation. Among other questions, the results suggest some scenarios in which it is or it is not worth investing in the development of new intelligent creativity-aiding tools.
△ Less
Submitted 10 March, 2021; v1 submitted 9 March, 2021;
originally announced March 2021.
-
Optimal Combination Forecasts on Retail Multi-Dimensional Sales Data
Authors:
Luis Roque,
Cristina A. C. Fernandes,
Tony Silva
Abstract:
Time series data in the retail world are particularly rich in terms of dimensionality, and these dimensions can be aggregated in groups or hierarchies. Valuable information is nested in these complex structures, which helps to predict the aggregated time series data. From a portfolio of brands under HUUB's monitoring, we selected two to explore their sales behaviour, leveraging the grou** proper…
▽ More
Time series data in the retail world are particularly rich in terms of dimensionality, and these dimensions can be aggregated in groups or hierarchies. Valuable information is nested in these complex structures, which helps to predict the aggregated time series data. From a portfolio of brands under HUUB's monitoring, we selected two to explore their sales behaviour, leveraging the grou** properties of their product structure. Using statistical models, namely SARIMA, to forecast each level of the hierarchy, an optimal combination approach was used to generate more consistent forecasts in the higher levels. Our results show that the proposed methods can indeed capture nested information in the more granular series, hel** to improve the forecast accuracy of the aggregated series. The Weighted Least Squares (WLS) method surpasses all other methods proposed in the study, including the Minimum Trace (MinT) reconciliation.
△ Less
Submitted 22 March, 2019;
originally announced March 2019.
-
On the design of an expert help system for computer algebra systems
Authors:
Renato P. dos Santos,
Waldir L. Roque
Abstract:
It is our intention here only to discuss the nature, complexity and tools concerning the design of Smart Help, an expert help facility for aiding users of Computer Algebra Systems. Although the expert help system presented here has been particularly oriented to REDUCE (as a consequence of our former experience with this system), we point out that the concept of Smart Help can be extended to other…
▽ More
It is our intention here only to discuss the nature, complexity and tools concerning the design of Smart Help, an expert help facility for aiding users of Computer Algebra Systems. Although the expert help system presented here has been particularly oriented to REDUCE (as a consequence of our former experience with this system), we point out that the concept of Smart Help can be extended to other Computer Algebra Systems. Technically, Smart Help is a Production System on the top of a particular implementation of MANTRA, a hybrid knowledge representation system, which has REDUCE integrated as an additional knowledge representation module. Since the heuristic level of MANTRA has not yet been implemented, being presently represented by the Lisp language itself, Smart Help is coded in Lisp and resides in the same Lisp session of MANTRA. A prototype of Smart Help is now running on a SUN work-station on an experimental basis.
△ Less
Submitted 28 May, 2014; v1 submitted 27 May, 2014;
originally announced May 2014.
