-
History repeats Itself: A Baseline for Temporal Knowledge Graph Forecasting
Authors:
Julia Gastinger,
Christian Meilicke,
Federico Errica,
Timo Sztyler,
Anett Schuelke,
Heiner Stuckenschmidt
Abstract:
Temporal Knowledge Graph (TKG) Forecasting aims at predicting links in Knowledge Graphs for future timesteps based on a history of Knowledge Graphs. To this day, standardized evaluation protocols and rigorous comparison across TKG models are available, but the importance of simple baselines is often neglected in the evaluation, which prevents researchers from discerning actual and fictitious progr…
▽ More
Temporal Knowledge Graph (TKG) Forecasting aims at predicting links in Knowledge Graphs for future timesteps based on a history of Knowledge Graphs. To this day, standardized evaluation protocols and rigorous comparison across TKG models are available, but the importance of simple baselines is often neglected in the evaluation, which prevents researchers from discerning actual and fictitious progress. We propose to close this gap by designing an intuitive baseline for TKG Forecasting based on predicting recurring facts. Compared to most TKG models, it requires little hyperparameter tuning and no iterative training. Further, it can help to identify failure modes in existing approaches. The empirical findings are quite unexpected: compared to 11 methods on five datasets, our baseline ranks first or third in three of them, painting a radically different picture of the predictive quality of the state of the art.
△ Less
Submitted 29 April, 2024; v1 submitted 25 April, 2024;
originally announced April 2024.
-
Towards Low-burden Responses to Open Questions in VR
Authors:
Dmitry Alexandrovsky,
Susanne Putze,
Alexander Schülke,
Rainer Malaka
Abstract:
Subjective self-reports in VR user studies is a burdening and often tedious task for the participants. To minimize the disruption with the ongoing experience VR research has started to administer the surveying directly inside the virtual environments. However, due to the tedious nature of text-entry in VR, most VR surveying tools focus on closed questions with predetermined responses, while open q…
▽ More
Subjective self-reports in VR user studies is a burdening and often tedious task for the participants. To minimize the disruption with the ongoing experience VR research has started to administer the surveying directly inside the virtual environments. However, due to the tedious nature of text-entry in VR, most VR surveying tools focus on closed questions with predetermined responses, while open questions with free-text responses remain unexplored. This neglects a crucial part of UX research. To provide guidance on suitable self-reporting methods for open questions in VR user studies, this position paper presents a comparative study with three text-entry methods in VR and outlines future directions towards low-burden qualitative responding.
△ Less
Submitted 24 April, 2021;
originally announced April 2021.
-
A study on Ensemble Learning for Time Series Forecasting and the need for Meta-Learning
Authors:
Julia Gastinger,
Sébastien Nicolas,
Dušica Stepić,
Mischa Schmidt,
Anett Schülke
Abstract:
The contribution of this work is twofold: (1) We introduce a collection of ensemble methods for time series forecasting to combine predictions from base models. We demonstrate insights on the power of ensemble learning for forecasting, showing experiment results on about 16000 openly available datasets, from M4, M5, M3 competitions, as well as FRED (Federal Reserve Economic Data) datasets. Whereas…
▽ More
The contribution of this work is twofold: (1) We introduce a collection of ensemble methods for time series forecasting to combine predictions from base models. We demonstrate insights on the power of ensemble learning for forecasting, showing experiment results on about 16000 openly available datasets, from M4, M5, M3 competitions, as well as FRED (Federal Reserve Economic Data) datasets. Whereas experiments show that ensembles provide a benefit on forecasting results, there is no clear winning ensemble strategy (plus hyperparameter configuration). Thus, in addition, (2), we propose a meta-learning step to choose, for each dataset, the most appropriate ensemble method and their hyperparameter configuration to run based on dataset meta-features.
△ Less
Submitted 23 April, 2021;
originally announced April 2021.
-
The Combinatorial Multi-Bandit Problem and its Application to Energy Management
Authors:
Tobias Jacobs,
Mischa Schmidt,
Sébastien Nicolas,
Anett Schülke
Abstract:
We study a Combinatorial Multi-Bandit Problem motivated by applications in energy systems management. Given multiple probabilistic multi-arm bandits with unknown outcome distributions, the task is to optimize the value of a combinatorial objective function map** the vector of individual bandit outcomes to a single scalar reward. Unlike in single-bandit problems with multi-dimensional action spac…
▽ More
We study a Combinatorial Multi-Bandit Problem motivated by applications in energy systems management. Given multiple probabilistic multi-arm bandits with unknown outcome distributions, the task is to optimize the value of a combinatorial objective function map** the vector of individual bandit outcomes to a single scalar reward. Unlike in single-bandit problems with multi-dimensional action space, the outcomes of the individual bandits are observable in our setting and the objective function is known. Guided by the hypothesis that individual observability enables better trade-offs between exploration and exploitation, we generalize the lower regret bound for single bandits, showing that indeed for multiple bandits it admits parallelized exploration. For our energy management application we propose a range of algorithms that combine exploration principles for multi-arm bandits with mathematical programming. In an experimental study we demonstrate the effectiveness of our approach to learn action assignments for 150 bandits, each having 24 actions, within a horizon of 365 episodes.
△ Less
Submitted 4 November, 2020; v1 submitted 30 October, 2020;
originally announced October 2020.
-
HAMLET -- A Learning Curve-Enabled Multi-Armed Bandit for Algorithm Selection
Authors:
Mischa Schmidt,
Julia Gastinger,
Sébastien Nicolas,
Anett Schülke
Abstract:
Automated algorithm selection and hyperparameter tuning facilitates the application of machine learning. Traditional multi-armed bandit strategies look to the history of observed rewards to identify the most promising arms for optimizing expected total reward in the long run. When considering limited time budgets and computational resources, this backward view of rewards is inappropriate as the ba…
▽ More
Automated algorithm selection and hyperparameter tuning facilitates the application of machine learning. Traditional multi-armed bandit strategies look to the history of observed rewards to identify the most promising arms for optimizing expected total reward in the long run. When considering limited time budgets and computational resources, this backward view of rewards is inappropriate as the bandit should look into the future for anticipating the highest final reward at the end of a specified time budget. This work addresses that insight by introducing HAMLET, which extends the bandit approach with learning curve extrapolation and computation time-awareness for selecting among a set of machine learning algorithms. Results show that the HAMLET Variants 1-3 exhibit equal or better performance than other bandit-based algorithm selection strategies in experiments with recorded hyperparameter tuning traces for the majority of considered time budgets. The best performing HAMLET Variant 3 combines learning curve extrapolation with the well-known upper confidence bound exploration bonus. That variant performs better than all non-HAMLET policies with statistical significance at the 95% level for 1,485 runs.
△ Less
Submitted 28 May, 2020; v1 submitted 30 January, 2020;
originally announced January 2020.
-
On the Performance of Differential Evolution for Hyperparameter Tuning
Authors:
Mischa Schmidt,
Shahd Safarani,
Julia Gastinger,
Tobias Jacobs,
Sebastien Nicolas,
Anett Schülke
Abstract:
Automated hyperparameter tuning aspires to facilitate the application of machine learning for non-experts. In the literature, different optimization approaches are applied for that purpose. This paper investigates the performance of Differential Evolution for tuning hyperparameters of supervised learning algorithms for classification tasks. This empirical study involves a range of different machin…
▽ More
Automated hyperparameter tuning aspires to facilitate the application of machine learning for non-experts. In the literature, different optimization approaches are applied for that purpose. This paper investigates the performance of Differential Evolution for tuning hyperparameters of supervised learning algorithms for classification tasks. This empirical study involves a range of different machine learning algorithms and datasets with various characteristics to compare the performance of Differential Evolution with Sequential Model-based Algorithm Configuration (SMAC), a reference Bayesian Optimization approach. The results indicate that Differential Evolution outperforms SMAC for most datasets when tuning a given machine learning algorithm - particularly when breaking ties in a first-to-report fashion. Only for the tightest of computational budgets SMAC performs better. On small datasets, Differential Evolution outperforms SMAC by 19% (37% after tie-breaking). In a second experiment across a range of representative datasets taken from the literature, Differential Evolution scores 15% (23% after tie-breaking) more wins than SMAC.
△ Less
Submitted 15 April, 2019;
originally announced April 2019.
