-
COPR -- Efficient, large-scale log storage and retrieval
Authors:
Julian Reichinger,
Thomas Krismayer,
Jan Rellermeyer
Abstract:
Modern, large scale monitoring systems have to process and store vast amounts of log data in near real-time. At query time the systems have to find relevant logs based on the content of the log message using support structures that can scale to these amounts of data while still being efficient to use. We present our novel Compressed Probabilistic Retrieval algorithm (COPR), capable of answering Mu…
▽ More
Modern, large scale monitoring systems have to process and store vast amounts of log data in near real-time. At query time the systems have to find relevant logs based on the content of the log message using support structures that can scale to these amounts of data while still being efficient to use. We present our novel Compressed Probabilistic Retrieval algorithm (COPR), capable of answering Multi-Set Multi-Membership-Queries, that can be used as an alternative to existing indexing structures for streamed log data. In our experiments, COPR required up to 93% less storage space than the tested state-of-the-art inverted index and had up to four orders of magnitude less false-positives than the tested state-of-the-art membership sketch. Additionally, COPR achieved up to 250 times higher query throughput than the tested inverted index and up to 240 times higher query throughput than the tested membership sketch.
△ Less
Submitted 27 March, 2024; v1 submitted 28 February, 2024;
originally announced February 2024.
-
McUDI: Model-Centric Unsupervised Degradation Indicator for Failure Prediction AIOps Solutions
Authors:
Lorena Poenaru-Olaru,
Luis Cruz,
Jan Rellermeyer,
Arie van Deursen
Abstract:
Due to the continuous change in operational data, AIOps solutions suffer from performance degradation over time. Although periodic retraining is the state-of-the-art technique to preserve the failure prediction AIOps models' performance over time, this technique requires a considerable amount of labeled data to retrain. In AIOps obtaining label data is expensive since it requires the availability…
▽ More
Due to the continuous change in operational data, AIOps solutions suffer from performance degradation over time. Although periodic retraining is the state-of-the-art technique to preserve the failure prediction AIOps models' performance over time, this technique requires a considerable amount of labeled data to retrain. In AIOps obtaining label data is expensive since it requires the availability of domain experts to intensively annotate it. In this paper, we present McUDI, a model-centric unsupervised degradation indicator that is capable of detecting the exact moment the AIOps model requires retraining as a result of changes in data. We further show how employing McUDI in the maintenance pipeline of AIOps solutions can reduce the number of samples that require annotations with 30k for job failure prediction and 260k for disk failure prediction while achieving similar performance with periodic retraining.
△ Less
Submitted 25 January, 2024;
originally announced January 2024.
-
Is Your Anomaly Detector Ready for Change? Adapting AIOps Solutions to the Real World
Authors:
Lorena Poenaru-Olaru,
Natalia Karpova,
Luis Cruz,
Jan Rellermeyer,
Arie van Deursen
Abstract:
Anomaly detection techniques are essential in automating the monitoring of IT systems and operations. These techniques imply that machine learning algorithms are trained on operational data corresponding to a specific period of time and that they are continuously evaluated on newly emerging data. Operational data is constantly changing over time, which affects the performance of deployed anomaly d…
▽ More
Anomaly detection techniques are essential in automating the monitoring of IT systems and operations. These techniques imply that machine learning algorithms are trained on operational data corresponding to a specific period of time and that they are continuously evaluated on newly emerging data. Operational data is constantly changing over time, which affects the performance of deployed anomaly detection models. Therefore, continuous model maintenance is required to preserve the performance of anomaly detectors over time. In this work, we analyze two different anomaly detection model maintenance techniques in terms of the model update frequency, namely blind model retraining and informed model retraining. We further investigate the effects of updating the model by retraining it on all the available data (full-history approach) and only the newest data (sliding window approach). Moreover, we investigate whether a data change monitoring tool is capable of determining when the anomaly detection model needs to be updated through retraining.
△ Less
Submitted 11 April, 2024; v1 submitted 17 November, 2023;
originally announced November 2023.
-
Log Parsing Evaluation in the Era of Modern Software Systems
Authors:
Stefan Petrescu,
Floris den Hengst,
Alexandru Uta,
Jan S. Rellermeyer
Abstract:
Due to the complexity and size of modern software systems, the amount of logs generated is tremendous. Hence, it is infeasible to manually investigate these data in a reasonable time, thereby requiring automating log analysis to derive insights about the functioning of the systems. Motivated by an industry use-case, we zoom-in on one integral part of automated log analysis, log parsing, which is t…
▽ More
Due to the complexity and size of modern software systems, the amount of logs generated is tremendous. Hence, it is infeasible to manually investigate these data in a reasonable time, thereby requiring automating log analysis to derive insights about the functioning of the systems. Motivated by an industry use-case, we zoom-in on one integral part of automated log analysis, log parsing, which is the prerequisite to deriving any insights from logs. Our investigation reveals problematic aspects within the log parsing field, particularly its inefficiency in handling heterogeneous real-world logs. We show this by assessing the 14 most-recognized log parsing approaches in the literature using (i) nine publicly available datasets, (ii) one dataset comprised of combined publicly available data, and (iii) one dataset generated within the infrastructure of a large bank. Subsequently, toward improving log parsing robustness in real-world production scenarios, we propose a tool, Logchimera, that enables estimating log parsing performance in industry contexts through generating synthetic log data that resemble industry logs. Our contributions serve as a foundation to consolidate past research efforts, facilitate future research advancements, and establish a strong link between research and industry log parsing.
△ Less
Submitted 17 August, 2023;
originally announced August 2023.
-
Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study
Authors:
Lorena Poenaru-Olaru,
Luis Cruz,
Arie van Deursen,
Jan S. Rellermeyer
Abstract:
As machine learning models increasingly replace traditional business logic in the production system, their lifecycle management is becoming a significant concern. Once deployed into production, the machine learning models are constantly evaluated on new streaming data. Given the continuous data flow, shifting data, also known as concept drift, is ubiquitous in such settings. Concept drift usually…
▽ More
As machine learning models increasingly replace traditional business logic in the production system, their lifecycle management is becoming a significant concern. Once deployed into production, the machine learning models are constantly evaluated on new streaming data. Given the continuous data flow, shifting data, also known as concept drift, is ubiquitous in such settings. Concept drift usually impacts the performance of machine learning models, thus, identifying the moment when concept drift occurs is required. Concept drift is identified through concept drift detectors. In this work, we assess the reliability of concept drift detectors to identify drift in time by exploring how late are they reporting drifts and how many false alarms are they signaling. We compare the performance of the most popular drift detectors belonging to two different concept drift detector groups, error rate-based detectors and data distribution-based detectors. We assess their performance on both synthetic and real-world data. In the case of synthetic data, we investigate the performance of detectors to identify two types of concept drift, abrupt and gradual. Our findings aim to help practitioners understand which drift detector should be employed in different situations and, to achieve this, we share a list of the most important observations made throughout this study, which can serve as guidelines for practical usage. Furthermore, based on our empirical results, we analyze the suitability of each concept drift detection group to be used as alarming system.
△ Less
Submitted 23 November, 2022;
originally announced November 2022.
-
Future Computer Systems and Networking Research in the Netherlands: A Manifesto
Authors:
Alexandru Iosup,
Fernando Kuipers,
Ana Lucia Varbanescu,
Paola Grosso,
Animesh Trivedi,
Jan Rellermeyer,
Lin Wang,
Alexandru Uta,
Francesco Regazzoni
Abstract:
Our modern society and competitive economy depend on a strong digital foundation and, in turn, on sustained research and innovation in computer systems and networks (CompSys). With this manifesto, we draw attention to CompSys as a vital part of ICT. Among ICT technologies, CompSys covers all the hardware and all the operational software layers that enable applications; only application-specific de…
▽ More
Our modern society and competitive economy depend on a strong digital foundation and, in turn, on sustained research and innovation in computer systems and networks (CompSys). With this manifesto, we draw attention to CompSys as a vital part of ICT. Among ICT technologies, CompSys covers all the hardware and all the operational software layers that enable applications; only application-specific details, and often only application-specific algorithms, are not part of CompSys. Each of the Top Sectors of the Dutch Economy, each route in the National Research Agenda, and each of the UN Sustainable Development Goals pose challenges that cannot be addressed without groundbreaking CompSys advances. Looking at the 2030-2035 horizon, important new applications will emerge only when enabled by CompSys developments. Triggered by the COVID-19 pandemic, millions moved abruptly online, raising infrastructure scalability and data sovereignty issues; but governments processing social data and responsible social networks still require a paradigm shift in data sovereignty and sharing. AI already requires massive computer systems which can cost millions per training task, but the current technology leaves an unsustainable energy footprint including large carbon emissions. Computational sciences such as bioinformatics, and "Humanities for all" and "citizen data science", cannot become affordable and efficient until computer systems take a generational leap. Similarly, the emerging quantum internet depends on (traditional) CompSys to bootstrap operation for the foreseeable future. Large commercial sectors, including finance and manufacturing, require specialized computing and networking or risk becoming uncompetitive. And, at the core of Dutch innovation, promising technology hubs, deltas, ports, and smart cities, could see their promise stagger due to critical dependency on non-European technology.
△ Less
Submitted 26 May, 2022;
originally announced June 2022.
-
In-Memory Indexed Caching for Distributed Data Processing
Authors:
Alexandru Uta,
Bogdan Ghit,
Ankur Dave,
Jan Rellermeyer,
Peter Boncz
Abstract:
Powerful abstractions such as dataframes are only as efficient as their underlying runtime system. The de-facto distributed data processing framework, Apache Spark, is poorly suited for the modern cloud-based data-science workloads due to its outdated assumptions: static datasets analyzed using coarse-grained transformations. In this paper, we introduce the Indexed DataFrame, an in-memory cache th…
▽ More
Powerful abstractions such as dataframes are only as efficient as their underlying runtime system. The de-facto distributed data processing framework, Apache Spark, is poorly suited for the modern cloud-based data-science workloads due to its outdated assumptions: static datasets analyzed using coarse-grained transformations. In this paper, we introduce the Indexed DataFrame, an in-memory cache that supports a dataframe abstraction which incorporates indexing capabilities to support fast lookup and join operations. Moreover, it supports appends with multi-version concurrency control. We implement the Indexed DataFrame as a lightweight, standalone library which can be integrated with minimum effort in existing Spark programs. We analyze the performance of the Indexed DataFrame in cluster and cloud deployments with real-world datasets and benchmarks using both Apache Spark and Databricks Runtime. In our evaluation, we show that the Indexed DataFrame significantly speeds-up query execution when compared to a non-indexed dataframe, incurring modest memory overhead.
△ Less
Submitted 8 February, 2022; v1 submitted 12 December, 2021;
originally announced December 2021.
-
Roadmap for Edge AI: A Dagstuhl Perspective
Authors:
Aaron Yi Ding,
Ella Peltonen,
Tobias Meuser,
Atakan Aral,
Christian Becker,
Schahram Dustdar,
Thomas Hiessl,
Dieter Kranzlmuller,
Madhusanka Liyanage,
Setareh Magshudi,
Nitinder Mohan,
Joerg Ott,
Jan S. Rellermeyer,
Stefan Schulte,
Henning Schulzrinne,
Gurkan Solmaz,
Sasu Tarkoma,
Blesson Varghese,
Lars Wolf
Abstract:
Based on the collective input of Dagstuhl Seminar (21342), this paper presents a comprehensive discussion on AI methods and capabilities in the context of edge computing, referred as Edge AI. In a nutshell, we envision Edge AI to provide adaptation for data-driven applications, enhance network and radio access, and allow the creation, optimization, and deployment of distributed AI/ML pipelines wit…
▽ More
Based on the collective input of Dagstuhl Seminar (21342), this paper presents a comprehensive discussion on AI methods and capabilities in the context of edge computing, referred as Edge AI. In a nutshell, we envision Edge AI to provide adaptation for data-driven applications, enhance network and radio access, and allow the creation, optimization, and deployment of distributed AI/ML pipelines with given quality of experience, trust, security and privacy targets. The Edge AI community investigates novel ML methods for the edge computing environment, spanning multiple sub-fields of computer science, engineering and ICT. The goal is to share an envisioned roadmap that can bring together key actors and enablers to further advance the domain of Edge AI.
△ Less
Submitted 27 November, 2021;
originally announced December 2021.
-
A Fresh Look at the Architecture and Performance of Contemporary Isolation Platforms
Authors:
Vincent van Rijn,
Jan S. Rellermeyer
Abstract:
With the ever-increasing pervasiveness of the cloud computing paradigm, strong isolation guarantees and low performance overhead from isolation platforms are paramount. An ideal isolation platform offers both: an impermeable isolation boundary while imposing a negligible performance overhead. In this paper, we examine various isolation platforms (containers, secure containers, hypervisors, unikern…
▽ More
With the ever-increasing pervasiveness of the cloud computing paradigm, strong isolation guarantees and low performance overhead from isolation platforms are paramount. An ideal isolation platform offers both: an impermeable isolation boundary while imposing a negligible performance overhead. In this paper, we examine various isolation platforms (containers, secure containers, hypervisors, unikernels), and conduct a wide array of experiments to measure the performance overhead and degree of isolation offered by the platforms. We find that container platforms have the best, near-native, performance while the newly emerging secure containers suffer from various overheads. The highest degree of isolation is achieved by unikernels, closely followed by traditional containers.
△ Less
Submitted 21 October, 2021;
originally announced October 2021.
-
RCURRENCY: Live Digital Asset Trading Using a Recurrent Neural Network-based Forecasting System
Authors:
Yapeng Jasper Hu,
Ralph van Gurp,
Ashay Somai,
Hugo Kooijman,
Jan S. Rellermeyer
Abstract:
Consistent alpha generation, i.e., maintaining an edge over the market, underpins the ability of asset traders to reliably generate profits. Technical indicators and trading strategies are commonly used tools to determine when to buy/hold/sell assets, yet these are limited by the fact that they operate on known values. Over the past decades, multiple studies have investigated the potential of arti…
▽ More
Consistent alpha generation, i.e., maintaining an edge over the market, underpins the ability of asset traders to reliably generate profits. Technical indicators and trading strategies are commonly used tools to determine when to buy/hold/sell assets, yet these are limited by the fact that they operate on known values. Over the past decades, multiple studies have investigated the potential of artificial intelligence in stock trading in conventional markets, with some success. In this paper, we present RCURRENCY, an RNN-based trading engine to predict data in the highly volatile digital asset market which is able to successfully manage an asset portfolio in a live environment. By combining asset value prediction and conventional trading tools, RCURRENCY determines whether to buy, hold or sell digital currencies at a given point in time. Experimental results show that, given the data of an interval $t$, a prediction with an error of less than 0.5\% of the data at the subsequent interval $t+1$ can be obtained. Evaluation of the system through backtesting shows that RCURRENCY can be used to successfully not only maintain a stable portfolio of digital assets in a simulated live environment using real historical trading data but even increase the portfolio value over time.
△ Less
Submitted 13 June, 2021;
originally announced June 2021.
-
Systematic Map** Study on the Machine Learning Lifecycle
Authors:
Yuanhao Xie,
Luís Cruz,
Petra Heck,
Jan S. Rellermeyer
Abstract:
The development of artificial intelligence (AI) has made various industries eager to explore the benefits of AI. There is an increasing amount of research surrounding AI, most of which is centred on the development of new AI algorithms and techniques. However, the advent of AI is bringing an increasing set of practical problems related to AI model lifecycle management that need to be investigated.…
▽ More
The development of artificial intelligence (AI) has made various industries eager to explore the benefits of AI. There is an increasing amount of research surrounding AI, most of which is centred on the development of new AI algorithms and techniques. However, the advent of AI is bringing an increasing set of practical problems related to AI model lifecycle management that need to be investigated. We address this gap by conducting a systematic map** study on the lifecycle of AI model. Through quantitative research, we provide an overview of the field, identify research opportunities, and provide suggestions for future research. Our study yields 405 publications published from 2005 to 2020, mapped in 5 different main research topics, and 31 sub-topics. We observe that only a minority of publications focus on data management and model production problems, and that more studies should address the AI lifecycle from a holistic perspective.
△ Less
Submitted 11 March, 2021;
originally announced March 2021.
-
SGD_Tucker: A Novel Stochastic Optimization Strategy for Parallel Sparse Tucker Decomposition
Authors:
Hao Li,
Zixuan Li,
Kenli Li,
Jan S. Rellermeyer,
Lydia Y. Chen,
Keqin Li
Abstract:
Sparse Tucker Decomposition (STD) algorithms learn a core tensor and a group of factor matrices to obtain an optimal low-rank representation feature for the \underline{H}igh-\underline{O}rder, \underline{H}igh-\underline{D}imension, and \underline{S}parse \underline{T}ensor (HOHDST). However, existing STD algorithms face the problem of intermediate variables explosion which results from the fact t…
▽ More
Sparse Tucker Decomposition (STD) algorithms learn a core tensor and a group of factor matrices to obtain an optimal low-rank representation feature for the \underline{H}igh-\underline{O}rder, \underline{H}igh-\underline{D}imension, and \underline{S}parse \underline{T}ensor (HOHDST). However, existing STD algorithms face the problem of intermediate variables explosion which results from the fact that the formation of those variables, i.e., matrices Khatri-Rao product, Kronecker product, and matrix-matrix multiplication, follows the whole elements in sparse tensor. The above problems prevent deep fusion of efficient computation and big data platforms. To overcome the bottleneck, a novel stochastic optimization strategy (SGD$\_$Tucker) is proposed for STD which can automatically divide the high-dimension intermediate variables into small batches of intermediate matrices. Specifically, SGD$\_$Tucker only follows the randomly selected small samples rather than the whole elements, while maintaining the overall accuracy and convergence rate. In practice, SGD$\_$Tucker features the two distinct advancements over the state of the art. First, SGD$\_$Tucker can prune the communication overhead for the core tensor in distributed settings. Second, the low data-dependence of SGD$\_$Tucker enables fine-grained parallelization, which makes SGD$\_$Tucker obtaining lower computational overheads with the same accuracy. Experimental results show that SGD$\_$Tucker runs at least 2$X$ faster than the state of the art.
△ Less
Submitted 8 December, 2020; v1 submitted 7 December, 2020;
originally announced December 2020.
-
A Survey on Distributed Machine Learning
Authors:
Joost Verbraeken,
Matthijs Wolting,
Jonathan Katzy,
Jeroen Kloppenburg,
Tim Verbelen,
Jan S. Rellermeyer
Abstract:
The demand for artificial intelligence has grown significantly over the last decade and this growth has been fueled by advances in machine learning techniques and the ability to leverage hardware acceleration. However, in order to increase the quality of predictions and render machine learning solutions feasible for more complex applications, a substantial amount of training data is required. Alth…
▽ More
The demand for artificial intelligence has grown significantly over the last decade and this growth has been fueled by advances in machine learning techniques and the ability to leverage hardware acceleration. However, in order to increase the quality of predictions and render machine learning solutions feasible for more complex applications, a substantial amount of training data is required. Although small machine learning models can be trained with modest amounts of data, the input for training larger models such as neural networks grows exponentially with the number of parameters. Since the demand for processing training data has outpaced the increase in computation power of computing machinery, there is a need for distributing the machine learning workload across multiple machines, and turning the centralized into a distributed system. These distributed systems present new challenges, first and foremost the efficient parallelization of the training process and the creation of a coherent model. This article provides an extensive overview of the current state-of-the-art in the field by outlining the challenges and opportunities of distributed machine learning over conventional (centralized) machine learning, discussing the techniques used for distributed machine learning, and providing an overview of the systems that are available.
△ Less
Submitted 20 December, 2019;
originally announced December 2019.
-
Is Big Data Performance Reproducible in Modern Cloud Networks?
Authors:
Alexandru Uta,
Alexandru Custura,
Dmitry Duplyakin,
Ivo Jimenez,
Jan Rellermeyer,
Carlos Maltzahn,
Robert Ricci,
Alexandru Iosup
Abstract:
Performance variability has been acknowledged as a problem for over a decade by cloud practitioners and performance engineers. Yet, our survey of top systems conferences reveals that the research community regularly disregards variability when running experiments in the cloud. Focusing on networks, we assess the impact of variability on cloud-based big-data workloads by gathering traces from mains…
▽ More
Performance variability has been acknowledged as a problem for over a decade by cloud practitioners and performance engineers. Yet, our survey of top systems conferences reveals that the research community regularly disregards variability when running experiments in the cloud. Focusing on networks, we assess the impact of variability on cloud-based big-data workloads by gathering traces from mainstream commercial clouds and private research clouds. Our data collection consists of millions of datapoints gathered while transferring over 9 petabytes of data. We characterize the network variability present in our data and show that, even though commercial cloud providers implement mechanisms for quality-of-service enforcement, variability still occurs, and is even exacerbated by such mechanisms and service provider policies. We show how big-data workloads suffer from significant slowdowns and lack predictability and replicability, even when state-of-the-art experimentation techniques are used. We provide guidelines for practitioners to reduce the volatility of big data performance, making experiments more repeatable.
△ Less
Submitted 19 December, 2019;
originally announced December 2019.
-
The Coming Age of Pervasive Data Processing
Authors:
Jan S. Rellermeyer,
Sobhan Omranian Khorasani,
Dan Graur,
Apourva Parthasarathy
Abstract:
Emerging Big Data analytics and machine learning applications require a significant amount of computational power. While there exists a plethora of large-scale data processing frameworks which thrive in handling the various complexities of data-intensive workloads, the ever-increasing demand of applications have made us reconsider the traditional ways of scaling (e.g., scale-out) and seek new oppo…
▽ More
Emerging Big Data analytics and machine learning applications require a significant amount of computational power. While there exists a plethora of large-scale data processing frameworks which thrive in handling the various complexities of data-intensive workloads, the ever-increasing demand of applications have made us reconsider the traditional ways of scaling (e.g., scale-out) and seek new opportunities for improving the performance. In order to prepare for an era where data collection and processing occur on a wide range of devices, from powerful HPC machines to small embedded devices, it is crucial to investigate and eliminate the potential sources of inefficiency in the current state of the art platforms. In this paper, we address the current and upcoming challenges of pervasive data processing and present directions for designing the next generation of large-scale data processing systems.
△ Less
Submitted 21 June, 2019;
originally announced June 2019.
-
Container Density Improvements with Dynamic Memory Extension using NAND Flash
Authors:
Jan S. Rellermeyer,
Maher Amer,
Richard Smutzer,
Karthick Rajamani
Abstract:
While containers efficiently implement the idea of operating-system-level application virtualization, they are often insufficient to increase the server utilization to a desirable level. The reason is that in practice many containerized applications experience a limited amount of load while there are few containers with a high load. In such a scenario, the virtual memory management system can beco…
▽ More
While containers efficiently implement the idea of operating-system-level application virtualization, they are often insufficient to increase the server utilization to a desirable level. The reason is that in practice many containerized applications experience a limited amount of load while there are few containers with a high load. In such a scenario, the virtual memory management system can become the limiting factor to container density even though the working set of active containers would fit into main memory. In this paper, we describe and evaluate a system for transparently moving memory pages in and out of DRAM and to a NAND Flash medium which is attached through the memory bus. This technique, called Diablo Memory Expansion (DMX), operates on a prediction model and is able to relieve the pressure on the memory system. We present a benchmark for container density and show that even under an overall constant workload, adding additional containers adversely affects performance-critical applications in Docker. When using the DMX technology of the Memory1 system, however, the performance of the critical workload remains stable.
△ Less
Submitted 24 June, 2019;
originally announced June 2019.
-
MAVIS: Managing Datacenters using Smartphones
Authors:
Raghav Shankar,
Benjamin Kobin,
Saurabh Bagchi,
Michael Kistler,
Jan Rellermeyer
Abstract:
Distributed monitoring plays a crucial role in managing the activities of cloud-based datacenters. System administrators have long relied on monitoring systems such as Nagios and Ganglia to obtain status alerts on their desktop-class machines. However, the popularity of mobile devices is pushing the community to develop datacenter monitoring solutions for smartphone-class devices. Here we lay out…
▽ More
Distributed monitoring plays a crucial role in managing the activities of cloud-based datacenters. System administrators have long relied on monitoring systems such as Nagios and Ganglia to obtain status alerts on their desktop-class machines. However, the popularity of mobile devices is pushing the community to develop datacenter monitoring solutions for smartphone-class devices. Here we lay out desirable characteristics of such smartphone-based monitoring and identify quantitatively the shortcomings from directly applying existing solutions to this domain. Then we introduce a possible design that addresses some of these shortcomings and provide results from an early prototype, called MAVIS, using one month of monitoring data from approximately 3,000 machines hosted by Purdue's central IT organization.
△ Less
Submitted 17 February, 2018;
originally announced February 2018.
