-
LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs
Authors:
Fabian Biester,
Mohamed Abdelaal,
Daniel Del Gaudio
Abstract:
Machine learning's influence is expanding rapidly, now integral to decision-making processes from corporate strategy to the advancements in Industry 4.0. The efficacy of Artificial Intelligence broadly hinges on the caliber of data used during its training phase; optimal performance is tied to exceptional data quality. Data cleaning tools, particularly those that exploit functional dependencies wi…
▽ More
Machine learning's influence is expanding rapidly, now integral to decision-making processes from corporate strategy to the advancements in Industry 4.0. The efficacy of Artificial Intelligence broadly hinges on the caliber of data used during its training phase; optimal performance is tied to exceptional data quality. Data cleaning tools, particularly those that exploit functional dependencies within ontological frameworks or context models, are instrumental in augmenting data quality. Nevertheless, crafting these context models is a demanding task, both in terms of resources and expertise, often necessitating specialized knowledge from domain experts.
In light of these challenges, this paper introduces an innovative approach, called LLMClean, for the automated generation of context models, utilizing Large Language Models to analyze and understand various datasets. LLMClean encompasses a sequence of actions, starting with categorizing the dataset, extracting or map** relevant models, and ultimately synthesizing the context model. To demonstrate its potential, we have developed and tested a prototype that applies our approach to three distinct datasets from the Internet of Things, healthcare, and Industry 4.0 sectors. The results of our evaluation indicate that our automated approach can achieve data cleaning efficacy comparable with that of context models crafted by human experts.
△ Less
Submitted 29 April, 2024;
originally announced April 2024.
-
Open-Source Drift Detection Tools in Action: Insights from Two Use Cases
Authors:
Rieke Müller,
Mohamed Abdelaal,
Davor Stjelja
Abstract:
Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smar…
▽ More
Data drifts pose a critical challenge in the lifecycle of machine learning (ML) models, affecting their performance and reliability. In response to this challenge, we present a microbenchmark study, called D3Bench, which evaluates the efficacy of open-source drift detection tools. D3Bench examines the capabilities of Evidently AI, NannyML, and Alibi-Detect, leveraging real-world data from two smart building use cases.We prioritize assessing the functional suitability of these tools to identify and analyze data drifts. Furthermore, we consider a comprehensive set of non-functional criteria, such as the integrability with ML pipelines, the adaptability to diverse data types, user-friendliness, computational efficiency, and resource demands. Our findings reveal that Evidently AI stands out for its general data drift detection, whereas NannyML excels at pinpointing the precise timing of shifts and evaluating their consequent effects on predictive accuracy.
△ Less
Submitted 10 May, 2024; v1 submitted 29 April, 2024;
originally announced April 2024.
-
LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks
Authors:
Truong Thanh Hung Nguyen,
Tobias Clement,
Phuc Truong Loc Nguyen,
Nils Kemmerzell,
Van Binh Truong,
Vo Thanh Khang Nguyen,
Mohamed Abdelaal,
Hung Cao
Abstract:
LangXAI is a framework that integrates Explainable Artificial Intelligence (XAI) with advanced vision models to generate textual explanations for visual recognition tasks. Despite XAI advancements, an understanding gap persists for end-users with limited domain knowledge in artificial intelligence and computer vision. LangXAI addresses this by furnishing text-based explanations for classification,…
▽ More
LangXAI is a framework that integrates Explainable Artificial Intelligence (XAI) with advanced vision models to generate textual explanations for visual recognition tasks. Despite XAI advancements, an understanding gap persists for end-users with limited domain knowledge in artificial intelligence and computer vision. LangXAI addresses this by furnishing text-based explanations for classification, object detection, and semantic segmentation model outputs to end-users. Preliminary results demonstrate LangXAI's enhanced plausibility, with high BERTScore across tasks, fostering a more transparent and reliable AI framework on vision tasks for end-users.
△ Less
Submitted 19 February, 2024;
originally announced February 2024.
-
Beyond explaining: XAI-based Adaptive Learning with SHAP Clustering for Energy Consumption Prediction
Authors:
Tobias Clement,
Hung Truong Thanh Nguyen,
Nils Kemmerzell,
Mohamed Abdelaal,
Davor Stjelja
Abstract:
This paper presents an approach integrating explainable artificial intelligence (XAI) techniques with adaptive learning to enhance energy consumption prediction models, with a focus on handling data distribution shifts. Leveraging SHAP clustering, our method provides interpretable explanations for model predictions and uses these insights to adaptively refine the model, balancing model complexity…
▽ More
This paper presents an approach integrating explainable artificial intelligence (XAI) techniques with adaptive learning to enhance energy consumption prediction models, with a focus on handling data distribution shifts. Leveraging SHAP clustering, our method provides interpretable explanations for model predictions and uses these insights to adaptively refine the model, balancing model complexity with predictive performance. We introduce a three-stage process: (1) obtaining SHAP values to explain model predictions, (2) clustering SHAP values to identify distinct patterns and outliers, and (3) refining the model based on the derived SHAP clustering characteristics. Our approach mitigates overfitting and ensures robustness in handling data distribution shifts. We evaluate our method on a comprehensive dataset comprising energy consumption records of buildings, as well as two additional datasets to assess the transferability of our approach to other domains, regression, and classification problems. Our experiments demonstrate the effectiveness of our approach in both task types, resulting in improved predictive performance and interpretable model explanations.
△ Less
Submitted 7 February, 2024;
originally announced February 2024.
-
XAI-Enhanced Semantic Segmentation Models for Visual Quality Inspection
Authors:
Tobias Clement,
Truong Thanh Hung Nguyen,
Mohamed Abdelaal,
Hung Cao
Abstract:
Visual quality inspection systems, crucial in sectors like manufacturing and logistics, employ computer vision and machine learning for precise, rapid defect detection. However, their unexplained nature can hinder trust, error identification, and system improvement. This paper presents a framework to bolster visual quality inspection by using CAM-based explanations to refine semantic segmentation…
▽ More
Visual quality inspection systems, crucial in sectors like manufacturing and logistics, employ computer vision and machine learning for precise, rapid defect detection. However, their unexplained nature can hinder trust, error identification, and system improvement. This paper presents a framework to bolster visual quality inspection by using CAM-based explanations to refine semantic segmentation models. Our approach consists of 1) Model Training, 2) XAI-based Model Explanation, 3) XAI Evaluation, and 4) Annotation Augmentation for Model Enhancement, informed by explanations and expert insights. Evaluations show XAI-enhanced models surpass original DeepLabv3-ResNet101 models, especially in intricate object segmentation.
△ Less
Submitted 18 January, 2024;
originally announced January 2024.
-
A Cyber-HIL for Investigating Control Systems in Ship Cyber Physical Systems under Communication Issues and Cyber Attacks
Authors:
Linh Vu,
Lam Nguyen,
Mahmoud Abdelaal,
Tuyen Vu,
Osama Mohammed
Abstract:
This paper presents a novel Cyber-Hardware-in-the-Loop (Cyber-HIL) platform for assessing control operation in ship cyber-physical systems. The proposed platform employs cutting-edge technologies, including Docker containers, real-time simulator $OPAL-RT$, and network emulator $ns3$, to create a secure and controlled testing and deployment environment for investigating the potential impact of cybe…
▽ More
This paper presents a novel Cyber-Hardware-in-the-Loop (Cyber-HIL) platform for assessing control operation in ship cyber-physical systems. The proposed platform employs cutting-edge technologies, including Docker containers, real-time simulator $OPAL-RT$, and network emulator $ns3$, to create a secure and controlled testing and deployment environment for investigating the potential impact of cyber attack threats on ship control systems. Real-time experiments were conducted using an advanced load-shedding controller as a control object in both synchronous and asynchronous manners, showcasing the platform's versatility and effectiveness in identifying vulnerabilities and improving overall Ship Cyber Physical System (SCPS) security. Furthermore, the performance of the load-shedding controller under cyber attacks was evaluated by conducting tests with man-in-the-middle (MITM) and denial-of-service (DoS) attacks. These attacks were implemented on the communication channels between the controller and the simulated ship system, emulating real-world scenarios. The proposed Cyber-HIL platform provides a comprehensive and effective approach to test and validate the security of ship control systems in the face of cyber threats.
△ Less
Submitted 25 August, 2023; v1 submitted 24 June, 2023;
originally announced June 2023.
-
AutoCure: Automated Tabular Data Curation Technique for ML Pipelines
Authors:
Mohamed Abdelaal,
Rashmi Koparde,
Harald Schoening
Abstract:
Machine learning algorithms have become increasingly prevalent in multiple domains, such as autonomous driving, healthcare, and finance. In such domains, data preparation remains a significant challenge in develo** accurate models, requiring significant expertise and time investment to search the huge search space of well-suited data curation and transformation tools. To address this challenge,…
▽ More
Machine learning algorithms have become increasingly prevalent in multiple domains, such as autonomous driving, healthcare, and finance. In such domains, data preparation remains a significant challenge in develo** accurate models, requiring significant expertise and time investment to search the huge search space of well-suited data curation and transformation tools. To address this challenge, we present AutoCure, a novel and configuration-free data curation pipeline that improves the quality of tabular data. Unlike traditional data curation methods, AutoCure synthetically enhances the density of the clean data fraction through an adaptive ensemble-based error detection method and a data augmentation module. In practice, AutoCure can be integrated with open source tools, e.g., Auto-sklearn, H2O, and TPOT, to promote the democratization of machine learning. As a proof of concept, we provide a comparative evaluation of AutoCure against 28 combinations of traditional data curation tools, demonstrating superior performance and predictive accuracy without user intervention. Our evaluation shows that AutoCure is an effective approach to automating data preparation and improving the accuracy of machine learning models.
△ Less
Submitted 26 April, 2023;
originally announced April 2023.
-
RTClean: Context-aware Tabular Data Cleaning using Real-time OFDs
Authors:
Daniel Del Gaudio,
Tim Schubert,
Mohamed Abdelaal
Abstract:
Nowadays, machine learning plays a key role in develo** plenty of applications, e.g., smart homes, smart medical assistance, and autonomous driving. A major challenge of these applications is preserving high quality of the training and the serving data. Nevertheless, existing data cleaning methods cannot exploit context information. Thus, they usually fail to track shifts in the data distributio…
▽ More
Nowadays, machine learning plays a key role in develo** plenty of applications, e.g., smart homes, smart medical assistance, and autonomous driving. A major challenge of these applications is preserving high quality of the training and the serving data. Nevertheless, existing data cleaning methods cannot exploit context information. Thus, they usually fail to track shifts in the data distributions or the associated error profiles. To overcome these limitations, we introduce, in this paper, a novel method for automated tabular data cleaning powered by dynamic functional dependency rules extracted from a live context model. As a proof of concept, we create a smart home use case to collect data while preserving the context information. Using two different data sets, our evaluations show that the proposed cleaning method outperforms a set of baseline methods in terms of the detection and repair accuracy.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines
Authors:
Mohamed Abdelaal,
Christian Hammacher,
Harald Schoening
Abstract:
Nowadays, machine learning (ML) plays a vital role in many aspects of our daily life. In essence, building well-performing ML applications requires the provision of high-quality data throughout the entire life-cycle of such applications. Nevertheless, most of the real-world tabular data suffer from different types of discrepancies, such as missing values, outliers, duplicates, pattern violation, a…
▽ More
Nowadays, machine learning (ML) plays a vital role in many aspects of our daily life. In essence, building well-performing ML applications requires the provision of high-quality data throughout the entire life-cycle of such applications. Nevertheless, most of the real-world tabular data suffer from different types of discrepancies, such as missing values, outliers, duplicates, pattern violation, and inconsistencies. Such discrepancies typically emerge while collecting, transferring, storing, and/or integrating the data. To deal with these discrepancies, numerous data cleaning methods have been introduced. However, the majority of such methods broadly overlook the requirements imposed by downstream ML models. As a result, the potential of utilizing these data cleaning methods in ML pipelines is predominantly unrevealed. In this work, we introduce a comprehensive benchmark, called REIN1, to thoroughly investigate the impact of data cleaning methods on various ML models. Through the benchmark, we provide answers to important research questions, e.g., where and whether data cleaning is a necessary step in ML pipelines. To this end, the benchmark examines 38 simple and advanced error detection and repair methods. To evaluate these methods, we utilized a wide collection of ML models trained on 14 publicly-available datasets covering different domains and encompassing realistic as well as synthetic error profiles.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
Scalability in Visualization
Authors:
Gaëlle Richer,
Alexis Pister,
Moataz Abdelaal,
Jean-Daniel Fekete,
Michael Sedlmair,
Daniel Weiskopf
Abstract:
We introduce a conceptual model for scalability designed for visualization research. With this model, we systematically analyze over 120 visualization publications from 1990-2020 to characterize the different notions of scalability in these works. While many papers have addressed scalability issues, our survey identifies a lack of consistency in the use of the term in the visualization research co…
▽ More
We introduce a conceptual model for scalability designed for visualization research. With this model, we systematically analyze over 120 visualization publications from 1990-2020 to characterize the different notions of scalability in these works. While many papers have addressed scalability issues, our survey identifies a lack of consistency in the use of the term in the visualization research community. We address this issue by introducing a consistent terminology meant to help visualization researchers better characterize the scalability aspects in their research. It also helps in providing multiple methods for supporting the claim that a work is "scalable". Our model is centered around an effort function with inputs and outputs. The inputs are the problem size and resources, whereas the outputs are the actual efforts, for instance, in terms of computational run time or visual clutter. We select representative examples to illustrate different approaches and facets of what scalability can mean in visualization literature. Finally, targeting the diverse crowd of visualization researchers without a scalability tradition, we provide a set of recommendations for how scalability can be presented in a clear and consistent way to improve fair comparison between visualization techniques and systems and foster reproducibility.
△ Less
Submitted 14 December, 2022; v1 submitted 12 October, 2022;
originally announced October 2022.
-
Comparative Evaluation of Bipartite, Node-Link, and Matrix-Based Network Representations
Authors:
Moataz Abdelaal,
Nathan D. Schiele,
Katrin Angerbauer,
Kuno Kurzhals,
Michael Sedlmair,
Daniel Weiskopf
Abstract:
This work investigates and compares the performance of node-link diagrams, adjacency matrices, and bipartite layouts for visualizing networks. In a crowd-sourced user study (n = 150), we measure the task accuracy and completion time of the three representations for different network classes and properties. In contrast to the literature, which covers mostly topology-based tasks (e.g., path finding)…
▽ More
This work investigates and compares the performance of node-link diagrams, adjacency matrices, and bipartite layouts for visualizing networks. In a crowd-sourced user study (n = 150), we measure the task accuracy and completion time of the three representations for different network classes and properties. In contrast to the literature, which covers mostly topology-based tasks (e.g., path finding) in small datasets, we mainly focus on overview tasks for large and directed networks. We consider three overview tasks on networks with 500 nodes: (T1) network class identification, (T2) cluster detection, and (T3) network density estimation, and two detailed tasks: (T4) node in-degree vs. out-degree and (T5) representation map**, on networks with 50 and 20 nodes, respectively. Our results show that bipartite layouts are beneficial for revealing the overall network structure, while adjacency matrices are most reliable across the different tasks.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
-
DiffML: End-to-end Differentiable ML Pipelines
Authors:
Benjamin Hilprecht,
Christian Hammacher,
Eduardo Reis,
Mohamed Abdelaal,
Carsten Binnig
Abstract:
In this paper, we present our vision of differentiable ML pipelines called DiffML to automate the construction of ML pipelines in an end-to-end fashion. The idea is that DiffML allows to jointly train not just the ML model itself but also the entire pipeline including data preprocessing steps, e.g., data cleaning, feature selection, etc. Our core idea is to formulate all pipeline steps in a differ…
▽ More
In this paper, we present our vision of differentiable ML pipelines called DiffML to automate the construction of ML pipelines in an end-to-end fashion. The idea is that DiffML allows to jointly train not just the ML model itself but also the entire pipeline including data preprocessing steps, e.g., data cleaning, feature selection, etc. Our core idea is to formulate all pipeline steps in a differentiable way such that the entire pipeline can be trained using backpropagation. However, this is a non-trivial problem and opens up many new research questions. To show the feasibility of this direction, we demonstrate initial ideas and a general principle of how typical preprocessing steps such as data cleaning, feature selection and dataset selection can be formulated as differentiable programs and jointly learned with the ML model. Moreover, we discuss a research roadmap and core challenges that have to be systematically tackled to enable fully differentiable ML pipelines.
△ Less
Submitted 5 July, 2022; v1 submitted 4 July, 2022;
originally announced July 2022.
-
Efficient Update of Redundancy Matrices for Truss and Frame Structures
Authors:
Tim Krake,
Malte von Scheven,
Jan Gade,
Moataz Abdelaal,
Daniel Weiskopf,
Manfred Bischoff
Abstract:
Redundancy matrices provide insights into the load carrying behavior of statically indeterminate structures. This information can be employed for the design and analysis of structures with regard to certain objectives, for example reliability, robustness, or adaptability. In this context, the structure is often iteratively examined with the help of slight adjustments. However, this procedure gener…
▽ More
Redundancy matrices provide insights into the load carrying behavior of statically indeterminate structures. This information can be employed for the design and analysis of structures with regard to certain objectives, for example reliability, robustness, or adaptability. In this context, the structure is often iteratively examined with the help of slight adjustments. However, this procedure generally requires a high computational effort for the recalculation of the redundancy matrix due to the necessity of costly matrix operations. This paper addresses this problem by providing generic algebraic formulations for efficiently updating the redundancy matrix (and related matrices). The formulations include various modifications like adding, removing, and exchanging elements and are applicable to truss and frame structures. With several examples, we demonstrate the interaction between the formulas and their mechanical interpretation. Finally, a performance test for a scaleable structure is presented.
△ Less
Submitted 28 November, 2022; v1 submitted 23 May, 2022;
originally announced May 2022.
-
Visualization of Nonlinear Programming for Robot Motion Planning
Authors:
David Hägele,
Moataz Abdelaal,
Ozgur S. Oguz,
Marc Toussaint,
Daniel Weiskopf
Abstract:
Nonlinear programming targets nonlinear optimization with constraints, which is a generic yet complex methodology involving humans for problem modeling and algorithms for problem solving. We address the particularly hard challenge of supporting domain experts in handling, understanding, and trouble-shooting high-dimensional optimization with a large number of constraints. Leveraging visual analyti…
▽ More
Nonlinear programming targets nonlinear optimization with constraints, which is a generic yet complex methodology involving humans for problem modeling and algorithms for problem solving. We address the particularly hard challenge of supporting domain experts in handling, understanding, and trouble-shooting high-dimensional optimization with a large number of constraints. Leveraging visual analytics, users are supported in exploring the computation process of nonlinear constraint optimization. Our system was designed for robot motion planning problems and developed in tight collaboration with domain experts in nonlinear programming and robotics. We report on the experiences from this design study, illustrate the usefulness for relevant example cases, and discuss the extension to visual analytics for nonlinear programming in general.
△ Less
Submitted 28 January, 2021;
originally announced January 2021.
