Showing 1–11 of 11 results for author: Ruskowski, M

Skills Composition Framework for Reconfigurable Cyber-Physical Production Modules

Authors: Aleksandr Sidorenko, Achim Wagner, Martin Ruskowski

Abstract: While the benefits of reconfigurable manufacturing systems (RMS) are well-known, there are still challenges to their development, including, among others, a modular software architecture that enables rapid reconfiguration without much reprogramming effort. Skill-based engineering improves software modularity and increases the reconfiguration potential of RMS. Nevertheless, a skills' composition fr… ▽ More While the benefits of reconfigurable manufacturing systems (RMS) are well-known, there are still challenges to their development, including, among others, a modular software architecture that enables rapid reconfiguration without much reprogramming effort. Skill-based engineering improves software modularity and increases the reconfiguration potential of RMS. Nevertheless, a skills' composition framework with a focus on frequent and rapid software changes is still missing. The Behavior trees (BTs) framework is a novel approach, which enables intuitive design of modular hierarchical control structures. BTs have been mostly explored from the AI and robotics perspectives, and little work has been done in investigating their potential for composing skills in the manufacturing domain. This paper proposes a framework for skills' composition and execution in skill-based reconfigurable cyber-physical production modules (RCPPMs). It is based on distributed BTs and provides good integration between low-level devices' specific code and AI-based task-oriented frameworks. We have implemented the provided models for the IEC 61499-based distributed automation controllers to show the instantiation of the proposed framework with the specific industrial technology and enable its evaluation by the automation community. △ Less

Submitted 22 May, 2024; originally announced May 2024.

Towards Using Behavior Trees in Industrial Automation Controllers

Authors: Aleksandr Sidorenko, Mahdi Rezapour, Achim Wagner, Martin Ruskowski

Abstract: The Industry 4.0 paradigm manifests the shift towards mass customization and cyber-physical production systems (CPPS) and sets new requirements for industrial automation software in terms of modularity, flexibility, and short development cycles of control programs. Though programmable logical controllers (PLCs) have been evolving into versatile and powerful edge devices, there is a lack of PLC sof… ▽ More The Industry 4.0 paradigm manifests the shift towards mass customization and cyber-physical production systems (CPPS) and sets new requirements for industrial automation software in terms of modularity, flexibility, and short development cycles of control programs. Though programmable logical controllers (PLCs) have been evolving into versatile and powerful edge devices, there is a lack of PLC software flexibility and integration between low-level programs and high-level task-oriented control frameworks. Behavior trees (BTs) is a novel framework, which enables rapid design of modular hierarchical control structures. It combines improved modularity with a simple and intuitive design of control logic. This paper proposes an approach for improving the industrial control software design by integrating BTs into PLC programs and separating hardware related functionalities from the coordination logic. Several strategies for integration of BTs into PLCs are shown. The first two integrate BTs with the IEC 61131 based PLCs and are based on the use of the PLCopen Common Behavior Model. The last one utilized event-based BTs and shows the integration with the IEC 61499 based controllers. An application example demonstrates the approach. The paper contributes in the following ways. First, we propose a new PLC software design, which improves modularity, supports better separation of concerns, and enables rapid development and reconfiguration of the control software. Second, we show and evaluate the integration of the BT framework into both IEC 61131 and IEC 61499 based PLCs, as well as the integration of the PLCopen function blocks with the external BT library. This leads to better integration of the low-level PLC code and the AI-based task-oriented frameworks. It also improves the skill-based programming approach for PLCs by using BTs for skills composition. △ Less

Submitted 22 April, 2024; originally announced April 2024.

Enhancing Object Detection Performance for Small Objects through Synthetic Data Generation and Proportional Class-Balancing Technique: A Comparative Study in Industrial Scenarios

Authors: Jibinraj Antony, Vinit Hegiste, Ali Nazeri, Hooman Tavakoli, Snehal Walunj, Christiane Plociennik, Martin Ruskowski

Abstract: Object Detection (OD) has proven to be a significant computer vision method in extracting localized class information and has multiple applications in the industry. Although many of the state-of-the-art (SOTA) OD models perform well on medium and large sized objects, they seem to under perform on small objects. In most of the industrial use cases, it is difficult to collect and annotate data for s… ▽ More Object Detection (OD) has proven to be a significant computer vision method in extracting localized class information and has multiple applications in the industry. Although many of the state-of-the-art (SOTA) OD models perform well on medium and large sized objects, they seem to under perform on small objects. In most of the industrial use cases, it is difficult to collect and annotate data for small objects, as it is time-consuming and prone to human errors. Additionally, those datasets are likely to be unbalanced and often result in an inefficient model convergence. To tackle this challenge, this study presents a novel approach that injects additional data points to improve the performance of the OD models. Using synthetic data generation, the difficulties in data collection and annotations for small object data points can be minimized and to create a dataset with balanced distribution. This paper discusses the effects of a simple proportional class-balancing technique, to enable better anchor matching of the OD models. A comparison was carried out on the performances of the SOTA OD models: YOLOv5, YOLOv7 and SSD, for combinations of real and synthetic datasets within an industrial use case. △ Less

Submitted 29 January, 2024; v1 submitted 23 January, 2024; originally announced January 2024.

Comments: Accepted and presented in conference ESAIM23 1st European Symposium on Artificial Intelligence in Manufacturing

State Derivative Normalization for Continuous-Time Deep Neural Networks

Authors: Jonas Weigand, Gerben I. Beintema, Jonas Ulmen, Daniel Görges, Roland Tóth, Maarten Schoukens, Martin Ruskowski

Abstract: The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a r… ▽ More The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature. △ Less

Submitted 14 May, 2024; v1 submitted 5 January, 2024; originally announced January 2024.

Comments: This work has been accepted for presentation at the 20th IFAC Symposium on System Identification 2024

Federated K-Means Clustering via Dual Decomposition-based Distributed Optimization

Authors: Vassilios Yfantis, Achim Wagner, Martin Ruskowski

Abstract: The use of distributed optimization in machine learning can be motivated either by the resulting preservation of privacy or the increase in computational efficiency. On the one hand, training data might be stored across multiple devices. Training a global model within a network where each node only has access to its confidential data requires the use of distributed algorithms. Even if the data is… ▽ More The use of distributed optimization in machine learning can be motivated either by the resulting preservation of privacy or the increase in computational efficiency. On the one hand, training data might be stored across multiple devices. Training a global model within a network where each node only has access to its confidential data requires the use of distributed algorithms. Even if the data is not confidential, sharing it might be prohibitive due to bandwidth limitations. On the other hand, the ever-increasing amount of available data leads to large-scale machine learning problems. By splitting the training process across multiple nodes its efficiency can be significantly increased. This paper aims to demonstrate how dual decomposition can be applied for distributed training of $ K $-means clustering problems. After an overview of distributed and federated machine learning, the mixed-integer quadratically constrained programming-based formulation of the $ K $-means clustering training problem is presented. The training can be performed in a distributed manner by splitting the data across different nodes and linking these nodes through consensus constraints. Finally, the performance of the subgradient method, the bundle trust method, and the quasi-Newton dual ascent algorithm are evaluated on a set of benchmark problems. While the mixed-integer programming-based formulation of the clustering problems suffers from weak integer relaxations, the presented approach can potentially be used to enable an efficient solution in the future, both in a central and distributed setting. △ Less

Submitted 25 July, 2023; originally announced July 2023.

MSC Class: 90

Federated Ensemble YOLOv5 -- A Better Generalized Object Detection Algorithm

Authors: Vinit Hegiste, Tatjana Legler, Martin Ruskowski

Abstract: Federated learning (FL) has gained significant traction as a privacy-preserving algorithm, but the underlying resemblances of federated learning algorithms like Federated averaging (FedAvg) or Federated SGD (Fed SGD) to ensemble learning algorithms have not been fully explored. The purpose of this paper is to examine the application of FL to object detection as a method to enhance generalizability… ▽ More Federated learning (FL) has gained significant traction as a privacy-preserving algorithm, but the underlying resemblances of federated learning algorithms like Federated averaging (FedAvg) or Federated SGD (Fed SGD) to ensemble learning algorithms have not been fully explored. The purpose of this paper is to examine the application of FL to object detection as a method to enhance generalizability, and to compare its performance against a centralized training approach for an object detection algorithm. Specifically, we investigate the performance of a YOLOv5 model trained using FL across multiple clients and employ a random sampling strategy without replacement, so each client holds a portion of the same dataset used for centralized training. Our experimental results showcase the superior efficiency of the FL object detector's global model in generating accurate bounding boxes for unseen objects, with the test set being a mixture of objects from two distinct clients not represented in the training dataset. These findings suggest that FL can be viewed from an ensemble algorithm perspective, akin to a synergistic blend of Bagging and Boosting techniques. As a result, FL can be seen not only as a method to enhance privacy, but also as a method to enhance the performance of a machine learning model. △ Less

Submitted 25 August, 2023; v1 submitted 30 June, 2023; originally announced June 2023.

Comments: 8 pages and submitted to FLTA2023 symposium under IEEE

Federated Object Detection for Quality Inspection in Shared Production

Authors: Vinit Hegiste, Tatjana Legler, Martin Ruskowski

Abstract: Federated learning (FL) has emerged as a promising approach for training machine learning models on decentralized data without compromising data privacy. In this paper, we propose a FL algorithm for object detection in quality inspection tasks using YOLOv5 as the object detection algorithm and Federated Averaging (FedAvg) as the FL algorithm. We apply this approach to a manufacturing use-case wher… ▽ More Federated learning (FL) has emerged as a promising approach for training machine learning models on decentralized data without compromising data privacy. In this paper, we propose a FL algorithm for object detection in quality inspection tasks using YOLOv5 as the object detection algorithm and Federated Averaging (FedAvg) as the FL algorithm. We apply this approach to a manufacturing use-case where multiple factories/clients contribute data for training a global object detection model while preserving data privacy on a non-IID dataset. Our experiments demonstrate that our FL approach achieves better generalization performance on the overall clients' test dataset and generates improved bounding boxes around the objects compared to models trained using local clients' datasets. This work showcases the potential of FL for quality inspection tasks in the manufacturing industry and provides valuable insights into the performance and feasibility of utilizing YOLOv5 and FedAvg for federated object detection. △ Less

Submitted 25 August, 2023; v1 submitted 30 June, 2023; originally announced June 2023.

Comments: Will submit it to an IEEE conference

Application of federated learning in manufacturing

Authors: Vinit Hegiste, Tatjana Legler, Martin Ruskowski

Abstract: A vast amount of data is created every minute, both in the private sector and industry. Whereas it is often easy to get hold of data in the private entertainment sector, in the industrial production environment it is much more difficult due to laws, preservation of intellectual property, and other factors. However, most machine learning methods require a data source that is sufficient in terms of… ▽ More A vast amount of data is created every minute, both in the private sector and industry. Whereas it is often easy to get hold of data in the private entertainment sector, in the industrial production environment it is much more difficult due to laws, preservation of intellectual property, and other factors. However, most machine learning methods require a data source that is sufficient in terms of quantity and quality. A suitable way to bring both requirements together is federated learning where learning progress is aggregated, but everyone remains the owner of their data. Federate learning was first proposed by Google researchers in 2016 and is used for example in the improvement of Google's keyboard Gboard. In contrast to billions of android users, comparable machinery is only used by few companies. This paper examines which other constraints prevail in production and which federated learning approaches can be considered as a result. △ Less

Submitted 21 November, 2022; v1 submitted 9 August, 2022; originally announced August 2022.

Comments: 8 pages, 15 figures, Submitted at IEEE conference I4Tech2022

Small Object Detection for Near Real-Time Egocentric Perception in a Manual Assembly Scenario

Authors: Hooman Tavakoli, Snehal Walunj, Parsha Pahlevannejad, Christiane Plociennik, Martin Ruskowski

Abstract: Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data… ▽ More Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data of objects are typically available and can be used to generate synthetic training data. We describe a near real-time small object detection pipeline for egocentric perception in a manual assembly scenario: We generate a training data set based on CAD data and realistic backgrounds in Unity. We then train a YOLOv4 model for a two-stage detection process: First, the context is recognized, then the small object of interest is detected. We evaluate our pipeline on the augmented reality device Microsoft Hololens 2. △ Less

Submitted 11 June, 2021; originally announced June 2021.

Comments: Accepted for presentation at EPIC@CVPR2021 workshop

Dynamic collision avoidance for multiple robotic manipulators based on a non-cooperative multi-agent game

Authors: Nigora Gafur, Gajanan Kanagalingam, Martin Ruskowski

Abstract: A flexible operation of multiple robotic manipulators in a shared workspace requires an online trajectory planning with static and dynamic collision avoidance. In this work, we propose a real-time capable motion control algorithm, based on non-linear model predictive control, which accounts for static and dynamic collision avoidance. The proposed algorithm is formulated as a non-cooperative game,… ▽ More A flexible operation of multiple robotic manipulators in a shared workspace requires an online trajectory planning with static and dynamic collision avoidance. In this work, we propose a real-time capable motion control algorithm, based on non-linear model predictive control, which accounts for static and dynamic collision avoidance. The proposed algorithm is formulated as a non-cooperative game, where each robot is considered as an agent. Each agent optimizes its own motion and accounts for the predicted movement of surrounding agents. We propose a novel approach to formulate the dynamic collision constraints. Additionally, we account for deadlocks that might occur in a setup of multiple robotic manipulators. We validate our algorithm on a pick and place scenario for four collaborative robots operating in a common workspace in the simulation environment Gazebo. The robots are controlled by the Robot Operating System (ROS). We demonstrate, that our approach is real-time capable and, due to the distributed nature of the approach, easily scales to an arbitrary number of robot manipulators in a shared workspace. △ Less

Submitted 13 February, 2022; v1 submitted 28 February, 2021; originally announced March 2021.

Comments: 12 pages, 11 figures

Flatness Based Control of an Industrial Robot Joint Using Secondary Encoders

Authors: Jonas Weigand, Nigora Gafur, Martin Ruskowski

Abstract: Due to their compliant structure, industrial robots without precision-enhancing measures are only to a limited extent suitable for machining applications. Apart from structural, thermal and bearing deformations, the main cause for compliant structure is backlash of transmission drives. This paper proposes a method to improve trajectory tracking accuracy by using secondary encoders and applying a f… ▽ More Due to their compliant structure, industrial robots without precision-enhancing measures are only to a limited extent suitable for machining applications. Apart from structural, thermal and bearing deformations, the main cause for compliant structure is backlash of transmission drives. This paper proposes a method to improve trajectory tracking accuracy by using secondary encoders and applying a feedback and a flatness based feed forward control strategy. For this purpose, a novel nonlinear, continuously differentiable dynamical model of a flexible robot joint is presented. The robot joint is modeled as a two-mass oscillator with pose-dependent inertia, nonlinear friction and nonlinear stiffness, including backlash. A flatness based feed forward control is designed to improve the guiding behaviour and a feedback controller, based on secondary encoders, is implemented for disturbance compensation. Using Automatic Differentiation, the nonlinear feed forward controller can be computed in a few microseconds online. Finally, the proposed algorithms are evaluated in simulations and experimentally on a real KUKA Quantec KR300 Ultra SE. △ Less

Submitted 13 January, 2021; originally announced January 2021.

Journal ref: Robotics and Computer-Integrated Manufacturing, Volume 68, 2021, 102039, ISSN 0736-5845