-
SMOF: Streaming Modern CNNs on FPGAs with Smart Off-Chip Eviction
Authors:
Petros Toupas,
Zhewen Yu,
Christos-Savvas Bouganis,
Dimitrios Tzovaras
Abstract:
Convolutional Neural Networks (CNNs) have demonstrated their effectiveness in numerous vision tasks. However, their high processing requirements necessitate efficient hardware acceleration to meet the application's performance targets. In the space of FPGAs, streaming-based dataflow architectures are often adopted by users, as significant performance gains can be achieved through layer-wise pipeli…
▽ More
Convolutional Neural Networks (CNNs) have demonstrated their effectiveness in numerous vision tasks. However, their high processing requirements necessitate efficient hardware acceleration to meet the application's performance targets. In the space of FPGAs, streaming-based dataflow architectures are often adopted by users, as significant performance gains can be achieved through layer-wise pipelining and reduced off-chip memory access by retaining data on-chip. However, modern topologies, such as the UNet, YOLO, and X3D models, utilise long skip connections, requiring significant on-chip storage and thus limiting the performance achieved by such system architectures. The paper addresses the above limitation by introducing weight and activation eviction mechanisms to off-chip memory along the computational pipeline, taking into account the available compute and memory resources. The proposed mechanism is incorporated into an existing toolflow, expanding the design space by utilising off-chip memory as a buffer. This enables the map** of such modern CNNs to devices with limited on-chip memory, under the streaming architecture design approach. SMOF has demonstrated the capacity to deliver competitive and, in some cases, state-of-the-art performance across a spectrum of computer vision tasks, achieving up to 10.65 X throughput improvement compared to previous works.
△ Less
Submitted 27 March, 2024;
originally announced March 2024.
-
From Detection to Action Recognition: An Edge-Based Pipeline for Robot Human Perception
Authors:
Petros Toupas,
Georgios Tsamis,
Dimitrios Giakoumis,
Konstantinos Votis,
Dimitrios Tzovaras
Abstract:
Mobile service robots are proving to be increasingly effective in a range of applications, such as healthcare, monitoring Activities of Daily Living (ADL), and facilitating Ambient Assisted Living (AAL). These robots heavily rely on Human Action Recognition (HAR) to interpret human actions and intentions. However, for HAR to function effectively on service robots, it requires prior knowledge of hu…
▽ More
Mobile service robots are proving to be increasingly effective in a range of applications, such as healthcare, monitoring Activities of Daily Living (ADL), and facilitating Ambient Assisted Living (AAL). These robots heavily rely on Human Action Recognition (HAR) to interpret human actions and intentions. However, for HAR to function effectively on service robots, it requires prior knowledge of human presence (human detection) and identification of individuals to monitor (human tracking). In this work, we propose an end-to-end pipeline that encompasses the entire process, starting from human detection and tracking, leading to action recognition. The pipeline is designed to operate in near real-time while ensuring all stages of processing are performed on the edge, reducing the need for centralised computation. To identify the most suitable models for our mobile robot, we conducted a series of experiments comparing state-of-the-art solutions based on both their detection performance and efficiency. To evaluate the effectiveness of our proposed pipeline, we proposed a dataset comprising daily household activities. By presenting our findings and analysing the results, we demonstrate the efficacy of our approach in enabling mobile robots to understand and respond to human behaviour in real-world scenarios relying mainly on the data from their RGB cameras.
△ Less
Submitted 6 December, 2023;
originally announced December 2023.
-
fpgaHART: A toolflow for throughput-oriented acceleration of 3D CNNs for HAR onto FPGAs
Authors:
Petros Toupas,
Christos-Savvas Bouganis,
Dimitrios Tzovaras
Abstract:
Surveillance systems, autonomous vehicles, human monitoring systems, and video retrieval are just few of the many applications in which 3D Convolutional Neural Networks are exploited. However, their extensive use is restricted by their high computational and memory requirements, especially when integrated into systems with limited resources. This study proposes a toolflow that optimises the mappin…
▽ More
Surveillance systems, autonomous vehicles, human monitoring systems, and video retrieval are just few of the many applications in which 3D Convolutional Neural Networks are exploited. However, their extensive use is restricted by their high computational and memory requirements, especially when integrated into systems with limited resources. This study proposes a toolflow that optimises the map** of 3D CNN models for Human Action Recognition onto FPGA devices, taking into account FPGA resources and off-chip memory characteristics. The proposed system employs Synchronous Dataflow (SDF) graphs to model the designs and introduces transformations to expand and explore the design space, resulting in high-throughput designs. A variety of 3D CNN models were evaluated using the proposed toolflow on multiple FPGA devices, demonstrating its potential to deliver competitive performance compared to earlier hand-tuned and model-specific designs.
△ Less
Submitted 31 May, 2023;
originally announced May 2023.
-
FMM-X3D: FPGA-based modeling and map** of X3D for Human Action Recognition
Authors:
Petros Toupas,
Christos-Savvas Bouganis,
Dimitrios Tzovaras
Abstract:
3D Convolutional Neural Networks are gaining increasing attention from researchers and practitioners and have found applications in many domains, such as surveillance systems, autonomous vehicles, human monitoring systems, and video retrieval. However, their widespread adoption is hindered by their high computational and memory requirements, especially when resource-constrained systems are targete…
▽ More
3D Convolutional Neural Networks are gaining increasing attention from researchers and practitioners and have found applications in many domains, such as surveillance systems, autonomous vehicles, human monitoring systems, and video retrieval. However, their widespread adoption is hindered by their high computational and memory requirements, especially when resource-constrained systems are targeted. This paper addresses the problem of map** X3D, a state-of-the-art model in Human Action Recognition that achieves accuracy of 95.5\% in the UCF101 benchmark, onto any FPGA device. The proposed toolflow generates an optimised stream-based hardware system, taking into account the available resources and off-chip memory characteristics of the FPGA device. The generated designs push further the current performance-accuracy pareto front, and enable for the first time the targeting of such complex model architectures for the Human Action Recognition task.
△ Less
Submitted 29 May, 2023;
originally announced May 2023.
-
HARFLOW3D: A Latency-Oriented 3D-CNN Accelerator Toolflow for HAR on FPGA Devices
Authors:
Petros Toupas,
Alexander Montgomerie-Corcoran,
Christos-Savvas Bouganis,
Dimitrios Tzovaras
Abstract:
For Human Action Recognition tasks (HAR), 3D Convolutional Neural Networks have proven to be highly effective, achieving state-of-the-art results. This study introduces a novel streaming architecture based toolflow for map** such models onto FPGAs considering the model's inherent characteristics and the features of the targeted FPGA device. The HARFLOW3D toolflow takes as input a 3D CNN in ONNX…
▽ More
For Human Action Recognition tasks (HAR), 3D Convolutional Neural Networks have proven to be highly effective, achieving state-of-the-art results. This study introduces a novel streaming architecture based toolflow for map** such models onto FPGAs considering the model's inherent characteristics and the features of the targeted FPGA device. The HARFLOW3D toolflow takes as input a 3D CNN in ONNX format and a description of the FPGA characteristics, generating a design that minimizes the latency of the computation. The toolflow is comprised of a number of parts, including i) a 3D CNN parser, ii) a performance and resource model, iii) a scheduling algorithm for executing 3D models on the generated hardware, iv) a resource-aware optimization engine tailored for 3D models, v) an automated map** to synthesizable code for FPGAs. The ability of the toolflow to support a broad range of models and devices is shown through a number of experiments on various 3D CNN and FPGA system pairs. Furthermore, the toolflow has produced high-performing results for 3D CNN models that have not been mapped to FPGAs before, demonstrating the potential of FPGA-based systems in this space. Overall, HARFLOW3D has demonstrated its ability to deliver competitive latency compared to a range of state-of-the-art hand-tuned approaches being able to achieve up to 5$\times$ better performance compared to some of the existing works.
△ Less
Submitted 29 May, 2023; v1 submitted 30 March, 2023;
originally announced March 2023.
-
Open Challenges in Synthetic Speech Detection
Authors:
Luca Cuccovillo,
Christoforos Papastergiopoulos,
Anastasios Vafeiadis,
Artem Yaroshchuk,
Patrick Aichroth,
Konstantinos Votis,
Dimitrios Tzovaras
Abstract:
In this paper the current status and open challenges of synthetic speech detection are addressed. The work comprises an initial analysis of available open datasets and of existing detection methods, a description of the requirements for new research datasets compliant with regulations and better representing real-case scenarios, and a discussion of the desired characteristics of future trustworthy…
▽ More
In this paper the current status and open challenges of synthetic speech detection are addressed. The work comprises an initial analysis of available open datasets and of existing detection methods, a description of the requirements for new research datasets compliant with regulations and better representing real-case scenarios, and a discussion of the desired characteristics of future trustworthy detection methods in terms of both functional and non-functional requirements. Compared to other works, based on specific detection solutions or presenting single dataset of synthetic speeches, our paper is meant to orient future state-of-the-art research in the domain, to quickly lessen the current gap between synthesis and detection approaches.
△ Less
Submitted 26 January, 2023; v1 submitted 15 September, 2022;
originally announced September 2022.
-
A Survey of Robotic Harvesting Systems and Enabling Technologies
Authors:
Leonidas Droukas,
Zoe Doulgeri,
Nikolaos L. Tsakiridis,
Dimitra Triantafyllou,
Ioannis Kleitsiotis,
Ioannis Mariolis,
Dimitrios Giakoumis,
Dimitrios Tzovaras,
Dimitrios Kateris,
Dionysis Bochtis
Abstract:
This paper presents a comprehensive review of ground agricultural robotic systems and applications with special focus on harvesting that span research and commercial products and results, as well as their enabling technologies. The majority of literature concerns the development of crop detection, field navigation via vision and their related challenges. Health monitoring, yield estimation, water…
▽ More
This paper presents a comprehensive review of ground agricultural robotic systems and applications with special focus on harvesting that span research and commercial products and results, as well as their enabling technologies. The majority of literature concerns the development of crop detection, field navigation via vision and their related challenges. Health monitoring, yield estimation, water status inspection, seed planting and weed removal are frequently encountered tasks. Regarding robotic harvesting, apples, strawberries, tomatoes and sweet peppers are mainly the crops considered in publications, research projects and commercial products. The reported harvesting agricultural robotic solutions, typically consist of a mobile platform, a single robotic arm/manipulator and various navigation/vision systems. This paper reviews reported development of specific functionalities and hardware, typically required by an operating agricultural robot harvester; they include (a) vision systems, (b) motion planning/navigation methodologies (for the robotic platform and/or arm), (c) Human-Robot-Interaction (HRI) strategies with 3D visualization, (d) system operation planning & gras** strategies and (e) robotic end-effector/gripper design. Clearly, automated agriculture and specifically autonomous harvesting via robotic systems is a research area that remains wide open, offering several challenges where new contributions can be made.
△ Less
Submitted 1 February, 2023; v1 submitted 21 July, 2022;
originally announced July 2022.
-
A Characterization of 3D Printability
Authors:
Ioannis Fudos,
Margarita Ntousia,
Vasiliki Stamati,
Paschalis Charalampous,
Theodora Kontodina,
Ioannis Kostavelis,
Dimitrios Tzovaras,
Leonardo Bilalis
Abstract:
Additive manufacturing technologies are positioned to provide an unprecedented innovative transformation in how products are designed and manufactured. Due to differences in the technical specifications of AM technologies, the final fabricated parts can vary significantly from the original CAD models, therefore raising issues regarding accuracy, surface finish, robustness, mechanical properties, f…
▽ More
Additive manufacturing technologies are positioned to provide an unprecedented innovative transformation in how products are designed and manufactured. Due to differences in the technical specifications of AM technologies, the final fabricated parts can vary significantly from the original CAD models, therefore raising issues regarding accuracy, surface finish, robustness, mechanical properties, functional and geometrical constraints. Various researchers have studied the correlation between AM technologies and design rules.
In this work we propose a novel approach to assessing the capability of a 3D model to be printed successfully (a.k.a printability) on a specific AM machine. This is utilized by taking into consideration the model mesh complexity and certain part characteristics. A printability score is derived for a model in reference to a specific 3D printing technology, expressing the probability of obtaining a robust and accurate end result for 3D printing on a specific AM machine. The printability score can be used either to determine which 3D technology is more suitable for manufacturing a specific model or as a guide to redesign the model to ensure printability. We verify this framework by conducting 3D printing experiments for benchmark models which are printed on three AM machines employing different technologies: Fused Deposition Modeling (FDM), Binder Jetting (3DP), and Material Jetting (Polyjet).
△ Less
Submitted 24 October, 2020;
originally announced October 2020.
-
A Deep Learning Framework for Simulation and Defect Prediction Applied in Microelectronics
Authors:
Nikolaos Dimitriou,
Lampros Leontaris,
Thanasis Vafeiadis,
Dimosthenis Ioannidis,
Tracy Wotherspoon,
Gregory Tinker,
Dimitrios Tzovaras
Abstract:
The prediction of upcoming events in industrial processes has been a long-standing research goal since it enables optimization of manufacturing parameters, planning of equipment maintenance and more importantly prediction and eventually prevention of defects. While existing approaches have accomplished substantial progress, they are mostly limited to processing of one dimensional signals or requir…
▽ More
The prediction of upcoming events in industrial processes has been a long-standing research goal since it enables optimization of manufacturing parameters, planning of equipment maintenance and more importantly prediction and eventually prevention of defects. While existing approaches have accomplished substantial progress, they are mostly limited to processing of one dimensional signals or require parameter tuning to model environmental parameters. In this paper, we propose an alternative approach based on deep neural networks that simulates changes in the 3D structure of a monitored object in a batch based on previous 3D measurements. In particular, we propose an architecture based on 3D Convolutional Neural Networks (3DCNN) in order to model the geometric variations in manufacturing parameters and predict upcoming events related to sub-optimal performance. We validate our framework on a microelectronics use-case using the recently published PCB scans dataset where we simulate changes on the shape and volume of glue deposited on an Liquid Crystal Polymer (LCP) substrate before the attachment of integrated circuits (IC). Experimental evaluation examines the impact of different choices in the cost function during training and shows that the proposed method can be efficiently used for defect prediction.
△ Less
Submitted 25 February, 2020;
originally announced February 2020.
-
Fault Diagnosis in Microelectronics Attachment via Deep Learning Analysis of 3D Laser Scans
Authors:
Nikolaos Dimitriou,
Lampros Leontaris,
Thanasis Vafeiadis,
Dimosthenis Ioannidis,
Tracy Wotherspoon,
Gregory Tinker,
Dimitrios Tzovaras
Abstract:
A common source of defects in manufacturing miniature Printed Circuits Boards (PCB) is the attachment of silicon die or other wire bondable components on a Liquid Crystal Polymer (LCP) substrate. Typically, a conductive glue is dispensed prior to attachment with defects caused either by insufficient or excessive glue. The current practice in electronics industry is to examine the deposited glue by…
▽ More
A common source of defects in manufacturing miniature Printed Circuits Boards (PCB) is the attachment of silicon die or other wire bondable components on a Liquid Crystal Polymer (LCP) substrate. Typically, a conductive glue is dispensed prior to attachment with defects caused either by insufficient or excessive glue. The current practice in electronics industry is to examine the deposited glue by a human operator a process that is both time consuming and inefficient especially in preproduction runs where the error rate is high. In this paper we propose a system that automates fault diagnosis by accurately estimating the volume of glue deposits before and even after die attachment. To this end a modular scanning system is deployed that produces high resolution point clouds whereas the actual estimation of glue volume is performed by (R)egression-Net (RNet), a 3D Convolutional Neural Network (3DCNN). RNet outperforms other deep architectures and is able to estimate the volume either directly from the point cloud of a glue deposit or more interestingly after die attachment when only a small part of glue is visible around each die. The entire methodology is evaluated under operational conditions where the proposed system achieves accurate results without delaying the manufacturing process.
△ Less
Submitted 25 February, 2020;
originally announced February 2020.
-
Blockchain 3.0 Smart Contracts in E-Government 3.0 Applications
Authors:
Sofia Terzi,
Konstantinos Votis,
Dimitrios Tzovaras,
Ioannis Stamelos,
Kelly Cooper
Abstract:
The adoption of Information Communication Technologies (ICT) and Web 3.0 contributes to the e-government sector by transforming how public administrations provide advanced and innovative services to interact with citizens. Blockchain (BC) and Artificial Intelligence (AI) disruptive technologies will reshape how we live, work, and interact with government sectors and industries. This paper presents…
▽ More
The adoption of Information Communication Technologies (ICT) and Web 3.0 contributes to the e-government sector by transforming how public administrations provide advanced and innovative services to interact with citizens. Blockchain (BC) and Artificial Intelligence (AI) disruptive technologies will reshape how we live, work, and interact with government sectors and industries. This paper presents how Blockchain 3.0 and Artificial Intelligence enhance robust, secure, scalable, and authenticity provenance solutions. Two validation scenarios are analyzed to present how blockchain smart contracts and AI agents support energy and health-oriented e-government services.
△ Less
Submitted 11 October, 2019;
originally announced October 2019.
-
Image-based Natural Language Understanding Using 2D Convolutional Neural Networks
Authors:
Erinc Merdivan,
Anastasios Vafeiadis,
Dimitrios Kalatzis,
Sten Hanke,
Johannes Kropf,
Konstantinos Votis,
Dimitrios Giakoumis,
Dimitrios Tzovaras,
Liming Chen,
Raouf Hamzaoui,
Matthieu Geist
Abstract:
We propose a new approach to natural language understanding in which we consider the input text as an image and apply 2D Convolutional Neural Networks to learn the local and global semantics of the sentences from the variations ofthe visual patterns of words. Our approach demonstrates that it is possible to get semantically meaningful features from images with text without using optical character…
▽ More
We propose a new approach to natural language understanding in which we consider the input text as an image and apply 2D Convolutional Neural Networks to learn the local and global semantics of the sentences from the variations ofthe visual patterns of words. Our approach demonstrates that it is possible to get semantically meaningful features from images with text without using optical character recognition and sequential processing pipelines, techniques that traditional Natural Language Understanding algorithms require. To validate our approach, we present results for two applications: text classification and dialog modeling. Using a 2D Convolutional Neural Network, we were able to outperform the state-of-art accuracy results of non-Latin alphabet-based text classification and achieved promising results for eight text classification datasets. Furthermore, our approach outperformed the memory networks when using out of vocabulary entities fromtask 4 of the bAbI dialog dataset.
△ Less
Submitted 6 November, 2018; v1 submitted 24 October, 2018;
originally announced October 2018.
-
Admission and Congestion Control for 5G Network Slicing
Authors:
Bin Han,
Antonio De Domenico,
Ghina Dandachi,
Anastasios Drosou,
Dimitrios Tzovaras,
Roberto Querio,
Fabrizio Moggio,
Ömer Bulakci,
Hans D. Schotten
Abstract:
Network Slicing has been widely accepted as essential feature of future 5th Generation (5G) mobile communication networks. Accounting the potentially dense demand of network slices as a cloud service and the limited resource of mobile network operators (MNOs), an efficient inter-slice management and orchestration plays a key role in 5G networks. This calls advanced solutions for slice admission an…
▽ More
Network Slicing has been widely accepted as essential feature of future 5th Generation (5G) mobile communication networks. Accounting the potentially dense demand of network slices as a cloud service and the limited resource of mobile network operators (MNOs), an efficient inter-slice management and orchestration plays a key role in 5G networks. This calls advanced solutions for slice admission and congestion control. This paper proposes a novel approach of inter-slice control that well copes with existing pre-standardized 5G architectures
△ Less
Submitted 31 August, 2018;
originally announced September 2018.
-
Security for Smart Mobile Networks: The NEMESYS Approach
Authors:
Erol Gelenbe,
Gokce Gorbil,
Dimitrios Tzovaras,
Steffen Liebergeld,
David Garcia,
Madalina Baltatu,
George Lyberopoulos
Abstract:
The growing popularity of smart mobile devices such as smartphones and tablets has made them an attractive target for cyber-criminals, resulting in a rapidly growing and evolving mobile threat as attackers experiment with new business models by targeting mobile users. With the emergence of the first large-scale mobile botnets, the core network has also become vulnerable to distributed denial-of-se…
▽ More
The growing popularity of smart mobile devices such as smartphones and tablets has made them an attractive target for cyber-criminals, resulting in a rapidly growing and evolving mobile threat as attackers experiment with new business models by targeting mobile users. With the emergence of the first large-scale mobile botnets, the core network has also become vulnerable to distributed denial-of-service attacks such as the signaling attack. Furthermore, complementary access methods such as Wi-Fi and femtocells introduce additional vulnerabilities for the mobile users as well as the core network. In this paper, we present the NEMESYS approach to smart mobile network security. The goal of the NEMESYS project is to develop novel security technologies for seamless service provisioning in the smart mobile ecosystem, and to improve mobile network security through a better understanding of the threat landscape. To this purpose, NEMESYS will collect and analyze information about the nature of cyber-attacks targeting smart mobile devices and the core network so that appropriate counter-measures can be taken. We are develo** a data collection infrastructure that incorporates virtualized mobile honeypots and honeyclients in order to gather, detect and provide early warning of mobile attacks and understand the modus operandi of cyber-criminals that target mobile devices. By correlating the extracted information with known attack patterns from wireline networks, we plan to reveal and identify the possible shift in the way that cyber-criminals launch attacks against smart mobile devices.
△ Less
Submitted 2 July, 2013;
originally announced July 2013.
-
NEMESYS: Enhanced Network Security for Seamless Service Provisioning in the Smart Mobile Ecosystem
Authors:
Erol Gelenbe,
Gokce Gorbil,
Dimitrios Tzovaras,
Steffen Liebergeld,
David Garcia,
Madalina Baltatu,
George Lyberopoulos
Abstract:
As a consequence of the growing popularity of smart mobile devices, mobile malware is clearly on the rise, with attackers targeting valuable user information and exploiting vulnerabilities of the mobile ecosystems. With the emergence of large-scale mobile botnets, smartphones can also be used to launch attacks on mobile networks. The NEMESYS project will develop novel security technologies for sea…
▽ More
As a consequence of the growing popularity of smart mobile devices, mobile malware is clearly on the rise, with attackers targeting valuable user information and exploiting vulnerabilities of the mobile ecosystems. With the emergence of large-scale mobile botnets, smartphones can also be used to launch attacks on mobile networks. The NEMESYS project will develop novel security technologies for seamless service provisioning in the smart mobile ecosystem, and improve mobile network security through better understanding of the threat landscape. NEMESYS will gather and analyze information about the nature of cyber-attacks targeting mobile users and the mobile network so that appropriate counter-measures can be taken. We will develop a data collection infrastructure that incorporates virtualized mobile honeypots and a honeyclient, to gather, detect and provide early warning of mobile attacks and better understand the modus operandi of cyber-criminals that target mobile devices. By correlating the extracted information with the known patterns of attacks from wireline networks, we will reveal and identify trends in the way that cyber-criminals launch attacks against mobile devices.
△ Less
Submitted 23 May, 2013;
originally announced May 2013.
