Showing 1–15 of 15 results for author: Breslin, J

LatentColorization: Latent Diffusion-Based Speaker Video Colorization

Authors: Rory Ward, Dan Bigioi, Shubhajit Basak, John G. Breslin, Peter Corcoran

Abstract: While current research predominantly focuses on image-based colorization, the domain of video-based colorization remains relatively unexplored. Most existing video colorization techniques operate on a frame-by-frame basis, often overlooking the critical aspect of temporal coherence between successive frames. This approach can result in inconsistencies across frames, leading to undesirable effects… ▽ More While current research predominantly focuses on image-based colorization, the domain of video-based colorization remains relatively unexplored. Most existing video colorization techniques operate on a frame-by-frame basis, often overlooking the critical aspect of temporal coherence between successive frames. This approach can result in inconsistencies across frames, leading to undesirable effects like flickering or abrupt color transitions between frames. To address these challenges, we harness the generative capabilities of a fine-tuned latent diffusion model designed specifically for video colorization, introducing a novel solution for achieving temporal consistency in video colorization, as well as demonstrating strong improvements on established image quality metrics compared to other existing methods. Furthermore, we perform a subjective study, where users preferred our approach to the existing state of the art. Our dataset encompasses a combination of conventional datasets and videos from television/movies. In short, by leveraging the power of a fine-tuned latent diffusion-based colorization system with a temporal consistency mechanism, we can improve the performance of automatic video colorization by addressing the challenges of temporal inconsistency. A short demonstration of our results can be seen in some example videos available at https://youtu.be/vDbzsZdFuxM. △ Less

Submitted 9 May, 2024; originally announced May 2024.

Multi-Component Optimization and Efficient Deployment of Neural-Networks on Resource-Constrained IoT Hardware

Authors: Bharath Sudharsan, Dineshkumar Sundaram, Pankesh Patel, John G. Breslin, Muhammad Intizar Ali, Schahram Dustdar, Albert Zomaya, Rajiv Ranjan

Abstract: The majority of IoT devices like smartwatches, smart plugs, HVAC controllers, etc., are powered by hardware with a constrained specification (low memory, clock speed and processor) which is insufficient to accommodate and execute large, high-quality models. On such resource-constrained devices, manufacturers still manage to provide attractive functionalities (to boost sales) by following the tradi… ▽ More The majority of IoT devices like smartwatches, smart plugs, HVAC controllers, etc., are powered by hardware with a constrained specification (low memory, clock speed and processor) which is insufficient to accommodate and execute large, high-quality models. On such resource-constrained devices, manufacturers still manage to provide attractive functionalities (to boost sales) by following the traditional approach of programming IoT devices/products to collect and transmit data (image, audio, sensor readings, etc.) to their cloud-based ML analytics platforms. For decades, this online approach has been facing issues such as compromised data streams, non-real-time analytics due to latency, bandwidth constraints, costly subscriptions, recent privacy issues raised by users and the GDPR guidelines, etc. In this paper, to enable ultra-fast and accurate AI-based offline analytics on resource-constrained IoT devices, we present an end-to-end multi-component model optimization sequence and open-source its implementation. Researchers and developers can use our optimization sequence to optimize high memory, computation demanding models in multiple aspects in order to produce small size, low latency, low-power consuming models that can comfortably fit and execute on resource-constrained hardware. The experimental results show that our optimization components can produce models that are; (i) 12.06 x times compressed; (ii) 0.13% to 0.27% more accurate; (iii) Orders of magnitude faster unit inference at 0.06 ms. Our optimization sequence is generic and can be applied to any state-of-the-art models trained for anomaly detection, predictive maintenance, robotics, voice recognition, and machine vision. △ Less

Submitted 20 April, 2022; originally announced April 2022.

A Demonstration of Smart Doorbell Design Using Federated Deep Learning

Authors: Vatsal Patel, Sarth Kanani, Tapan Pathak, Pankesh Patel, Muhammad Intizar Ali, John Breslin

Abstract: Smart doorbells have been playing an important role in protecting our modern homes. Existing approaches of sending video streams to a centralized server (or Cloud) for video analytics have been facing many challenges such as latency, bandwidth cost and more importantly users' privacy concerns. To address these challenges, this paper showcases the ability of an intelligent smart doorbell based on F… ▽ More Smart doorbells have been playing an important role in protecting our modern homes. Existing approaches of sending video streams to a centralized server (or Cloud) for video analytics have been facing many challenges such as latency, bandwidth cost and more importantly users' privacy concerns. To address these challenges, this paper showcases the ability of an intelligent smart doorbell based on Federated Deep Learning, which can deploy and manage video analytics applications such as a smart doorbell across Edge and Cloud resources. This platform can scale, work with multiple devices, seamlessly manage online orchestration of the application components. The proposed framework is implemented using state-of-the-art technology. We implement the Federated Server using the Flask framework, containerized using Nginx and Gunicorn, which is deployed on AWS EC2 and AWS Serverless architecture. △ Less

Submitted 19 October, 2020; originally announced October 2020.

Comments: 6

Demonstration of a Cloud-based Software Framework for Video Analytics Application using Low-Cost IoT Devices

Authors: Bhavin Joshi, Tapan Pathak, Vatsal Patel, Sarth Kanani, Pankesh Patel, Muhammad Intizar Ali, John Breslin

Abstract: The design of products and services such as a Smart doorbell, demonstrating video analytics software/algorithm functionality, is expected to address a new kind of requirements such as designing a scalable solution while considering the trade-off between cost and accuracy; a flexible architecture to deploy new AI-based models or update existing models, as user requirements evolve; as well as seamle… ▽ More The design of products and services such as a Smart doorbell, demonstrating video analytics software/algorithm functionality, is expected to address a new kind of requirements such as designing a scalable solution while considering the trade-off between cost and accuracy; a flexible architecture to deploy new AI-based models or update existing models, as user requirements evolve; as well as seamlessly integrating different kinds of user interfaces and devices. To address these challenges, we propose a smart doorbell that orchestrates video analytics across Edge and Cloud resources. The proposal uses AWS as a base platform for implementation and leverages Commercially Available Off-The-Shelf(COTS) affordable devices such as Raspberry Pi in the form of an Edge device. △ Less

Submitted 29 September, 2020; originally announced October 2020.

Comments: arXiv admin note: substantial text overlap with arXiv:2009.09065

Block Chain and Internet of Nano-Things for Optimizing Chemical Sensing in Smart Farming

Authors: Dixon Vimalajeewa, Subhasis Thakur, John Breslin, Donagh P. Berry, Sasitharan Balasubramaniam

Abstract: The use of Internet of Things (IoT) with the Internet of Nano Things (IoNT) can further expand decision making systems (DMS) to improve reliability as it provides a new spectrum of more granular level data to make decisions. However, growing concerns such as data security, transparency and processing capability challenge their use in real-world applications. DMS integrated with Block Chain (BC) te… ▽ More The use of Internet of Things (IoT) with the Internet of Nano Things (IoNT) can further expand decision making systems (DMS) to improve reliability as it provides a new spectrum of more granular level data to make decisions. However, growing concerns such as data security, transparency and processing capability challenge their use in real-world applications. DMS integrated with Block Chain (BC) technology can contribute immensely to overcome such challenges. The use of IoNT and IoT along with BC for making DMS has not yet been investigated. This study proposes a BC-powered IoNT (BC-IoNT) system for sensing chemicals level in the context of farm management. This is a critical application for smart farming, which aims to improve sustainable farm practices through controlled delivery of chemicals. BC-IoNT system includes a novel machine learning model formed by using the Langmuir molecular binding model and the Bayesian theory, and is used as a smart contract for sensing the level of the chemicals. A credit model is used to quantify the traceability and credibility of farms to determine if they are compliant with the chemical standards. The accuracy of detecting the chemicals of the distributed BC-IoNT approach was >90% and the centralized approach was <80%. Also, the efficiency of sensing the level of chemicals depends on the sampling frequency and variability in chemical level among farms. △ Less

Submitted 5 October, 2020; originally announced October 2020.

Comments: 16 pages, 12 figures

A Distributed Framework to Orchestrate Video Analytics Applications

Authors: Tapan Pathak, Vatsal Patel, Sarth Kanani, Shailesh Arya, Pankesh Patel, Muhammad Intizar Ali, John Breslin

Abstract: The concept of the Internet of Things (IoT) is a reality now. This paradigm shift has caught everyones attention in a large class of applications, including IoT-based video analytics using smart doorbells. Due to its growing application segments, various efforts exist in scientific literature and many video-based doorbell solutions are commercially available in the market. However, contemporary of… ▽ More The concept of the Internet of Things (IoT) is a reality now. This paradigm shift has caught everyones attention in a large class of applications, including IoT-based video analytics using smart doorbells. Due to its growing application segments, various efforts exist in scientific literature and many video-based doorbell solutions are commercially available in the market. However, contemporary offerings are bespoke, offering limited composability and reusability of a smart doorbell framework. Second, they are monolithic and proprietary, which means that the implementation details remain hidden from the users. We believe that a transparent design can greatly aid in the development of a smart doorbell, enabling its use in multiple application domains. To address the above-mentioned challenges, we propose a distributed framework to orchestrate video analytics across Edge and Cloud resources. We investigate trade-offs in the distribution of different software components over a bespoke/full system, where components over Edge and Cloud are treated generically. This paper evaluates the proposed framework as well as the state-of-the-art models and presents comparative analysis of them on various metrics (such as overall model accuracy, latency, memory, and CPU usage). The evaluation result demonstrates our intuition very well, showcasing that the AWS-based approach exhibits reasonably high object-detection accuracy, low memory, and CPU usage when compared to the state-of-the-art approaches, but high latency. △ Less

Submitted 17 September, 2020; originally announced September 2020.

Comments: 9

Inferring User Interests in Microblogging Social Networks: A Survey

Authors: Guangyuan Piao, John G. Breslin

Abstract: With the growing popularity of microblogging services such as Twitter in recent years, an increasing number of users are using these services in their daily lives. The huge volume of information generated by users raises new opportunities in various applications and areas. Inferring user interests plays a significant role in providing personalized recommendations on microblogging services, and als… ▽ More With the growing popularity of microblogging services such as Twitter in recent years, an increasing number of users are using these services in their daily lives. The huge volume of information generated by users raises new opportunities in various applications and areas. Inferring user interests plays a significant role in providing personalized recommendations on microblogging services, and also on third-party applications providing social logins via these services, especially in cold-start situations. In this survey, we review user modeling strategies with respect to inferring user interests from previous studies. To this end, we focus on four dimensions of inferring user interest profiles: (1) data collection, (2) representation of user interest profiles, (3) construction and enhancement of user interest profiles, and (4) the evaluation of the constructed profiles. Through this survey, we aim to provide an overview of state-of-the-art user modeling strategies for inferring user interest profiles on microblogging social networks with respect to the four dimensions. For each dimension, we review and summarize previous studies based on specified criteria. Finally, we discuss some challenges and opportunities for future work in this research domain. △ Less

Submitted 14 August, 2018; v1 submitted 20 December, 2017; originally announced December 2017.

Comments: pre-print, accepted at UMUAI, final version DOI 10.1007/s11257-018-9207-8

Factorization Machines Leveraging Lightweight Linked Open Data-enabled Features for Top-N Recommendations

Authors: Guangyuan Piao, John G. Breslin

Abstract: With the popularity of Linked Open Data (LOD) and the associated rise in freely accessible knowledge that can be accessed via LOD, exploiting LOD for recommender systems has been widely studied based on various approaches such as graph-based or using different machine learning models with LOD-enabled features. Many of the previous approaches require construction of an additional graph to run graph… ▽ More With the popularity of Linked Open Data (LOD) and the associated rise in freely accessible knowledge that can be accessed via LOD, exploiting LOD for recommender systems has been widely studied based on various approaches such as graph-based or using different machine learning models with LOD-enabled features. Many of the previous approaches require construction of an additional graph to run graph-based algorithms or to extract path-based features by combining user- item interactions (e.g., likes, dislikes) and background knowledge from LOD. In this paper, we investigate Factorization Machines (FMs) based on particularly lightweight LOD-enabled features which can be directly obtained via a public SPARQL Endpoint without any additional effort to construct a graph. Firstly, we aim to study whether using FM with these lightweight LOD-enabled features can provide competitive performance compared to a learning-to-rank approach leveraging LOD as well as other well-established approaches such as kNN-item and BPRMF. Secondly, we are interested in finding out to what extent each set of LOD-enabled features contributes to the recommendation performance. Experimental evaluation on a standard dataset shows that our proposed approach using FM with lightweight LOD-enabled features provides the best performance compared to other approaches in terms of five evaluation metrics. In addition, the study of the recommendation performance based on different sets of LOD-enabled features indicate that property-object lists and PageRank scores of items are useful for improving the performance, and can provide the best performance through using them together for FM. We observe that subject-property lists of items does not contribute to the recommendation performance but rather decreases the performance. △ Less

Submitted 28 July, 2017; v1 submitted 18 July, 2017; originally announced July 2017.

Comments: This draft has been accepted at WISE2017

Data Selection Strategies for Multi-Domain Sentiment Analysis

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: Domain adaptation is important in sentiment analysis as sentiment-indicating words vary between domains. Recently, multi-domain adaptation has become more pervasive, but existing approaches train on all available source domains including dissimilar ones. However, the selection of appropriate training data is as important as the choice of algorithm. We undertake -- to our knowledge for the first ti… ▽ More Domain adaptation is important in sentiment analysis as sentiment-indicating words vary between domains. Recently, multi-domain adaptation has become more pervasive, but existing approaches train on all available source domains including dissimilar ones. However, the selection of appropriate training data is as important as the choice of algorithm. We undertake -- to our knowledge for the first time -- an extensive study of domain similarity metrics in the context of sentiment analysis and propose novel representations, metrics, and a new scope for data selection. We evaluate the proposed methods on two large-scale multi-domain adaptation settings on tweets and reviews and demonstrate that they consistently outperform strong random and balanced baselines, while our proposed selection strategy outperforms instance-level selection and yields the best score on a large reviews corpus. △ Less

Submitted 8 February, 2017; originally announced February 2017.

Comments: 10 pages, 2 figures, 4 tables

Knowledge Adaptation: Teaching to Adapt

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: Domain adaptation is crucial in many real-world applications where the distribution of the training data differs from the distribution of the test data. Previous Deep Learning-based approaches to domain adaptation need to be trained jointly on source and target domain data and are therefore unappealing in scenarios where models need to be adapted to a large number of domains or where a domain is e… ▽ More Domain adaptation is crucial in many real-world applications where the distribution of the training data differs from the distribution of the test data. Previous Deep Learning-based approaches to domain adaptation need to be trained jointly on source and target domain data and are therefore unappealing in scenarios where models need to be adapted to a large number of domains or where a domain is evolving, e.g. spam detection where attackers continuously change their tactics. To fill this gap, we propose Knowledge Adaptation, an extension of Knowledge Distillation (Bucilua et al., 2006; Hinton et al., 2015) to the domain adaptation scenario. We show how a student model achieves state-of-the-art results on unsupervised domain adaptation from multiple sources on a standard sentiment analysis benchmark by taking into account the domain-specific expertise of multiple teachers and the similarities between their domains. When learning from a single teacher, using domain similarity to gauge trustworthiness is inadequate. To this end, we propose a simple metric that correlates well with the teacher's accuracy in the target domain. We demonstrate that incorporating high-confidence examples selected by this metric enables the student model to achieve state-of-the-art performance in the single-source scenario. △ Less

Submitted 7 February, 2017; originally announced February 2017.

Comments: 11 pages, 4 figures, 2 tables

Towards a continuous modeling of natural language domains

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: Humans continuously adapt their style and language to a variety of domains. However, a reliable definition of `domain' has eluded researchers thus far. Additionally, the notion of discrete domains stands in contrast to the multiplicity of heterogeneous domains that humans navigate, many of which overlap. In order to better understand the change and variation of human language, we draw on research… ▽ More Humans continuously adapt their style and language to a variety of domains. However, a reliable definition of `domain' has eluded researchers thus far. Additionally, the notion of discrete domains stands in contrast to the multiplicity of heterogeneous domains that humans navigate, many of which overlap. In order to better understand the change and variation of human language, we draw on research in domain adaptation and extend the notion of discrete domains to the continuous spectrum. We propose representation learning-based models that can adapt to continuous domains and detail how these can be used to investigate variation in language. To this end, we propose to use dialogue modeling as a test bed due to its proximity to language modeling and its social component. △ Less

Submitted 28 October, 2016; originally announced October 2016.

Comments: 5 pages, 3 figures, published in Uphill Battles in Language Processing workshop, EMNLP 2016

Character-level and Multi-channel Convolutional Neural Networks for Large-scale Authorship Attribution

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: Convolutional neural networks (CNNs) have demonstrated superior capability for extracting information from raw signals in computer vision. Recently, character-level and multi-channel CNNs have exhibited excellent performance for sentence classification tasks. We apply CNNs to large-scale authorship attribution, which aims to determine an unknown text's author among many candidate authors, motivate… ▽ More Convolutional neural networks (CNNs) have demonstrated superior capability for extracting information from raw signals in computer vision. Recently, character-level and multi-channel CNNs have exhibited excellent performance for sentence classification tasks. We apply CNNs to large-scale authorship attribution, which aims to determine an unknown text's author among many candidate authors, motivated by their ability to process character-level signals and to differentiate between a large number of classes, while making fast predictions in comparison to state-of-the-art approaches. We extensively evaluate CNN-based approaches that leverage word and character channels and compare them against state-of-the-art methods for a large range of author numbers, shedding new light on traditional approaches. We show that character-level CNNs outperform the state-of-the-art on four out of five datasets in different domains. Additionally, we present the first application of authorship attribution to reddit. △ Less

Submitted 21 September, 2016; originally announced September 2016.

Comments: 9 pages, 5 figures, 3 tables

INSIGHT-1 at SemEval-2016 Task 5: Deep Learning for Multilingual Aspect-based Sentiment Analysis

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: This paper describes our deep learning-based approach to multilingual aspect-based sentiment analysis as part of SemEval 2016 Task 5. We use a convolutional neural network (CNN) for both aspect extraction and aspect-based sentiment analysis. We cast aspect extraction as a multi-label classification problem, outputting probabilities over aspects parameterized by a threshold. To determine the sentim… ▽ More This paper describes our deep learning-based approach to multilingual aspect-based sentiment analysis as part of SemEval 2016 Task 5. We use a convolutional neural network (CNN) for both aspect extraction and aspect-based sentiment analysis. We cast aspect extraction as a multi-label classification problem, outputting probabilities over aspects parameterized by a threshold. To determine the sentiment towards an aspect, we concatenate an aspect vector with every word embedding and apply a convolution over it. Our constrained system (unconstrained for English) achieves competitive results across all languages and domains, placing first or second in 5 and 7 out of 11 language-domain pairs for aspect category detection (slot 1) and sentiment polarity (slot 3) respectively, thereby demonstrating the viability of a deep learning-based approach for multilingual aspect-based sentiment analysis. △ Less

Submitted 22 September, 2016; v1 submitted 9 September, 2016; originally announced September 2016.

Comments: Published in Proceedings of SemEval-2016, 7 pages

Journal ref: Proceedings of SemEval (2016): 330-336

INSIGHT-1 at SemEval-2016 Task 4: Convolutional Neural Networks for Sentiment Classification and Quantification

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: This paper describes our deep learning-based approach to sentiment analysis in Twitter as part of SemEval-2016 Task 4. We use a convolutional neural network to determine sentiment and participate in all subtasks, i.e. two-point, three-point, and five-point scale sentiment classification and two-point and five-point scale sentiment quantification. We achieve competitive results for two-point scale… ▽ More This paper describes our deep learning-based approach to sentiment analysis in Twitter as part of SemEval-2016 Task 4. We use a convolutional neural network to determine sentiment and participate in all subtasks, i.e. two-point, three-point, and five-point scale sentiment classification and two-point and five-point scale sentiment quantification. We achieve competitive results for two-point scale sentiment classification and quantification, ranking fifth and a close fourth (third and second by alternative metrics) respectively despite using only pre-trained embeddings that contain no sentiment information. We achieve good performance on three-point scale sentiment classification, ranking eighth out of 35, while performing poorly on five-point scale sentiment classification and quantification. An error analysis reveals that this is due to low expressiveness of the model to capture negative sentiment as well as an inability to take into account ordinal information. We propose improvements in order to address these and other issues. △ Less

Submitted 9 September, 2016; originally announced September 2016.

Comments: Published in Proceedings of SemEval-2016, 5 pages

Journal ref: Proceedings of SemEval (2016): 178-182

A Hierarchical Model of Reviews for Aspect-based Sentiment Analysis

Authors: Sebastian Ruder, Parsa Ghaffari, John G. Breslin

Abstract: Opinion mining from customer reviews has become pervasive in recent years. Sentences in reviews, however, are usually classified independently, even though they form part of a review's argumentative structure. Intuitively, sentences in a review build and elaborate upon each other; knowledge of the review structure and sentential context should thus inform the classification of each sentence. We de… ▽ More Opinion mining from customer reviews has become pervasive in recent years. Sentences in reviews, however, are usually classified independently, even though they form part of a review's argumentative structure. Intuitively, sentences in a review build and elaborate upon each other; knowledge of the review structure and sentential context should thus inform the classification of each sentence. We demonstrate this hypothesis for the task of aspect-based sentiment analysis by modeling the interdependencies of sentences in a review with a hierarchical bidirectional LSTM. We show that the hierarchical model outperforms two non-hierarchical baselines, obtains results competitive with the state-of-the-art, and outperforms the state-of-the-art on five multilingual, multi-domain datasets without any hand-engineered features or external resources. △ Less

Submitted 9 September, 2016; originally announced September 2016.

Comments: To be published at EMNLP 2016, 7 pages