-
Nadine: An LLM-driven Intelligent Social Robot with Affective Capabilities and Human-like Memory
Authors:
Hangyeol Kang,
Maher Ben Moussa,
Nadia Magnenat-Thalmann
Abstract:
In this work, we describe our approach to develo** an intelligent and robust social robotic system for the Nadine social robot platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models to achieve advanced human-like affective and cognitive capabilities. This approach is nov…
▽ More
In this work, we describe our approach to develo** an intelligent and robust social robotic system for the Nadine social robot platform. We achieve this by integrating Large Language Models (LLMs) and skilfully leveraging the powerful reasoning and instruction-following capabilities of these types of models to achieve advanced human-like affective and cognitive capabilities. This approach is novel compared to the current state-of-the-art LLM-based agents which do not implement human-like long-term memory or sophisticated emotional appraisal. The naturalness of social robots, consisting of multiple modules, highly depends on the performance and capabilities of each component of the system and the seamless integration of the components. We built a social robot system that enables generating appropriate behaviours through multimodal input processing, bringing episodic memories accordingly to the recognised user, and simulating the emotional states of the robot induced by the interaction with the human partner. In particular, we introduce an LLM-agent frame for social robots, SoR-ReAct, serving as a core component for the interaction module in our system. This design has brought forth the advancement of social robots and aims to increase the quality of human-robot interaction.
△ Less
Submitted 30 May, 2024;
originally announced May 2024.
-
A computational medical XR discipline
Authors:
George Papagiannakis,
Walter Greenleaf,
Michael Cole,
Mark Zhang,
Rabi Datta,
Mathias Delahaye,
Eleni Grigoriou,
Manos Kamarianakis,
Antonis Protopsaltis,
Philippe Bijlenga,
Nadia Magnenat-Thalmann,
Eleftherios Tsiridis,
Eustathios Kenanidis,
Kyriakos Vamvakidis,
Ioannis Koutelidakis,
Oliver A Kannape
Abstract:
Computational Medical Extended Reality (CMXR), brings together life sciences and neuroscience with mathematics, engineering and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It significantly differs from previous "Clinical XR" or "Medical XR" terms, as it is focusing on how to integrate compu…
▽ More
Computational Medical Extended Reality (CMXR), brings together life sciences and neuroscience with mathematics, engineering and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It significantly differs from previous "Clinical XR" or "Medical XR" terms, as it is focusing on how to integrate computational methods from neural simulation to computational geometry, computational vision and computer graphics with deep learning models to solve specific hard problems in medicine and neuroscience: from low/no-code/genAI authoring platforms to deep learning XR systems for training, planning, operative navigation, therapy and rehabilitation.
△ Less
Submitted 1 July, 2024; v1 submitted 9 August, 2021;
originally announced August 2021.
-
Can a Humanoid Robot be part of the Organizational Workforce? A User Study Leveraging Sentiment Analysis
Authors:
Nidhi Mishra,
Manoj Ramanathan,
Ranjan Satapathy,
Erik Cambria,
Nadia Magnenat-Thalmann
Abstract:
Hiring robots for the workplaces is a challenging task as robots have to cater to customer demands, follow organizational protocols and behave with social etiquette. In this study, we propose to have a humanoid social robot, Nadine, as a customer service agent in an open social work environment. The objective of this study is to analyze the effects of humanoid robots on customers at work environme…
▽ More
Hiring robots for the workplaces is a challenging task as robots have to cater to customer demands, follow organizational protocols and behave with social etiquette. In this study, we propose to have a humanoid social robot, Nadine, as a customer service agent in an open social work environment. The objective of this study is to analyze the effects of humanoid robots on customers at work environment, and see if it can handle social scenarios. We propose to evaluate these objectives through two modes, namely, survey questionnaire and customer feedback. We also propose a novel approach to analyze customer feedback data (text) using sentic computing methods. Specifically, we employ aspect extraction and sentiment analysis to analyze the data. From our framework, we detect sentiment associated to the aspects that mainly concerned the customers during their interaction. This allows us to understand customers expectations and current limitations of robots as employees.
△ Less
Submitted 30 May, 2019; v1 submitted 21 May, 2019;
originally announced May 2019.
-
SLRMA: Sparse Low-Rank Matrix Approximation for Data Compression
Authors:
Junhui Hou,
Lap-Pui Chau,
Nadia Magnenat-Thalmann,
Ying He
Abstract:
Low-rank matrix approximation (LRMA) is a powerful technique for signal processing and pattern analysis. However, its potential for data compression has not yet been fully investigated in the literature. In this paper, we propose sparse low-rank matrix approximation (SLRMA), an effective computational tool for data compression. SLRMA extends the conventional LRMA by exploring both the intra- and i…
▽ More
Low-rank matrix approximation (LRMA) is a powerful technique for signal processing and pattern analysis. However, its potential for data compression has not yet been fully investigated in the literature. In this paper, we propose sparse low-rank matrix approximation (SLRMA), an effective computational tool for data compression. SLRMA extends the conventional LRMA by exploring both the intra- and inter-coherence of data samples simultaneously. With the aid of prescribed orthogonal transforms (e.g., discrete cosine/wavelet transform and graph transform), SLRMA decomposes a matrix into a product of two smaller matrices, where one matrix is made of extremely sparse and orthogonal column vectors, and the other consists of the transform coefficients. Technically, we formulate SLRMA as a constrained optimization problem, i.e., minimizing the approximation error in the least-squares sense regularized by $\ell_0$-norm and orthogonality, and solve it using the inexact augmented Lagrangian multiplier method. Through extensive tests on real-world data, such as 2D image sets and 3D dynamic meshes, we observe that (i) SLRMA empirically converges well; (ii) SLRMA can produce approximation error comparable to LRMA but in a much sparse form; (iii) SLRMA-based compression schemes significantly outperform the state-of-the-art in terms of rate-distortion performance.
△ Less
Submitted 19 February, 2016; v1 submitted 7 July, 2015;
originally announced July 2015.
-
Low-latency compression of mocap data using learned spatial decorrelation transform
Authors:
Junhui Hou,
Lap-Pui Chau,
Nadia Magnenat-Thalmann,
Ying He
Abstract:
Due to the growing needs of human motion capture (mocap) in movie, video games, sports, etc., it is highly desired to compress mocap data for efficient storage and transmission. This paper presents two efficient frameworks for compressing human mocap data with low latency. The first framework processes the data in a frame-by-frame manner so that it is ideal for mocap data streaming and time critic…
▽ More
Due to the growing needs of human motion capture (mocap) in movie, video games, sports, etc., it is highly desired to compress mocap data for efficient storage and transmission. This paper presents two efficient frameworks for compressing human mocap data with low latency. The first framework processes the data in a frame-by-frame manner so that it is ideal for mocap data streaming and time critical applications. The second one is clip-based and provides a flexible tradeoff between latency and compression performance. Since mocap data exhibits some unique spatial characteristics, we propose a very effective transform, namely learned orthogonal transform (LOT), for reducing the spatial redundancy. The LOT problem is formulated as minimizing square error regularized by orthogonality and sparsity and solved via alternating iteration. We also adopt a predictive coding and temporal DCT for temporal decorrelation in the frame- and clip-based frameworks, respectively. Experimental results show that the proposed frameworks can produce higher compression performance at lower computational cost and latency than the state-of-the-art methods.
△ Less
Submitted 18 February, 2016; v1 submitted 29 June, 2015;
originally announced June 2015.
-
Human Motion Capture Data Tailored Transform Coding
Authors:
Junhui Hou,
Lap-Pui Chau,
Nadia Magnenat-Thalmann,
Ying He
Abstract:
Human motion capture (mocap) is a widely used technique for digitalizing human movements. With growing usage, compressing mocap data has received increasing attention, since compact data size enables efficient storage and transmission. Our analysis shows that mocap data have some unique characteristics that distinguish themselves from images and videos. Therefore, directly borrowing image or video…
▽ More
Human motion capture (mocap) is a widely used technique for digitalizing human movements. With growing usage, compressing mocap data has received increasing attention, since compact data size enables efficient storage and transmission. Our analysis shows that mocap data have some unique characteristics that distinguish themselves from images and videos. Therefore, directly borrowing image or video compression techniques, such as discrete cosine transform, does not work well. In this paper, we propose a novel mocap-tailored transform coding algorithm that takes advantage of these features. Our algorithm segments the input mocap sequences into clips, which are represented in 2D matrices. Then it computes a set of data-dependent orthogonal bases to transform the matrices to frequency domain, in which the transform coefficients have significantly less dependency. Finally, the compression is obtained by entropy coding of the quantized coefficients and the bases. Our method has low computational cost and can be easily extended to compress mocap databases. It also requires neither training nor complicated parameter setting. Experimental results demonstrate that the proposed scheme significantly outperforms state-of-the-art algorithms in terms of compression performance and speed.
△ Less
Submitted 17 October, 2014;
originally announced October 2014.
