-
Bridging the Communication Gap: Artificial Agents Learning Sign Language through Imitation
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Artificial agents, particularly humanoid robots, interact with their environment, objects, and people using cameras, actuators, and physical presence. Their communication methods are often pre-programmed, limiting their actions and interactions. Our research explores acquiring non-verbal communication skills through learning from demonstrations, with potential applications in sign language compreh…
▽ More
Artificial agents, particularly humanoid robots, interact with their environment, objects, and people using cameras, actuators, and physical presence. Their communication methods are often pre-programmed, limiting their actions and interactions. Our research explores acquiring non-verbal communication skills through learning from demonstrations, with potential applications in sign language comprehension and expression. In particular, we focus on imitation learning for artificial agents, exemplified by teaching a simulated humanoid American Sign Language. We use computer vision and deep learning to extract information from videos, and reinforcement learning to enable the agent to replicate observed actions. Compared to other methods, our approach eliminates the need for additional hardware to acquire information. We demonstrate how the combination of these different techniques offers a viable way to learn sign language. Our methodology successfully teaches 5 different signs involving the upper body (i.e., arms and hands). This research paves the way for advanced communication skills in artificial agents.
△ Less
Submitted 14 June, 2024;
originally announced June 2024.
-
Signs of Language: Embodied Sign Language Fingerspelling Acquisition from Demonstrations for Human-Robot Interaction
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Learning fine-grained movements is a challenging topic in robotics, particularly in the context of robotic hands. One specific instance of this challenge is the acquisition of fingerspelling sign language in robots. In this paper, we propose an approach for learning dexterous motor imitation from video examples without additional information. To achieve this, we first build a URDF model of a robot…
▽ More
Learning fine-grained movements is a challenging topic in robotics, particularly in the context of robotic hands. One specific instance of this challenge is the acquisition of fingerspelling sign language in robots. In this paper, we propose an approach for learning dexterous motor imitation from video examples without additional information. To achieve this, we first build a URDF model of a robotic hand with a single actuator for each joint. We then leverage pre-trained deep vision models to extract the 3D pose of the hand from RGB videos. Next, using state-of-the-art reinforcement learning algorithms for motion imitation (namely, proximal policy optimization and soft actor-critic), we train a policy to reproduce the movement extracted from the demonstrations. We identify the optimal set of hyperparameters for imitation based on a reference motion. Finally, we demonstrate the generalizability of our approach by testing it on six different tasks, corresponding to fingerspelled letters. Our results show that our approach is able to successfully imitate these fine-grained movements without additional information, highlighting its potential for real-world applications in robotics.
△ Less
Submitted 5 June, 2023; v1 submitted 12 September, 2022;
originally announced September 2022.
-
WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language
Authors:
Federico Tavella,
Viktor Schlegel,
Marta Romeo,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonolo…
▽ More
Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.
△ Less
Submitted 11 March, 2022;
originally announced March 2022.
-
Phonology Recognition in American Sign Language
Authors:
Federico Tavella,
Aphrodite Galata,
Angelo Cangelosi
Abstract:
Inspired by recent developments in natural language processing, we propose a novel approach to sign language processing based on phonological properties validated by American Sign Language users. By taking advantage of datasets composed of phonological data and people speaking sign language, we use a pretrained deep model based on mesh reconstruction to extract the 3D coordinates of the signers ke…
▽ More
Inspired by recent developments in natural language processing, we propose a novel approach to sign language processing based on phonological properties validated by American Sign Language users. By taking advantage of datasets composed of phonological data and people speaking sign language, we use a pretrained deep model based on mesh reconstruction to extract the 3D coordinates of the signers keypoints. Then, we train standard statistical and deep machine learning models in order to assign phonological classes to each temporal sequence of coordinates.
Our paper introduces the idea of exploiting the phonological properties manually assigned by sign language users to classify videos of people performing signs by regressing a 3D mesh. We establish a new baseline for this problem based on the statistical distribution of 725 different signs. Our best-performing models achieve a micro-averaged F1-score of 58% for the major location class and 70% for the sign type using statistical and deep learning algorithms, compared to their corresponding baselines of 35% and 39%.
△ Less
Submitted 1 October, 2021;
originally announced October 2021.
-
A Machine Learning-based Approach to Detect Threats in Bio-Cyber DNA Storage Systems
Authors:
Federico Tavella,
Alberto Giaretta,
Mauro Conti,
Sasitharan Balasubramaniam
Abstract:
Data storage is one of the main computing issues of this century. Not only storage devices are converging to strict physical limits, but also the amount of data generated by users is growing at an unbelievable rate. To face these challenges, data centres grew constantly over the past decades. However, this growth comes with a price, particularly from the environmental point of view. Among various…
▽ More
Data storage is one of the main computing issues of this century. Not only storage devices are converging to strict physical limits, but also the amount of data generated by users is growing at an unbelievable rate. To face these challenges, data centres grew constantly over the past decades. However, this growth comes with a price, particularly from the environmental point of view. Among various promising media, DNA is one of the most fascinating candidate. In our previous work, we have proposed an automated archival architecture which uses bioengineered bacteria to store and retrieve data, previously encoded into DNA. This storage technique is one example of how biological media can deliver power-efficient storing solutions. The similarities between these biological media and classical ones can also be a drawback, as malicious parties might replicate traditional attacks on the former archival system, using biological instruments and techniques. In this paper, first we analyse the main characteristics of our storage system and the different types of attacks that could be executed on it. Then, aiming at identifying on-going attacks, we propose and evaluate detection techniques, which rely on traditional metrics and machine learning algorithms. We identify and adapt two suitable metrics for this purpose, namely generalized entropy and information distance. Moreover, our trained models achieve an AUROC over 0.99 and AUPRC over 0.91.
△ Less
Submitted 28 September, 2020;
originally announced September 2020.
-
Enabling high repetition rate nonlinear THz science with a kilowatt-class sub-100 fs laser source
Authors:
Patrick L. Kramer,
Matthew Windeler,
Katalin Mecseki,
Elio G. Champenois,
Matthias C. Hoffmann,
Franz Tavella
Abstract:
Manipulating the atomic and electronic structure of matter with strong terahertz (THz) fields while probing the response with ultrafast pulses at x-ray free electron lasers (FELs) has offered unique insights into a multitude of physical phenomena in solid state and atomic physics. Recent upgrades of x-ray FEL facilities are pushing to much higher repetition rates, enabling unprecedented signal to…
▽ More
Manipulating the atomic and electronic structure of matter with strong terahertz (THz) fields while probing the response with ultrafast pulses at x-ray free electron lasers (FELs) has offered unique insights into a multitude of physical phenomena in solid state and atomic physics. Recent upgrades of x-ray FEL facilities are pushing to much higher repetition rates, enabling unprecedented signal to noise for pump probe experiments. This requires the development of suitable THz pump sources that are able to deliver intense pulses at compatible repetition rates. Here we present a high power laser-driven THz source based on optical rectification in LiNbO3 using tilted pulse front pum**. Our source is driven by a kilowatt-level Yb:YAG amplifier system operating at 100 kHz repetition rate and employing nonlinear spectral broadening and recompression to achieve sub-100 fs pulses at 1030 nm wavelength. We demonstrate a maximum of 144 mW average THz power (1.44 uJ pulse energy), consisting of single-cycle pulses centered at 0.6 THz with a peak electric field strength exceeding 150 kV/cm. These high field pulses open up a range of possibilities for nonlinear time-resolved experiments with x-ray probing at unprecedented rates.
△ Less
Submitted 12 February, 2020;
originally announced February 2020.
-
Melting and phase change for laser-shocked iron
Authors:
S. White,
B. Kettle,
C. L. S. Lewis,
D. Riley,
J. Vorberger,
S. H. Glenzer,
E. Gamboa,
B. Nagler,
F. Tavella,
H. J. Lee,
C. D. Murphy,
D. O. Gericke
Abstract:
Using the LCLS facility at the SLAC National Accelerator Laboratory, we have observed X-ray scattering from iron compressed with laser driven shocks to Earth-core like pressures above 400GPa. The data shows shots where melting is incomplete and we observe hexagonal close packed (hcp) crystal structure at shock compressed densities up to 14.0 gcm-3 but no evidence of a double-hexagonal close packed…
▽ More
Using the LCLS facility at the SLAC National Accelerator Laboratory, we have observed X-ray scattering from iron compressed with laser driven shocks to Earth-core like pressures above 400GPa. The data shows shots where melting is incomplete and we observe hexagonal close packed (hcp) crystal structure at shock compressed densities up to 14.0 gcm-3 but no evidence of a double-hexagonal close packed (dhcp) crystal. The observation of a crystalline structure at these densities, where shock heating is expected to be in excess of the equilibrium melt temperature, may indicate superheating of the solid. These results are important for equation of state modelling at high strain rates relevant for impact scenarios and laser-driven shock wave experiments.
△ Less
Submitted 23 November, 2018;
originally announced November 2018.
-
DNA Molecular Storage System: Transferring Digitally Encoded Information through Bacterial Nanonetworks
Authors:
Federico Tavella,
Alberto Giaretta,
Triona Marie Dooley-Cullinane,
Mauro Conti,
Lee Coffey,
Sasitharan Balasubramaniam
Abstract:
Since the birth of computer and networks, fuelled by pervasive computing and ubiquitous connectivity, the amount of data stored and transmitted has exponentially grown through the years. Due to this demand, new solutions for storing data are needed, and one promising media is the DNA. This storage solution provides numerous advantages, which includes the ability to store dense information while ac…
▽ More
Since the birth of computer and networks, fuelled by pervasive computing and ubiquitous connectivity, the amount of data stored and transmitted has exponentially grown through the years. Due to this demand, new solutions for storing data are needed, and one promising media is the DNA. This storage solution provides numerous advantages, which includes the ability to store dense information while achieving long-term stability. However, the question as how the data can be retrieved from a DNA-based archive, still remains. In this paper, we aim to address this question by proposing a new storage solution that relies upon molecular communication, and in particular bacterial nanonetworks. Our solution allows digitally encoded information to be stored into non-motile bacteria, which compose an archival architecture of clusters, and to be later retrieved by engineered motile bacteria, whenever reading operations are needed. We conducted extensive simulations, in order to determine the reliability of data retrieval from non-motile storage clusters, placed at different locations. Aiming to assess the feasibility of our solution, we have also conducted wet lab experiments that show how bacteria nanonetworks can effectively retrieve a simple message, such as "Hello World", by conjugation with non-motile bacteria, and finally mobilize towards a final point.
△ Less
Submitted 18 January, 2018; v1 submitted 15 January, 2018;
originally announced January 2018.
-
Soft x-rays induce femtosecond solid-to-solid phase transition
Authors:
Franz Tavella,
Hauke Höppner,
Victor Tkachenko,
Nikita Medvedev,
Flavio Capotondi,
Torsten Golz,
Yun Kai,
Michele Manfredda,
Emanuele Pedersoli,
Mark Prandolini,
Nikola Stojanovic,
Takanori Tanikawa,
Ulrich Teubner,
Sven Toleikis,
Beata Ziaja
Abstract:
Soft x-rays were applied to induce graphitization of diamond through a non-thermal solid-to-solid phase transition. This process was observed within poly-crystalline diamond with a time-resolved experiment using ultrashort soft x-ray pulses of duration 52.5 fs and cross correlated by an optical pulse of duration 32.8 fs. This scheme enabled for the first time the measurement of a phase transition…
▽ More
Soft x-rays were applied to induce graphitization of diamond through a non-thermal solid-to-solid phase transition. This process was observed within poly-crystalline diamond with a time-resolved experiment using ultrashort soft x-ray pulses of duration 52.5 fs and cross correlated by an optical pulse of duration 32.8 fs. This scheme enabled for the first time the measurement of a phase transition on a timescale of ~150 fs. Excellent agreement between experiment and theoretical predictions was found, using a dedicated code that followed the non-equilibrium evolution of the irradiated diamond including all transient electronic and structural changes. These observations confirm that soft x-rays can induce a non-thermal ultrafast solid-to-solid phase transition on a hundred femtosecond timescale.
△ Less
Submitted 20 December, 2016;
originally announced December 2016.
-
The Phase-Contrast Imaging Instrument at the Matter in Extreme Conditions Endstation at LCLS
Authors:
Bob Nagler,
Andreas Schropp,
Eric C. Galtier,
Brice Arnold,
Shaughnessy B. Brown,
Alan Fry,
Arianna Gleason,
Eduardo Granados,
Akel Hashim,
Jerome B. Hastings,
Dirk Samberg,
Frank Seiboth,
Franz Tavella,
Zhou Xing,
Hae Ja Lee,
Christian G. Schroer
Abstract:
We describe the Phase-Contrast Imaging instrument at the Matter in Extreme Conditions (MEC) endstation of the Linac Coherent Light Source. The instrument can image phenomena with a spatial resolution of a few hundreds of nanometers and at the same time reveal the atomic structure through X-ray diffraction, with a temporal resolution better than 100 femtosecond. It was specifically designed for stu…
▽ More
We describe the Phase-Contrast Imaging instrument at the Matter in Extreme Conditions (MEC) endstation of the Linac Coherent Light Source. The instrument can image phenomena with a spatial resolution of a few hundreds of nanometers and at the same time reveal the atomic structure through X-ray diffraction, with a temporal resolution better than 100 femtosecond. It was specifically designed for studies relevant to High-Energy-Density Science and can monitor, e.g., shock fronts, phase transitions, or void collapses. This versatile instrument was commissioned last year and is now available to the MEC user community.
△ Less
Submitted 26 October, 2016;
originally announced October 2016.
-
Field-free molecular alignment probed by the free electron laser in Hamburg (FLASH)
Authors:
P Johnsson,
A Rouzee,
W Siu,
Y Huismans,
F Lepine,
T Marchenko,
S Duesterer,
F Tavella,
N Stojanovic,
A Azima,
R Treusch,
M F Kling,
M J J Vrakking
Abstract:
We report experiments on field-free molecular alignment performed at FLASH, the free electron laser (FEL) in Hamburg. The impulsive alignment induced by a 100 fs near-infrared laser pulse in a rotationally cold CO_2 sample is characterized by ionizing and dissociating the molecules with a time delayed extreme ultra-violet (XUV) FEL pulse. The time-dependent angular distributions of ionic fragmen…
▽ More
We report experiments on field-free molecular alignment performed at FLASH, the free electron laser (FEL) in Hamburg. The impulsive alignment induced by a 100 fs near-infrared laser pulse in a rotationally cold CO_2 sample is characterized by ionizing and dissociating the molecules with a time delayed extreme ultra-violet (XUV) FEL pulse. The time-dependent angular distributions of ionic fragments measured by a velocity map imaging spectrometer shows rapid changes associated with the induced rotational dynamics. The experimental results also show hints of a dissociation process that depends non-linearly on the XUV intensity. With samples of aligned molecules at FLASH, experiments using ultrashort XUV pulses become possible in the molecular frame, which will enable new insights into the understanding of molecules and their interactions.
△ Less
Submitted 1 February, 2009;
originally announced February 2009.
