Showing 1–50 of 69 results for author: Hernandez, J

A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms

Authors: Gavin Abercrombie, Djalel Benbouzid, Paolo Giudici, Delaram Golpayegani, Julio Hernandez, Pierre Noro, Harshvardhan Pandit, Eva Paraschou, Charlie Pownall, Jyoti Prajapati, Mark A. Sayre, Ushnish Sengupta, Arthit Suriyawongful, Ruby Thelot, Sofia Vei, Laura Waltersdorfer

Abstract: This paper introduces a collaborative, human-centered taxonomy of AI, algorithmic and automation harms. We argue that existing taxonomies, while valuable, can be narrow, unclear, typically cater to practitioners and government, and often overlook the needs of the wider public. Drawing on existing taxonomies and a large repository of documented incidents, we propose a taxonomy that is clear and und… ▽ More This paper introduces a collaborative, human-centered taxonomy of AI, algorithmic and automation harms. We argue that existing taxonomies, while valuable, can be narrow, unclear, typically cater to practitioners and government, and often overlook the needs of the wider public. Drawing on existing taxonomies and a large repository of documented incidents, we propose a taxonomy that is clear and understandable to a broad set of audiences, as well as being flexible, extensible, and interoperable. Through iterative refinement with topic experts and crowdsourced annotation testing, we propose a taxonomy that can serve as a powerful tool for civil society organisations, educators, policymakers, product teams and the general public. By fostering a greater understanding of the real-world harms of AI and related technologies, we aim to increase understanding, empower NGOs and individuals to identify and report violations, inform policy discussions, and encourage responsible technology development and deployment. △ Less

Submitted 1 July, 2024; originally announced July 2024.

A Queuing Envelope Model for Estimating Latency Guarantees in Deterministic Networking Scenarios

Authors: Nataliia Koneva, Alfonso Sánchez-Macián, José Alberto Hernández, Farhad Arpanaei, Óscar González de Dios

Abstract: Accurate estimation of queuing delays is crucial for designing and optimizing communication networks, particularly in the context of Deterministic Networking (DetNet) scenarios. This study investigates the approximation of Internet queuing delays using an M/M/1 envelope model, which provides a simple methodology to find tight upper bounds of real delay percentiles. Real traffic statistics collecte… ▽ More Accurate estimation of queuing delays is crucial for designing and optimizing communication networks, particularly in the context of Deterministic Networking (DetNet) scenarios. This study investigates the approximation of Internet queuing delays using an M/M/1 envelope model, which provides a simple methodology to find tight upper bounds of real delay percentiles. Real traffic statistics collected at large Internet Exchange Points (like Amsterdam and San Francisco) have been used to fit polynomial regression models for transforming packet queuing delays into the M/M/1 envelope models. We finally propose a methodology for providing delay percentiles in DetNet scenarios where tight latency guarantees need to be assured. △ Less

Submitted 24 June, 2024; originally announced June 2024.

What Are the Odds? Language Models Are Capable of Probabilistic Reasoning

Authors: Akshay Paruchuri, Jake Garrison, Shun Liao, John Hernandez, Jacob Sunshine, Tim Althoff, Xin Liu, Daniel McDuff

Abstract: Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a… ▽ More Language models (LM) are capable of remarkably complex linguistic tasks; however, numerical reasoning is an area in which they frequently struggle. An important but rarely evaluated form of reasoning is understanding probability distributions. In this paper, we focus on evaluating the probabilistic reasoning capabilities of LMs using idealized and real-world statistical distributions. We perform a systematic evaluation of state-of-the-art LMs on three tasks: estimating percentiles, drawing samples, and calculating probabilities. We evaluate three ways to provide context to LMs 1) anchoring examples from within a distribution or family of distributions, 2) real-world context, 3) summary statistics on which to base a Normal approximation. Models can make inferences about distributions, and can be further aided by the incorporation of real-world context, example shots and simplified assumptions, even if these assumptions are incorrect or misspecified. To conduct this work, we developed a comprehensive benchmark distribution dataset with associated question-answer pairs that we will release publicly. △ Less

Submitted 18 June, 2024; originally announced June 2024.

Comments: 21 pages, 9 figures, 2 tables

Reinforcement-Learning based routing for packet-optical networks with hybrid telemetry

Authors: A. L. García Navarro, Nataliia Koneva, Alfonso Sánchez-Macián, José Alberto Hernández, Óscar González de Dios, J. M. Rivas-Moscoso

Abstract: This article provides a methodology and open-source implementation of Reinforcement Learning algorithms for finding optimal routes in a packet-optical network scenario. The algorithm uses measurements provided by the physical layer (pre-FEC bit error rate and propagation delay) and the link layer (link load) to configure a set of latency-based rewards and penalties based on such measurements. Then… ▽ More This article provides a methodology and open-source implementation of Reinforcement Learning algorithms for finding optimal routes in a packet-optical network scenario. The algorithm uses measurements provided by the physical layer (pre-FEC bit error rate and propagation delay) and the link layer (link load) to configure a set of latency-based rewards and penalties based on such measurements. Then, the algorithm executes Q-learning based on this set of rewards for finding the optimal routing strategies. It is further shown that the algorithm dynamically adapts to changing network conditions by re-calculating optimal policies upon either link load changes or link degradation as measured by pre-FEC BER. △ Less

Submitted 21 June, 2024; v1 submitted 18 June, 2024; originally announced June 2024.

Journal ref: The 28th International Conference on Optical Network Design and Modelling (ONDM 2024)

On optimizing Inband Telemetry systems for accurate latency-based service deployments

Authors: Nataliia Koneva, Alfonso Sánchez-Macián, José Alberto Hernández, Óscar González de Dios

Abstract: The power of Machine Learning and Artificial Intelligence algorithms based on collected datasets, along with the programmability and flexibility provided by Software Defined Networking can provide the building blocks for constructing the so-called Zero-Touch Network and Service Management systems. However, the fuel towards this goal relies on the availability of sufficient and good-quality data co… ▽ More The power of Machine Learning and Artificial Intelligence algorithms based on collected datasets, along with the programmability and flexibility provided by Software Defined Networking can provide the building blocks for constructing the so-called Zero-Touch Network and Service Management systems. However, the fuel towards this goal relies on the availability of sufficient and good-quality data collected from measurements and telemetry. This article provides a telemetry methodology to collect accurate latency measurements, as a first step toward building intelligent control planes that make correct decisions based on precise information. △ Less

Submitted 21 June, 2024; v1 submitted 18 June, 2024; originally announced June 2024.

Journal ref: The 28th International Conference on Optical Network Design and Modelling (ONDM 2024)

Count-Min sketches for Telemetry: analysis of performance in P4 implementations

Authors: José A. Hernández, Davide Scano, Filippo Cugini, Gonzalo Martínez, Natalia Koneva, Alvaro Sánchez-Macián, Óscar González de Dios

Abstract: Monitoring streams of packets at 100~Gb/s and beyond requires using compact and efficient hashing-techniques like HyperLogLog (HLL) or Count-Min Sketch (CMS). In this work, we evaluate the uses and applications of Count-Min Sketch for Metro Networks employing P4-based packet-optical nodes. We provide dimensioning rules for CMS at 100~Gb/s and 400~Gb/s and evaluate its performance in a real impleme… ▽ More Monitoring streams of packets at 100~Gb/s and beyond requires using compact and efficient hashing-techniques like HyperLogLog (HLL) or Count-Min Sketch (CMS). In this work, we evaluate the uses and applications of Count-Min Sketch for Metro Networks employing P4-based packet-optical nodes. We provide dimensioning rules for CMS at 100~Gb/s and 400~Gb/s and evaluate its performance in a real implementation testbed. △ Less

Submitted 21 June, 2024; v1 submitted 18 June, 2024; originally announced June 2024.

Journal ref: The 28th International Conference on Optical Network Design and Modelling (ONDM 2024)

Lexidate: Model Evaluation and Selection with Lexicase

Authors: Jose Guadalupe Hernandez, Anil Kumar Saini, Jason H. Moore

Abstract: Automated machine learning streamlines the task of finding effective machine learning pipelines by automating model training, evaluation, and selection. Traditional evaluation strategies, like cross-validation (CV), generate one value that averages the accuracy of a pipeline's predictions. This single value, however, may not fully describe the generalizability of the pipeline. Here, we present Lex… ▽ More Automated machine learning streamlines the task of finding effective machine learning pipelines by automating model training, evaluation, and selection. Traditional evaluation strategies, like cross-validation (CV), generate one value that averages the accuracy of a pipeline's predictions. This single value, however, may not fully describe the generalizability of the pipeline. Here, we present Lexicase-based Validation (lexidate), a method that uses multiple, independent prediction values for selection. Lexidate splits training data into a learning set and a selection set. Pipelines are trained on the learning set and make predictions on the selection set. The predictions are graded for correctness and used by lexicase selection to identify parent pipelines. Compared to 10-fold CV, lexicase reduces the training time. We test the effectiveness of three lexidate configurations within the Tree-based Pipeline Optimization Tool 2 (TPOT2) package on six OpenML classification tasks. In one configuration, we detected no difference in the accuracy of the final model returned from TPOT2 on most tasks compared to 10-fold CV. All configurations studied here returned similar or less complex final pipelines compared to 10-fold CV. △ Less

Submitted 17 June, 2024; originally announced June 2024.

Generative Visual Instruction Tuning

Authors: Jefferson Hernandez, Ruben Villegas, Vicente Ordonez

Abstract: We propose to use machine-generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruct… ▽ More We propose to use machine-generated instruction-following data to improve the zero-shot capabilities of a large multimodal model with additional support for generative and image editing tasks. We achieve this by curating a new multimodal instruction-following set using GPT-4V and existing datasets for image generation and editing. Using this instruction set and the existing LLaVA-Finetune instruction set for visual understanding tasks, we produce GenLLaVA, a Generative Large Language, and Visual Assistant. GenLLaVA is built through a strategy that combines three types of large pre-trained models through instruction finetuning: LLaMA for language modeling, SigLIP for image-text matching, and StableDiffusion for text-to-image generation. Our model demonstrates visual understanding capabilities on par with LLaVA and additionally demonstrates competitive results with native multimodal models such as Unified-IO 2, paving the way for building advanced general-purpose visual assistants by effectively re-using existing multimodal models. We open-source our dataset, codebase, and model checkpoints to foster further research and application in this domain. △ Less

Submitted 17 June, 2024; originally announced June 2024.

Towards a Personal Health Large Language Model

Authors: Justin Cosentino, Anastasiya Belyaeva, Xin Liu, Nicholas A. Furlotte, Zhun Yang, Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, Robby Bryant, Ryan G. Gomes, Allen Jiang, Roy Lee, Yun Liu, Javier Perez, Jameson K. Rogers, Cathy Speed, Shyam Tailor, Megan Walker, Jeffrey Yu, Tim Althoff, Conor Heneghan, John Hernandez, Mark Malhotra , et al. (9 additional authors not shown)

Abstract: In health, most large language model (LLM) research has focused on clinical tasks. However, mobile and wearable devices, which are rarely integrated into such tasks, provide rich, longitudinal data for personal health monitoring. Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data. We… ▽ More In health, most large language model (LLM) research has focused on clinical tasks. However, mobile and wearable devices, which are rarely integrated into such tasks, provide rich, longitudinal data for personal health monitoring. Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data. We created and curated three datasets that test 1) production of personalized insights and recommendations from sleep patterns, physical activity, and physiological responses, 2) expert domain knowledge, and 3) prediction of self-reported sleep outcomes. For the first task we designed 857 case studies in collaboration with domain experts to assess real-world scenarios in sleep and fitness. Through comprehensive evaluation of domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not statistically different from expert performance in fitness and, while experts remain superior for sleep, fine-tuning PH-LLM provided significant improvements in using relevant domain knowledge and personalizing information for sleep insights. We evaluated PH-LLM domain knowledge using multiple choice sleep medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on fitness, exceeding average scores from a sample of human experts. Finally, we trained PH-LLM to predict self-reported sleep quality outcomes from textual and multimodal encoding representations of wearable data, and demonstrate that multimodal encoding is required to match performance of specialized discriminative models. Although further development and evaluation are necessary in the safety-critical personal health domain, these results demonstrate both the broad knowledge and capabilities of Gemini models and the benefit of contextualizing physiological data for personal health applications as done with PH-LLM. △ Less

Submitted 10 June, 2024; originally announced June 2024.

Comments: 72 pages

An economically-consistent discrete choice model with flexible utility specification based on artificial neural networks

Authors: Jose Ignacio Hernandez, Niek Mouter, Sander van Cranenburgh

Abstract: Random utility maximisation (RUM) models are one of the cornerstones of discrete choice modelling. However, specifying the utility function of RUM models is not straightforward and has a considerable impact on the resulting interpretable outcomes and welfare measures. In this paper, we propose a new discrete choice model based on artificial neural networks (ANNs) named "Alternative-Specific and Sh… ▽ More Random utility maximisation (RUM) models are one of the cornerstones of discrete choice modelling. However, specifying the utility function of RUM models is not straightforward and has a considerable impact on the resulting interpretable outcomes and welfare measures. In this paper, we propose a new discrete choice model based on artificial neural networks (ANNs) named "Alternative-Specific and Shared weights Neural Network (ASS-NN)", which provides a further balance between flexible utility approximation from the data and consistency with two assumptions: RUM theory and fungibility of money (i.e., "one euro is one euro"). Therefore, the ASS-NN can derive economically-consistent outcomes, such as marginal utilities or willingness to pay, without explicitly specifying the utility functional form. Using a Monte Carlo experiment and empirical data from the Swissmetro dataset, we show that ASS-NN outperforms (in terms of goodness of fit) conventional multinomial logit (MNL) models under different utility specifications. Furthermore, we show how the ASS-NN is used to derive marginal utilities and willingness to pay measures. △ Less

Submitted 19 April, 2024; originally announced April 2024.

Memristor-Based Lightweight Encryption

Authors: Muhammad Ali Siddiqi, Jan Andrés Galvan Hernández, Anteneh Gebregiorgis, Rajendra Bishnoi, Christos Strydis, Said Hamdioui, Mottaqiallah Taouil

Abstract: Next-generation personalized healthcare devices are undergoing extreme miniaturization in order to improve user acceptability. However, such developments make it difficult to incorporate cryptographic primitives using available target technologies since these algorithms are notorious for their energy consumption. Besides, strengthening these schemes against side-channel attacks further adds to the… ▽ More Next-generation personalized healthcare devices are undergoing extreme miniaturization in order to improve user acceptability. However, such developments make it difficult to incorporate cryptographic primitives using available target technologies since these algorithms are notorious for their energy consumption. Besides, strengthening these schemes against side-channel attacks further adds to the device overheads. Therefore, viable alternatives among emerging technologies are being sought. In this work, we investigate the possibility of using memristors for implementing lightweight encryption. We propose a 40-nm RRAM-based GIFT-cipher implementation using a 1T1R configuration with promising results; it exhibits roughly half the energy consumption of a CMOS-only implementation. More importantly, its non-volatile and reconfigurable substitution boxes offer an energy-efficient protection mechanism against side-channel attacks. The complete cipher takes 0.0034 mm$^2$ of area, and encrypting a 128-bit block consumes a mere 242 pJ. △ Less

Submitted 29 March, 2024; originally announced April 2024.

Journal ref: Proceedings of the 2023 26th Euromicro Conference on Digital System Design (DSD)

Open Source Conversational LLMs do not know most Spanish words

Authors: Javier Conde, Miguel González, Nina Melero, Raquel Ferrando, Gonzalo Martínez, Elena Merino-Gómez, José Alberto Hernández, Pedro Reviriego

Abstract: The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-source chat LLMs. These models are evaluated on a wide range of benchmarks to assess their capabilities in answering questions or solving problems on almost any possible topic or to test their ability to reason or interpr… ▽ More The growing interest in Large Language Models (LLMs) and in particular in conversational models with which users can interact has led to the development of a large number of open-source chat LLMs. These models are evaluated on a wide range of benchmarks to assess their capabilities in answering questions or solving problems on almost any possible topic or to test their ability to reason or interpret texts. Instead, the evaluation of the knowledge that these models have of the languages has received much less attention. For example, the words that they can recognize and use in different languages. In this paper, we evaluate the knowledge that open-source chat LLMs have of Spanish words by testing a sample of words in a reference dictionary. The results show that open-source chat LLMs produce incorrect meanings for an important fraction of the words and are not able to use most of the words correctly to write sentences with context. These results show how Spanish is left behind in the open-source LLM race and highlight the need to push for linguistic fairness in conversational LLMs ensuring that they provide similar performance across languages. △ Less

Submitted 21 March, 2024; originally announced March 2024.

Comments: Under Review at SEPLN-2024

Journal ref: SEPLN Journal 2024

Prototipo de video juego activo basado en una cámara 3D para motivar la actividad física en niños y adultos mayores

Authors: Benjamín Ojeda Magaña, José Guadalupe Robledo Hernández, Leopoldo Gómez Barba, Victor Manuel Rangel Cobián

Abstract: This document describes the development of a video game prototype designed to encourage physical activity among children and older adults. The prototype consists of a laptop, a camera with 3D sensors, and optionally requires an LCD screen or a projector. The programming component of this prototype was developed in Scratch, a programming language geared towards children, which greatly facilitates t… ▽ More This document describes the development of a video game prototype designed to encourage physical activity among children and older adults. The prototype consists of a laptop, a camera with 3D sensors, and optionally requires an LCD screen or a projector. The programming component of this prototype was developed in Scratch, a programming language geared towards children, which greatly facilitates the creation of a game tailored to the users' preferences. The idea to create such a prototype originated from the desire to offer an option that promotes physical activity among children and adults, given that a lack of physical exercise is a predominant factor in the development of chronic degenerative diseases such as diabetes and hypertension, to name the most common. As a result of this initiative, an active video game prototype was successfully developed, based on a **-pong game, which allows both children and adults to interact in a fun way while encouraging the performance of physical activities that can positively impact the users' health. △ Less

Submitted 19 March, 2024; originally announced March 2024.

Comments: 13 pages, in Spanish language, 11 figures

ACM Class: I.4.9

Beware of Words: Evaluating the Lexical Richness of Conversational Large Language Models

Authors: Gonzalo Martínez, José Alberto Hernández, Javier Conde, Pedro Reviriego, Elena Merino

Abstract: The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasoning or maths to answering questions on a myriad of topics. Instead, much less attention is being devoted to the study of the linguistic features of the texts generated by these LLMs. This is surprising since LLMs are model… ▽ More The performance of conversational Large Language Models (LLMs) in general, and of ChatGPT in particular, is currently being evaluated on many different tasks, from logical reasoning or maths to answering questions on a myriad of topics. Instead, much less attention is being devoted to the study of the linguistic features of the texts generated by these LLMs. This is surprising since LLMs are models for language, and understanding how they use the language is important. Indeed, conversational LLMs are poised to have a significant impact on the evolution of languages as they may eventually dominate the creation of new text. This means that for example, if conversational LLMs do not use a word it may become less and less frequent and eventually stop being used altogether. Therefore, evaluating the linguistic features of the text they produce and how those depend on the model parameters is the first step toward understanding the potential impact of conversational LLMs on the evolution of languages. In this paper, we consider the evaluation of the lexical richness of the text generated by LLMs and how it depends on the model parameters. A methodology is presented and used to conduct a comprehensive evaluation of lexical richness using ChatGPT as a case study. The results show how lexical richness depends on the version of ChatGPT and some of its parameters, such as the presence penalty, or on the role assigned to the model. The dataset and tools used in our analysis are released under open licenses with the goal of drawing the much-needed attention to the evaluation of the linguistic features of LLM-generated text. △ Less

Submitted 11 February, 2024; originally announced February 2024.

Synthesis of 3D on-air signatures with the Sigma-Lognormal model

Authors: Miguel A. Ferrer, Moises Diaz, Cristina Carmona-Duarte, Jose J. Quintana Hernandez, Rejean Plamondon

Abstract: Signature synthesis is a computation technique that generates artificial specimens which can support decision making in automatic signature verification. A lot of work has been dedicated to this subject, which centres on synthesizing dynamic and static two-dimensional handwriting on canvas. This paper proposes a framework to generate synthetic 3D on-air signatures exploiting the lognormality princ… ▽ More Signature synthesis is a computation technique that generates artificial specimens which can support decision making in automatic signature verification. A lot of work has been dedicated to this subject, which centres on synthesizing dynamic and static two-dimensional handwriting on canvas. This paper proposes a framework to generate synthetic 3D on-air signatures exploiting the lognormality principle, which mimics the complex neuromotor control processes at play as the fingertip moves. Addressing the usual cases involving the development of artificial individuals and duplicated samples, this paper contributes to the synthesis of: (1) the trajectory and velocity of entirely 3D new signatures; (2) kinematic information when only the 3D trajectory of the signature is known, and (3) duplicate samples of 3D real signatures. Validation was conducted by generating synthetic 3D signature databases mimicking real ones and showing that automatic signature verifications of genuine and skilled forgeries report performances similar to those of real and synthetic databases. We also observed that training 3D automatic signature verifiers with duplicates can reduce errors. We further demonstrated that our proposal is also valid for synthesizing 3D air writing and gestures. Finally, a perception test confirmed the human likeness of the generated specimens. The databases generated are publicly available, only for research purposes, at . △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: Accepted Version. Published on Knowledge-Based Systems

Journal ref: Knowledge-Based Systems, Vol. 265,2023

From User Surveys to Telemetry-Driven Agents: Exploring the Potential of Personalized Productivity Solutions

Authors: Subigya Nepal, Javier Hernandez, Talie Massachi, Kael Rowan, Judith Amores, **a Suh, Gonzalo Ramos, Brian Houck, Shamsi T. Iqbal, Mary Czerwinski

Abstract: We present a comprehensive, user-centric approach to understand preferences in AI-based productivity agents and develop personalized solutions tailored to users' needs. Utilizing a two-phase method, we first conducted a survey with 363 participants, exploring various aspects of productivity, communication style, agent approach, personality traits, personalization, and privacy. Drawing on the surve… ▽ More We present a comprehensive, user-centric approach to understand preferences in AI-based productivity agents and develop personalized solutions tailored to users' needs. Utilizing a two-phase method, we first conducted a survey with 363 participants, exploring various aspects of productivity, communication style, agent approach, personality traits, personalization, and privacy. Drawing on the survey insights, we developed a GPT-4 powered personalized productivity agent that utilizes telemetry data gathered via Viva Insights from information workers to provide tailored assistance. We compared its performance with alternative productivity-assistive tools, such as dashboard and narrative, in a study involving 40 participants. Our findings highlight the importance of user-centric design, adaptability, and the balance between personalization and privacy in AI-assisted productivity tools. By building on the insights distilled from our study, we believe that our work can enable and guide future research to further enhance productivity solutions, ultimately leading to optimized efficiency and user experiences for information workers. △ Less

Submitted 16 January, 2024; originally announced January 2024.

ACM Class: H.5.0; H.5.3; H.5.m; J.0

Repeatability, Reproducibility, Replicability, Reusability (4R) in Journals' Policies and Software/Data Management in Scientific Publications: A Survey, Discussion, and Perspectives

Authors: José Armando Hernández, Miguel Colom

Abstract: With the recognized crisis of credibility in scientific research, there is a growth of reproducibility studies in computer science, and although existing surveys have reviewed reproducibility from various perspectives, especially very specific technological issues, they do not address the author-publisher relationship in the publication of reproducible computational scientific articles. This aspec… ▽ More With the recognized crisis of credibility in scientific research, there is a growth of reproducibility studies in computer science, and although existing surveys have reviewed reproducibility from various perspectives, especially very specific technological issues, they do not address the author-publisher relationship in the publication of reproducible computational scientific articles. This aspect requires significant attention because it is the basis for reliable research. We have found a large gap between the reproducibility-oriented practices, journal policies, recommendations, publisher artifact Description/Evaluation guidelines, submission guides, technological reproducibility evolution, and its effective adoption to contribute to tackling the crisis. We conducted a narrative survey, a comprehensive overview and discussion identifying the mutual efforts required from Authors, Journals, and Technological actors to achieve reproducibility research. The relationship between authors and scientific journals in their mutual efforts to jointly improve the reproducibility of scientific results is analyzed. Eventually, we propose recommendations for the journal policies, as well as a unified and standardized Reproducibility Guide for the submission of scientific articles for authors. The main objective of this work is to analyze the implementation and experiences of reproducibility policies, techniques and technologies, standards, methodologies, software, and data management tools required for scientific reproducible publications. Also, the benefits and drawbacks of such an adoption, as well as open challenges and promising trends, to propose possible strategies and efforts to mitigate the identified gaps. To this purpose, we analyzed 200 scientific articles, surveyed 16 Computer Science journals, and systematically classified them according to reproducibility strategies, technologies, policies, code citation, and editorial business. We conclude there is still a reproducibility gap in scientific publications, although at the same time also the opportunity to reduce this gap with the joint effort of authors, publishers, and technological providers. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models

Authors: Gonzalo Martínez, Javier Conde, Elena Merino-Gómez, Beatriz Bermúdez-Margaretto, José Alberto Hernández, Pedro Reviriego, Marc Brysbaert

Abstract: Vocabulary tests, once a cornerstone of language modeling evaluation, have been largely overlooked in the current landscape of Large Language Models (LLMs) like Llama, Mistral, and GPT. While most LLM evaluation benchmarks focus on specific tasks or domain-specific knowledge, they often neglect the fundamental linguistic aspects of language understanding and production. In this paper, we advocate… ▽ More Vocabulary tests, once a cornerstone of language modeling evaluation, have been largely overlooked in the current landscape of Large Language Models (LLMs) like Llama, Mistral, and GPT. While most LLM evaluation benchmarks focus on specific tasks or domain-specific knowledge, they often neglect the fundamental linguistic aspects of language understanding and production. In this paper, we advocate for the revival of vocabulary tests as a valuable tool for assessing LLM performance. We evaluate seven LLMs using two vocabulary test formats across two languages and uncover surprising gaps in their lexical knowledge. These findings shed light on the intricacies of LLM word representations, their learning mechanisms, and performance variations across models and languages. Moreover, the ability to automatically generate and perform vocabulary tests offers new opportunities to expand the approach and provide a more complete picture of LLMs' language skills. △ Less

Submitted 29 January, 2024; v1 submitted 23 October, 2023; originally announced October 2023.

Learning to bag with a simulation-free reinforcement learning framework for robots

Authors: Francisco Munguia-Galeano, Jihong Zhu, Juan David Hernández, Ze Ji

Abstract: Bagging is an essential skill that humans perform in their daily activities. However, deformable objects, such as bags, are complex for robots to manipulate. This paper presents an efficient learning-based framework that enables robots to learn bagging. The novelty of this framework is its ability to perform bagging without relying on simulations. The learning process is accomplished through a rei… ▽ More Bagging is an essential skill that humans perform in their daily activities. However, deformable objects, such as bags, are complex for robots to manipulate. This paper presents an efficient learning-based framework that enables robots to learn bagging. The novelty of this framework is its ability to perform bagging without relying on simulations. The learning process is accomplished through a reinforcement learning algorithm introduced in this work, designed to find the best gras** points of the bag based on a set of compact state representations. The framework utilizes a set of primitive actions and represents the task in five states. In our experiments, the framework reaches a 60 % and 80 % of success rate after around three hours of training in the real world when starting the bagging task from folded and unfolded, respectively. Finally, we test the trained model with two more bags of different sizes to evaluate its generalizability. △ Less

Submitted 22 October, 2023; originally announced October 2023.

Comments: IET Cyber-Systems and Robotics

Affective Conversational Agents: Understanding Expectations and Personal Influences

Authors: Javier Hernandez, **a Suh, Judith Amores, Kael Rowan, Gonzalo Ramos, Mary Czerwinski

Abstract: The rise of AI conversational agents has broadened opportunities to enhance human capabilities across various domains. As these agents become more prevalent, it is crucial to investigate the impact of different affective abilities on their performance and user experience. In this study, we surveyed 745 respondents to understand the expectations and preferences regarding affective skills in various… ▽ More The rise of AI conversational agents has broadened opportunities to enhance human capabilities across various domains. As these agents become more prevalent, it is crucial to investigate the impact of different affective abilities on their performance and user experience. In this study, we surveyed 745 respondents to understand the expectations and preferences regarding affective skills in various applications. Specifically, we assessed preferences concerning AI agents that can perceive, respond to, and simulate emotions across 32 distinct scenarios. Our results indicate a preference for scenarios that involve human interaction, emotional support, and creative tasks, with influences from factors such as emotional reappraisal and personality traits. Overall, the desired affective skills in AI agents depend largely on the application's context and nature, emphasizing the need for adaptability and context-awareness in the design of affective AI conversational agents. △ Less

Submitted 19 October, 2023; originally announced October 2023.

How many words does ChatGPT know? The answer is ChatWords

Authors: Gonzalo Martínez, Javier Conde, Pedro Reviriego, Elena Merino-Gómez, José Alberto Hernández, Fabrizio Lombardi

Abstract: The introduction of ChatGPT has put Artificial Intelligence (AI) Natural Language Processing (NLP) in the spotlight. ChatGPT adoption has been exponential with millions of users experimenting with it in a myriad of tasks and application domains with impressive results. However, ChatGPT has limitations and suffers hallucinations, for example producing answers that look plausible but they are comple… ▽ More The introduction of ChatGPT has put Artificial Intelligence (AI) Natural Language Processing (NLP) in the spotlight. ChatGPT adoption has been exponential with millions of users experimenting with it in a myriad of tasks and application domains with impressive results. However, ChatGPT has limitations and suffers hallucinations, for example producing answers that look plausible but they are completely wrong. Evaluating the performance of ChatGPT and similar AI tools is a complex issue that is being explored from different perspectives. In this work, we contribute to those efforts with ChatWords, an automated test system, to evaluate ChatGPT knowledge of an arbitrary set of words. ChatWords is designed to be extensible, easy to use, and adaptable to evaluate also other NLP AI tools. ChatWords is publicly available and its main goal is to facilitate research on the lexical knowledge of AI tools. The benefits of ChatWords are illustrated with two case studies: evaluating the knowledge that ChatGPT has of the Spanish lexicon (taken from the official dictionary of the "Real Academia Española") and of the words that appear in the Quixote, the well-known novel written by Miguel de Cervantes. The results show that ChatGPT is only able to recognize approximately 80% of the words in the dictionary and 90% of the words in the Quixote, in some cases with an incorrect meaning. The implications of the lexical knowledge of NLP AI tools and potential applications of ChatWords are also discussed providing directions for further work on the study of the lexical knowledge of AI tools. △ Less

Submitted 28 September, 2023; originally announced September 2023.

Hyper-reduction for Petrov-Galerkin reduced order models

Authors: S. Ares de Parga, J. R. Bravo, J. A. Hernandez, R. Zorrilla, R. Rossi

Abstract: Projection-based Reduced Order Models minimize the discrete residual of a "full order model" (FOM) while constraining the unknowns to a reduced dimension space. For problems with symmetric positive definite (SPD) Jacobians, this is optimally achieved by projecting the full order residual onto the approximation basis (Galerkin Projection). This is sub-optimal for non-SPD Jacobians as it only minimi… ▽ More Projection-based Reduced Order Models minimize the discrete residual of a "full order model" (FOM) while constraining the unknowns to a reduced dimension space. For problems with symmetric positive definite (SPD) Jacobians, this is optimally achieved by projecting the full order residual onto the approximation basis (Galerkin Projection). This is sub-optimal for non-SPD Jacobians as it only minimizes the projection of the residual, not the residual itself. An alternative is to directly minimize the 2-norm of the residual, achievable using QR factorization or the method of the normal equations (LSPG). The first approach involves constructing and factorizing a large matrix, while LSPG avoids this but requires constructing a product element by element, necessitating a complementary mesh and adding complexity to the hyper-reduction process. This work proposes an alternative based on Petrov-Galerkin minimization. We choose a left basis for a least-squares minimization on a reduced problem, ensuring the discrete full order residual is minimized. This is applicable to both SPD and non-SPD Jacobians, allowing element-by-element assembly, avoiding the use of a complementary mesh, and simplifying finite element implementation. The technique is suitable for hyper-reduction using the Empirical Cubature Method and is applicable in nonlinear reduction procedures. △ Less

Submitted 28 September, 2023; originally announced September 2023.

Journal ref: Computer Methods in Applied Mechanics and Engineering, vol. 416, pp. 116298, 2023

Open Source Robot Localization for Non-Planar Environments

Authors: Francisco Martín Rico, José Miguel Guerrero Hernández, Rodrigo Pérez Rodríguez, Juan Diego Peña Narváez, Alberto García Gómez-Jacinto

Abstract: The operational environments in which a mobile robot executes its missions often exhibit non-flat terrain characteristics, encompassing outdoor and indoor settings featuring ramps and slopes. In such scenarios, the conventional methodologies employed for localization encounter novel challenges and limitations. This study delineates a localization framework incorporating ground elevation and inclin… ▽ More The operational environments in which a mobile robot executes its missions often exhibit non-flat terrain characteristics, encompassing outdoor and indoor settings featuring ramps and slopes. In such scenarios, the conventional methodologies employed for localization encounter novel challenges and limitations. This study delineates a localization framework incorporating ground elevation and incline considerations, deviating from traditional 2D localization paradigms that may falter in such contexts. In our proposed approach, the map encompasses elevation and spatial occupancy information, employing Gridmaps and Octomaps. At the same time, the perception model is designed to accommodate the robot's inclined orientation and the potential presence of ground as an obstacle, besides usual structural and dynamic obstacles. We provide an implementation of our approach fully working with Nav2, ready to replace the baseline AMCL approach when the robot is in non-planar environments. Our methodology was rigorously tested in both simulated environments and through practical application on actual robots, including the Tiago and Summit XL models, across various settings ranging from indoor and outdoor to flat and uneven terrains. Demonstrating exceptional precision, our approach yielded error margins below 10 centimeters and 0.05 radians in indoor settings and less than 1.0 meters in extensive outdoor routes. While our results exhibit a slight improvement over AMCL in indoor environments, the enhancement in performance is significantly more pronounced when compared to 3D SLAM algorithms. This underscores the considerable robustness and efficiency of our approach, positioning it as an effective strategy for mobile robots tasked with navigating expansive and intricate indoor/outdoor environments. △ Less

Submitted 30 March, 2024; v1 submitted 22 September, 2023; originally announced September 2023.

A Comparative Study on Routing Selection Algorithms for Dynamic Planning of EONs over C+L Bands

Authors: Farhad Arpanaei, José Manuel Rivas-Moscoso, Mahdi Ranjbar Zefreh, José Alberto Hernández, Juan Pedro Fernández-Palacios, David Larrabeiti

Abstract: The performance of three routing selection algorithms is compared in terms of bandwidth blocking probability, quality of transmission, and run time in EONs over the C+L band. The min-max frequency algorithm shows the best performance on all metrics. The performance of three routing selection algorithms is compared in terms of bandwidth blocking probability, quality of transmission, and run time in EONs over the C+L band. The min-max frequency algorithm shows the best performance on all metrics. △ Less

Submitted 25 August, 2023; originally announced August 2023.

Playing with Words: Comparing the Vocabulary and Lexical Richness of ChatGPT and Humans

Authors: Pedro Reviriego, Javier Conde, Elena Merino-Gómez, Gonzalo Martínez, José Alberto Hernández

Abstract: The introduction of Artificial Intelligence (AI) generative language models such as GPT (Generative Pre-trained Transformer) and tools such as ChatGPT has triggered a revolution that can transform how text is generated. This has many implications, for example, as AI-generated text becomes a significant fraction of the text, would this have an effect on the language capabilities of readers and also… ▽ More The introduction of Artificial Intelligence (AI) generative language models such as GPT (Generative Pre-trained Transformer) and tools such as ChatGPT has triggered a revolution that can transform how text is generated. This has many implications, for example, as AI-generated text becomes a significant fraction of the text, would this have an effect on the language capabilities of readers and also on the training of newer AI tools? Would it affect the evolution of languages? Focusing on one specific aspect of the language: words; will the use of tools such as ChatGPT increase or reduce the vocabulary used or the lexical richness? This has implications for words, as those not included in AI-generated content will tend to be less and less popular and may eventually be lost. In this work, we perform an initial comparison of the vocabulary and lexical richness of ChatGPT and humans when performing the same tasks. In more detail, two datasets containing the answers to different types of questions answered by ChatGPT and humans, and a third dataset in which ChatGPT paraphrases sentences and questions are used. The analysis shows that ChatGPT tends to use fewer distinct words and lower lexical richness than humans. These results are very preliminary and additional datasets and ChatGPT configurations have to be evaluated to extract more general conclusions. Therefore, further research is needed to understand how the use of ChatGPT and more broadly generative AI tools will affect the vocabulary and lexical richness in different types of text and languages. △ Less

Submitted 31 August, 2023; v1 submitted 14 August, 2023; originally announced August 2023.

Research Protocol for the Google Health Digital Well-being Study

Authors: Daniel McDuff, Andrew Barakat, Ari Winbush, Allen Jiang, Felicia Cordeiro, Ryann Crowley, Lauren E. Kahn, John Hernandez, Nicholas B. Allen

Abstract: The impact of digital device use on health and well-being is a pressing question to which individuals, families, schools, policy makers, legislators, and digital designers are all demanding answers. However, the scientific literature on this topic to date is marred by small and/or unrepresentative samples, poor measurement of core constructs (e.g., device use, smartphone addiction), and a limited… ▽ More The impact of digital device use on health and well-being is a pressing question to which individuals, families, schools, policy makers, legislators, and digital designers are all demanding answers. However, the scientific literature on this topic to date is marred by small and/or unrepresentative samples, poor measurement of core constructs (e.g., device use, smartphone addiction), and a limited ability to address the psychological and behavioral mechanisms that may underlie the relationships between device use and well-being. A number of recent authoritative reviews have made urgent calls for future research projects to address these limitations. The critical role of research is to identify which patterns of use are associated with benefits versus risks, and who is more vulnerable to harmful versus beneficial outcomes, so that we can pursue evidence-based product design, education, and regulation aimed at maximizing benefits and minimizing risks of smartphones and other digital devices. We describe a protocol for a Digital Well-Being (DWB) study to help answer these questions. △ Less

Submitted 11 July, 2023; originally announced July 2023.

Towards Understanding the Interplay of Generative Artificial Intelligence and the Internet

Authors: Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, Rik Sarkar

Abstract: The rapid adoption of generative Artificial Intelligence (AI) tools that can generate realistic images or text, such as DALL-E, MidJourney, or ChatGPT, have put the societal impacts of these technologies at the center of public debate. These tools are possible due to the massive amount of data (text and images) that is publicly available through the Internet. At the same time, these generative AI… ▽ More The rapid adoption of generative Artificial Intelligence (AI) tools that can generate realistic images or text, such as DALL-E, MidJourney, or ChatGPT, have put the societal impacts of these technologies at the center of public debate. These tools are possible due to the massive amount of data (text and images) that is publicly available through the Internet. At the same time, these generative AI tools become content creators that are already contributing to the data that is available to train future models. Therefore, future versions of generative AI tools will be trained with a mix of human-created and AI-generated content, causing a potential feedback loop between generative AI and public data repositories. This interaction raises many questions: how will future versions of generative AI tools behave when trained on a mixture of real and AI generated data? Will they evolve and improve with the new data sets or on the contrary will they degrade? Will evolution introduce biases or reduce diversity in subsequent generations of generative AI tools? What are the societal implications of the possible degradation of these models? Can we mitigate the effects of this feedback loop? In this document, we explore the effect of this interaction and report some initial results using simple diffusion models trained with various image datasets. Our results show that the quality and diversity of the generated images can degrade over time suggesting that incorporating AI-created data can have undesired effects on future versions of generative models. △ Less

Submitted 8 June, 2023; originally announced June 2023.

Phylogeny-informed fitness estimation

Authors: Alexander Lale**i, Matthew Andres Moreno, Jose Guadalupe Hernandez, Emily Dolson

Abstract: Phylogenies (ancestry trees) depict the evolutionary history of an evolving population. In evolutionary computing, a phylogeny can reveal how an evolutionary algorithm steers a population through a search space, illuminating the step-by-step process by which any solutions evolve. Thus far, phylogenetic analyses have primarily been applied as post-hoc analyses used to deepen our understanding of ex… ▽ More Phylogenies (ancestry trees) depict the evolutionary history of an evolving population. In evolutionary computing, a phylogeny can reveal how an evolutionary algorithm steers a population through a search space, illuminating the step-by-step process by which any solutions evolve. Thus far, phylogenetic analyses have primarily been applied as post-hoc analyses used to deepen our understanding of existing evolutionary algorithms. Here, we investigate whether phylogenetic analyses can be used at runtime to augment parent selection procedures during an evolutionary search. Specifically, we propose phylogeny-informed fitness estimation, which exploits a population's phylogeny to estimate fitness evaluations. We evaluate phylogeny-informed fitness estimation in the context of the down-sampled lexicase and cohort lexicase selection algorithms on two diagnostic analyses and four genetic programming (GP) problems. Our results indicate that phylogeny-informed fitness estimation can mitigate the drawbacks of down-sampled lexicase, improving diversity maintenance and search space exploration. However, the extent to which phylogeny-informed fitness estimation improves problem-solving success for GP varies by problem, subsampling method, and subsampling level. This work serves as an initial step toward improving evolutionary algorithms by exploiting runtime phylogenetic analysis. △ Less

Submitted 6 June, 2023; originally announced June 2023.

Comments: Submitted as contribution to GPTP XX

A brief introduction to satellite communications for Non-Terrestrial Networks (NTN)

Authors: Jose Alberto Hernandez, Pedro Reviriego

Abstract: At present (year 2023), approximately 2,500 satellites are currently orbiting the Earth. This number is expected to reach 50,000 satellites (that is, 20 times growth) for the next 10 years, thanks to the recent advances concerning launching satellites at low cost and with high probability of success. In this sense, it is expected that next years the world will witness a massive increase in mobile… ▽ More At present (year 2023), approximately 2,500 satellites are currently orbiting the Earth. This number is expected to reach 50,000 satellites (that is, 20 times growth) for the next 10 years, thanks to the recent advances concerning launching satellites at low cost and with high probability of success. In this sense, it is expected that next years the world will witness a massive increase in mobile connectivity thanks to the combination of 5G deployments and satellites, building the so-called Space-Terrestrial Integrated Network (STIN), thanks to the emergence of Non-Terrestrial Networks (NTNs). This document overviews the foundations of satellite communications as a short tutorial for those interested in research and development on Space-Terrestrial Integrated Networks (STIN) and Non-Terrestrial Networks (NTN) for supporting 5G in remote areas. △ Less

Submitted 8 May, 2023; originally announced May 2023.

ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders

Authors: Jefferson Hernandez, Ruben Villegas, Vicente Ordonez

Abstract: We propose ViC-MAE, a model that combines both Masked AutoEncoders (MAE) and contrastive learning. ViC-MAE is trained using a global featured obtained by pooling the local representations learned under an MAE reconstruction loss and leveraging this representation under a contrastive objective across images and video frames. We show that visual representations learned under ViC-MAE generalize well… ▽ More We propose ViC-MAE, a model that combines both Masked AutoEncoders (MAE) and contrastive learning. ViC-MAE is trained using a global featured obtained by pooling the local representations learned under an MAE reconstruction loss and leveraging this representation under a contrastive objective across images and video frames. We show that visual representations learned under ViC-MAE generalize well to both video and image classification tasks. Particularly, ViC-MAE obtains state-of-the-art transfer learning performance from video to images on Imagenet-1k compared to the recently proposed OmniMAE by achieving a top-1 accuracy of 86% (+1.3% absolute improvement) when trained on the same data and 87.1% (+2.4% absolute improvement) when training on extra data. At the same time ViC-MAE outperforms most other methods on video benchmarks by obtaining 75.9% top-1 accuracy on the challenging Something something-v2 video benchmark . When training on videos and images from a diverse combination of datasets, our method maintains a balanced transfer-learning performance between video and image classification benchmarks, coming only as a close second to the best supervised method. △ Less

Submitted 30 November, 2023; v1 submitted 21 March, 2023; originally announced March 2023.

Comments: More results on Video an Image datasets, ViC-MAE now supports training on videos and images

Combining Generative Artificial Intelligence (AI) and the Internet: Heading towards Evolution or Degradation?

Authors: Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, Rik Sarkar

Abstract: In the span of a few months, generative Artificial Intelligence (AI) tools that can generate realistic images or text have taken the Internet by storm, making them one of the technologies with fastest adoption ever. Some of these generative AI tools such as DALL-E, MidJourney, or ChatGPT have gained wide public notoriety. Interestingly, these tools are possible because of the massive amount of dat… ▽ More In the span of a few months, generative Artificial Intelligence (AI) tools that can generate realistic images or text have taken the Internet by storm, making them one of the technologies with fastest adoption ever. Some of these generative AI tools such as DALL-E, MidJourney, or ChatGPT have gained wide public notoriety. Interestingly, these tools are possible because of the massive amount of data (text and images) available on the Internet. The tools are trained on massive data sets that are scraped from Internet sites. And now, these generative AI tools are creating massive amounts of new data that are being fed into the Internet. Therefore, future versions of generative AI tools will be trained with Internet data that is a mix of original and AI-generated data. As time goes on, a mixture of original data and data generated by different versions of AI tools will populate the Internet. This raises a few intriguing questions: how will future versions of generative AI tools behave when trained on a mixture of real and AI generated data? Will they evolve with the new data sets or degenerate? Will evolution introduce biases in subsequent generations of generative AI tools? In this document, we explore these questions and report some very initial simulation results using a simple image-generation AI tool. These results suggest that the quality of the generated images degrades as more AI-generated data is used for training thus suggesting that generative AI may degenerate. Although these results are preliminary and cannot be generalised without further study, they serve to illustrate the potential issues of the interaction between generative AI and the Internet. △ Less

Submitted 17 February, 2023; originally announced March 2023.

Comments: First version

Beyond 5G Domainless Network Operation enabled by Multiband: Toward Optical Continuum Architectures

Authors: Oscar Gonzalez de Dios, Ramon Casellas, Filippo Cugini, Jose Alberto Hernandez

Abstract: Both public and private innovation projects are targeting the design, prototy** and demonstration of a novel end-to-end integrated packet-optical transport architecture based on Multi-Band (MB) optical transmission and switching networks. Essentially, MB is expected to be the next technological evolution to deal with the traffic demand and service requirements of 5G mobile networks, and beyond,… ▽ More Both public and private innovation projects are targeting the design, prototy** and demonstration of a novel end-to-end integrated packet-optical transport architecture based on Multi-Band (MB) optical transmission and switching networks. Essentially, MB is expected to be the next technological evolution to deal with the traffic demand and service requirements of 5G mobile networks, and beyond, in the most cost-effective manner. Thanks to MB transmission, classical telco architectures segmented into hierarchical levels and domains can move forward toward an optical network continuum, where edge access nodes are all-optically interconnected with top-hierarchical nodes, interfacing Content Delivery Networks (CDN) and Internet Exchange Points (IXP). This article overviews the technological challenges and innovation requirements to enable such an architectural shift of telco networks both from a data and control and management planes. △ Less

Submitted 16 February, 2023; originally announced February 2023.

Round Trip Time (RTT) Delay in the Internet: Analysis and Trends

Authors: Gonzalo Martínez, José Alberto Hernández, Pedro Reviriego, Paul Reinheimer

Abstract: Both capacity and latency are crucial performance metrics for the optimal operation of most networking services and applications, from online gaming to futuristic holographic-type communications. Networks worldwide have witnessed important breakthroughs in terms of capacity, including fibre introduction everywhere, new radio technologies and faster core networks. However, the impact of these capac… ▽ More Both capacity and latency are crucial performance metrics for the optimal operation of most networking services and applications, from online gaming to futuristic holographic-type communications. Networks worldwide have witnessed important breakthroughs in terms of capacity, including fibre introduction everywhere, new radio technologies and faster core networks. However, the impact of these capacity upgrades on end-to-end delay is not straightforward as traffic has also grown exponentially. This article overviews the current status of end-to-end latency on different regions and continents worldwide and how far these are from the theoretical minimum baseline, given by the speed of light propagation over an optical fibre. We observe that the trend in the last decade goes toward latency reduction (in spite of the ever-increasing annual traffic growth), but still there are important differences between countries. △ Less

Submitted 8 June, 2023; v1 submitted 18 January, 2023; originally announced January 2023.

Searching for Uncollected Litter with Computer Vision

Authors: Julian Hernandez, Dr. Clark Fitzgerald

Abstract: This study combines photo metadata and computer vision to quantify where uncollected litter is present. Images from the Trash Annotations in Context (TACO) dataset were used to teach an algorithm to detect 10 categories of garbage. Although it worked well with smartphone photos, it struggled when trying to process images from vehicle mounted cameras. However, increasing the variety of perspectives… ▽ More This study combines photo metadata and computer vision to quantify where uncollected litter is present. Images from the Trash Annotations in Context (TACO) dataset were used to teach an algorithm to detect 10 categories of garbage. Although it worked well with smartphone photos, it struggled when trying to process images from vehicle mounted cameras. However, increasing the variety of perspectives and backgrounds in the dataset will help it improve in unfamiliar situations. These data are plotted onto a map which, as accuracy improves, could be used for measuring waste management strategies and quantifying trends. △ Less

Submitted 27 November, 2022; originally announced November 2022.

Comments: 17 pages, 6 figures

Link and Network-wide Study of Incoherent GN/EGN Models

Authors: Farhad Arpanaei, M. Ranjbar Zefreh, Jose A. Hernandez, Andrea Carena, David Larrabeiti

Abstract: An unprecedented comparison of closed-form incoherent GN (InGN) models is presented with heterogeneous spans and partially loaded links in elastic optical networks. Results reveal that with accumulated dispersion correction and modulation format terms, the InGN shows higher accuracy. An unprecedented comparison of closed-form incoherent GN (InGN) models is presented with heterogeneous spans and partially loaded links in elastic optical networks. Results reveal that with accumulated dispersion correction and modulation format terms, the InGN shows higher accuracy. △ Less

Submitted 17 October, 2022; originally announced October 2022.

SCAMPS: Synthetics for Camera Measurement of Physiological Signals

Authors: Daniel McDuff, Miah Wander, Xin Liu, Brian L. Hill, Javier Hernandez, Jonathan Lester, Tadas Baltrusaitis

Abstract: The use of cameras and computational algorithms for noninvasive, low-cost and scalable measurement of physiological (e.g., cardiac and pulmonary) vital signs is very attractive. However, diverse data representing a range of environments, body motions, illumination conditions and physiological states is laborious, time consuming and expensive to obtain. Synthetic data have proven a valuable tool in… ▽ More The use of cameras and computational algorithms for noninvasive, low-cost and scalable measurement of physiological (e.g., cardiac and pulmonary) vital signs is very attractive. However, diverse data representing a range of environments, body motions, illumination conditions and physiological states is laborious, time consuming and expensive to obtain. Synthetic data have proven a valuable tool in several areas of machine learning, yet are not widely available for camera measurement of physiological states. Synthetic data offer "perfect" labels (e.g., without noise and with precise synchronization), labels that may not be possible to obtain otherwise (e.g., precise pixel level segmentation maps) and provide a high degree of control over variation and diversity in the dataset. We present SCAMPS, a dataset of synthetics containing 2,800 videos (1.68M frames) with aligned cardiac and respiratory signals and facial action intensities. The RGB frames are provided alongside segmentation maps. We provide precise descriptive statistics about the underlying waveforms, including inter-beat interval, heart rate variability, and pulse arrival time. Finally, we present baseline results training on these synthetic data and testing on real-world datasets to illustrate generalizability. △ Less

Submitted 8 June, 2022; originally announced June 2022.

A suite of diagnostic metrics for characterizing selection schemes

Authors: Jose Guadalupe Hernandez, Alexander Lale**i, Charles Ofria

Abstract: Benchmark suites are crucial for assessing the performance of evolutionary algorithms, but the constituent problems are often too complex to provide clear intuition about an algorithm's strengths and weaknesses. To address this gap, we introduce DOSSIER ("Diagnostic Overview of Selection Schemes In Evolutionary Runs"), a diagnostic suite initially composed of eight handcrafted metrics. These metri… ▽ More Benchmark suites are crucial for assessing the performance of evolutionary algorithms, but the constituent problems are often too complex to provide clear intuition about an algorithm's strengths and weaknesses. To address this gap, we introduce DOSSIER ("Diagnostic Overview of Selection Schemes In Evolutionary Runs"), a diagnostic suite initially composed of eight handcrafted metrics. These metrics are designed to empirically measure specific capacities for exploitation, exploration, and their interactions. We consider exploitation both with and without constraints, and we divide exploration into two aspects: diversity exploration (the ability to simultaneously explore multiple pathways) and valley-crossing exploration (the ability to cross wider and wider fitness valleys). We apply DOSSIER to six popular selection schemes: truncation, tournament, fitness sharing, lexicase, nondominated sorting, and novelty search. Our results confirm that simple schemes (e.g., tournament and truncation) emphasized exploitation. For more sophisticated schemes, however, our diagnostics revealed interesting dynamics. Lexicase selection performed moderately well across all diagnostics that did not incorporate valley crossing, but faltered dramatically whenever valleys were present, performing worse than even random search. Fitness sharing was the only scheme to effectively contend with valley crossing but it struggled with the other diagnostics. Our study highlights the utility of using diagnostics to gain nuanced insights into selection scheme characteristics, which can inform the design of new selection methods. △ Less

Submitted 23 October, 2023; v1 submitted 28 April, 2022; originally announced April 2022.

Comments: Incorporated valley crossing diagnostics and results. Also refactored paper to focus on three key problem characteristics

Predicting the impact of treatments over time with uncertainty aware neural differential equations

Authors: Edward De Brouwer, Javier González Hernández, Stephanie Hyland

Abstract: Predicting the impact of treatments from observational data only still represents a majorchallenge despite recent significant advances in time series modeling. Treatment assignments are usually correlated with the predictors of the response, resulting in a lack of data support for counterfactual predictions and therefore in poor quality estimates. Developments in causal inference have lead to meth… ▽ More Predicting the impact of treatments from observational data only still represents a majorchallenge despite recent significant advances in time series modeling. Treatment assignments are usually correlated with the predictors of the response, resulting in a lack of data support for counterfactual predictions and therefore in poor quality estimates. Developments in causal inference have lead to methods addressing this confounding by requiring a minimum level of overlap. However,overlap is difficult to assess and usually notsatisfied in practice. In this work, we propose Counterfactual ODE (CF-ODE), a novel method to predict the impact of treatments continuously over time using Neural Ordinary Differential Equations equipped with uncertainty estimates. This allows to specifically assess which treatment outcomes can be reliably predicted. We demonstrate over several longitudinal data sets that CF-ODE provides more accurate predictions and more reliable uncertainty estimates than previously available methods. △ Less

Submitted 24 February, 2022; originally announced February 2022.

Journal ref: AISTATS 2022

Synthetic Data for Multi-Parameter Camera-Based Physiological Sensing

Authors: Daniel McDuff, Xin Liu, Javier Hernandez, Erroll Wood, Tadas Baltrusaitis

Abstract: Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camera-based physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded… ▽ More Synthetic data is a powerful tool in training data hungry deep learning algorithms. However, to date, camera-based physiological sensing has not taken full advantage of these techniques. In this work, we leverage a high-fidelity synthetics pipeline for generating videos of faces with faithful blood flow and breathing patterns. We present systematic experiments showing how physiologically-grounded synthetic data can be used in training camera-based multi-parameter cardiopulmonary sensing. We provide empirical evidence that heart and breathing rate measurement accuracy increases with the number of synthetic avatars in the training set. Furthermore, training with avatars with darker skin types leads to better overall performance than training with avatars with lighter skin types. Finally, we discuss the opportunities that synthetics present in the domain of camera-based physiological sensing and limitations that need to be overcome. △ Less

Submitted 10 October, 2021; originally announced October 2021.

Panoptic Narrative Grounding

Authors: C. González, N. Ayobi, I. Hernández, J. Hernández, J. Pont-Tuset, P. Arbeláez

Abstract: This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as step** stone for future work. We exploit the intrinsic semantic richness in an image by includ… ▽ More This paper proposes Panoptic Narrative Grounding, a spatially fine and general formulation of the natural language visual grounding problem. We establish an experimental framework for the study of this new task, including new ground truth and metrics, and we propose a strong baseline method to serve as step** stone for future work. We exploit the intrinsic semantic richness in an image by including panoptic categories, and we approach visual grounding at a fine-grained level by using segmentations. In terms of ground truth, we propose an algorithm to automatically transfer Localized Narratives annotations to specific regions in the panoptic segmentations of the MS COCO dataset. To guarantee the quality of our annotations, we take advantage of the semantic structure contained in WordNet to exclusively incorporate noun phrases that are grounded to a meaningfully related panoptic segmentation region. The proposed baseline achieves a performance of 55.4 absolute Average Recall points. This result is a suitable foundation to push the envelope further in the development of methods for Panoptic Narrative Grounding. △ Less

Submitted 10 September, 2021; originally announced September 2021.

Comments: 10 pages, 6 figures, to appear at ICCV 2021 (Oral presentation)

What can phylogenetic metrics tell us about useful diversity in evolutionary algorithms?

Authors: Jose Guadalupe Hernandez, Alexander Lale**i, Emily Dolson

Abstract: It is generally accepted that "diversity" is associated with success in evolutionary algorithms. However, diversity is a broad concept that can be measured and defined in a multitude of ways. To date, most evolutionary computation research has measured diversity using the richness and/or evenness of a particular genotypic or phenotypic property. While these metrics are informative, we hypothesize… ▽ More It is generally accepted that "diversity" is associated with success in evolutionary algorithms. However, diversity is a broad concept that can be measured and defined in a multitude of ways. To date, most evolutionary computation research has measured diversity using the richness and/or evenness of a particular genotypic or phenotypic property. While these metrics are informative, we hypothesize that other diversity metrics are more strongly predictive of success. Phylogenetic diversity metrics are a class of metrics popularly used in biology, which take into account the evolutionary history of a population. Here, we investigate the extent to which 1) these metrics provide different information than those traditionally used in evolutionary computation, and 2) these metrics better predict the long-term success of a run of evolutionary computation. We find that, in most cases, phylogenetic metrics behave meaningfully differently from other diversity metrics. Moreover, our results suggest that phylogenetic diversity is indeed a better predictor of success. △ Less

Submitted 28 August, 2021; originally announced August 2021.

Comments: 21 page, 7 figures. Presented Genetic Programming in Theory and Practice, 2021

ACM Class: I.2.2

Cooperation dynamics under pandemic risks and heterogeneous economic interdependence

Authors: Manuel Chica, Juan M. Hernandez, Francisco C. Santos

Abstract: The spread of COVID-19 and ensuing containment measures have accentuated the profound interdependence among nations or regions. This has been particularly evident in tourism, one of the sectors most affected by uncoordinated mobility restrictions. The impact of this interdependence on the tendency to adopt less or more restrictive measures is hard to evaluate, more so if diversity in economic expo… ▽ More The spread of COVID-19 and ensuing containment measures have accentuated the profound interdependence among nations or regions. This has been particularly evident in tourism, one of the sectors most affected by uncoordinated mobility restrictions. The impact of this interdependence on the tendency to adopt less or more restrictive measures is hard to evaluate, more so if diversity in economic exposures to citizens' mobility are considered. Here, we address this problem by develo** an analytical and computational game-theoretical model encompassing the conflicts arising from the need to control the economic effects of global risks, such as in the COVID-19 pandemic. The model includes the individual costs derived from severe restrictions imposed by governments, including the resulting economic interdependence among all the parties involved in the game. By using tourism-based data, the model is enriched with actual heterogeneous income losses, such that every player has a different economic cost when applying restrictions. We show that economic interdependence enhances cooperation because of the decline in the expected payoffs by free-riding parties (i.e., those neglecting the application of mobility restrictions). Furthermore, we show (analytically and through numerical simulations) that these cross-exposures can transform the nature of the cooperation dilemma each region or country faces, modifying the position of the fixed points and the size of the basins of attraction that characterize this class of games. Finally, our results suggest that heterogeneity among regions may be used to leverage the impact of intervention policies by ensuring an agreement among the most relevant initial set of cooperators. △ Less

Submitted 30 July, 2021; originally announced August 2021.

An Exploration of Exploration: Measuring the ability of lexicase selection to find obscure pathways to optimality

Authors: Jose Guadalupe Hernandez, Alexander Lale**i, Charles Ofria

Abstract: Parent selection algorithms (selection schemes) steer populations through a problem's search space, often trading off between exploitation and exploration. Understanding how selection schemes affect exploitation and exploration within a search space is crucial to tackling increasingly challenging problems. Here, we introduce an "exploration diagnostic" that diagnoses a selection scheme's capacity… ▽ More Parent selection algorithms (selection schemes) steer populations through a problem's search space, often trading off between exploitation and exploration. Understanding how selection schemes affect exploitation and exploration within a search space is crucial to tackling increasingly challenging problems. Here, we introduce an "exploration diagnostic" that diagnoses a selection scheme's capacity for search space exploration. We use our exploration diagnostic to investigate the exploratory capacity of lexicase selection and several of its variants: epsilon lexicase, down-sampled lexicase, cohort lexicase, and novelty-lexicase. We verify that lexicase selection out-explores tournament selection, and we show that lexicase selection's exploratory capacity can be sensitive to the ratio between population size and the number of test cases used for evaluating candidate solutions. Additionally, we find that relaxing lexicase's elitism with epsilon lexicase can further improve exploration. Both down-sampling and cohort lexicase -- two techniques for applying random subsampling to test cases -- degrade lexicase's exploratory capacity; however, we find that cohort partitioning better preserves lexicase's exploratory capacity than down-sampling. Finally, we find evidence that novelty-lexicase's addition of novelty test cases can degrade lexicase's capacity for exploration. Overall, our findings provide hypotheses for further exploration and actionable insights and recommendations for using lexicase selection. Additionally, this work demonstrates the value of selection scheme diagnostics as a complement to more conventional benchmarking approaches to selection scheme analysis. △ Less

Submitted 26 July, 2021; v1 submitted 20 July, 2021; originally announced July 2021.

Comments: Changes to the axis labels and added funding sources to acknowledgments

Blockchains' federation for integrating distributed health data using a patient-centered approach

Authors: Javier Rojo, Juan Hernandez, Juan Manuel Murillo, Jose Garcia-Alonso

Abstract: Today's world is a globalized and connected one, where people are increasingly moving around and interacting with a greater number of services and devices of all kinds, including those that allow them to monitor their health. However, each company, institution or health system usually store its patients' data in an isolated way. Although this approach can have some benefits related with privacy, s… ▽ More Today's world is a globalized and connected one, where people are increasingly moving around and interacting with a greater number of services and devices of all kinds, including those that allow them to monitor their health. However, each company, institution or health system usually store its patients' data in an isolated way. Although this approach can have some benefits related with privacy, security, etc., it also implies that each one of them generates different, incomplete and possibly contradictory views of a patient's health data, losing part of the value that this information could bring to the patient. That is the reason why researchers from all over the world are determined to replace the current institution-centered health systems with new patient-centered ones. In these new systems, all the health information of a patient is integrated into a unique global vision. However, some questions are still unanswered. Specifically, who should store and maintain the information of a given patient and how should this information be made available for other systems. To address this situation, this work proposes a new solution towards making the Personal Health Trajectory of patients available for both, the patients themselves and health institutions. By using the concept of blockchains' federation and web services access to the global vision of a person health can be granted to existing and new solutions. To demonstrate the viability of the proposal, an implementation is provided alongside the obtained results in a potential scenario. △ Less

Submitted 9 March, 2021; originally announced March 2021.

Comments: 8 pages, 4 figures

DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization

Authors: Javier Hernandez, Daniel McDuff, Ognjen, Rudovic, Alberto Fung, Mary Czerwinski

Abstract: Facial action unit recognition has many applications from market research to psychotherapy and from image captioning to entertainment. Despite its recent progress, deployment of these models has been impeded due to their limited generalization to unseen people and demographics. This work conducts an in-depth analysis of performance across several dimensions: individuals(40 subjects), genders (male… ▽ More Facial action unit recognition has many applications from market research to psychotherapy and from image captioning to entertainment. Despite its recent progress, deployment of these models has been impeded due to their limited generalization to unseen people and demographics. This work conducts an in-depth analysis of performance across several dimensions: individuals(40 subjects), genders (male and female), skin types (darker and lighter), and databases (BP4D and DISFA). To help suppress the variance in data, we use the notion of self-supervised denoising autoencoders to design a method for deep face normalization(DeepFN) that transfers facial expressions of different people onto a common facial template which is then used to train and evaluate facial action recognition models. We show that person-independent models yield significantly lower performance (55% average F1 and accuracy across 40 subjects) than person-dependent models (60.3%), leading to a generalization gap of 5.3%. However, normalizing the data with the newly introduced DeepFN significantly increased the performance of person-independent models (59.6%), effectively reducing the gap. Similarly, we observed generalization gaps when considering gender (2.4%), skin type (5.3%), and dataset (9.4%), which were significantly reduced with the use of DeepFN. These findings represent an important step towards the creation of more generalizable facial action unit recognition systems. △ Less

Submitted 3 March, 2021; originally announced March 2021.

Journal ref: 2022 10th International Conference on Affective Computing and Intelligent Interaction (ACII)

AffectiveSpotlight: Facilitating the Communication of Affective Responses from Audience Members during Online Presentations

Authors: Prasanth Murali, Javier Hernandez, Daniel McDuff, Kael Rowan, **a Suh, Mary Czerwinski

Abstract: The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N = 175), we assessed the most relevant audience responses such as confusion… ▽ More The ability to monitor audience reactions is critical when delivering presentations. However, current videoconferencing platforms offer limited solutions to support this. This work leverages recent advances in affect sensing to capture and facilitate communication of relevant audience signals. Using an exploratory survey (N = 175), we assessed the most relevant audience responses such as confusion, engagement, and head-nods. We then implemented AffectiveSpotlight, a Microsoft Teams bot that analyzes facial responses and head gestures of audience members and dynamically spotlights the most expressive ones. In a within-subjects study with 14 groups (N = 117), we observed that the system made presenters significantly more aware of their audience, speak for a longer period of time, and self-assess the quality of their talk more similarly to the audience members, compared to two control conditions (randomly-selected spotlight and default platform UI). We provide design recommendations for future affective interfaces for online presentations based on feedback from the study. △ Less

Submitted 28 January, 2021; originally announced January 2021.

An Evolutionary Game Model for Understanding Fraud in Consumption Taxes

Authors: M. Chica, J. Hernandez, C. Manrique-de-Lara-Peñate, R. Chiong

Abstract: This paper presents a computational evolutionary game model to study and understand fraud dynamics in the consumption tax system. Players are cooperators if they correctly declare their value added tax (VAT), and are defectors otherwise. Each player's payoff is influenced by the amount evaded and the subjective probability of being inspected by tax authorities. Since transactions between companies… ▽ More This paper presents a computational evolutionary game model to study and understand fraud dynamics in the consumption tax system. Players are cooperators if they correctly declare their value added tax (VAT), and are defectors otherwise. Each player's payoff is influenced by the amount evaded and the subjective probability of being inspected by tax authorities. Since transactions between companies must be declared by both the buyer and seller, a strategy adopted by one influences the other's payoff. We study the model with a well-mixed population and different scale-free networks. Model parameters were calibrated using real-world data of VAT declarations by businesses registered in the Canary Islands region of Spain. We analyzed several scenarios of audit probabilities for high and low transactions and their prevalence in the population, as well as social rewards and penalties to find the most efficient policy to increase the proportion of cooperators. Two major insights were found. First, increasing the subjective audit probability for low transactions is more efficient than increasing this probability for high transactions. Second, favoring social rewards for cooperators or alternative penalties for defectors can be effective policies, but their success depends on the distribution of the audit probability for low and high transactions. △ Less

Submitted 12 January, 2021; originally announced January 2021.

Advancing Non-Contact Vital Sign Measurement using Synthetic Avatars

Authors: Daniel McDuff, Javier Hernandez, Erroll Wood, Xin Liu, Tadas Baltrusaitis

Abstract: Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring. However, machine vision approaches are often limited by the availability and diversity of annotated video datasets resulting in poor generalization to complex real-life conditions. To address these challenges, this work proposes the use of synthetic avatars that display facial blood flow ch… ▽ More Non-contact physiological measurement has the potential to provide low-cost, non-invasive health monitoring. However, machine vision approaches are often limited by the availability and diversity of annotated video datasets resulting in poor generalization to complex real-life conditions. To address these challenges, this work proposes the use of synthetic avatars that display facial blood flow changes and allow for systematic generation of samples under a wide variety of conditions. Our results show that training on both simulated and real video data can lead to performance gains under challenging conditions. We show state-of-the-art performance on three large benchmark datasets and improved robustness to skin type and motion. △ Less

Submitted 24 October, 2020; originally announced October 2020.

Kids Today: Remote Education in the time of COVID-19

Authors: Adriana Mejia Castaño, Javier E Hernandez, Angie Mendez Llanos

Abstract: With the recent COVID-19 breakup, it became necessary to implement remote classes in schools and universities to safeguard health and life. However, many students (teachers and parents, also) face great difficulties accessing and staying in class due to technology limitations, affecting their education. Using several nationally representative datasets in Colombia, this article documents how the ac… ▽ More With the recent COVID-19 breakup, it became necessary to implement remote classes in schools and universities to safeguard health and life. However, many students (teachers and parents, also) face great difficulties accessing and staying in class due to technology limitations, affecting their education. Using several nationally representative datasets in Colombia, this article documents how the academic performance of students in their final high school year is affected due to technologies, aggregated by municipalities. We conclude that internet access strongly affects these results, and little improvement on the internet/computer access will reflect better academic performance. Under these conditions, belonging to an ethnic group or high rurality (non-geographic centralized municipalities) has a negative impact. Policy implications are discussed. △ Less

Submitted 14 October, 2020; originally announced October 2020.

Defining Adaptive Proxemic Zones for Activity-aware Navigation

Authors: Jonatan Gines Clavero, Francisco Martin Rico, Francisco J. Rodriguez-Lera, Jose Miguel Guerrero Hernandez, Vicente Matellan Olivera

Abstract: Many of the tasks that a service robot can perform at home involve navigation skills. In a real world scenario, the navigation system should consider individuals beyond just objects, theses days it is necessary to offer particular and dynamic representation in the scenario in order to enhance the HRI experience. In this paper, we use the proxemic theory to do this representation. The proxemic zone… ▽ More Many of the tasks that a service robot can perform at home involve navigation skills. In a real world scenario, the navigation system should consider individuals beyond just objects, theses days it is necessary to offer particular and dynamic representation in the scenario in order to enhance the HRI experience. In this paper, we use the proxemic theory to do this representation. The proxemic zones are not static. The culture or the context influences them and, if we have this influence into account, we can increase humans' comfort. Moreover, there are collaborative tasks in which these zones take different shapes to allow the task's best performance. This research develops a layer, the social layer, to represent and distribute the proxemics zones' information in a standard way, through a cost map and using it to perform a social navigate task. We have evaluated these components in a simulated scenario, performing different collaborative and human-robot interaction tasks and reducing the personal area invasion in a 32\%. The material developed during this research can be found in a public repository, as well as instructions to facilitate the reproducibility of the results. △ Less

Submitted 10 September, 2020; originally announced September 2020.