Showing 1–10 of 10 results for author: Parker, R

Continually Learned Pavlovian Signalling Without Forgetting for Human-in-the-Loop Robotic Control

Authors: Adam S. R. Parker, Michael R. Dawson, Patrick M. Pilarski

Abstract: Artificial limbs are sophisticated devices to assist people with tasks of daily living. Despite advanced robotic prostheses demonstrating similar motion capabilities to biological limbs, users report them difficult and non-intuitive to use. Providing more effective feedback from the device to the user has therefore become a topic of increased interest. In particular, prediction learning methods fr… ▽ More Artificial limbs are sophisticated devices to assist people with tasks of daily living. Despite advanced robotic prostheses demonstrating similar motion capabilities to biological limbs, users report them difficult and non-intuitive to use. Providing more effective feedback from the device to the user has therefore become a topic of increased interest. In particular, prediction learning methods from the field of reinforcement learning -- specifically, an approach termed Pavlovian signalling -- have been proposed as one approach for better modulating feedback in prostheses since they can adapt during continuous use. One challenge identified in these learning methods is that they can forget previously learned predictions when a user begins to successfully act upon delivered feedback. The present work directly addresses this challenge, contributing new evidence on the impact of algorithmic choices, such as on- or off-policy methods and representation choices, on the Pavlovian signalling from a machine to a user during their control of a robotic arm. Two conditions of algorithmic differences were studied using different scenarios of controlling a robotic arm: an automated motion system and human participant piloting. Contrary to expectations, off-policy learning did not provide the expected solution to the forgetting problem. We instead identified beneficial properties of a look-ahead state representation that made existing approaches able to learn (and not forget) predictions in support of Pavlovian signalling. This work therefore contributes new insight into the challenges of providing learned predictive feedback from a prosthetic device, and demonstrates avenues for more dynamic signalling in future human-machine interactions. △ Less

Submitted 16 May, 2023; originally announced May 2023.

Comments: 12 pages inc. supplementary, 7 figures, 3 algorithms, Published the NeurIPS Workshop on Human in the Loop Learning, Nov 28 - Dec 8 2022

Joint Action is a Framework for Understanding Partnerships Between Humans and Upper Limb Prostheses

Authors: Michael R. Dawson, Adam S. R. Parker, Heather E. Williams, Ahmed W. Shehata, Jacqueline S. Hebert, Craig S. Chapman, Patrick M. Pilarski

Abstract: Recent advances in upper limb prostheses have led to significant improvements in the number of movements provided by the robotic limb. However, the method for controlling multiple degrees of freedom via user-generated signals remains challenging. To address this issue, various machine learning controllers have been developed to better predict movement intent. As these controllers become more intel… ▽ More Recent advances in upper limb prostheses have led to significant improvements in the number of movements provided by the robotic limb. However, the method for controlling multiple degrees of freedom via user-generated signals remains challenging. To address this issue, various machine learning controllers have been developed to better predict movement intent. As these controllers become more intelligent and take on more autonomy in the system, the traditional approach of representing the human-machine interface as a human controlling a tool becomes limiting. One possible approach to improve the understanding of these interfaces is to model them as collaborative, multi-agent systems through the lens of joint action. The field of joint action has been commonly applied to two human partners who are trying to work jointly together to achieve a task, such as singing or moving a table together, by effecting coordinated change in their shared environment. In this work, we compare different prosthesis controllers (proportional electromyography with sequential switching, pattern recognition, and adaptive switching) in terms of how they present the hallmarks of joint action. The results of the comparison lead to a new perspective for understanding how existing myoelectric systems relate to each other, along with recommendations for how to improve these systems by increasing the collaborative communication between each partner. △ Less

Submitted 28 December, 2022; originally announced December 2022.

Comments: Submitted to Frontiers in Neurorobotics

The Frost Hollow Experiments: Pavlovian Signalling as a Path to Coordination and Communication Between Agents

Authors: Patrick M. Pilarski, Andrew Butcher, Elnaz Davoodi, Michael Bradley Johanson, Dylan J. A. Brenneis, Adam S. R. Parker, Leslie Acker, Matthew M. Botvinick, Joseph Modayil, Adam White

Abstract: Learned communication between agents is a powerful tool when approaching decision-making problems that are hard to overcome by any single agent in isolation. However, continual coordination and communication learning between machine agents or human-machine partnerships remains a challenging open problem. As a step** stone toward solving the continual communication learning problem, in this paper… ▽ More Learned communication between agents is a powerful tool when approaching decision-making problems that are hard to overcome by any single agent in isolation. However, continual coordination and communication learning between machine agents or human-machine partnerships remains a challenging open problem. As a step** stone toward solving the continual communication learning problem, in this paper we contribute a multi-faceted study into what we term Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent with different perceptual access to their shared environment. We seek to establish how different temporal processes and representational choices impact Pavlovian signalling between learning agents. To do so, we introduce a partially observable decision-making domain we call the Frost Hollow. In this domain a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that seeks to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: 1) machine prediction and control learning in a linear walk, and 2) a prediction learning machine interacting with a human participant in a virtual reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning. Our results therefore point to an actionable, constructivist path towards continual communication learning between reinforcement learning agents, with potential impact in a range of real-world settings. △ Less

Submitted 17 March, 2022; originally announced March 2022.

Comments: 54 pages, 29 figures, 4 tables

Breath analysis by ultra-sensitive broadband laser spectroscopy detects SARS-CoV-2 infection

Authors: Qizhong Liang, Ya-Chu Chan, Jutta Toscano, Kristen K. Bjorkman, Leslie A. Leinwand, Roy Parker, Eva S. Nozik, David J. Nesbitt, Jun Ye

Abstract: Rapid testing is essential to fighting pandemics such as COVID-19, the disease caused by the SARS-CoV-2 virus. Exhaled human breath contains multiple volatile molecules providing powerful potential for non-invasive diagnosis of diverse medical conditions. We investigated breath detection of SARS-CoV-2 infection using cavity-enhanced direct frequency comb spectroscopy (CE-DFCS), a state-of-the-art… ▽ More Rapid testing is essential to fighting pandemics such as COVID-19, the disease caused by the SARS-CoV-2 virus. Exhaled human breath contains multiple volatile molecules providing powerful potential for non-invasive diagnosis of diverse medical conditions. We investigated breath detection of SARS-CoV-2 infection using cavity-enhanced direct frequency comb spectroscopy (CE-DFCS), a state-of-the-art laser spectroscopic technique capable of a real-time massive collection of broadband molecular absorption features at ro-vibrational quantum state resolution and at parts-per-trillion volume detection sensitivity. Using a total of 170 individual breath samples (83 positive and 87 negative with SARS-CoV-2 based on Reverse Transcription Polymerase Chain Reaction tests), we report excellent discrimination capability for SARS-CoV-2 infection with an area under the Receiver-Operating-Characteristics curve of 0.849(4). Our results support the development of CE-DFCS as an alternative, rapid, non-invasive test for COVID-19 and highlight its remarkable potential for optical diagnoses of diverse biological conditions and disease states. △ Less

Submitted 13 February, 2023; v1 submitted 4 February, 2022; originally announced February 2022.

Pavlovian Signalling with General Value Functions in Agent-Agent Temporal Decision Making

Authors: Andrew Butcher, Michael Bradley Johanson, Elnaz Davoodi, Dylan J. A. Brenneis, Leslie Acker, Adam S. R. Parker, Adam White, Joseph Modayil, Patrick M. Pilarski

Abstract: In this paper, we contribute a multi-faceted study into Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict th… ▽ More In this paper, we contribute a multi-faceted study into Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent. Signalling is intimately connected to time and timing. In service of generating and receiving signals, humans and other animals are known to represent time, determine time since past events, predict the time until a future stimulus, and both recognize and generate patterns that unfold in time. We investigate how different temporal processes impact coordination and signalling between learning agents by introducing a partially observable decision-making domain we call the Frost Hollow. In this domain, a prediction learning agent and a reinforcement learning agent are coupled into a two-part decision-making system that works to acquire sparse reward while avoiding time-conditional hazards. We evaluate two domain variations: machine agents interacting in a seven-state linear walk, and human-machine interaction in a virtual-reality environment. Our results showcase the speed of learning for Pavlovian signalling, the impact that different temporal representations do (and do not) have on agent-agent coordination, and how temporal aliasing impacts agent-agent and human-agent interactions differently. As a main contribution, we establish Pavlovian signalling as a natural bridge between fixed signalling paradigms and fully adaptive communication learning between two agents. We further show how to computationally build this adaptive signalling process out of a fixed signalling process, characterized by fast continual prediction learning and minimal constraints on the nature of the agent receiving signals. Our results therefore suggest an actionable, constructivist path towards communication learning between reinforcement learning agents. △ Less

Submitted 10 January, 2022; originally announced January 2022.

Comments: 9 pages, 7 figures

5G Network Slicing with QKD and Quantum-Safe Security

Authors: Paul Wright, Catherine White, Ryan C. Parker, Jean-Sébastien Pegon, Marco Menchetti, Joseph Pearse, Arash Bahrami, Anastasia Moroz, Adrian Wonfor, Richard V. Penty, Timothy P. Spiller, Andrew Lord

Abstract: We demonstrate how the 5G network slicing model can be extended to address data security requirements. In this work we demonstrate two different slice configurations, with different encryption requirements, representing two diverse use-cases for 5G networking: namely, an enterprise application hosted at a metro network site, and a content delivery network. We create a modified software-defined net… ▽ More We demonstrate how the 5G network slicing model can be extended to address data security requirements. In this work we demonstrate two different slice configurations, with different encryption requirements, representing two diverse use-cases for 5G networking: namely, an enterprise application hosted at a metro network site, and a content delivery network. We create a modified software-defined networking (SDN) orchestrator which calculates and provisions network slices according to the requirements, including encryption backed by quantum key distribution (QKD), or other methods. Slices are automatically provisioned by SDN orchestration of network resources, allowing selection of encrypted links as appropriate, including those which use standard Diffie-Hellman key exchange, QKD and quantum-resistant algorithms (QRAs), as well as no encryption at all. We show that the set-up and tear-down times of the network slices takes of the order of 1-2 minutes, which is an order of magnitude improvement over manually provisioning a link today. △ Less

Submitted 8 January, 2021; v1 submitted 7 July, 2020; originally announced July 2020.

Comments: 9 pages, 7 figures

Journal ref: J. Opt. Commun. Netw. 13, 33-40 (2021)

Learned human-agent decision-making, communication and joint action in a virtual reality environment

Authors: Patrick M. Pilarski, Andrew Butcher, Michael Johanson, Matthew M. Botvinick, Andrew Bolt, Adam S. R. Parker

Abstract: Humans make decisions and act alongside other humans to pursue both short-term and long-term goals. As a result of ongoing progress in areas such as computing science and automation, humans now also interact with non-human agents of varying complexity as part of their day-to-day activities; substantial work is being done to integrate increasingly intelligent machine agents into human work and play… ▽ More Humans make decisions and act alongside other humans to pursue both short-term and long-term goals. As a result of ongoing progress in areas such as computing science and automation, humans now also interact with non-human agents of varying complexity as part of their day-to-day activities; substantial work is being done to integrate increasingly intelligent machine agents into human work and play. With increases in the cognitive, sensory, and motor capacity of these agents, intelligent machinery for human assistance can now reasonably be considered to engage in joint action with humans---i.e., two or more agents adapting their behaviour and their understanding of each other so as to progress in shared objectives or goals. The mechanisms, conditions, and opportunities for skillful joint action in human-machine partnerships is of great interest to multiple communities. Despite this, human-machine joint action is as yet under-explored, especially in cases where a human and an intelligent machine interact in a persistent way during the course of real-time, daily-life experience. In this work, we contribute a virtual reality environment wherein a human and an agent can adapt their predictions, their actions, and their communication so as to pursue a simple foraging task. In a case study with a single participant, we provide an example of human-agent coordination and decision-making involving prediction learning on the part of the human and the machine agent, and control learning on the part of the machine agent wherein audio communication signals are used to cue its human partner in service of acquiring shared reward. These comparisons suggest the utility of studying human-machine coordination in a virtual reality environment, and identify further research that will expand our understanding of persistent human-machine joint action. △ Less

Submitted 7 May, 2019; originally announced May 2019.

Comments: 5 pages, 3 figures. Accepted to The 4th Multidisciplinary Conference on Reinforcement Learning and Decision Making, July 7-10, 2019, McGill University, Montreal, Quebec, Canada

A parallel algorithm for Gaussian elimination over finite fields

Authors: Stephen Linton, Gabriele Nebe, Alice Niemeyer, Richard Parker, Jon Thackray

Abstract: In this paper we describe a parallel Gaussian elimination algorithm for matrices with entries in a finite field. Unlike previous approaches, our algorithm subdivides a very large input matrix into smaller submatrices by subdividing both rows and columns into roughly square blocks sized so that computing with individual blocks on individual processors provides adequate concurrency. The algorithm al… ▽ More In this paper we describe a parallel Gaussian elimination algorithm for matrices with entries in a finite field. Unlike previous approaches, our algorithm subdivides a very large input matrix into smaller submatrices by subdividing both rows and columns into roughly square blocks sized so that computing with individual blocks on individual processors provides adequate concurrency. The algorithm also returns the transformation matrix, which encodes the row operations used. We go to some lengths to avoid storing any unnecessary data as we keep track of the row operations, such as block columns of the transformation matrix known to be zero. The algorithm is accompanied by a concurrency analysis which shows that the improvement in concurrency is of the same order of magnitude as the number of blocks. An implementation of the algorithm has been tested on matrices as large as $1 000 000\times 1 000 000$ over small finite fields. △ Less

Submitted 8 June, 2018; originally announced June 2018.

MSC Class: 15A06; 68W10; 68W05

Communicative Capital for Prosthetic Agents

Authors: Patrick M. Pilarski, Richard S. Sutton, Kory W. Mathewson, Craig Sherstan, Adam S. R. Parker, Ann L. Edwards

Abstract: This work presents an overarching perspective on the role that machine intelligence can play in enhancing human abilities, especially those that have been diminished due to injury or illness. As a primary contribution, we develop the hypothesis that assistive devices, and specifically artificial arms and hands, can and should be viewed as agents in order for us to most effectively improve their co… ▽ More This work presents an overarching perspective on the role that machine intelligence can play in enhancing human abilities, especially those that have been diminished due to injury or illness. As a primary contribution, we develop the hypothesis that assistive devices, and specifically artificial arms and hands, can and should be viewed as agents in order for us to most effectively improve their collaboration with their human users. We believe that increased agency will enable more powerful interactions between human users and next generation prosthetic devices, especially when the sensorimotor space of the prosthetic technology greatly exceeds the conventional control and communication channels available to a prosthetic user. To more concretely examine an agency-based view on prosthetic devices, we propose a new schema for interpreting the capacity of a human-machine collaboration as a function of both the human's and machine's degrees of agency. We then introduce the idea of communicative capital as a way of thinking about the communication resources developed by a human and a machine during their ongoing interaction. Using this schema of agency and capacity, we examine the benefits and disadvantages of increasing the agency of a prosthetic limb. To do so, we present an analysis of examples from the literature where building communicative capital has enabled a progression of fruitful, task-directed interactions between prostheses and their human users. We then describe further work that is needed to concretely evaluate the hypothesis that prostheses are best thought of as agents. The agent-based viewpoint developed in this article significantly extends current thinking on how best to support the natural, functional use of increasingly complex prosthetic enhancements, and opens the door for more powerful interactions between humans and their assistive technologies. △ Less

Submitted 9 November, 2017; originally announced November 2017.

Comments: 33 pages, 10 figures; unpublished technical report undergoing peer review

Using Learned Predictions as Feedback to Improve Control and Communication with an Artificial Limb: Preliminary Findings

Authors: Adam S. R. Parker, Ann L. Edwards, Patrick M. Pilarski

Abstract: Many people suffer from the loss of a limb. Learning to get by without an arm or hand can be very challenging, and existing prostheses do not yet fulfil the needs of individuals with amputations. One promising solution is to provide greater communication between a prosthesis and its user. Towards this end, we present a simple machine learning interface to supplement the control of a robotic limb w… ▽ More Many people suffer from the loss of a limb. Learning to get by without an arm or hand can be very challenging, and existing prostheses do not yet fulfil the needs of individuals with amputations. One promising solution is to provide greater communication between a prosthesis and its user. Towards this end, we present a simple machine learning interface to supplement the control of a robotic limb with feedback to the user about what the limb will be experiencing in the near future. A real-time prediction learner was implemented to predict impact-related electrical load experienced by a robot limb; the learning system's predictions were then communicated to the device's user to aid in their interactions with a workspace. We tested this system with five able-bodied subjects. Each subject manipulated the robot arm while receiving different forms of vibrotactile feedback regarding the arm's contact with its workspace. Our trials showed that communicable predictions could be learned quickly during human control of the robot arm. Using these predictions as a basis for feedback led to a statistically significant improvement in task performance when compared to purely reactive feedback from the device. Our study therefore contributes initial evidence that prediction learning and machine intelligence can benefit not just control, but also feedback from an artificial limb. We expect that a greater level of acceptance and ownership can be achieved if the prosthesis itself takes an active role in transmitting learned knowledge about its state and its situation of use. △ Less

Submitted 8 August, 2014; originally announced August 2014.

Comments: 7 pages, 5 figures