Guiding Soft Robots with Motor-Imagery Brain Signals and Impedance Control

Abstract

Integrating Brain-Machine Interfaces into non-clinical applications like robot motion control remains difficult - despite remarkable advancements in clinical settings. Specifically, EEG-based motor imagery systems are still error-prone, posing safety risks when rigid robots operate near humans. This work presents an alternative pathway towards safe and effective operation by combining wearable EEG with physically embodied safety in soft robots. We introduce and test a pipeline that allows a user to move a soft robot’s end effector in real time via brain waves that are measured by as few as three EEG channels. A robust motor imagery algorithm interprets the user’s intentions to move the position of a virtual attractor to which the end effector is attracted, thanks to a new Cartesian impedance controller. We specifically focus here on planar soft robot-based architected metamaterials, which require the development of a novel control architecture to deal with the peculiar nonlinearities - e.g., non-affinity in control. We preliminarily but quantitatively evaluate the approach on the task of setpoint regulation. We observe that the user reaches the proximity of the setpoint in 66% of steps and that for successful steps, the average response time is 21.5s. We also demonstrate the execution of simple real-world tasks involving interaction with the environment, which would be extremely hard to perform if it were not for the robot’s softness.

I Introduction

BMI [1] facilitate the translation of neural activity into actionable commands, enabling individuals to control external devices and systems through their thoughts and attention [2, 3]. Compared to traditional bulky EEG setups [4], one of the emerging avenues towards practical and wearable EEG devices are systems based on motor imagery signals due to their intuitiveness and no external dependency on (e.g., visual) stimuli. These have been use in stroke rehabilitation [5]. Several works in literature have considered using this technology to control robot manipulators [6, 7, 8].

However, the state-of-the-art classifiers on few-channel, online EEG signals are still limited in achieving an accuracy of 65-75 $\mathrm{\char 37}$ [9, 8] and are prone to producing outliers, which make it very challenging to operate robots safely and robustly using these techniques [1]. In (rigid) robotics literature, this has been addressed by relying on force-based (i.e., impedance) control [6] and by making the robot’s behavior more predictable [7]. In this work, we follow a different path and investigate embodying safety by pairing soft robots [10, 11] with BMI. This way, risks can be mitigated, and more natural interactions with an unstructured environment can be achieved by relying on structural compliance.

Figure 1: Scheme of the proposed approach to control HSA robots with motor imagery. Brain signals steer an attractor in operational space: first, we switch the active coordinate axis when we detect jaw clenching. If no jaw clenching is detected, we classify the EEG signals based on left/right motor imaginations into positive and negative movements along the active axis. Next, we regulate the robot towards the chosen attractor position $x^{\mathrm{at}}$ with a Cartesian impedance controller. This controller first cancels all static forces and the residual coupling of the null space on the operational space dynamics. This allows us to now shape our own potential with a PD term in operational space. As the robot is underactuated, we optimize least-squares to identify the actuation $\phi^{\mathrm{d}}$ so that the residual between the desired and actual torques in the configuration space is minimized. Icons created by Flaticon©.

While BMI-based assistance has been investigated with a focus on soft exosuits assisting hand-rehabilitation  [12] or with strenuous arm acitivites [13], it is still an open challenge how BMI can be used for controlling soft manipulators. In this work, we make a first step towards solving this challenge by proposing a pipeline (see Fig. 1) that lets the user steer the soft robot’s end-effector in Cartesian space. The two key ingredients are a novel map** strategy transforming the brain signals into meaningful references and Cartesian impedance control. The latter is essential because it allows for preserving the robot’s compliance in closed-loop [14]. We build the proposed BMI pipeline around a HSA soft robot [15, 16], which relies on architected metamaterials and electrical actuation to elongate, bend, and twist. This makes the control problem especially challenging because of the peculiarity of these systems’ dynamics, namely underactuation and non-affinity in control. We provide more details on innovation from the model-based control standpoint in Section II-D.

We quantitatively verify the entire approach on mind-controlled setpoint regulation involving tracking a reference consisting of nine-step functions and demonstrate the qualitative behavior when assisting with a simple daily living activity. Furthermore, we compare the performance of our motor imagery-based control to approaches giving the computational controller access to the privileged information of setpoints, which can considered to be an upper bound on performance.

Our contributions are: (i) Establishing a BMI strategy for continuum soft robots. This strategy is supported by experiments in which we perform setpoint regulation with a planar HSA robot and motor imagery, (ii) A Cartesian impedance controller for HSA robots, which we experimentally validated on a simple ADL task involving environment interaction: the user needs to steer the end-effector towards the tip of a hairspray container, apply force for releasing the fluid, and finally let the robot retreat from the contact.

A video attachment to this paper including recordings of experimental results can be found on YouTube. Furthermore, we have open-sourced our code including the OpenVibe pipeline on GitHub.

II Task-space Impedance Control

In this work, we let the user steer with motor imaginary brain signals the Cartesian position $x\in\mathbb{R}^{2}$ of the end-effector (i.e., the platform) of a planar HSA robot. We realize this strategy by first classifying the motor imaginary signals into Cartesian-space movement directions (e.g., the active axis and sign of the movement). We use this information to adjust the position of a task-space attractor iteratively (see Section II-B). Section II-D describes how a model-based computational controller establishes this attractor. Importantly, we preserve the soft robot’s compliance by sha** the closed-loop system’s impedance in Cartesian space.

II-A Background: Motor Imagery-based BMI systems

Imagining the movement of body parts or limbs (e.g., hands, legs, tongue) without moving it or the mental rehearsal of a motor act without overt movement execution is termed Motor Imagery [17]. The neuronal activities observable inside a frequency range of $8\text{\,}\mathrm{H}\mathrm{z}$ to $12\text{\,}\mathrm{H}\mathrm{z}$ (Mu) and $12\text{\,}\mathrm{H}\mathrm{z}$ to $30\text{\,}\mathrm{H}\mathrm{z}$ (Beta) are associated with cortical areas directly connected to the brain’s motor output (activating primary sensorimotor areas that can be modulated with imaginary mental movement in healthy as well people with neuromuscular disabilities).

The motor imagery BMI framework typically consists of four integral components:

1. Signal acquisition: The initial stage involves the recording of neural signals while the person imagines the movements of the limbs, generally acquired using noninvasive methodologies (e.g., EEG).

2. Feature extraction: Following signal acquisition, signal processing techniques are applied to extract salient features from the neural patterns associated with specific cognitive processes or intentions.

3. Feature translation: This translation phase interprets the user’s cognitive intent, converting it into actionable instructions for external devices.

4. Device output: The culmination of the BMI process is the application of the interpreted commands to external devices.

As detailed further in Sec. III-B, we leverage the difference in signals when imagining motor actions vs. rest state to control the sign of movement. The active axis of movement can be switched by clenching the yaw.

II-B Planning attractors with brain signals

Our brain signal processing pipeline provides us with two pieces of information at each time step $k$: i) the unit vector $e_{\mathrm{a}}(k)\in\{[1,0]^{\mathrm{T}},[0,1]^{\mathrm{T}}\}$ corresponding to the current active axis of movement and ii) the sign of movement $s(k)\in\{-1,1\}$. We use $e_{\mathrm{a}}(k)$ and $s(k)$ to incrementally steer a virtual attractor defined in operational space $x^{\mathrm{at}}\in\mathbb{R}^{2}$ as follows

$$\small u(k)=s(k)\,e_{\mathrm{a}}(k)\in\mathbb{R}^{2},\quad x^{\mathrm{at}}(k)=x^{\mathrm{at}}(k-1)+\Delta_{\mathrm{x}}\,u(k),$$ (1)

where $\Delta_{\mathrm{x}}\in\mathbb{R}^{+}$ is a tunable constant influencing the velocity of the attractor movement. Later, we will shape the potential field with a computational controller such that the attractor becomes a globally asymptotically stable equilibrium (see Section II-D).

Figure 2: EEG data processing pipeline: The EEG data is acquired in real-time, pre-processed, and divided into episodes and subbands. Next, we extract power features and pass them to two LDA classifiers: the first outputs the axis of movement (for example, moving along the x- or y-axis), and the second provides the sign of movement (for example, positive or negative movement along the active axis). These commands are then used to move the attractor in Cartesian space.

II-C Background: modeling planar HSA robots

Robots based on HSA rely on rods made of architected metamaterials to generate motion. More specifically, twisting the rods along their handedness leads to an elongation of the rod [18]. Combining multiple HSA in the setting of a parallel robot and actuating them with servo motors allows us to generate complex motion primitives and offer beneficial mechanical characteristics such as a high stiffness-to-weight ratio [15, 18].

Prior work [16] has shown that the shape of planar HSA robots can be approximated by one CS segment. Therefore, we define the configuration of the system as $q=\begin{bmatrix}\kappa_{\mathrm{be}}&\sigma_{\mathrm{sh}}&\sigma_{\mathrm{ax}}\end{bmatrix}^{\mathrm{T}}\in\mathbb{R}^{3}$. We also have access to closed-form formulations of the forward kinematics $\pi:q\rightarrow\chi$ and inverse kinematics $\varrho:\chi\rightarrow q$ where $\chi=\begin{bmatrix}x^{\mathrm{T}}&\theta\end{bmatrix}^{\mathrm{T}}\in SE(2)$ is the pose in task space and $\theta$ represents the end-effector orientation [16]. We use the notation $J(q)=\frac{\partial x}{\partial q}\in\mathbb{R}^{2\times 3}$ to refer to the kinematic Jacobian.

We can then derive the dynamics of the planar HSA robot in Euler-Lagrangian form

$$\small M(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q)+K\,(q-q^{0})+D\,\dot{q}=\alpha(q,\phi),$$ (2)

where $M(q),C(q,\dot{q})\in\mathbb{R}^{3\times 3}$ captures the inertial and Coriolis effects, $G(q)\in\mathbb{R}^{3}$ contributes the gravitational forces and $K\in\mathbb{R}^{3\times 3}$ is the stiffness of the robot in its un-actuated state $q^{0}$. Furthermore, $D\in\mathbb{R}^{3\times 3}$ is a positive-definite dam** matrix. Finally, for the planar case, two HSA rods are assumed to be actuated by the motor/twist angle $\phi\in\mathbb{R}^{2}$. As the handedness of the rods will be accounted for later, we state the actuation bounds as $0\leq\phi_{i}\leq\phi_{\mathrm{max}}\>\forall i\in\{1,2\}$. The auxetic trajectory [18] causes the motors to act through the elasticity of the rods on the system and modify the axial rest length of the rod as a function of the twist strain [15]. Furthermore, the stiffness of the rod can be modeled to be an affine function with respect to the twist strain [18, 15]. Both effects are captured in the actuation function $\alpha(q,\phi)$, which is nonlinear with respect to the actuation coordinate $\phi$ and affine in the configuration $q$. Although this has never been done in the context of HSA robots, it is immediate to see that their dynamics (2) can be projected into operational space yielding the form [19, 20]

$$\small\Lambda(q)\,\ddot{x}+\mu(q,\dot{q})\dot{q}+J_{\mathrm{B}}^{+\mathrm{T}}(G(q)+K(q-q^{0})+D\,\dot{q})=J_{\mathrm{B}}^{+\mathrm{T}}\alpha(q,\phi),$$ (3)

where $J_{\mathrm{B}}^{+}(q)=B^{-1}J^{\mathrm{T}}(JB^{-1}J^{\mathrm{T}})^{-1}\in\mathbb{R}^{3\times 2}$ is the dynamically consistent pseudo-inverse, $\Lambda(q)=(J\,B^{-1}J^{\mathrm{T}})^{-1}\in\mathbb{R}^{2\times 2}$ is the inertia matrix in task space, and $\mu(q,\dot{q})=\Lambda(q)\,(JB^{-1}C-\dot{J})\in\mathbb{R}^{2\times 3}$ collects the Cartesian Coriolis and centrifugal terms.

II-D Cartesian impedance controller

In previous work [16], we have devised a model-based control strategy for regulating a planar HSA robot towards a desired position in task space. We introduce below a novel control strategy that addresses some limitations of our previous work that are critical for the BMI application. Namely, we (i) avoid computationally demanding planning procedures, (ii) remove integral terms that are unsafe for environment interaction, and (iii) enable impedance sha** in operational space. This Cartesian-space impedance controller is inspired by  [21, 20], but specifically designed for and tailored to HSA robots. Crucially, we need to overcome the challenges of underactuation and the nonlinearity in the actuation - which were not present in that original work.

II-D1 Proposed controller

We propose the following dynamic feedback law that renders $x^{\mathrm{at}}$ an attractor of the closed-loop system

$$\small\begin{split}\tau=&J^{\mathrm{T}}(q)\,\left(K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}\right)+G(q)+K\,(q-q^{0})\\ &+J^{\mathrm{T}}(q)\,J_{\mathrm{B}}^{+\mathrm{T}}(q)\,D\,\dot{q}+J^{\mathrm{T}}(q)\,\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}\end{split}$$ (4)

where $\tau\in\mathbb{R}^{3}$ is the desired torque in configuration space, $G(q)+K\,(q-q^{0})$ cancels the acting gravitational and elastic forces, and $J^{\mathrm{T}}J_{\mathrm{B}}^{+\mathrm{T}}D\,\dot{q}$ removes the natural dissipation in operational space. We emphasize that because the system is underactuated, we need to cancel the stiffness directly in the configuration instead of operational space as done in previous work [20]. We can shape our desired impedance characteristics in Cartesian space with the PD term $f_{\mathrm{PD}}=K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}$ which is then projected into configuration space by premultiplying with $J^{\mathrm{T}}(q)$.

The term $\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}$ decouples the operational space dynamics from the residual of the null-space dynamics [20][21, Ch. 4]. The identity $\dot{q}=J_{\mathrm{B}}^{+}\,\dot{x}+Z^{\mathrm{T}}\,\nu_{\mathrm{N}}$, where $Z^{\mathrm{T}}\in\mathbb{R}^{3\times 1}$ is the dynamically-consistent pseudo-inverse of the null space, allows us to formulate $\dot{q}$ as a sum of the task-space velocity $\dot{x}$ and the null-space velocity $\nu_{\mathrm{N}}$. Leveraging this identity, the Coriolis and centrifugal matrix $\mu(q,\dot{q})$ can be split into a term $\mu_{x}(q,\dot{q})=\mu\,J_{\mathrm{B}}^{+}\in\mathbb{R}^{2\times 2}$ excited by $x$ and the expression $\mu_{\mathrm{N}}(q,\dot{q})=\mu\,Z^{\mathrm{T}}\in\mathbb{R}^{2\times 1}$ that is excited by the null-space coordinates resulting in $\mu(q,\dot{q})\,\dot{q}=\mu_{x}(q,\dot{q})\,\dot{x}+\mu_{\mathrm{N}}(q,\dot{q})\,\nu_{\mathrm{N}}$. This allows us to cancel the term $\mu_{\mathrm{N}}(q,\dot{q})\,\nu_{\mathrm{N}}$ through $\mu(q,\dot{q})\left(I_{3}-J_{\mathrm{B}}^{+}(q)J(q)\right)\dot{q}$ without having to compute the null space explicitly.

In summary, the closed-loop dynamics in operational space can be stated as

$$\small\Lambda(q)\,\ddot{x}+\mu(q,\dot{q})\,J_{\mathrm{B}}^{+}\,\dot{x}+K_{x}\,(x^{\mathrm{at}}-x)-D_{x}\,\dot{x}=0,$$ (5)

which results in $x^{\mathrm{at}}$ being the globally asymptotically stable equilibrium of the operational space dynamics.

(a) Operational workspace

(b) Experimental setup

Figure 3: Panel (a: Operational workspace of a HSA robot with attached end-effector: the color displays the mean steady-state actuation $\frac{\phi_{1}^{\mathrm{ss}}+\phi_{2}^{\mathrm{ss}}}{2}$ necessary for the end-effector to remain at the position. Additionally, we visualize three example shapes: the straight configuration with $\phi^{\mathrm{ss}}=(0,0)$ (blue), maximum clockwise bending with $\phi^{\mathrm{ss}}=(3.49,0)\,$\mathrm{r}\mathrm{a}\mathrm{d}$$ (red), and maximum counter-clockwise bending with $\phi^{\mathrm{ss}}=(0,3.49)\,$\mathrm{r}\mathrm{a}\mathrm{d}$$ (green). Panel (b): The HSA robot is mounted platform-down to a motion capture cage with 8x Optitrack PrimeX 13 cameras, which track the 3D pose of the platform (i.e., the end-effector). A Dynamixel MX-28 servo actuates each of the four HSAs. We project a rendering of the current (white dot) and desired (red dot) end-effector position, the attractor (green square), and the operational workspace (grey area) onto the black screen in the background. The study subject wears a cap with the Neuroconcise FlexEEG sensor, and we acquire the data of three electrodes connected to the motor cortex.

Figure 4: ERD/S (overall average) over a time period of $2.5\text{\,}\mathrm{s}$ of training data for right-hand Imagination v/s rest state including the Alpha and Beta bands of the EEG signals where the cue is presented at $0\text{\,}\mathrm{s}$. We plot the data of three sensors (i.e., channels): FC3-CP3 (left), FCZ-CPZ (middle) and FC4-CP4 (right).

(a) End-effector x-coordinate

(b) End-effector y-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 5: Experimental results for tracking a reference trajectory of nine step functions with motor imagery. Panel (a) & (b): The x/y-coordinate of the end-effector position with the solid line denoting the actual position, the dotted line the attractor position, and the dashed line the reference (i.e., the setpoint). Panel (c): The evolution of the configuration. Panel(d): The saturated planar control inputs.

II-D2 Map** to Lagrangian forces

Now that we have formulated our control law $\tau$ in configuration space, we need to identify a strategy to specify the motor angles $\phi\in\mathbb{R}^{2}$ such that $\alpha(q,\phi)\approx\tau$. Note that, in contrast to other continuum soft robots studied in literature [22], the actuation term $\alpha(q,\phi)$ is not affine in control. In previous work [16], we side-stepped this challenge by linearizing with respect to the steady-state actuation $\phi^{\mathrm{ss}}$: $A(q)=\lVert\frac{\partial\alpha}{\partial\phi}\rVert_{\phi=\phi^{\mathrm{ss}}}$ therefore recovering the usual scenario of an affine actuation function. Unfortunately, this is not possible in the setting of this work as i) we do not have access to such $\phi^{\mathrm{ss}}$, and ii) linearizing around $\phi$ causes the closed-loop system to become unstable. We, therefore, propose to formulate instead a nonlinear least-squares problem $\phi^{\mathrm{d}}=\operatorname*{arg\!\min}_{\phi}\frac{1}{2}\lVert\tau-\alpha(q,\phi)\rVert^{2}$ and solve it in real-time with a Levenberg Marquardt solver implemented in JAX [23].

We note that this approach is not guaranteed to be valid for the general case of an underactuated soft robot but for this particular structure of $\alpha(q,\phi)\in\mathbb{R}^{3}$ with $\phi\in\mathbb{R}^{2}$ it is possible to identify solutions $\phi$ with the Euclidean norm of the residual being smaller than $0.001$. The source code of the controller is available on GitHub.

III Experiments

III-A Experimental setup

In the following, we detail the EEG data processing procedure (see also Fig. 2) and present our experimental setup, which is annotated in Fig. 3(b).

III-A1 EEG data processing

We integrate the 3-channel flexEEG Neuroconcise device with the OpenVibe software to acquire the EEG data and process it in real-time. This configuration facilitates data recording and cue presentation. We process the EEG signals at a sampling frequency of $125\text{\,}\mathrm{H}\mathrm{z}$ with a pipeline that involves three bi-polar channels around the motor cortex: FC3-CP3, FCZ-CPZ, and FC4-CP4. After a notch filter of $50\text{\,}\mathrm{H}\mathrm{z}$, we apply Independent Component Analysis (ICA) to extract three independent components from the recorded EEG data, which is represented by the equation $S(t)=W\,X(t)$, where $S(t)$ are the extracted independent components and $W$ represents the unmixing matrix, allowing us to separate eye blink artifacts in EEG signals, which is critical for enhancing the accuracy. Subsequently, we apply a Butterworth filter bank to isolate specific frequency bands of interest, including 8-15 $\mathrm{H}\mathrm{z}$ and 15-42 $\mathrm{H}\mathrm{z}$. This enhances the ability to analyze EEG data by isolating and examining different frequency band components within the EEG signals. Once the signals are filtered in sub-bands and epoched with a duration of $2\text{\,}\mathrm{s}$ and time interval of $0.065\text{\,}\mathrm{s}$, the features are extracted by the log of the power: $L_{i}(t)=\log\left(P_{i}(t)\right)$, where the power $P_{i}(t)=|E_{i}(t)|^{2}$ is represented by square of magnitude of the EEG signal $E_{i}(t)$ at time instance $t$. These features are then provided to a classifier, which we select as LDA due to its simplicity [24].

We implement a second classifier with the same pipeline, where jaw clenching is provided as a muscle artifact that is classified v/s raw EEG data.

III-A2 Robotic system

We consider a robot consisting of four HSA rods, which were 3D printed via digital projection lithography from the photopolymer resin Carbon FPU 50. Each HSA rod is electrically actuated by a Dynamixel M-28 servo up to a maximum twist angle of $\phi_{\mathrm{max}}=$3.49\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$. The robot is attached platform-down to a motion capture cage with eight Optitrack Prime X13 cameras tracking at $200\text{\,}\mathrm{H}\mathrm{z}$ the pose of reflective markers attached to the end-effector of the HSA robot. We estimate the current Cartesian-space velocity of the end-effector with a Savitzky-Golay filter. Subsequently, we leverage a closed-form expression of the inverse kinematics of a CS model [16] to compute the current configuration $q$ of the robot. We render an image of the current shape of the robot together with the present end-effector position (white dot), the attractor planned by the user (green square), the operational workspace (grey, see also Fig. 3(a)) and if applicable, the goal position (red dot). We specify the currently active axis of movement $e_{\mathrm{a}}$ with a double arrow and project the resulting image onto a black screen in the background of the motion capture cage. The robot is operated with a ROS2 software framework. We receive the predicted and classified brain signals via TCP at a frequency of $18\text{\,}\mathrm{H}\mathrm{z}$ and move the attractor subsequently with $\Delta_{\mathrm{x}}=$0.2\text{\,}\mathrm{m}\mathrm{m}$$. We evaluate the Cartesian impedance controller using the gains $K_{\mathrm{p}}=$300\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$, $K_{\mathrm{d}}=$1.5\text{\,}\mathrm{N}\mathrm{s}\mathrm{/}\mathrm{m}$$ at a frequency of $50\text{\,}\mathrm{H}\mathrm{z}$ and finally send the desired motor positions to the servos.

The entire control pipeline from measuring EEG signals to sending the actuation commands to the motors exhibits a maximum latency of (i.e., is upper-bounded by) $52\text{\,}\mathrm{m}\mathrm{s}$.

III-B BMI protocol:

In the following, we will describe the protocol for collecting the motor imagery dataset, training the EEG classifiers, and map** classifier outputs into actionable robot commands.

III-B1 Training protocol

The participant is given brief instructions about motor imagery signals. We follow the Graz-BCI [25] paradigm, which assists with training, where the display of the cue instructs the participant to perform imagination of movements: when a left arrow appears, the participant is asked to rest and not focus on motor movement. When the right arrow appears on the screen, the participant is asked to imagine the motor activity from the dominant hand (here, the right hand), such as gras** or squeezing an object. The training protocol for right-hand motor imagery v/s rest demands fifty cues per class. We similarly collect data for the second classifier by asking participants to clench their jaw, which can be detected as muscular artifacts (vs. no muscular movement) in the EEG signals. At the end of the trial, we train both classifiers (see Sections II-A and III-A1 for more information) and repeat the procedure until an accuracy of $75\text{\,}\mathrm{\char 37}$ is obtained for right-hand motor imagery v/s rest and accuracy of $85\text{\,}\mathrm{\char 37}$ for jaw clench artifact v/s raw EEG.

(a) End-effector x-coordinate

(b) End-effector y-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 6: Experimental results for tracking a reference trajectory of nine step functions directly with the Cartesian impedance controller with access to the privileged information $x^{\mathrm{d}}$. Panel (a) & (b): The x/y-coordinate of the end-effector position with the solid line denoting the actual position, the dotted line the attractor position, and the dashed line the reference (i.e., the setpoint). Panel (c): The evolution of the configuration. Panel(d): The saturated planar control inputs.

III-B2 Online robot control

Now, the participant operates the HSA robot in real time, with both classifiers being executed online. Moreover, we map the outputs of the classifier into actionable commands: we initialize the x-axis as the active axis of movement (i.e., $e_{\mathrm{a}}=[1,0]$). When the first classifier detects jaw clenching for at least $80\text{\,}\mathrm{\char 37}$ of samples over a duration of $2.8\text{\,}\mathrm{s}$, we switch to the y-axis: $e_{\mathrm{a}}=[0,1]$ and vice-versa. If we do not detect any yaw clenching artifacts, we evaluate the output of the second classifier in parallel: if there is motor imagery activity identified in the EEG classifier, the attractor will be moved in the positive direction (i.e., $s=1$) along $e_{\mathrm{a}}$. In contrast, if the EEG signals are classified as the participant being at rest, $s=-1$ (i.e., movement in the negative direction).

III-C Setpoint regulation

We randomly generate nine setpoints $x^{\mathrm{d}}(t)\in\mathbb{R}^{2}$ within the operational workspace of the robot (see Fig. 3(a) and display them as a red circle to the user over a duration of $540\text{\,}\mathrm{s}$. The user can freely move the attractor to reach and keep the end-effector at the setpoint. Furthermore, we execute an experiment in which the computational controller has access to normally privileged information: as we substitute $x^{\mathrm{at}}$ with $x^{\mathrm{d}}$ in (4), the Cartesian impedance controller now immediately regulates the robot towards the setpoint. By excluding the BMI from the pipeline, this provides us with a reference of the performance we can expect from the computational controller and, with that, also represents a performance upper bound.

(a) $t=$0\text{\,}\mathrm{s}$$

(b) $t=$12\text{\,}\mathrm{s}$$

(c) $t=$24\text{\,}\mathrm{s}$$

(d) $t=$36\text{\,}\mathrm{s}$$

(e) $t=$48\text{\,}\mathrm{s}$$

(f) $t=$60\text{\,}\mathrm{s}$$

(g) $t=$72\text{\,}\mathrm{s}$$

(h) $t=$84\text{\,}\mathrm{s}$$

(i) $t=$96\text{\,}\mathrm{s}$$

(j) $t=$108\text{\,}\mathrm{s}$$

Figure 7: Sequence of stills for completing a basic Activity of Daily Living (ADL) by controlling the robot with EEG-based motor imagery. Note: Fig. 7(i) is edited for improved contrast.

III-D Interacting with the environment on a real-world task

We consider the ADL task of releasing hairspray by actuating the button of its container with the HSA robot’s end-effector. For successful execution, the end-effector must be very stiff in the normal direction of the contact. On the other hand, we might want to benefit from the physical intelligence of the system by being relatively flexible in the tangential direction. Therefore, we first define the perpendicular stiffness $k_{\perp}=$500\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$ and the tangential stiffness as $k_{\parallel}=$50\text{\,}\mathrm{N}\mathrm{/}\mathrm{m}$$. We assume that the normal direction of the contact can be described by the polar angle $\theta_{\perp}$ (with respect to the x-axis). We envision that in the future, the user can adjust such stiffness characteristics online via a BMI system similar to [6]. In this work, however, we estimate by visual inspection that $\theta_{\perp}=$1.31\text{\,}\mathrm{r}\mathrm{a}\mathrm{d}$$. The Cartesian stiffness matrix in global coordinates is then given by $K_{x}=R(\theta_{\perp})\,\mathrm{diag}(k_{\perp},k_{\parallel})\,R(\theta_{\perp})^{\mathrm{T}}$ where $R(\theta_{\perp})\in SO(2)$ is the rotation matrix between global and contact frames.

(a) End-effector $x$-coordinate

(b) End-effector $y$-coordinate

(c) Configuration $q$

(d) Control input $\phi$

Figure 8: Experimental results for completing a basic Activity of Daily Living (ADL) by controlling the robot with EEG-based motor imagery. Panel (a) & (b): The x/y-coordinate of the end-effector with the solid line and dotted lines denoting the actual and attractor position, respectively. Panel (c): The evolution of the configuration. Panel(d): The saturated planar control inputs.

III-E Evaluation metrics

In the following, we will introduce and define a few metrics that help us assess the performance of the approach.

III-E1 Event-Related Synchronization/De-Synchronization:

We apply ERD / ERS [26] to demonstrate the difference between EEG signals when the participant imagines right-hand movement vs. rest, i.e., no activity. ERD/ERS corresponds to a shift in power during imagination with respect to a baseline. It is defined by

$$\mathrm{ERD/ERS}(t,f)=\frac{P(t,f)-P_{\mathrm{base}}(f)}{P_{\mathrm{base}}(f)},$$ (6)

where $\mathrm{ERD/ERS}(t,f)$ represents the ERD or ERS at a specific time $t$ and frequency $f$, $P(t,f)$ stands for the power of brain activity during imagination, and $P_{\mathrm{base}}(f)$ denotes the baseline power.

III-F Step response metrics

For the task of setpoint regulation, we analyze primarily two aspects: (a) is the participant able to reach the proximity of setpoint within the (generously) allotted time of $60\text{\,}\mathrm{s}$? We define the proximity of the setpoints as $\lVert x^{\mathrm{d}}-x(t)\rVert_{2}\leq$2\text{\,}\mathrm{m}\mathrm{m}$$. And (b) what is the response time for reaching for the first time the proximity of the setpoint?

IV Results and Discussion

First, we analyze the ERD/ERS behavior with respect to rest vs. motor imaginations in Fig. 4. It is evident that the baseline of rest remained the same in both scenarios when the participant did not perform motor imagery, but as soon as the cue is presented at $0.0\text{\,}\mathrm{s}$, a shift in power for the right-hand motor imagery with comparison to rest state is noticeable.

We present the results for setpoint regulation employing motor imagery in Fig. 5. We observe that the participant can reach the proximity of the setpoint within the allotted time of $60\text{\,}\mathrm{s}$ six out of nine times (i.e., $66.6\text{\,}\mathrm{\char 37}$). For the successful steps, the average response time is $21.5\text{\,}\mathrm{s}$. However, as our protocol does not contain a command to let the attractor rest, it is challenging to keep the end-effector at the setpoint and we observe oscillations, particularly with respect to the x-coordinate. In our third experiment, we ask the Cartesian impedance controller to track the setpoints directly. The fast response time, a well-known characteristic of model-based control approaches, is evident. However, the errors in the model (for example, caused by hysteresis or unmodelled nonlinearities) [16], together with the lack of integral action, lead to steady-state errors.

Finally, we consider the ADL task of releasing hair spray using the end-effector of the HSA robot. We present a sequence of stills in Fig. 7 and plots of the entire sequence in Fig. 8. Already during the first attempt, the participant can steer the end-effector toward the button, apply force, and release the fluid within $86\text{\,}\mathrm{s}$. The impedance of the controller is clearly visible in Fig. 8(b) when the manipulator is in contact with the object at time $74\text{\,}\mathrm{s}$ to $104\text{\,}\mathrm{s}$. Also, we noticed that the end-effector does not need to be perfectly aligned with the center of the button and can still complete the task successfully due to the compliance of the closed-loop system in the tangential direction.

We noticed that the variability of setting up the EEG device on each study participant and the EEG sensor noise caused by external factors (e.g., floor vibrations) still pose a considerable challenge for deploying motor imagery-based tools in practice. Furthermore, subject-specific factors such as the ability to focus on imagining motor actions, mental tiredness, etc., significantly affected the performance (e.g., classification accuracy, setpoint tracking error).

V Conclusion

In this paper, we proposed to combine motor imagery-based BMI systems with continuum soft robots. This symbiosis promises the safe and compliant operation of robots that can assist people with limb impairments in their daily lives. As demonstrated in the ADL experiment, the physical intelligence of the soft robot can compensate for errors and deviations in the output of the BMI classifier. Furthermore, we introduced a Cartesian impedance controller for planar HSA robots that can deal with the peculiar characteristics of these robots (e.g., underactuation, non-affinity in control, etc.), and allows for model-based control without interfering with the structural compliance of the system.