Haptic-Based Bilateral Teleoperation of Aerial Manipulator for Extracting Wedged Object with Compensation of Human Reaction Time

Abstract

Bilateral teleoperation of an aerial manipulator facilitates the execution of industrial missions thanks to the combination of the aerial platform’s maneuverability and the ability to conduct complex tasks with human supervision. Heretofore, research on such operations has focused on flying without any physical interaction or exerting a pushing force on a contact surface that does not involve abrupt changes in the interaction force. In this paper, we propose a human reaction time compensating haptic-based bilateral teleoperation strategy for an aerial manipulator extracting a wedged object from a static structure (i.e., plug-pulling), which incurs an abrupt decrease in the interaction force and causes additional difficulty for an aerial platform. A haptic device composed of a 4-degree-of-freedom robotic arm and a gripper is made for the teleoperation of aerial wedged object-extracting tasks, and a haptic-based teleoperation method to execute the aerial manipulator by the haptic device is introduced. We detect the extraction of the object by the estimation of the external force exerted on the aerial manipulator and generate reference trajectories for both the aerial manipulator and the haptic device after the extraction. As an example of the extraction of a wedged object, we conduct comparative plug-pulling experiments with a quadrotor-based aerial manipulator. The results validate that the proposed bilateral teleoperation method reduces the overshoot in the aerial manipulator’s position and ensures fast recovery to its initial position after extracting the wedged object.

I INTRODUCTION

Unmanned aerial manipulators (UAMs) have gained increasing attention owing to their maneuverability and versatility. Thanks to the maneuverability, they can be utilized in places that are dangerous for human operators or inappropriate for ground-based robots including walls or windows located in tall buildings, wind turbines, decommissioned reactors, or power transmission towers. Also, thanks to their versatility, there exist several studies on a UAM conducting physically interacting tasks such as drawer opening [1], peg-in-hole operation [2], non-destructive inspection [3], door opening [4] and window-cleaning [5].

Even though there exist works on fully autonomous control of UAMs [6, 7, 8], these controllers might face safety issues while operating in a priorly unknown environment or require complicated algorithms with high computational power to perform complex tasks. To resolve these problems, the “human-in-the-loop” control of a UAM is adopted to utilize humans’ decision-making ability while conducting complex tasks. Also, sensory information gathered by the UAM can be transmitted to the human operator to aid the decision-making process. Hence, research on the control of a UAM using the mutual exchange of information between the human operator and the aerial vehicle, or bilateral teleoperation [9], is meaningful.

Figure 1: Illustration of the haptic-based bilateral teleoperation of a UAM (remote environment) extracting a wedged object from a static structure with a haptic device (local environment).

However, there are a few situations in which a human operator cannot quickly pilot the UAM due to the limitation of the humans’ reaction time. Those situations include the tasks involving abrupt decrease in the interaction force such as extracting a wedged object from a static structure [10]. When those situations occur, the instantaneous fully autonomous control of the UAM and the haptic device after the abrupt force change can prevent an excessive overshoot in the UAM’s position or system destabilization induced by the operator’s long reaction time.

This paper proposes a haptic-based bilateral teleoperation of an aerial manipulator for extracting a wedged object from a static structure with the compensation of human reaction time, as shown in Fig. 1. To this end, we fabricate a haptic device configured with a 4-degree-of-freedom (DOF) and a gripper for gras** and pulling a wedged object using a UAM. Then, we divide the aerial object-extracting task into two flight phases including nominal and recovery phases where the recovery flight phase refers to a short time interval after object extraction while the nominal flight phase means the remaining interval. Also, the reference trajectory generation methods for the haptic device and the UAM during two flight phases are designed and the detection algorithm for switching from the nominal to the recovery flight phase is developed. As an example of the wedged object-extracting tasks, we conduct two plug-pulling experiments using a quadrotor-based aerial manipulator with and without the proposed bilateral teleoperation method, and we show that the proposed method enhances the behavior of the UAM after the object extraction in the perspective of reducing the overshoot in the UAM’s position and the time duration for the UAM to return to its original position.

I-A Related Works

Several studies on the bilateral teleoperation of aerial vehicles for missions with no physical interaction have been conducted. [11] and [12] present the teleoperation method using the haptic feedback reflecting obstacles nearby. In [13], the human operator indirectly guides a micro aerial vehicle by adjusting parameters related to its path generation with haptic feedback including information about tracking performance or obstacle presence. In [14], the authors introduce the teleoperation approach which transmits a wall-contact feeling to a human operator when the base approaches an unreachable position.

There also exist a few works on the bilateral teleoperation of UAMs involving contact with the surrounding environment. In [15], a method for the teleoperation of a UAM is introduced to enable a smooth transition between the free-flight and force-control phases and the regulation of the contact force exerted on the UAM’s end-effector. Hereby, the haptic feedback to a human operator includes the contact force that UAM receives. In [16], the human operator leverages haptic feedback and 3D visual feedback provided by virtual reality for the teleoperation of a UAM conducting several aerial manipulation tasks including gras**, pick-and-place, force exertion, and peg-in-hole insertion. [17] introduces a haptic device for the teleoperation of aerial drilling, and in [18], a method for the teleoperation of a 6-DOF fully actuated UAM that physically interacts with the surrounding environment is designed and tested. However, those works focus on the interaction tasks only involving the pushing motion where the effect of the interaction force change can be mitigated by slowing down before the physical contact.

I-B Contributions

To the best of the authors’ knowledge, it is the first attempt to conduct the haptic-based bilateral teleoperation of a UAM extracting a wedged object from a static structure. Also, we produce the haptic device configured with a 4-DOF robotic arm and a gripper for the emulation of the movement of the UAM grabbing and extracting an object wedged in a static structure. Moreover, an admittance controller that makes the haptic device compliant with the external torque exerted by a human’s hand is designed without a force/torque sensor. Furthermore, to avoid destabilization or an excessive overshoot in the position of the UAM after extracting the wedged object, we design an algorithm for the detection of the object extraction and the trajectory generation method for the UAM and the haptic device after the detection. Lastly, for the validation of the proposed method, we conduct an actual plug-pulling experiment with a quadrotor-based aerial manipulator with our haptic device.

I-C Outline

In Section II, we formulate two dynamic equations of the haptic device and the UAM, and describe the bilateral teleoperation method during the nominal flight phase in Section III. Section IV presents the conditions for detecting the extraction of a wedged object and the trajectory generation methods during the recovery flight. In Section VI, the proposed teleoperation strategy is validated experimentally.

Notations: $\boldsymbol{0_{i\times j}}$, $\boldsymbol{I_{i}}$ and $\boldsymbol{e_{3}}$ represent the $i\times j$ zero matrix, $i\times i$ identity matrix and $[0\ 0\ 1]^{\top}$, respectively. For scalars $a$, $b$ and $c$, we let $[a;b;c]$ denote $[a\ b\ c]^{\top}$. Also, $\|\boldsymbol{\alpha}\|$ and $\alpha_{i}$ mean the Euclidean norm and the $i$-th element of a vector $\boldsymbol{\alpha}$, respectively. Moreover, $c_{*}$ and $s_{*}$ are the abbreviations of $\cos{*}$ and $\sin{*}$, respectively.

II MODELING

For the bilateral teleoperation of a UAM extracting a wedged object, we manipulate the UAM consisting of a multirotor and a 2-DOF robotic arm with a gripper using the haptic device configured with a 4-DOF robotic arm and a 1-DOF gripper. In this section, the dynamic models of the haptic device and the UAM are derived.

II-A Haptic Device

Figure 2: Illustration of the robotic arm that is used for our haptic device. Key coordinate frames are also depicted.

In Fig. 2, $\boldsymbol{\mathcal{O}_{i}}$-$\boldsymbol{xyz}$ ($i=0,\cdots,4$) and $\boldsymbol{\mathcal{O}_{H,E}}$-$\boldsymbol{xyz}$ are the coordinate frames of the $i^{\textrm{th}}$ joint and the robotic arm’s end-effector, respectively. Also, $\boldsymbol{p_{H}}$ means the displacement from the initial position of $\boldsymbol{O_{H,E}}$ to its current position with respect to $\boldsymbol{\mathcal{O}_{0}}$-$\boldsymbol{xyz}$. The generalized coordinate of the robotic arm is $\boldsymbol{\theta_{H}}\triangleq[\theta_{H,1};\theta_{H,2};\theta_{H,3};\theta_{H,4}]$, and the joint angle of the gripper is defined as $\theta_{g}$.

Since a ball joint is attached between the robotic arm and the gripper, the human hand freely rotates. Then, we can derive dynamic models of the robotic arm and the gripper, respectively. For the derivation of the robotic arm’s dynamic model, we assume that the gripper is considered as a point mass attached on $\boldsymbol{O_{H,E}}$. Hence, $\boldsymbol{\ddot{\theta}_{H}}$ satisfies the following equation:

$$\boldsymbol{M_{H}}(\boldsymbol{\theta_{H}})\boldsymbol{\ddot{\theta}_{H}}+\boldsymbol{C_{H}}(\boldsymbol{\theta_{H}},\boldsymbol{\dot{\theta}_{H}})\boldsymbol{\dot{\theta}_{H}}+\boldsymbol{G_{H}}(\boldsymbol{\theta_{H}})\\ =\boldsymbol{\tau_{H}}+\boldsymbol{\tau_{H,E}}$$ (1)

where $\boldsymbol{M}(\boldsymbol{\theta_{H}})\in{\mathbb{R}}^{4\times 4}$, $\boldsymbol{C_{H}}(\boldsymbol{\theta_{H}},\boldsymbol{\dot{\theta}_{H}})\in{\mathbb{R}}^{4\times 4}$ and $\boldsymbol{G_{H}}(\boldsymbol{\theta_{H}})\in{\mathbb{R}}^{4}$ are the mass, Coriolis and gravity matrices of the robotic arm, respectively, and we let $\boldsymbol{\tau_{H}}\in{\mathbb{R}}^{4}$ and $\boldsymbol{\tau_{H,E}}\in{\mathbb{R}}^{4}$ denote the joint torque vector and the external torque applied by the human operator, respectively.

Meanwhile, the model of the gripper is:

$$J_{g}\ddot{\theta}_{g}=\tau_{g}$$ (2)

where $J_{g}$ and $\tau_{g}$ are the moment of inertia of the gripper’s servo motor with respect to the rotation axis and the corresponding joint torque, respectively.

II-B UAM

Figure 3: Illustration of the UAM with the key coordinate frames.

The key coordinate frames of the UAM are shown in Fig. 3. $\boldsymbol{\mathcal{O}_{W}}$-$\boldsymbol{xyz}$, $\boldsymbol{\mathcal{O}_{b}}$-$\boldsymbol{xyz}$ and $\boldsymbol{\mathcal{O}_{A,E}}$-$\boldsymbol{xyz}$ represent the coordinate frames of the world, multirotor, and the end-effector of the UAM, respectively. In this paper, the translational movement of the vehicle is described by the position of the UAM’s center of mass, $\boldsymbol{p_{c}}$, and its rotational movement is expressed with the Euler angles of the multirotor, $\boldsymbol{\phi}\in{\mathbb{R}}^{3}$. Also, the joint angles of the attached robotic arm are defined as $\boldsymbol{\theta_{a}}\in{\mathbb{R}}^{3}$.

According to [19], the dynamic model with respect to $\boldsymbol{p_{c}}$ is formulated as follows:

$$\begin{split}m_{t}\boldsymbol{\ddot{p}_{c}}=&-m_{t}g\boldsymbol{e_{3}}+\boldsymbol{u_{c}}+\boldsymbol{R^{\top}_{b}}\boldsymbol{f_{A,E}}\end{split}$$ (3)

where $m_{t}$ and $g$ are the total mass of the UAM and the gravitational acceleration, respectively, $\boldsymbol{R_{b}}\in SO(3)$ means the rotational matrix from $\boldsymbol{O_{b}}$-$\boldsymbol{xyz}$ to $\boldsymbol{O_{W}}$-$\boldsymbol{xyz}$, and

$$\begin{split}\boldsymbol{u_{c}}=&F\underset{\triangleq\boldsymbol{\Psi}}{\underline{\begin{bmatrix}c\phi_{3}&s\phi_{3}&0\\ s\phi_{3}&-c\phi_{3}&0\\ 0&0&1\end{bmatrix}}}\begin{bmatrix}c\phi_{1}s\phi_{2}\\ s\phi_{1}\\ c\phi_{1}c\phi_{2}\end{bmatrix}\end{split}$$

with $F$ representing the total thrust of the UAM. Also, $\boldsymbol{f_{A,E}}\in{\mathbb{R}}^{3}$ represents the force exerted on the end-effector expressed in $\boldsymbol{O_{b}}$-$\boldsymbol{xyz}$.

Meanwhile, with the assumption that the attitude of the multirotor and the joint angles of the robotic arm are well controlled by their built-in controllers (e.g., PX4, Dynamixel), the control inputs of the multirotor’s attitude and the joint angles become the desired angular velocity of the multirotor, $\boldsymbol{\omega_{b,d}}$, and the desired joint angles, $\boldsymbol{\theta_{A,d}}\in{\mathbb{R}}^{3}$, respectively. Hence, the dynamic equations with respect to $\boldsymbol{\phi}$ and $\boldsymbol{\theta}_{A}$ are expressed as follows:

$$\begin{split}\boldsymbol{\dot{\phi}}=&\boldsymbol{Q}^{-1}(\boldsymbol{\phi})\boldsymbol{\omega_{b,d}},\quad\boldsymbol{{\theta}_{A}}=\boldsymbol{{\theta}_{A,d}}\end{split}$$ (4)

where $\boldsymbol{Q}(\boldsymbol{\phi})\triangleq\begin{bmatrix}1&0&-s\phi_{2}\\ 0&c\phi_{1}&s\phi_{1}c\phi_{2}\\ 0&-s\phi_{1}&c\phi_{1}c\phi_{2}\end{bmatrix}$.

III BILATERAL TELEOPERATION DURING NOMINAL FLIGHT

Figure 4: Diagram including the teleoperation method during nominal flight (blue lines) and reference trajectory generation methods for the haptic device and the UAM during recovery flight (red lines).

The nominal flight phase means that there does not occur sudden change in $\boldsymbol{f_{A,E}}$, i.e., it does not involve the extraction of a wedged object from a static structure. In this section, the bilateral teleoperation strategy during the nominal flight phase based on [18] is presented. The overall flow map of this phase is described in Fig. 4 with the blue lines.

III-A UAM

III-A1 Reference Trajectory Generation

From the displacement of the haptic device’s tooltip, $\boldsymbol{p_{H}}$, the desired velocity of the UAM’s COM with respect to $\boldsymbol{O_{b}}$-$\boldsymbol{xyz}$, $\boldsymbol{\dot{p}^{b}_{c,d}}$, is designed as follows:

$$\begin{split}\boldsymbol{\dot{p}^{b}_{c,d}}=\begin{bmatrix}v_{\textrm{max},x}\tanh{(\tfrac{p_{H,1}}{p_{H,\textrm{max},x}})}\\ v_{\textrm{max},y}\tanh{(\tfrac{p_{H,2}}{p_{H,\textrm{max},y}})}\\ v_{\textrm{max},z}\tanh{(\tfrac{p_{H,3}}{p_{H,\textrm{max},z}})}\end{bmatrix}.\end{split}$$ (5)

where $v_{\textrm{max},x}$, $v_{\textrm{max},y}$ and $v_{\textrm{max},z}$ represent the UAM’s maximum speeds of $x$, $y$ and $z$-axes of the body frame, respectively, and we let $p_{H,\textrm{max},x}$, $p_{H,\textrm{max},y}$ and $p_{H,\textrm{max},z}$ denote the maximum absolute values of each element of $\boldsymbol{p_{H}}$. Then from the above equation, the desired velocity and position of $\boldsymbol{p_{c}}$ are obtained as follows:

$$\begin{split}\boldsymbol{\dot{p}_{c,d}}=&\boldsymbol{R_{z}}(\phi_{3})\boldsymbol{\dot{p}^{b}_{c,d}}\\ \boldsymbol{p_{c,d}}=&\boldsymbol{p_{c,0}}+\int^{t}_{t_{0}}\boldsymbol{\dot{p}_{c,d}}(\tau)d\tau\end{split}$$ (6)

where $\boldsymbol{p_{c,0}}$ is the initial value of $\boldsymbol{p_{c}}$.

While the desired yaw angle of the multirotor, $\phi_{d,3}$, is set from the keyboard of the ground control station by the human operator, the desired joint angles are set as follows:

$$\begin{split}\boldsymbol{\theta_{A,d}}=[-\phi_{1};-\phi_{2};\theta_{g}].\end{split}$$ (7)

This equation makes the attitude of the end-effector parallel to the ground and commands the UAM’s gripper to follow the signal from the haptic device.

III-A2 Control

For the regulation of $\boldsymbol{p_{c}}$, we adopt the disturbance-observer (DOB)-based controller introduced in [20] to generate the desired values of the total thrust $F$, the roll angle $\phi_{1}$ and the pitch angle $\phi_{2}$ from the error between $\boldsymbol{p_{c}}$ and $\boldsymbol{p_{c,d}}$. Then, based on $\boldsymbol{\phi_{d}}=[\phi_{d,1};\phi_{d,2};\phi_{d,3}]$, we calculate $\boldsymbol{\omega^{b}_{b,d}}$ for the control of the multirotor’s attitude. Lastly, we simply transmit $\boldsymbol{\theta_{A,d}}$ in (7) to the built-in controller of the servo motors.

The desired value of $\boldsymbol{u_{c}}$ in (3), $\boldsymbol{u_{c,d}}$, is calculated as follows:

$$\boldsymbol{u_{c,d}}=\hat{m}_{t}(\hat{g}\boldsymbol{e_{3}}\\ -\boldsymbol{K_{D}}(\boldsymbol{\dot{p}_{c}}-\boldsymbol{\dot{p}_{c,d}})-\boldsymbol{K_{P}}(\boldsymbol{p_{c}}-\boldsymbol{p_{c,d}}))-\boldsymbol{R_{b}}\boldsymbol{\hat{f}_{A,E}}$$ (8)

where $\hat{m}_{t}$ and $\hat{g}$ are the nominal values of $m_{t}$ and $g$, respectively, and $\boldsymbol{\hat{f}_{A,E}}$ is the estimated value of $\boldsymbol{f_{A,E}}$ which will be presented later. Then, $F$ and the desired roll and pitch angles, $\phi_{d,1}$ and $\phi_{d,2}$, are extracted from $\boldsymbol{u}_{c,d}$ as follows:

$$\begin{split}F=&\tfrac{1}{c\phi_{1}c\phi_{2}}\boldsymbol{e_{3}^{\top}}\boldsymbol{\Psi}\boldsymbol{u_{c,d}},\ \phi_{d,1}=\sin^{-1}{(\tfrac{1}{F}\boldsymbol{e_{2}^{\top}}\boldsymbol{\Psi}\boldsymbol{u_{c,d}})},\\ \phi_{d,2}=&\sin^{-1}{(\tfrac{1}{Fc\phi_{1}}\boldsymbol{e_{1}^{\top}}\boldsymbol{\Psi}\boldsymbol{u_{c,d}})}\end{split}$$ (9)

Meanwhile, $\boldsymbol{\hat{f}_{A,E}}$ is calculated based on $\boldsymbol{u_{c}}$ as follows:

$$\begin{split}\boldsymbol{\dot{\zeta}_{c}}=&-\tfrac{1}{\nu_{c}}\boldsymbol{\Gamma_{c,\zeta}}(\boldsymbol{\zeta_{c}}-\boldsymbol{\dot{p}_{c}}),\qquad\ \ \boldsymbol{\zeta_{c}}(t_{0})=\boldsymbol{p_{c,0}}\\ \boldsymbol{\dot{\chi}_{c}}=&-\tfrac{1}{\nu_{c}}\boldsymbol{\Gamma_{c,\chi}}(\boldsymbol{\chi_{c}}-\tfrac{1}{\hat{m}_{t}}\boldsymbol{u_{c}}),\quad\boldsymbol{\chi_{c}}(t_{0})=g\boldsymbol{e_{3}}\\ \boldsymbol{\hat{f}_{A,E}}=&-\tfrac{\hat{m}_{t}}{\nu_{c}}\boldsymbol{\Gamma_{c,\zeta}}(\boldsymbol{\zeta_{c}}-\boldsymbol{\dot{p}_{c}})+\hat{m}_{t}\hat{g}\boldsymbol{e_{3}}-\hat{m}_{t}\boldsymbol{\chi_{c}}\end{split}$$ (10)

where $\boldsymbol{\zeta_{c}}\in{\mathbb{R}}^{3}$ and $\boldsymbol{\chi_{c}}\in{\mathbb{R}}^{3}$ represents the estimation of $\boldsymbol{\dot{p}_{c}}$ and the filtered value of $\tfrac{1}{\hat{m}_{t}}\boldsymbol{u_{c}}$, respectively, and $\boldsymbol{\Gamma_{c,\zeta}}$, $\boldsymbol{\Gamma_{c,\chi}}\in{\mathbb{R}}^{3\times 3}$ and $\nu_{c}<1$ are the user-defined positive definite matrices and positive scalar value smaller than 1. Note that $\boldsymbol{\hat{f}_{A,E}}$ is calculated based on $\boldsymbol{u_{c}}$, not $\boldsymbol{u_{c,d}}$, to exclude the effects of the difference between $\boldsymbol{u_{c}}$ and $\boldsymbol{u_{c,d}}$.

Meanwhile, $\boldsymbol{\omega_{b,d}}$ is obtained as follows:

$$\begin{split}\boldsymbol{\omega_{b,d}}=\boldsymbol{K_{R}}(\boldsymbol{\phi_{d}}-\boldsymbol{\phi})\end{split}$$ (11)

where $\boldsymbol{K_{R}}\in{\mathbb{R}}^{3\times 3}$ is the user-defined positive definite matrix, and we directly input $\boldsymbol{\theta_{A,d}}$ generated in (7) into the built-in controller to control the joint angle control.

III-B Haptic Device: Robotic Arm

Let the positive definite matrices $\boldsymbol{M_{H,d}}\in{\mathbb{R}}^{4\times 4}$ and $\boldsymbol{D_{H,d}}\in{\mathbb{R}}^{4\times 4}$ denote the desired inertia and dam** matrices of the haptic device, respectively, then the admittance control law for $\boldsymbol{\theta_{H}}$ is designed as follows:

$$\boldsymbol{M_{H,d}}\boldsymbol{\ddot{\theta}_{H,d}}+\boldsymbol{D_{H,d}}\boldsymbol{\dot{\theta}_{H,d}}\\ =\boldsymbol{\hat{\tau}_{H,E}}+\boldsymbol{\tau_{fb}}+\boldsymbol{J_{H,E}}(\boldsymbol{\theta_{H}})\boldsymbol{\hat{f}_{A,E}}$$ (12)

where we let $\boldsymbol{\hat{\tau}_{H,E}}\in{\mathbb{R}}^{4}$ and $\boldsymbol{\tau_{fb}}\in{\mathbb{R}}^{4}$ denote the estimated value of $\boldsymbol{\tau_{H,E}}$ and recentering torque, respectively.

III-B1 Estimation of External Torque Applied to Haptic Device

The external torque from the human operator is estimated by the momentum-based estimator introduced in [21] as follows:

$$\boldsymbol{\hat{\tau}_{H,E}}=\boldsymbol{K_{I,H}}\Bigg{[}\boldsymbol{M}(\boldsymbol{\theta_{H}})\boldsymbol{\dot{\theta}_{H}}-\int^{t}_{t_{0}}(\boldsymbol{\tau_{H}}-\boldsymbol{C}(\boldsymbol{\theta_{H}},\boldsymbol{\dot{\theta}_{H}})\boldsymbol{\dot{\theta}_{H}}\\ -\boldsymbol{G}(\boldsymbol{\theta_{H}})+\boldsymbol{\hat{\tau}_{H,E}})d\tau-\boldsymbol{M}(\boldsymbol{\theta_{H,0}})\boldsymbol{\dot{\theta}_{H,0}}\Bigg{]},\ \boldsymbol{\hat{\tau}_{H,E,0}}=\boldsymbol{0_{4\times 1}}$$ (13)

where $\boldsymbol{K_{I,H}}\in{\mathbb{R}}^{4\times 4}$ is the user-defined positive definite matrix and we let $\boldsymbol{\theta_{H,0}}$ and $\boldsymbol{\hat{\tau}_{H,E,0}}$ denote the initial values of $\boldsymbol{\theta_{H}}$ and $\boldsymbol{\hat{\tau}_{H,E}}$, respectively.

III-B2 Recentering Torque

To drive the haptic device to its initial state when $\boldsymbol{\tau_{H,E}}$ is $\boldsymbol{0_{3}}$, the recentering torque, $\boldsymbol{\tau_{fb}}\in{\mathbb{R}}^{4}$, is designed as follows:

$$\begin{split}\boldsymbol{\tau_{fb}}=&-\boldsymbol{K_{fb}}(\boldsymbol{\theta_{H}}-\boldsymbol{\theta_{H,0}})\end{split}$$ (14)

where $\boldsymbol{K_{fb}}\in{\mathbb{R}}^{4\times 4}$ represents a user-defined positive definite matrix

III-C Haptic Device: Gripper

To make the gripper of the haptic device compliant with the gras** motion of the human hand, the desired angle of the gripper, $\theta_{g,d}$, is updated as follows:

$$\begin{split}\ddot{\theta}_{g,d}=-k_{d,g}\dot{\theta}_{g,d}-k_{fb,g}(\theta_{g}-\theta_{g}(t_{0}))+k_{\tau}\tau_{g}\end{split}$$ (15)

with the positive parameters $k_{d,g}$, $k_{fb,g}$ and $k_{\tau}$

IV RECOVERY FLIGHT

When the extraction of the wedged object is detected, an excessive overshoot in the magnitude of $\boldsymbol{p_{c}}$ might occur since a human operator cannot quickly react to the extraction of a wedged object. Hence, the recovery flight, which means the fully autonomous maneuver of the UAM returning to its state at the moment of extraction, is necessary to avoid the destabilization of the UAM or the large magnitude of position overshoot.

In this section, a method for detecting a wedged object extraction is introduced. Then, we present the methods of reference trajectory generation for the UAM and the haptic device during the recovery flight phase. During this phase, the UAM and the haptic device do not mutually exchange information and they are controlled fully autonomously. The detailed description is shown in Fig. 4 with the red lines.

Note that $T_{e}$ and $t_{e}$ represent the user-defined time duration of the recovery flight and the time instant of the wedged object extraction, respectively.

IV-A Detection of Wedged Object Extraction

By using the linear Kalman filter, we calculate $\boldsymbol{\dot{\hat{f}}_{A,E}}$. Then, we can determine whether the wedged object is extracted or not by examining the decrease speed of $\|\boldsymbol{\dot{\hat{f}}_{A,E}}\|$. From this concept, the switching from the nominal to recovery flight is detected by a condition shown below:

$$\begin{split}\hat{\dot{f}}_{A,E,thres}\leq\|\boldsymbol{\dot{\hat{f}}_{A,E}}\|\end{split}$$ (16)

where $\hat{\dot{f}}_{A,E,thres}$ represents the minimum threshold.

IV-B Reference Trajectory Generation for UAM

According to [22], the motor commands and the acceleration of the Euler angles are known to be proportional to the snap of the position trajectory. Therefore, we design a minimum-snap trajectory of $\boldsymbol{p_{c}}$ for the recovery flight as follows:

$$\begin{split}\boldsymbol{p_{c,d}}=\Sigma^{7}_{i=0}\boldsymbol{c_{i}}t^{i},\quad\boldsymbol{\dot{p}_{c,d}}=\Sigma^{7}_{i=1}i\boldsymbol{c_{i}}t^{i-1}\end{split}$$ (17)

where the coefficient $\boldsymbol{c_{i}}$ is calculated by solving the optimization problem below:

$$\begin{split}\underset{\boldsymbol{c_{0}},\cdots,\boldsymbol{c_{7}}}{\textrm{min}}&\int^{t_{e}+T_{e}}_{t_{e}}\|\boldsymbol{\ddddot{p}_{c,d}}\|^{2}d\tau\\ \textrm{s.t.}\ &\boldsymbol{p_{c,d}}(t_{e})=\boldsymbol{p_{e}},\quad\boldsymbol{p_{c,d}}(t_{e}+T_{e})=\boldsymbol{p_{e}}\\ &\boldsymbol{\dot{p}_{c,d}}(t_{e})=\boldsymbol{\dot{p}_{e}},\quad\boldsymbol{\dot{p}_{c,d}}(t_{e}+T_{e})=\boldsymbol{0_{3\times 1}}\\ &\boldsymbol{\ddot{p}_{c,d}}(t_{e}+T_{e})=\boldsymbol{\dddot{p}_{c,d}}(t_{e}+T_{e})=\boldsymbol{0_{3\times 1}}\end{split}$$ (18)

where $\boldsymbol{p_{e}}$ and $\boldsymbol{\dot{p}_{e}}$ represent the values of $\boldsymbol{p_{c}}$ and $\boldsymbol{\dot{p}_{c}}$ at $t=t_{e}$.

Meanwhile, $\phi_{d,3}$, $\theta_{A,d,1}$ and $\theta_{A,d,2}$ are the same as the value during the nominal flight.

For the control of the UAM, we use the same control laws introduced in (8) - (11).

IV-C Reference Trajectory Generation for Haptic Device

If there exists an error between $\boldsymbol{\theta_{H,d}}$ and $\boldsymbol{\theta_{H}}$ at the moment of re-switching from the recovery to nominal phases, the UAM will face a sudden movement. Hence, the robotic arm of the haptic device is commanded to quickly recover its initial pose during the recovery flight as follows:

$$\begin{split}\boldsymbol{\dot{\theta}_{H,d}}=-\boldsymbol{K_{recovery}}(\boldsymbol{\theta_{H}}-\boldsymbol{\theta_{H,0}})\end{split}$$ (19)

with the user defined positive definite matrix $\boldsymbol{K_{recovery}}\in{\mathbb{R}}^{4\times 4}$.

To hold onto the extracted object during the recovery flight phase, we do not change the value of $\theta_{g}$ from its value at $t=t_{e}$.

V RESULTS

As an example of the wedged object extracting using a UAM, we conduct plug-pulling experiments with a UAM configured with a quadrotor and a 3-DOF robotic arm including a gripper. This section compares the results of the aerial plug-pulling experiments with and without the recovery flight control strategy.

V-A Experimental Setups

Figure 5: (a) UAM utilized for our plug-pulling experiments. (b) Structure of the electric socket.

For the UAM, we use the platform configured with an underactuated quadrotor and a 3-DOF robotic arm with a gripper shown in Fig. 5a. The quadrotor is made up of the off-the-shelf frame DJI S500, four U3-KV700 motors with corresponding ALPHA-40A-LV electronic speed controllers (ESCs), four 11-inch T-Motor carbon fiber propellers, a 4S Turnigy Lipo battery with Intel NUC for computing, and a 6S Polytronics Lipo battery for the power supplement of the four motors and robotic arm. On Intel NUC, the algorithm for the reference trajectory generation of the position and joint angles, a position controller, and a navigation algorithm using the data obtained by OptiTrack are executed with Robot Operating System (ROS) installed in Ubuntu 18.04. Based on the total thrust and the desired Euler angles generated by the position controller, the motor commands are transmitted to the four ESCs by Pixhawk 4 which is connected to Intel NUC. The robotic arm is configured with an XM540 servo motor, two XM430 servo motors, and an FR12-G101GM gripper manufactured by ROBOTIS dynamixel and is controlled by U2D2. For the plug-pulling setting, a stand for a 110 V socket firmly attached to a black plastic plate is installed as in Fig. 5b.

V-B Parameter and Controller Gain Settings

V-B1 Haptic Device

To control the haptic device, we use the parameters and controller gains shown in Table I.

TABLE I: Parameters and Controller Gains for the Haptic Device.

$\boldsymbol{M_{H,d}}$	$\boldsymbol{D_{H,d}}$
diag$\{0.20,0.20,0.20,0.20\}$	diag$\{0.10,0.50,0.10,0.10\}$
$\boldsymbol{K_{I,H}}$	$\boldsymbol{K_{fb}}$
diag$\{30.0,30.0,30.0,30.0\}$	diag$\{3.00,1.00,7.50,7.50\}$
$\boldsymbol{K_{recovery}}$	$k_{d,g}$	$k_{fb,g}$	$k_{\tau}$
diag$\{6.00,2.00,15.0,15.0\}$	0.50	8.00	15.0

V-B2 UAM

For the reference trajectory generation and control of the UAM, we utilize the parameters and controller gains in Table II.

TABLE II: Parameters and controller gains utilized for the UAM

$v_{\textrm{max},x}$	$v_{\textrm{max},y}$	$v_{\textrm{max},z}$	$p_{H,\textrm{max},x}$	$p_{H,\textrm{max},y}$
0.4	0.4	0.4	0.2	0.2
$p_{H,\textrm{max},z}$	$\hat{f}_{A,E,\textrm{th}}$	$T_{e}$	$\hat{m}_{t}$	$\hat{g}$
0.2	7.50	5.0	2.50	9.81
$\boldsymbol{K_{P}}$		$\boldsymbol{K_{D}}$		$\nu_{c}$
diag$\{4.00,4.00,8.00\}$		diag$\{3.00,3.00,4.80\}$		0.90
$\boldsymbol{K_{R}}$		$\boldsymbol{\Gamma_{c,\zeta}}$ = $\boldsymbol{\Gamma_{c,\chi}}$
diag$\{12.0,12.0,10.0\}$		diag$\{1.00,1.00,1.00\}$

However, since the excessively fast estimation of $\boldsymbol{f_{A,E}}$ might induce the undesirable peak in $\boldsymbol{\hat{f}_{A,E}}$ so that the control performance is degraded, we reduce $\nu_{c}$ from 0.90 to 0.20 only for the estimation of $\boldsymbol{f_{A,E}}$.

V-C Scenario

First, we manipulate our haptic device to drive the UAM toward the 110V plug attached to the electric socket. Second, when the gripper of the UAM envelopes the plug-grabbing region, we close the gripper of the UAM by closing that of the haptic device by hand. Third, we pull the haptic device to make the UAM try to extract the plug from the socket.

To validate our proposed teleoperation strategy, we conduct two experiments: One with the baseline method, and another with the proposed method. The baseline method is simply utilizing the teleoperation strategy during the nominal flight for the entire steps of our scenario. Otherwise, the proposed method means that we utilize all of the methods introduced in Sections III - V.

V-D Results and Discussion

Figure 6: History of the UAM’s COM position ($\boldsymbol{p_{c}}$), estimation of the force exerted on the UAM ($\boldsymbol{\hat{f}_{A,E}}$), speed of $\boldsymbol{\hat{f}_{A,E}}$ ($\|\boldsymbol{\dot{\hat{f}}_{A,E}}\|$), and snapshots. The vertical black dashed lines represent the time instant of plug separation. In the plots of $\boldsymbol{p_{c}}$ and $\boldsymbol{\hat{f}_{A,E}}$, the black, red, and blue lines represent the first, second and third elements, respectively, and the solid and dashed lines mean the actual and desired trajectories, respectively. For the speed of $\boldsymbol{\hat{f}_{A,E}}$, the red line represents the calculated value while the black dashed horizontal lines mean $\dot{\hat{f}}_{A,E,thres}$. (a) Results with the baseline method and (b) results with the proposed method.

Fig. 6a presents the results with the baseline teleoperation method. The results of $\boldsymbol{\hat{f}_{A,E}}$ and $\|\boldsymbol{\dot{\hat{f}}_{A,E}}\|$ shows an abrupt change in the interaction force after the plug separation. Also, from the plot of $\boldsymbol{p_{c}}$, we can observe that there occur the continuous increases of $\boldsymbol{p_{c,d,1}}$, $\boldsymbol{p_{c,d,2}}$ and $\boldsymbol{p_{c,d,3}}$ along with the pulling direction even after the plug separation due to the limited reaction time of the human operator. It leads to a large difference between $\boldsymbol{p_{c}}$ and $\boldsymbol{p_{e}}$ and prevents the UAM from quickly returning to the plug-pulling location.

Fig. 6b reports the results with the proposed teleoperation method. As in Fig. 6a, the results of $\boldsymbol{\hat{f}_{A,E}}$ and $\|\boldsymbol{\dot{\hat{f}}_{A,E}}\|$ also validate that an abrupt change in the pulling force occurs after the plug separation. Compared to the history of $\boldsymbol{p_{c}}$ with the baseline method, we can notice that the proposed method helps the UAM recover the plug-pulling position within a short time. Also, even though there also exists an overshoot after the plug separation due to the time delay in estimating $\boldsymbol{\hat{f}_{A,E}}$, from Table III, we can note that the magnitude of overshoot in $\boldsymbol{p_{c}}$ is remarkably reduced compared to the previous result. From this quantitative result, with the same controller setting for the UAM, we confirm that the proposed teleoperation strategy helps reduce the overshoot in the UAM’s position after the plug is extracted from the socket.

TABLE III: Quantitative Results of Overshoot After Plug Separation. Best value is highlighted with bold.

	Baseline	Proposed
$\underset{t_{e}\leq t\leq t_{e}+T_{e}}{\textrm{max}}\|\boldsymbol{p_{c}}(t)-\boldsymbol{p_{e}}\|$ [m]	0.8934	0.6422

Figure 7: History of $\boldsymbol{p_{H}}$ of the haptic device. The blue, red, and yellow lines represent the trajectory of $p_{H,1}$, $p_{H,2}$ and $p_{H,3}$, respectively. The vertical black dashed line expresses the time instant of the plug separation. (a) Results with the baseline method and (b) results with the proposed method.

In Fig. 7a, it can be seen that $\boldsymbol{p_{H}}$ slowly returns to $\boldsymbol{0}_{3\times 1}$ due to the limitation of the human operator’s reaction time. This result leads to the excessive overshoot in $\boldsymbol{p_{c}}$. On the other hand, we can notice that $\boldsymbol{p_{H}}$ automatically returns to its original point in Fig. 7b. Since $\boldsymbol{p_{H}}$ becomes $\boldsymbol{0_{3\times 1}}$ after the plug separation, the human operator can manipulate the haptic device from its initial configuration when the nominal flight phase is revisited.

VI CONCLUSIONS

In this paper, we propose a haptic-based bilateral teleoperation strategy to compensate for human reaction time for a UAM extracting a wedged object from a static structure (i.e., plug-pulling), which incurs an abrupt decrease in the interaction force. To this end, the haptic device composed of a 4-DOF robotic arm and a gripper is manufactured for the teleoperation of aerial wedged object-extracting tasks, and their dynamic models are derived. Then, we introduce the haptic-based bilateral teleoperation strategy to execute the UAM by the haptic device during the nominal flight phase. Also, we construct the algorithm for detecting the extraction of the wedged object, and the reference trajectory generation method for the avoidance of destabilization or an excessive overshoot in the position of the UAM is designed. As an example of wedged object-extracting tasks, we conduct plug-pulling experiments with a quadrotor-based aerial manipulator. From the comparative experiments with the baseline and proposed teleoperation strategies, the results indicate that the proposed one ensures fast recovery to its initial position after the wedged object extraction and reduces the overshoot in the aerial manipulator’s position. For future works, we may adopt a hybrid control strategy or model predictive controller for more reduction of the overshoot in the position of the UAM. Also, an additional device for visual or tactile feedback may be attached to our UAM.