Revolutionizing Packaging: A Robotic Bagging Pipeline with Constraint-aware Structure-of-Interest Planning

Abstract

Bagging operations, common in packaging and assisted living applications, are challenging due to a bag’s complex deformable properties. To address this, we develop a robotic system for automated bagging tasks using an adaptive structure-of-interest (SOI) manipulation approach. Our method relies on real-time visual feedback to dynamically adjust manipulation without requiring prior knowledge of bag materials or dynamics. We present a robust pipeline featuring state estimation for SOIs using Gaussian Mixture Models (GMM), SOI generation via optimization-based bagging techniques, SOI motion planning with Constrained Bidirectional Rapidly-exploring Random Trees (CBiRRT), and dual-arm manipulation coordinated by Model Predictive Control (MPC). Experiments demonstrate the system’s ability to achieve precise, stable bagging of various objects using adaptive coordination of the manipulators. The proposed framework advances the capability of dual-arm robots to perform more sophisticated automation of common tasks involving interactions with deformable objects.

I Introduction

The field of deformable object manipulation (DOM) has garnered considerable attention for its potential to automate many advanced tasks in human environments. Everyday objects, from garments to soft furnishings, present highly deformable behaviors that complicate its automatic handling. Providing robots with the sufficient dexterity to manipulate this type of objects is crucial for their seamless integration into daily human environments. However, due to their infinite degrees-of-freedom and nonlinear dynamics, most research work has focused on simpler cases, such as 1-D and 2-D deformable bodies. The manipulation of complex 3D deformable structures such as common household bags (whose topology is modelled as a 2-torus), remains an underexplored problem in the robotics research community.

To address this gap in the literature, our work introduces a dual-arm robotic system empowered by constraint-aware structure-of-interest (SOI) planning, which advances DOM into the realm of 3D objects. This system is a significant leap towards sophisticated automation, capable of performing intricate tasks such as robotic bagging with precision and adaptability, marking a pivotal step in DOM research. In this paper, we introduce a novel approach to this problem through a dual-arm robotic system that leverages constraint-aware structure-of-interest (SOI) planning.

Figure 1: The dual-arm grasps two handles of a fabric bag to manipulate the SOI (i.e., the opening rim) for the bagging task.

Our approach is based on the insight that similar to the concept of Region of Interest (ROI) in the image processing domain, complete state estimation of a manipulated deformable object is not essential for robotic interaction. For specific deformable object manipulation tasks, it is sufficient to focus exclusively on state estimation of the critical structure-related components. Take, for instance, a robotic bagging task: the opening rim of the fabric bag can be considered the Structure of Interest (SOI). By concentrating on state estimation related to just the opening rim, the robotic system can successfully accomplish the bagging task. This targeted approach simplifies the state estimation process and improves the efficiency and effectiveness of the manipulation task. This system is specifically designed to address the automation of bagging tasks, a common yet challenging operation in both industrial and everyday contexts. The core of our approach is the use of two robotic arms that work in unison, guided by a sophisticated planning system that accounts for the constraints imposed by the object’s structure and desired final state. This is achieved using 3D-printed connectors, which allow the robots to manipulate the bag with an unprecedented level of precision and stability.

Our contribution is as follows:

• We propose a constraint-aware SOI planning framework that enables dual-arm robots to perform complex bagging tasks by manipulating a bag over an object to achieve a desired configuration.

• We integrate an adaptive vision-based control system that does not require prior knowledge of the bag’s material properties or system dynamics, making the setup more flexible and broadly applicable.

• We present a comprehensive methodological framework that encompasses SOI state estimation, bagging SOI generation, SOI planning, and motion planning, evidencing the system’s adaptability and sensitivity to environmental constraints.

II Related Work

The manipulation of deformable objects by robotic systems has been an area of increasing interest within the robotics community [1]. Early research efforts primarily addressed the manipulation of 1-D and 2-D deformable objects [2], using techniques such as tension-based strategies [3] and computational geometry [4] to model and control the behaviour of ropes, cloths, and sheets [5].

With the shift towards 3D deformable objects, researchers have explored various methods to handle increased complexity [6]. Notably, work by [7] delved into the dynamics of soft body manipulation using dual-arm robots, while [8] focused on non-prehensile manipulation techniques for cloth folding tasks [9]. Both approaches laid the groundwork for understanding the intricate interplay between robotic control and deformable object dynamics.

Recent advancements in vision-based control systems, such as those by [10], have shown how real-time feedback can enhance the adaptability of robots to the unpredictable nature of deformable objects [11, 12]. These works have informed the development of our constraint-aware SOI planning, which integrates real-time visual servoing to adjust the robot’s actions on the fly.

Our work builds on these foundational studies and takes a significant step forward by focusing on dual-arm manipulation for the specific task of robotic bagging—a complex application that has received limited attention thus far. We leverage the principles of constraint-aware planning to address the intricate problem of envelo** 3D objects with a deformable bag, which requires a high level of coordination and sensitivity to the dynamic constraints of the object and its environment.

Figure 2: The dual robot grasp two handles of a deformable fabric bag to manipulate the SOI (i.e., the opening rim) for the bagging task.

III Problem Statement

Notation. Subscript $(\cdot)_{t}$ is the discrete-time instant. $\mathbf{I}_{n\times m}$ is the $n\times m$ matrix of ones, and the identity matrix as $\mathbf{E}_{n}$. $\mathbf{L}_{n}$ is the low triangle matrix of $\mathbf{I}_{n\times n}$, and $\otimes$ is the Kronecker product. ${}^{\mathcal{F}_{x}}\mathbf{p}$ is a point $\mathbf{p}$ in the frame $\mathcal{F}_{x}$. In this work, unless otherwise specified, all points are expressed in the world frame $\mathcal{F}_{w}$ and omitted for clear.

In this work, we consider a novel robotic bagging task, as illustrated in Fig. 2. We propose a dual-arm robotic system where two robot manipulators grasp a deformable fabric bag to envelop a baggable object, denoted by $\mathbf{B}$, which is suspended in the air. The system’s objective is to manipulate the fabric bag from an initial state to a goal state, where the bag completely covers the object. We posit that it is unnecessary to estimate the entire fabric bag; instead, focusing on the Section of Interest (SOI) of the bag, specifically the bag opening rim, is sufficient for this task. Consequently, we define the SOI state of the bagging task as the opening rim of the bag, represented as $\mathbf{x}$, which can be captured by a depth camera configured in an eye-to-hand calibration style. In contrast to [13], which uses the entire point cloud as the SOI, we opt for a simpler representation by selecting contour keypoints to depict our SOI:

$$\mathbf{x}=\left[\mathbf{x}_{1}^{\intercal},\ldots,\mathbf{x}_{n_{x}}^{\intercal}\right]^{\intercal}\in\mathbb{R}^{3n_{x}},\quad\mathbf{x}_{i}=\left[x_{i},y_{i},z_{i}\right]^{\intercal}\in\mathbb{R}^{3}$$ (1)

where $n_{x}$ denotes the number of contour keypoints, and $\mathbf{x}_{i}$ represents the Cartesian coordinates of the $i$-th point in $\mathcal{F}_{w}$.

The task’s goal is to manipulate the SOI of the bag from its initial state $\mathbf{x}_{0}$ to the target state $\mathbf{x}^{*}$. We address this problem by planning with constraints: 1) The SOI keypoints should approximate the shape of an oval; 2) The perimeter formed by the SOI keypoints must be constant, indicating that the size of the fabric bag’s opening rim does not change during manipulation. Depending on whether the bag is in contact with the object $\mathbf{B}$, we divide the planning into pre-bagging and bagging stages:

$$\mathcal{G}:=\left[\mathcal{G}_{\text{pre-bagging}},\mathcal{G}_{\text{bagging}}\right]\Big{|}\underbrace{\mathbf{g}_{0},\mathbf{g}_{1},\ldots,\mathbf{g}^{{\dagger}}}_{\text{pre-bagging}},\underbrace{\mathbf{g}^{{\dagger}},\ldots,\mathbf{g}^{\ast}}_{\text{bagging}},$$ (2)

where $\mathbf{g}^{{\dagger}}$ is an SOI shape tailored to the baggable object $\mathbf{B}$ that can perfectly envelop its bottom. To reach each subgoal $\mathbf{g}_{i}$, we employ an MPC-based shape servoing approach to generate the velocity command $\mathbf{u}$ based on a measurable error function $\mathcal{E}_{\text{subgoal}}(\cdot)$ between the resulting SOI state $\mathbf{x}_{i}$ and the current subgoal $\mathbf{g}_{i}$:

$$\mathbf{u}_{i}=\underset{\mathbf{u}\in\mathcal{A}}{\arg\min}\leavevmode\nobreak\ \mathcal{E}_{\text{subgoal}}(\mathbf{x}_{i},\mathbf{g}_{i}),$$ (3)

The bagging task is thus accomplished through a sequence of actions $\{\mathbf{u}_{1},\mathbf{u}_{2},\ldots,\mathbf{u}^{*}\}$.

Figure 3: Schematic diagram of dual-arm manipulation approach of the bagging task. The algorithm aims to command the robot to manipulate the bag into the specified shape to cover the bottom part of $\mathbf{B}$, i.e., bagging manner. Experiments are conducted in the Cartesian space.

IV Methodology

In this section, a dual-arm manipulation approach of the bagging task is proposed, which is comprised of: 1) SOI State Estimation. Extracting meaningful representations of the Structure of Interest (SOI) from the raw dense and noisy point cloud. 2) Bagging SOI Generation. Generate a pre-enclosing shape $\mathbf{x}_{\ast}$ to cover the bottom part of $\mathbf{B}$. 3) SOI Planning. Generate a collision-free deformation path $\mathcal{G}$ from $\mathbf{x}_{0}$ to $\mathbf{x}_{\ast}$. 4) Motion Planning. Formulating the bagging process as shape servoing, drive the dual-arm move along $\mathcal{G}$ to complete the bagging task. Fig. 3 presents the block diagram of the proposed manipulation approach.

IV-A SOI State Estimation

In this work, the bag’s SOI state is defined as a sequence of contour keypoints, i.e., $\mathcal{Q}_{t}=\{\mathbf{x}_{t}^{i}\},i\in[1,n_{x}]$. The raw point cloud perceived by the depth camera is $\mathcal{P}_{t}=\{\mathbf{p}_{t}^{i}\},i\in[1,n_{p}]$, usually $n_{p}\gg n_{x}$. The state estimator aims to obtain a concise representation $\mathbf{x}_{t}^{i}$ by aligning $\mathcal{Q}_{t}$ to $\mathcal{P}_{t}$ in real-time.

We adopt the sampling approach in [14], i.e., Structure preserved registration (SPR), formulating the alignment process as a probability density estimation problem for Gaussian mixture model (GMM). By treating $\mathcal{P}_{t}$ as the points randomly sampled from GMM, thereby obtaining Gaussian’s centroids as $\mathcal{Q}_{t}$. Considering that $\mathcal{P}_{t}$ is dense, noisy and contains outliers, a uniform distribution for $\mathcal{Q}_{t}$ is added to GMM. The sampling probability of $\mathbf{p}^{m}_{t}$ is taken as below:

$$\kappa(\mathbf{p}^{m}_{t})=\Sigma_{n=1}^{n_{x}+1}\kappa(n)\kappa(\mathbf{p}^{m}_{t}|n)$$ (4)

where $\kappa(n)$ is the sampling weight of $n$-th mixture component, and $\kappa(\mathbf{p}^{m}_{t}|n)$ denotes the sampling probability of $\mathbf{p}^{m}_{t}$ from $n$-th mixture component. Both are given in [14].

The optimal estimation of $\mathcal{Q}_{t}$ can be obtained by maximizing the log-likelihood function $\mathcal{O}$ of the observation process:

$$\mathcal{O}(\mathbf{x}_{t}^{n})=\sum_{m=1}^{n_{p}}\sum_{n=1}^{n_{x}+1}\kappa(n|\mathbf{p}_{t}^{m})\log(\kappa(n)\kappa(\mathbf{p}_{t}^{m}|n))$$ (5)

The maximization of (5) can be processed through the EM algorithm [14], the optimal result $\mathbf{x}_{t}^{n,\ast}$ is regarded the concise SOI representation of the bag. Fig. 4 visualizes the GMM-based representation, where the black dots are $\mathcal{P}_{t}$, and the red dots connected with the blue line are the precise $\mathcal{Q}_{t}$.

Figure 4: The visual description of the GMM-based representation.

Figure 5: (a) The visualization of the bagging SOI. (b) The projection of $\mathcal{F}_{m}$ in $\mathcal{F}_{w}$, and visualizing three constraints. The red square is the centroid of ${}^{\mathcal{F}_{m}}\Omega^{v}$ and blue is that of ${}^{\mathcal{F}_{m}}\Omega^{e}$.

IV-B Bagging SOI Generation

This section introduces how to generate two shapes, i.e., a bagging SOI $\mathbf{x}_{{\dagger}}$ covering the bottom of $\mathbf{B}$, while another is the goal SOI $\mathbf{x}_{\ast}$ for surrounding the entire lower part of $\mathbf{B}$. As the dual robot manipulates the bag in an almost symmetrical way, thus the elliptical configuration is used as a reference to determine $\mathbf{x}_{{\dagger}}$ and $\mathbf{x}_{\ast}$. In this work, the bottom vertex points $\mathbf{V}$ of $\mathbf{B}$ are assumed to be coplanar, i.e., $\mathbf{V}=[\mathbf{v}_{1},\ldots,\mathbf{v}_{n_{v}}]$, and $\mathbf{v}_{i}\in\mathbb{R}^{3}$ is the $i$-th vertex point’s Cartesian coordinate in $\mathcal{F}_{w}$. The essence is to change the SOI generation in $\mathcal{F}_{w}$ into 2D generation in the $xy$-plane in the map** frame $\mathcal{F}_{m}$, which is built on the plane where $\mathbf{V}$ is located.

Step 1: Calculate $\mathcal{F}_{m}$ on the plane consisting $\mathbf{V}$.

Two auxiliary vectors are given as: ${\xi}_{i}=\mathbf{v}_{i}-\bar{\mathbf{v}},i=1,2.$ where $\bar{\mathbf{v}}$ is the centroid of $\mathbf{V}$. The $z$-axis of $\mathcal{F}_{m}$ is calculated as ${\mathbf{a}}=({\xi}_{1}\times{\xi}_{2})/\|{\xi}_{1}\times{\xi}_{2}\|$. Any point is selected to determine the $y$-axis: ${\mathbf{o}}=(\mathbf{v}_{3}-\bar{\mathbf{v}})/\|\mathbf{v}_{3}-\bar{\mathbf{v}}\|$. Then, $x$-axis is given as ${\mathbf{n}}=({\mathbf{o}}\times{\mathbf{a}})/\|{\mathbf{o}}\times{\mathbf{a}}\|$. For the uniqueness of $\mathcal{F}_{m}$, taking ${\mathbf{p}}=\bar{\mathbf{v}}$ as the originate of $\mathcal{F}_{m}$. The transformation matrix from $\mathcal{F}_{w}$ to $\mathcal{F}_{m}$ is constructed as:

$$^{\mathcal{F}_{w}}\mathbf{T}_{\mathcal{F}_{m}}=\left[\begin{array}[]{c:c:c:c}{\mathbf{n}}&{\mathbf{o}}&{\mathbf{a}}&{\mathbf{p}}\\ \hdashline 0&0&0&1\end{array}\right]\in\mathbb{R}^{4\times 4}$$ (6)

Adopting (6) to map $\mathbf{V}$ into $\mathcal{F}_{m}$, denoted as ${}^{\mathcal{F}_{m}}\mathbf{V}=\{^{\mathcal{F}_{m}}\mathbf{v}_{i}\}\in\mathbb{R}^{n_{v}\times 3}$ where ${}^{\mathcal{F}_{m}}\mathbf{v}_{i}=[^{\mathcal{F}_{m}}v_{i,x},^{\mathcal{F}_{m}}v_{i,y},{}^{\mathcal{F}_{m}}v_{i,z}]$, then the centroid of ${}^{\mathcal{F}_{m}}\mathbf{V}$ is denoted as ${}^{\mathcal{F}_{m}}\bar{\mathbf{v}}$. Normally, the $z$-axis of ${}^{\mathcal{F}_{m}}\mathbf{V}$ is close to zero, as $\mathbf{V}$ is assumed to be coplanar.

Step 2: Calculate bagging ellipse in $xy$-plane of $\mathcal{F}_{m}$.

The 2D ellipse parametric equation is constructed as:

$$x =\tau_{x}+\rho_{a}\cos(\theta)\cos(\alpha)-\rho_{b}\sin(\theta)\sin(\alpha)$$

$$y =\tau_{y}+\rho_{a}\cos(\theta)\sin(\alpha)+\rho_{b}\sin(\theta)\cos(\alpha)$$ (7)

where $\tau_{x},\tau_{y}$ are the centroid. $\rho_{a},\rho_{b}$ are the axes lengths. $\theta$ is the parameter belong to $[0,2\pi]$. $\alpha$ is the rotation angle. Let $\theta_{i}=2\pi i/1800,i\in[1800]$, the generated 2D ellipse is given as ${}^{\mathcal{F}_{m}}\Omega^{e}:=\{(x_{i},y_{i})|\theta_{i}\}\in\mathbb{R}^{1800\times 2}$, and the $xy$-coordinates of ${}^{\mathcal{F}_{m}}\mathbf{V}$ is extracted as ${}^{\mathcal{F}_{m}}\Omega^{v}:=\in\mathbb{R}^{n_{v}\times 2}$.

The 2D ellipse standard equation is constructed as:

$${}^{\mathcal{F}_{m}}f_{s}(x,y)={{{{\left({\left({x-{\tau_{x}}}\right)\cos\alpha+\left({y-{\tau_{y}}}\right)\sin\alpha}\right)}^{2}}}}/{{{\rho_{a}^{2}}}}$$

$$+{{{{\left({\left({{\tau_{x}}-x}\right)\sin\alpha+\left({y-{\tau_{y}}}\right)\cos\alpha}\right)}^{2}}}}/{{{\rho_{b}^{2}}}}\geq 0$$ (8)

Whether $(x,y)$ is inside the ellipse can be judged by (IV-B), which is used construct the subsequent constraint. Let the perimeter of the bag’s rim as $\omega$, and the cost function that satisfies the perimeter limitation is constructed as:

$$\mathcal{J}_{1}(\tau_{x},\tau_{y},\rho_{a},\rho_{b},\alpha)=\big{\|}2\pi\sqrt{{(\rho_{a}^{2}+\rho_{b}^{2})}/{2}}-\omega\big{\|}^{2}$$ (9)

Further, three additional constraints are defined as:

Constraint $C_{1}$: it regulates the covering of ${}^{\mathcal{F}_{m}}\Omega^{e}$ to ${}^{\mathcal{F}_{m}}\Omega^{v}$:

$$0\leq^{\mathcal{F}_{m}}f_{s}(^{\mathcal{F}_{m}}v_{i,x},^{\mathcal{F}_{m}}v_{i,y})\leq\lambda_{1},\ \ \ i\in[1,\ldots,n_{v}]$$ (10)

where $\lambda_{1}$ controls the enclosing degree. The smaller $\lambda_{1}$ is, the center alignment between ${}^{\mathcal{F}_{m}}\Omega^{e}$ and ${}^{\mathcal{F}_{m}}\Omega^{v}$ more obvious. The default value is $\lambda_{1}=0.87$.

Constraint $C_{2}$: it limits the Euclidean distance between the centers of ${}^{\mathcal{F}_{m}}\Omega^{e}$ and ${}^{\mathcal{F}_{m}}\Omega^{v}$:

$$0\leq\big{\|}[\tau_{x},\tau_{y},0]-^{\mathcal{F}_{m}}\bar{\mathbf{v}}\big{\|}\leq\lambda_{2}$$ (11)

where $\lambda_{2}$ specifies the proximity of the two centers, it has the similar control effect to $\lambda_{1}$. The default value is $\lambda_{2}=0.003$.

Constraint $C_{3}$: it adjusts the parallelism of the respective principal axes of ${}^{\mathcal{F}_{m}}\Omega^{e}$ and ${}^{\mathcal{F}_{m}}\Omega^{v}$, denoted as $\eta_{e}\in\mathbb{R}^{2},\eta_{v}\in\mathbb{R}^{2}$ and calculated by PCA [15]. Afterwards, the inner product is used to evaluate the parallelism:

$$-\lambda_{3}\leq|{\rm{dot}}(\eta_{e},\eta_{v})|-1\leq\lambda_{3}$$ (12)

where $\lambda_{3}$ controls the parallel degree. As we only consider the parallelism, and ignore the direction (same/opposite), so we take absolute operation and subtract 1. The default value is $\lambda_{3}=0.0001$.

The optimal values $(\tau_{x}^{*},\tau_{y}^{*},\rho_{a}^{*},\rho_{b}^{*},\alpha^{*})$ can be obtained by minimizing $\mathcal{J}_{1}$, and considering three constraints $C_{1},C_{2},C_{3}$. The nonlinear optimizer is adopted to obtain the optimal values: ${}^{\mathcal{F}_{m}}\Omega^{e,\ast}:=\{(x_{i},y_{i})|\tau_{x}^{*},\tau_{y}^{*},\rho_{a}^{*},\rho_{b}^{*},\alpha^{*}\}$, then concatenate a zero vector horizontally to make ${}^{\mathcal{F}_{m}}\Omega^{e,\ast}$ three-dimensional.

Step 3: Bagging SOI Generation.

Similarly, adopting (6) to map ${}^{\mathcal{F}_{m}}\Omega^{e,\ast}$ into $\mathcal{F}_{w}$ to obtain ${}^{\mathcal{F}_{w}}\Omega^{e,\ast}$, and whose dimension should be consistent with that of $\mathbf{x}$. Thus, farthest point sampling (FPS) [16] extracts $n_{x}$ samples from ${}^{\mathcal{F}_{w}}\Omega^{e,\ast}$ to obtain:

$$\mathbf{x}_{{\dagger}}={\rm{FPS}}(^{\mathcal{F}_{w}}\Omega^{e,\ast},\ n_{x})\in\mathbb{R}^{n_{x}\times 3}$$ (13)

Step 4: Goal SOI Generation.

Note that $\mathbf{x}_{{\dagger}}$ is coplanar with $\mathbf{V}$, and seen as an transient shape, i.e., $\mathbf{x}_{{\dagger}}$ is at the bottom of $\mathbf{B}$. Our goal is to generate a shape that surrounds the bottom part of $\mathbf{B}$, so we simply translate $\mathbf{x}_{{\dagger}}$ along $\mathbf{a}$ by a safety threshold $\gamma$ to get $\mathbf{x}_{\ast}$, it yields

$$\mathbf{x}_{\ast}=\mathbf{x}_{{\dagger}}+\gamma\cdot\mathbf{a}$$ (14)

Finally, $\mathbf{x}_{\ast}$ is the goal SOI, covering the bottom part of $\mathbf{B}$. Fig. 5 visualizes the bagging/goal SOI and three constraints.

Remark 1.

In this work, $\mathbf{a}$ should hold an acute angle with the positive direction of the $z$-axis, which can be adjusted by calculating $\rm{dot}(\mathbf{a},[0,0,1])$. If negative, we should reverse ${\mathbf{a}}$.

Figure 6: Illustration of SOI planning. $\mathbf{x}_{0}$ and $\mathbf{x}_{{\dagger}}$ are the initial and bagging SOI. $\mathbf{x}_{\ast}$ is goal SOI covering the bottom part of $\mathbf{B}$.

IV-C SOI Planning

In this section, we introduce how to generate a collision-free deformation path $\mathcal{G}$ of the bag from the initial SOI $\mathbf{x}_{0}$ to $\mathbf{x}_{\ast}$, where $\mathbf{x}_{\ast}$ is the bagging SOI as the pre-enclosing configuration of $\mathbf{B}$. Note that $\mathcal{G}$ includes two stages, $\mathbf{x}_{0}\rightarrow\mathbf{x}_{{\dagger}}$ and $\mathbf{x}_{{\dagger}}\rightarrow\mathbf{x}_{\ast}$ Then, $\mathcal{G}$ serves as the desired trajectory of the subsequent controller to assist in completing the bagging task. The SOI planning is conducted in the world frame $\mathcal{F}_{w}$. The studied planning task can be seen as the shape planning of bag’s SOI, as a global trajectory guiding the robot.

The 3D ellipse parametric equation is constructed as:

$$^{\mathcal{F}_{w}}f_{p}(\mathbf{c},\beta_{a},\beta_{b},{\mathbf{u}},{\mathbf{v}})=\mathbf{c}+\beta_{a}\cos(\theta){\mathbf{u}}+\beta_{b}\sin(\theta){\mathbf{v}}$$ (15)

where $\mathbf{c}=[c_{x},c_{y},c_{z}]$ is the centroid. $\beta_{a}$ and $\beta_{b}$ determine the semi-major and semi-minor axes lengths, respectively. $\theta$ is the parametric angle belong to $[0,2\pi]$. ${\mathbf{u}}\in\mathbb{R}^{3},{\mathbf{v}}\in\mathbb{R}^{3}$ are the direction vectors. Let $\theta_{i}=2\pi i/2000,i\in[1,2000]$, and the 3-dimension ellipse in $\mathcal{F}_{w}$ is constructed as:

$$\Upsilon_{e}:=\{\Upsilon_{i}=(x_{i},y_{i},z_{i})|\theta_{i}\leftarrow\eqref{eq34}\}\in\mathbb{R}^{2000\times 3}$$ (16)

The perimeter of $\Upsilon_{e}$ is numerically obtained as $\chi=\Sigma_{i=1}^{2001}\|\Upsilon_{i}-\Upsilon_{i-1}\|$ with $\Upsilon_{2001}=\Upsilon_{1}$ for the circle calculation.

Figure 7: Visualization of map** a randomly sampled configuration in the raw space into the stable manifold. The blue dot-line represents the randomly sampled configuration, and the yellow one is the stable configuration in the manifold through ProjectStableConfig.

Projection of Stable Configuration Manifold

The bag’s raw configuration space $\mathbf{x}_{t}$ has a dimensionality of $3n_{x}$. However, its stable state is confined to a specific subspace known as a manifold within this larger space. Therefore, it can enhance the planning credibility if the planning process is performed specifically on this manifold that contains the stable state of the bag. However, it’s challenging to obtain this manifold through random sampling in the raw space as the dimensions of the raw space significantly exceed those of the stable space. To discover this constraint manifold within the current shape configuration, it would be suitable to employ the projection method, which allows for a more targeted exploration of the stable state [17].

By formulating a random sampling in the raw space as a local minimization problem of the energy, it can project it onto the stable manifold, with $\mathbf{x}_{t}$ as the initial value.

$$\mathbf{x}_{t}^{\rm{stable}} =\mathop{\arg\min}_{\mathbf{x}}\ \mathcal{J}_{2}(\mathbf{x}),\ \ \ {\rm{s.t.}}\ \ \mathbf{x}=\mathbf{x}_{t}$$ (17)

A geometric index is used as the projection model, where the cost function $\mathcal{J}_{2}$ for a configuration $\mathbf{x}$ is presented as:

$$\mathcal{J}_{2}(\mathbf{c},\beta_{a},\beta_{b},{\mathbf{u}},{\mathbf{v}})=|{\rm{CD}}(\Upsilon_{e},\mathbf{x}_{t})|^{2}$$ (18)

where $\rm{CD}(\cdot)$ is the Chamfer Distance, to evaluate the similarity between two unordered dataset with different dimensions.

Constraint $C_{4}$: it ensure the perimeter $\chi$ of $\Upsilon_{e}$ on the manifold be consistent with the bag rim’s perimeter $\omega$.

$$1-\lambda_{4}\leq{\chi}\ /\ {\omega}\leq 1+\lambda_{4}$$ (19)

where $\lambda_{4}$ controls the scale between $\chi$ and the ground-truth perimeter $\omega$. The default value is set to $\lambda_{4}=0.001$.

Constraint $C_{5}$: it ensures that the projection process completed without large offsets.

$$0\leq\|\mathbf{c}-\bar{\mathbf{x}}_{t}\|<\lambda_{5}$$ (20)

where $\bar{\mathbf{x}}_{t}$ is the centroid of $\mathbf{x}_{t}$. The default value is $\lambda_{5}=0.01$.

The optimal values of (15) can be obtained by minimizing $\mathcal{J}_{2}$ with (19). Afterwards, $\mathbf{x}_{t}^{\rm{stable}}={{}^{\mathcal{F}_{w}}f_{p}}(\mathbf{c}^{*},\beta_{a}^{*},\beta_{b}^{\ast},{\mathbf{u}}^{*},{{{\mathbf{v}}}^{\ast}})$. Additional constraints can indeed be incorporated to cater to specific tasks. The projection process from a raw configuration space to a neighboring stable manifold is denoted as $\mathbf{x}^{\rm{stable}}=\rm{ProjectStableConfig}(\mathbf{x}_{t})$. Four examples of ProjectStableConfig are shown in Fig. 7.

Step 1: Pre-Bagging SOI Planning

Our shape planning algorithm follows the same streamline as the Constrained Bi-directional Rapidly-Exploring Random Tree (CBiRRT) [17]. ProjectStableConfig ensures the validity of nodes, and CBiRRT contains two independent trees, growing from the initial configuration and the goal configuration, respectively. Both trees expand and explore the configuration space, gradually moving towards each other, until they eventually become connected to generate the final path. For our planning, the bag’s state $\mathbf{x}_{t}$ is regarded as the tree’s node. The procedure of CBiRRT is introduced in [18], interesting readers could refer to it.

In planning, each random node $\mathcal{G}_{\rm{rand}}$ is projected to the stable manifold using ProjectStableConfig before the next-step planning. If the bag is in collision, $\mathcal{G}_{\rm{rand}}$ is discarded and regenerated. The Chamfer Distance is utilized to calculate the distance between two nodes, this point is different from [18].

Similiiar to [17], the constrained extension is denoted as $\mathcal{G}_{\rm{reached}}$ = ConstrainedExtend($\mathcal{G}_{\rm{from}},\mathcal{G}_{\rm{to}})$. This function aims to make progress from $\mathcal{G}_{\rm{from}}$ towards reaching $\mathcal{G}_{\rm{to}}$ while adhering to the constraints and limitations imposed by the planning problem. During each step of the process, a new configuration for the bag is generated by interpolating from the last reached configuration $\mathbf{x}_{\rm{last}}$ to $\mathbf{x}_{\rm{to}}$, using a small step size. To ensure the overall shape of the bag is preserved and prevent excessive stretching, the displacement limitation for the relative deformation of the bag’s rim is enforced. Afterwards, a stable configuration $\mathbf{x}_{\rm{new}}$ is obtained using ProjectStableConfig on the interpolated configuration.

After planning, the bidirectional path of CBiRRT is extracted as the deformation path $\mathcal{G}$, then refined for the subsequent smooth dual-arm manipulation. The pre-bagging path is constructed as $\mathcal{G}_{\rm{pre\text{-}bagging}}:=\{\mathbf{g}_{0},\mathbf{g}_{1},\ldots,\mathbf{g}_{{\dagger}}\}$

Step 2: Bagging SOI Planning

This stage presents the deformation path from $\mathbf{x}_{{\dagger}}$ to $\mathbf{x}_{\ast}$, adopting the same planning procedure as the Step 1. The bagging path is constructed as $\mathcal{G}_{\rm{bagging}}:=\{\mathbf{g}_{{\dagger}},\ldots,\mathbf{g}_{\ast}\}$.

The final path $\mathcal{G}$ from $\mathbf{x}_{0}$ to $\mathbf{x}_{\ast}$ is constructed as:

$$\mathcal{G}:=\{{\left[\mathcal{G}_{\rm{pre\text{-}bagging}},\mathcal{G}_{\rm{bagging}}\right]|\underbrace{\mathbf{g}_{0},\mathbf{g}_{1},\ldots,\mathbf{g}_{{\dagger}}}_{\rm{pre\text{-}bagging}},\underbrace{\mathbf{g}_{{\dagger}},\ldots,\mathbf{g}_{\ast}}_{\rm{bagging}}}\}$$ (21)

IV-D Motion Planning

The process of the proposed bagging manipulation approach is: (a) the vision system perceives $\mathbf{B}$, and the robot generates the bagging/goal SOI. (b) an collision-free deformation path $\mathcal{G}$ is obtained using CBiRRT. (c) the dual robot completes the bagging task along $\mathcal{G}$ in a constrained environment. For providing a clear and intuitive visual effect, the robot adopts 3D translation and 3D rotation. The end-effector’s pose is denoted by $\mathbf{r}=[\mathbf{p}^{\rm{[left]}},\mathbf{p}^{[\rm{right}]}]\in\mathbb{R}^{12}$. We assume that the material properties of the bag and the robot movements remain relatively stable during the manipulation process. The robot is able to execute the given velocity commands accurately and without delay. For the controller design, we formulate this manipulation process as the shape servoing [19], i.e., tiny movements of the robot can produce tiny deformations of the bag. Inspired by [20], the local first-order kinematic model can be obtained as below:

$$\mathbf{y}_{t}=\mathbf{J}_{t}\mathbf{u}_{t},\ \ \mathbf{y}_{t}=\mathbf{x}_{t}-\mathbf{x}_{t-1},\ \ \mathbf{u}_{t}=\mathbf{r}_{t}-\mathbf{r}_{t-1}$$ (22)

where $\mathbf{J}_{t}$ is the deformation Jacobian matrix (DJM), which represents the kinematic relationship between $\mathbf{y}_{t}$ and $\mathbf{u}_{t}$. We make the assumption that $\mathbf{J}_{t}$ maintains full column rank while performing the manipulation task, which is straightforward to fulfill in practical scenarios since the dimension of $\mathbf{x}$ is significantly greater than $\mathbf{u}$. Since the bag has strong unknown nonlinearity, it’s difficult to obtain accurate analytical expression of $\mathbf{J}_{t}$. Therefore, the Broyden approach is used to computes local approximations of $\mathbf{J}_{t}$ in real time instead of identifying the full mechanical model.

$$\hat{\mathbf{J}}_{t}=\hat{\mathbf{J}}_{t-1}+\varepsilon\cdot({{{{\mathbf{y}_{t}}-\hat{\mathbf{J}}_{t-1}{\mathbf{u}_{t}}}}})\ /\ ({{\mathbf{u}_{t}^{\intercal}{\mathbf{u}_{t}}}})\cdot\mathbf{u}_{t}^{\intercal}\in\mathbb{R}^{3n_{x}\times 12}$$ (23)

where $\varepsilon\in(0,1]$ regulates the convergence speed.

Considering $\mathcal{G}$ contains a sequence of trajectories, thus we adopt a model predictive control (MPC) to drive the dual-arm manipulate along $\mathcal{G}$. To simplify calculation burden, $\mathbf{J}_{t}$ is assumed to be estimated accurately, such that it satisfies $\mathbf{y}_{t}={\hat{\mathbf{J}}_{t}}{\mathbf{u}_{t}}$. Two prediction vectors are defined as follows:

$$\bar{\mathbf{x}}_{t}=\{\mathbf{x}_{t+i|t}\}\in{\mathbb{R}^{3n_{x}h}},\ \bar{\mathbf{u}}_{t}=\{\mathbf{u}_{t+i-1|t}\}\in{\mathbb{R}^{12h}},i\in[1,h]$$ (24)

where $\bar{\mathbf{x}}_{t}$ and $\bar{\mathbf{u}}_{t}$ represent the predictions of $\mathbf{x}_{t}$ and $\mathbf{u}_{t}$ in the next $h$ periods, respectively. $\mathbf{x}_{t+i|t}$ and $\mathbf{u}_{t+i|t}$ denote the $i$th predictions of $\mathbf{x}_{t}$ and $\mathbf{u}_{t}$ from the time instant $t$, where $\mathbf{x}_{t|t}=\mathbf{x}_{t}$, and $\mathbf{u}_{t|t}=\mathbf{u}_{t}$ must hold. $\bar{\mathbf{x}}_{t}$ can be calculated from $\hat{\mathbf{J}}_{t}$ by noting that $\hat{\mathbf{\mathbf{J}}}_{t}\approx\hat{\mathbf{\mathbf{J}}}_{t+h}$ is satisfied during period $[t,t+h]$ (which is reasonable, given the slow manipulation of the bag). In this way, $\bar{\mathbf{x}}_{t}$ are computed as the augmented format:

$$\bar{\mathbf{x}}_{t} =\mathbf{D}\mathbf{x}_{t}+\boldsymbol{\Theta}{\bar{\mathbf{u}}_{t}},\ \ \ \mathbf{D}=\mathbf{I}_{h\times 1}\otimes\mathbf{E}_{3n_{x}},\ \ \ \boldsymbol{\Theta}=\mathbf{L}_{h}\otimes\hat{\mathbf{J}}_{t}$$ (25)

The target $\bar{\mathbf{x}}_{t}^{*}$ is constructed as $\bar{\mathbf{x}}_{t}^{*}=[\mathcal{G}_{t+1},\mathcal{G}_{t+2},\ldots,\mathcal{G}_{t+h}]$. The optimization function of $\bar{\mathbf{u}}_{t}$ is formulated as:

$$\mathcal{Q}\left(\bar{\mathbf{u}}_{t}\right)={\left({{\bar{\mathbf{x}}_{t}}-{\bar{\mathbf{x}}_{t}^{*}}}\right)^{\intercal}}\boldsymbol{\Lambda}_{1}\left({{\bar{\mathbf{x}}_{t}}-{\bar{\mathbf{x}}_{t}^{*}}}\right)+\bar{\mathbf{u}}_{t}^{\intercal}\boldsymbol{\Lambda}_{2}{\bar{\mathbf{u}}_{t}}$$ (26)

where $\boldsymbol{\Lambda}_{1}$ and $\boldsymbol{\Lambda}_{2}$ are symmetric positive-definite matrices, regulating the convergence speed and the smoothness of $\bar{\mathbf{u}}_{t}$, respectively. Finally, ${\mathbf{u}}_{t}$ is obtained by the receding horizon:

$$\mathbf{u}_{t}=[\mathbf{E}_{12},\mathbf{0},\ldots,\mathbf{0}]\cdot\bar{\mathbf{u}}_{t}\in\mathbb{R}^{12}$$ (27)

Figure 8: Experimental set-up. (a) Overview of the experimental set-up to validate our SOI-based control for the dual-arm bagging task. (b)-(e) Four samples of dual-arm manipulation bags. (f) - (i) Four baggable objects, i.e., coffee box, canned pineapple, grapefruit, and 3D-printed triangular prism.

V Experiments

V-A Experimental Setup

As shown in Fig. 8, we describe The experimental setup used to validate the proposed SOI-based control of dual-arm bagging task, including four baggable objects $\mathbf{B}$. A D455 camera is in the eye-to-hand configuration, and used to observe the manipulation process from a top-down perspective with the resolution 640x480. Visual perception is processed with OpenCV on a Linux-based PC, and the point cloud $\mathcal{P}_{t}$ are obtained through RealSense libraries. Dual-CR5 robots are equipped with 3D-printer holders to grasp the both ends of the bag with zip ties in advance, and assume that no drops occur during manipulation. The custom bag with a green rim is adopted for ease of perception. The velocity command $\mathbf{u}_{t}$ has a hard saturation to meet the assumption in Sec. III to ensure the estimation validity of $\mathbf{J}_{t}$. The motion control algorithm is implemented on ROS, which runs with a servo-control loop of around 11 Hz.

We use professional 3D scanners (Model: CR-Scan Ferret Pro) to obtain the vertex points $\mathbf{V}$ of each $\mathbf{B}$. Meanwhile, the ArUco markers are attached to $\mathbf{B}$ to ensure that the robot can determine the type through the camera before manipulation, then call the corresponding configuration of $\mathbf{V}$.

V-B Evaluation of GMM-based State Estimation

In this section, we verify the GMM-based state estimator introduced in (5), it aims to extract clear state $\mathcal{Q}_{t}$ from the raw dense and noisy point cloud $\mathcal{P}_{t}$. As the used bag has an obvious rim, so $\mathcal{P}_{t}$ can be obtained simply. EM algorithm [14] is used to solve $\mathcal{O}(\mathbf{x}_{t}^{n})$, we can get the concise $\mathcal{Q}_{t}$.

Fig. 9 shows the extraction effect of the GMM-based state estimator, and the bag’s rim are marked by green, as shown in Fig. 9a. The results in Fig. 9b show that the GMM-based state estimator can propose a relatively completely $\mathcal{Q}_{t}$, and $\mathcal{Q}_{t}$ is equidistantly distributed, this echoes the uniform distribution assumption (4). Furthermore, Fig. 9c is added to evaluate ProjectStableConfig in (17). The results show that ProjectStableConfig can find a stable manifold under the current shape configuration $\mathcal{Q}_{t}$, presented as the red curve in Fig. 9c distributed along $\mathcal{Q}_{t}$. This proves the effectiveness of ProjectStableConfig and can find a stable manifold projection, which is helpful for subsequent planning and control.

Figure 9: Extraction results of the GMM-based state estimator. (a) the bag’s SOI. (b) the blue dots are $\mathcal{P}_{t}$, and the red ones are $\mathcal{Q}_{t}$ estimated by GMM. (c) the smooth state by refining $\mathcal{Q}_{t}$ using ProjectStableConfig.

Figure 10: (a) - (f): Generations of bagging SOI $\mathbf{x}_{{\dagger}}$. (g)-(l): Deformation path $\mathcal{G}$.

V-C Evaluation of bagging SOI Generation

In this section, we evaluate the bagging SOI generation presented in Sec. IV-B, to generate a pre-enclosed shape $\mathbf{x}_{{\dagger}}$ to cover the bottom of $\mathbf{B}$. Six types of baggable objects are adopted with the known $\mathbf{V}$. The parameters are set to $\omega=0.68,\lambda_{1}=0.85,\lambda_{2}=0.005,\lambda_{3}=0.001$.

Fig. 10a - Fig. 10f shows the results of the bagging SOI $\mathbf{x}_{{\dagger}}$. The blue points represent $\mathbf{V}$, and the red one is $\mathbf{x}_{{\dagger}}$ through (13). The results show that $\mathbf{x}_{{\dagger}}$ satisfies the perimeter $\omega$, and can surround $\mathbf{V}$ to the greatest extent along the principal axis of $\mathbf{V}$. And $\mathbf{x}_{{\dagger}}$ is generated evenly distributed around $\mathbf{V}$, which verifies the regulation of $C_{1}$. As $\mathbf{x}_{{\dagger}}$ is generated by a parametric equation, $\mathbf{x}_{{\dagger}}$ is continuous, which is helpful for subsequent SOI planning. Moreover, in the experiment, we found that by adjusting $\lambda_{1}$ of constraint $C_{1}$ (10) and $\lambda_{3}$ of constraint $C_{3}$ (12), it can efficiently regulate $\mathbf{x}_{{\dagger}}$ to adapt to various $\mathbf{B}$, so that $\mathbf{x}_{{\dagger}}$ can best meet different task requirements. The average values of $C_{1}$, $C_{2}$, and $C_{3}$ are 0.731, 0.004, and $0.0001$, respectively.

V-D Evaluation of SOI Planning

In this section, we evaluate the SOI planning presented in Sec. IV-C, which aims to generate a collision-free deformation path $\mathcal{G}$ from the initial SOI $\mathbf{x}_{0}$ to the goal SOI $\mathbf{x}_{*}$ via the bagging SOI $\mathbf{x}_{{\dagger}}$. The parameter is $\lambda_{4}=0.002,\lambda_{5}=0.02$.

Fig. 10g - Fig. 10l give six planning results $\mathcal{G}$ of different configurations of $(\mathbf{x}_{0},\mathbf{x}_{{\dagger}},\mathbf{x}_{\ast})$. The gradient curves from blue to red is $\mathcal{G}_{\rm{pre\text{-}bagging}}$, and that from red to green is $\mathcal{G}_{\rm{bagging}}$. As ProjectStableConfig is used, thus each node in $\mathcal{G}$ is smooth, and satisfies the physical perimeter constraint $\omega$. Since the distance from $\mathbf{x}_{{\dagger}}$ to $\mathbf{x}_{\ast}$ is short and the distance to $\mathbf{B}$ is close, $\mathcal{G}_{\rm{bagging}}$ shows a certain degree of fluctuation. Fig. 10j shows that CBiRRT can generate an effective deformation path $\mathcal{G}$ even when there are obstacles. The black cuboid is the used-defined obstacle. The planning results show that continuous $\mathcal{G}$ can be obtained using the CBiRRT, and (19) guarantees the perimeter limitation of each node in $\mathcal{G}$. This proves the rationality of the optimization manner (18), and various constraints can be added to improve the planning accuracy and meet the task requirements.

Note that the proposed SOI planning is a top-level planning framework, the kind where the initial SOI $\mathbf{x}_{0}$, bagging SOI $\mathbf{x}_{{\dagger}}$, and the goal $\mathbf{x}_{\ast}$ are given, the two-stage deformation trajectories are planned, i.e., $\mathcal{G}_{\rm{pre\text{-}bagging}}$ and $\mathcal{G}_{\rm{bagging}}$. It’s just that in this article, $\mathbf{x}_{\ast}$ is done by simply translating $\mathbf{x}_{{\dagger}}$, but $\mathbf{x}_{\ast}$ may have more complex format actually.

Figure 11: Manipulation trajectories of the bag’s SOI in the bagging task. The last column gives the deformation error of each step of MPC.

V-E Dual-arm Bagging Manipulation

The dual-arm bagging experiments are conducted to evaluate the proposed SOI-based bagging manipulation approach. The used baggable objects contain four types, i.e., coffee box, canned pineapple, grapefruit, and 3D-printed triangular prism, for Exp 1 to Exp 4, respectively. The fundamental process is that the dual-CR5 manipulates the bag to first deform along $\mathcal{G}_{\rm{pre\text{-}bagging}}$ to $\mathbf{x}_{{\dagger}}$, then deform along $\mathcal{G}_{\rm{bagging}}$ to $\mathbf{x}_{\ast}$, and finally complete the bagging task. For analyzing the bagging approach, we compare two planning algorithms (FFG-RRT [21], TS-RRT [22]) and two manipulation algorithms (IBVS [23], SSVS [24]) respectively in Sec. IV-C and Sec. IV-D.

Fig. 11 shows the bagging results of four experiments, with each row representing a type of $\mathbf{B}$. The first five columns of each row represent the deformation process, the sixth column represents $\mathcal{G}$, and the last column represents the deformation error $\|\mathbf{x}_{t}-\mathcal{G}_{t}\|$ of each step of MPC. In order to quantitatively compare performance, three indicators are introduced, i.e., planning success rate, planning time, and manipulation success rate, corresponding to different planning algorithms and control algorithms respectively. Table I gives the detailed comparative analysis outcomes.

Planning success rate shows that CBiRRT outperforms the other counterpart, with the acceptable computation time, while FFG-RRT is the fastest. This is because FFG-RRT directly explores forward and rushes to the desired configuration at the fastest speed, while CBiRRT conducts two-way exploration based on stability, this results in CBiRRT have more exploration steps. From the manipulation success rate, we know that the MPC used in this article has the highest value, while the other two control approaches are slightly worse. This is because the desired command of the traditional shape servoing is stationary, while that of our bagging task is actually a sequence of deformation trajectories. This point is very consistent with the MPC processing manner, and can ensure the stability of tracking in the future prediction time domain. The manipulation results prove the effectiveness of MPC in such robot manipulation tasks.

Besides, $\mathbf{x}_{{\dagger}}$ is equivalent to an intermediate buffer shape, thus dividing the entire bagging task into two subtasks, namely pre-bagging and bagging, thereby improving the success rate of manipulation.

Table I: Performance of Different Sensorimotor Models on Different Tasks for Motor-robot Experiments

Method

Coffee box (Exp 1)

Canned pineapple (Exp 2)

Grapefruit (Exp 3)

Triangular prism (Exp 4)

Planning success rate

Planning time (s)

Manipulation success rate

Planning success rate

Planning time (s)

Manipulation success rate

Planning success rate

Planning time (s)

Manipulation success rate

Planning success rate

Planning time (s)

Manipulation success rate

FFG-RRT [21]

6/10

$3.87\pm 1.97$

8/8

8/10

2.37 $\pm$ 0.87

8/8

7/10

3.89 $\pm$ 1.18

8/8

6/10

3.58 $\pm$ 1.11

8/8

TS-RRT [22]

7/10

6.32 $\pm$ 1.08

8/8

8/10

5.58 $\pm$ 1.13

8/8

9/10

6.85 $\pm$ 0.56

8/8

7/10

7.32 $\pm$ 1.34

8/8

IBVS [23]

-

-

4/8

-

-

7/8

-

-

5/8

-

-

6/8

SSVS [24]

-

-

5/8

-

-

7/8

-

-

6/8

-

-

7/8

Ours

9/10

5.13 $\pm$ 1.26

8/8

10/10

4.21 $\pm$ 0.98

8/8

10/10

4.98 $\pm$ 1.93

8/8

9/10

5.32 $\pm$1.56

8/8

VI Conclusion

Our study introduced a dual-arm robotic system for automating bagging tasks, employing a novel constraint-aware SOI planning approach for manipulating 3D deformable objects. The system’s innovation lies in its targeted SOI state estimation, which simplifies the control of the bag’s opening rim, enhancing task efficiency. Key contributions include a flexible, adaptive vision-based control system and a comprehensive framework demonstrating the system’s adaptability to environmental constraints. This research not only advances DOM in handling complex tasks but also has potential implications for enhancing robotic assistance in everyday activities. Future efforts will aim to improve system adaptability and extend its application to further realize the benefits of robotic automation in diverse real-world settings.