ProbRadarM3F: mmWave Radar-based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion

Abstract

Millimeter wave (mmWave) radar is a non-intrusive privacy and relatively convenient and inexpensive device, which has been demonstrated to be applicable in place of RGB cameras in human indoor pose estimation tasks. However, mmWave radar relies on the collection of reflected signals from the target, and the radar signals containing information is difficult to be fully applied. This has been a long-standing hindrance to the improvement of pose estimation accuracy. To address this major challenge, this paper introduces a probability map guided multi-format feature fusion model, ProbRadarM3F. This is a novel radar feature extraction framework using a traditional FFT method in parallel with a probability map based positional encoding method. ProbRadarM3F fuses the traditional heatmap features and the positional features, then effectively achieves the estimation of 14 keypoints of the human body. Experimental evaluation on the HuPR dataset proves the effectiveness of the model proposed in this paper, outperforming other methods experimented on this dataset with an AP of 69.9 $\%$. The emphasis of our study is focusing on the position information that is not exploited before in radar singal. This provides direction to investigate other potential non-redundant information from mmWave rader.

I Introduction

Smart medical care is paramount in hospitals and nursing homes in today’s society. To improve healthcare outcomes and ensure patient safety, there is an increasing need for human pose monitoring. The Internet of Things (IoT) has become instrumental in this regard, providing continuous and efficient monitoring capabilities [1, 2]. As shown in Fig. 1, IoT with human activity monitoring is essential to improve the convenience, safety, and interactivity of medical care. Traditionally, monitoring systems have relied on RGB cameras to estimate human pose and track human activity [3]. However, these methods pose significant privacy concerns and can be less effective in low-light or obscured environments.

Recent advancements focus on reducing privacy concerns by using non-visual signal-based sensing methods [4, 5]. These signals do not contain human facial information or visual photographs, thus addressing the need for human privacy in IoT application scenarios [6, 7]. Millimeter-wave (mmWave) radar has emerged as a prominent technology due to its high resolution and cost-effectiveness. MmWave radar can accurately estimate human skeletal poses without capturing identifiable visual information, making it ideal for preserving privacy. This paper focuses on mmWave radar-based human skeletal pose estimation, a critical component in smart sensor systems for medical care or smart home IoT applications.

Figure 1: Consider an illustration of a medical IoT application: In this scenario, if a person receiving care at home experiences a fall, it is immediately detected, triggering an alarm at a remote care center and prompting the automatic dispatch of an ambulance. Our paper focuses on utilizing mmWave radar to estimate the human skeletal pose, a critical sensor component in this example of a medical IoT application. This approach preserves the user’s privacy while still providing the necessary monitoring information to the care center.

MmWave radar, a robust yet low-cost sensor, has been applied for environment perception in advanced driver-assistance systems (ADAS) [8, 9, 10, 11, 12, 13] and cooperative intelligent transportation systems (C-ITS) [14] in recent years. Considerable frontier research on mmWave radar has focused on achieving human activity sensing tasks. However, significant challenges remain.

In the classical radar processing pipeline, various levels of data representation exist. For human skeletal pose estimation, point clouds and heatmaps are the most widely used forms of data representation. However, point cloud generation relies on hand-crafted parameters, and the process itself loses Doppler information and some positional data, leading to difficulties in capturing finer joint features. Heatmaps are often generated using the Fast Fourier Transform (FFT) method [15], which can extract range, Doppler, and angle of arrival (AoA) information from the raw radar signal to create different types of heatmaps. However, much of the information in the heatmap is considered redundant and not valued. Recently, many pose estimation works have been realized through a point cloud $+$ neural network pipeline or a heatmap $+$ neural network pipeline. Some studies have focused on improving the deep learning capability of networks and algorithms to address the mmWave radar-based precision problem. However, the importance of capturing the information present in the radar signal itself is often neglected, leading to the loss of significant effective information.

To address the inefficiencies in traditional heatmap-based methods, a new method is proposed, constructing an efficient feature extraction framework. Specifically, the method incorporates two feature extraction branches. One branch adheres to conventional procedures, utilizing the FFT method to generate heatmaps for extracting range, Doppler, azimuth, and elevation information from mmWave radar signals. The other branch introduces a probability map generation and positional encoding method, designed to enhance the utilization of positional information in radar signals. A model is also proposed for fusing probability-guided positional features with traditional heatmap features. The contributions of this work are summarized below:

• A mmWave radar-based multi-format feature fusion model, ProbRadarM3F, is presented for human skeletal pose estimation. The model is designed for raw mmWave radar signals, with features extracted from two branches: the FFT branch and the ProbPE branch. In the ProbPE branch, probability maps are generated, introducing a positional encoding method. To the best of our knowledge, this study represents the first attempt to extract additional probabilistic and positional features from mmWave radar signals to aid in pose estimation using a probability map-guided positional encoding approach. By fusing features in different formats, the keypoint prediction accuracy is significantly increased. ProbRadarM3F can serve as a new baseline for research and application in radar position information exploitation.

• To validate the effectiveness of ProbRadarM3F by fair comparison with [16], our model was built upon the model used in [16] and experiments were conducted on the same dataset, HuPR [16], which provides purely raw radar signals. The results demonstrate that the design of a probability map-guided positional encoding strategy greatly improves the recognition performance of the model, leading to a significant enhancement in the precision of human skeletal pose prediction.

The subsequent sections of this paper are organized as follows: Section II provides a comprehensive review of the research and methodology of mmWave radar for indoor sensing applications, particularly for human skeletal pose prediction. Section III details the implementation of the proposed ProbRadarM3F model. In Section IV, experiments conducted with ProbRadarM3F on the HuPR dataset are presented, and the results are evaluated. The advantages of its pose prediction capabilities are demonstrated. Finally, Section V summarizes the approach and findings and suggests potential directions for future research.

Figure 2: Illustration of our proposed ProbRadarM3F model. This model consists of two main branches: the FFT branch and the ProbPE branch. Different colored arrows represent data streams from different radars. The initial input is the raw ADC data from two vertically placed mmWave radars. After processing the raw radar data into a radar data cube, the FFT branch processes the data to extract features from range-Doppler-azimuth heatmaps and range-Doppler-elevation heatmaps based on 4D-FFT. The ProbPE branch applies positional encoding to extract features from generated radar probability maps. In the following module, the multi-frame and multi-format features are fused, and a cross- and self-attention module is introduced to generate skeletal pose estimation.

II Related Works

II-A Millimeter-Wave Radar Indoor Sensing

The field of millimeter-wave (mmWave) radar for indoor sensing has gained substantial advancements in recent years, driven by the growing demand for non-invasive, privacy-preserving technologies for human presence and activity detection. MmWave radar offers unique advantages for indoor sensing applications. The mmWave frequency can penetrate non-metallic objects while providing high-resolution imaging capabilities [17]. Its utility is increasingly evident in through-wall sensing [18], human detection and recognition [19], and vital sign monitoring [20, 21, 22].

Initially, researchers used radar signal processing to extract waveform features, combining them with target distance, orientation, altitude, and motion information to perform simple action classification tasks [23]. Later, more advanced technologies such as deep learning were introduced for more accurate indoor sensing projects based on basic mmWave radar data. For instance, Wang et al. proposed a sequence-to-sequence (seq2seq) 3D temporal convolutional network with a self-attention method to estimate human hand pose [24]. Yi et al. utilized a multi-person detection model based on long short-term memory (LSTM) to determine human presence and localize their positions using a single 60GHz mmWave radio [25]. Wu et al. pioneered the work of segmenting human silhouettes from millimetre-wave RF signals by locating human reflections in radar frames and extracting features from surrounding signals with a human detection module and from aggregated frames with an attention-based mask generation module [26].

Although these studies have been highly successful, they have focused more on changes in the structure of deep learning networks and neglected how to extract more valid features from the radar signal. Given the inherent limitations of mmWave radar data, this oversight may result in the loss of valuable information during processing. Therefore, in this paper, we pay special attention to the position information contained in the radar signals, which is long unnoticed but rich in information. The positional feature is extracted and combined with traditional feature to depict human body poses.

II-B MmWave based Human Skeletal Pose Estimation

Since the application demands both privacy and precision, the advent of mmWave radar technology has markedly expanded the horizons of human pose estimation and recognition. A notable research direction in human pose estimation is the recognition of skeletal pose. The advantage of human skeletal pose estimation lies in its ability to provide a simplified but informative representation of human pose and movements, facilitating efficient and accurate recognition of human activity. Skeletal pose prediction derived from mmWave radar signals features critical spatial and temporal information about human activity without the privacy concerns associated with optical imaging.

RF-Pose [27] is one of the pioneering works that first considered human skeleton reconstruction. Zhao et al. proposed a deep network from FWCM signals to estimate coarse parts of the human skeleton under the supervision of visual information. They subsequently improved the model to successfully predict 14 human joints, including the head, neck, shoulders, elbows, wrists, hips, knees, and feet. RF-Pose and its follow-up work utilized expensive customized hardware systems. Nowadays, more studies are based on low-cost mass-produced industrial millimeter-wave radar and have demonstrated good performance. Ding et al. presented a kinematic-constrained learning architecture that incorporates kinematic constraints with neural network learning for skeleton estimation based on range-Doppler heatmaps [28]. Kong et al. performed a convolution operation on the Range-Doppler Profile to detect the corresponding ranges and designed a two-stream deep learning architecture to extract body shape and motion features to predict skeleton joint coordinates and reconstruct body posture [29]. Cao et al. developed a model that incorporates part-level range-Doppler maps for individual body parts with local kinematic constraints and global constraints for reconstructing the human skeleton [30]. In this work, we also take advantage of human skeletal pose. Our extracted positional features based on probability maps are beneficial in predicting human key joints positions and predicting human posture.

III Proposed Methods

In this section, detailed descriptions of ProbRadarM3F for human skeletal pose estimation, built upon [16], with additional fusion of multi-format features from mmWave radar data, are provided. The structure of the model is illustrated in Fig. 2. After processing the raw signal from mmWave radar, features are extracted from the radar data through the FFT branch and the ProbPE branch. Multi-frame features in different formats from the two branches are fused. Finally, the coordinates of human skeletal joints are predicted from the fused features via a cross-self-attention mechanism.

III-A FFT Branch: Radar Data Processing and Range-Doppler-Angle Map Generating

The dataset used provides only unprocessed raw radar data acquired by the DAC1000 system. Therefore, the radar signal is pre-processed to convert the raw analogue-to-digital converter (ADC) data into a structured format, specifically a radar data cube, for efficient signal processing and analysis.

The raw ADC data, captured in binary format, is initially one-dimensional, representing interleaved ADC samples across multiple channels. It is structured by organizing the data into a two-dimensional matrix to separate the individual low voltage differential signaling (LVDS) channels. By segregating the samples from even and odd channels, the real and imaginary components of the complex signal are constructed, ensuring the retention of phase information necessary for subsequent processing tasks. Ultimately, the complex data is reformatted into a radar data cube to accurately reflect the sequence of radar data acquisition. This reorganization involves structuring the data into blocks corresponding to each chirp across multiple transmit (TX) and receive (RX) channels. The radar data cube accurately retains information about the spatial and temporal dimensions of the observed scene.

As shown in the FFT branch in Fig. 2, a comprehensive analysis is initiated by performing a 4-dimensional FFT (4D FFT) along the four axes: ADC samples, chirps, horizontal antennas, and vertical antennas as follows:

$$F(h,i,j,k)=\sum_{n=0}^{N-1}\sum_{m=0}^{M-1}\sum_{p=0}^{P-1}\sum_{q=0}^{Q-1}f(n,m,p,q)$$ (1)

$$\times e^{-j2\pi\left(\frac{hn}{N}+\frac{im}{M}+\frac{jp}{P}+\frac{kq}{Q}\right)},$$

where ${f(n,m,p,q)}$ represents the original input signal across ADC samples ${n}$, chirps ${m}$, horizontal antennas ${p}$, and vertical antennas ${q}$, and ${F(h,i,j,k)}$ denotes the results of the Fourier transform. ${N,M,P,Q}$ are the sizes of the dimensions, while ${h,i,j,k}$ are the indices in the frequency domain for the corresponding dimensions. This process yields detailed range-Doppler-azimuth-elevation maps. Additionally, a certain amount of chirps within a specific velocity range is uniformly sampled in the Doppler dimension to filter out irrelevant information in the radar signal. This strategy effectively eliminates extraneous signal components, thereby streamlining the dataset for enhanced processing efficiency.

The dataset utilized in this study provides data collected by two vertically placed radars. Due to the limited resolution and detection range in elevation angle directions of the adopted radars, the elevation information from each radar is averaged in the processing as follows:

$$F_{\mathrm{avg}}=\frac{1}{Q}\sum_{k_{q}=0}^{Q-1}F(k_{m},k_{n},k_{p},k_{q}).$$ (2)

While the azimuth information of the horizontally placed radar is retained, the azimuth information of the vertically placed radar is considered as elevation angle information for the overall data. Eventually, the range-Doppler-azimuth map from the horizontal radar and the range-Doppler-elevation map from the vertical radar are obtained. To combine dynamic information across varying chirps within the range-Doppler-angle maps, the features from multiple chirps are merged by employing the M-Net [31]. The M-Net module utilizes a neural network to fuse different chirp information within a frame, instead of the traditional method, to solve for Doppler velocities, thereby outputting the merged features of the frame.

III-B ProbPE Branch: Probability Map Generating and Positional Encoding

As shown in Fig. 2, in the ProbPE branch, the radar data cube is initially subjected to a two-dimensional FFT (2D FFT). The FFT is executed in the fast-time dimension (the dimension of digitized chirp samples) to obtain range information. Due to the significant impact of velocity frequency effects in the slow-time dimension, where multiple frames correspond to the same range unit, a FFT is also conducted in this dimension to obtain the Doppler frequency.

To minimize irrelevant information that could increase computational load or interfere with feature extraction, a constant false-alarm rate (CFAR) method [32] is employed. This method is crucial in distinguishing between targets and interference noise based on intensity differences. A two-dimensional CFAR (2D-CFAR) is applied to range-Doppler maps to select the elements containing more valid information. In the 2D-CFAR detection process, targets are detected as accurately as possible to reduce false alarms. Guard cells and reference cells are introduced near each detection cell, as shown in the lower left corner of Fig. 3. Cell Averaging CFAR (CA-CFAR) is a common variant of CFAR, which uses the following formula to calculate the threshold for each cell under examination:

$$T=\alpha\cdot\frac{1}{N}\sum_{i=1}^{N}X_{i},$$ (3)

where ${T}$ represents the threshold for the cell, ${X_{i}}$ represents the amplitude values of the reference cells surrounding the cell under test, ${N}$ represents the number of reference cells considered, and ${\alpha}$ is a scaling factor determined by the desired false alarm rate. If the intensity of the target surpasses that of interference noise, as ${I_{\mathrm{target}}>I_{\mathrm{noise}}\times T}$, it indicates the presence of the target.

Subsequently, probability maps are derived from the filtered range bins, which serve as a foundational step in understanding the spatial distribution of potential targets. The concatenation set for the filtered range bins from the two radars is taken separately. The phase differences among various antennas yield distinct spikes, indicating frequency components from different directions. Angle information is obtained by performing FFT on target data from distinct virtual antennas. Angle FFT is executed to derive azimuth from the horizontal radar and elevation from the vertical radar. An averaging calculation in the Doppler dimension is performed because the focus at this point is on probability and position information. For the non-overlap** range bins filtered out from two radars, the Doppler dimensions are averaged directly to fill the corresponding range-angle vectors, facilitating the subsequent multiplication operations. Thus, range-azimuth and range-elevation vectors from the two radars are extracted, respectively. These vectors contain information on the radar’s intensity at specific ranges and angles, effectively characterizing the probability of a target’s presence at a given location. Consequently, radar probability maps are constructed, as illustrated in Fig. 3.

Figure 3: An illustration of probability map generation. It starts by filtering out the range bin through 2D CFAR on the range-Doppler heatmaps. Angle information is then extracted from the selected range, producing range-azimuth and range-elevation vectors. These vectors are normalized, transposed, and multiplied to create probability maps, indicating the likelihood of the target appearing at a specific position. The probability map serves as a guide for the positional encoding method.

The normalization of these vectors is performed to ensure uniformity in the data. Following normalization, the range-azimuth and range-elevation vectors at identical ranges are transposed and multiplied to generate range-azimuth-elevation probability maps:

$$P_{\mathrm{rae}}(r,\theta,\phi)=\hat{V}_{\mathrm{ra}}(r,\theta)^{T}\cdot\hat{V}_{\mathrm{re}}(r,\phi),$$ (4)

where ${P_{\mathrm{rae}}(r,\theta,\phi)}$ represents the probability map indicating the likelihood of detecting a target at range ${r}$, azimuth ${\theta}$, and elevation ${\phi}$. ${\hat{V}\mathrm{ra}(r,\theta)}$ represents the normalized range-azimuth vector, and ${\hat{V}\mathrm{re}(r,\phi)}$ denotes the normalized range-elevation vector.

These probability maps not only indicate the probability of detected elements but also contain significant geometric information, suggesting further exploitable position data. In advanced deep learning methodologies, the Transformer [33] enhances spatial perception through a sine coding formula for positional encoding. DETR [34] incorporates the Transformer model into target recognition, applying sine positional encoding to 2D images.

In our project, for each range $r$, the azimuth-elevation probability map can be treated as a two-dimensional representation. Therefore, sine positional encoding is introduced to extract positional information from the maps. The sine encoding formula is as follows:

$${PE}^{r}_{\left(\mathrm{{pos}}_{\theta},2i\right)}=\sin\left(\mathrm{pos}_{\theta}/10000^{\frac{2i}{d}}\right),$$ (5)

$${PE}^{r}_{\left(\mathrm{{pos}}_{\theta},2i+1\right)}=\cos\left(\mathrm{pos}_{\theta}/10000^{\frac{2i}{d}}\right),$$ (6)

$${PE}^{r}_{\left(\mathrm{pos}_{\phi},2i\right)}=\sin\left(\mathrm{pos}_{\phi}/10000^{\frac{2i}{d}}\right),$$ (7)

$${PE}^{r}_{\left(\mathrm{pos}_{\phi},2i+1\right)}=\cos\left(\mathrm{pos}_{\phi}/10000^{\frac{2i}{d}}\right),$$ (8)

where $(\mathrm{pos}_{\theta},\mathrm{pos}_{\phi})$ represents the position being computed and $d$ represents the dimension of the positional encoding vector. This formula intricately encodes positional information by encoding each coordinate, $\mathrm{pos}_{\theta}$ and $\mathrm{pos}_{\phi}$, into a unique 32-dimensional vector through the application of sine and cosine functions. This approach captures the essence of spatial variations across the probability maps.

By integrating positional encoding with the original probability features, the model is endowed with an enhanced capacity to recognize and interpret spatial relationships, significantly improving its performance on subsequent tasks. The features are then fed into the subsequent feature extraction module, which aligns with the structure of the other branch.

III-C Multi-format Feature Fusion and Estimation Head

The features obtained from both branches are based on single-frame radar data. During human body movement, both previous and subsequent frames can be used as a reference for the current frame’s action information. To better utilize temporal information, the multi-frame information is jointly processed, considering the inherent temporal continuity of human motion. Data from multiple frames before and after the target frame are integrated, thus harnessing richer temporal features.

To integrate multi-format spatio-temporal features, multiple 3D convolutional layers are used to aggregate information. These 3D convolutional blocks help extract and consolidate features across both spatial and temporal dimensions. For each spatial scale, a 3D convolution block aggregates the residual temporal information, yielding three-layer encoded features. In addressing the challenge of integrating multi-scale features from distinct processing branches, the output dimensions of encoded features across layers are standardized. This uniformity facilitates the direct summation of positional and probability features with features from the other branch. Unlike the conventional concatenation method, this direct addition method preserves spatial coherency, particularly for positional features. The fused features are obtained as follows:

$$F_{f}^{i}=F_{1}^{i}+F_{2}^{i}\quad(i=1,2,3),$$ (9)

where $i$ indicates the layers after 3D convolution. $F_{f}$ represents the fused feature of each layer, and $F_{1}$, $F_{2}$ represent features from two branches.

Following the pioneering work [16], the layer features obtained by summing go through a cross- and self-attention decoding module, specifically designed to enhance human pose detection by leveraging contextual information across and within the frames captured by both horizontal and vertical radar systems. The core consists of multiple decoder layers, each designed to handle features at varying scales and complexities. At each scale, the network combines basic 2D blocks for initial feature transformation and flattens to get an attention map in each attention system. As shown in the lower right corner of Fig. 2, the cross-attention mechanism combines the same layer of features from both radars. In the cross-attention mechanism, the features from the horizontal radar are used as key and value, and the features from the vertical radar as queries. The roles of key, query, and value are then switched to perform the same operation. Since the horizontal and vertical radar capture features in different aspects, i.e., azimuth and elevation data, a residual connection is crucial to avoid directly correlating these features and to ensure training stability. In addition, all keys, queries, and values in the self-attention system are from the same radar to enhance the internal structure of individual features. There are no skipped connections to generate self-participating residual features.

The predicted keypoint heatmaps are obtained through the attention decoding module from the multi-layer features. Knowing the location of certain keypoints can help in predicting the location of other keypoints. Therefore, similar to [16], the pose refinement module of the Graph Convolutional Network (GCN) is used to refine the keypoint heatmap [35]. A 3-layer GCN is used to perform feature propagation and inference, refining the keypoint predictions using the mutual position information of the physical connections between keypoints. The generated new heatmap is used as a common reference to generate the final keypoint locations. There is a mismatch between the coordinate systems for radar signals and keypoint heatmaps, making it impossible to directly use keypoint coordinates to locate radar features.

To facilitate the end-to-end training of the model, a strategy of imposing a pixel-wise binary cross-entropy loss on both the initial keypoint heatmaps and the GCN-refined keypoint heatmaps is employed. The objective function is formulated as follows:

$$L=L_{\mathrm{bce}}(\hat{H},G)+L_{\mathrm{bce}}(H,G),$$ (10)

where ${H}$ and ${\hat{H}}$ represent the initial keypoint heatmaps and refined keypoint heatmaps, respectively, and ${G}$ represents the generated groundtruth keypoint heatmaps based on Gaussian distribution. To illustrate, the binary cross-entropy loss $L_{\mathrm{bce}}(H,G)$ is defined by the equation:

$$L_{\mathrm{bce}}(H,G)=-\sum_{c,i,j}G_{c,i,j}\log(H_{c,i,j})$$ (11)

$$+(1-G_{c,i,j})\log(1-H_{c,i,j}),$$

where $c,i,j$ is derived from the channel, width and height of the joint prediction heatmaps $H$ and groudtruth heatmaps $G$, which follow Gaussian distribution.

IV Experiments and Results

Figure 4: Visualisation of comparison in state-of-the-art approaches and our proposed model.

Figure 5: Visualisation of predicted keypoints in inaccurate condition.

IV-A Dataset and Evaluation

The HuPR dataset was chosen for our experiments [16]. This dataset acquires data from two identical radars. One radar sensor is rotated 90° in the antenna plane with respect to the other, with one radar focusing on the horizontal plane and the other on the vertical plane. Unlike some radar datasets that provide processed data in formats such as point clouds [36, 37, 38], heatmaps [39], etc., which lose the purity of radar data, HuPR provides raw radar ADC signals. This allows more possibilities for our data processing and feature extraction methods.

The HuPR dataset includes data from 235 acquisition sequences in an indoor environment. Each sequence contains RGB camera frames, horizontal radar frames, and vertical radar frames that are one minute in duration. The two radars and the camera are synchronized and configured to capture 10 frames per second (FPS), so each sequence has a set of 600 camera-radar-radar frames. In each sequence, one person performs a static action, a standing hand wave, and a walking hand wave. These poses imitate and demonstrate the basic movements of a person in a medical care scenario, such as performing arm and walking rehabilitation therapy. The dataset also provides ground truth generated by the human pose estimation network HRNet [40] from RGB frames.

In alignment with established norms in the field, our evaluation framework for 2D keypoints employs average precision (AP) metrics calculated across various levels of object keypoint similarity (OKS) [41]. The OKS is calculated using the following formula:

$$\mathrm{OKS}_{p}=\frac{\sum_{i}\exp(-d_{p_{i}}^{2}/2S_{p}^{2}\sigma_{i}^{2})\delta\left(v_{p_{i}}=1\right)}{\sum_{i}\delta\left(v_{p_{i}}=1\right)},$$ (12)

where $p_{i}$ denotes the ID of the targeted keypoint. $d_{p_{i}}$ represents the Euclidean distance between the predicted keypoint and the ground truth keypoint for the $i$th keypoint on person $p$. $S_{p}$ is the scale of the person based on its area, calculated from the ground truth box of the person. $\sigma_{i}$ is the normalization factor of the corresponding skeletal point, reflecting the influence of the current skeletal point on the whole. $\delta\left(v_{p_{i}}=1\right)$ indicates that the predicted keypoint $p_{i}$ is visible within the observation range.

This allows the evaluation of the model’s accuracy in detecting 2D keypoints across 14 critical human body positions, including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. To offer a nuanced understanding of the model’s performance, three distinct AP metrics are used: $AP^{50}$, $AP^{75}$, and $AP$. These metrics represent varying degrees of OKS stringency, with $AP^{50}$ and $AP^{75}$ indicating more lenient and stringent OKS constraints, respectively. The metric denoted as $AP$ calculates the mean average precision across a range of 10 OKS thresholds, specifically at 0.5, 0.55, and incrementally up to 0.95, providing a comprehensive overview of the model’s performance across a broad spectrum of precision requirements.

IV-B Implementation Details

We take 193 sequences from HuPR for training, 21 for validation, and 21 for test. The sequence are chosen as HuPR network, which is considered as baseline, for fair comparison of results.

We implement ProbRadarM3F using Python and Pytorch. The training of our network is processed on a single NVIDIA V100 GPU. The network is trained using the step decay learning rate strategy, with an initial learning rate of 0.00005 and a reduction of 0.999 times per 2000 iterations. The Adam is employed as optimizer, and we set the batch size to 24. Some other parameters in the experiment are set as follows. In CFAR processing, the settings of the guard unit and the reference unit are set to 16 and 8. The depth of positional encoding for the probability map is set to 32. The number of input frames is 8 for joint processing of multi-frame data. The convolutional of network contains basic blocks from ResNet [42]. The size of convolution kernel for 3D convolutional processing is $3\times 3\times 3$ and ReLU activation function is used. These parameters are selected based on optimal performance in experiments.

IV-C Results and Analysis

Experiments on the HuPR dataset were performed to evaluate the effectiveness of the proposed strategies. Table II compares the average precision values of our proposed methods with RF-Pose [27] and the baseline method [16]. ProbRadarM3F shows advantages in AP at various precision levels, illustrating that our method outperforms RF-Pose on the same dataset. Compared to our baseline method, which is state-of-the-art, our method achieves higher scores on every metric, especially in $AP$ and $AP^{75}$, with gains of 6.5$\%$ and 12.9$\%$, respectively. The increase in $AP^{75}$ is a stringent indicator of our method’s ability to accurately predict human keypoints.

The specific precision of each keypoint is detailed in Table I. Our method has improved precision in every joint keypoint compared to the state-of-the-art. The best performance is observed in the hip joints, where $AP$ reaches 92.9$\%$. The greatest improvement is seen in the shoulder joints, with an $AP$ increase of 8.2$\%$. Reflected signals from the torso joints are strong and contain significant information with little noise, resulting in more accurate predictions. However, estimating arm and wrist joints remains challenging due to richer and finer arm movements and the radar’s limitations in capturing information from small reflective surfaces away from the torso. Despite this, our work shows accuracy gains of 6.7$\%$ in the elbow and 5.9$\%$ in the wrist.

Fig. 4 illustrates the performance of our model under different types of actions in comparison. GT represents the ground truth keypoints generated from HRNet [40] that precisely follow the actions in RGB frames, while HuPR and Ours are the predictions of the baseline and our ProbRadarM3F for the same frame, respectively. As shown, the predicted keypoints mostly align with the ground truth. To present realistic results, some frames where the prediction is not accurate enough are displayed in Fig. 5. It can be seen that our model loses accuracy when the target moves quickly. Although it predicts the torso and head locations well, there is still room for improvement in the wrist and foot joints. This observation highlights a critical area for future improvement, particularly in enhancing the model’s sensitivity to high-motion extremities.

Therefore, analyzing both the overall and individual results, our work effectively improves the precision of mmWave radar-based human skeletal pose estimation. This demonstrates that the probability map-guided positional encoding method effectively mines information from radar signals that might have been previously overlooked.

TABLE I: Comparison of keypoint Accurate Precison

	$AP$
Model	Head	Neck	Shoulder	Elbow	Wrist	Hip	Knee	Ankle
RF-Pose [27]	61.0	65.3	52.5	16.1	6.3	73.5	65.7	62.0
HuPR [16]	77.5	81.9	70.3	45.5	22.3	88.1	82.2	73.1
Ours	81.1	83.6	78.1	52.2	28.2	92.9	88.1	75.8

TABLE II: Comparison of State-of-the-art Approaches and Our Proposed Model

Model	$AP$	$AP^{50}$	$AP^{75}$
RF-Pose [27]	41.4	82.9	37.0
HuPR [16]	63.4	97.0	74.0
Ours	69.9	98.5	86.9

IV-D Ablation Study

A series of ablation studies were conducted to evaluate the individual contributions of various components within our proposed framework, ProbRadarM3F. These studies were performed on the HuPR dataset under consistent training, validation, and testing set settings.

• To determine the impact of the ProbPE branch, the network was operated with only the FFT branch, excluding the integration of features from the positional encoding guided by probability maps. The experimental results are shown in Table III. In the absence of the ProbPE branch, the model achieved lower precision in keypoint prediction compared to the complete ProbRadarM3F model, particularly under stringent OKS constraints. Specifically, excluding the ProbPE branch resulted in a decrease of 3.8$\%$ in $AP$ and a more pronounced 7.7$\%$ reduction in $AP^{75})$. The importance of the ProbPE branch is evident, highlighting the significant role of position information in the model’s prediction performance.

• In addition to isolating the effect of the ProbPE branch as a whole, the influence of probability maps in the ProbPE branch was specifically assessed. The complete model conducts positional encoding guided by probability maps, which are pivotal in generating refined positional features. To quantify the contribution of these probability maps, an ablation experiment was conducted by excluding the probability map generation step and directly applying positional encoding to the data after DOA. As shown in Table III, precision improved compared to using the FFT branch independently. However, the absence of probability maps resulted in a noticeable decrease in performance, leading to a reduction of $2.3\%$ in $AP$ when compared to the complete model. The results demonstrate the importance of probability maps. Without the guidance of probability maps, positional encoding becomes less effective. Consequently, the probability maps are not merely auxiliary but integral to ensuring that the positional encoding maximally captures and utilizes the spatial context inherent in the radar signals. The probability maps enhance the model’s ability to accurately determine the probability of target presence at specific azimuth and elevation coordinates, which is crucial for accurately encoding positional information.

TABLE III: Ablation Study

Model	$AP$	$AP^{50}$	$AP^{75}$
Ours(without ProbPE Branch)	66.1	97.1	79.2
Ours(without Probability Maps Generation)	67.6	98.4	84.2
Ours(Complete ProbRadarM3F Model)	69.9	98.5	86.9

V Conclusion

In our paper, we have explored the use of mmWave radar for estimating human skeletal poses. This is an important component of smart sensor systems used in medical care IoT applications, particularly in addressing privacy concerns. We introduced ProbRadarM3F, a new model that uses millimeter-wave radar for human skeletal pose estimation. This model significantly improves the extraction and use of hidden information from radar signals. We also introduced the ProbPE branch, which generates probability maps based on estimating the likelihood of a specific position. These probability maps are then used to effectively extract position features using a positional encoding method. Additionally, we combined positional features with features obtained from the FFT branch to enhance the human body keypoint features produced by the model. Our experiments on the HuPR dataset demonstrated that ProbRadarM3F outperforms existing methods, indicating the effectiveness of our approach. Our results highlight that there is valuable information, such as positional data, present in the radar heatmap or even in the radar signal itself, which should not be considered redundant.