A Medical Low-Back Pain Physical Rehabilitation Dataset for Human Body Movement Analysis

Abstract

While automatic monitoring and coaching of exercises are showing encouraging results in non-medical applications, they still have limitations such as errors and limited use contexts. To allow the development and assessment of physical rehabilitation by an intelligent tutoring system, we identify in this article four challenges to address and propose a medical dataset of clinical patients carrying out low back-pain rehabilitation exercises. The dataset includes 3D Kinect skeleton positions and orientations, RGB videos, 2D skeleton data, and medical annotations to assess the correctness, and error classification and localisation of body part and timespan. Along this dataset, we perform a complete research path, from data collection to processing, and finally a small benchmark. We evaluated on the dataset two baseline movement recognition algorithms, pertaining to two different approaches: the probabilistic approach with a Gaussian Mixture Model (GMM), and the deep learning approach with a Long-Short Term Memory (LSTM). This dataset is valuable because it includes rehabilitation relevant motions in a clinical setting with patients in their rehabilitation program, using a cost-effective, portable, and convenient sensor, and because it shows the potential for improvement on these challenges.

I Introduction

(a) torso rotation

(b) flank stretch

(c) hiding face

Figure 1: The three rehabilitation exercises in our dataset

Physical rehabilitation

Back pain is the 3rd common musculoskeletal disabling condition in the 45-65 years old population, and therapists consider that regular physical exercises is essential to alleviate back pain. 50 to 80% of the world population suffers at a given moment from back pain [1, 2], caused by accident, surgery, old age or unfit working habits. While active rehabilitation is considered more effective for physical rehabilitation than usual care [3, 4], therapists are concerned about the decreasing engagement of the patients throughout the months-long repetition of physical exercises [5], primarily attributed to the lack of supervision and monitoring of the patient performance.

To promote physical activity, games [6], mobile phone applications [7], virtual agents [8, 9] and robots [10, 11, 12, 13] can enhance the engagement of the patient. These approaches rely on body motion analysis of patients. Leveraging machine learning and computer vision, 3D physical body movements analysis has improved the performance of its algorithms. Unfortunately, their performance relies on the quality of the training dataset, and most human motion datasets have not been carried out in the medical context [14]: they lack medical annotation and variability representative of patients, and most contain data from only healthy subjects.

Challenges

Our main motivation for creating the presented dataset is to create an intelligent tutoring system capable of supervising rehabilitation sessions autonomously such as presented in [15], by providing instructions for each exercise of the program and real-time feedback how to improve the efficiency of the patient’s performance. With the aim of rehabilitation using human body movement analysis, intelligent tutoring systems (ITS) need to analyse complex full-body exercises that can involve several parts of the body but not necessarily all parts of the body. The assessment algorithm should be able to understand which parts are important, and what are the ranges of freedom that are acceptable. The ITS should encapsulate the tolerated variance for each joint and time frame. Thus, we have identified 4 challenges in rehabilitation movements analysis:

1. Rehabilitation motion assessment. The goal is to assess an observed motion sequence by detecting if the rehabilitation exercise is correctly performed or not.

2. Rehabilitation error recognition. The goal is to classify the observed error among a set of known errors, so as to explain and give feedback.

3. Spatial localization of the error. In addition to recognizing the error, the goal is also to identify which body part is responsible of the error.

4. Temporal localization of the error. The goal is to detect the temporal segment where the detected error occurred along the sequence.

While most rehabilitation datasets only provide annotations for the rehabilitation motion assessment, we propose a medical dataset of clinical patients carrying out low back-pain rehabilitation exercises. The dataset includes 3D Kinect skeleton positions and orientations, RGB videos, 2D skeleton data, and medical annotations to assess the correctness, label and timing errors of each movement part. The article also provides initial benchmarks with 2 movement analysis algorithms.

II Related Work

Capture system

A number of human activity datasets have been recently reported. They were mostly captured in two modalities: marker-based motion capture (MoCap) tracking and vision-based (marker-less) tracking [14]. On the other hand, vision-based technologies do not hinder movements. RGB-D cameras, such as the Microsoft Kinect, are affordable, require little calibration or setup. Moreover, the Kinect has been validated against standard motion capture systems despite a lower precision, and against other vision-based tracking algorithms, and wearable sensors [16, 17, 18, 19, 20, 21]. More recently, human pose estimation from RGB images [22, 23, 24, 25, 26, 27, 28] have also made impressive improvements, leveraging deep learning models and large datasets. Cameras can easily blend into patient’s homes or a clinical environment.They are affordable, require little calibration or setup. This makes them suitable for use in our clinical test where patients are recorded in daily sessions of 30 minutes. Thus, we provide data from both RGB cameras and the Kinect.

TABLE I: Comparison with other datasets (Participants column reports the total number of participants including the patients. NA = Not Available, annotators column report the labelling method : if the cases were recorded under instructions or if a medical or non medical human annotator labelled the data )

Dataset	Activities	Exercises	Nb exer.	Pat-ients	Par-tici-pant	EMG	RGB D JP JO A	MoCap /wearable	Nb record.	Annotations	Annotators
K3Da  [29]	clinical assess. of gait, balance	posture tests	13	0	54	NA	Kinect2 D, JP	NA	525	None	None
EMG Squat [30]	Therapy to protect the anterior cruciate ligament (ACL)	Squat	3	0	9	leg	NA	NA	81	Exercise type	instructions
HPTE [31]	Physiotherapy exercises at home	Shoulder and knee exercises	8	0	5	NA	Kinect1 RGB D	NA	240	Exercise type	instructions
UI-PRMD [32]	Physical therapy and rehabilitation	Whole-body exercises	10	0	10	NA	Kinect1JP JO	Vicon	1000	Correct/ Incorrect	instructions
TRSp [33]	Post-stroke physical rehab	Arm movements using a haptic robot tabletop	2	9	19	NA	Kinect1 JP JO	NA	190	Error label	2 Med
IRDS [34]	general physical rehabilitation	arms and legs	9	15	29	NA	Kinect2 JP	NA	2589	Exercise type position (sitting or standing), correctness label (correct, incorrect, unrecognisable)	2 Non Med
EmoPain [35]	Chronic low back pain	Physical exercises	7	22	50	back	multi-RGB A	IMU	unk.	pain expression & pain related mvt	8 Non Med + 6 Med
KIMORE [36]	low back pain	whole body	5	34	78	NA	Kinect2 RDB D JP JO	NA	1000	3 scores (quantify the accuracy of the subjects) for total error and body part	3 Med
Keraal	Posture and back pain rehabilitation	Upper-body exercises	3	12	21	NA	Kinect2 RGB, JP, JP	Vicon (Gr3)	2622	Error label + (Group1a,2a: body part, timespan)	2 Med

Medical Human Movement Dataset

Although several datasets contribute to research in human motion analysis, such as the Kinect datasets [37, 38, 39, 40] or the multi-sensor datasets [41, 42, 43], they can hardly be applied in the medical context, including physical rehabilitation.

The K3Da dataset [29] is the first Kinect based dataset in a healthcare setting based on common clinical assessments of gait and balance. Nevertheless, they did not specifically recruit patients and the data were not labelled with medical criteria. The HPTE dataset [31] records with a Kinect therapy movements for computerized monitoring at home of 8 shoulder and knee exercise movements. Again, the participants are healthy. The EMG Squat dataset [30] is restricted to EMG electrodes recordings of 3 exercises of lower limbs performed by 9 healthy participants. The UI-PRMD [32] proposes a dataset of 10 healthy subjects performing physical therapy recorded by a Vicon marker-based tracker and a Kinect. Moreover, the labeling only indicates if the execution is correct, but not the type of error or their timing.

Human Movement Dataset with Patients

Targeting low hack pain too, the EmoPain dataset [35], to study the effects of pain during physical rehabilitation, provides high-resolution face videos, audios, full body joint motions, and electromyographs (EMG) from back muscles. The labels include pain facial expressions and pain-related movements. Like K3Da, EmoPain is labeled for recognition of exercises or mental states, they do not address the challenges stated in Sec. I. The TRSP dataset [33] proposes a dataset of clinically relevant motions during robotic rehabilitation exercises focusing on strength exercises with a haptic robot tabletop, captured with a Kinect. Likewise, IRDS [34] provides Kinect data of patients doing general rehabilitation. In TRSP and IRDS, The data from both healthy and patients are assessed by clinicians with scores and error labels, so they can address the challenges 2 and 3. But the labels do detail which body part and timespan of the execution was wrong. The Kimore dataset [36] proposes 5 whole body exercises from patients and healthy subjects. It is the closest to ours : it also targets low-back pain and is labelled by medical annotators. The labels are both at the global and the body part level to address challenges 1, 2 and 3. However, Kimore does not yet provide with the temporal information of errors to address challenge 4.

In comparison with these datasets, our Keraal dataset has been recorded within a long-term rehabilitation program, targeting low back pain. Contrarily to EMG Squat, HPTE and K3Da, UI-PRMD, we have recruited rehabilitation patients and have data labelled by a doctor. Like the HPTE dataset [31], our work is in a long-term effort of enabling physiotherapy exercises at home and our data is extracted from a 4-weeks evolution of each patient. A full comparison can be read in Table I. Thus our Keraal dataset is the only benchmarking set with clinical patients and with labels provided by a physician for the four challenges : motion assessment, error recognition, spatial and temporal localization.

III Dataset and Framework

This section describes the protocol and rationale for how the Keraal dataset was created, the participants included, the hardware setup and the experimental protocol. The dataset and the code are available on http://nguyensmai.free.fr/KeraalDataset.html and https://github.com/nguyensmai/KeraalDataset.

Rehabilitation program

31 patients, aged 18 to 70 years, were recruited in the double blind study. This prospective, centrally randomized, controlled, single-blind and bi-centric study was conducted from October 2017 to May 2019. 12 patients suffering from low-back pain were included and were asked to perform each of the three predefined exercises the best they can from its demonstration. The details on this clinical trial, including the patient care, the rehabilitation sessions, the inclusion and exclusion criteria, the characteristics of the patients, the efficiency of the care have been reported in [15]. This study has received a Legal authorisation for clinical tests from the medical ethics board of the hospital at Brest (CHRU Brest). All subjects have given their informed consent to participate in the study.

III-A Exercises and errors

A list of three exercises have been chosen in conjunction with therapists as common rehabilitation exercises that are also used for low-back pain treatment, under the condition that they can be coached by an intelligent tutoring system using visual assessment. Illustrations of these exercises can be seen in Fig. 1 . The 3 exercises are centered on spine stretching: a left rotation of the trunk followed by a the same right rotation, a left and right lateral bending of the trunk and a breathing exercise with the upper limbs flexed 90°at shoulder and elbow.

A list of common errors was defined in conjunction with the experience of the therapists of CHRU Brest. Errors are illustrated in Fig. 2.

(a) torso rotation – error1:
Arms are not raised enough

(b) flank stretch – error1:
Opposite arm is not along the body

(c) hiding face – error1:
Arms are not raised enough

(d) torso rotation – error2:
The torso’s rotation is not sufficient

(e) flank stretch – error2:
Body is not tilted

(f) hiding face – error2:
Arms are not outspread enough

(g) torso rotation – error3:
The body is leaned on the side

(h) flank stretch – error3:
The above arm is not bent

(i) hiding face – error3:
Arms are not raised enough

Figure 2: Each column illustrates 3 errors for each exercise.

III-B Participants

The dataset contains data from three groups of participants:

• Group1 : Rehabilitation Patients :

  The data of the daily sessions of the 12 patients recruited is split into two subsets:

  • Group1a: 14 recordings per exercise among 6 patients were annotated as detailed in Sec.III-D.

  • Group1b: the remaining recordings of the 12 patients without annotation.

• Group2 : Healthy participants with Kinect V2 recordings : Six healthy adults performed the exercises after the same instructions. They are free to execute the exercise correctly or with errors. As for group1, the recordings of the 6 healthy adults are split into two subsets:

  • Group2a: 51 recordings per exercise were annotated as detailed in Sec. III-D.

  • Group2b: the remaining recordings of the 6 healthy adults without annotation.

• Group3 : Healthy participants : Three healthy adults performed correct execution of exercises and simulated the identified common errors described in Sec. III-D.

III-C Sensor system

Using the Microsoft Kinect V2 sensor, we obtained the RGB video with the skeleton drawn, and the skeleton joint positions and orientations information. From the RGB videos, we can also obtain additional estimation of joint positions and orientations using the human body keypoint detection libraries OpenPose [44] and Blazepose [25]. Moreover, as the Vicon system is considered the best system for precision, for Group3, we also recorded with MoCap using the Vicon system. For synchronisation purpose, the two systems were activated simultaneously.

Comparing the human pose estimation methods, [45] showed that OpenPose and BlazePose data lead to comparable performances of GMM to classify correct/incorrect exercises (challenge 1). Thus in the following, we report results with Kinect data.

III-D Dataset annotation

The videos collected from patients with the Kinect V2 skeleton drawn were annotated by a medical doctor in physiotherapy and a physiotherapist, using the Anvil video annotation research tool. The videos without blurred faces) are annotated at three levels related to the 4 challenges described in Sec. I. On a global evaluation level, an assessment is given as either correct or incorrect (Challenge 1). In the case of an incorrect error, they can indicate if the execution has no errors but finished before the end (label code $4$: incomplete) or the participant did not start the execution of the exercise (label code $5$ : motionless). On the error classification level, in the case of an incorrect movement, annotations first indicate whether the error is significant or small as well as the label of the error (Fig. 2) (Challenge 2). Moreover the body part causing the error is also indicated (Challenge 3). On a temporal level, the annotators can also indicate the time window where the error occurs (Challenge 4), and the same information as previously: whether the error is significant or small, the error label, and the body part causing the error. The annotation is carried out a frame level.

For challenge 1 to annotate a movement as correct or incorrect, we carried out an interannotator agreement analysis. The results show substantial agreement between the two medical annotators ( Cohen’s $\kappa$ = 0.63 and Krippendorff’s $\alpha$ = 0.62 [46],[47]).

III-E Dataset description

Our dataset is composed of :

• The therapist’s annotations indicate whether the execution is correct, the label of the error, the body-part causing the error and the temporal description of the beginning and ending timestamps of the error.

• Anonymised RGB videos.

• The positions and orientations of each joint of the Microsoft Kinect V2 skeleton.

• The 2D positions of each joint of the OpenPose, Alphapose and Blazepose skeletons in the COCO pose or the 33 3D landmarks output format.

• the positions and orientations of each joint of the Vicon skeleton.

We have summarized the data and the number of recordings available per participant group in Table II.

TABLE II: Available recording modalities per group and the format of the data

Gr.	Annotation	RGB videos	Kinect	Openpose/Blazepose	Vicon	Nb rec
1a	xml anvil : err label, bodypart, timespan	mp4, 480x360	tabular	dictionary	NA	249
1b	NA	mp4, 480x360	tabular	dictionary	NA	1631
2a	xml anvil : err label, bodypart, timespan	mp4, 480x360	tabular	dictionary	NA	51
2b	NA	mp4, 480x360	tabular	dictionary	NA	151
3	error label	avi, 960x544	tabular	dictionary	tabular	540

The Keraal dataset has a total of 2622 recordings, with 1881 recordings of patients and 741 recordings of healthy participants. Therefore, our number of patients recording already outnumbers the total number of recordings of all the other datasets, but the IRDS dataset.

IV Benchmarks of the Keraal Dataset

IV-A Two algorithms

In order to evaluate the dataset, we propose to assess the performance of human motion analysis algorithms in the context of rehabilitation. These baseline performances can serve as reference for future comparison with more sophisticated approaches. The first approach is a probabilistic approach uses a Gaussian Mixture Model (GMM) on a Riemannian Manifold [48, 49] . The second method is a deep learning-based approach employing a LSTM [50] as applied for motion analysis in [51, 52] as the main part of its architecture. We use a 3-layers LSTM with tanh activation followed by dropout layer as the core of two different architectures employed for rehabilitation motion assessment and rehabilitation motion recognition. For both configurations, we use the Adam optimizer with a learning rate of $0.01$ and a batch size of $32$ on a simple laptop. These two approaches are evaluated and compared for the challenges of rehabilitation motion assessment (Challenge 1) and rehabilitation motion recognition (Challenge 2). Position and orientation features of the upper body of the Kinect skeleton data are used in the following. For rehabilitation motion assessment, we employ our LSTM model within an autoencoder architecture in order to assess movements as correct or incorrect (Fig. 3(a)). For rehabilitation motion error classification, we add a fully-connected layer with softmax activation to train over $1000$ epochs (Fig. 3(b)).

(a) Architecture of the LSTM-autoencoder

(b) Architecture of the LSTM classifier

Figure 3: LSTM models for assessment and classification

IV-B Rehabilitation motion assessment

The training data correspond to healthy subjects’ correct demonstrations (Group2 and Group3 cf. III-B), while testing data are patients’ performances acquired during rehabilitation sessions. and labelled (Group 1a cf. III-B). The unlabelled data from patients (Group 1b) can be used for unsupervised or semi-supervised learning.

We compare the ability of the two baselines to detect incorrect motion sequences while only correct demonstrations are available during training. After determining by grid search the best threshold values for classification, Fig. 5 shows the detection results for the best F1-score of the GMM baseline and LSTM baseline, respectively. While colors represent normalized values, we left absolute values within the matrices so as to emphasize that classes are imbalanced.

(a) GMM torso rotation

(b) GMM flank stretch

(c) GMM hiding face

(d) LSTM torso rotation

(e) LSTM flank stretch

(f) LSTM hiding face

Figure 5: Confusion matrix of error detection using SVM and LSTM autoencoder.

Most of the sequences are classified as correct, as shown Fig. 5. The majority of incorrect motion sequences are misclassified as correct, as highlighted in red color. This is a critical point showing that the proposed baselines may not be appropriate for rehabilitation motion analysis as it does not allow patients to improve their performance. This suggests that selecting a lower threshold may allow the approaches to correctly detect errors with the cost of also misclassifying correct sequences as incorrect. Depending on the requirement, a trade-off can be chosen. To further facilitate the choice, we show in Fig. 7 the evolution of the number of true positives (correct motion) and true negatives (incorrect motion) with respect to the thresholds for both baselines.

Further, [53] studied the impact of the amount of training data on this performance for GMM and Spatio-Temporal Graph Convolutional Networks with this dataset.

(a) GMM torso rotation

(b) GMM flank stretch

(c) GMM hiding face

Figure 6: GMM baseline : true positives and true negatives evolution with respect to different thresholds employed to evaluate a movement as correct or incorrect.

(a) LSTM torso rotation

(b) LSTM flank stretch

(c) LSTM hiding face

Figure 7: LSTM baseline : true positives and true negatives evolution with respect to different thresholds employed to evaluate a movement as correct or incorrect.

IV-C Rehabilitation motion error classification

For the challenge of error classification, we aim to compare the two baselines for supervised classification. The goal is to classify an observed exercise among 4 classes corresponding to correct, error1, error2 and error3, for each exercise separately. We evaluate the two baselines: 1) GMM-based features combined with a SVM classifier and 2) LSTM classifier. These two approaches are compared in term of classification accuracy. Classification results of GMM-based features combined with SVM classifier is reported in TableIII. For the LSTM classifier, as it may depend on initialization, we run the experiment $10$ times and report mean accuracy with standard deviation. The best accuracy among the 10 runs is also reported.

TABLE III: Accuracies of GMM-based SVM classifier and the LSTM classifier.

	GMM & SVM	LSTM    |
Exercise	Accuracy	Accuracy (mean$\pm$std)	Best Accur.
torso rotation	27.78 %	53.89$\pm$4.82 %	64.44 %
flank stretch	25.32 %	31.64$\pm$6.48 %	43.03%
hiding face	33.33 %	49.1$\pm$3.28 %	56.19%

The LSTM classifier obtains much better accuracy for the three exercises. However, the latter approach obtains a maximum accuracy of $64.44\%$ for the torso rotation exercise, showing that recognizing errors is still quite challenging.

(a) SVM torso rotation

(b) SVM flank stretch

(c) SVM hiding face

Figure 8: Confusion matrices of rehabilitation movements classification using GMM-based features with SVM. Rows full of $0.00$ mean that the corresponding error is not present in test set.

(a) LSTM torso rotation

(b) LSTM flank stretch

(c) LSTM hiding face

Figure 9: Confusion matrices for rehabilitation movements classification using LSTM. Rows full of $0.00$ mean that the corresponding error is not present in test set.

From the confusion matrices in Fig. 9, we indeed notice that the majority of correct sequences are not recognized using the GMM-based features with SVM classifier in comparison to the LSTM classifier which is more accurate to recognize correct sequences. Nevertheless, the LSTM classifier is not able to efficiently recognize performed errors for all exercises. This especially explains the lower accuracy obtained for the flank stretch exercise in contrast to the two other exercises, as reported in TableIII. Indeed, for this exercise, fewer test sequences are annotated as correct by the medical expert. Moreover, for the two other exercises (torso rotation and hiding face), we observe a lot of confusion for the error2. As described in Fig.2, this error is related to the insufficient medial rotation (yaw) of arms. While it shows that the proposed baseline is not able to handle this medial rotation, it can also be explained by the precision of the skeleton data provided by Kinect.

V Conclusion

In this paper, we formalized the challenges of automatic coaching of physical rehabilitation exercises from human pose movements as four folds, including motion assessment, error classification, spatial and temporal localization. To benchmark the performances of machine learning algorithms with respect to these four challenges, we introduced the Keraal dataset for human body movement analysis in the context of low-back pain physical rehabilitation. This dataset has been acquired during a clinical study where patients were performing rehabilitation exercises. The limitations of this article are : the dataset includes a small number of exercises and patients, and the baseline algorithms have only standard performance. However, the dataset has detailed annotations. It includes 3D skeleton sequences captured by a Kinect, color video sequences and 2D skeleton data estimated from videos. Moreover, medical expert annotations are associated to each patient’s performance for assessment of correctness, recognition of errors, spatio-temporal localization of errors. The performance of the two proposed baselines show that the dataset is challenging enough to be a benchmark. We believe it can serve the research community of various fields from computer vision, machine learning, robotics, virtual agents, physical medicine and bio-mechanics, in the long run, allowing the patients to have limited access to perform rehabilitation movements on a regular basis.

In addition, to further facilitate its use, we evaluate and compare two baseline motion analysis algorithms, pertaining to two different approaches, for the tasks of rehabilitation motion assessment and error recognition. While the experiment introduces how to use the dataset, it also demonstrates that the targeted tasks are still challenging. This suggests that specific and more accurate methods should be designed so as to deeply assess rehabilitation movements and differentiate slight errors from correct sequences. This latter investigation is part of our future work. Moreover, we also aim to investigate the challenges of spatial and temporal localization of errors along a motion sequence. Finally, we also want to extend the number of rehabilitation exercises considered by our dataset, as well as annotating more samples.

This dataset allows the development of better intelligent tutoring systems for physical rehabilitation and for physical exercises in general. Its social impact is to enable telemedecine and allow access to better exercising for those with difficult access to rehabilitation centers.

Appendix : Articulated figures

(a) Kinect

(b) OpenPose Coco

(c) 33 pose landmarks for BlazePose

(d) Vicon

Figure 10: Pose (Skeleton) output formats of the Kinect, OpenPose , BlazePose and Vicon.