-
OccFusion: Rendering Occluded Humans with Generative Diffusion Priors
Authors:
Adam Sun,
Tiange Xiang,
Scott Delp,
Li Fei-Fei,
Ehsan Adeli
Abstract:
Most existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion mod…
▽ More
Most existing human rendering methods require every part of the human to be fully visible throughout the input video. However, this assumption does not hold in real-life settings where obstructions are common, resulting in only partial visibility of the human. Considering this, we present OccFusion, an approach that utilizes efficient 3D Gaussian splatting supervised by pretrained 2D diffusion models for efficient and high-fidelity human rendering. We propose a pipeline consisting of three stages. In the Initialization stage, complete human masks are generated from partial visibility masks. In the Optimization stage, 3D human Gaussians are optimized with additional supervision by Score-Distillation Sampling (SDS) to create a complete geometry of the human. Finally, in the Refinement stage, in-context inpainting is designed to further improve rendering quality on the less observed human body parts. We evaluate OccFusion on ZJU-MoCap and challenging OcMotion sequences and find that it achieves state-of-the-art performance in the rendering of occluded humans.
△ Less
Submitted 29 June, 2024;
originally announced July 2024.
-
AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scale
Authors:
Keenon Werling,
Janelle Kaneda,
Alan Tan,
Rishi Agarwal,
Six Skov,
Tom Van Wouwe,
Scott Uhlrich,
Nicholas Bianco,
Carmichael Ong,
Antoine Falisse,
Shardul Sapkota,
Aidan Chandra,
Joshua Carter,
Ezio Preatoni,
Benjamin Fregly,
Jennifer Hicks,
Scott Delp,
C. Karen Liu
Abstract:
While reconstructing human poses in 3D from inexpensive sensors has advanced significantly in recent years, quantifying the dynamics of human motion, including the muscle-generated joint torques and external forces, remains a challenge. Prior attempts to estimate physics from reconstructed human poses have been hampered by a lack of datasets with high-quality pose and force data for a variety of m…
▽ More
While reconstructing human poses in 3D from inexpensive sensors has advanced significantly in recent years, quantifying the dynamics of human motion, including the muscle-generated joint torques and external forces, remains a challenge. Prior attempts to estimate physics from reconstructed human poses have been hampered by a lack of datasets with high-quality pose and force data for a variety of movements. We present the AddBiomechanics Dataset 1.0, which includes physically accurate human dynamics of 273 human subjects, over 70 hours of motion and force plate data, totaling more than 24 million frames. To construct this dataset, novel analytical methods were required, which are also reported here. We propose a benchmark for estimating human dynamics from motion using this dataset, and present several baseline results. The AddBiomechanics Dataset is publicly available at https://addbiomechanics.org/download_data.html.
△ Less
Submitted 16 May, 2024;
originally announced June 2024.
-
Countrywide natural experiment reveals impact of built environment on physical activity
Authors:
Tim Althoff,
Boris Ivanovic,
Jennifer L. Hicks,
Scott L. Delp,
Abby C. King,
Jure Leskovec
Abstract:
While physical activity is critical to human health, most people do not meet recommended guidelines. More walkable built environments have the potential to increase activity across the population. However, previous studies on the built environment and physical activity have led to mixed findings, possibly due to methodological limitations such as small cohorts, few or single locations, over-relian…
▽ More
While physical activity is critical to human health, most people do not meet recommended guidelines. More walkable built environments have the potential to increase activity across the population. However, previous studies on the built environment and physical activity have led to mixed findings, possibly due to methodological limitations such as small cohorts, few or single locations, over-reliance on self-reported measures, and cross-sectional designs. Here, we address these limitations by leveraging a large U.S. cohort of smartphone users (N=2,112,288) to evaluate within-person longitudinal behavior changes that occurred over 248,266 days of objectively-measured physical activity across 7,447 relocations among 1,609 U.S. cities. By analyzing the results of this natural experiment, which exposed individuals to differing built environments, we find that increases in walkability are associated with significant increases in physical activity after relocation (and vice versa). These changes hold across subpopulations of different genders, age, and body-mass index (BMI), and are sustained over three months after moving.The added activity observed after moving to a more walkable location is predominantly composed of moderate-to-vigorous physical activity (MVPA), which is linked to an array of associated health benefits across the life course. A simulation experiment demonstrates that substantial walkability improvements (i.e., bringing all US locations to the walkability level of Chicago or Philadelphia) may lead to 10.3% or 33 million more Americans meeting aerobic physical activity guidelines. Evidence against residential self-selection confounding is reported. Our findings provide robust evidence supporting the importance of the built environment in directly improving health-enhancing physical activity, in addition to offering potential guidance for public policy activities in this area.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Wild2Avatar: Rendering Humans Behind Occlusions
Authors:
Tiange Xiang,
Adam Sun,
Scott Delp,
Kazuki Kozuka,
Li Fei-Fei,
Ehsan Adeli
Abstract:
Rendering the visual appearance of moving humans from occluded monocular videos is a challenging task. Most existing research renders 3D humans under ideal conditions, requiring a clear and unobstructed scene. Those methods cannot be used to render humans in real-world scenes where obstacles may block the camera's view and lead to partial occlusions. In this work, we present Wild2Avatar, a neural…
▽ More
Rendering the visual appearance of moving humans from occluded monocular videos is a challenging task. Most existing research renders 3D humans under ideal conditions, requiring a clear and unobstructed scene. Those methods cannot be used to render humans in real-world scenes where obstacles may block the camera's view and lead to partial occlusions. In this work, we present Wild2Avatar, a neural rendering approach catered for occluded in-the-wild monocular videos. We propose occlusion-aware scene parameterization for decoupling the scene into three parts - occlusion, human, and background. Additionally, extensive objective functions are designed to help enforce the decoupling of the human from both the occlusion and the background and to ensure the completeness of the human model. We verify the effectiveness of our approach with experiments on in-the-wild videos.
△ Less
Submitted 31 December, 2023;
originally announced January 2024.
-
DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion
Authors:
Tom Van Wouwe,
Seunghwan Lee,
Antoine Falisse,
Scott Delp,
C. Karen Liu
Abstract:
Motion capture from a limited number of body-worn sensors, such as inertial measurement units (IMUs) and pressure insoles, has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. While a common goal across applications is to use the minimal number of senso…
▽ More
Motion capture from a limited number of body-worn sensors, such as inertial measurement units (IMUs) and pressure insoles, has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. While a common goal across applications is to use the minimal number of sensors to achieve required accuracy, the optimal arrangement of the sensors might differ from application to application. We propose a single diffusion model, DiffusionPoser, which reconstructs human motion in real-time from an arbitrary combination of sensors, including IMUs placed at specified locations, and, pressure insoles. Unlike existing methods, our model grants users the flexibility to determine the number and arrangement of sensors tailored to the specific activity of interest, without the need for retraining. A novel autoregressive inferencing scheme ensures real-time motion reconstruction that closely aligns with measured sensor signals. The generative nature of DiffusionPoser ensures realistic behavior, even for degrees-of-freedom not directly measured. Qualitative results can be found on our website: https://diffusionposer.github.io/.
△ Less
Submitted 28 March, 2024; v1 submitted 31 August, 2023;
originally announced August 2023.
-
Ten Steps to Becoming a Musculoskeletal Simulation Expert: A Half-Century of Progress and Outlook for the Future
Authors:
Scott D. Uhlrich,
Thomas K. Uchida,
Marissa R. Lee,
Scott L. Delp
Abstract:
Over the past half-century, musculoskeletal simulations have deepened our knowledge of human and animal movement. This article outlines ten steps to becoming a musculoskeletal simulation expert so you can contribute to the next half-century of technical innovation and scientific discovery. We advocate looking to the past, present, and future to harness the power of simulations that seek to underst…
▽ More
Over the past half-century, musculoskeletal simulations have deepened our knowledge of human and animal movement. This article outlines ten steps to becoming a musculoskeletal simulation expert so you can contribute to the next half-century of technical innovation and scientific discovery. We advocate looking to the past, present, and future to harness the power of simulations that seek to understand and improve mobility. Instead of presenting a comprehensive literature review, we articulate a set of ideas intended to help researchers use simulations effectively and responsibly by understanding the work on which today's musculoskeletal simulations are built, following established modeling and simulation principles, and branching out in new directions.
△ Less
Submitted 1 June, 2023;
originally announced June 2023.
-
Measuring Physical and Electrical Parameters in Free-Living Subjects: Motivating an Instrument to Characterize Analytes of Clinical Importance in Blood Samples
Authors:
Barry K. Gilbert,
Clifton R. Haider,
Daniel J. Schwab,
Gary S. Delp
Abstract:
Significance: A path is described to increase the sensitivity and accuracy of body-worn devices used to monitor patient health. This path supports improved health management. A wavelength-choice algorithm developed at Mayo demonstrates that critical biochemical analytes can be assessed using accurate optical absorption curves over a wide range of wavelengths. Aim: Combine the requirements for moni…
▽ More
Significance: A path is described to increase the sensitivity and accuracy of body-worn devices used to monitor patient health. This path supports improved health management. A wavelength-choice algorithm developed at Mayo demonstrates that critical biochemical analytes can be assessed using accurate optical absorption curves over a wide range of wavelengths. Aim: Combine the requirements for monitoring cardio/electrical, movement, activity, gait, tremor, and critical biochemical analytes including hemoglobin makeup in the context of body-worn sensors. Use the data needed to characterize clinically important analytes in blood samples to drive instrument requirements. Approach: Using data and knowledge gained over previously separate research threads, some providing currently usable results from more than eighty years back, determine analyte characteristics needed to design sensitive and accurate multiuse measurement and recording units. Results: Strategies for wavelength selection are detailed. Fine-grained, broad-spectrum measurement of multiple analytes transmission, absorption, and anisotropic scattering are needed. Post-Beer-Lambert, using the propagation of error from small variations, and utility functions that include costs and systemic error sources, improved measurements can be performed. Conclusions: The Mayo Double-Integrating Sphere Spectrophotometer (referred hereafter as MDISS), as described in the companion report arXiv:2212.08763, produces the data necessary for optimal component choice. These data can provide for robust enhancement of the sensitivity, cost, and accuracy of body-worn medical sensors. Keywords: Bio-Analyte, Spectrophotometry, Body-worn monitor, Propagation of error, Double-Integrating Sphere, Mt. Everest medical measurements, O2SAT
Please see also arXiv:2212.08763
△ Less
Submitted 6 January, 2023; v1 submitted 2 January, 2023;
originally announced January 2023.
-
An Experimental Double-Integrating Sphere Spectrophotometer for In Vitro Optical Analysis of Blood and Tissue Samples, Including Examples of Analyte Measurement Results
Authors:
Daniel J. Schwab,
Clifton R. Haider,
Gary S. Delp,
Stefan K. Grebe,
Barry K. Gilbert
Abstract:
Data-driven science requires data to drive it. Being able to make accurate and precise measurement of biomaterials in the body means that medical assessments can be more accurate. There are differences between how blood absorbs and how it reflects light. The Mayo Clinic's Double-Integrating Sphere Spectrophotometer (MDISS) is an automated measurement device that detects both scattered and direct e…
▽ More
Data-driven science requires data to drive it. Being able to make accurate and precise measurement of biomaterials in the body means that medical assessments can be more accurate. There are differences between how blood absorbs and how it reflects light. The Mayo Clinic's Double-Integrating Sphere Spectrophotometer (MDISS) is an automated measurement device that detects both scattered and direct energy as it passes through a sample in a holder. It can make over 1,200 evenly spaced color measurements from the very deep purple (300-nm) through the visible light spectrum into the near infrared (2800-nm). The MDISS samples measured have been also measured by commercial laboratory equipment. The MDISS measurements are as accurate and more precise than those devices now in use.
With so many measurements to be made during the time that the sample remains undegraded, mechanical and data collection automation was required. The MDISS sample holders include different thicknesses, versions that can operate at high pressure (such as divers may experience), and versions that can pump and rotate the measured material to maintain consistency of measurement. Although the data obtained are preliminary, they have potential to guide the design of new devices for more accurate assessments. There is an extensive "lessons learned" section. Please also see the companion report arXiv:2301.00938
△ Less
Submitted 4 January, 2023; v1 submitted 16 December, 2022;
originally announced December 2022.
-
Open source software for automatic subregional assessment of knee cartilage degradation using quantitative T2 relaxometry and deep learning
Authors:
Kevin A. Thomas,
Dominik Krzemiński,
Łukasz Kidziński,
Rohan Paul,
Elka B. Rubin,
Eni Halilaj,
Marianne S. Black,
Akshay Chaudhari,
Garry E. Gold,
Scott L. Delp
Abstract:
Objective: We evaluate a fully-automated femoral cartilage segmentation model for measuring T2 relaxation values and longitudinal changes using multi-echo spin echo (MESE) MRI. We have open sourced this model and corresponding segmentations. Methods: We trained a neural network to segment femoral cartilage from MESE MRIs. Cartilage was divided into 12 subregions along medial-lateral, superficial-d…
▽ More
Objective: We evaluate a fully-automated femoral cartilage segmentation model for measuring T2 relaxation values and longitudinal changes using multi-echo spin echo (MESE) MRI. We have open sourced this model and corresponding segmentations. Methods: We trained a neural network to segment femoral cartilage from MESE MRIs. Cartilage was divided into 12 subregions along medial-lateral, superficial-deep, and anterior-central-posterior boundaries. Subregional T2 values and four-year changes were calculated using a musculoskeletal radiologist's segmentations (Reader 1) and the model's segmentations. These were compared using 28 held out images. A subset of 14 images were also evaluated by a second expert (Reader 2) for comparison. Results: Model segmentations agreed with Reader 1 segmentations with a Dice score of 0.85 +/- 0.03. The model's estimated T2 values for individual subregions agreed with those of Reader 1 with an average Spearman correlation of 0.89 and average mean absolute error (MAE) of 1.34 ms. The model's estimated four-year change in T2 for individual regions agreed with Reader 1 with an average correlation of 0.80 and average MAE of 1.72 ms. The model agreed with Reader 1 at least as closely as Reader 2 agreed with Reader 1 in terms of Dice score (0.85 vs 0.75) and subregional T2 values. Conclusions: We present a fast, fully-automated model for segmentation of MESE MRIs. Assessments of cartilage health using its segmentations agree with those of an expert as closely as experts agree with one another. This has the potential to accelerate osteoarthritis research.
△ Less
Submitted 22 December, 2020;
originally announced December 2020.
-
Medical device surveillance with electronic health records
Authors:
Alison Callahan,
Jason A Fries,
Christopher Ré,
James I Huddleston III,
Nicholas J Giori,
Scott Delp,
Nigam H Shah
Abstract:
Post-market medical device surveillance is a challenge facing manufacturers, regulatory agencies, and health care providers. Electronic health records are valuable sources of real world evidence to assess device safety and track device-related patient outcomes over time. However, distilling this evidence remains challenging, as information is fractured across clinical notes and structured records.…
▽ More
Post-market medical device surveillance is a challenge facing manufacturers, regulatory agencies, and health care providers. Electronic health records are valuable sources of real world evidence to assess device safety and track device-related patient outcomes over time. However, distilling this evidence remains challenging, as information is fractured across clinical notes and structured records. Modern machine learning methods for machine reading promise to unlock increasingly complex information from text, but face barriers due to their reliance on large and expensive hand-labeled training sets. To address these challenges, we developed and validated state-of-the-art deep learning methods that identify patient outcomes from clinical notes without requiring hand-labeled training data. Using hip replacements as a test case, our methods accurately extracted implant details and reports of complications and pain from electronic health records with up to 96.3% precision, 98.5% recall, and 97.4% F1, improved classification performance by 12.7- 53.0% over rule-based methods, and detected over 6 times as many complication events compared to using structured data alone. Using these events to assess complication-free survivorship of different implant systems, we found significant variation between implants, including for risk of revision surgery, which could not be detected using coded data alone. Patients with revision surgeries had more hip pain mentions in the post-hip replacement, pre-revision period compared to patients with no evidence of revision surgery (mean hip pain mentions 4.97 vs. 3.23; t = 5.14; p < 0.001). Some implant models were associated with higher or lower rates of hip pain mentions. Our methods complement existing surveillance mechanisms by requiring orders of magnitude less hand-labeled training data, offering a scalable solution for national medical device surveillance.
△ Less
Submitted 3 April, 2019;
originally announced April 2019.
-
Artificial Intelligence for Prosthetics - challenge solutions
Authors:
Łukasz Kidziński,
Carmichael Ong,
Sharada Prasanna Mohanty,
Jennifer Hicks,
Sean F. Carroll,
Bo Zhou,
Hongsheng Zeng,
Fan Wang,
Rongzhong Lian,
Hao Tian,
Wojciech Jaśkowski,
Garrett Andersen,
Odd Rune Lykkebø,
Nihat Engin Toklu,
Pranav Shyam,
Rupesh Kumar Srivastava,
Sergey Kolesnikov,
Oleksii Hrinchuk,
Anton Pechenko,
Mattias Ljungström,
Zhen Wang,
Xu Hu,
Zehong Hu,
Minghui Qiu,
Jun Huang
, et al. (25 additional authors not shown)
Abstract:
In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector. Top participants were invited to describe their algorithms. In this work, we describe the challenge and present thirteen solutions that used deep reinforcement learning approaches. Many s…
▽ More
In the NeurIPS 2018 Artificial Intelligence for Prosthetics challenge, participants were tasked with building a controller for a musculoskeletal model with a goal of matching a given time-varying velocity vector. Top participants were invited to describe their algorithms. In this work, we describe the challenge and present thirteen solutions that used deep reinforcement learning approaches. Many solutions use similar relaxations and heuristics, such as reward sha**, frame skip**, discretization of the action space, symmetry, and policy blending. However, each team implemented different modifications of the known algorithms by, for example, dividing the task into subtasks, learning low-level control, or by incorporating expert knowledge and using imitation learning.
△ Less
Submitted 6 February, 2019;
originally announced February 2019.
-
Learning to Run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments
Authors:
Łukasz Kidziński,
Sharada Prasanna Mohanty,
Carmichael Ong,
Zhewei Huang,
Shuchang Zhou,
Anton Pechenko,
Adam Stelmaszczyk,
Piotr Jarosik,
Mikhail Pavlov,
Sergey Kolesnikov,
Sergey Plis,
Zhibo Chen,
Zhizheng Zhang,
Jiale Chen,
Jun Shi,
Zhuobin Zheng,
Chun Yuan,
Zhihui Lin,
Henryk Michalewski,
Piotr Miłoś,
Błażej Osiński,
Andrew Melnik,
Malte Schilling,
Helge Ritter,
Sean Carroll
, et al. (4 additional authors not shown)
Abstract:
In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course. Top participants were invited to describe their algorithms. In this work, we present eight solutions that used deep reinforcement learning approaches, based on algorithms such as Deep Deterministic Policy Gradient…
▽ More
In the NIPS 2017 Learning to Run challenge, participants were tasked with building a controller for a musculoskeletal model to make it run as fast as possible through an obstacle course. Top participants were invited to describe their algorithms. In this work, we present eight solutions that used deep reinforcement learning approaches, based on algorithms such as Deep Deterministic Policy Gradient, Proximal Policy Optimization, and Trust Region Policy Optimization. Many solutions use similar relaxations and heuristics, such as reward sha**, frame skip**, discretization of the action space, symmetry, and policy blending. However, each of the eight teams implemented different modifications of the known algorithms.
△ Less
Submitted 1 April, 2018;
originally announced April 2018.
-
Learning to Run challenge: Synthesizing physiologically accurate motion using deep reinforcement learning
Authors:
Łukasz Kidziński,
Sharada P. Mohanty,
Carmichael Ong,
Jennifer L. Hicks,
Sean F. Carroll,
Sergey Levine,
Marcel Salathé,
Scott L. Delp
Abstract:
Synthesizing physiologically-accurate human movement in a variety of conditions can help practitioners plan surgeries, design experiments, or prototype assistive devices in simulated environments, reducing time and costs and improving treatment outcomes. Because of the large and complex solution spaces of biomechanical models, current methods are constrained to specific movements and models, requi…
▽ More
Synthesizing physiologically-accurate human movement in a variety of conditions can help practitioners plan surgeries, design experiments, or prototype assistive devices in simulated environments, reducing time and costs and improving treatment outcomes. Because of the large and complex solution spaces of biomechanical models, current methods are constrained to specific movements and models, requiring careful design of a controller and hindering many possible applications. We sought to discover if modern optimization methods efficiently explore these complex spaces. To do this, we posed the problem as a competition in which participants were tasked with develo** a controller to enable a physiologically-based human model to navigate a complex obstacle course as quickly as possible, without using any experimental data. They were provided with a human musculoskeletal model and a physics-based simulation environment. In this paper, we discuss the design of the competition, technical difficulties, results, and analysis of the top controllers. The challenge proved that deep reinforcement learning techniques, despite their high computational cost, can be successfully employed as an optimization method for synthesizing physiologically feasible motion in high-dimensional biomechanical systems.
△ Less
Submitted 31 March, 2018;
originally announced April 2018.
-
ShortFuse: Biomedical Time Series Representations in the Presence of Structured Information
Authors:
Madalina Fiterau,
Suvrat Bhooshan,
Jason Fries,
Charles Bournhonesque,
Jennifer Hicks,
Eni Halilaj,
Christopher Ré,
Scott Delp
Abstract:
In healthcare applications, temporal variables that encode movement, health status and longitudinal patient evolution are often accompanied by rich structured information such as demographics, diagnostics and medical exam data. However, current methods do not jointly optimize over structured covariates and time series in the feature extraction process. We present ShortFuse, a method that boosts th…
▽ More
In healthcare applications, temporal variables that encode movement, health status and longitudinal patient evolution are often accompanied by rich structured information such as demographics, diagnostics and medical exam data. However, current methods do not jointly optimize over structured covariates and time series in the feature extraction process. We present ShortFuse, a method that boosts the accuracy of deep learning models for time series by explicitly modeling temporal interactions and dependencies with structured covariates. ShortFuse introduces hybrid convolutional and LSTM cells that incorporate the covariates via weights that are shared across the temporal domain. ShortFuse outperforms competing models by 3% on two biomedical applications, forecasting osteoarthritis-related cartilage degeneration and predicting surgical outcomes for cerebral palsy patients, matching or exceeding the accuracy of models that use features engineered by domain experts.
△ Less
Submitted 15 May, 2017; v1 submitted 13 May, 2017;
originally announced May 2017.
-
Self-tracking Energy Transfer for Neural Stimulation in Untethered Mice
Authors:
John S. Ho,
Yuji Tanabe,
Shrivats Mohan Iyer,
Amelia J. Christensen,
Logan Grosenick,
Karl Deisseroth,
Scott L. Delp,
Ada S. Y. Poon
Abstract:
Optical or electrical stimulation of neural circuits in mice during natural behavior is an important paradigm for studying brain function. Conventional systems for optogenetics and electrical microstimulation require tethers or large head-mounted devices that disrupt animal behavior. We report a method for wireless powering of small-scale implanted devices based on the strong localization of energ…
▽ More
Optical or electrical stimulation of neural circuits in mice during natural behavior is an important paradigm for studying brain function. Conventional systems for optogenetics and electrical microstimulation require tethers or large head-mounted devices that disrupt animal behavior. We report a method for wireless powering of small-scale implanted devices based on the strong localization of energy that occurs during resonant interaction between a radio-frequency cavity and intrinsic modes in mice. The system features self-tracking over a wide (16 cm diameter) operational area, and is used to demonstrate wireless activation of cortical neurons with miniaturized stimulators (10 mm$^{3}$, 20 mg) fully implanted under the skin.
△ Less
Submitted 4 March, 2015;
originally announced March 2015.
