EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting

Abstract

Human activities are inherently complex, and even simple household tasks involve numerous object interactions. To better understand these activities and behaviors, it is crucial to model their dynamic interactions with the environment. The recent availability of affordable head-mounted cameras and egocentric data offers a more accessible and efficient means to understand dynamic human-object interactions in 3D environments. However, most existing methods for human activity modeling either focus on reconstructing 3D models of hand-object or human-scene interactions or on map** 3D scenes, neglecting dynamic interactions with objects. The few existing solutions often require inputs from multiple sources, including multi-camera setups, depth-sensing cameras, or kinesthetic sensors. To this end, we introduce EgoGaussian, the first method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We leverage the uniquely discrete nature of Gaussian Splatting and segment dynamic interactions from the background. Our approach employs a clip-level online learning pipeline that leverages the dynamic nature of human activities, allowing us to reconstruct the temporal evolution of the scene in chronological order and track rigid object motion. Additionally, our method automatically segments object and background Gaussians, providing 3D representations for both static scenes and dynamic objects. EgoGaussian outperforms previous NeRF and Dynamic Gaussian methods in challenging in-the-wild videos and we also qualitatively demonstrate the high quality of the reconstructed models.

1 Introduction

Human activities are inherently complex and performing simple household tasks involves numerous interactions with objects. For example, making a coffee in the morning involves multiple steps: taking a mug from a shelf, placing it under the coffee machine, pressing a button for the preferred type of coffee, and adding milk or sugar. Even this seemingly simple task includes various object interactions and movements. To better understand human activities and behaviors, it is important to be able to model these dynamic interactions with the environment. The recent availability of affordable head-mounted cameras [44, 53] and egocentric data [11, 20, 21, 38] offers a more accessible and efficient means to understand dynamic human-object interactions in 3D environments. Toward this goal, we tackle the challenging task of reconstructing 3D scenes and dynamic interactions of objects from RGB egocentric videos.

Most existing methods for modeling human-object interactions either focus on reconstructing 3D hand-object [16, 33, 61, 69] or human-scene interaction models [26, 1, 28, 71, 32, 76] or on map** 3D scenes [56]. These approaches often neglect dynamic interactions with objects, resulting in static representations with motion-induced artifacts, commonly known as the “ghost effect”. The few existing solutions often require inputs from multiple sources, including multi-camera setups [37], depth-sensing cameras [62], or kinesthetic sensors [25]. While these methods achieve 3D reconstruction, they do not consider changes caused by interactions and thus fail to capture the dynamics depicted in egocentric videos.

In this paper, we go beyond prior works to tackle the task of dynamic scene reconstruction from RGB egocentric videos. Our proposed method EgoGaussian simultaneously reconstructs 3D scenes and dynamically tracks 3D object motions within them. Our key insight is that the uniquely discrete nature of Gaussian Splatting makes it especially suitable for spatial segmentation, allowing objects to be trained separately from the background. Given that human activities involve continuous motion over time, we identify critical contact points in time and distinguish dynamic interactions from static captures that only contain camera movements. We propose a clip-level online learning pipeline that leverages the dynamic nature of human activities, allowing us to reconstruct the temporal evolution of the scene in chronological order and track rigid object motion.

To reconstruct the dynamic scenes from an egocentric video, EgoGaussian first obtains hand-object segmentation using an off-the-shelf method and derives camera poses through structure-from-motion. By leveraging the natural trajectories of interactions, we partition the input video into static and dynamic clips. The static clips are used to reconstruct the background scenes and initialize the shapes of the object that will be interacted with. Subsequently, we refine the object’s shapes and track their motion through the dynamic clips. We empirically show that EgoGaussian achieves better reconstruction of dynamic scenes than previous NeRF and Dynamic Gaussian methods. We quantitatively evaluate our method on two in-the-wild egocentric video datasets following the evaluation protocol for novel-view synthesis. We also qualitatively demonstrate the high quality of the reconstructed scenes and the tracked object shapes and their motion.

Our main contributions can be summarized as follows: 1) We present a novel method that accurately reconstructs 3D scenes and dynamic object motion within them from RGB egocentric videos. 2) We leverage the dynamic nature of interactions that consist of transitions between static and dynamic phases, which facilitates the reconstruction of the static scenes, the object shapes, and the tracking of their motion. 3) Through both qualitative and quantitative evaluation, we demonstrate that our method outperforms previous approaches and provides better 4D reconstruction that captures the dynamic object interactions. Our model and code will be released upon acceptance.

2 Related Work

Hand-Object Segmentation. Many works have studied hand-object interaction in egocentric vision from different aspects. One significant area of focus is segmentation, specifically obtaining image segmentation masks of hands and the objects they hold. Ren et al. [49] proposed a motion-based approach to robustly segment both hand and object using optical flow and domain-specific cues from egocentric video. Concurrent with the emergence of deep neural networks-based hand-object segmentation is the scaling-up of egocentric data that includes pixel-level annotations and involves diverse daily activities [11, 12, 20]. VISOR[13] annotates videos from EPIC-KITCHENS[11, 12] dataset and provides masks for 67k hand-object relations covering 36 hours of videos. EgoHOS [74] further introduces the notion of a dense contact boundary to explicitly model the interaction and a context-aware compositional data augmentation technique to generate semantically consistent hand-object segmentation on out-of-distribution egocentric videos. Cheng et al. [9] produces a rich, unified 2D output of interaction by converting predicted bounding boxes to segments with Segment Anything (SAM) [30]. Our method takes egocentric videos with hand-object segmentation masks as input and creates dynamic 3D models.

Hand-Object Reconstruction. Another highly related direction is to reconstruct the hand-object interaction, featuring 3D pose estimation for hands and objects. Recent works often jointly reconstruct hands and objects to favor physically plausible interactions [16, 33, 61, 69, 70, 78]. These approaches can be grouped into two categories. One assumes a known 3D object model and fits that model into 2D image [6, 10, 33, 55, 67]. For example, RHO [6] adapts an optimization-based approach that’s able to reconstruct hands and objects from single images in the wild, by leveraging 2D image cues and 3D contact priors to provide constraints. Recent works eliminate the need for a known 3D model and directly reconstruct 3d object shapes from the input [16, 48, 70]. However, they either require multiview input [48], specific hand-object interaction supervision [70], or can only reconstruct simple object shapes [16]. Current shape-agnostic methods struggle in in-the-wild scenarios [16, 78]. In contrast, our method does not require prior knowledge and obtains 3D object shapes through differentiable 3D Gaussian-based rendering.

Static Scene Modeling. In the past few years, the domain of static scene modeling has garnered considerable attention. Mildenhall et al. [42] introduce the groundbreaking Neural Radiance Fields (NeRF), which utilizes a large Multilayer Perceptron (MLP) to represent 3D scenes and renders via volume rendering technique. However, their method queries the MLP at hundreds of points for each ray, resulting in slow training and rendering speed. Additionally, the original NeRF’s performance can diminish in scenes with highly dynamic elements due to its static, volumetric nature. Therefore, some subsequent works have tried to enhance the quality by (1) mitigating existing problems, such as aliasing [3, 4, 8] and reflection [24, 58] (2) incorporating image processing [41, 39] (3) employing per-image transient latent codes [40, 51], and (4) introducing supervision of expected depth with point clouds [14, 65]. There also exist some other follow-up works aiming to improve the speed, for example, by caching precomputed MLP results [27, 72], employing well-designed data structures [7, 54], removing the neural network [18], or utilizing multi-resolution hash encoding [43]. Yet, most of these methods still use ray marching, which involves sampling millions of points and slows down real-time rendering. Recently, Kerbl et al. [29] propose a different approach in the modeling and rendering of complex static 3D scenes - 3D Gaussian Splatting (3DGS). They model static scenes with Gaussians whose position, opacity, shape, and color are learned through a differentiable splatting-based renderer, achieving real-time rendering speed.

Dynamic Scene Modeling. Motivated by the success of NeRF [42] in static scene modeling, numerous studies have adopted neural representations to model dynamic scenes. One strategy to extend 3D into 4D scenes is by using time stamps as an additional conditioning factor. [2, 64]. Another set of ’dynamic NeRF’ works [5, 17, 50, 52, 59] involves employing 4D space-time grid-based representations. Representing the 3D scene at a certain timestamp as a canonical space and then explicitly modeling deformation fields to warp 3D points into the canonical space is a common strategy as well [15, 34, 45, 47]. Another strategy is to combine the two approaches, using a conditional neural volume together with a deformation field [31, 46]. However, these methods all suffer from the same issues as static NeRFs, in that they require raymarching and despite advances in performance, still are not sufficiently fast for real-time rendering. Similarly, many dynamic extensions to 3D Gaussian splatting were also proposed [36, 37, 63, 68]. The most common approach is to learn for every timestep a set of deformations for each Gaussians. This can be done explicitly, or implicitly using a deformation which is evaluated for each Gaussian. This results in substantially faster training and rendering speed, with comparable levels of rendering quality. Although these methods result in decent quality renders, upon closer inspection all of them result in noticeably blurrier results than are possible with static reconstructions, especially when strong motion is involved.

3 Methodology

Figure 1 summarizes our method, EgoGaussian, for dynamic scene reconstruction from RGB egocentric videos. EgoGaussian first obtains camera poses and hand-object segmentation masks and the segmentation masks are further used to partition the videos into static and dynamic clips (Sec. 3.2). The static clips are used to initialize the static background and object shapes (Sec. 3.3), while the dynamic clips are used to track object motion and gradually refine their shapes (Sec. 3.4.

Figure 1: EgoGaussian Pipeline. Given an egocentric video input, our framework first estimates camera poses via structure-from-motion and obtains hand-object segmentation masks using an off-the-shelf approach. We also partition the video input into static and dynamic clips in the preprocessing step. The static clips are used to reconstruct the background scenes and initialize the shapes of the object that will be interacted with. Subsequently, we refine the object’s shapes and track their motion through the dynamic clips.

3.1 Preliminary: 3D Gaussian Splatting

We use 3D Gaussian Splatting (3D-GS) as our modeling structure because it provides an explicit 3D scene representation with a set of point-cloud-like 3D Gaussians. Each Gaussian is characterized by a position (mean) $\mathbf{\mu}$, a covariance matrix $\Sigma$, an opacity, and color features $\mathbf{c}$. The Gaussians are defined using the standard multivariate Gaussian distribution $G(\mathbf{x})=e^{-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^{T}\Sigma^{-1}(\mathbf{x}-\mathbf{\mu})}$, providing a flexible optimization framework and a compact 3D scene representation.

3D Gaussian Splatting utilizes a differentiable point-based $\alpha$-blending rendering to compute the color $C$ of pixel $\mathbf{x}_{p}$. Specifically, it adapts a typical neural point-based approach and blends $N$ ordered points overlap** the pixel: $C(\mathbf{x}_{p})=\sum_{i\in\mathcal{N}}\mathbf{c}_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right)$, where $\alpha_{i}$ is calculated by evaluating a 2D Gaussian with covariance $\Sigma$, projected from the 3D Gaussian, and then multiplied with its opacity; and $\mathbf{c}_{i}$ is the color of each Gaussian. The original 3D-GS implementation treats color as a directional appearance component represented via spherical harmonics (SH). For simplicity, we disable the view-dependent color by setting the maximum SH degree to $0$.

3.2 Data preprocessing

Camera pose estimation. For egocentric dataset where camera poses are already computed and made public, e.g., EPIC Fields [56] provides estimated camera poses for EPIC-KITCHENS [11], we will employ them directly; otherwise, we estimate camera poses ourselves. We will provide a frame filtering pipeline here to alleviate the computational load of running COLMAP on lengthy videos. COLMAP’s SfM also creates a sparse point cloud corresponding to the camera poses estimated, which we use as an initialization for 3D Gaussian Splatting.

Hand-object segmentation. We require pixel-level segmentation of hand-object interaction either provided [35] or estimated beforehand. The onset and offset frames of each hand-object interaction should also be estimated throughout the video. For the HOI4D dataset this is provided, whereas for the EPIC-KITCHENS dataset we use EgoHOS [74] to generate hand masks and Track-Anything model [66] for object masks and human body masks. Furthermore, these masks are dilated by 2 pixels for better robustness.

Video partitioning. We then partition the egocentric video into static and dynamic clips according to the onset and offset of interactions. We define static clips as ones where only the actor’s hands or body are moving, but objects are all static, while dynamic clips contain both actor and object motion.

3.3 Static clip reconstruction

As some other works have pointed out, 3D Gaussian Splatting tends to overfit to training views by generating excessive floaters when there are scene inconsistencies among 3D views [22]. In order to eliminate such inconsistencies, any objects that move at all in the scene should be identified and masked out to provide a purely static scene reconstruction. Currently moving objects can be identified using the masks mentioned in the previous section. We explain in the following sections how we also identify objects that have moved or will move, and we provide an illustration of this process in Figure 2

Figure 2: Static reconstruction pipeline. Given a static clip partitioned from egocentric video, we adopt a standard 3D Gaussian Splatting training schema first. Based on the trained Gaussians, we then train an object identity variable with respect to 5 object masks, so that object Gaussians can be detached from the background Gaussians.

Initial Static training. We first train a static version of the scene with each static clip. To do so, we use a set of $M$ observations/frames from this clip $S_{i}=\{\mathbf{I}_{s_{j}},\mathbf{H}_{s_{j}},\theta_{s_{j}}|$  $j=1,\dots,M\}$, where $\mathbf{I}_{j}$ is an input RGB egocentric frame, $\mathbf{H}_{j}$ is the binary hand/body mask where pixel value $=0$ represents body part and pixel value $=1$ is for rest of the frame, and $\theta_{j}$ is the corresponding camera parameters for frame $j$. We then follow the similar optimization pipeline as the original 3DGS, including pruning and densification. We use a masked version of their loss function:

$$\mathcal{L}=(1-\lambda)\mathcal{L}_{1}\left(\mathbf{I}_{\text{input}},\mathbf{I}_{\text{render}}\right)+\lambda\mathcal{L}_{\text{D-SSIM }}\left(\mathbf{I}_{\text{input}},\mathbf{I}_{\text{render}}\right),$$ (1)

with the gradients zeroed out according to the hand/body mask $\mathbf{H}$. Similar to SuGaR [23], after around 30K iterations, we append an additional entropy loss on the opacity $\alpha$ of Gaussians, i.e.

$$\mathcal{L}_{\text{entropy}_{\alpha}}=-\alpha\log(\alpha)-(1-\alpha)\log(1-\alpha),$$ (2)

as a way to enforce Gaussians to be either fully transparent or completely opaque and train for another 10K iterations while disabling pruning and densification. Instead, we prune the transparent Gaussians once at the end of this phase of training.

Object identity training. This produces a set of 3D Gaussians $\mathcal{G}$ reconstructing the scene captured by this static clip, which includes both the static background and any objects that might move during dynamic portions of the video. As previously mentioned, we require masks not only for currently moving objects, but objects that move at any time in the video. As these objects are static and not segmented during static clips, we instead use the masks from the closest adjacent dynamic frame, and copy them forwards and backwards in time. For example, for a static clip $S_{1}$ with $M$ frames between two dynamic ones where different objects are moved, masks for two different objects A and B should be determined for the first and last 5 frames of this clip respectively: $\{\tilde{\mathbf{O}}_{A,1},\dots\tilde{\mathbf{O}}_{A,5}\}$, $\{\tilde{\mathbf{O}}_{B,M-4},\dots\tilde{\mathbf{O}}_{B,M}\}$. Although this assumes that the camera moves minimally during these transitions, this works well in practice for all tested settings. An additional trainable parameter of label $l$ is then attached to each Gaussian and initialized to a very small value. This label can then be rendered similar to the RGB value as: $L(\mathbf{x}_{p})=\sum_{i\in\mathcal{N}}l_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right)$. This produces a segmentation mask $L_{render}$, upon which we can apply a binary cross entropy loss using the dynamic object masks $\tilde{\mathbf{O}}$ we previous discussed:

$$\mathcal{L}_{\text{BCEWithLogitsLoss}}=-\left[(1-\tilde{\mathbf{O}})\cdot\log(\sigma(\mathbf{I}_{\text{render, obj}}))+\tilde{\mathbf{O}}\cdot\log(1-\sigma(\mathbf{I}_{\text{render, obj}})))\right],$$ (3)

where $\sigma(X)=\frac{1}{1+e^{-X}}$ is the sigmoid function. This allows us to separate the Gaussians into object and background Gaussians based on thresholding $\sigma(l)$ and can be rendered to compute movable object masks.

Combination of static and dynamic clips. The previously mentioned static scene still includes objects that are only static for some parts of the video. However, parts of the background scene are obscured by these objects, and only become visible during dynamic clips. We therefore retrain the static scene using the movable object masks in order to mask out objects that move during any time in the video. This allows us to train a more complete static scene which includes portions of the background that are only visible during dynamic parts of the video.

3.4 Dynamic object modeling

Object pose estimation. The pre-trained and segmented set of Gaussians $\mathcal{G}_{\text{obj}}$ can be used not only to mask out the objects for purpose of background reconstruction, but also as an initial estimate of the object appearance. Then, drawing inspiration from the previous hand-object reconstruction works that use a differentiable renderer such as [6, 73], we estimate the object pose for each video frame in the dynamic clip. Specifically, we estimate for each frame $f_{j}$ a corresponding pose $\mathbf{p}_{j}$. We further decompose the pose $\mathbf{p}_{j}$ into a 3D translation vector $\mathbf{t}_{j}$ and a rotation matrix $\mathbf{R}_{j}$. Unlike previous dynamic Gaussian Splatting methods [63, 37], our method applies one set of transformation parameters to the whole collection as a whole, treating it as a single rigid object.

To ensure the transformation is rigid, any estimated $3\times 3$ rotation matrix must be $\mathbf{R}_{i}\in SO(3)$. We optimize the rotation $\mathbf{R}_{j}$ using the 6D continuous rotation representation proposed by [77] and translation $\mathbf{t}_{j}$. Hence for each input frame $f_{j}$, assuming we have the Gaussians representing the object’s state in the previous frame, $\mathcal{G}_{\text{obj},j-1}$, we apply the current trainable translation and rotation parameters to it: $\mathbf{X}_{\mathcal{G}_{\text{obj},j}}=\mathbf{X}_{\mathcal{G}_{\text{obj},j-1}}\cdot g(\tilde{\mathbf{R}}_{j})+\mathbf{t}_{j}$, where $\mathbf{X}_{\mathcal{G}_{\text{obj},j}}$ is the 3D coordinates of Gaussians $\mathcal{G}_{\text{obj},j}$ corresponding to the object’s state at frame $j$, $g(\cdot)$ is a function defined in [77] that transforms the 6D representation of rotation to a standard $3\times 3$ rotation matrix.

In order to prioritize learning the object pose, during this stage the learning rate on the parameters of the Gaussians such as position and color are all lowered by a factor of 10. In order to better constrain the shape of the object and prevent it from extending to outside of the mask, we also apply a silhouette loss onto the computed alpha value, which can be computed using the following equation: $A(\mathbf{x}_{p})=\sum_{i\in\mathcal{N}}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right)$ which effectively equates the RGB rendering equation without color. Our final object loss is then:

$$\mathcal{L}_{\text{obj}}=\mathcal{L}_{1}\left(\mathbf{I}_{\text{obj}},\mathbf{I}_{\text{render},\mathcal{G}_{\text{obj}}}\right)+\lambda\mathcal{L}_{2}\left(1-\mathbf{O},\mathbf{A}_{\text{render},\mathcal{G}_{\text{obj}}}\right),$$ (4)

where $\mathbf{I}_{\text{obj}}$ is cropped from $\mathbf{I}_{\text{input}}$ with the object mask $\mathbf{O}$, $\mathbf{I}_{\text{render},\mathcal{G}_{\text{obj}}}$ is rendered from $\mathcal{G}_{\text{obj}}$ so it contains the object only and black background, $\mathbf{A}_{\text{render},\mathcal{G}_{\text{obj}}}$ is the rendered alpha. We experimentally observe that $0.5$ is a suitable $\lambda$ value.

As we optimize the time-dependent pose parameters $\mathbf{t}_{j}$ and $\tilde{\mathbf{R}}_{j}$ one frame at a time, the Gaussians can easily overfit to the current frame. To address this, we train not only on the current frame, instead for every training iteration we train either on the current frame or a random previous frame with a probability of $0.5$. A problem is that during the Gaussian splatting optimization process, the opacity is frequently set to zero in order to prune floaters. However, this would produce a very noisy signal for the pose optimization. As such, we instead alternate between optimizing the rigid object pose, and densifying/pruning the Gaussians. In practice, for every dynamic frame, we first train for 4k iterations only optimizing the object pose without pruning or densification. We then freeze the object pose, and train another 4k iterations to better incorporate any new information from the object pose, and finally train another 4k iterations again only the pose without densification or pruning. For all 12k iterations, all Gaussian parameters such as color and position are continuously optimized. After iterating through the whole dynamic clip with $M$ frames, we obtain a coarse object pose for each frame $P=\{\mathbf{t}_{j},\mathbf{R}_{j}|j=1,\ldots,M\}$. Finally, we perform one final round of training on all frames jointly, with 6k iterations of pose estimation, 6k iterations of pruning/densifications, and finally another 6k iterations of pose estimation. This ensures that our object model is more equally fit onto all frames, rather than focused on the last seen ones. We provide an illustration of this process in Figure 3

Figure 3: Object pose estimation pipeline. We use a sequential pipeline with regularization from previous frames, allowing us to simultaneously estimate the object poses and reconstruct the object for each transformation we model, while preserving the correct geometry of the object.

Combining the static scene and dynamic object. As a final step, we combine the object model estimated in Section3.4 with the full background model in Section3.3. In practice, we note that at this stage, there are often floaters belonging to the background that obscure parts of the object. To eliminate these, we perform a final fine-tuning stage using all training frames and Gaussians. As we focus here on optimizing how the background and dynamic object interact and fit with each other, we again freeze the per-frame object pose. This produces then the full scene reconstruction, including per-frame data of the object pose.

4 Experiments

We compare our method with existing baselines for the dynamic scene reconstruction task. The goal of this task is to reconstruct both static 3D scenes and dynamic objects from RGB egocentric videos.

4.1 Novel View Synthesis

Datasets. We evaluate our method for dynamic novel view synthesis on in-the-wild videos extracted from the two commonly used egocentric video datasets.

HOI4D [35] is a large-scale egocentric video dataset of human-object interactions, where each video is $20$ seconds long. We randomly select $4$ videos involving active objects undergoing rigid transformations from this dataset. Among these, $2$ videos contain mostly translations, while the other $2$ include both translations and rotations. Compared to the original dataset, we downsample the image resolution to one-quarter of its original size, resulting in a resolution of $480$ x $270$ pixels.

EPIC-KITCHENS [11] is a large-scale dataset featuring in-the-wild egocentric videos of human-object interactions in native kitchen environments. We select $4$ video clips involving rigid transformations from the EPIC-KITCHENS dataset. Of these clips, 2 contain mostly translations, while the other 2 include both translations and rotations. Similar to the HOI4D dataset, we also downsample the image resolution to $455$ x $256$. The average length of these clips is $10.43$ seconds with $60$ FPS.

Evaluation protocol. For each video, we train on every second frame, and evaluate on the rest. Although we are able to correctly track the trajectory of the motion with fewer training frames, the fast motion that comes with egocentric videos leads to a significant loss of information when doing so. As such, we use a much denser concentration of training frames.

Metrics. To assess the performance of our model, we use the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM)[60], and the VGG-based perceptual similarity metric LPIPS [75]. As we aim to reconstruct the background and object without the actor, we mask out the arm and body of any actors within the scene when computing these metrics, and only evaluate the quality of the object and background reconstruction.

Baselines. We compare our model’s results with the current state-of-the-art (SOTA) methods, Deformable 3DGS [68] and 4DGS [63]. All the compared methods have the publicly available codebase. We can run the code as it is and report the results. All the numbers reported in the tables are benchmarked on a single NVIDIA RTX 6000 or TITAN Xp GPU.

Results. Table 1 and Figure 5 compares our method with existing dynamic Gaussian Splatting methods. We observe that EgoGaussian significantly outperforms existing methods on all evaluation metrics over two datasets. The two SOTA methods have very close performance on both datasets and all methods have better results on static evaluation frames than on dynamic ones.

Table 1: Comparison with SOTA dynamic Gaussian Splatting methods. we evaluate our method and two other SOTA baselines on the HOI4D and EK datasets. The best and second best results are bolded and italicized respectively.

	HOI4D								Epic-Kitchen
Method	Static				Dynamic				Static				Dynamic
	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
4DGS [63]	0.89	25.45	0.13		0.89	25.25	0.13		0.91	33.25	0.12		0.79	22.77	0.24
Def-3DGS [68]	0.90	25.96	0.11		0.89	25.39	0.12		0.93	33.73	0.11		0.81	23.38	0.22
Ours	0.96	31.52	0.07		0.95	30.29	0.09		0.94	34.22	0.10		0.88	28.30	0.17

4.2 Dynamic modeling

Figure 4: More qualitative results of the reconstructed dynamic scenes. The center figure shows the motion path, while the four F panels display individual trajectories of the reconstructed frames compared to the corresponding ground truth input frames.

We show in Figure 4 how we can reconstruct the scene including the object trajectory both using the original camera motion, or from a fixed point of view.

4.3 Ablation study

Without full scene fine-tuning. We show the necessity of fine tuning the static background and dynamic object as described in Section 3.4 jointly by comparing how our method performs when the background and object are only trained in isolation without fine-tuning on all frames or on the combined scene. As can be seen in Table 2 without full scene fine tuning, quality drops significantly.

Table 2: Ablation study of full scene fine tuning.

	HOI4D								Epic-Kitchen
Method	Static				Dynamic				Static				Dynamic
	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\uparrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
With Fine-tuning	0.96	31.52	0.07		0.95	30.29	0.09		0.94	34.22	0.10		0.88	28.30	0.17
Without Fine-tuning	0.87	23.90	0.15		0.86	23.03	0.17		0.78	21.58	0.24		0.79	21.04	0.24

Estimate poses with larger time gap. We show that our method is also able to estimate poses with larger time gaps, by training on every 6 frames instead of every 2 frames. As seen in Table 3, although PSNR drops, we are still nonetheless able to produce an accurate reconstruction. Note that the metrics were computed only on the dynamic object itself.

Table 3: Ablation study of step size over the object on object reconstruction.

	HOI4D								Epic-Kitchen
Method	Static				Dynamic				Static				Dynamic
	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
With Original Step Size	0.98	31.82	0.03		0.98	29.79	0.04		0.97	28.87	0.05		0.97	31.33	0.05
With Larger Step Size	0.98	28.61	0.03		0.98	28.35	0.03		0.94	24.01	0.06		0.92	26.68	0.08

5 Conclusion

In this paper, we introduce EgoGaussian, the first method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We present a novel Gaussian Splatting framework that leverages the dynamic nature of human activities and distinguishes between dynamic interactions and static captures. This approach allows us to reliably reconstruct the static background and track object motion during the dynamic phase while progressively refining the object shape. Our method significantly outperforms previous SOTA baselines evaluated on two in-the-wild egocentric video datasets. Qualitative results demonstrate the high quality of the reconstructed dynamic scenes.

Limitations and discussion. While our method successfully reconstructs dynamic scenes from egocentric videos, several challenges remain. First, as we constrain objects to be rigid, we cannot model elastic or stretchable objects. Furthermore, although we substantially improve on previous methods, notable image artifacts and floaters still remain. Finally, the iterative process of estimating the object pose significantly increases training time.

Potential negative societal impacts.. EgoGaussian shares most of the societal impacts brought by others 3D modeling methods including potential unsolicited 3D reconstruction of, for example, private properties. With EgoGaussian and its focus on rigid inanimate objects, there is a heightened risk of industrial espionage. However, this risk is mitigated by the requirement that individuals must have access to images of the specific place or product to create a 3D model. This limitation reduces the likelihood of such privacy and security breaches.

GT

Ours

4DGS [63]

Def-3DGS [68]

HOI Scene1

HOI Scene2

HOI Scene3

GT

Ours

4DGS [63]

Def-3DGS [68]

HOI Scene4

EK Scene1

EK Scene2

EK Scene3 (Fail)

Figure 5: Qualitative comparison with SOTA. We show reconstructions produced by our method and SOTA baselines (4DGS [63] and Deformable 3DGS [68]) from both HOI4D and EPIC-KITCHENS. Our reconstruction demonstrates more accurate reconstructions while baseline approaches fail to handle dynamic interactions.

Appendix A Appendix / supplemental material

A.1 NeRF-based method experiments

Datasets and evaluation protocol.

In this section we present some supplementary results led with the same dataset as in section 4.1 but with a different division of training and validation frames. For a single given video, We use for training nine out of ten frames from static parts and one out of six frames for dynamic parts.

Metrics and Baseline.

We compare our model to some well known NeRF-based architecture able to model dynamic scene. For it we use Neuraldiff [57] and T-NeRF [47]. We used the official implementation with default parameter for NeuralDiff and the implementation provided by [19] for T-NeRF. For T-NeRF we used the default setting for "nerfies" type dataset and the additional preprocessing was borrowed from [45].

Table 4: Results on dynamic NeRF methods. We evaluate two baselines on the HOI4D and EK datasets.

	HOI4D								Epic-Kitchen
Method	Static				Dynamic				Static				Dynamic
	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$		SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
NeuralDiff [57]	0.71	21.15	0.32		0.76	23.98	0.26		0.89	31.06	0.13		0.79	24.85	0.24
T-NeRF [47]	0.84	23.00	0.10		0.78	19.96	0.14		0.80	14.52	0.17		0.75	21.33	0.22

GT

NeuralDiff [57]

T-NeRF [47]

HOI Scene 2

HOI Scene 3

HOI Scene 4

Figure 6: Three different scenes from HOI4D, we show reconstructions produced by NeuralDiff [57] and T-NeRF [19]