A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets

3DGS creates a scene representation based on volumetric primitives, that each have the following set of parameters: position (also referred to as mean) $\mu$, covariance matrix $\mathbf{\Sigma}$ that, in practice, is decomposed into scale and rotation, opacity $\mathrm{o}$, and spherical harmonics (SH) coefficients to represent appearance, or view-dependent color.

The 3D primitives are projected to 2D screen space, and rasterized using $\alpha$-blending. The $\alpha$-blending weights are given as

with the projected Gaussian contribution on pixel $(x,y)$ given as:

where $\mu^{\prime}$ and $\mathbf{\Sigma}^{\prime}$ are the projected 2D mean and covariance matrix. The combined effect of converting SHs to per-view color values and $\alpha$-blending them recreates the appearance of the captured scene.

We create a tree-based hierarchy with interior and leaf nodes for each chunk. Every node is associated with a 3D Gaussian, which is either a leaf node coming from the original optimization, or a merged interior node. Our requirements for intermediate nodes are that they should: 1) Maintain the same fast rasterization routine as leaf nodes; 2) Represent the appearance of children as accurately as possible. We thus need to define intermediate nodes that are 3D Gaussians that have all the attributes of 3DGS primitives, i.e., mean $\mu$ (position), covariance $\mathbf{\Sigma}$, SH coefficients and opacity (see Sec. 3).

For mean and covariance, there is exhaustive literature related to Gaussians that we build on to define our merging process. Specifically, to merge $N$ Gaussian primitives with means $\mu_{i}^{(l)}$ and covariances $\mathbf{\Sigma}_{i}^{(l)}$ of level $l$ s.t. the 3D Kullback-Leibler divergence between the merged node and its children’s weighted distributions is minimized, we use (Goldberger and Roweis, 2004; Jakob et al., 2011):

We can similarly use the weighted average of the SH coefficients for the merged node using the same weights:

Gaussians in a typical scene may violate the above assumption of isotropy. In Sec. 5.1, we describe additional measures to address these cases and improve quality of higher LOD levels in such cases.

We now have a merging procedure for our 3D Gaussians. Given a set of 3DGS primitives, we first build an axis-aligned bounding box (AABB) Bounding Volume Hierarchy (BVH) top-down over them. We start from an AABB that encloses all Gaussian primitives, using $3\times$ their stored size to capture their extent (using the cutoff from the original paper (Kerbl et al., 2023)). The initial AABB is the root node of the BVH. To obtain the child nodes, we recursively perform a binary median split on the current node. First we project the means of all Gaussians in a group onto the longest axis of their bounding box. We then partition the group by assigning each primitive based on the position of its projected mean with respect to the median of all projections. The resulting BVH tree ensures that the children of each interior node will be spatially compact. We then compute the intermediate node Gaussians from their respective children, starting from the leaves and recursively merging up the tree.

Given a hierarchical tree structure for a 3DGS model, and a view $V$ we select a cut through the tree that will maximize rendering performance while maintaining visual quality. We first define the granularity $\epsilon(n)$ of a given hierarchy node $n$ as the projected size on the screen for a given view (Fig. 3(a)). Specifically, we use the bounding box over leaf Gaussians contained in a node, then take the largest dimension of the bounding box and use this to compute the projected size. We find the cut by identifying the nodes whose projected bounding box is smaller than a given target granularity $\tau_{\epsilon}$ on screen, e.g., 1 pixel. Based on the generation of the hierarchy nodes and its bounds, the AABB of a parent can never appear smaller than one of its children. This enables us to find the appropriate cut in linear time or, in a massively parallel setting, in constant time per node: if the bounds of a node $n$ fulfill the granularity condition, but its parent’s do not, then node $n$ is chosen for the given setting and included in the cut (green curve in Fig. 3(b)). Note that either all children or the parent are chosen by this logic.

A key element of any hierarchical rendering solution is the ability to allow smooth transitions between hierarchy levels. We achieve smooth transitions by interpolation on the individual Gaussian attributes. When a node is no longer the best fit for the current target granularity, it is replaced by its children by interpolating between parent and children Gaussian attributes. The cut returned by hierarchy selection with target granularity $\tau_{\epsilon}$ will contain nodes that (over-)fulfill the criterion. Interpolation weights are selected by evaluating the granularity $\epsilon(n)$ of each node in the cut, as well as the granularity $\epsilon(p)$, where $p$ is the parent of $n$. The interpolation weight $t_{n}$ is then found as:

Position, color and spherical harmonics can be interpolated linearly using the interpolation weights. For covariance, we found that interpolating scale and rotation separately results in superior appearance to covariance matrix interpolation, even if we use linear interpolation for rotations instead of the more expensive ”slerp”. However, Gaussians may have the same shape (covariance), even though they differ significantly in their rotation and scaling. For instance, a Gaussian that is scaled by $s$ along its $x$-axis will appear the same as a Gaussian that is scaled by $s$ along it’s $y$-axis and then rotated by $90^{\circ}$ around its z-axis (Fig. 4(a)). As a result, when directly interpolating their properties, an undesired rotation will occur which is visually disturbing.

To avoid this, during hierarchy generation we also perform orientation matching: starting from the root node, we recursively reinterpret the axes of orientation for each child Gaussian such that it minimizes the relative rotation between the child node and its parent, by exhaustive search. Using a non-exhaustive approach, e.g., matching eigenvalues, unnecessary rotations still occur.

Interpolation of opacity also requires careful treatment. At the start of the transition from a parent to the children, the children share all other attributes with the parent that are progressively interpolated to those of each individual child (Fig. 4(b)). However, we need to modify the falloff of the child nodes so that the resulting blending of the overlap** Gaussians gives the same result as the parent at the start of the transition.

Our top-down hierarchy construction results in a data-structure that works well (see Tab. 2). However, once constructed, the intermediate nodes can be rendered the same way as leaf 3DGS primitives, and thus can be optimized the same way. As a result, we can optimize the intermediate nodes of the hierarchy to improve the visual quality they can represent. This poses the question of how to perform this optimization between different scales. One solution would be to optimize the hierarchy by randomly selecting an input view and a downsampling factor (Barron et al., 2021); a lower resolution directly implies a different target granularity, and the corresponding hierarchy cut. However, this method has drawbacks that we illustrate in Fig. 5 and Fig. 7: when reducing resolution, there are high-frequency details that cannot be represented.

Instead of downsampling, we always render at full resolution during optimization, and simply choose random target granularities. This has the desired effect of optimizing nodes in many different cuts, while preserving visual details. This can be particularly useful when we have limited resources, where we can apply a more aggressive LOD strategy while maintaining better visual quality.

Concretely, we load the generated hierarchy and optimize it by randomly selecting one of the training views and a target granularity $\tau_{\epsilon}$ in a given target range at random. The target range $[\tau_{min},\tau_{max})$, and the training views chosen define the corresponding cuts selected. To achieve good quality sampling of all levels of the hierarchy, we sample the target granularities $\tau_{\epsilon}$ given a canonical random variable $\xi\in\left[0,1\right)$ using $\tau_{\epsilon}=\tau_{max}^{\xi}~{}\tau_{min}^{1-\xi}$.

Both cut selection and interpolation (Sec. 4) are used during this optimization. To optimize child and parent nodes simultaneously with smooth switching enabled, we must propagate gradients correctly to two layers of the hierarchy, optimizing child and parent nodes simultaneously. This requires gradients to propagate through the interpolation weights and the expression of $\alpha^{\prime}$ in Eq. 12. By optimizing multiple levels of the hierarchy, we tackle a more complex setting than 3DGS: Optimizing higher LOD levels may also degrade the quality of the leaves due to interpolation. To avoid this, we do not change the leaf nodes during optimization.

Note that, as observed by recent work (Yu et al., 2023), the original 3DGS method does not handle antialiasing properly; since hierarchies operate at different scales, correct anti-aliasing is required. We thus use the EWA filtering solution of Yu et al. (2023).

The hierarchy adds some overhead in terms of memory. More importantly, for the hierarchy optimization itself we want to avoid parent-child settings where the parent’s size is only marginally larger than the children’s. Otherwise, such nodes might be selected rarely and not be properly optimized during training. To avoid this we sparsify the generated tree structure.

We begin by marking all leaf nodes — the output of 3DGS optimization — as relevant, i.e., they should not be removed from the tree. Next, we find the union of cuts in the tree (according to Sec. 4.2) over all training views for the minimum target granularity $\tau_{min}=3$ pixels (the minimum extent of a primitive 3DGS primitive due to low-pass filtering). We then find the bottom-most nodes in this union, which again yields a cut. These nodes are considered to be the highest-detail nodes that are relevant for the selected granularity. All nodes between them and already marked nodes are removed from the tree. We then raise the target granularity by $2\times$ and repeat this process until $\tau_{max}$, half the image’s resolution, is reached. Note that this may result in nodes with $K$ children, in which case Eq. 12 generalizes to: $\alpha^{\prime}~{}=~{}1-(1-\alpha_{p})^{(\frac{1}{K})}$.

We first calibrate cameras of the entire dataset; we discuss our solution to this engineering challenge in Sec. 7 and in the Appendix. To allow consistent training of all chunks, we need to provide a basic scaffold and skybox for all ensuing steps. We do this by running a very coarse initial optimization on the full dataset. Specifically, we initiate a default 3DGS optimization of the entire scene, using the available SfM points and add an auxiliary skybox (see Sec. 6.2). Additionally, we disable densification and the position of primitives is not optimized during this step, since the SfM points are well placed. This coarse model serves as a minimal basis for providing backdrop details, i.e., parts of the scene outside a given chunk. In the case of extremely large scenes where storage of the SfM points would exceed (V-)RAM capacities, the coarse optimization itself could be broken into multiple steps, with intermediate, partial results being streamed out to disk. We split our scenes into chunks that are large enough to establish sufficient context for common real-world elements, including cars, buildings and monuments. For each chunk, we select all cameras that are inside the bounds of the chunk, or are within $2\times$ the chunk bounds and have more than 50 SfM points within the chunk bounds.

We train each chunk independently; the result is then used to create a hierarchy (Sec. 4) for each chunk, followed by the optimization and consolidation steps (Sec. 6.3). A typical chunk in large datasets differs significantly from the data used in the original 3DGS setting. In particular, the extent of the scene is significantly larger than the small scenes of datasets such as Mip-NeRF 360 (Barron et al., 2022) or Deep Blending (Hedman et al., 2018). Also the capture density is much lower, and is not “object-centric”. This makes it harder to optimize, since the space of rays is not covered uniformly (Kopanas and Drettakis, 2023). The data also contain exposure changes, humans and moving objects (cars, bicycles etc.) that need to be removed and ignored in the optimization.

We define a skybox that surrounds the extent of the scene, i.e., 100 000 3DGS primitives on a sphere $10\times$ the diameter of the scene, to capture the effect of the sky. We load the coarse optimization of the scene that will be used for all content outside the chunk; it also prevents each chunk from creating inconsistent content for the sky. We train the content inside the chunk using the optimization of 3DGS (Kerbl et al., 2023), using correct anti-aliasing (Yu et al., 2023). We perform a small, temporary optimization of the coarse environment and the skybox outside the chunk. Specifically we only optimize opacity and SH coefficients.

The original 3DGS optimization collects statistics to decide whether or not Gaussians should be densified at regular intervals. Specifically, the densification policy is based on the mean of the screen-space positional gradients over a fixed number of iterations. There are two main problems in the context of unbounded scenes. First, this policy rarely discourages Gaussians from densifying, regardless of whether they are fine enough to model local detail. Second, data sets with sparse, scattered cameras that observe separate portions of the scene (as is, e.g., the case for urban drive-through captures) result in a much lower tendency to densify overall. We address both of these issues by changing the densification policy to consider the maximum of the observed screen-space gradient rather than its mean, that is no longer reliable in the context of sparse capture.

The sparse camera captures that we have do not provide enough information for good quality reconstruction, e.g., for the street in urban driving scenarios. We perform monocular depth prediction, scale and shift the depth based on the SfM points and use it to regularize the optimization. This results in improvement in visual quality, especially for the road (see Sec. 8). We provide the details of this process in Appendix C.

Each chunk starts with the SfM points from the per-chunk refinement from COLMAP (Sec. 7.2), and the 3D Gaussians contained in the scaffold that are in its neighboring chunks. The resulting 3D Gaussians associated with a chunk and its hierarchy are thus sometimes outside the chunk itself. During the consolidation phase, if a primitive associated to—but outside—chunk $i$ is closer to another chunk $j\neq i$, it is deleted. The consolidation also creates a global hierarchy with a root node for the entire scene. Rendering is performed by setting a granularity threshold, and finding the corresponding cut. We update the cut every 2 frames to add details by transferring nodes to the GPU from CPU RAM, and run cleanup every 100 frames.

We implemented our method on top of the original 3DGS implementation in C++ and Python/PyTorch, modifying the SIBR (Bonopera et al., 2020) viewer for fast rendering. We will provide all the source code and the data on publication of our paper, including all scripts for dataset preprocessing (see project page). We use auto differentation available in PyTorch for the hierarchy optimization, except for the gradients of Eq. 12 which we derive manually. Also, during the optimization of the hierarchy, we avoid optimizing leaf nodes by using the stop gradient operator on all Gaussians corresponding to leaf nodes in the hierarchy, effectively freezing their attributes.

We captured three outdoors scenes: SmallCity, BigCity, Campus. We list the statistics of each scene in Tab. 1 as well as those of an additional scene provided by Wayve.

Several intermediate nodes will generate $\alpha$ values $>1$, required for merged Gaussians to represent more opaque-appearing primitives with a delayed fall-off (Sec. 4). However, in the presence of these nodes, we can no longer apply the original 3DGS exponential activation for opacity during training. Instead, we use an absolute value activation function for post optimization. Since the $\alpha$-blending values are already internally clamped to $0.99$ in the rasterization, no additional changes are necessary to the 3DGS forward routine. However, for robustness, we must account for this clam** step and zero a Gaussian’s opacity gradients whenever it occurs.

The results are best appreciated in the supplemental videos, where we see that we can navigate in the captured large scenes in real-time. The paths we show in the video illustrate the full extent of captured area to show how large they are. For each scene we stop and show a free-viewpoint walk-around session. This works particularly well in areas where camera density is higher, such as places where multiple passes of capture crossed, etc.

Our smallest scene, SmallCity, contains only a single urban block. We cover more than a kilometer distance in the Campus, Wayve scenes, and several kilometers in the BigCity scene. During training, they are subdivided into 4, 11, 22, and 46 chunks, respectively. Each chunk has 2–8M leaf Gaussians. The coarse, single chunk, and hierarchy optimization take one hour each on one NVIDIA V100 GPU (32 GB), with 2–5 GB, 8–10 GB, and 11–16 GB peak memory usage respectively. After per-chunk optimization and consolidation, disk sizes for the hierarchical scenes are 6, 17, 27, and 88 GB, $\approx$68% larger than non-hierarchical 3DGS files. While the quality is globally good, there are some artifacts. Most such visual artifacts are due to the sparse nature of these large captures: contrary to traditional radiance field captures, a given point in the scene is only seen by a handful of cameras. Other artifacts are due to distractors in the scene that were not completely removed: we are able to remove most moving cars, but cars coming to a stop are not correctly identified, leading to inconsistent data and bad optimization. Develo** a complete solution to this problem is orthogonal to our contribution (see also Sec. 9 and Appendix A, B).

We also show the effect of progressive interpolation between hierarchy levels in the video.

Comparing to other methods is difficult, since no other approach with code available can handle our large scenes. We thus decided to compare the quality of our method, given different options on a single chunk that can be reasonably handled by previous methods. All methods, including ours, receive the same input, i.e., only the images that belong to that chunk. In particular, we compare to F2-NeRF, Instant-NGP, the original 3DGS, Mip-NeRF 360 and Mega-NeRF (Turki et al., 2022) on one chunk of each dataset. Finally, we perform ablations on several elements in our method, illustrating the importance of each corresponding algorithmic component. All real-time rendering results and performance metrics were collected on an NVIDIA A6000 GPU.

We perform several ablations to assess the effect of different aspects of our algorithm in our datasets. In Fig. 11, we show the visual effects of hierarchy consolidation and depth supervision on the result. Clearly, consolidation by removing redundant scaffold Gaussians is essential for visual quality. For urban scenes, we find that the ability for view extrapolation can be diminished if we do not use depth in our datasets. While depth supervision does not improve per-image metrics (see also quantitative single-chunk ablations in Tab. 6), we note that its inclusion particularly improves the quality of the appearance of roads, which often lack salient features. We also show the effect of the additional bundle adjustment step for each chunk; the COLMAP hierarchical mapper only provides approximate camera poses that need to be refined. These higher-fidelity poses demonstrably reduce blurriness and increase detail in the resulting novel views. Finally, we show the effects of exposure compensation and hierarchy optimization: The former removes spurious Gaussians attempting to model differences in intensity between images. The latter improves the quality of intermediate nodes in the hierarchy, leading to sharper shapes and outlines, especially for distant objects.

In addition to the above, for a single chunk of Wayve, we further examined the effect of including or excluding our smooth interpolation during hierarchy optimization. Excluding interpolation both from training and evaluation results in PSNR metrics of 25.21 db, 24.73 db and 23.44 db for granularity settings $\tau_{1},\tau_{2}$ and $\tau_{3}$, respectively, yielding a clear drop in quality across multiple hierarchy levels (compare with results in Tab. 2).