StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering

We propose a fast tile-based culling approach that bounds Gaussians to exactly those tiles they contribute to. For each ray, Kerbl et al. (2023) disregard Gaussians with a contribution below $\epsilon_{O}=1/255$, which forms an exact culling condition. Like 3DGS, we start with an axis-aligned bounding rectangle using the largest eigenvalue of the 2D covariance matrix to determine which tiles may potentially be touched during both Preprocess and Duplication. This conservative estimate gives very large bounds for highly anisotropic Gaussians.

For exact culling, we then calculate the point $\mathbf{\hat{\mathbf{x}}}$ inside each tile $X$ that maximizes the 2D Gaussian’s contribution $G_{2}(\mathbf{x})$:

If $\mu_{2}\in X\Rightarrow\mathbf{\hat{\mathbf{x}}}=\mathbf{\mu_{2}}$. If $\mu_{2}\notin X$, then $\mathbf{\hat{\mathbf{x}}}$ must lie on one of the two tile edges closest to $\mu_{2}$, due to Gaussians being monotonic along rays pointing away from $\mathbf{\mu_{2}}$. We can then compute the maximum along those two edges (similar to Eqn. (4), but in 2D) and clamp the resulting values to obtain ${\hat{\mathbf{x}}}$ (see the Appendix B.1 for the full algorithm). Finally, we evaluate $G_{2}(\mathbf{\hat{\mathbf{x}}})$ to perform the comparison with $\epsilon_{O}$, which significantly reduces the number of Gaussians per tile (cf. Fig. 6).

For pre-sorting we require a representative $t_{opt}$ per tile. Intuitively, the center ray of the tile should be a valid compromise for all rays in the tile. However, this completely ignores the fact that a Gaussian in general does not uniformly contribute to all rays in a tile. Especially for small Gaussians whose main extent is approximately parallel to the view rays, the center ray may result in depth estimates far away from any contribution made by the Gaussian.

Arguably, the weighted integral $\int_{X}G_{2}(\mathbf{x})t_{opt}(\mathbf{x})d\mathbf{x}$ is a better estimate. Yet, even a numerical approximation considering all rays in the tile $X$ is too compute-intensive. Thus, we approximate it with a single sample: the one with the highest weight within a tile, i.e., $\mathbf{\hat{\mathbf{x}}}$. Since $\mathbf{\hat{\mathbf{x}}}$ was already calculated during culling, we only need to construct the corresponding ray to evaluate $t_{opt}$. The optimized depth location reduces $(\delta_{\text{max}},\delta_{\text{avg}})$ from $(1.553$, $0.006)$ to $(0.575$, $0.003)$ and $(3.917$, $0.014)$ to $(3.098$, $0.006)$ for the views in Tab. 1.

Similar to other compute-mode rasterization methods, primitives that cover a large portion of the screen may become an issue if a single thread evaluates their coverage. For 3DGS, this is the case in the first two stages of the rendering pipeline, which operate on a per-Gaussian basis. For our method, tile-culling and per-tile depth calculations increase the workload of these stages, which further amplifies this problem.

To remedy this issue, we propose a two-stage load balancing scheme: In the first phase, each thread responsible for a Gaussian which covers fewer than a predetermined maximum number of tiles, performs its own processing. We empirically determined that a maximum of $32$ tiles results in good performance. Most threads are typically idle after this initial phase. In the second phase, we distribute the remaining workload within each warp using warp voting and shuffle instructions. For close-ups and high-resolution rendering, where single Gaussians often cover a large portion of the screen, our approach can speed up Preprocess and Duplication by up to $10\times$.

With the goal of establishing a hierarchical rendering pipeline, a naïve approach is to design one kernel per hierarchy level. However, such an approach would require communication via slow global memory between the levels and would prohibit early ray termination after reaching the opacity threshold. Thus, we opt for combining the final three levels of our rendering hierarchy in a single kernel, where multiple threads cooperatively sort and manage shared queues, as detailed in Fig. 7. We use a large $4{\times}4$ tile-queue of 64 elements (managed by 16 threads), feeding into four eight-element $2{\times}2$ tile-queues. Finally, each $2{\times}2$ tile-queue feeds into four per-pixel queues with four elements, managed by one thread each. For one $16{\times}16$ tile, we thus start 256 threads, allocate 16 $4{\times}4$ tile-queues and 64 $2{\times}2$ tile-queues in shared memory as well as one per-pixel queue per thread in registers. Each queue stores only the Gaussian’s id and the current level’s depth $t_{opt}$. Additional information is loaded on demand from global memory and shared between threads of the hierarchical level via shuffle operations, e.g. $\mu,\Sigma^{-1}$ during depth calculation, or $\mu_{2},\Sigma_{2}^{-1}$ during culling and blending.

The queues follow a push methodology to keep queue fill rates as high as possible, ensuring that resorting remains effective. While 16 threads (a halfwarp) are assigned to each $4{\times}4$ tile-queue, we load and feed batches of 32 into two $4{\times}4$ tile-queues at once, allowing all threads within a warp to load data together. After loading, each thread performs tile-based culling (as described before, but for a $4{\times}4$ tile), followed by computing $t_{opt}$. For culled Gaussians, we set $t_{opt}=\infty$. Then, each halfwarp sorts the 32 newly loaded elements using Batcher Merge Sort (Batcher, 1968) before writing them to the back of the $4{\times}4$ tile-queue. Typically, there are now two individually sorted parts in the $4{\times}4$ tile-queue: the already present elements (up to 32) and the newly added (up to 32). As both are sorted, we use efficient merge sort to combine them. Culled Gaussians are now at the back of the queue and can be discarded.

While there are more than 32 elements in the $4{\times}4$ tile-queue, we push batches of size 16 into the $2{\times}2$ tile-queue. Each thread in the halfwarp re-fetches the data needed for computing $t_{opt}$ for a single Gaussian. Each group of four threads then pushes sub-batches of size four into their $2{\times}2$ tile-queue, relying on shuffle instructions to update $t_{opt}$ for each $2{\times}2$ tile. We follow the same approach as before: we sort the four new entries according to depth, for which we use a simple coordination using shuffle instructions. We then use merge sort to combine the new elements with the existing ones.

After the $2{\times}2$ tile-queue is filled, we draw four elements from it and insert them into the per-pixel queue. Again, we batch-load the needed data using the four threads assigned to the respective $2{\times}2$ tile-queue, and again use shuffle instructions to communicate all relevant information for each Gaussian to all other threads in the $2{\times}2$ tile. We evaluate $t_{opt}$ and $\alpha$ for the respective rays and insert the newly computed data into the per-pixel queue. If the Gaussian’s $\alpha$ is below $\epsilon_{O}$, we simply discard it. As we add elements one by one into the per-pixel queue, we rely on simple insertion sort. Only if the per-pixel queue is full, we take one element from it and perform blending, freeing up space for the next element from the $2{\times}2$ tile-queue.

Due to the hierarchical structure, we effectively construct an overall sort window varying between $25$ and $72$, where the minimum is hit if the $4{\times}4$ tile-queue is drained down to $17$ elements, with $4$ elements remaining in the other queues. $72$ elements are sorted if we fill the $4{{\times}}4$ tile-queue with $64$ elements and then move $4$ elements through the half-filled $2{\times}2$ tile-queue and the filled per-pixel queue. While our sort setup typically achieves better sorting than a simple per-thread sort window of $25$, we may occasionally achieve worse sorting, as the higher-level queues are shared between threads.

The sizes of the three queues are variable, with some restrictions. The $4{\times}4$ tile-queue size is constrained to $32n+32$, with $n\in\mathbb{Z}^{+}$, as this enables efficient warp-wide merge sort. Similarly, the $2{\times}2$ tile-queue must be of size $4m+4$, with $m\in\mathbb{Z}^{+}$, as it is managed by four threads. The per-pixel queue size can be chosen arbitrarily. We heuristically decided on $(64/8/4)$ for the three queue sizes, as this achieves a large enough sort window, while limiting shared memory requirements and register pressure, ultimately leading to better performance. We provide ablations for our chosen queue sizes and load balancing thresholds in Appendix E.

For our quantitative evaluation, we report PSNR, SSIM, LPIPS (Zhang et al., 2018) and F LIP (Andersson et al., 2020) in Tab. 2. To facilitate cross-method comparisons, we reproduce the results from Kerbl et al. (2023) for Mip-NeRF 360 (Barron et al., 2022), Instant-NGP (Müller et al., 2022) and Plenoxels (Fridovich-Keil et al., 2022). For Deep Blending and Mip-NeRF 360 Outdoor, we outperform 3DGS. For Tanks & Temples and Mip-NeRF 360 Indoor, our model performs slightly worse, which we attribute to 3DGS’s ability to fake view-dependent effects with pop**. When enabling Opacity Decay, which results in ${\sim}50\%$ fewer Gaussians, our method retains more quality than 3DGS. In general, our approach performs comparably to 3DGS in terms of standard image quality metrics.

View inconsistencies between subsequent frames, such as pop**, cannot be detected with standard image quality metrics. To detect such artifacts, we follow recent best practice in 3D style transfer (Nguyen-Phuoc et al., 2022) and measure the consistency between novel views and warped novel views with optical flow (Lai et al., 2018). While ground-truth images or videos may seem attractive, they vary significantly in location and thus view-dependent effects or only exist for a small subset of our used datasets. For our method and 3DGS, we capture videos from three separate camera paths per scene, exhibiting both rotation and translation. We then directly warp each frame $\mathbf{F}_{i}$ to a subsequent frame $\mathbf{F}_{i+t}$ with offset $t$ using optical flow predictions from state-of-the-art RAFT (Teed and Deng, 2020).

Measuring the error between the warped frame $\hat{\mathbf{F}}_{i+t}$ and the rendered frame $\mathbf{F}_{i+t}$ with MSE does not prove effective to detect pop** artifacts (see Fig. 9). MSE tends to weigh small inaccuracies that originate from war** higher than pop** artifacts. F LIP (Andersson et al., 2020) proves significantly more reliable in our experiments, as it approximates the difference perceived by humans when flip** between images — a scenario in which pop** artifacts are particularly disturbing. For each frame, we calculate a consistency error $E_{i+t}=\text{\reflectbox{F}LIP}(\hat{\mathbf{F}}_{i+t},\mathbf{F}_{i+t})$. For each video, consisting of $N_{F}$ frames, we then compute the mean F LIP error as

Note that the error metric includes a base error floor due to dis-occlusions under translation and correct view-dependent shading. To mitigate these issues, we use an occlusion detection method from Ruder et al. (2016), do not consider the outermost $20$ pixels, and subtract the per-pixel minimum $\text{\reflectbox{F}LIP}_{t}$ score — clearly, this does not perturb the inter-method differences.

We use $t=1$ and $t=7$ to measure short-range and long-range consistency, following Nguyen-Phuoc et al. (2022). Tab. 3 shows our obtained results. The large margins, particularly for $\text{\reflectbox{F}LIP}_{7}$, highlight that our method is more view-consistent than 3DGS. We argue that $\text{\reflectbox{F}LIP}_{7}$ is a more reliable metric, allowing errors due to pop** to accumulate over multiple frames, as can be seen in Fig. 8. Please see the supplementary video for further evidence. With Opacity Decay, our approach achieves virtually identical results, indicating that our method can handle large Gaussians. For 3DGS, pop** is significantly increased, indicating that 3DGS may increase the number of Gaussians to hide imperfections in the renderer, while our approach achieves comparable view-consistency scores.

3DGS enables efficient extraction of depth values $0pt\in\mathbb{R}_{+}$ with volumetric rendering:

where $\phi(\cdot)$ describes the depth of a single Gaussian with location $\mu$ (in 3DGS’s case, $\phi(\mu;\mathbf{o},\mathbf{d})=\|\mu-\mathbf{o}\|_{2}$). Clearly, this depth estimate is dependent on the sort order, leading to problems for 3DGS’s approximate global sort. Our approach improves sort quality and places 2D splats at the points of maximum contribution ($\phi(\mu;\mathbf{o},\mathbf{d})=t_{opt}$, cf. Eqn. (4)).

We establish a metric to compare multi-view consistency in depth estimates, leveraging the sparse point cloud $\mathcal{P}=\{\mathbf{p}_{i}\}$ from COLMAP (Schönberger and Frahm, 2016), which serves as initialization for 3DGS. If $\mathbf{p}_{i}$ is visible from a camera with position $\mathbf{o}$, we reconstruct the estimated location $\hat{\mathbf{\mathbf{p}}}_{i}=\mathbf{o}+0pt\cdot\mathbf{d}$, with rendered depth $0pt$ and view direction $\mathbf{d}$ of the corresponding pixel of $\mathbf{p}_{i}$. The black background for real-world scenes used by 3DGS enables cheating by not fully accumulating opacity and blending the background color. For a fair comparison, if any of the tested methods has $T_{N_{\mathbf{r}}}>1\times 10^{-2}$ for a point $\mathbf{p}_{i}$, we do not consider this point in our set of tested points $\bar{\mathcal{P}}$. To minimize errors due to resolution, we render at the resolution used for COLMAP when computing $\hat{\mathbf{\mathbf{p}}}_{i}$. Finally, we establish the depth error $E_{depth}$ as

We compute $E_{depth}$ for all test set views and report our results in Tab. 4. On average, our method achieves better scores than 3DGS, especially for the outdoor scenes of Mip-NeRF 360 (Barron et al., 2022). Opacity decay leads to significantly fewer and larger Gaussians, resulting often in lower accumulated opacity and, consequently, more discarded points. Both methods achieve better results for $E_{depth}$ in this case, as these removed points often correspond to the background, where depth estimates are generally less precise.

We provide a performance comparison between 3DGS and our renderer with different configurations in Tab. 5. On average, the Render stage takes considerably longer for our hierarchical renderer (A-E) due to additional per-ray sorting. Not computing the per-tile depth (B) only marginally speeds up the Duplicate stage. Without our load balancing scheme (C), Duplicate takes 5$\times$ longer, as it is mostly dominated by very large Gaussians. Disabling tile-based culling (D) slightly accelerates Preprocess but leads to many more entries in the global sorting data structure, which increases Sort and Render times. Disabling hierarchical culling inside the render kernel (E) leads to a drastic increase in Render time as all Gaussians move through the entire pipeline. Our final approach (A) with all optimizations achieves competitive runtimes on all evaluated scenes. Both methods see a drastic performance increase with Opacity Decay due to the significantly lower number of Gaussians—however, while our approach stays view-consistent, 3DGS shows even more pop** artifacts.

Individual scenes with a similar number of Gaussians can exhibit sharp differences in runtime behavior. In Tab. 6 and Tab. 7, we show detailed timings and metrics for two exemplary scenes - Bonsai and Train - which display the largest inter-method differences in performance, despite their comparable number of Gaussians $N$. Even though the Train scene contains slightly fewer Gaussians than Bonsai, the average number of visible (inside the view-frustum) Gaussians $N_{V}$, as well as their average screen-space size (indicated by avg./std. corresponding image tiles $N_{t}$), is considerably larger.

As larger Gaussian splats provide more opportunities for culling, our tile-based culling results in a larger reduction of avg. $N_{t}$ for Train than Bonsai (${\sim}3.5{\times}$ vs. ${\sim}2.5{\times}$). The resulting lower number of sort entries allows Train to amortize the slower Render stage with a much faster Sort, while Bonsai does not experience the same gains.

The relative performance of our backward Render pass compared to 3DGS is only $1.1\times$ compared to the $1.5\times$ we see for the forward Render stage. This is mostly due to the backward Render executing a large number of atomics, which are equal between both approaches. Although the backward pass skips Duplicate and Sort—which are faster in our renderer—the final change in training time is only about 3%. The backward Render pass is only a single step in the entire training pipeline and thus, the overall time loss is close to negligible. Again, if we turn on Opacity Decay, training becomes proportionally faster.