Scalable Human-Machine Point Cloud Compression

Abstract

Due to the limited computational capabilities of edge devices, deep learning inference can be quite expensive. One remedy is to compress and transmit point cloud data over the network for server-side processing. Unfortunately, this approach can be sensitive to network factors, including available bitrate. Luckily, the bitrate requirements can be reduced without sacrificing inference accuracy by using a machine task-specialized codec. In this paper, we present a scalable codec for point-cloud data that is specialized for the machine task of classification, while also providing a mechanism for human viewing. In the proposed scalable codec, the “base” bitstream supports the machine task, and an “enhancement” bitstream may be used for better input reconstruction performance for human viewing. We base our architecture on PointNet++, and test its efficacy on the ModelNet40 dataset. We show significant improvements over prior non-specialized codecs.

I Introduction

Point clouds representing 3D visual data are increasingly being used in many applications, including augmented reality, robotics, and autonomous driving. Advances in deep learning have led to the development of deep models for performing machine vision tasks on point cloud data, including classification, object detection, and segmentation. However, most deep learning models are computationally expensive. This poses a challenge for computationally-limited edge devices that want to perform machine vision tasks, and yet are limited in size, energy consumption, cost, and other factors.

One option for performing a machine task on the edge device is to limit the complexity of the model. Unfortunately, this usually comes at the cost of model accuracy. Another option is to transmit the input data to a server for machine analysis. In this approach, the edge device compresses the input data prior to transmission, often using a codec designed to reconstruct the input for human viewing. However, such input reconstruction codecs are not optimized for machine analysis. Thus, they often spend large amounts of bits on encoding information that is not relevant to the machine task. This results in a worse rate-accuracy trade-off than is possible with a more specialized codec. In situations where a low rate is desired — for instance, in areas of poor network connectivity, or congested networks — using a non-specialized codec may result in excessively high machine task latency [1].

In order to improve the rate-accuracy trade-off, we may instead use a codec that is specialized for the machine task. Such codecs often perform part of the machine task on the edge device itself. In this hybrid approach, the edge device simultaneously compresses the input and performs part of the machine task. This allows the model to discard unnecessary information, thus reducing the rate, resulting in a system that is more robust to changing network conditions, and may reduce system latency over a certain range of available bitrates [2].

In [3], a novel codec for point cloud classification was proposed. This learned codec, based on PointNet [4], compresses the input point cloud into a highly compressed representation that is intended solely for machine analysis, in this case classification. This codec was shown to achieve a significantly better rate-accuracy trade-off in comparison with alternative methods using standard codecs that are not specialized for machine analysis. This was achieved by removing not only statistical redundancy, but also task-irrelevant information, during the compression process.

While the codec in [3] achieves a good rate-accuracy trade-off for point cloud classification, it is not suitable for other purposes. Most applications involving automated machine-based analysis are expected to run the machine task continuously, but may occasionally require human verification or review. Hence, it is important to develop codecs that are able to support both tasks — machine vision and human viewing — efficiently. In this paper, we present such a scalable codec, the first in the point cloud literature, which supports point cloud classification, while also providing a mechanism for human viewing. Our code is available online.

II Related work

Point cloud classification is among the most researched point cloud analysis tasks. Related classification models accept different input formats, including point lists (e.g., PointNet [4] and PointNet++ [5]), 3D voxel grids (e.g., VoxNet [6]), and Octrees (e.g., OctNet [7]). Since our work builds on PointNet and PointNet++, we assume the input format is a point list.

Conventional handcrafted point cloud codecs include Draco [8] and G-PCC [9] (implemented as TMC13 [10]). More recently, the research focus has shifted towards learned codecs, following the seminal work of Ballé et al. [11] who proposed a variational autoencoder (VAE) based architecture for image compression. In their architecture, the input ${\boldsymbol{x}}$ is first transformed into a latent representation ${\boldsymbol{y}}$, which is then quantized and entropy-coded using a learned entropy model. Such an architecture can be trained end-to-end using the loss

$$\mathcal{L}=R+\lambda\cdot D({\boldsymbol{x}},{\boldsymbol{\hat{x}}}),$$ (1)

where $D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ is the distortion measure between the input ${\boldsymbol{x}}$ and decoded ${\boldsymbol{\hat{x}}}$, and $R$ is the estimate of the entropy of ${\boldsymbol{\hat{y}}}$. This mechanism has been successful in various fields of learned compression, including point cloud compression [12, 13, 14, 15, 16].

Specialized compression for machine tasks — often referred to as coding for machines — has been explored for images [17] and video [18] and, recently, for point clouds [3]. Moreover, scalable multi-task coding approaches [19, 20] have shown that one can perform a machine vision task at a fairly low bitrate, while enabling other tasks, such as input reconstruction for human viewing, with an additional enhancement bitstream.

While quality-scalable point cloud coding has been studied before [21], this paper presents the first scalable multi-task point cloud codec: the base bitstream supports point cloud classification, while the enhancement bitstream allows point cloud reconstruction. Unlike our earlier work [3], which presented a classification-optimized codec based on PointNet [4], our scalable codec is based on PointNet++ [5], which is a hierarchical extension of PointNet.

III Proposed codec

(a) Input compression.

(b) Machine task compression, as used in e.g. [3].

(c) Scalable multi-task compression.

Figure 1: High-level comparison of codec architectures.

Figure 2: Proposed codec architecture.

Figure 3: Proposed codec architecture (details).

III-A Preliminaries

In fig. 0(a), we show an abstract representation of an input compression codec. The input point cloud ${\boldsymbol{x}}$ is encoded and decoded as ${\boldsymbol{\hat{x}}}$ by any desired point cloud codec, including non-learned codecs such as G-PCC [9]. Then, the reconstructed point cloud ${\boldsymbol{\hat{x}}}$ is fed into a classification model (e.g., PointNet) in order to obtain the class prediction ${\boldsymbol{\hat{t}}}$. This approach provides a baseline for comparison with our proposed codec.

In fig. 0(b), we show an abstract representation of a machine task codec, as was explored by [3] for point cloud classification. Using the same terminology as in [11], $g_{a}$ refers to the analysis transform, and $g_{s}$ refers to the synthesis transform. In this codec, the input point cloud ${\boldsymbol{x}}$ is first encoded into a latent representation ${\boldsymbol{y}}=g_{a}({\boldsymbol{x}})$. This is then quantized as ${\boldsymbol{\hat{y}}}=Q({\boldsymbol{y}})$, and then losslessly compressed using a learned entropy model. For instance, in [3], a fully-factorized entropy model was used. The reconstructed latent representation ${\boldsymbol{\hat{y}}}$ may then be used to predict the classes ${\boldsymbol{\hat{t}}}=g_{s,t}({\boldsymbol{\hat{y}}})$.

III-B Scalable human-machine compression codec

In fig. 0(c), we show a high-level representation of a scalable multi-task codec. Following the principle of latent space scalability [20], the scalable multi-task codec splits the latent space into two parts, $[{\boldsymbol{\hat{y}}}_{1},{\boldsymbol{\hat{y}}}_{2}]=\operatorname{split}({\boldsymbol{\hat{y}}})$, along the channel dimension. The first part ${\boldsymbol{\hat{y}}}_{1}$ is called the “base” layer and the second part ${\boldsymbol{\hat{y}}}_{2}$ is called the “enhancement” layer. The base layer is used for the machine task (i.e., classification) to predict the class ${\boldsymbol{\hat{t}}}=g_{s,t}({\boldsymbol{\hat{y}}}_{1})$. Both base and enhancement layers are used for input reconstruction: ${\boldsymbol{\hat{x}}}=g_{s,x}(\operatorname{concat}[\operatorname{detach}({\boldsymbol{\hat{y}}}_{1}),{\boldsymbol{\hat{y}}}_{2}])$. This approach allows for scalability: the enhancement bitstream only needs to be computed and transmitted when human viewing is desired. Note that the $\operatorname{detach}$ operation is used to disable gradient propagation of $D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ backwards through ${\boldsymbol{\hat{y}}}_{1}$. This improves the specialization of ${\boldsymbol{\hat{y}}}_{1}$ towards the machine task; otherwise, ${\boldsymbol{\hat{y}}}_{1}$ may end up with some enhancement-layer information [22]. Two separate bitstreams (base and enhancement) are produced, whose total rate is $R_{{\boldsymbol{\hat{y}}}}=R_{{\boldsymbol{\hat{y}}}_{1}}+R_{{\boldsymbol{\hat{y}}}_{2}}$.

III-C Proposed architecture

Our complete proposed codec architecture, based on PointNet++ [5], is shown in fig. 2, and the details of each block are shown in fig. 3. Our codec is provided in “full” and “lite” configurations, as detailed in table I. The input point cloud of $P$ points is represented as a matrix ${\boldsymbol{x}}\in\mathbb{R}^{3\times P}$ of $(x,y,z)$ coordinates. ${\boldsymbol{u}}$ is an additional set of attribute features (e.g., normals, color, etc.) that optionally may also be compressed. At the beginning of the encoder, ${\boldsymbol{x}}$ and ${\boldsymbol{u}}$ are concatenated along the channel dimension so that ${\boldsymbol{u}}^{(0)}=\operatorname{concat}[{\boldsymbol{x}},{\boldsymbol{u}}]$. This ensures that the same encoding capabilities are available for both. We feed ${\boldsymbol{x}}^{(0)}={\boldsymbol{x}}$ and ${\boldsymbol{u}}^{(0)}$ into a sequence of downsampling blocks. Each $i$-th downsampling block takes in ${\boldsymbol{x}}^{(i-1)}$ and ${\boldsymbol{u}}^{(i-1)}$, and outputs and a smaller set of centroids ${\boldsymbol{x}}^{(i)}$ and features ${\boldsymbol{u}}^{(i)}$, along with features ${\boldsymbol{u_{g}}}^{(i-1)}$ that are grouped by the centroids. The final ${\boldsymbol{u}}^{(3)}$ is compressed in a multi-task scalable manner, from which ${\boldsymbol{\hat{u}}}^{(3)}$ is derived. Each ${\boldsymbol{u_{g}}}^{(i-1)}$ is compressed using a standard transform-encoder-decoder-transform compression pipeline to obtain ${\boldsymbol{\hat{u}_{g}}}^{(i-1)}$. A sequence of upsampling blocks is then applied, where each $i$-th upsampling block takes in ${\boldsymbol{\hat{u}}}^{(i)}$. For $i>0$, the output of the $i$-th upsampling block is concatenated with ${\boldsymbol{\hat{u}_{g}}}^{(i-1)}$ to give ${\boldsymbol{\hat{u}}}^{(i-1)}$. For $i=0$, the output is ${\boldsymbol{\hat{x}}}$.

III-C1 Downsampling

Each downsampling block is a PointNet++ “set abstraction” layer (see [5] for more details), with minor modifications. We used single-scale grou** (SSG) of points for our proposed codec, though it may likely be improved with multi-scale grou** (MSG) or multi-resolution grou** (MRG) as described in [5]. The “set abstraction” layer takes as input a (subsampled) point cloud ${\boldsymbol{x}}^{(i-1)}$ of shape $3\times P^{(i-1)}$ and features ${\boldsymbol{u}}^{(i-1)}$ of shape $D^{(i-1)}\times P^{(i-1)}$ containing information about each corresponding point. Using farthest point sampling (FPS), a set of $P^{(i)}$ centroids ${\boldsymbol{x}}^{(i)}$ of shape $3\times P^{(i)}$ is selected from ${\boldsymbol{x}}^{(i-1)}$. Then, the points in ${\boldsymbol{x}}^{(i-1)}$ are grouped into $P^{(i)}$ groups of $S^{(i)}$ points each. Then for each centroid, a ball query [5] is performed to find the first $S^{(i)}$ points that are within a radius of $R^{(i)}$ from the given centroid. The relative positions of each group of points is then computed with respect to its associated centroid, resulting in the residuals $r^{(i)}=\operatorname{ball\_query}({\boldsymbol{x}}^{(i-1)},{\boldsymbol{x}}^{(i)},R^{(i)})-\operatorname{repeat}({\boldsymbol{x}}^{(i)},S^{(i)})$ of shape $3\times P^{(i)}\times S^{(i)}$. Each point in ${\boldsymbol{r}}^{(i)}$ is concatenated with its respective feature vector in ${\boldsymbol{u}}^{(i-1)}$, resulting in a grouped feature tensor ${\boldsymbol{u_{g}}}^{(i-1)}$ of shape $(D^{(i-1)}+3)\times P^{(i)}\times S^{(i)}$. Then, ${\boldsymbol{u_{g}}}^{(i-1)}$ is fed into a miniature group-wise “PointNet encoder” block consisting of a shared multi-layer perceptron (MLP) with a max pooling operation at the end. That is, the same PointNet encoder is applied to each group of features independently and identically. This results in ${\boldsymbol{u}}^{(i)}$ of shape $D^{(i)}\times P^{(i)}$. Three downsampling blocks are used in our proposed codec. The last downsampling block groups all the remaining points into a single group, i.e., $P^{(3)}=1$.

III-C2 Feature compression

At each level $i<3$, a latent representation ${\boldsymbol{y}}^{(i)}={h_{a}}^{(i)}({\boldsymbol{u_{g}}}^{(i)})$ of shape $M^{(i)}\times P^{(i)}$ is computed, then quantized as ${\boldsymbol{\hat{y}}}^{(i)}=Q({\boldsymbol{y}}^{(i)})$, and then entropy-coded using a learned entropy model. The resulting bitstream is transmitted and then decoded as ${\boldsymbol{\hat{y}}}^{(i)}$. From this, a set of grouped features ${\boldsymbol{\hat{u}_{g}}}^{(i)}=h_{s}^{\,(i)}({\boldsymbol{\hat{y}}}^{(i)})$ is computed for usage during upsampling. (Note that ${\boldsymbol{\hat{u}_{g}}}^{(i)}$ and ${\boldsymbol{u_{g}}}^{(i)}$ may inhabit entirely different feature spaces, and are unrelated.)

The final feature vector ${\boldsymbol{u}}^{(3)}$ is fed into the scalable multi-task compression pipeline described in section III-B. The latent representation ${\boldsymbol{y}}^{(3)}={h_{a}}^{(3)}({\boldsymbol{u}}^{(3)})$ is computed, then quantized as ${\boldsymbol{\hat{y}}}^{(3)}=Q({\boldsymbol{y}}^{(3)})$, and then entropy-coded to generate a base bitstream for ${\boldsymbol{\hat{y}}}_{1}^{(3)}$ and an enhancement bitstream for ${\boldsymbol{\hat{y}}}_{2}^{(3)}$. The base bitstream is decoded as ${\boldsymbol{\hat{y}}}_{1}^{(3)}$, and then fed into a decoder-side task backend to obtain the class prediction ${\boldsymbol{\hat{t}}}=g_{s,t}^{(3)}({\boldsymbol{\hat{y}}}_{1}^{(3)})$. The enhancement bitstream is decoded as ${\boldsymbol{\hat{y}}}_{2}^{(3)}$, and then its concatenation with ${\boldsymbol{\hat{y}}}_{1}^{(3)}$ (detached) is fed into a decoder-side transform to obtain ${\boldsymbol{\hat{u}_{g}}}^{(3)}=h_{s}^{(3)}(\operatorname{concat}[\operatorname{detach}({\boldsymbol{\hat{y}}}_{1}^{(3)}),{\boldsymbol{\hat{y}}}_{2}^{(3)}])$.

Note that we split only ${\boldsymbol{y}}^{(3)}$ into ${\boldsymbol{y}}^{(3)}_{1}$ and ${\boldsymbol{y}}^{(3)}_{2}$. This is because ${\boldsymbol{y}}^{(3)}_{1}$ suffices for the classification task, while ${\boldsymbol{y}}^{(3)}_{2}$ and ${\boldsymbol{y}}^{(j)}$ for $j<3$ are only needed for input reconstruction. For brevity, in the experiments we only test input reconstruction with all enhancement layers, but the proposed architecture also allows for quality- or density-scalable input reconstruction.

III-C3 Upsampling

An upsampling block takes in $E^{(i+1)}+D^{(i)}+3\cdot(1-\delta_{i,3})$ channels and outputs $E^{(i)}$ channels, where $\delta_{i,j}$ is the Kronecker delta ($\delta_{i,j}=1$ if $i=j$, $0$ otherwise). Each upsampling block consists of a simple two-layer point-wise MLP. For upsampling blocks at levels deeper than $i>0$, batch normalizations are also present. Furthermore, each such block is followed by a reshape-transpose-reshape combination that converts the resulting shape $(E^{(i)}\cdot S^{(i)})\times P^{(i)}$ into the output shape $E^{(i)}\times(S^{(i)}\cdot P^{(i)})$. To ensure compatibility of shapes among various components during reconstruction, we enforce the constraint $P^{(i-1)}=P^{(i)}\cdot S^{(i)}$.

TABLE I: Architecture hyperparameter configurations

Codec	$i$	$P^{(i)}$	$S^{(i)}$	$R^{(i)}$	$D^{(i)}$	$E^{(i)}$	$M^{(i)}$
full	0	1024			3	3	0
	1	256	4	0.2	128	64	0
	2	64	4	0.4	192	32	64
	3	1	64		256	16	(48, 16)
lite	0	1024			3	3	0
	1	256	4	0.2	32	32	0
	2	64	4	0.4	48	16	16
	3	1	64		64	8	(48, 16)

IV Experiments

We trained our models on the ModelNet40 [23] dataset, sampling $P=1024$ points per object, and reconstructed the same number of points. Our implementation uses the PyTorch, CompressAI [24], and CompressAI Trainer [25] libraries.

(a) RA for base task.

(b) RA for base task (zoomed in).

(c) RD for reconstruction task.

Figure 4: Rate-accuracy (RA) and rate-distortion (RD) curves on the ModelNet40 dataset, with rate units of bits per point (bpp) scaled for 1024 points.

The loss function used for training was

$$\mathcal{L}=\sum_{i=0}^{3}R_{{\boldsymbol{\hat{y}}}^{(i)}}+\lambda_{x}D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})+\lambda_{t}D({\boldsymbol{t}},{\boldsymbol{\hat{t}}}).$$ (2)

The rate estimate $R_{{\boldsymbol{\hat{y}}}^{(i)}}$ for each level is the negative log likelihood outputted by the respective entropy models. The distortion $D({\boldsymbol{t}},{\boldsymbol{\hat{t}}})$ is the cross-entropy between the one-hot encoded labels ${\boldsymbol{t}}$ and the softmax of the model’s prediction ${\boldsymbol{\hat{t}}}$. The distortion $D({\boldsymbol{x}},{\boldsymbol{\hat{x}}})$ is the Chamfer distance between the input and reconstructed point clouds. We trained different models to operate at different base/enhancement rate points by varying the hyperparameters $\lambda_{t}\in\{2^{-7},2^{-6},\ldots,2^{0}\}$ and $\lambda_{x}\in[1,8000]$. These values are reported for rates in units of bits per point (bpp), i.e., the rates are divided by $P=1024$. To improve stability during training, we disabled the first two levels by setting $M^{(0)}=0$ and $M^{(1)}=0$. We also set $M^{(2)}=64$ and $M^{(3)}=64$, and used a partitioning ratio of $0.75$ for the last layer, i.e., $M^{(3)}_{1}=48$ (base) and $M^{(3)}_{2}=16$ (enhancement).

The input compression codecs were evaluated using the same methodology for varying $P$ and input scaling as in [3]. Since codecs like TMC13 [10] and OctAttention [15] are tailored to point clouds with a large number of points, the coding overhead they produce may be significant when applied to smaller point clouds such as those used in classification. To improve the fairness when measuring the rate, we ignored such overhead within the bitstream when possible.

V Results

figs. 3(b) and 3(a) show the rate-accuracy curves for the base task of point-cloud classification. Only the rate of the base bitstream is measured for the base task. Scalable codec curves are grouped by their worst-case reconstruction ability measured in terms of Chamfer distance [26]. Our proposed codec achieves better classification performance than all existing codecs, including [3] (denoted “Ulhaq2023”), which in turn is better than input compression codecs (“ICC”) that use a point-cloud codec (TMC13 [10] and OctAttention [15]) followed by a PointNet/PointNet++ classifier. Only the best ICC curves are shown. Note that “$P{=}{}^{*}$” indicates the best results attained when varying $P\in\{8,16,\ldots,1024\}$.

There is a large gap in base task performance between task-specialized codecs (the proposed one and [3]) and non-specialized codecs because, in addition to statistical redundancy, the former codecs remove a significant amount of task-irrelevant information. Meanwhile, the proposed codec outperforms [3] due to its reliance on the more powerful PointNet++, rather than PointNet.

fig. 3(c) show the rate-distortion curves for the reconstruction task in terms of the rate versus the Chamfer distance. For the proposed codec, the rate includes both the base and enhancement bitstreams. The figures show that our proposed codec is also capable of achieving better reconstruction quality than conventional codecs at low rates, below 1.6 bpp. As the rate increases, the conventional codecs eventually catch up in terms of reconstruction quality.

VI Conclusion

In this paper, we proposed a scalable multi-task compression codec for point clouds. Our proposed codec produces a base bitstream that supports point cloud classification and an enhancement bitstream that, together with the base bitstream, supports reconstruction of the point cloud for human viewing. The proposed codec showed a significant improvement in rate-accuracy performance for the base task over existing codecs, while also achieving competitive performance against conventional codecs on input reconstruction at reasonably low rates. In the future, we hope that this work will inspire further research into scalable multi-task compression codecs for point clouds on more complex real-world datasets, and for other tasks such as semantic segmentation.