CountFormer: Multi-View Crowd Counting Transformer

Abstract

Multi-view counting (MVC) methods have shown their superiority over single-view counterparts, particularly in situations characterized by heavy occlusion and severe perspective distortions.

However, hand-crafted heuristic features and identical camera layout requirements in conventional MVC methods limit their applicability and scalability in real-world scenarios. In this work, we propose a concise 3D MVC framework called CountFormer to elevate multi-view image-level features to a scene-level volume representation and estimate the 3D density map based on the volume features. By incorporating a camera encoding strategy, CountFormer successfully embeds camera parameters into the volume query and image-level features, enabling it to handle various camera layouts with significant differences. Furthermore, we introduce a feature lifting module capitalized on the attention mechanism to transform image-level features into a 3D volume representation for each camera view. Subsequently, the multi-view volume aggregation module attentively aggregates various multi-view volumes to create a comprehensive scene-level volume representation, allowing CountFormer to handle images captured by arbitrary dynamic camera layouts. The proposed method performs favorably against the state-of-the-art approaches across various widely used datasets, demonstrating its greater suitability for real-world deployment compared to conventional MVC frameworks.

1 Introduction

Single-view counting (SVC) has exhibited promising effectiveness, yielding remarkable achievements on well-established datasets [93, 81, 1, 28, 5, 41, 42, 52, 69, 62, 31, 53, 6, 57, 74], while the inherent limitations of existing SVC approaches hinder their practical application, thereby impeding their effective deployment in real-world scenarios.

Recently, there has been a growing trend towards addressing the multi-view counting (MVC) problem [86, 90, 87, 88, 91, 89, 82]. Specifically, existing approaches in the field of MVC typically make use of flat ground-plane assumptions to transform the image-level features onto the ground plane. Subsequently, fusing the multi-view (MV) ground features to recover scene-level features, and estimating the scene-level density using predictors based on fused ground features.

However, the flat ground assumptions [86, 90, 87, 88, 91, 89, 95] are not always guaranteed, which can result in misalignment between the scene-level features and the real-world environment, leading to less accurate counting performance. Additionally, in the MV feature fusion module, the attention weights are determined solely based on the distance from the ground plane to each camera without considering the features themselves [86, 90, 91, 89, 95], which may limit the effectiveness of the fusion strategy. Moreover, most current approaches[86, 90, 87, 88, 89, 95] can only handle statically fixed camera layouts, i.e., the camera configurations are identical during the training and inference stage, lacking the ability to perform the MVC task with images from arbitrary dynamic camera settings.

Simultaneously, noteworthy advancements have been accomplished in the domain of multi-view perception (MVP). Specifically, research works [55, 15, 29, 17, 30, 35, 26] focus on lifting image-level features of MV images to the cohesive scene-level space, followed by the execution of specific perception tasks using the derived scene-level features. Despite the concise architecture design and the promising performances on the MVP task, it is infeasible to naively adopt existing MVP methods without considering the specific challenges of the MVC settings because contemporary MVP approaches necessitate the assumption of stable and fixed camera layouts, which may not hold in MVC scenarios. Moreover, the inherent philosophies of employing multi-camera layouts in MVC and MVP are different. The MVP tasks require multi-view cameras with limited overlapped field of view (FOV) to provide a 360^∘ FOV of the scene, while MVC tasks rely on multi-view settings that the FOV of each camera has significant overlap with the others to address occlusion and scale variation challenges.

In this work, we incorporate the recent advanced ideas in MVP and propose an innovative MV learning framework called CountFormer to extend the applicability and scalability of the existing MVC approaches. CountFormer is founded upon a primary consideration, where it shall be adequate to process images captured with arbitrary dynamic camera layout settings and robust enough to alleviate the performance drop caused by extrinsic parameter perturbation. To accomplish this objective, a feature lifting module is first proposed to lift the image-level feature in 3D space for each view. Specifically, the 3D scene is voxelized into individual voxels, and features corresponding to each voxel and camera view are obtained using a deformable attention mechanism [98]. Since the feature lifting module does not necessitate the flat ground assumption, the CountFormer is more suitable to deal with challenging situations such as congested crowds or uneven terrain. In addition, the integration of the attention mechanism enhances its robustness against fluctuations in camera extrinsic parameters, which is particularly beneficial in practical settings where such perturbations are inevitable [79].

Subsequently, an MV volume aggregation module is introduced to attentively fuse the MV volume features to generate the comprehensive scene-level volume representation, where the blending weights are estimated by implicitly joint considering the voxel features and the geometry property of the cameras, ensuring that both the visual features and the geometric information are considered during the fusion process, leading to more accurate and robust results than previous methods [86, 90, 89, 95, 91]. Due to the sophisticated design, the aggregation module is capable of efficiently handling a dynamic number of volume features, allowing the CountFormer to adapt to scenarios where the number of cameras may vary, ensuring robust performance in various applications. Afterwards, CountFormer employs 3D convolution operators to estimate the 3D scene-level density map from the aggregated volume representation.

Figure 1: Framework of the CountFormer. The Image Encoder extracts multi-view and multi-level features (MVML) from the multi-view images of the scene. Image-Level Camera Embedding Module fuses camera intrinsic and extrinsic with the MVML features. The elaborate Cross-View Attention Module, a sophisticated attention component, transforms the image-level features into scene-level volume representations. Besides main components, a 2D Density Predictor is used to estimate the image space density, 3D Density Predictors are employed to regress for the 3D scene-level density, and a simple feature pyramid network fuses the multi-scale voxel features.

To enhance the robustness and representation capability of CountFormer in handling arbitrary dynamic camera layouts, the camera encoding strategy is incorporated, which involves implicitly encoding the extrinsic and intrinsic camera parameters. Specifically, the camera information is implicitly encoded into the volume query that is used to lift the corresponding image-level features in 3D space. Additionally, the camera information is also encoded into the corresponding image-level features. By incorporating the camera information into both the volume query and image-level features, CountFormer effectively integrates the camera-specific characteristics into its processing pipeline, leading to improved performance in various camera configurations and scene understanding tasks.

In summary, CountFormer is highly versatile and scalable, making it well-suited and efficient for usage in real-world scenarios, and our main contributions are as follows:

1. We creatively design a revolutionary multi-view counting (MVC) framework, called CountFormer, which is the first attempt to solve the 3D MVC problem to fit a real-world environment.

2. A feature lifting module and an MV volume aggregation module are conceived to transform the MV image-level features w.r.t arbitrary dynamic camera layouts into a unified scene-level volume representation.

3. We present an effective strategy to embed the camera parameters into the image-level features and the volume query, facilitating accurate and adaptable representation among diverse camera setups.

2 Related Work

Due to the extensive scope of related works, we only discuss works that are strongly related to our framework.

Single-View Counting (SVC). Since the work of [84], density estimation has been the main paradigm for crowd counting. Subsequent works, such as [48, 85, 50, 60, 40, 93, 3, 1, 61, 39, 59, 21, 13, 25, 37] continue improving performance by designing more powerful model structures that capable of learning multi-scale representation and perspective-free feature detection. Simultaneously, another line of works [16, 52, 28, 8, 74, 5, 70, 63, 83, 44] adopt the insight from semantic image segmentation to exploit the encoder-decoder architecture to facilitate the multi-scale learning capability. Recently, by adopting the transformer architecture[68, 7], works such as [11, 66, 64, 33, 73, 32, 53, 78, 9, 54] substantially advance performances because of their remarkable representation and generalization capability. In addition to the intricate structures, [49, 43, 80, 12, 19, 97, 57] advances the progress by optimizing the training process or improving the cross-scene generalization ability. Despite the promising performance, SVC methods encounter challenges that necessitate resolution, including effectively dealing with scale variation, mitigating occlusion difficulties, and handling congested crowds.

Multi-View Perception (MVP). The pioneering research [55] introduces a novel approach that involves the transformation of multi-view multi-level features into a cohesive scene-level space, followed by the execution of specific perception tasks using the derived scene-level features. Subsequently, certain studies [15, 34, 47, 27] enhance the framework by incorporating sophisticated fundamental components. For instance, these studies utilize the transformer structure as the foundational framework [77], employ a robust perception head [76, 35], integrate multiple modal sensors to enhance performance [47, 34], and leverage the attention mechanism to effectively encode image-level features to the scene-level space [29, 96]. Furthermore, researchers tackle the task of 3D semantic occupancy prediction by employing a voxelization technique to convert the scene into discrete voxels and subsequently conduct 3D semantic segmentation to retrieve semantic features for each voxel [75, 94, 30, 18, 71, 67]. Despite the solid theoretical foundation and the extensive experimental validation of their effectiveness, naively transferring existing MVP approaches to solve the MVC task proves to be difficult, where the biggest challenge remains that MVC necessitates the capability of handling arbitrary dynamic camera layouts, which is infeasible for existing methods.

Multi-View Counting (MVC). Pioneering works [86, 90, 91] propose to transform each camera view’s image-level features to the scene’s ground plane and then fuse these features to estimate the scene-level density map. To further improve the performance, [87, 88] consider individuals’ variable height in the 3D environment by introducing multi-height ground planes along the z-axis. Although demonstrating promising performance, the strong assumption limits its applicability and scalability. Specifically, the feature transform module [86, 90, 89, 91, 87, 88] necessitated a flat ground assumption, which is not guaranteed in real-world situations. Furthermore, the fusion module’s blending weights are exclusively determined by the geometric position of the cameras [86, 90, 91, 89], ignoring the critical semantic features. However, these approaches seem counterintuitive, as occlusions and other crucial information related to the scene are primarily encoded in the semantic features. Alternatively, [87, 88] attempt to achieve reasonable fusion results by jointly considering geometric and semantic features, while these approaches come at the expense of limiting the flexibility and adaptability of the methods to handle dynamic camera layout settings.

3 Methodology

3.1 Image Encoder

The image encoder aims to extract multi-view and multi-level (MVML) features $\left\{\mathcal{F}_{\mathrm{n}}^{\mathrm{l}}\right\}_{\mathrm{n}\leq\mathrm{N},~{}\mathrm{l}\leq\mathrm{L}}$ from multiple synchronized captured images $\left\{\mathcal{I}_{\mathrm{n}}\right\}_{\mathrm{n}\leq\mathrm{N}}$, where varying $\mathrm{N}$ is the number of views, and $\mathrm{L}$ indicates the maximum feature pyramid level. Each image $\mathcal{I}_{\mathrm{n}}$ is first fed into a vision transformer [46] to extract multi-level features. Subsequently, considering the significance of multi-scale feature fusion, the feature pyramid network [36] is adopted to aggregate and dispatch the multi-scale features to enrich the multi-scale representation and contextual information encoding capability, which are also employed by extensive research works [14, 92, 27, 17, 24].

3.2 Image-Level Camera Embedding

The camera parameters play a key role in transforming image-level features into scene-level representation. Existing approaches tend to employ channel-wise camera encoding to differentiate features across different views [17, 29, 45, 26, 51, 38]. However, the existing strategy proves inefficient when dealing with the challenging CVCS dataset [91], which includes tens of thousands of diverse camera configurations. In this study, we devise both channel-wise and spatial-wise camera encoding to facilitate learning from MV images with significantly varied camera layouts. Formally, given the extrinsic parameters ${\mathrm{M}}\in\mathbb{R}^{3\times 4}$ representing transformation from scene space to camera space, the intrinsic parameters $\mathrm{K}\in\mathbb{R}^{3\times 3}$, and the image-level augmentation matrix $\mathrm{A_{c}}\in\mathbb{R}^{2\times 3}$, we flatten, concatenate, and extend the $\{\mathrm{M,K,A_{c}}\}$ to build the spatial-aligned camera-parameter vector $\xi\in\mathbb{R}^{27\times\mathrm{H}\times\mathrm{W}}$. Then we generate the positional encoding $\mathcal{{P}}_{\rm{c}}\in\mathbb{R}^{2\times\mathrm{H}\times\mathrm{W}}$ defined with $\mathcal{P}_{\rm{c}}(\cdot,\mathrm{u,v})=[\nicefrac{{\mathrm{u}}}{{\mathrm{W}}},\nicefrac{{\rm{v}}}{{\mathrm{H}}}]^{\mathrm{T}}$. Suppose the feature map is $\mathcal{F}\in\mathbb{R}^{\mathrm{C}\times\mathrm{H}\times\mathrm{W}}$, then the image-level camera embedding is performed with

$$\mathrm{E}_{\rm{c}}\Big{(}\mathcal{F},\{\mathrm{M,K,A_{c}}\}\Big{)}=\mathcal{F}\otimes\phi_{\rm{c}}\Big{(}[\xi,\mathcal{P}_{\rm{c}}]\Big{)},$$ (1)

where $[\cdot]$ denotes tensor concatenation, $\phi_{\rm{c}}$$:\mathbb{R}^{29}\rightarrow\mathbb{R}^{\mathrm{C}}$ represents a Multi-Layer Perceptron (MLP) that aggregates the positional encoding and the camera parameters, and $\otimes$ remains a binary operator, such as the Hadamard Product [23] or the widely-used element-wise addition.

3.3 Multi Scale Transformers

The multi-scale transformers aim to generate the multi-scale volume representations $\{\mathcal{V}_{1},\mathcal{V}_{2},\cdots,\mathcal{V}_{\mathrm{L}}\}$ from the MVML features $\left\{\mathcal{F}_{\mathrm{n}}^{\mathrm{l}}\right\}_{\rm{n}\leq\mathrm{N},l\leq\mathrm{L}}$, where $\mathrm{N}$ is the number of employed view and $\mathrm{L}$ refers the maximum feature pyramid level. To accomplish this objective, for each level $\mathrm{l}$, a CountFormer is dedicated to lift the MV image-level features $\{\mathcal{F}^{\mathrm{l}}_{\mathrm{n}}\}_{\mathrm{n}\leq\mathrm{N}}$ to the scene-level volume representation $\mathcal{V}_{\mathrm{l}}$.

Specifically, the CountFormer consists of multiple encoder layers, each following the standard design principles of transformers [68], with three specialized components, i.e., the learnable volume query, the cross-view attention, and a computationally efficient 3D convolution that replaces the self-attention mechanism. To elaborate further, the volume query is used as the initial query for each camera view, which is then combined with camera extrinsic and intrinsic parameters to create the view-dependent query. The primary purpose of the cross-view attention module is to lift the image-level features to volume features for each camera view and then aggregate the MV volumes to produce a comprehensive scene-level volume representation. In detail, it comprises three indispensable components, i.e., volume-level query embedding, the feature lifting module, and the MV volume aggregation module. Note that, in the following discussion, the subscripts $\rm{l}$ and $\rm{n}$ may be omitted to keep simplicity.

Volume Query Representation. The scene is discretized into voxels with shape $\mathrm{Z\times Y\times X}$, and assigned with a group of volume-shaped learnable parameters $\mathcal{Q}\in\mathbb{R}^{\mathrm{C\times Z\times Y\times X}}$ as the queries, where $\mathrm{X,Y,Z}$ are the spatial shape of the volume and $\mathrm{C}$ governs the hidden dimension. Specifically, the query $\mathcal{Q}^{\mathrm{p}}\in\mathbb{R}^{\mathrm{C}}$ located at $\rm{p}=\rm{(d,h,w)}$ is responsible for the corresponding voxel in the volume, each voxel in the volume corresponds to a real-world size with $\mathrm{s}$ meter, and the center of the volume is aligned with the origin point in the 3D scene coordinate system. In comparison to existing approaches [86, 95, 87] that utilize Inverse Perspective Map** (IPM) [2] strategy or Spatial Transformer Networks (STN) [20] module for deriving the volume representation of the scene, the query-based attention paradigm provides greater scalability, does not rely on the flat ground assumption, and requires less hyperparameter tuning, making it a more practical approach.

Volume-Level Camera Embedding. Existing MVP approaches [67, 18, 77, 29, 96, 35] consider $\mathcal{Q}$ as the universal query and do not take into account the camera’s intrinsic and extrinsic parameters, which is comprehensible since MVP task comprises a fixed and stable camera layout [4, 65], e.g., the golden MVP benchmark nuScene [4] comprises 6 stable surrounding cameras to provide 360^∘ FOV. Nevertheless, the view-agnostic global query $\mathcal{Q}$ is inadequate in addressing the complexities posed by challenging surveillance environments, where significantly diverse dynamic camera layouts are inevitable in such scenarios. For instance, the CVCS benchmark [91] consists of images from tens of thousands of camera views. To tackle this problem, we develop a crucial volume-level camera embedding module that utilizes a similar strategy as the image-level camera embedding, which encodes the camera parameters $\{\mathrm{M,K}\}$ and volume-level augmentation matric $\rm A_{v}\in\mathbb{R}^{2\times 3}$ into the versatile $\mathcal{Q}$ to create the view-dependent volume query with

$$\small\mathrm{E}_{\mathrm{v}}\Big{(}\mathcal{Q},\{\mathrm{M,K,A_{v}}\}\Big{)}=\mathcal{Q}\otimes\phi_{\mathrm{v}}\Big{(}[\xi,\mathcal{P}_{\mathrm{v}}]\Big{)},$$ (2)

where $\phi_{\mathrm{v}}:\mathbb{R}^{30}\rightarrow\mathbb{R}^{\mathrm{C}}$ fuses the positional encoding with the camera parameters, $\mathcal{P}_{\mathrm{v}}\in\mathbb{R}^{3\times\mathrm{Z\times Y\times X}}$ stands for the positional encoding with $\mathcal{P}_{\mathrm{v}}(\cdot,\rm{d,h,w})=[\nicefrac{{\rm{d}}}{{\mathrm{Z}}},\nicefrac{{\rm{h}}}{{\mathrm{Y}}},\nicefrac{{\rm{w}}}{{\mathrm{X}}}]^{\mathrm{T}}$, and $\xi$ takes similar definition as in Equation 1.

Feature Lifting Module. Considering the need for supporting dynamic camera layouts and the complex deployment environment in the MVC task, employing deformable attention [98, 29] would be more suitable than relying on the IPM paradigm for constructing the feature lifting module. Employing deform attention to lift image feature to 3D scene space has been widely used in multi-view tasks [29, 22, 72, 30]. In this work, the CountFormer also adopts this method but replaces the global versatile query with a view-dependent query to address the complexities in surveillance environments. Mathematically, given the camera dependent query $\mathcal{Q}$ encoded with Equation 2, and the corresponding image-level feature $\mathcal{F}$, for each voxel query $\mathcal{Q}^{\mathrm{p}}$ located at $\mathrm{p=(d,h,w)}$, one may first recover the corresponding 3D location $\mathrm{(x^{\prime},y^{\prime},z^{\prime})}$ w.r.t. the scene with

$$\mathrm{x^{\prime}=s\cdot(w-\nicefrac{{\mathrm{X}}}{{2}});~{}~{}y^{\prime}=s\cdot(h-\nicefrac{{\mathrm{Y}}}{{2}});~{}~{}z^{\prime}=s\cdot(d-\nicefrac{{\mathrm{Z}}}{{2}})},$$ (3)

then obtain the homogeneous reference point $\mathrm{p=(u,v)}$ by projecting the 3D scene-level point $\mathrm{(x^{\prime},y^{\prime},z^{\prime})}$ according to the intrinsic $\mathrm{K}$ and extrinsic $\mathrm{M}$ as

$$\mathrm{[u,v,1]^{\mathrm{T}}=\mathrm{K}\cdot\mathrm{M}\cdot[x^{\prime},y^{\prime},z^{\prime},1]^{T}},$$ (4)

and finally, perform the sophisticated deformable attention mechanism at the projected location $\rm(u,v)$.

Multi-View Volume Aggregation. The MV volume aggregation module aggregates the MV volume representations to produce a comprehensive volume of the scene. Existing fusion approaches can only deal with stable and fixed camera layouts [29, 86, 90, 88, 87], or are compromised in conducting the fusion process by only considering the geometric position of the cameras and ignoring the critical semantic features [90, 86, 91], or assume that different views contribute equally to the scene representation [29, 17, 27]. In summary, all existing approaches lack the necessary scalability to effectively handle dynamic camera layouts. In this work, a compact yet effective attention mechanism is devised to aggregate the MV volume representations $\{\mathcal{V}_{\rm n}\}_{\rm n\leq{\mathrm{N}}}$ with

$$\small\mathcal{V}=\sum\nolimits_{\rm n}\mathcal{W}_{\rm n}\odot\mathcal{V}_{\rm n},$$ (5)

where $\mathcal{W}_{\rm n}\in\mathbb{R}^{1\times\mathrm{Z\times Y\times X}}$ denotes the attention weight, $\odot$ refers the element-wise product operation, and $\mathcal{V}_{\rm n}\in\mathbb{R}^{\mathrm{C\times Z\times Y\times X}}$ is the volume feature w.r.t. the $\rm n_{\mathrm{th}}$ view. Moreover, instead of employing the computationally heavily vanilla self-attention [68], we tend to estimate the weight of the ${\rm p}_{\mathrm{th}}$ voxel in the $\rm n_{\mathrm{th}}$ view $\rm\mathcal{W}_{n}^{p}$ much cheaper,

$$\mathcal{W}_{\rm n}^{\rm p}=\begin{cases}\nicefrac{{\exp\{\phi(\mathcal{Q}_{\rm n}^{\rm p})\}}}{{\sum_{\rm k\in\mathbf{S}}\exp\{\phi(\mathcal{Q}_{\rm k}^{\rm p})\}}},&\mathrm{if}~{}{\rm n}\in\mathbf{S}\\ 0,&\mathrm{others}\end{cases}$$ (6)

where $\mathbf{S}$ represents the whole camera-views that voxel $\rm p$ hits, $\phi:$ $~{}\mathbb{R}^{\mathrm{C}}\rightarrow\mathbb{R}_{+}$ measures the importance of voxel, and $\mathcal{Q}_{\rm n}$ referes to the volume query with camera-encoding w.r.t. the $\rm n_{\mathrm{th}}$ view. It is worth noting that the aggregation strategy possesses significant advantages over previous methods. For instance, it is independent of the permutation of the camera views and thus is capable of handling dynamic camera layouts. Besides, the blending weights $\rm\mathcal{W}_{n}^{p}$ inherently encode the semantic contextual and geometric position, making it more practical than previous fusion strategies [91, 29, 17], especially dealing with occlusion. It shall point out that the MVP tasks necessitate the multi-cameras to provide 360^∘ FOV comprehensive perception of the scene, where marginal overlap** exists between different cameras [4, 65], making the cross-view fusion strategy not necessitates elaborated design [29, 17, 75]. In contrast, MVC tasks heavily rely on the overlap** between different camera views to address the ambiguity caused by occlusion and the scale variation resulting from perspective projection [86, 91]. As a result, the multi-view fusion strategy needs to be carefully designed to effectively integrate information from multiple views.

3.4 Density Predictor

As 3D density estimation requires more low-level features to enable the network to learn fine-grained density, we incorporate the 3D FPN [36] to perform multi-scale volume feature fusion. Formally, given multi-scale 3D volumes $\{\mathcal{V}^{\rm l}\}_{\rm l\leq\mathrm{L}}$, we upsample ${\rm l-1}_{\mathrm{th}}$ level 3D density features $\mathcal{X}_{\rm l-1}$ with 3D deconvolution layer and fuse it with $\rm l_{\rm th}$ volume representation $\mathcal{V}_{\rm l}$ as

$$\small\mathcal{X}_{\rm l}=\mathcal{V}_{\rm l}+\mathrm{DeConv}(\mathcal{X}_{\rm l-1}),$$ (7)

we apply 3D convolution layer on $\mathcal{X}_{\rm l}$ to estimate the 3D density map $\mathcal{G}_{\rm l}$, and supervise the training procedure with $\mathrm{L}_{2}~{}\mathrm{Norm}$. Considering that the high-resolution prediction remains more important, loss weight $\alpha_{\rm l}$ is employed to balance the training losses among various resolutions. Besides, as shown in Figure 1, we add the 2D density estimation task as an intermediate supervision to accelerate the training process, which may aid in gradient propagation and improve the overall training procedure. This design is reasonable because the complicated CountFormer architecture makes the gradient feedback from the 3D density supervision too long, while the intermediate 2D density estimation task elegantly improves the gradient feedback instead. To this end, the training objective of the CountFormer is

$$\small\mathcal{L}=\lambda\left|\mathcal{H}-\bar{\mathcal{H}}\right|_{2}+\sum\nolimits_{\rm l}\alpha_{\rm l}\left|\mathcal{G}_{\rm l}-\bar{\mathcal{G}_{\rm l}}\right|_{2},$$ (8)

where $\left|\cdot\right|_{2}$ is the $\mathrm{L}_{2}~{}\mathrm{Norm}$, $\lambda$ trades between the 2D density supervision and 3D density estimation, $\mathcal{H}$ and $\mathcal{G}_{\rm l}$ denote the GT 2D and 3D density, and $\bar{\cdot}$ is the corresponding prediction. Compared to the complicated loss design strategy adopted in [86, 90, 87, 88, 89], the training objective in CountFormer is much simpler yet more straightforward to tune.

4 Experiments

4.1 Experiment Settings

We leverage all existing MVC datasets to evaluate the effectiveness of the CountFormer, including CityStreet [86], PETS2009 [10], DukeMTMC [58], and CVCS [91]. To make a fair comparison, following conventional works [86, 90, 89, 87, 88], we employ the mean absolute error ($\mathrm{MAE}\downarrow$) and normalized mean absolute error ($\mathrm{NAE}\downarrow$) as the evaluation criteria to quantify the counting performances on both image-level and scene-level. Our code and model is available at https://github.com/MandyMo/ECCV_Countformer for research purpose.

4.2 Qualitative Experiments

To better demonstrate the robustness of CountFormer in the challenging scenarios, we draw some representative samples from CityStreet and PETS2009 testing sets, as illustrated in Figure 2, i.e., in the presence of occlusion and the congested crowds. Experiments demonstrate the capability of CountFormer in dealing with occlusion. Specifically, the CVCS approach[91] encounters difficulties in accurately reconstructing the density in spatial regions obstructed by objects such as buses or trees. By employing the MV volume aggregation module, our CountFormer can alleviate this issue and make a reasonable estimation, because the aggregation module is capable of dynamic blend voxel features from all views. Moreover, both the CVCS [91] and the 3D Counting [87] demonstrate limited efficacy in managing densely populated crowds. As expected, CountFormer effectively tackled these challenges by utilizing the feature lifting module to transform image-level features to 3D volume for each view and adopting the MV volume aggregation module to attentively fuse suitable features for each voxel.

Figure 2: Qualitative Results. The figure exhibits several typical scenarios on the CityStreet (with 3 views) and PETS2009 (with 3 views) datasets, including occlusion and congested crowds. For each sample, the multi-view images, the ground truth scene-level density and estimated density from CVCS method[91], 3D Counting approach[87], and the CountFormer are presented in the bird’s eye view, respectively.

Figure 3: Qualitative Results. The figure visualizes 3 challenging scenarios on the CVCS benchmark. Regarding each sample, the visualization includes the multi-view images (with 5 views), ground truth density, density obtained with the MV volume aggregation module, and density estimated without this module.

To gain a better understanding of the effectiveness of the feature aggregation module in CountFormer, we select a 3D reference point for each sample and report the fusion weight of each point across all views (the projected reference points are marked as $\bigstar$, $\square$, $\vartriangle$, and $\lozenge$ in Figure 2). As Table 1 reports the attention weights of each 3D reference point, CountFormer assigns smaller weights to occluded views because the MV volume aggregation module jointly consider the geometric information and semantic feature of each view, while the CVCS approach focuses on geometric camera position only (see $\bigstar$, $\square$, and $\vartriangle$ ). Additionally, CountFormer also automatically assigns appropriate weights to views with congested crowds, enhancing the count accuracy (see $\lozenge$).

Sample	Method	Atten-Weight
		C1	C3	C4
$\bigstar$	CVCS	0.15	0.23	0.62
	CountFormer	0.40	0.51	0.09
$\square$	CVCS	0.67	0.22	0.11
	CountFormer	0.18	0.33	0.49

(a) Attention Weights on CityStreet.

Sample	Method	Atten-Weight
		C1	C2	C3
$\vartriangle$	CVCS	0.38	0.41	0.21
	CountFormer	0.19	0.45	0.36
$\lozenge$	CVCS	0.35	0.43	0.28
	CountFormer	0.45	0.47	0.08

(b) Attention Weights on PETS2009.

Table 1: Quantitative Analysis of Attention Weights. The table presents the attention weights according to the MV features on CityStreet and PETS2009, where the 3D reference points $\bigstar$ , $\square$, $\vartriangle$, and $\lozenge$ according each view are marked on Figure 2.

Figure 3 depicts the density maps w/ and w/o the volume aggregation module on the challenging CVCS benchmark, one may observe that, w/o the aggregation module, it tends to undercount in spatial regions that are occluded in some views (2_nd sample), the model fails to conduct exact counting at the present of exaggerated views (1_st sample), and the CountFormer may struggle to predict sharp density maps with large-scale variation (3_rd sample). However, w/ the aggregate module, CounFormer is capable of dealing with these challenges, highlighting this module’s ability to effectively fuse MV volume features, especially in situations with significant scale variations or occlusions.

(a) Histogram Comparison

Dataset	SOTA	Ours
CityStreet [86]	6.98 [90]	4.72
CVCS [91]	7.22 [91]	4.79
PETS2009 [10]	3.08 [95]	0.74
DukeMTMC[58]	0.87 [95]	0.88

(a) Quantitative Comparison

Figure 4: Comparisons with state-of-the-art (SOTA) methods. The figure presents the comparisons between [86, 90, 87, 88, 91, 95, 56, 89] and our CountFormer, where the mean absolute error (MAE $\downarrow$) is used to evaluate the performance on the CityStreet dataset[86], CVCS dataset [91], PETS2009 dataset[10], and DukeMTMC dataset[58]. For better visualization, we plot the best performance among [86, 90, 87, 88, 91, 95, 56, 89] to compare with ours on each dataset.

4.3 Quantitative Experiments

As Figure 4 briefly summarizes the comparison of CountFormer against the latest state-of-the-art (SOTA) performances, the CountFormer substantially outperforms the SOTA by a large margin on PETS2009 [10], CityStreet[86], and CVCS [91] datasets, and achieves comparable performance on DukeMTMC [58].

Specifically, on the DukeMTMC dataset[58], our method achieves competitive results among newly launched approaches [86, 90, 87, 88, 91, 95, 89, 56]. Although the performance seems not overwhelming, it is reasonable considering the saturated performance [90, 88, 91, 95], the inaccurate annotation quality [86], and the non-overlap** camera view. In contrast, the PETS2009 dataset [10] contains a congested crowd distribution and overlap** views. As a result, the CountFormer significantly outperforms all existing approaches [86, 90, 87, 88, 91, 95, 56, 89] and achieves exceptional performance, reducing the scene-level $\mathrm{MAE}$ by $76.0\%$. Similarly, a similar trend can be seen in the CityStreet benchmark [86], which contains a larger crowd distribution, severe dynamic occlusions from the environment, and diverse scale variations caused by perspective projection, the CountFormer significantly outperforms all current approaches [86, 90, 87, 88, 91, 95, 56, 89, 89] in terms of $\mathrm{MAE/NAE}$ for both scene-level and single-view level, maintaining a state-of-the-art performance and reducing the scene-level MAE/NAE by nearly half. On the large-scale and most challenging MVC benchmark CVCS [91], CountFormer achieves an impressive $\mathrm{MAE/NAE}$ and sets a new state-of-the-art performance, outperforming the latest 3D counting approach [88] by $178\%$.

It is not astonishing when considering the superior architecture of the CountFormer. Specifically, (1) the Feature Lifting Module employs the attention mechanism to retrieve semantic features from image space, which proves more robust than the IPM [2] strategy that previous works adopted [90, 88] because the IPM assumes the world to be flat on a plane, which seems difficult to be guaranteed. Moreover, it remains complicated to consider the height of people in the crowd for the IPM strategy, making the IPM tend to commit misaligned feature transformation; (2) the MV Volume Aggregation Module is capable of solving the occlusion and scale variation problem by picking features from the appropriate view, yet contemporary works [91] turn to fuse multi-view features based on the distance of the IPM plane to the camera while ignoring the semantic feature themselves. While distance prior is capable of solving the scale variation dilemma, it seems powerless when dealing with occlusion since occlusion necessitates the semantic features for further validation; (3) the camera encoding strategy embeds the camera parameters into the volume query and the image features, implicitly allowing the CountFormer to model the camera extrinsic and intrinsic and facilitating the CountFormer to deal with arbitrary camera layouts.

$\mathrm{~{}L~{}}$	$\mathrm{~{}V~{}}$	$\mathrm{~{}A~{}}$	$\mathrm{~{}I~{}}$	PETS2009		CityStreet		CVCS
				$\mathrm{MAE}$	$\mathrm{NAE}$	$\mathrm{MAE}$	$\mathrm{NAE}$	$\mathrm{MAE}$	$\mathrm{NAE}$
				2.76	0.114	7.12	0.084	11.3	0.088
✓				1.31	0.054	5.65	0.071	9.51	0.074
✓	✓			1.17	0.047	5.15	0.062	6.22	0.048
✓	✓	✓		0.80	0.033	4.81	0.057	4.96	0.041
✓	✓	✓	✓	0.74	0.030	4.72	0.058	4.79	0.039

Table 2: Ablation Study. The table presents the ablation results w.r.t. various combinations of the critical components, where A adopts the MV Volume Aggregation Module, A simply averages the MV volume for each voxel, I and  I  denotes injecting camera-encoding into image-level features or not, V and  V  governs whether adopting volume-level camera encoding, L refers to the Feature Lifting Module, and  L  naively employs the IPM to transform image features to 3D space.

4.4 Ablation Study

In this section, various ablation experiments are conducted to comprehend the merit of the CountFormer competently.

Firstly, we evaluate the impact of the critical designs that constitute the CountFormer. As Table 2 summarizes the ablation results, adopting the Feature Lifting Module substantially improves the performances on the PETS2009 and CityStreet datasets because PETS2009 and CityStreet comprise fixed camera layouts, making the Feature Lifting Module adequate to lift image features to volume representation and favoring the volume queries to encode the camera layouts implicitly, overwhelming the boosting of the performances.

Method	$\mathrm{~{}L~{}}$	$\mathrm{~{}V~{}}$	$\mathrm{~{}A~{}}$	PETS2009		CityStreet		CVCS
				$\mathrm{MAE}$	$\mathrm{NAE}$	$\mathrm{MAE}$	$\mathrm{NAE}$	$\mathrm{MAE}$	$\mathrm{NAE}$
3D Counting [87]				3.25	0.136	7.63	0.102	12.8	0.116
	✓			2.53	0.102	6.76	0.091	12.9	0.114
	✓	✓		2.18	0.084	6.09	0.082	8.52	0.069
	✓	✓	✓	1.29	0.051	5.54	0.074	6.21	0.046
CVCS [91]				3.81	0.142	7.43	0.101	7.27	0.061
	✓			2.79	0.108	7.14	0.098	7.41	0.063
	✓	✓		2.54	0.097	6.86	0.094	7.01	0.059
	✓	✓	✓	1.72	0.063	6.17	0.080	6.76	0.054
BEVFormer [29]	✓			3.04	0.119	7.17	0.096	9.56	0.087
	✓	✓		2.63	0.105	6.63	0.085	7.32	0.061
	✓	✓	✓	1.45	0.058	5.81	0.078	6.33	0.048

Table 3: Ablation Study. The table shows the effects of integrating critical components into existing approaches [88, 91, 29], where L, V, A, L, V, and A shares a similar definition as in Table 2.

Furthermore, it is noteworthy that the integration of the Volume-Level Camera Embedding alongside the MV volume Aggregation Module exhibits a substantial enhancement in performance when applied to the CVCS dataset, which currently stands as one of the most challenging benchmarks for MVC analysis. We believe the notable enhancement in performance attributed to the Volume-Level Camera Embedding’s explicit incorporation of camera extrinsic and intrinsic parameters, as well as the MV Volume Aggregation Module’s aptitude for selecting suitable features for each voxel. Furthermore, the introduction of camera encoding into image features yields marginal performance improvement, which is reasonable as the camera encoding complements the attention mechanism within the feature-lifting module.

#Train-View	#Test-View
	3	5	7	9
$\star$	23.7/0.178	11.0/0.081	6.24/0.047	3.88/0.029
5	26.8/0.197	11.0/0.081	8.06/0.059	4.52/0.034
$\mathrm{U}(2,11)$	24.0/0.179	11.9/0.088	6.72/0.052	4.02/0.030

Table 4: Ablation Study. The table investigates the robustness of CountFormer against various numbers of camera views on the CVCS dataset [91], where the number of image views used for training is not necessarily equal to that in the testing stage and the ground-truth count is all the people in the scene. Specifically, $\star$ indicates that the #Train-view equals the #Test-view, $5$ means the CountFormer is trained with #Train-View=5 and evaluated with various #Test-View, and $\mathrm{U}(\rm a,b)$ denotes the #Train-view uniformly sampled from $\rm a$ to $\rm b$ in each iteration.

Table 3 presents the effectiveness of the components when integrated with established MVC methodologies [87, 91] and MVP approach[29]. We may observe that the volume-level camera embedding, the MV volume aggregation module, and the feature lifting module can also substantially improve the performances of existing MVC methods [91, 87]. Additionally, experimental results on the challenging CVCS dataset demonstrate that the MV volume aggregation and camera-level volume embedding are indispensable in dealing with arbitrary dynamic camera layouts. Moreover, it is infeasible to naively adopt the established MVP architecture BEVFormer [29] without considering the specific challenges of the MVC settings. Fortunately, equipped with the proposed components, e.g., volume-level camera embedding and MV volume aggregation module, the BEVFormer achieves promising performances.

#Train-View	#Test-View	PETS2009		CityStreet
		$\mathrm{MAE}$	$\mathrm{NAE}$	$\mathrm{MAE}$	$\mathrm{NAE}$
$\star$	1	5.43	0.217	7.75	0.091
	2	2.07	0.082	6.03	0.071
	3	0.74	0.030	4.72	0.058
3	1	7.05	0.267	8.42	0.100
	2	3.46	0.139	6.71	0.080
	3	0.74	0.030	4.72	0.058
$\mathrm{U}(1,3)$	1	6.01	0.257	7.91	0.098
	2	2.56	0.991	6.29	0.074
	3	0.89	0.035	5.16	0.063

Table 5: Ablation Study. The table demonstrates the robustness of CountFormer with various numbers of cameras on the PETS2009 [10] and CityStreet [86] datasets, where $\star$ and $\mathrm{U}(\cdot,\cdot)$ take similar definition as in Table 4.

Moreover, in challenging surveillance environments, it is inevitable in practical situations that some cameras stand down, necessitating the capability of dealing with dynamic camera layouts, e.g., various numbers of camera views and cameras with significantly different views. As Table 4 and Table 5 demonstrate the performances of CountFormer with dynamic arbitrary camera layouts, CountFormer achieves encouraging performances even when some cameras are deactivated during the inference stage. By incorporating dynamic view selection during the training stage, CountFormer achieves comparable performance to models trained with a predefined number of views. This robustness in handling dynamic camera layouts contributes to the effectiveness of CountFormer in challenging surveillance scenarios.

5 Conclusion and Discussion

3D multi-view counting (MVC) is a challenging research area with many potential real-world applications. Nevertheless, there is currently no method that can solve the 3D MVC problem with arbitrary dynamic camera layouts. To address this gap, we developed a concise multi-view learning framework, CountFormer. Experimental results demonstrated that CountFormer is capable of handling challenging scenarios that single-view counting approaches struggle with. Quantitatively, CountFormer substantially outperforms all existing MVC approaches and achieves state-of-the-art performance on most MVC benchmarks. We believe that CountFormer can provide valuable insights for further research on MVC in real-world scenarios. In addition, the training process of CountFormer requires labor-intensive 3D annotations of the head point, posing challenges for deployment in real-world scenarios. However, labeling 2D head points in image space has proven to be more efficient. Therefore, it deserves to explore methods to leverage 2D annotations for training CountFormer in the future. Furthermore, Resource consumption is a significant problem that requires careful consideration in real-world scenarios. Employing more efficient attention mechanisms, using channel pruning methods, as well as adopting the quantization techniques to accelerate the inference speed is a promising work.