CDC: A Simple Framework for Complex Data Clustering^††thanks: Z. Kang, X. Xie, B. Li, E. Pan are with the School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China (e-mail: [email protected]; [email protected]; bingheng86, [email protected]).

MVC methods generally focus on enhancing performance by utilizing the global consensus and complementary information among multiple views. [20] mines view-shared information via adding a sample-level contrastive module to align angles between representations. [21] uses Hilbert Schmidt Independence Criterion (HSIC) to explore underlying cluster structure shared by multiple views. [22] generates an automatic partitioning with data of multiple views via a multi-objective clustering framework. [23] achieves cross-view consensus by projecting data points into a space where geometric and cluster assignments are consistent.
Different from shallow methods, deep MVC methods learn good representations via designed neural networks. [24] applies attention encoder and multi-view mutual information maximization to capture the complementary information, consistency information, and internal relations of each view. Recently, some methods have combined the contrastive learning mechanisms to obtain clustering-favorable representations. For example, [25] performs instance-level and category-level contrastive learning to improve cross-view consistency. However, these methods are not scalable to large-scale data. To reduce the complexity, [5, 26] construct bipartite graphs between cluster centroids of $K$-means and raw data points, where the anchors are chosen randomly and fixed for subsequent learning. [27] leverages features, anchors, and neighbors jointly to construct bipartite graphs. [28] captures the view-specific and consistent information by constructing a consensus graph from view-independent anchors. Although they all have linear complexity, their performance could be sub-optimal since the pre-defined anchors are not updated according to the downstream task. Differently, we generate high-quality anchors adaptively, which is efficient and stable on complex data.

Graph clustering methods aim to group nodes based on node attributes and topological structure information. Some representation learning methods, such as Node2vec [29] and GAE [30], can be used to learn embeddings for traditional clustering techniques. However, the obtained embeddings might not be appropriate for clustering because they are not specific to learn representations of cluster-ability. MVGRL [31], BGRL [32], and GRACE [33] obtain classification-favorable representation via contrastive graph learning, but are not applicable on large-scale graph due to their computation cost of data augmentation. Although MCGC [11] is augmentation-free by regarding k-nearest neighbors as positive pairs, the algorithm has a square complexity. Other deep graph clustering methods, like SDCN [34], DFCN [35], and DCRN [36], achieve promising performance via training MLP and GNNs jointly on small-scale/medium-scale graph. MvAGC [37] has low complexity, but is not efficient owing to its anchor sampling strategy. Hence, these graph clustering methods cannot effectively and efficiently handle large-scale graph. Though S${}^{3}$GC [18] obtains promising results on extra-large-scale graph via random walk-based sampler and light-weight encoder, there is expensive computation cost of training. Our method can handle graph clustering in linear time with promising performance.

Filtered features are more clustering-favorable [38], and we apply graph filtering to remove undesirable high-frequency noise while preserving the graph’s geometric features. Similar to [11], smoothed $H$ is obtained by solving the following optimization problem:

We keep the first-order Taylor series of $H$ from Eq. (1) and apply $k$-order filtering, which yields:

where $k$ is a non-negative integer and it controls the depth of feature aggregation and smoothness of representation. In addition to learning smooth features, graph filtering is also used to unify different types of data into our framework.

We use the idea of data self-expression to capture the relations among data points, i.e., each sample can be presented as a linear combination of other data points. The combination coefficient matrix can be regarded as a reconstructed graph [11]. To reduce the computation complexity, $m$ representative samples $B\in\mathbb{R}^{m\times d}$ called anchors are selected to construct anchor graph $Z\in\mathbb{R}^{m\times N}$ [39]. However, the performance of this approach is unstable since it introduces anchors in a probabilistic way. Moreover, once the anchors are chosen, they won’t be updated, which could lead to sub-optimal performance. To get rid of uncertainty in anchor selection, we propose to learn anchors from data, i.e., anchors $B$ are generated adaptively. To guarantee the quality of anchors, we enforce that the similarity between $B$ and $H$ is preserved, i.e., $BH^{\top}=Z$. Then we formalize the graph learning problem as:

where $\beta$ is a balance parameter. To make it easy to solve, we relax the above problem to:

It has two advantages over other anchor-based methods: efficient and adaptive generation of high-quality anchors. First, existing methods often repeat many times to reduce the uncertainty in results, which is time-consuming and is not suitable for large-scale data. Second, existing methods perform anchor selection and graph learning in two separate steps. By contrast, we follow a joint learning approach, where anchors and anchor graphs will be mutually boosted by each other.
For a multi-view scenario, each view could contribute differently. Therefore, we introduce learnable weights $\{\lambda_{v}\}_{v=1}^{V}$ and achieve a consensus anchor graph $Z$ by solving the following model:

Note that we learn anchors for each view to capture distinctive information. After constructing anchor graph $Z$, $Z^{\top}\Delta Z$ can be used as input to obtain the spectral embedding for clustering in traditional anchor-based methods, where $\Delta$ is a diagonal matrix with $\Delta_{ii}=\sum_{j=1}^{N}Z_{ji}$. According to [40], the right singular vectors of $Z$ are the same as the eigenvectors of $Z^{\top}\Delta Z$. Consequently, we perform singular value decomposition (SVD) on $Z$ and then run $K$-means on the right vector to produce the final result, which needs $\mathcal{O}(m^{2}N)$ instead of $\mathcal{O}(N^{3})$.

To solve Eq. (5), we use an alternative strategy.

The adjacency graph is often sparse in real-world scenarios. Consequently, we implement graph filtering with sparse matrix techniques, which takes linear time while multiplication operation takes $\mathcal{O}(f_{1}N^{2})$ in general, where $f_{1}=\sum_{v=1}^{V}d_{v}$. Assume there are $t$ iterations in total, then the optimization of $Z$ takes $\mathcal{O}(t\operatorname{max}(m^{3},mf_{1}N))$. Specifically, all multiplications and additions take $\mathcal{O}(tmf_{1}N)$ and the inverse operation needs $\mathcal{O}(tm^{3})$. Then optimization of $B^{v}$ and $\{\lambda_{v}\}_{v=1}^{V}$ takes $\mathcal{O}(tf_{2})$ and $\mathcal{O}(tmf_{1}N)$, where $f_{2}=\sum_{v=1}^{V}d_{v}^{3}$. It is worth pointing out that anchor generation has a constant complexity, which won’t be limited by the size of the data. We perform SVD on $Z$ and implement $K$-means to obtain clustering result, which takes $\mathcal{O}(m^{2}N)$ and $\mathcal{O}(\bar{t}cmN)$ respectively, where $\bar{t}$ is the iteration number of $K$-means and $c$ is cluster number. In practice, $d\ll N$, $m\ll N$, and $t\ll N$, $c$ and $\bar{t}$ are constants, so the proposed method has a linear time complexity. Moreover, the largest space cost is $m\times N$ or $N\times d$, which means our approach has a linear space complexity.

We compare our complexity with baselines in Table I. The iteration number $t$ is omitted. The $\widehat{P}$ represents the average degree of the graph in S${}^{3}$GC. $l$ and $K$ are the number of view groups and nearest neighbors for each view group, where view group is defined as a group of multiple randomly selected views. $B$ is the batch size, remaining symbols are the same as those in the main body of CDC. It can be seen that our method has clear advantages and only suffers from feature dimension. What’s more, for high-dimensional data, dimension reduction techniques can be applied.

To show the effectiveness and efficiency of the CDC, we evaluate CDC on 10 benchmark datasets, including 6 multi-view data and 4 single-view data. More specifically, ACM and DBLP [17] are multi-view graphs with multiple relations, AMAP and AMAC [37] are multi-attribute graphs, YTF-31 [7] and YTF-400 [27] are multi-view non-graph data (YouTube-Faces); Citeseer, Pubmed [12], Products and Papers100M [42] are single-view graphs, where the latter two are from Open Graph Benchmarks [43]. The statistical information of these datasets is shown in Table II. Most notably, YTF-400 represents the largest multi-view non-graph dataset, while Papers100M is the largest graph used in the clustering task. We adopt four popular clustering metrics, including ACCuracy (ACC), Normalized Mutual Information (NMI), F1 score, and Adjusted Rand Index (ARI). A higher value of them indicates a better performance.

We compare CDC with a number of single-view methods as well as multi-view methods.
Single-view graph Baselines include MinCutPool [44], METIS [45], Node2vec [29], DGI [46], DMoN [47], GRACE [33], BGRL [32], MVGRL [31], and S${}^{3}$GC [18].

METIS uses only structural information to partition graphs. Node2vec is a well-known graph embedding algorithm based on random walks. MinCutPool and DMoN integrate spectral clustering with graph neural networks. DGI learns node representations by maximizing mutual information between patch representations and corresponding high-level summaries of graphs. GRACE, BGRL, and MVGRL are three contrastive graph representation learning methods. S${}^{3}$GC is a recent scalable graph clustering method, which uses light-weight encoder and random walk-based sampler.

Multi-view graph There are 10 baselines on multi-view graphs clustering, including SDCN [34], DAEGC [14], O2MAC [17], HDMI [48], CMGEC [24], COMPLETER [49], MvAGC [37], MCGC [11], MVGRL [31], and MAGCN [16]. The first six methods are only applicable to data with multiple graphs or multiple attributes, whereas the last four are applicable to general multi-view graph data.
Graph attention auto-encoders and GCNs are used in SDCN and DAEGC, respectively.

To get consistent embedding, CMGEC adds a graph fusion network to multiple graph auto-encoders. In O2MAC, the most informative view is selected to learn cluster representation. HDMI learns node embeddings by using high-order mutual information. MAGCN applies graph auto-encoder on both attributes and topological graphs to learn consensus representations. Through contrastive mechanisms, COMPLETER and MVGRL learn a common representation shared across multiple views and graphs. MCGC uses a contrastive regularizer to boost the quality of the learned graph. In MvAGC, high-order topological interactions are explored to improve clustering performance.

Non-graph We compare CDC with six scalable MVC methods on non-graph data, including BMVC [9], LMVSC [5], MSGL [26], FPMVS [40], EOMSC-CA [7], and FastMICE [27].

BMVC learns discrete representations and binary cluster structures jointly to integrate collaborative information. MSGL and LMVSC are two scalable subspace clustering methods. FPMVS and EMOMSC-CA are two adaptive anchor-based algorithms. The differences between CDC and them are: 1) CDC uses a similarity preservation regularizer while anchor matrices are assumed to be unitary matrices in FPMVS and EMOMSC; 2) the complexity of anchor generation in CDC is not linked with data size. FastMICE constructs anchor graphs by using features, anchors, and neighbors jointly.

Parameter setting The balance parameters $\alpha$ and $\beta$ are set as $[1e-3,1,1e1,1e3,1e4]$. The number of anchors $m$ is set as $[c,10,30,50,70,100]$. All experiments are conducted on the same machine with the Intel(R) Core(TM) i9-12900k CPU, two GeForce GTX 3090 GPUs, and 128GB RAM.