Optimal Rate of Kernel Regression
in Large Dimensions

Suppose we have observed $n$ i.i.d. samples $(X_{i},Y_{i})$ from a joint distribution $(X,Y)$ supported on $\mathbb{R}^{d+1}\times\mathbb{R}$. The regression problem, one of the most fundamental problems in statistics, aims to find a function $\hat{f}_{n}$ based on these samples such that the excess risk,

$\displaystyle\left\|\hat{f}_{n}-f_{\star}\right\|_{L^{2}}^{2}=\mathbb{E}_{X}\left[\left(f_{\star}(X)-\hat{f}_{n}(X)\right)^{2}\right],$

is small, where $f_{\star}(x)=\mathbb{E}[Y|x]$ is the regression function. Many non-parametric regression methods are proposed to solve the regression problem, such as polynomial splines [75], local polynomials [20, 73], the kernel methods [14, 15, 16], etc. When the dimension $d$ of data is small, these methods produce reasonable results; however, when $d$ is relatively large, the convergence rate of the excess risk can be extremely slow. What’s worse, though some additional assumptions such as low intrinsic dimensionality (that data falls into a subspace with dimension far smaller than $d$) and sparsity of features can improve the theoretical performance of certain non-parametric regression problems [2, 27], few successful real-world examples/applications have been reported. On the other hand, neural network methods have gained tremendous successes in many large-dimensional problems, such as computer vision [35, 43] and natural language processing [23]. For example, the ILSVRC competition [65] has a dataset of 1.2 million samples with a dimensionality of approximately 200K, while the pre-train dataset of the well-known language representation model, Bidirectional Encoder Representations from Transformers (BERT) [23], consists of 13 million samples with a dimensionality of approximately 400K.

Several groups of researchers tried to explain the superior performance of neural networks on large dimensional data from various aspects. However, the highly non-linear dynamic of the differential equation associated with the gradient descent/flow of training the neural network[33, 45, 63] makes the analysis on the dynamic of training the neural network notoriously hard. When the width of a neural network is sufficiently large, the training process falls into the ‘lazy regime’, i.e., its parameters/weights stay in a small neighborhood of their initial position during the training process [3, 25, 26, 48]. Since [39] observed that the time-varying neural network kernel (NNK) converges to a time-invariant neural tangent kernel (NTK) point-wisely as the width $m$ of the neural network $\rightarrow\infty$, it has been widely believed that the generalization ability of early-stop** kernel regression with NTK could be served as a proper surrogate of the generalization ability of neural networks in the ‘lazy regime’ [4, 37, 76]. Recently, a sequence of works [44, 46] further showed that the NNK uniformly converges to the NTK as the width $m\rightarrow\infty$ which rigorously justified this belief. Thus, understanding the generalization ability of the kernel regression (with respect to NTK) in large dimensions will help us understand the superior performance of (wide) neural networks.

Kernel regression (or regression over an RKHS), as a classical topic, has been studied since the 1990s. Most work imposes the polynomial eigenvalue decay assumption over a kernel $K$ (i.e., there exist constants $0<\mathfrak{c}\leq\mathfrak{C}<\infty$, such that the eigenvalues of the kernel satisfy $\mathfrak{c}j^{-\beta}\leq\lambda_{j}\leq\mathfrak{C}j^{-\beta}$ for some constant $\beta>1$) and assume that the target function $f_{\star}$ belongs to the RKHS associated with $K$ [14, 15, 51, 64]. They then showed that the minimax rate of the excess risk of regression over the corresponding RKHS is lower bounded by $n^{-{\beta}/({\beta+1})}$ and that some kernel methods (e.g., the kernel ridge regression and the early-stop** kernel regression) can produce estimators achieving this optimal rate. Thus, verifying that if an NTK satisfies the polynomial eigenvalue decay assumption and determining the eigenvalue decay rate of it becomes a natural strategy to discuss the generalization ability of the NTK ( or equivalently, the wide neural networks ) regression. When the NTK is defined on sphere $\mathbb{S}^{d}$, it is an inner product kernel. Hence, the eigenvalues of NTK can be obtained through a detailed calculation with the help of the spherical harmonic polynomials. It is shown in [10, 11] that when $d$ is fixed, the eigenvalues of the NTK defined on $\mathbb{S}^{d}$ polynomially decayed at rate $({d+1})/{d}$. When the domain is other than a sphere, [44, 46] further illustrated that the eigenvalues decay rate of NTK on any bounded open set in $\mathbb{R}^{d}$ is still $({d+1})/{d}$. Some works then claimed that the optimal tuned neural network on $\mathbb{S}^{d}$ or on any bounded open set in $\mathbb{R}^{d}$ can achieve the optimal rate $n^{-({d+1})/({2d+1})}$ [37, 44, 46].

When dimension $d$ is large, much less is known about the convergence rate of the excess risk of kernel methods. There are several works devoted to the high-dimensional setting where $n\asymp d$. For example, motivated by the linear approximation of kernel matrices in high dimensional data proposed by [40], [49] provided an upper bound on the excess risk of kernel interpolation and claimed that kernel interpolation generalizes well in high dimensions. Similar results for kernel ridge regression are proven in [52]. These results are widely interpreted as evidence of the benign overfitting phenomenon (e.g., [9, 11, 53, 67]): overfitted models can still generalize well. Building on the work of [49], the benign overfitting phenomenon has been extensively investigated in the literature, and we referred to [7, 13, 34, 62, 78] for details. There is another line of research considering the large dimensional setting where $n\asymp d^{\gamma}$ for some ${\gamma}>0$. For example, [30] studied the square-integrable function space on the sphere $\mathbb{S}^{d}$ and proved that when ${\gamma}$ is a non-integer, kernel ridge regression is consistent if and only if the regression function is a polynomial with a fixed degree $\leq\gamma$. Inspired by the techniques presented in [30], several follow-up works extended the results to different settings [1, 29, 31, 55, 56, 61]. Additionally, [24] established an upper bound for kernel methods with specific kernels when ${\gamma}$ is an integer. Surprisingly, a recent work ([12]) numerically reported a ‘periodic plateau behavior’ in Figure 5 (b) of their paper: when $\gamma$ varies within certain specific ranges, the excess risk of kernel regression decays very slowly. All these inspirational works hint that determining the convergence rate of kernel regression in large dimensions is a hard but fruitful question, and we are probably to find many new phenomena if we can determine its convergence rate.

In this paper, we consider the generalization ability of kernel regression, especially kernel regression with inner product kernel defined on sphere $\mathbb{S}^{d}$, with respect to large-dimensional data where $n\asymp d^{\gamma}$. More precisely, assuming the target function $f_{\star}\in\mathcal{H}^{\mathtt{in}}$, the RKHS associated with an inner product kernel defined on $\mathbb{S}^{d}$, we will provide a sharp convergence rate of the excess risk of kernel regression with respect to data of large dimension. We will further show that this rate is actually (nearly) minimax optimal for any ${\gamma}>0$.

Traditional technical tools for kernel regression are developed implicitly under the assumption that the dimension $d$ of the domain $\mathcal{X}$ is fixed or bounded. The recent successes of neural networks in high dimensional data urge us to investigate the convergence rate of the excess risk of the NTK regression for data with large $d$.

Suppose that we have observed $n$ i.i.d. samples $(X_{i},Y_{i}),i\in[n]$ from the model:

$$y=f_{\star}(\boldsymbol{x})+\epsilon,$$

(2)

where $X_{i}$’s are sampled from $\rho_{\mathcal{X}}$, $\rho_{\mathcal{X}}$ is the marginal distribution on $\mathcal{X}\subset\mathbb{R}^{d+1}$, $f_{\star}$ is some function defined on a compact set $\mathcal{X}$, and $\epsilon\sim\mathcal{N}(0,\sigma^{2})$ for some fixed $\sigma>0$. Denote the $n\times 1$ data vector of ${Y_{i}}$’s and the $n\times d$ data matrix of $X_{i}$’s by $\boldsymbol{y}$ and $\boldsymbol{X}$ respectively.

Let us make the following assumptions on the kernel $K$ and the candidate function class $\mathcal{B}$ throughout this paper.

Given a positive definite kernel function $K$ and a positive measure $\rho_{\mathcal{X}}$ on $\mathcal{X}$, the integral operator $T_{K}$ defined by

$\displaystyle T_{K}(f)(x)=\int K(x,y)f(y)~{}\mathsf{d}\rho_{\mathcal{X}}(y)$

is a self-adjoint compact operator. The celebrated Mercer’s decomposition theorem further assures that

$\displaystyle K(x,y)=\sum\nolimits_{j}\lambda_{j}\phi_{j}(x)\phi_{j}(y),$

(4)

where $\{\lambda_{j},j=1,2,...\}$ and ortho-normal eigen-functions $\{\phi_{j}(x),j=1,2,...\}$ are the non-increasing ordered eigenvalues and corresponding eigen-functions of $T_{K}$. After a little bit of abuse of notations, we may call $\{\lambda_{j},j=1,2,...\}$ and $\{\phi_{j}(x),j=1,2,...\}$ the eigenvalues and eigenvectors(or eigen-functions) of the kernel function $K$ as well.

Suppose that $f_{\star}\in\mathcal{H}$, a reproducible kernel Hilbert space (RKHS) [21, 41, 70] associated with a positive definite kernel function $K(\cdot,\cdot)$ defined on $\mathcal{X}$. The gradient flow of the loss function $\mathcal{L}=\frac{1}{2n}\sum_{j}(y_{j}-f(X_{j}))^{2}$ induced a gradient flow in $\mathcal{H}$ which is given by

$\displaystyle\frac{\mathsf{d}}{\mathsf{d}t}{f}_{t}(\boldsymbol{x})=-\frac{1}{n}K(\boldsymbol{x},\boldsymbol{X})(f_{t}(\boldsymbol{X})-\boldsymbol{y}),$

(5)

where $\boldsymbol{X}=(X_{1},\cdots,X_{n})$, $\boldsymbol{y}=(Y_{1},\cdots,Y_{n})^{\tau}$. If we further assume that ${f}_{0}(\boldsymbol{x})=0$, then we have

$$f_{t}(\boldsymbol{x})=K(\boldsymbol{x},\boldsymbol{X})K(\boldsymbol{X},\boldsymbol{X})^{-1}(\boldsymbol{I}_{n}-e^{-\frac{1}{n}K(\boldsymbol{X},\boldsymbol{X})t})\boldsymbol{y}.$$

(6)

This $f_{t}(\boldsymbol{x})$ is referred to as the estimator given by kernel regression stopped at time $t$.

In this section, as a warm-up, we will show that the optimal rate of kernel regression with respect to the inner product kernel is $n^{-1/2}$ when $n\propto d^{\gamma},\gamma=2,4,6,\cdots$.

We first specify the following large-dimensional scenario for kernel regression where we perform our analysis:

In this paper, we only consider the inner product kernels defined on the sphere. An inner product kernel is a kernel function $K$ defined on $\mathbb{S}^{d}$ such that there exists a function $\Phi:[-1,1]\to\mathbb{R}$ satisfying that for any $x,x^{\prime}\in\mathbb{S}^{d}$, we have $K(x,x^{\prime})=\Phi(\left\langle x,x^{\prime}\right\rangle)$. If we further assume that the marginal distribution $\rho_{\mathcal{X}}$ is the uniform distribution on $\mathcal{X}=\mathbb{S}^{d}$, then the Mercer’s decomposition for ${K}$ can be rewritten as

$\displaystyle{K}(x,x^{\prime})=\sum_{k=0}^{\infty}\mu_{k}\sum_{j=1}^{N(d,k)}Y_{k,j}(x)Y_{k,j}\left(x^{\prime}\right),$

(8)

where $Y_{k,j}$ for $j=1,\cdots,N(d,k)$ are spherical harmonic polynomials of degree $k$ and $\mu_{k}$’s are the eigenvalues of $K$ with multiplicity $N(d,0)=1$; $N(d,k)=\frac{2k+d-1}{k}\cdot\frac{(k+d-2)!}{(d-1)!(k-1)!},k=1,2,\cdots$. For more details of the inner product kernels, readers can refer to [28].

To avoid unnecessary notation, let us make the following assumption on the inner product kernel $K$.

The purpose of Assumption 4 is to keep the main results and proofs clean. Notice that, by Theorem 1.b in [32], the inner product kernel $K$ on the sphere is semi-positive definite for all dimensions if and only if all coefficients $\{a_{j},j=0,1,2,...\}$ are non-negative. One can easily extend our results in this paper when certain coefficients $a_{k}$’s are zero (e.g., one can consider the two-layer NTK defined as in Section 5, with $a_{i}=0$ for any $i=3,5,7,\cdots$).

With this assumption, we have the following lemma which is borrowed from [30].

Thanks to Lemma 3.2, we can now use Theorem 6.3 to provide an upper bound on the excess risk of kernel regression with the inner product kernel $K^{\mathtt{in}}$ in large dimensions.

Recall that the eigenvalues $\lambda_{j}$’s in (4) are of non-increasing order, while the eigenvalues $\mu_{k}$’s in (8) are not necessarily non-increasing. However, the minimax lower bound on the excess risk with respect to the RKHS is determined by large eigenvalues. Therefore, the following property of the eigenvalues $\{\mu_{k}\}_{k\geq 0}$ is crucial to determining the minimax lower bound of large-dimensional kernel regression.

We then use Theorem 6.10 to show that kernel regression with $K^{\mathtt{in}}$ achieves the optimal rate under specific asymptotic frameworks.

Notice that Theorem 3.3 and Theorem 3.5 only show the optimality of kernel regression with $K^{\mathtt{in}}$ when $n\asymp d^{\gamma}$ for $\gamma=2,4,6,\cdots$. In the next section, we will modify the existing tools for bounding the excess risk of kernel regression with $K^{\mathtt{in}}$, and show the optimality of kernel regression with $K^{\mathtt{in}}$ when $n\asymp d^{\gamma}$ for any $\gamma>0$.

We have shown that in the large dimensional setting where $n\asymp d^{\gamma},\gamma=2,4,6,\cdots$, the optimal rate of the kernel regression with $K^{\mathtt{in}}$ for large dimensional data is $n^{-1/2}$.

However, when $\gamma\neq 2,4,6,\cdots$, Theorem 6.10 can not be applied to large-dimensional kernel regression. For example, when $\gamma\in(2p,2p+1]$ for some integer $p\geq 0$, we have $\mathfrak{C}_{4}d^{p-\gamma}\log(d)\leq\bar{\varepsilon}_{n}\leq\mathfrak{C}_{3}d^{p-\gamma}\log(d)\ll n^{-1/2}$, where $\mathfrak{C}_{3}$ and $\mathfrak{C}_{4}$ are constants only depending on $\gamma$, $c_{1}$, and $c_{2}$ (see e.g., Remark C.2). However, the inequality (33) does not hold (see, e.g., Lemma C.1). Furthermore, the upper bound $n^{-1/2}$ provided by Theorem 6.3 is no longer matching the metric entropy $\bar{\varepsilon}_{n}^{2}$.

The main focus of this section is trying to determine the optimal rate for all the $\gamma>0$. To construct a minimax lower bound for regression over the unit ball $\mathcal{B}\subset\mathcal{H}^{\mathtt{in}}$, we need the following modification of Proposition 6.7.