Texture Edge detection by Patch consensus (TEP)

Abstract

We propose Texture Edge detection using Patch consensus (TEP) which is a training-free method to detect the boundary of texture. We propose a new simple way to identify the texture edge location, using the consensus of segmented local patch information. While on the boundary, even using local patch information, the distinction between textures are typically not clear, but using neighbor consensus give a clear idea of the boundary. We utilize local patch, and its response against neighboring regions, to emphasize the similarities and the differences across different textures. The step of segmentation of response further emphasizes the edge location, and the neighborhood voting gives consensus and stabilize the edge detection. We analyze texture as a stationary process to give insight into the patch width parameter verses the quality of edge detection. We derive the necessary condition for textures to be distinguished, and analyze the patch width with respect to the scale of textures. Various experiments are presented to validate the proposed model.

1 Introduction

Texture has been explored for decades [32] and fruitful results are established for different type of textures: Markov random field [25] is widely used for texture synthesis. Its generative feature fits well with the randomness of certain types of textures (wood surface, sand); Lattice based method [18] is powerful in modeling highly symmetric and periodic texture, when the texton is relatively well defined (wall paper, honeycomb); Frequency/Wavelet analysis [20, 31] utilize spacial filters to vectorize textures and plays an important role in image compression [17] and texture classification and segmentation [33, 14]. We refer to [30, 25] for a comprehensive review of classical models. Textures remains to be a challenging topic, since there is no general nor precise definition of textures, and the boundaries of different textures are especially difficult to recognize.

We explore texture edge detection and segmentation. The classical Canny edge detection [5] detects edge locations by thinning the mask function, which is computed by applying thresholds to the magnitude of $|\nabla U_{0}|$. One of the most well-known variational segmentation models is the Mumford-Shah functional [23],

$$E_{MS}(U,\Gamma)=\alpha\int_{\Omega\backslash\Gamma}|\nabla U|^{2}dx+\beta\mathcal{H}^{1}(\Gamma)+\int_{\Omega\backslash\Gamma}(U-U_{0})^{2}dx.$$ (1)

Here, $\alpha$ and $\beta$ are positive parameters, $\Omega\subset\mathbb{R}$ is a bounded image domain, $U_{0}\mathrel{\mathop{:}}\Omega\to\mathbb{R}$ is the given image, and $\mathcal{H}^{1}(\Gamma)$ denotes one-dimensional Hausdorff measure of the object boundary $\Gamma$. Chan and Vese [7] proposed using level set method, and it gives very effective piece-wise constant segmentation. For textured image segmentation, some texture descriptors, such as Gabor filter, can be used with these models. Gabor filter [22] can detect localized frequency response in varied orientations and scales. Figure 1 illustrates challenges of texture edge detection. For a comprehensive review of classical texture segmentation models, we refer to [13]. In real images, there are many different types of textures and it is typically difficult to find a proper filter bank that is suitable for all types of images. More recent network based approaches with data-adaptive property is capable of accomplishing high level tasks as semantic segmentation, e.g., [3, 29, 34, 10, 19], assuming the network is well-trained.

(a)	(b)	(c)

Figure 1: Challenges of texture edge detection. (a) An given image with textures. (b) Canny edge detection. (c) Chan-Vese segmentation.

In this paper, we propose filter-free and training-free approach utilizing local patches response to capture the similarities and the differences between textures. Non-local filter [4] for image denoising averages pixel intensity values over similar image patches effectively. In [9], the authors extend this to texture synthesis algorithm, where the pixels are selected over similar image patches to regenerate textures. One of the difficulties in applying such nonlocal filter for texture edge detection is that near the boundary of texture, there may not be similar patches to give the clear boundary, different from denoising non textures edges.

We use local patch information utilizing similar responses within one texture, and different responses against different textures, and propose a Texture Edge detection method by Patch consensus (TEP). We propose a consensus based edge detection, which utilize the fact that away from the boundary often gives a clear idea about where the boundary of the texture should be located. We analyze the statistical condition for the texture edge to be detected by patch-wise similarity, and explore the relation between the patch width and the performance of the proposed method. The contributions of this paper are as follows:

• We propose a simple training-free filter-free Texture Edge detection method using patch consensus (TEP).

• We statistically analyze when the texture can be separated by the patch consensus.

• Numerical results are presented to validate the proposed model.

The paper is outlined as follows: In Section 2, the details of texture edge detection with Patch consensus (TEP) is presented. Statistical analysis of the proposed model is provided in Section 3. In Section 4, we present the algorithms and numerical implementation details, and in Section 5 various experiments with comparisons and applications are presented.

2 The proposed model: Texture Edge detection by Patch consensus (TEP)

Let the discrete image domain be $\Omega=[1,2,\dots,M]\bigoplus[1,2,\dots,N]$, and let the matrix $U\in\mathbb{R}^{M\times N}$ denote the given image, where $U[\mathbf{x}]=U[x_{1},x_{2}]$ represents the intensity value at a pixel location $\mathbf{x}\in\Omega$. We consider a square neighborhood of $\mathbf{x}$ with the width 2$r+1\in\mathbb{Z}^{+}$ to be

$$\mathcal{B}_{r}(\mathbf{x})=\{\mathbf{y}\in\Omega\mid\|\mathbf{y}-\mathbf{x}\|_{\infty}\leq r\},$$

and we denote the vector version of the image patch of $\mathcal{B}_{r}(\mathbf{x})$ to be $\vec{\mathcal{P}}(\mathbf{x})\in\mathbb{R}^{d}$ that $d=(2r+1)^{2}$. We refer to $r$ as the patch width parameter. We set the order of the entry in the vector to be column-wise from left to right, i.e., let the matrix $C$ be the $\sqrt{d}\times\sqrt{d}$ image, the transformation matrix pair $A\in\mathbb{R}^{{d}\times\sqrt{d}},B\in\mathbb{R}^{\sqrt{d}\times 1}$, where

$$A=\left(\begin{array}[]{c}1\\ 1\\ \vdots\\ 1\end{array}\right)\bigotimes I,\qquad B=\left(\begin{array}[]{c}1\\ 0\\ \vdots\\ 0\end{array}\right),\qquad\text{then}\quad\vec{\mathcal{P}}(\mathbf{x})=ACB$$

which transforms a square patch in $\mathbb{R}^{\sqrt{d}\times\sqrt{d}}$ to a vector in $\mathbb{R}^{d}$. Here $\bigotimes$ denotes Konecker product, and $I$ is $\sqrt{d}$ dimensional identity matrix.

The main idea of the proposed method, Texture Edge detection by Path consensus (TEP), is as follows. From a local patch $\vec{\mathcal{P}}(\mathbf{x})$, a patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$ is considered. We segment these patch responses $\mathcal{R}(\mathbf{y};\mathbf{x})$ to emphasize the similarities and the differences of patch responses. Then, we collect these segmentation boundaries and construct the edge function $V$ in $\Omega$.

[Step 1] For each patch $\vec{\mathcal{P}}(\mathbf{x})$, we define the patch response in a larger domain as

$$\mathcal{R}(\mathbf{y};\mathbf{x})=\frac{1}{(2r+1)^{2}}\|\vec{\mathcal{P}}(\mathbf{y})-\vec{\mathcal{P}}(\mathbf{x})\|_{2}^{2}\geq 0.$$ (2)

Here $\mathbf{y}\in\mathcal{B}_{R}(\mathbf{x})$ with $R>r$ represents the half width of the comparison neighborhood. We refer to $R$ as the large comparison region width parameter. This $\mathcal{R}(\mathbf{y};\mathbf{x})$ measures the similarity of a patch at $\mathbf{x}$ and a patch at $\mathbf{y}$. When the patches $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{P}}(\mathbf{y})$ are similar, it gives near zero value, and when they are very different, it gives a high value. For computational efficiency, we take $\mathbf{y}$ from the neighborhood $\mathcal{B}_{R}(\mathbf{x})$, but one may use $\mathbf{y}\in\Omega$.

[Step 2] To emphasize the texture differences and capture the edge information more clearly, we segment the response $\mathcal{R}(\mathbf{y};\mathbf{x})$ on $\mathcal{B}_{R}(\mathbf{x})$ using the following unsupervised multiphase segmentation model [26]:

$$E_{\text{seg}}(\chi_{i},c_{i},K|\mathcal{R})=\lambda\left(\sum_{i=1}^{K}\frac{P_{i}}{A_{i}}\right)\mathcal{H}^{1}(\Gamma)+\sum_{i=1}^{K}\int_{\chi_{i}}|\mathcal{R}(\mathbf{y};\mathbf{x})-c_{i}|^{2}d\mathbf{x}$$ (3)

where $\chi_{i}$ is the indicator function of each phase $i$ which partitions $B_{R}(\mathbf{x})=\bigcup_{i=1}^{K}\chi_{i}$, $K$ is the number of phases, $\mathcal{H}^{1}$ denotes one-dimensional Hausdorff measure, $\Gamma=\cup_{i=1}^{K}\{\partial\chi_{i}\}$ is the set of all boundaries, and $c_{i}=\int_{\chi_{i}}\mathcal{R}(\mathbf{y};\mathbf{x})\;d\mathbf{x}/\int_{\chi_{i}}d\mathbf{x}$ is the intensity average of the phase $i$. Here the scale term $P_{i}/A_{i}=\mathcal{H}^{1}(\partial\chi_{i})/\int_{\chi_{i}}d\mathbf{x}$ is the perimeter over the area of each phase $i$. This model (3) automatically finds the number of phases $K$ by a greedy algorithm. In this paper, we bound the number of phases to be $K\in\{1,2\}$, thus it finds either one or two phases within the response $\mathcal{R}(\mathbf{y};\mathbf{x})$. We define the local edge function to be

$$\mathcal{W}(\mathbf{y};\mathbf{x})=\frac{1}{2}\sum_{i=1}^{K}|\nabla\chi_{i}|.$$ (4)

This represents the edge from the point of view of patch $\vec{\mathcal{P}}(\mathbf{x})$.

[Step 3] A local response for points on the boundary of texture doesn’t give a good edge information in general, thus we use consensus and collect the segmented patches to determine the edge function $V(\mathbf{x})$, by superposing $\mathcal{W}(\mathbf{y};\mathbf{x})$ for $\forall\mathbf{x}\in\Omega$;

$$V(\mathbf{x})=\frac{1}{\mathinner{\!\left\lvert\mathcal{B}_{R}(\mathbf{x})\right\rvert}}\sum_{\mathbf{y}\in\mathcal{B}_{R}(\mathbf{x})}\mathcal{W}(\mathbf{x};\mathbf{y}).$$ (5)

This becomes a non-binary edge function $V(\mathbf{x})\mathrel{\mathop{:}}\Omega\to[0,1]$ representing the ratio of $\mathbf{x}$’s neighbors $\mathbf{y}\in\mathcal{B}_{R}(\mathbf{x})$ that voted $\mathbf{x}$ as an edge pixel. This superposition gives consensus among patch responses. Even when the texture boundary is not very clear, points away from the boundary can still give a good information about the edge location.

Figure 2: Texture Edge detection by Patch consensus (TEP). For each patch $\vec{\mathcal{P}}(\mathbf{x})$ in the given image $U$, the patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$ is computed. The patch response is segmented, and the boundary of the phases gives the local edge $\mathcal{W}(\mathbf{y},\mathbf{x})$. The edge function $V(\mathbf{x})$ is computed by the consensus of the local edge function $\mathcal{W}(\mathbf{y},\mathbf{x})$.

The flowchart of the proposed model TEP is presented in Figure 2: for each pixel $\mathbf{x}\in\Omega$ and its patch $\vec{\mathcal{P}}(\mathbf{x})$, the patch responses on a larger domain $B_{R}(\mathbf{x})$ is computed as $\mathcal{R}(\mathbf{y};\mathbf{x})$. We use unsupervised segmentation to segment the patch responses to emphasize the similarities and the differences among these patch responses. The gradient of phases is used to compute the local edges $\mathcal{W}(\mathbf{y};\mathbf{x})$. Finally, the consensus is used to get the edge map $V(\mathbf{x})$. Since we use the observer patch $\vec{\mathcal{P}}(\mathbf{x})$ as input, the proposed method is self-adaptive to the image without the need of training. This also reduces the number of hyper-parameters needed in filter based approaches. The parameters needed for TEP are the patch width parameter $r$ of $\vec{\mathcal{P}}(\mathbf{x})$, the large comparison region width parameter $R$, and one regularity parameter $\lambda$ for the unsupervised multi-phase segmentation.

3 Analytical properties of the proposed model

In this section, we statistically analyze when the texture can be separated by the patch consensus. In particular, we model the texture as random fields, derive the necessary conditions for our model to generate distinguishable patch responses for different textures in the sense of patch-wise Euclidean distance, and study the roles of the patch width parameter $r$, and the large comparison region width parameter $R$.

3.1 Texture as Stationary Random Field

Random field models the self-similarity property of the natural stochastic textures well that statistical approaches are proposed for structure-texture decomposition and image denoising [16, 36, 35]. In this paper, we model texture as a two dimensional Gaussian random field [1] defined on pixels $\Omega$, and study how the decay of correlation of the texture random field helps to identify texture boundaries from the patch responses for stochastic textures. In the context of discussing image patches as random vectors, we use calligraphic letter $\vec{\mathcal{P}}$ to denote random vector and lowercase letter $\vec{v}$ to denote a concrete vector of the same size as $\vec{\mathcal{P}}$. We start with introducing the definitions of Gaussian random field and its related properties.

Definition 1.

Let $\mathbf{x}\in\Omega\subset\mathbb{Z}^{2}$ be the pixel index. The set of random variables $\mathcal{P}=\{\mathcal{P}(\mathbf{x})\}_{\mathbf{x}\in\Omega}$ is a Gaussian random field, if $\vec{\mathcal{P}}(\mathbf{x})=[\mathcal{P}(\mathbf{x}_{1}),\mathcal{P}(\mathbf{x}_{2}),\dots,\mathcal{P}(\mathbf{x}_{d})]^{T}$ is a $d$-dimensional Gaussian random vector for arbitrary choices of indices $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{d}\in\Omega$, where $d\in\mathbb{Z}^{+}$ and $\mathcal{P}(\mathbf{x}_{i})$ denotes a Gaussian variable indexed by pixel location $\mathbf{x}_{i}$. The probability density of $\vec{\mathcal{P}}(\mathbf{x})=\vec{v}$ is given by

$$\phi(\vec{v})=\frac{1}{(2\pi)^{d/2}|\Sigma|^{1/2}}e^{-\frac{1}{2}(\vec{v}-\vec{\mu}_{p})^{T}\Sigma_{p}^{-1}(\vec{v}-\vec{\mu}_{p})},$$

where $\vec{\mu}_{p}=\mathbb{E}(\vec{\mathcal{P}}(\mathbf{x}))$ is the expectation vector and $\Sigma_{p}=\mathrm{Cov}(\vec{\mathcal{P}}(\mathbf{x}))$ is the nonnegative definite $d\times d$ covariance matrix.

A Gaussian random field is completely determined by its first and the second moments, i.e., its mean $\vec{\mu}$ and covariance $\Sigma$, and Gaussian distribution is suitable for many natural stochastic textures [36]. In this paper, we assume fast decaying of the correlation with respect to pixelwise distance $\|\mathbf{x}-\mathbf{y}\|_{2}$ and choose a squared exponential covariance function such as

$$\mathrm{Cov}(\mathcal{P}(\mathbf{x}_{1}),\mathcal{P}(\mathbf{x}_{2}))=\gamma_{p}(\mathbf{x}_{1},\mathbf{x}_{2})=\sigma_{p}^{2}\exp\left(-\frac{\|\mathbf{x}_{1}-\mathbf{x}_{2}\|_{2}^{2}}{2l_{p}^{2}}\right),$$ (6)

which makes the random field $\mathcal{P}$ stationary and isotropic, here $\sigma_{p}>0$ is the magnitude parameter and $l_{p}>0$ is the decaying rate parameter. We remark that we choose the squared exponential decaying covariance (6) for the convenience of computation, and the derivations of this section can be generalized to decaying covariance functions of any order. For textures with spatially repetitive patterns, it is natural to assume the corresponding random field to be stationary [36].

Definition 2.

A Gaussian random field $\mathcal{P}$ is called stationary, if for every $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{d}\in\Omega$ and $\mathbf{z}\in\mathbb{Z}^{2}$, the joint distribution of the Gaussian random vector $[\mathcal{P}(\mathbf{x}_{1}+\mathbf{z}),\mathcal{P}(\mathbf{x}_{2}+\mathbf{z}),\dots,\mathcal{P}(\mathbf{x}_{d}+\mathbf{z})]$ is independent of $\mathbf{z}$.

Definition 3.

A stationary Gaussian random field $\mathcal{P}$ is called isotropic, if its covariance function $\gamma_{p}(\mathbf{x},\mathbf{y})$ only depends on the relative distance of pixels $\mathbf{x}$ and $\mathbf{y}$, i.e., $\gamma_{p}(\mathbf{x}-\mathbf{y})=\gamma_{p}(\|{\mathbf{x}}-\mathbf{y}\|_{2})$.

An immediate consequence of Definition 2 is that the distribution of the $d$-dimensional image patch is independent of the choice of the patch center, i.e. $\vec{\mathcal{P}}(\mathbf{x})\sim\mathcal{N}\left(\vec{\mu}_{p},\Sigma_{p}\right)$ for all $\mathbf{x}\in\Omega$. The patch response $\mathcal{R}(\mathbf{y},\mathbf{x})$ involves the observation of two patches $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{P}}(\mathbf{y})$. The mutual distribution of the two patches follows

$$\begin{pmatrix}\vec{\mathcal{P}}(\mathbf{x})\\ \vec{\mathcal{P}}(\mathbf{y})\end{pmatrix}\sim\mathcal{N}\left(\begin{pmatrix}\vec{\mu}_{p}\\ \vec{\mu}_{p}\end{pmatrix},\begin{pmatrix}\Sigma_{p}&\Sigma_{\mathrm{c}}(\tau)\\ \Sigma_{\mathrm{c}}^{T}(\tau)&\Sigma_{p}\end{pmatrix}\right)$$ (13)

where the $d\times d$ covariance matrix $\Sigma_{\mathrm{c}}(\tau)=\mathrm{Cov}(\vec{\mathcal{P}}(\mathbf{x}),\vec{\mathcal{P}}(\mathbf{y}))$ only depends on the relative distance $\tau=\|\mathbf{y}-\mathbf{x}\|_{2}$ of pixels $\mathbf{x},\mathbf{y}$, as a consequence of Definition 3. The entries of the covariance function is given by (6), i.e.,

$$\Sigma_{\mathrm{c}}(\tau)[i,j]=\sigma_{p}^{2}\exp\left(-\frac{\tau_{i,j}^{2}}{2l_{p}^{2}}\right)$$

where $\tau_{i,j}$ denotes the relative distance of $i$’th pixel in $\vec{\mathcal{P}}(\mathbf{x})$ and $j$’th pixel in $\vec{\mathcal{P}}(\mathbf{y})$. Let $d=(2r+1)^{2}$ as in section 2, and we assume $\tau>2\sqrt{2}r$, which guarantees that $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{P}}(\mathbf{y})$ do not overlap, hence $\tau_{i,j}>\tau-2\sqrt{2}r$ for all $i,j\in[1,(2r+1)^{2}]\cap\mathbb{Z}$. This leads to an upper bound of the Frobenius norm of the covariance matrix $\Sigma_{\mathrm{c}}$:

$$\|\Sigma_{\mathrm{c}}(\tau)\|_{F}\;\;=\;\;\sqrt{\sum_{i,j=1}^{(2r+1)^{2}}\sigma_{p}^{4}\exp{\left(-\frac{\tau_{i,j}^{2}}{l_{p}^{2}}\right)}}\;\;\leq\;\;\sigma_{p}^{2}(2r+1)^{2}\exp\left(-\frac{(\tau-2\sqrt{2}r)^{2}}{2l_{p}^{2}}\right).$$ (14)

Fixing $r$, the cross term $\Sigma_{\mathrm{c}}(\tau)\to O_{d}$ as $\tau\to\infty$, where $O_{d}$ is $\mathbb{R}^{d\times d}$ null matrix. Comparing (6) and (14), the decaying rate of correlation of the image patches is consistent with the rate of the pixel-wise covariance function $\gamma(\tau)$.

The conditional distribution of $\vec{\mathcal{P}}(\mathbf{y})$ with respect to $\vec{\mathcal{P}}(\mathbf{x})$ is again multivariate Gaussian, which is fully determined by its mean and variance functions

$$\vec{\mu}_{p}(\mathbf{y};\mathbf{x}) =\vec{\mu}_{p}+\Sigma_{\mathrm{c}}^{T}(\tau)\Sigma_{p}^{-1}(\vec{\mathcal{P}}(\mathbf{x})-\vec{\mu}_{p}),$$ (15)

$$\Sigma_{p}(\mathbf{y};\mathbf{x}) =\Sigma_{p}-\Sigma_{\mathrm{c}}^{T}(\tau)\Sigma_{p}^{-1}\Sigma_{\mathrm{c}}(\tau).$$ (16)

Combining with (14), $\vec{\mu}_{p}(\mathbf{y};\mathbf{x})$ and $\Sigma_{p}(\mathbf{y};\mathbf{x})$ converge to $\vec{\mu}_{p}$ and $\Sigma_{p}$ as $\tau\to\infty$, i.e,

$$\lim_{\tau\to\infty}\|\vec{\mu}_{p}(\mathbf{y};\mathbf{x})-\vec{\mu}_{p}\|_{F}=0,\quad\lim_{\tau\to\infty}\|\Sigma_{p}(\mathbf{y};\mathbf{x})-\Sigma_{p}\|_{F}=0.$$

3.2 Characteristics of the patch response

In the following, we provide the main results of the section. In order to compute the expectation of $\mathcal{R}(\mathbf{y};\mathbf{x})$, we need the following lemma:

Lemma 1 (Expectation of quadratic form [27]).

Let $\vec{\mathcal{P}}$ be a $d\times 1$ random vector with mean $\vec{\mu}_{p}$ and variance $\Sigma_{p}$, and let $A$ be an $d\times d$ symmetric matrix. Then

$$\mathbb{E}(\vec{\mathcal{P}}^{T}A\vec{\mathcal{P}})=\vec{\mu}_{p}^{T}A\vec{\mu}_{p}+\mathrm{tr}(A\Sigma_{p})$$

where $\mathrm{tr}(\cdot)$ is the trace operator.

Theorem 1.

Let the random field $\mathcal{P}$ be defined as in Definition 1, equipped with the covariance function (6). Then the patch response $\mathcal{R}(\mathbf{y};\mathbf{x})=\frac{1}{d}\|\vec{\mathcal{P}}(\mathbf{y})-\vec{\mathcal{P}}(\mathbf{x})\|_{2}^{2}$, where $d=(2r+1)^{2}$, has expectation

$$\mathbb{E}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)=2\sigma_{p}^{2}\left(1-\exp(-\frac{\tau^{2}}{2l_{p}^{2}})\right).$$ (17)

The proof is presented in Appendix A. Theorem 1 describes the expectation of $\mathcal{R(\mathbf{y};\mathbf{x})}$ when the patch centered at location $\mathbf{y}$ is drawn from $\mathcal{P}$.

When it is not, i.e. comparing two different textures, let $\vec{\mathcal{Q}}(\mathbf{y})\sim\mathcal{N}\left(\vec{\mu}_{q},\Sigma_{q}\right)$ be another random field independent from $\mathcal{P}$, where the covariance function is given as

$$\mathrm{Cov}(\mathcal{Q}(\mathbf{x}_{1}),\mathcal{Q}(\mathbf{y}_{2}))=\gamma_{q}(\mathbf{x}_{1},\mathbf{x}_{2})=\sigma_{q}^{2}\exp\left(-\frac{\|\mathbf{x}_{1}-\mathbf{x}_{2}\|_{2}^{2}}{2l_{q}^{2}}\right)$$

for some $\sigma_{q},l_{q}>0$. If the patch $\vec{\mathcal{P}}(\mathbf{x})$ is observing $\vec{\mathcal{Q}}(\mathbf{y})$, we simply have

$$\vec{\mu}_{q}(\mathbf{y};\mathbf{x})=\vec{\mu}_{q},\quad\Sigma_{q}(\mathbf{y};\mathbf{x})=\Sigma_{q},$$

since $\mathrm{Cov}(\mathcal{P}(\mathbf{x}),\mathcal{Q}(\mathbf{y}))=0$. The expectation of the patch response is given as

$$\mathbb{E}\left(\frac{1}{d}\|\vec{\mathcal{P}}(\mathbf{x})-\vec{\mathcal{Q}}(\mathbf{y})\|_{2}^{2}\right) =\frac{1}{d}\mathbb{E}_{\mathbf{x}}\left(\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\vec{\mathcal{P}}(\mathbf{x})^{T}\vec{\mathcal{P}}(\mathbf{x})-2\vec{\mathcal{P}}(\mathbf{x})^{T}\vec{\mathcal{Q}}(\mathbf{y})+\vec{\mathcal{Q}}(\mathbf{y})^{T}\vec{\mathcal{Q}}(\mathbf{y})\right)\right)$$

$$=\frac{1}{d}\left(\vec{\mu}_{p}^{T}\vec{\mu}_{p}+\mathrm{tr}(\Sigma_{p})-2\vec{\mu}_{p}^{T}\vec{\mu}_{q}+\vec{\mu_{q}}^{T}\vec{\mu_{q}}+\mathrm{tr}(\Sigma_{q})\right)$$

$$=\frac{1}{d}\left(\|\vec{\mu}_{p}-\vec{\mu}_{q}\|_{2}^{2}+\mathrm{tr}(\Sigma_{p})+\mathrm{tr}(\Sigma_{q})\right)=(\mu_{p}-\mu_{q})^{2}+\sigma_{p}^{2}+\sigma_{q}^{2}.$$ (18)

Suppose there are two textures $\mathcal{P},\mathcal{Q}$ in $\mathcal{B}_{R}(\mathbf{x})$ while the patch in $\mathcal{B}_{r}(\mathbf{x})$ is drawn from $\mathcal{P}$. Texture edge can be detected if the quantities (17) and (18) differs, preferably significantly differs. This difference is described by

$$\mathrm{diff}(\tau) =\mathinner{\!\left\lvert\mathbb{E}\left(\frac{1}{d}\|\vec{\mathcal{P}}(\mathbf{y})-\vec{\mathcal{P}}(\mathbf{x})\|_{2}^{2}-\frac{1}{d}\|\vec{\mathcal{P}}(\mathbf{x})-\vec{\mathcal{Q}}(\mathbf{y})\|_{2}^{2}\right)\right\rvert}$$

$$=\mathinner{\!\left\lvert(\mu_{p}-\mu_{q})^{2}+(\sigma_{p}^{2}-\sigma_{q}^{2})-2\sigma_{p}^{2}\exp{(-\frac{\tau^{2}}{2l_{p}^{2}})}\right\rvert}.$$ (19)

Notice that this difference (19) is a function of $\tau$. For larger $\tau$, this separation is clearer, yet, more likely $\vec{\mathcal{P}}(\mathbf{x})$ may encounter another texture $\vec{\mathcal{Q}}(\mathbf{y})$ in a real image when $\mathbf{y}$ is far away from $\mathbf{x}$. It is rather important to search for edges in $\mathcal{R}(\cdot;\mathbf{x})$ in the region that $\tau$ is large.

3.3 Stability of the patch response w.r.t. the patch width parameter $r$

We explore how the value of the patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$ concentrates to its expectation with respect to the patch width parameter $r$. Since $\mathcal{R}(\mathbf{y};\mathbf{x})$ is a quadratic form of two Gaussian vectors, its distribution can be described by a variation of the $\chi^{2}$ distribution [2].

Theorem 2.

Let $\mathcal{P}$ be defined as in Theorem 1, then the patch response $\mathcal{R}(\mathbf{y};\mathbf{x})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}}$ follows generalized $\chi^{2}$ distribution [2] in $\vec{\mathcal{P}}(\mathbf{y})$. Assuming $1/d\|\vec{v}-\vec{\mu}_{p}\|_{2}^{2}\sim\mathcal{O}(\sigma_{p}^{2})$, the variance of the patch response becomes

$$\mathrm{Var}\left(\mathcal{R}(\mathbf{y};\mathbf{x})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}}\right)\sim\mathcal{O}(\frac{\sigma_{p}^{4}}{r^{2}}).$$

Proof.

Fix $\vec{\mathcal{P}}(\mathbf{x})=\vec{v}$, and denote $\vec{\mathcal{S}}\vcentcolon=\frac{1}{\sqrt{d}}\left(\vec{\mathcal{P}}(\mathbf{y})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}}-\vec{v}\right)$, then $\vec{\mathcal{S}}$ follows the Gaussian distribution $\vec{\mathcal{S}}\sim\mathcal{N}\left(\vec{\mu}_{*},\Sigma_{*}\right)$, where according to (16), we have

$$\vec{\mu}_{*}=\frac{1}{\sqrt{d}}\left(\mathbb{E}\left(\vec{\mathcal{P}}(\mathbf{y})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}}\right)-\vec{v}\right),\mathrm{~{}~{}and~{}~{}}\Sigma_{*}=\frac{1}{d}\left(\Sigma_{p}-\Sigma_{\mathrm{c}}^{T}(\tau)\Sigma_{p}^{-1}\Sigma_{\mathrm{c}}(\tau)\right).$$

Let $Q$ be an orthogonal matrix that diagonalize $\Sigma_{*}$, that is, $Q^{T}\Sigma_{*}Q=\text{diag}(\lambda_{1},\lambda_{2},\dots,\lambda_{d})=\Lambda$, where $\lambda_{i}>0$ are the eigenvalues of $\Sigma_{*}$. Define a new random vector

$$\vec{\mathcal{U}}=Q^{T}\Sigma^{-\frac{1}{2}}_{*}(\vec{\mathcal{S}}-\vec{\mu}_{*}),$$

here $\vec{\mathcal{U}}$ is standard Gaussian, i.e., $\vec{\mathcal{U}}\sim\mathcal{N}(\vec{0},I_{d})$. The observed patch response can be reformulated as

$$\mathcal{R}(\mathbf{y};\mathbf{x})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}} =\|\vec{\mathcal{S}}\|_{2}^{2}=(\vec{\mathcal{U}}+\vec{b})^{T}Q^{T}\Sigma_{*}Q(\vec{\mathcal{U}}+\vec{b})=(\vec{\mathcal{U}}+\vec{b})^{T}\Lambda(\vec{\mathcal{U}}+\vec{b})=\sum_{j=1}^{d}\lambda_{j}(\mathcal{U}_{j}+b_{j})^{2},$$ (20)

where $\vec{b}=Q^{T}\Sigma^{-\frac{1}{2}}_{*}\vec{\mu}_{*}$, and $\mathcal{U}_{j}$ and $b_{j}$ denote the $j$’th element of vectors $\vec{\mathcal{U}}$, and $\vec{b}$, respectively. The response (20) is a weighted sum of squares of $d$ independent Gaussian variables $(\mathcal{U}_{j}+b_{j})\sim\mathcal{N}(b_{j},1)$. Each $(\mathcal{U}_{j}+b_{j})^{2}$ follows noncentral chi-squared distribution $\chi_{\nu}^{2}(\delta)$, which is fully described by the degree of freedom $\nu$ and noncentrality parameter $\delta$, and the mean and variance of such distribution is given by $\nu+\delta$ and $2(\nu+2\delta)$. Specifically, we have $(\mathcal{U}_{j}+b_{j})^{2}\sim\chi^{2}_{1}(b^{2}_{j})$. The density of the patch response (20) in general does not have a closed form [8]. With $d=(2r+1)^{2}$, its variance becomes

$$\mathrm{Var}\left(\mathcal{R}(\mathbf{y};\mathbf{x})|_{\vec{\mathcal{P}}(\mathbf{x})=\vec{v}}\right) =\sum_{j=1}^{d}\lambda_{j}^{2}\mathrm{Var}(\mathcal{U}_{j}+b_{j})^{2}=2\sum_{j=1}^{d}\lambda_{j}^{2}(1+2b_{j}^{2})$$

$$=2\mathrm{tr}(\Lambda^{2})+4\vec{b}^{T}\Lambda^{2}\vec{b}=2\mathrm{tr}(\Sigma_{*}^{2})+4\vec{\mu}_{*}^{T}\Sigma_{*}\vec{\mu}_{*}$$

$$=\frac{2}{d^{2}}\left(\mathrm{tr}\left(\Sigma_{p}(\mathbf{y};\mathbf{x})^{2}\right)+2(\vec{\mu}_{p}-\vec{v})^{T}\Sigma_{p}(\mathbf{y};\mathbf{x})(\vec{\mu}_{p}-\vec{v})\right)\sim\mathcal{O}(\frac{\sigma_{p}^{4}}{d}).$$

∎

In Figure 3, (a) shows a synthetic image consists of two textures $\mathcal{P}$ (left) and $\mathcal{Q}$ (right) from Brodatz texture images set. Two patches $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{Q}}(\mathbf{y})$ are marked with blue and yellow squares correspondingly. (b) and (c) show two patch responses $\mathcal{R}(\cdot;\mathbf{x})$ and $\mathcal{R}(\cdot;\mathbf{y})$. The brightness is proportional to the value of the patch responses. This shows that with a suitable patch width parameter $r$, the texture edge is clearly emphasized, which is consistent with the analysis in section 3.2. The contrast of two textured regions clearly indicates the edge location.

(a)	(b)	(c)

Figure 3: (a) Synthetic texture image with two textures $\mathcal{P}$ (left) and $\mathcal{Q}$ (right). Two patches $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{Q}}(\mathbf{y})$ are marked with blue and yellow. (b) and (c) show two patch responses $\mathcal{R}(\cdot;\mathbf{x})$ and $\mathcal{R}(\cdot;\mathbf{y})$ respectively. Note that the texture edge is clearly emphasized with a suitable patch width parameter $r$.

The edge function $V$ is given by the consensus of many patch responses. For accurate edge detection, these responses from different observer patches should be consistent, that is many patch responses should recognize there is an edge. This can be measured by the distribution of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$, the expected response from the perspective of patch $\vec{\mathcal{P}}(\mathbf{x})$. In Figure 4 (a), the histograms of the two textures in Figure 3 (a) are presented. The intensity values of the two textures are heavily overlapped which indicates the challenges of using intensity based method to detect the texture boundaries. By using the patch based consensus, Figure 4 (b) and (c) show that as the patch width parameter increases, the more concentrated the expectation becomes. This is consistent with Theorem 2, thus hel** to distinguish two textures. Figure 4 (b) and (c) show the estimated distribution of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$, (b) is assuming the pixel $\mathbf{x}$ is from a random field $\mathcal{P}$, and (c) is assuming the pixel $\mathbf{x}$ is from a random field $\mathcal{Q}$. Note that $\mathbb{E}_{\mathbf{x}}\left(\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)\right)$ is given from (17) in Theorem 1, if $\mathbf{y}$ is equipped with $\mathcal{P}$, and from (18), if equipped with $\mathcal{Q}$. Two sets of the distributions are concentrated around estimated expectations, which are computed from the pixel-wise means and variances of the texture images. We observe the concentration effect with a larger concentration rate, which is due to the variance difference of two textures. In particular, we have $\sigma_{p}>\sigma_{q}$, and according to Theorem 2, the variance of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$ is $\mathcal{O}(\sigma_{p}^{4}/r^{2})$ for Figure 4 (b) and $\mathcal{O}(\sigma_{q}^{4}/r^{2})$ for Figure 4 (c). Neither of the textures $\mathcal{P}$ and $\mathcal{Q}$ are strictly stationary nor isotropic, yet our model well-describes the behavior of the patch response.

(a)	(b)	(c)

Figure 4: (a) The intensity histograms of two textures in Figure 3 (a). (b) Estimated distributions of the patch response $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$ as $r$ increases, assuming the patch centered at $\mathbf{x}$ is sampled from $\mathcal{P}$, and $\mathbf{y}$ from $\mathcal{P}$ or $\mathcal{Q}$ as indicated in the legend. (c) Same as (b) assuming the patch centered at $\mathbf{x}$ is sampled from $\mathcal{Q}$.

3.4 The patch width parameter $r$ and edge detection

The quality of edge detection depends on the intensity contrast of the patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$. This contrast is given by the responses observing $\mathcal{P}$ or $\mathcal{Q}$ by the patch centered at $\mathbf{x}$, and the regularity of $\mathcal{R}(\mathbf{y};\mathbf{x})$ is related to the choice of $r$.

For two texture separation, we first assume $R$ is chosen that $\mathcal{B}_{R}(\mathbf{x})$ contains two different textures $\mathcal{P}$ and $\mathcal{Q}$ with a fixed pixel $\mathbf{x}$ away from any texture boundary. We use the squared Hellinger distance [11] of two probability density functions $f_{1},f_{2}$ to compare the two different patch responses:

$$\mathcal{H}^{2}(f_{1},f_{2})=1-\sqrt{\langle f_{1},f_{2}\rangle}\in[0,1],$$ (21)

here $\langle\cdot,\cdot\rangle$ denotes the inner product. Squared Hellinger distance (21) is a bounded metric that measures the similarity of the probability density functions $f_{1},f_{2}$ in terms of the overlap. In Figure 5, the blue curve indicates the squared Hellinger distance of the patch responses of observing textures $\mathcal{P}$ and $\mathcal{Q}$ from the perspective of patch $\vec{\mathcal{P}}(\mathbf{x})$ and the red curve is from the perspective of patch $\vec{\mathcal{Q}}(\mathbf{x})$. These curves represents the differences of the density functions shown in Figure 4 (b) and (c). As $r$ increases, two responses get better separated in Figure 4 (b) and (c), which is represented as the increasing value of squared Hellinger distance. The growth of two blue and red curves are different as $r$ increases, which is due to the difference in the variance of two textures $\mathcal{P}$ and $\mathcal{Q}$ in Figure 4. The horizontal dash line in Figure 5 shows a wide range of $r$ which gives the separation of two textures.

Figure 5: Change of distance of distribution function of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}(\mathcal{R}(\mathbf{y};\mathbf{x}))$ with respect to the patch width parameter $r$. Blue line assumes the observer $\mathbf{x}$ being equipped with $\mathcal{P}$, while the red line assumes $\mathbf{x}$ being equipped with $\mathcal{Q}$. The horizontal dashed line indicates the required minimal distance for two textures to be distinguished by the segmentation model.

In practice, the patch width parameter $r$ only needs to meet segmentation requirement of one of two adjacent textures.

4 Numerical Details

We summarize the proposed method in Algorithm 1 which includes following modifications for an efficient computation.

Input : The given image $U$, the patch width parameter $r$, the large comparison region width parameter $R$ , the regularity parameter $\lambda$ for the segmentation model (3), and the parameter $\delta$ for modification (22).

1 Initialize $V$ as a zero matrix of the size of $U$;

2 for $\mathbf{x}\in\Omega$ do

3       for $\mathbf{y}\in\mathcal{B}_{R}(\mathbf{x})$ do

4             Compute $\mathcal{R}(\mathbf{y};\mathbf{x})$ in (2), and modify to $\hat{\mathcal{R}}(\cdot;\mathbf{x})$ as in (22);

5            

6       end for

7      Compute $\mathcal{W}(\cdot;\mathbf{x})$ from the segmentation (4), and modify to $\hat{\mathcal{W}}(\cdot;\mathbf{x})$ as in (23);

8       Update $V|_{\mathcal{B}_{R}(\mathbf{x})}\leftarrow V|_{\mathcal{B}_{R}(\mathbf{x})}+\frac{1}{(2R+1)^{2}}\hat{\mathcal{W}}(\cdot;\mathbf{x})$;

9      

10 end for

Output : $V$ the edge function of the given image $U$.

Algorithm 1 Texture Edge Detection by Patch consensus

First, when computing the patch response $\mathcal{R}(\cdot;\mathbf{x})$, if two points $\mathbf{x}$ and $\mathbf{y}$ are very close, i.e. $\mathbf{y}$ is inside $\mathcal{B}_{\delta}(\mathbf{x})$ for $\delta$ small, the patches $\vec{\mathcal{P}}(\mathbf{y})$ and $\vec{\mathcal{P}}(\mathbf{x})$ overlapped in most parts. This results in unwanted singularity around the center of $\mathcal{R}(\cdot;\mathbf{x})$. We remove this center singularity with a local average:

$$\hat{\mathcal{R}}(\mathbf{y};\mathbf{x})=\begin{cases}\frac{1}{\mathinner{\!\left\lvert\mathcal{B}_{R}(\mathbf{x})/\mathcal{B}_{\delta}(\mathbf{x})\right\rvert}}\sum_{\textbf{z}\in\mathcal{B}_{R}(\mathbf{x})/\mathcal{B}_{\delta}(\mathbf{x})}\mathcal{R}(\textbf{z};\mathbf{x})&\quad\text{if }\|\mathbf{y}-\mathbf{x}\|_{\infty}\leq\delta,\\ \mathcal{R}(\mathbf{y};\mathbf{x}),&\quad\text{otherwise}.\end{cases}$$ (22)

The patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$ in the subdomain $\mathcal{B}_{\delta}(\mathbf{x})$ is replaced by the average over its complement $\sum_{\textbf{z}\in\mathcal{B}_{R}(\mathbf{x})/\mathcal{B}_{\delta}(\mathbf{x})}\mathcal{R}(\textbf{z};\mathbf{x})$. In practice, we choose $\delta=5$ when the patch width parameter $r\in[10,30]$.

Secondly, when $\mathcal{B}_{R}(\mathbf{x})$ is close to, but not overlapped with, any texture edge, the patch centered at some pixel $\mathbf{y}\in\mathcal{B}_{R}(\mathbf{x})$ may still see the texture edge outside $\mathcal{B}_{R}(\mathbf{x})$. This can cause the local edge function $\mathcal{W}(\mathbf{y};\mathbf{x})$ to report a false positive edge inside $\mathcal{B}_{R}(\mathbf{x})$, and give thick and blurry edge on $V$. We make the local edge function $\mathcal{W}(\mathbf{y};\mathbf{x})$ to only respond within $\mathcal{B}_{R}(\mathbf{x})$, by the following modification

$$\hat{\mathcal{W}}(\mathbf{y};\mathbf{x})=\begin{cases}\mathcal{W}(\mathbf{y};\mathbf{x})&\quad\text{if }\mathrm{d}(\mathbf{y},\partial B_{R}(\mathbf{x}))>r,\\ 0&\quad\text{otherwise,}\end{cases}$$ (23)

where $\mathrm{d}(\mathbf{y},\partial B_{R}(\mathbf{x}))$ is the distance of pixel $\mathbf{y}$ to the boundary of $\mathcal{B}_{R}(\mathbf{x})$, i.e.,

$$\mathrm{d}(\mathbf{y},\partial B_{R}(\mathbf{x}))=\min\left\{\|\mathbf{y}-\mathbf{z}\|_{\infty}~{}|~{}\mathbf{z}\in\partial\mathcal{B}_{R}(\mathbf{x})\right\}.$$

Thirdly, we bound the number of phases to be $K\in\{1,2\}$ in the segmentation step. When $K=1$, the energy (3) reduces to the variance of the given image. The effect of the parameter $\lambda$ can be interpreted as a threshold on the segmentation model to give one or two phases. We set $\lambda$ = 0.01 to 0.05, when normalized patch response $\mathcal{R}\in[0,1]$ is used. When the given image range is $U\in[0,255]$ and the patch response is not normalize, we use $\lambda$ = 450 to 1,000. In Figure 5, the horizontal dashed line represents the distance threshold for the two textures to be separated, i.e. the segmentation model to choose $K=2$. The $\lambda$ controls the regularity for the local edge function $\mathcal{W}(\mathbf{y};\mathbf{x})$, and efficiently reduce the unwanted edge detected. With $\lambda$ fixed, textures requires different patch width parameters $r$ to find an edge (if there is one).

5 Numerical Experiments

In this section, we present numerical results exploring different aspects of the proposed model. First, Figure 6 represents the procedure of the proposed method. In the center figure, for each point $\mathbf{x}$, yellow boxes show the local patch $\vec{\mathcal{P}}(\mathbf{x})$ with $r=3$, and the blue boxes show the patch responses $\mathcal{R}(\mathbf{y};\mathbf{x})$ in $\mathcal{B}_{R}(\mathbf{x})$. For each zoomed location, we present the yellow local patch $\vec{\mathcal{P}}(\mathbf{x})$, patch response $\mathcal{R}(\mathbf{y};\mathbf{x})$ and the local edge function $\mathcal{W}(\mathbf{y};\mathbf{x})$. For $\mathbf{x}_{1}$, $\mathbf{x}_{3}$, and $\mathbf{x}_{4}$, two regions are identified and an edge is found between two textures. For $\mathbf{x}_{6}$, two edges are found separating the patch response to three regions, here two of these three regions represents the same textured region. Notice for $\mathbf{x}_{2}$ and $\mathbf{x}_{5}$, although textures are changing and patch response shows some textures, they are identified to be the same textured regions and no edges are found.

Figure 6: For various locations $\mathbf{x}$, yellow boxes show the local patch $\mathcal{P}(\mathbf{x})$ with $r=3$, and the blue boxes show the patch responses $\mathcal{R}(\mathbf{y};\mathbf{x})$ in $\mathcal{B}_{R}$. The zoomed-in images also show the local edge function $\mathcal{W}(\mathbf{y};\mathbf{x})$. For $\mathbf{x}_{1}$, $\mathbf{x}_{3}$, $\mathbf{x}_{4}$ and $\mathbf{x}_{6}$, edges are clearly found, while for $\mathbf{x}_{2}$ and $\mathbf{x}_{5}$, although patch responses show some textures, they are identified to be the same textured regions and no edges are found.

5.1 Real images with texture

We represent the texture edge detection result for real textured images, and show comparison with Canny edge detection [5]. In Figure 7, TEP finds texture and object boundaries without finding edges within textures. Zoom-in of the red and the yellow boxes in (a) are presented in (d)-(g), where (d) and (f) shows how TEP $V(\mathbf{x})$ only finds the boundary of the textures. In (d), TEP result considers the checkerboard texture as one region, and is able to detect the subtle transitions at the corner of the table. The train rail is considered as an entity, despite the track lines in (d), while, the Canny edge detection in (e) finds sharp gradient change as edges, and finds the edges of the checkerboard pattern also. In (f), notice that the shades caused by wrinkles are ignored by the proposed model, while it is captured by Canny edge detection in (g). For TEP, $r=3,R=35$, and $\lambda=1000$ are used, while for the Canny edge detection, we used $(0.04,0.1)$ for hysteresis thresholding and $\sigma=2$ for Gaussian blurring.

(a)	(b)	(c)

(d)	(e)	(f)	(g)

Figure 7: (a) The given image. (b) TEP result $V(\mathbf{x})$. (c) Canny edge detection. (d) and (e) are zoom-in of the red box, and (f) and (g) are that of the yellow box in (a). TEP edges $V(\mathbf{x})$ are red edge in (d) and yellow edge in (f). (e) and (g) show Canny edge detection in Cyan. TEP finds the texture edges without finding the edges inside one texture.

In Figure 8, first two rows (a)-(g), the worm details are understood as texture in (d). For TEP, $r=5,R=20$, and $\lambda=800$ are used, and for Canny edge detector, threshold parameters $\{0.04,0.1\}$ and $\sigma=2$ for Gaussian filter are used. In Figure 8 last row, the details of the hair is understood as texture by TEP, while Canny edge detection finds the details. For TEP, $r=5,R=30$, and $\lambda=450$ are used, and for Canny edge detector, threshold parameters $\{0.12,0.3\}$ and $\sigma=2$ for Gaussian filter are used. TEP consistently represents the region better even for textures with complicated and large scale patterns.

(a)	(b)	(c)

(d)	(e)	(f)	(g)

(h)	(i)	(j)

Figure 8: (a) The given image. (b) TEP result $V(\mathbf{x})$. (c) Canny edge detection. (d) and (e) are zoom-in of the red box, and (f) and (g) are that of the yellow box in (a). TEP edges $V(\mathbf{x})$ are red edge in (d) and yellow edge in (f). (e) and (g) show Canny edge detection in Cyan. Texture of the worm using TEP is clearly understood as texture in (d) and red edge finds the boundary of the worm. (h) The given image. (i) and (j) are zoom-in of the yellow box in (h). (i) TEP result $V(\mathbf{x})$ in yellow. (j) Canny edge detection in cyan. TEP finds hairy region as one texture.

TEP is a training-free method for texture edge detection. Yet, in Figure 9, we present images from the Berkeley segmentation dataset BSDS500, and compare with the state-of-the-art machine learning model, Edge Detector with Transformer (EDTER) [24] as an example. Since the Berkeley Segmentation dataset for edge detection was published [21], it has been a benchmark for contour detection, especially in machine learning community [34, 19, 24, 10]. These methods are trained with color images with ground-truth data provided by human experts [21] that these methods aim at object detection. On the other hand, We apply TED only on gray scale images without any a priori knowledge of the image. TEP detects local texture edges, and this is not an object detection method. Even then, in Figure 9, TEP shows good edge detection and provides comparable results to the deep learning model. In the first row images, TEP and EDTER both finds large scale region with bricks (while Canny edge detection finds details of the bricks). TEP gives different strength to the edge, some parts are weaker edges than others, while EDTER gives the same strength, since it is object oriented contour detection. In the second and third rows, TEP edge is closer to the given image, grou** different texture correctly, and TEP finds irregular texture boundary. In the last row, while TEP finds more details of the dress, EDTER is simplified, and TEP sees the texture of the flood and finds the edge of the texture, while EDTER finds the edges in the floor tiles. With texture edge detection, TEP can give comparable good edge detection results.

(a)	(b)	(c)	(d)

Figure 9: Images from BDSD500 Dataset. First column (a) are the given images. (b) TEP results. (c) Canny edge detection. (d) EDTER results.

5.2 The scale of the texture vs the patch width parameter

The patch width parameter $r$ can be adjusted to find different scales in the image. In Figure 10, we experiment with image (a) which has different scales of texture for each object. The background has the smallest texton - the smallest repeating unit in the texture. The triangle, the circle and the square all have different sizes of texton in increasing order. From image (b) to (d), the patch width parameter $r$ is increasing from $r=4,6$ to 8, and as $r$ increases, TEP sees bigger patches as a texture. In (b) with a small $r$, each texton in the square is identifies as a separate region, since it understands the texture only in the small scale that each texton within the square is understood as a separate region. In the circle while the edge of the circle is identified, within the circle it also shows some texture details. In image (d), even the big texton is captured by a large $r$, that all objects clearly shows the texture edge boundary. When $r$ is large enough, TEP ignores the fine details within the textured region.

(a)	(b)	(c)	(d)

Figure 10: (a) Synthetic texture image with different sizes of texton. For different $r$, (b) $r=4$, (c) $r=6$, (d) $r=8$, TEP finds different scale of texture. (Other parameters are fixed as $\lambda=0.02,R=20$.) When $r$ is small, only texture with small textons are identified as one region. In (b) the square, each texton is identified as one region. When $r$ is large, even the large texture is identified as one region, and clearer texture edge is found.

Figure 11 shows real example in (a), using $r=1$ for (b) and $r=7$ for (c). In (b), the starfish shape is identified but with many details, since with a small $r$, only very small scale texture is identified as one region. In (c), with a large $r$, larger textures, e.g., inside the starfish, is identified as one textured region, and only boundary of the starfish is emphasized. As the patch width parameter increase, TEP focuses on large scale structure of the given image, while grou** small details as one region.

(a)	(b)	(c)

Figure 11: (a) The given image. The TEP results are shown with (b) $r=1$, (c) $r=7$. The comparison window parameter $R=35$ is fixed for both results.

5.3 Robustness Against Noise and Multiple junctions

We test the robustness of TEP against different levels and types of noise. Figure 12 shows the TEP result against additive Gaussian noise with increasing variance, and the TEP result against increasing salt-and-pepper noise. In the first row (a), Gaussian noise with variance 0.02, 0.04, 0.06, 0.08 to 0.1 are added from the left to the right. In the second row (b), they are TEP results with parameters $r=5,R=20$, and $\lambda=0.018$ (using normalized patch response). In the third row (c), textured images with Salt and Pepper noise ranging from $10\%,20\%,30\%,40\%$, to $50\%$ are shown from the left to the right. In the forth row (d), TEP results are shown with the same parameters $r=5,R=20$, and $\lambda=0.018$ (with normalized patch response). As more noise are added, some parts of edge strength gets weaker. Otherwise, TEP shows robustness against Gaussian and Salt and Pepper noise. In Figure 12 (d) row, clear edge is detected up to 40-50% of Salt and Pepper noise.

(a)
(b)

(c)
(d)

Figure 12: First row (a): Textured image with additive Gaussian noise. From the first to the last image, Gaussian noise with variance 0.02, 0.04, 0.06, 0.08 to 0.1 are added (which corresponds to SNR = $17.01,14.55,13.30,12.48$, to $11.89$). Second row (b) shows the results of TEP showing the edge detection. Third row (c): Textured image with Salt and Pepper noise ranging from $10\%,20\%,30\%,40\%$, to $50\%$. Forth row (d) shows the results of TEP showing the edge detection. Parameters $r=5,R=20$, and $\lambda=0.018$ (with normalized patch response) are used for both set of experiments. TEP shows robustness against noise.

When detecting edges, finding sharp junctions can be challenging, e.g., due to multiple edges meeting at one point, it can easily get blurry. In Figure 13, we experiment with images with multiple junctions of various textures. TEP computes the non-binary edge function $V$ by collecting local segmentation results, that as long as the texture can be separated, multiple junction can also be identified. This is consistent with equation (19) in Section 3, the differences of average intensities help the proposed method to identify the texture boundaries. Figure 13 (b) and (d) show TEP results showing clear edges near the junction point. The experiment results show TEP well handles $N-$junction problem with $N=4$ and $N=8$. In (d), the strength of the edge, $V$ value may be not as high for some points near the junction in the center, and for a very accurate edge detection, multiple junction will impose challenges. One can further improve the sharpness of edge detection with small modifications, which we further discussed in Appendix B.

(a)	(b)	(c)	(d)

Figure 13: (a) Texture image with 4 junction. (b) TEP result of (a). (c) Texture image with 8-junction. (d) TEP result of (c). TEP well handles $N-$junction problem with $N=4$ and $N=8$.

5.4 Image segmentation using the edge function $V$ and image decomposition

Using the edge function $V$, we can design a color segmentation method, using chromaticity and brightness model [6]. We separate the given color image $\mathbf{U}_{0}\mathrel{\mathop{:}}\Omega\to\mathbb{R}^{3}$ to the brightness $U_{b}\mathrel{\mathop{:}}\Omega\to\mathbb{R}$ and the chromaticity $\mathbf{U}_{c}\mathrel{\mathop{:}}\Omega\to\mathbb{S}^{3}=\{\mathbf{x}\in\mathbb{R}^{3}\mid\|\mathbf{x}\|_{2}=1\}$:

$$\mathbf{U}_{0}=U_{b}\cdot\mathbf{U}_{c},\text{ where }U_{b}=|\mathbf{U}_{0}|\text{ and }\mathbf{U}_{c}=\frac{\mathbf{U}_{0}}{U_{b}}.$$

We use the edge function, and propose the following segmentation functional for each chromaticity and brightness components, with $\widetilde{\mathbf{U}}=\widetilde{U}_{b}\cdot\widetilde{\mathbf{U}}_{c}$,

$$\widetilde{U}_{b} =\operatorname*{argmin}_{U}\int_{\Omega}\left\{g_{\alpha}(V)|\nabla U|^{2}+\gamma_{1}|U-U_{b}|^{2}\right\}d\mathbf{x},$$ (24)

$$\widetilde{\mathbf{U}}_{c} =\operatorname*{argmin}_{\mathbf{U}}(\mathbf{U})=\int_{\Omega}\left\{g_{\alpha}(V)|\nabla\mathbf{U}|^{2}+\gamma_{2}|\mathbf{U}-\mathbf{U}_{c}|^{2}+\beta(1-|\mathbf{U}|)^{2}\right\}d\mathbf{x},$$

where $g_{\alpha}(x)\mathrel{\mathop{:}}[0,1]\to[0,1]$ is an edge indication function, $\displaystyle{g_{\alpha}(x)=\frac{1-x^{\alpha}}{1+x^{\alpha}}}$, such that $g_{\alpha}$ is strictly decreasing, and $g_{\alpha}(0)=1,~{}g_{\alpha}(1)=0$ for $\alpha>0$. In order to utilize texture edge $V$ in an effective way, $g_{\alpha}$ needs to control the smoothness of $U$ inversely proportional to the strength of $V$ within the range of $V\in[0,1]$. In application, since $V$ is generated through consensus, $1-V$ is far from zero at texture edge, we choose $\alpha<1$ to enhance the convexity of $g_{\alpha}(V)$, which creates wider region near $V=1$ thus properly control the smoothness of $U$.

The functionals (24) are minimized by considering the Euler-Lagrange equations with gradient decent with time evolution:

$$\frac{\partial\widetilde{U}_{b}}{\partial t} =\operatorname{div}(g\nabla\widetilde{U}_{b})+\gamma_{1}(\widetilde{U}_{b}-U_{b}),$$

$$\frac{\partial\widetilde{\mathbf{U}}_{c}}{\partial t} =\operatorname{div}(g\nabla\widetilde{\mathbf{U}}_{c})+\gamma_{2}(\widetilde{\mathbf{U}}_{c}-\mathbf{U}_{c})+\beta(1-\frac{1}{|\widetilde{\mathbf{U}}_{c}|})\widetilde{\mathbf{U}}_{c},$$

using finite difference scheme. We only used the brightness of the image to compute $V(\textbf{x})$. Figure 14 (a) is the given image, (b) two-phase clustering of image (a), (c) shows the segmentation result of (24) and (d) is two-phase clustering of image (c). Within each region, small scale details are removed, while the edge is kept very sharp.

(a)	(b)	(c)	(d)

Figure 14: (a) The given color image. (b) Two-phase clustering of image (a). (c) The proposed segmentation result in (24). (d) Two-phase clustering of image (c). While within each region is diffused, the texture edge is kept very sharp, clearly segmenting the image. Notice how well texture edge is found using $V$.

We present more segmentation results in Figure 15. The brightness of the image is used to compute $V(\textbf{x})$, and we used parameters $r=5,R=25$, and $\lambda=400$, with $U\in[0,255]$. For $g_{\alpha}$, $\alpha=0.2$ is used and $dt=0.1$ for evolution. The first row, the grains are identified as one texture, as well as some textures on the ground as another texture. In the second row, branches with leafs, and grass regions are identifies as different textures. In the third row, grains on the rock are identified as one texture, and they are well separated from the fur of the animal, even when the colors are similar. In the forth row, the texture within the coral are identified as a texture, and detail of the oscillatory boundary are well identified.

(a)	(b)	(c)

Figure 15: (a) The given image. (b) The edge function $g$ using $V$, which ignores the texture within regions. (c) The segmentation result of (24). Textures of the images are well grouped as each region, e.g., in the top image, grains are identified as one region, and in the third row, grains on the rock are identified as one texture, and they are will separated from the fur of the animal, even when the colors are similar.

This method can be naturally extended to image decomposition, and in Figure 16, we present the remainder after the segmentation showing the details of the image.

(a)	(b)	(c)

Figure 16: (a) The given image. (b) The segmentation result using (24). There is a natural extension to image decomposition and (c) shows the reminder, subtracting image (b) from (a), representing the details of the image.

In Appendix, we further discuss more details of the proposed method, e.g. proof of Theorem 1, behavior of periodic texture, and junction refinement.

6 Concluding Remarks

We proposed a texture edge detection method which utilize patch based consensus. We use the local patch and its response, to emphasize the similarities and the differences among textures, and segmentation of the patch response helps to locate the edge location clearly. On the boundary of texture, local patch information and patch response is not as accurate to identify edge location, that we utilize the neighbor consensus to stabilize the process. We statistically analyze when the texture can be separated, and derive necessary conditions to distinguish textures. The proposed method has three parameters which are not very sensitive to choose, and show how patch width and the size of texton is related. This method is robust to different type of noise, and can handle multiple junctions. This method is training-free and filter-free approach. We provided numerical details and various experiments which illustrate the properties of the proposed model.

In the future, one may consider refining and thinning the edge thickness, which will also improve image decomposition application in Figure 16. We can also consider using multi-scale approach to further separate more object related edges, e.g., using different scales of patch responses. We may improve the performance of TEP via utilizing a scheme with adaptive patch size which can handle more complicated real images. Also, different types of kernels can be used instead of squared euclidean distance when comparing image patches, in order to enhance the sensitivity to certain types of textures.

Appendix A Proof of Theorem 1.

Proof.

Note that $\mathcal{R}(\mathbf{y};\mathbf{x})$ is a quadratic function of two random vectors $\vec{\mathcal{P}}(\mathbf{x})$ and $\vec{\mathcal{P}}(\mathbf{y})$, the expectation needs to be computed with double integral

$$\mathbb{E}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)=\mathbb{E}_{\mathbf{x}}\left(\mathbb{E}_{\mathbf{y}|\mathbf{x}}\mathcal{R}(\mathbf{y};\mathbf{x})\right).$$

To handle the conditional expectation $\mathbb{E}_{\mathbf{y}|\mathbf{x}}(\cdot)$, one needs the conditional distribution of $\vec{\mathcal{P}}(\mathbf{y})$, which is Gaussian with mean (15) and variance (16). Then

$$\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\|\vec{\mathcal{P}}(\mathbf{y})-\vec{\mathcal{P}}(\mathbf{x})\|_{2}^{2}\right) =\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\vec{\mathcal{P}}^{T}(\mathbf{y})\vec{\mathcal{P}}(\mathbf{y})-2\vec{\mathcal{P}}^{T}(\mathbf{y})\vec{\mathcal{P}}(\mathbf{x})+\vec{\mathcal{P}}^{T}(\mathbf{x})\vec{\mathcal{P}}(\mathbf{x})\right).$$ (25)

$$=\mathrm{tr}\left(\Sigma_{p}(\mathbf{y;x})\right)+\vec{\mu}_{p}^{T}(\mathbf{y};\mathbf{x})\vec{\mu}_{p}(\mathbf{y};\mathbf{x})-2\vec{\mu}_{p}^{T}(\mathbf{y};\mathbf{x})\vec{P}(\mathbf{x})+\vec{\mathcal{P}}^{T}(\mathbf{x})\vec{\mathcal{P}}(\mathbf{x})$$ (26)

where Lemma 1 is applied to the first term of the right hand side of (25). To compute the expectation of (26) with respect to $\vec{\mathcal{P}}(\mathbf{x})$, we have the following identities:

$$\mathrm{tr}\left(\Sigma_{p}(\mathbf{y};\mathbf{x})\right) =\mathrm{tr}\left(\Sigma_{p}\right)-\mathrm{tr}\left(\Sigma_{p}^{-1}\Sigma_{\mathrm{c}}^{T}(\tau)\Sigma_{\mathrm{c}}(\tau)\right)$$

$$\mathbb{E}_{\mathbf{x}}\left(\vec{\mu}_{p}^{T}(\mathbf{y};\mathbf{x})\vec{\mu}_{p}(\mathbf{y};\mathbf{x})\right) =\vec{\mu}_{p}^{T}\vec{\mu}_{p}+\mathrm{tr}\left(\Sigma_{p}^{-1}\Sigma_{\mathrm{c}}^{T}(\tau)\Sigma_{\mathrm{c}}(\tau)\right)$$

$$\mathbb{E}_{\mathbf{x}}\left(\vec{\mu}_{p}^{T}(\mathbf{y};\mathbf{x})\vec{\mathcal{P}}(\mathbf{x})\right) =\vec{\mu}_{p}^{T}\vec{\mu}_{p}+\mathrm{tr}\left(\Sigma_{\mathrm{c}}(\tau)\right)$$

$$\mathbb{E}_{\mathbf{x}}\left(\vec{\mathcal{P}}^{T}(\mathbf{x})\vec{\mathcal{P}}(\mathbf{x})\right) =\vec{\mu}_{p}^{T}\vec{\mu}_{p}+\mathrm{tr}(\Sigma_{p}).$$

With substitution, the expectation of $\mathcal{R}(\mathbf{y};\mathbf{x})$ is then given as

$$\mathbb{E}_{\mathbf{x}}\left(\mathbb{E}_{\mathbf{y}|\mathbf{x}}\mathcal{R}(\mathbf{y};\mathbf{x})\right) =\frac{1}{d}\mathbb{E}_{\mathbf{x}}\left(\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\|\vec{\mathcal{P}}(\mathbf{y})-\vec{\mathcal{P}}(\mathbf{x})\|_{2}^{2}\right)\right)$$

$$=\frac{1}{d}\left(2\mathrm{tr}\left(\Sigma_{p}\right)-2\mathrm{tr}\left(\Sigma_{c}(\tau)\right)\right)=2\sigma_{p}^{2}\left(1-\exp(-\frac{\tau^{2}}{2l_{p}^{2}})\right).$$

∎

Appendix B Junction edge refinement

For the edges near a junction point, the strength of $V$ may be weaker than straight edges as seem in Figure 13. This is because the local patches at a location x which is near a junction point, may observed another textures which is a bit further from the two immediate two texture edge near x and confuse the segmentation. In Figure 17, (b) shows this effect. This can be improved by a simple dilation-erosion operation from mathematical morphology [28]. By using line shaped structuring element, where the orientation of the line should be parallel to the existing edge direction, the edge function $V$ is improved as in Figure 17 (c) and (d).

(a)	(b)	(c)	(d)

Figure 17: (a) The given image with 4-junction. (b) The edge function $V$. (c) Improved edge function $V$ using straight line direction enhancing. (d) TEP result with the improved $V$.

Appendix C Periodic texture and its patch response

Extending from the analysis in Section 3, we numerically present the cases for periodic texture. For periodic texture, it is interesting to notice that the variance of expectation is also strongly correlated to the period of the texture. In Figure 18, we show the distribution of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$ with varying patch width parameter $r$. The variance is near zero whenever the patch width parameter $r$ matches the periods of the texture, while the general decreasing effect discussed in section 3 still exists. This is consistent with the work by Hong, et al. [12], where the authors observed that for a periodic image, some statistical distance measurement of the image patch vs the entire image vanishes whenever the patch width parameter is a multiple of the texture period. In another work [15], authors measured the scale of the texture by applying time varying Gaussian kernel to the image, and observe when the averaging process has big jump, and use it to measure the scale.

In this paper, we choose relatively small $r$ while kee** the patch response consistent and stable. For periodic texture, we can use these estimations to help find the scale of the texture.

(a)	(b)

Figure 18: (a) Given synthetic periodic texture. (b) Estimation of the distribution of $\mathbb{E}_{\mathbf{y}|\mathbf{x}}\left(\mathcal{R}(\mathbf{y};\mathbf{x})\right)$. The black horizontal line in (b) shows empirical mean, and the blue horizontal line in (b) shows the theoretical mean. The dark red region shows the location of distribution median. The vertical lines indicate multiples of the texture period. Notice that there is a sharp decrease of the variance when the patch width parameter $r$ hits the period.