Guided Intra-Patch Smoothing Graph Filtering for Single-Image Denoising

: Graph filtering is an important part of graph signal processing and a useful tool for image denoising. Existing graph filtering methods, such as adaptive weighted graph filtering (AWGF), focus on coefficient shrinkage strategies in a graph-frequency domain. However, they seldom consider the image attributes in their graph-filtering procedure. Consequently, the denoising performance of graph filtering is barely comparable with that of other state-of-the-art denoising methods. To fully exploit the image attributes, we propose a guided intra-patch smoothing AWGF (AWGF-GPS) method for single-image denoising. Unlike AWGF, which employs graph topology on patches, AWGF-GPS learns the topology of superpixels by introducing the pixel smoothing attribute of a patch. This operation forces the restored pixels to smoothly evolve in local areas, where both intra- and inter-patch relationships of the image are utilized during patch restoration. Meanwhile, a guided-patch regularizer is incorporated into AWGF-GPS. The guided patch is obtained in advance using a maximum-a-posteriori probability estimator. Because the guided patch is considered as a sketch of a denoised patch, AWGF-GPS can effectively supervise patch restoration during graph filtering to increase the reliability of the denoised patch. Experiments demonstrate that the AWGF-GPS method suitably rebuilds denoising images. It outperforms most state-of-the-art single-image denoising methods and is competitive with certain deep-learning methods. In particular, it has the advantage of managing images with significant noise.


Introduction
Image denoising aims to remove noise from images, thereby benefitting the subsequent analysis and processing of images and videos. Current image denoising methods are divided into two categories: Model-and deep-learning-based methods. Prior to the success of deep learning methods, model-based methods have been used for decades to exploit the intrinsic attributes of images [1][2][3]. Most of them resolve the denoising problem using the internal prior of the target noisy image without considering other images. Therefore, they are often considered as single-image denoising methods. In contrast, deep-learning-based methods employ the external prior from other image databases to recover denoised images. Thus, they can be considered as data-driven methods that require numerous image data [4][5][6]. Flexible feature learning strategies are adopted in these methods via various deep networks. Because of the feature analysis of external images instead of the target image, deep-learning-based methods have a higher denoising performance than the model-based methods. However, in this study, we focus on the model-based denoising method assuming that only a single noisy image is provided.
In general, model-based denoising methods involve a filtering process, where the input and output data are noisy and denoised images, respectively. Thus, various filtering methods are used in different domains such as spatial, transform, and learned domains. In earlier studies, spatial-domain methods, such as bilateral filtering [7] and non-local means filtering [8], were used for the direct operation on pixels or patches. A spatial smoothing filter was used to remove noise-like components from the target image. However, since this smoothing procedure is only performed in local areas or patch groups, spatial filtering has a limited ability to exploit the statistical information of the entire image. To overcome this problem, transform-domain filters are presented by considering the directional structures in the images. Wavelet and Curvelet filters have been successfully employed for their adequate structural description of basis [9][10][11]. A popular denoising method, named block matching and three-dimensional (BM3D) filtering, was proposed using a collaborative spatial-wavelet filter [12,13]. It adopts a spatial filtering result to guide sequential wavelet filtering on similar patches. Unfortunately, none of the aforementioned filters are contented-based due to their fixed filter coefficients. Therefore, more advanced models, such as sparse representation and low-rank representation models [3,[14][15][16], have been deployed in different learned domains in recent years. For example, a trilateral weighted sparse coding (TWSC) method is used to estimate data fidelity based on the sparse representation theory [17]. A low-rank approximation approach with adaptive regularizer learning (ARLLR) [18] is presented to shrink the eigenvalues of patches and is highly successful in image denoising. However, it is not yet considered as a filter-based model. The current study proves that the low-rank model is equivalent to a subspace graph filter [19]. The filtering procedure takes place in a graph subspace supported by the eigenvectors of the patch group.
Graph filtering is an essential component of graph signal processing. Its basic idea is to filter the input signal on the network nodes [20]. Once pixels or patches are chosen according to the nodes, graph filtering can adequately fit image denoising. Certain graph polynomial filtering methods are presented to employ various Laplacian matrix regularizers in the existing denoising model [21][22][23]. Several adaptive graph filtering methods are also proposed, by applying different coefficient shrinkage strategies in the graph-frequency domain. In a pioneer work, an idea lowpass graph filter was designed using a full shrinkage approach in the high graph-frequency band, achieving denoising performance comparable with that of BM3D [24][25][26]. Given a patch group, the study proves that the eigenvectors of the Laplacian matrix are a set of graph Fourier bases [27]. Recently, an adaptive weighted graph filtering (AWGF) method introduced an effective shrinkage approach in the entire band [19]. Although it theoretically builds a bridge from the existing low-rank model to graph filtering, its denoising performance is barely comparable with that of low-rank denoising methods. In the traditional AWGF method, the graph filter is prioritized, whereas patch attributes are seldom considered for image denoising.
Motivated by the recent progress, we propose a guided intra-patch smoothing AWGF (AWGF-GPS) method for single-image denoising. Our contributions are twofold. (1) Unlike AWGF that uses graph topology on patches, AWGF-GPS learns the superpixel graph to exploit the pixel smoothing attribute in patches. This operation forces the restored pixels to smoothly evolve in the local area, where both intra-and inter-patch relationships are utilized during patch restoration.
(2) A guided-patch regularizer is incorporated into AWGF-GPS. The guided patch is obtained in advance using a maximum-a-posteriori (MAP) probability estimator. By considering the guided patch as a sketch of a denoised patch, AWGF-GPS effectively supervises the patch restoration procedure. Consequently, the reliability of the denoised patch is increased. Experiments demonstrate that our AWGF-GPS method suitably rebuilds denoising images. It outperforms most state-of-the-art model-based methods and is competitive with certain deep-learning methods.

Related Work
We briefly review the existing AWGF model [19]. Given a noisy patch group, Y, with vectorized patches y i in its column, the AWGF model is expressed as where τ is a weighted coefficient, U g is a graph Fourier basis matrix of the graph filter, N is the patch number, and g is a diagonal matrix with the shrinkage coefficients σ g,i as its diagonal entries. Moreover, g σ g,i is a prior regularizer, where different regularizers can be adopted to generate various shrinkage strategies on σ g,i [28,29]. Sequentially, the denoised patch group, X, is obtained as where˜ g is a diagonal matrix with the optimal shrinkage coefficients σ g,i as its diagonal entries.
Note that the noisy patch group, Y, in Eq. (2) is right-multiplied by the filtering operation of U g˜ g U T g . Because the graph filter is built on the graph topology constructed among patch nodes, the AWGF model is an intrinsic patch-based graph filtering model. It focuses on the relationship between patches, where the similarity attribute of the pixels is not considered.

Proposed Method
We propose the AWGF-GPS method to employ both the self-similarity of patches and the local similarity of intra-patch pixels. Fig. 1 depicts the filtering flowchart. The superpixel is defined as a pixel set sharing the same location on the patches. A graph is learned using these superpixels to strengthen the smooth attributes of the neighboring pixels. Sequentially, the corresponding graph Fourier bases are obtained from the Laplacian matrix during the graph analysis. The guided patches are evaluated based on their corresponding noisy patches via a MAP estimator. Combined with the graph Fourier bases and the guided patches, the AWGF-GPS model is finally implemented to restore the denoised patches.

Graph Learning on Superpixels
We obtain the superpixels from the noisy patch group, Y, where each row vector of Y represents a superpixel node. Graph learning is then performed on these nodes. Various graph learning methods have been proposed to form graphs with different strategies. However, we prefer a log-model graph learning approach [30], which builds a connected graph without any isolated nodes. This graph learning model is described as where W is an adjacent matrix to measure the weighted edges among the superpixel nodes, Z is a pairwise distance matrix for the superpixels, α and β are two weighted coefficients, 1 represents a vector of ones, and symbol • denotes the Hadamard product.

Figure 1: Flowchart of AWGF-GPS denoising
The optimization problem in Eq. (3) can be conveniently solved using the GSP Toolbox [31]. Then, the graph Laplacian matrix is given as L = D − W, where the diagonal entry of the degree matrix, D, is the corresponding row sum of W. Finally, the graph Fourier bases, U g , are obtained from the eigenvectors of L.

Guided Patch Estimation
We employ the guided patches as a regularizer in our AWGF-GPS model. Here, a MAP estimator is used to generate the guided patch group,X, that is viewed as a sketch of the denoised patch group, X. For convenience, we interpret the MAP estimator at the pixel level [32] and then extend it to the patch operation level. Given a noisy pixel, y = x + n, with clean pixel x and noise n, where n follows a Gaussian distribution, N 0, σ 2 n , with a zero mean value and standard deviation σ n , the conditional probability of x can be expressed as In Eq. (4), symbolsμ andσ 2 x are respectively defined aŝ wherex and σ x are the mean value and standard deviation of x, respectively. Hence, we define the guided pixel,x, as the conditional expected value of x witĥ where S = σ 2 x /σ 2 n is the signal-to-noise ratio (SNR). Inspired by the pixel estimation ofx, the guided group,X, can be written aŝ whereX is the mean patch group composed of the corresponding guided pixels. In Eq. (8), we assume that each pixel in the patch group shares the same SNR.
Unfortunately, the guided group,X, in Eq. (8) is impractical because there is no exquisite formulation forX and S. As a result, we generate these two parameters by an alternative approach. To estimate the SNR, we approximate S as where var (Y) is the mean deviation of pixel for patch group Y.
We estimate the mean patch group,X, using an ideal lowpass graph filter as where the diagonal matrix,˜ g,M , contains only the first M nonzero diagonal entries. These entries are described as The optimal value of M is further achieved under the noise control of σ 2 n as

AWGF-GPS Model
Using graph Fourier bases and guided patches, we define the AWGF-GPS model as where γ is a weighted coefficient and N is the superpixel number. Then the denoised patch group, X, is obtained as We should mention some aspects of the filtering procedure from Eqs. (13) to (14). Unlike the definition in Eq. (2), the noisy patch group, Y, in Eq. (14) is left-multiplied by the filtering operation of U g˜ g U T g . In this case, the row combination process is employed for Y. Since each row vector of Y represents a superpixel, it conducts a superpixel-based filtering model. Moreover, we aim for the denoising patches described in Eq. (13) to be centralized withX, because the guided patch group,X, is another denoised result for clean patches. These guide patches provide an optimal direction for denoising.
To simplify Eq. (13), we transform the model by merging its first and third terms as Subsequently, we cast Eq. (15) in the trace form as where Then, we provide the setting of the prior regularizer, g σ g,i . According to the AWGF approach [19], regularizer g σ g,i is expressed as where ε is a small positive constant and η i is a weighted coefficient. By defining the projection matrix, P = Y T U g , with its vectors p i , we set η i as the projected energy with η 2 i = p T i p i . Because the shrinkage coefficient matrix, g , is diagonal, the optimization problem in Eq. (16) can be decomposed into a set of subproblems by the given g σ g,i of Eq. (17). For each shrinkage coefficient, σ g,i , the corresponding subproblem is written as where f i is the i-th diagonal entry of U g T YE T U g . Considering the shrinkage coefficient solution in AWGF as a reference, Eq. (18) also provides a closed-form solution as Thus, the AWGF-GPS method completes its coefficient shrinkage procedure.

AWGF-GPS Denoising Framework
We present the iterative AWGF-GPS denoising framework in Algorithm 1. Given an intermediate noisy image, Y (k) , in the k-th iteration, we deal with its j-th noisy patch group, Y j , in four steps.

4:
For each patch y j of Y (k) do 5: Find similar patches of y j to form patch group Y j . 6: Learn the graph from superpixels using Eq. (3). 7: Calculate eigenvectors U g from adjacent matrix W.
First, we obtain the Laplacian matrix, L, from the learned superpixel-based graph, W, in Eq. (3) and obtain its graph Fourier bases, U g . Second, the j-th guided patch group,X j , is estimated using Eq. (8), where the corresponding SNR, S j , and the mean patch group,X j , are defined by Eqs. (9) and (10), respectively. Here, the noise standard variance, σ n,j , of Y j is given by where Y j is the patch group of noisy image Y sharing the same patch index of Y j . Third, the AWGF-GPS model is applied to the patch group, Y j , to achieve its shrinkage coefficients σ g,i using Eq. (19) and the intermediate denoised patches, X j , using Eq. (14). Finally, we assemble all denoised patches X j into the k-th iterative denoised image, X (k) . The optimal denoised image, X, is obtained after the K-th iteration.

Experimental Results
We compare the AWGF-GPS method with several state-of-the-art model-based denoising methods, including BM3D [12], TWSC [17], ARLLR [18], and AWGF [19]. A deep-learning image denoising method, named the fast and flexible denoising convolutional neural network (FFDNET) [5], is adopted to show the gap between the model-and deep-learning-based denoising methods. Moreover, because our method is derived from AWGF, the denoising parameters are also inherited from those in AWGF. However, for the weighted coefficients in Eq. (13), parameters τ and γ are set as τ = √ N σ 2 n,j and γ = 0.1τ −1 , respectively. A few clean images are shown in Fig. 2 as test materials. The noisy images are generated by adding Gaussian noise. A denoised image comparison of the noise standard deviation of σ n = 40 is shown in Fig. 3. As a benchmark, BM3D achieves an acceptable denoising performance through collaborative spatial-wavelet filtering. TWSC is better than BM3D with its efficiently learned features of noisy images via sparse representation. ARLLR outperforms the other methods. This indicates that the eigenvalue shrinkage approach is powerful for managing noise. The noise is sufficiently removed in the high "rank-bands" corresponding to the small eigenvalues of the patch group. AWGF emphasizes graph filtering but has less consideration for the image attributes. This leads to a denoising performance that is barely comparable with that of ARLLR. Our method is proven to be the best. The denoised images derived from AWG-GPS are smoothed with a few artificial textures by employing the intra-patch smoothing attribute among the superpixels and the supervision from guided patches. However, FFDNET achieves an outstanding performance in restoring images more naturally. Unlike model-based methods, it benefits from learning the prior from the external images.
The denoised image comparison of the noise standard deviation of σ n = 70 is shown in Fig. 4. BM3D obtains good results. TWSC and ARLLR are better than BM3D due to their self-learned features from noisy images. We observe that ARLLR suffers from stronger artificial textures (House). ARLLR is an eigenvalue shrinkage method that uses the statistical information of patch groups via singular value decomposition (SVD). However, it seldom focuses on the pixel smoothing attribute in local areas. AWGF outperforms the aforementioned methods. This proves that graph filtering on the learned graph is useful to alleviate the artificial textures, since the restored patches are required to smoothly evolve on the graph. The proposed AWGF-GPS further smoothens the denoised images, which is owing to the superpixel-based graph, where the similarities between the patches and intra-patch pixels are fully exploited. The shrinkage approach is also efficiently conducted using guided patches. Note that the denoising performance of FFDNET is unsatisfactory. For example, the eaves area of the house is not sufficiently restored. This phenomenon is caused by feature mismatch. In this case, the external prior from the image databases cannot precisely guide the patch reconstruction for denoised images because the features of clean images are significantly contaminated by noise and are difficult to recognize.  The statistical results of the PSNR and SSIM are listed in Tabs. 1 and 2. We use BM3D as a baseline, for it achieves an acceptable performance for all the images and noise levels. TWSC outperforms BM3D because it is a learned domain method. In the sparse representation model, dictionary learning aims to catch patch features as atoms, whereas sparse coding focuses on patch restoration. However, as the noise level increases, the performance of the TWSC degenerates significantly. In this case, the atoms are distorted by direct feature learning from the noisy image. ARLLR is better than the former methods. The shrinkage approach on the patch group is effective in dealing with noise because the relationship among patches is exploited. Our AWGF-GPS method performs the best among these model-based denoising methods. As previously indicated, its coefficient shrinkage strategy is presented on the superpixel graph, where both attributes, within and among the patches, are considered. The guided patch regularizer further enhances the denoising performance. We also note that our method may suffer from an over-smoothing problem. It is inappropriate to deal with images (e.g., Barbara) that contain strong textures. The AWGF-GPS model attempts to design a lowpass graph filter. Because the components of the texture are centralized in high graph-frequency bands, the AWGF-GPS filter is unsatisfactory. FFDNET is best in terms of the noise standard deviation of σ n ≤ 70 but the worst for σ n = 100. The use of an external prior is effective in improving denoising performance. However, once the feature mismatch phenomenon occurs, the denoised result significantly deteriorates and becomes inferior to the results of the model-based methods. Compared with FFDNET, our method has the advantage in coping with large noise.  Notes: * For each test image, the corresponding denoised images with the best SSIM are marked in blue, and those with the second highest SSIM values are marked in red.

Conclusion
We have proposed a guided intra-patch smoothing graph filtering method for single-image denoising. Unlike the traditional AWGF, which only focuses on coefficient shrinkage, the proposed AWGF-GPS method considers more image attributes for denoising. The similarities between patches and intra-patch pixels are exploited by introducing the superpixel operation. Moreover, the guided patches from the MAP estimator provide a reliable optimal direction for the AWGF-GPS model. This provides an additional way to supervise patch restoration during graph filtering. Experiments have demonstrated that the AWGF-GPS method outperforms several state-of-the-art model-based denoising methods and is comparable with certain deep-learning methods.

Conflicts of Interest:
The authors declare that they have no conflicts of interest to report regarding the present study.