IJCRR - 9(13), July, 2017
Pages: 01-05
Date of Publication: 03-Jul-2017
Print Article
Download XML Download PDF
Graphical Processing Unit Accelerated Face Resolution Enhancement using Pixels - Homogeneity and Relative- Ratios
Author: Vikram Mutneja, Satvir Singh
Category: General Sciences
Abstract:This work presents a GPU (Graphical Processing Unit) accelerated spatial domain oriented face resolution enhancement algorithm based on the homogeneity levels and relative-ratios of the pixels with respect to its surrounding pixels. The algorithm has been developed, implemented as well as tested in the MATLAB environment. MATLAB is slow in processing but at the same time a resourceful environment for the development in the area of image processing owing to its extremely rich set of functions and programmer-friendly integrated development environment. However, to compensate for the speed loss in testing and implementation phase, we have made use of GPU computing i.e. done parallelization of the algorithm on NVIDIA GPU using CUDA (Compute Unified Device Architecture) interface in the MATLAB environment. It is a simple but efficient algorithm in which kernel matrices are created encoding the homogeneity levels and relative-ratios of all pixels in surrounding four quadrants. Kernel matrices are subsequently applied to reconstruct the HR (High-Resolution) version from input LR (Low-Resolution) facial image.
Keywords: Image processing, GPU computing, Face resolution enhancement, Surveillance videos, Spatial domain processing
DOI: 10.7324/IJCRR.2017.9131
Full Text:
Introduction
The processing of the facial images is an essential task in the smart video surveillance systems. The resolution of the facial images from surveillance videos is usually very less owing to the various factors such as hardware constraints, distance between the subjects and camera. Therefore enhancement concerning resolution is a major step for the purpose of overall improvement in the task of face detection and recognition. As the task of video processing demaLiterature Survey nds of fast computational processing, the parallelization of proposed algorithm has been done on the NVIDIA Graphical Processing Unit (GPU) in CUDA (Compute Unified Device Architecture) under MATLAB environment.
The facial image processing systems can be classified based on different attributes, such as the number of input facial images, face angle, quality and source of images etc. Some of the techniques use single input LR (Low-Resolution) image, while some use multiple LR images which are captured generally from consecutive frames of video. The latter technique reconstructs the HR image from multiple LR images, therefore called as reconstruction based techniques. The other type of techniques which have attracted large amount of research work in past one decade are the learning based techniques. The relationship between single or multiple LR images and corresponding HR image is learned via a machine learning or intelligent technique. Further, the optimization techniques may be applied for tuning of weights to improve the results. Most of the learning based methods are based on the single facial image as input and use the trained system or dictionary to estimate the missing HR details. Reconstruction based models are based on the generalized smoothness priors while the learning based techniques use recognition based priors.
Based upon the region of interest, the human facial image processing systems can be classified in two major categories. The methods which use the positions of facial landmarks such as left eye, right eye, nose tip and other facial regions are called as local methods ( Yang et al. (2010), Liu and Yang (2014) ), while another type of technique is based on template matching, called as global methods. Supposedly the local methods require the input facial images of reasonable high quality so that facial landmarks can be extracted, which also depends on the facial image resolution. It is easier to localize the facial landmarks in the high-resolution facial images as compared to the low-resoultion facial image, which necessitates the task of face resolution enhancement.
The development techniques for image processing have certain barriers such as the computational load of handling huge amount of graphics data of images which may involve computation on pixels or their extracted features. At the same time, support from the development environment largely affects the development time. MATLAB is an ideal environment for doing research in the area of image processing because of rich set of functions available in its Image Processing, Computer Vision and related toolboxes. However, MATLAB suffers severely in computational performance as applications written in MATLAB do not run directly on CPU or operating system but the MATLAB engine. On the other hand, developing CUDA (Compute Unified Device Architecture) kernels to harness the power of GPU directly into C or FORTRAN requires writing a lot of glue code for tasks such as communicating data between GPU and CPU, managing GPU memory, initializing and launching CUDA kernels and visualizing their output, which makes It difficult to evaluate and test CUDA kernels for whole of input data space and visualizing their corresponding output. To overcome this limitation, MATLAB's parallel processing toolbox comes to rescue as its GPU computing libraries facilitate to quickly develop, evaluate, visualize and analyze CUDA kernels.
In this work, we have done improvements and GPU acceleration of our previous work on face resolution enhancement Mutneja and Singh (2017). This paper is structured as follows: Section 2 provides
Literature Survey.
The proposed technique based on pixels-homogeneity and relative-ratios has been explained in Section 3, Section 4 describes experimental setup, Sections 5 and 6 elaborates Results and Discussion respectively, and finally, Sections 7 and 8 gives Conclusions and Future Scope respectively.
Literature Survey
Many researchers have worked in the area of reconstruction based face super-resolution such as by Baker and Kanade (2002), Jiang et al. (2016a), Yang et al. (2010). A lot of work has been done in the learning based face resolution enhancement known as face hallucination such as by Yin et al. (2016), Tu et al. (2017). Some of the works have done hybridization of reconstruction and learning, such as by Yeganli et al. (2016). Many spatial domain based techniques are based on neighbor embedding such as by Chang et al. (2004), Huang et al. (2016), Jiang et al. (2016b).
Some researchers have worked in the field of sparse representation based face super resolution. Yang et al. (2010) presented a single image super-resolution by estimating sparse representation for each patch of the low-resolution input to be applied with the high-resolution image patch dictionary. Zhang et al. (2016) proposed a noise robust method based on information entropy and regression, to weight facial patches for estimating high frequency details from input low-resolution facial images. Huang et al. (2017) Proposed a novel sparse coding-based face hallucination method, by incorporating the intrinsic geometric structure of training samples for dictionary learning. Authors worked to minimize artificial effects using graph construction in HR manifold, and K-selection mean constraints for finding optimal weight HR face reconstruction. Rahiman and George (2017) worked on single image super-resolution using neighbor embedding and sparse representation based learning using partitioned feature space and statistical prediction model. Mao et al. (2016) proposed a new weighted-patch super-resolution method using AdaBoost iteratively to focus more on the patches (face regions) with richer information for improving the reconstruction power.
Dahl et al. (2017) presented a pixel recursive super resolution model using convolutional neural networks to synthesize realistic details into images while enhancing their resolution, represented a multi-modal conditional distribution by accurately modeling the statistical dependencies among the high-resolution image pixels, conditioned on a low-resolution input.
Chen et al. (2017) proposed noisy facial images super-resolution using the contour features to handle noise and standard deviation prior as statistical measure to enhance the low quality contour feature. Jiang et al. (2017) developed facial image super-resolution method based on missing image interpolation information using 'Smooth Regression and a novel Local Structure Prior' (SRLSP). Tu et al. (2017) proposed face hallucination method using direct combined approach, independent of size and number of training samples . Hui et al. (2017) developed appearance, geometrical features, and optical flow model single LR based face resolution enhancement technique. Gong and Wang (2017) developed face hallucination based on similarity measurements between single LR and corresponding multiple HR training images.
Proposed Technique
In this work, GPU acceleration of face resolution enhancement technique has been proposed based on hybridization of homogeneity levels and relative-ratios of pixels in LR patches, by using window based scanning of whole input facial image. Kernel matrices are generated for encoding homogeneity levels as well as relative-ratios to construct resultant HR image. LR patches are slid to the nucleus of HR patches and remaining pixels in HR patch are computed by application of kernel matrices. The size of LR patch has been selected as 3×3 which is swelled to make the HR patch of size 5×5. For generating the homogeneity kernel matrix, LR window is slid over whole input LR image and checks the homogeneity levels of pixels in the surrounding four quadrants. To check for the homogeneity levels, the summation of pixel intensities is done in the respective quadrant and if found less than the set threshold, is marked as homogeneous otherwise non-homogenous. Figure 1 shows the division of LR patch into four surrounding quadrants. Algorithm 1 shows the working of GPU CUDA Kernel for generating the homogeneity matrix.
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512201259639.png)
To generate the kernel matrix for the relative-ratios, the ratio of intensity levels of all the pixel is computed with respect to the central pixel in LR patch. The process of iterating the pixels is same as depicted in the Algorithm 1. After generation of kernel matrices, they are applied to the input image to create the corresponding HR image. Algorithm 2 shows the flow chart of proposed technique to generate the HR image by applying the homogeneity and relative-ratios kernel matrices. The GPU kernel thread block size has been set as 1×1×1 while the grid size has been set as NR×NC×1.
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512201843324.png)
Experimental Setup
The machine running Windows 7 (64 Bit) on Intel Core i3 1.9 GHz and having NVIDIA graphics processing unit GeForce GT 740M has been used to develop and test the proposed method.
Results
Figure 2 shows the results of application of proposed algorithm (shown in column 4) on some of the test LR images (shown in first column) along with the existing techniques (‘Bicubic’ and ‘Box’).
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512201659587.png)
Table 1 and Table 2 shows the results of application of proposed algorithm on the Gray-scale and Color test images respectively, for different values of scaling factors.
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512203253230.png)
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512203432725.png)
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512203588217.png)
Discussion
The quality metrics Structural Similarity Index Measure (SSIM), Peak Signal To Noise Ratio (PSNR), Signal To Noise Ratio (SNR) and Mean Square Error (MSE) have been measured from the generated HR image with respect to the original HR image, along with the measurement of GPU acceleration achieved with respect to the CPU (Central Processing Unit) code (refer Table 1 and 2). As the work has been done in MATLAB, the comparison of results of proposed algorithm has been done with some methods (‘Bicubic’ and ‘Box’) available in MATLAB (refer Figure 3).
The proposed method has been tested on color as well as gray-scale images. Two folders have been created for testing, each containing 20 test images. One of the folders contains gray-scale images and second constitutes color facial images from database color FERET (Phillips et al. (2000, 1998)). Portions of the research in this paper use the FERET database of facial images collected under the FERET program, sponsored by the DOD Counterdrug Technology Development Program Office FERET Phillips et al. (1998), Phillips et al. (2000). Table 1 and Table 2 shows the mean values of measured quality metrics and time taken for gray-scale and color images respectively. The term SF refers to the scaling factor i.e. face resolution enhancement factor. By comparison, it is inferred that application of proposed technique using color based processing (i.e. of channels R,G and B) gives better results as compared to gray-scale based only processing on color images.
Further, the efficiency of the proposed algorithm has been tested on different resolution factors. For this, the test images from FERET are down-scaled to the factors 0.6000, 0.3600, 0.2160, 0.1296 and 0.0778. The percentage reduction in the time cost has been calculated in each case. Figure 3 shows the comparison of time cost of proposed algorithm using GPU Computing with respect to the CPU code. The maximum percentage time cost reduction of the order of 28.38% has been achieved corresponding to the scaling factor 1.67 and 22.39% at scaling factor of 12.86.
Conclusion
We have successfully developed and tested the GPU accelerated face resolution enhancement algorithm based on pixels-homogeneity and relative-ratios in MATLAB environment using CUDA based parallel computing. The testing of the proposed algorithm has been done on test facial images from web and FERET database (Phillips et al. (2000, 1998)) and comparison done with some of the existing techniques in terms of quality metrics SSIM, PSNR, SNR and MSE. It has been observed that considerable speedup was achieved by the parallel execution of proposed face resolution enhancement algorithm as compared to its serial version.
As MATLAB interprets the code, and it is the interpreter that slows down the processing, therefore by using GPU computing, we have tried to compensate for this time efficiency loss and at the same time benefitting user friendly environment of MATLAB for efficient development of GPU computing based algorithm. From the results of quality metrics and time efficiency improvement achieved by parallelization, we conclude that proposed system is very effective and efficient in resolution enhancement of facial images.
![](https://ijcrr.com/admin/public/uploads/1/myfiles/1_1512203784950.png)
Future Scope
In the near future, we intend to make improvements in the proposed technique by making use of shared memory in GPU, increasing the extent of parallelization for further time acceleration, introducing adaptability in terms of selection of size of LR and HR patches for achieving the desired resolution enhancement factors. Further, we intend to hybridize the proposed technique with boosting based analysis for improvement in the quality of resultant HR facial image.
AcknowledgementAuthors gratefully acknowledge the support provided by I.K.G. Punjab Technical University, Kapurthala, Punjab, India. Authors deeply acknowledge the immense help received from the scholars whose articles are cited and included in references of this manuscript. The authors are also grateful to authors / editors / publishers of all those articles, journals and books from where the literature for this article has been reviewed and discussed.
References:
- Baker, S. and Kanade, T. (2002). Limits on super - resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167 - 1183.
- Chang, H., Yeung, D.-Y., and Xiong, Y. (2004). Super-resolution through neighbor embedding. In Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, volume 1, pages I - I. IEEE.
- Chen, L., Hu, R., Liang, C., Li, Q., and Han, Z. (2017). A novel face super resolution approach for noisy images using contour feature and standard deviation prior. Multimedia Tools and Applications, 76(2):2467 - 2493.
- Dahl, R., Norouzi, M., and Shlens, J. (2017). Pixel recursive super resolution. arXiv preprint arXiv:1702.00783.
- Gong, M. and Wang, K. (2017). Single face hallucination via local neighbor patches. AEU-International Journal of Electronics and Communications, 74:88 - 93.
- Huang, K., Hu, R., Jiang, J., Han, Z., and Wang, F. (2016). Face image super-resolution through improved neighbor embedding. In International Conference on Multimedia Modeling, pages 409 - 420. Springer.
- Huang, K., Hu, R., Jiang, J., Han, Z., and Wang, F. (2017). Hrm graph constrained dictionary learning for face image super-resolution. Multimedia Tools and Applications, 76(2):3139 - 3162.
- Hui, Z., Liu, W., and Lam, K.-M. (2017). A novel correspondence-based face-hallucination method. Image and Vision Computing.
- Jiang, J., Chen, C., Huang, K., Cai, Z., and Hu, R. (2016a). Noise robust position-patch based face super-resolution via tikhonov regularized neighbor representation. Information Sciences, 367:354 - 372.
- Jiang, J., Chen, C., Ma, J., Wang, Z., Wang, Z., and Hu, R. (2017). Srlsp: A face image super-resolution algorithm using smooth regression with local structure prior. IEEE Transactions on Multimedia, 19(1):27 - 40.
- Jiang, J., Hu, R., Wang, Z., Han, Z., and Ma, J. (2016b). Facial image hallucination through coupled-layer neighbor embedding. IEEE Transactions on Circuits and Systems for Video Technology, 26(9):1674 - 1684.
- Liu, S. and Yang, M.-H. (2014). Compressed face hallucination. In Image Processing (ICIP), 2014 IEEE International Conference on, pages 4032 - 4036. IEEE.
- Mao, S., Zhou, D., Zhang, Y., and Zhang, Z. (2016). A boosting method to face image super-resolution. arXiv preprint arXiv:1609.01805.
- Mutneja, V. and Singh, S. (2017). A novel facial image resolution enhancement algorithm using pixels homogeneity. International Journal, 8(3).
- Phillips, P. J., Moon, H., Rizvi, S. A., and Rauss, P. J. (2000). The feret evaluation methodology for face-recognition algorithms. IEEE Transactions on pattern analysis and machine intelligence, 22(10):1090 - 1104.
- Phillips, P. J., Wechsler, H., Huang, J., and Rauss, P. J. (1998). The feret database and evaluation procedure for face-recognition algorithms. Image and vision computing, 16(5):295 - 306.
- Rahiman, V. A. and George, S. N. (2017). Single image super resolution using neighbor embedding and statistical prediction model. Computers and Electrical Engineering.
- Tu, C.-T., Ho, M.-C., and Lin, M.-Y. (2017). A new approach for face hallucination based on a two - dimensional direct combined model. Pattern Recognition, 62:1 - 20.
- Yang, J., Wright, J., Huang, T. S., and Ma, Y. (2010). Image super - resolution via sparse representation. IEEE transactions on image processing, 19(11):2861 - 2873.
- Yeganli, F, Nazzal, M., Unal, M., and Ozkaramanli, H. (2016). Image super-resolution via sparse representation over multiple learned dictionaries based on edge sharpness. Signal, Image and Video Processing, 10(3):535 - 542.
- Yin, Q, Cao, Z., and Zhou, E. (2016). Face hallucination using convolutional neural networks. US Patent 9,405,960.
- Zhang, Y, Zhang, Z, Hu, G, and Hancock, E. R. (2016). Face image super - resolution via weighted patches regression. In Pattern Recognition (ICPR), 2016 23rd International Conference on, pages 3892 - 3897. IEEE.
|