A Remote Sensing Image Semantic Segmentation Method by Combining Deformable Convolution with Conditional Random Fields

doi:10.11947/j.JGGS.2020.0304

Abstract

Abstract:

Currently, deep convolutional neural networks have made great progress in the field of semantic segmentation. Because of the fixed convolution kernel geometry, standard convolution neural networks have been limited the ability to simulate geometric transformations. Therefore, a deformable convolution is introduced to enhance the adaptability of convolutional networks to spatial transformation. Considering that the deep convolutional neural networks cannot adequately segment the local objects at the output layer due to using the pooling layers in neural network architecture. To overcome this shortcoming, the rough prediction segmentation results of the neural network output layer will be processed by fully connected conditional random fields to improve the ability of image segmentation. The proposed method can easily be trained by end-to-end using standard backpropagation algorithms. Finally, the proposed method is tested on the ISPRS dataset. The results show that the proposed method can effectively overcome the influence of the complex structure of the segmentation object and obtain state-of-the-art accuracy on the ISPRS Vaihingen 2D semantic labeling dataset.

Key words: high-resolution remote sensing image; semantic segmentation; deformable convolution network; conditions random fields

Zongcheng ZUO, Wen ZHANG, Dongying ZHANG. A Remote Sensing Image Semantic Segmentation Method by Combining Deformable Convolution with Conditional Random Fields[J]. Journal of Geodesy and Geoinformation Science, 2020, 3(3): 39-49.

Figures/Tables 7

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Tab.1

References 33

[1]	LIU Jing , LIPeijun. A high resolution image segmentation method by combined structural and spectral characteristics[J]. Acta Geodaetica et Cartographica Sinica, 2014,43(5):466-473. DOI: 10.13485/j.cnki.11-2089.2014.0087.
[2]	ZHOU Chenghu, LUO Jiancheng. Geo-computing of high resolution satellite remote sensing image[M]. Beijing: Science Press, 2009.
[3]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation [C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015: 3431-3440.
[4]	WANG Hongzhen, WANG Ying, ZHANG Qian , et al. Gated convolutional neural network for semantic segmentation in high-resolution images[J]. Remote Sensing, 2017,9(5):446.
[5]	GARCIA-GARCIA A, ORTS-ESCOLANO S, OPREA S , et al. A review on deep learning techniques applied to semantic segmentation[J]. arXiv:1704. 06857, 2017.
[6]	HUBEL D H . The visual cortex of the brain[J]. Scientific American, 1963,209:54-62. DOI: 10.1038/scientificamerican1163-54.
[7]	KRÄHENBVHL P, KOLTUN V . Efficient inference in fully connected crfs with gaussian edge potentials[J]. arXiv:1210.5644, 2012: 109-117.
[8]	Rother C, Kolmogorov V, Blake A . “Grabcut”-interactive foreground extraction using iterated graph cuts[J]. ACM Transactions on Graphics, 2004: 23(3):309-314. DOI: 10.1049/ el.2009.2014.
[9]	SHOTTON J, WINN J, ROTHER C , et al. Textonboost for image understanding:multi-class object recognition and segmentation by jointly modeling texture, layout, and context[J]. International Journal of Computer Vision, 2009,81(1):2-23.
[10]	DAI Jifeng, QI Haozhi, XIONG Yuwen , et al. Deformable convolutional networks[J]. arXiv:1703.06211, 2017: 764-773.
[11]	LUO Wenjie, LI Yujia, URTASUN R , et al. Understanding the effective receptive field in deep convolutional neural networks[J]. arXiv:1701.04128, 2017.
[12]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[13]	HOLSCHNEIDER M, KRONLAND-MARTINET R, MORLET J , et al. A real-time algorithm for signal analysis with the help of the wavelet transform[M]. COMBES J M, GROSSMANN A, TCHAMITCHIAN P. Wavelets:Time-Frequency Methods and Phase Space. Berlin:Springer, 1990: 286-297.
[14]	DENG Jia, DONG Wei, SOCHER R, et al. Imagenet:a large-scale hierarchical image database[C]∥Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL, USA:IEEE, 2009.
[15]	DAI Jifeng, LI Yi, HE Kaiming, et al. R-FCN:object detection via region-based fully convolutional networks [C]∥Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain:ACM, 2016: 379-387.
[16]	BADRINARAYANAN V, KENDALL A, CIPOLLA R . SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017,39(12):2481-2495.
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs [C]∥Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[18]	CHATFIELD K, SIMONYAN K, VEDALDI A , et al. Return of the devil in the details: delving deep into convolutional nets[C]∥Proceedings of the British Machine Vision Conference. Dundee, Britain:BMVA Press, 2014.
[19]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.1556, 2014.
[20]	IOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]∥Proceedings of the 32nd International Conference on Machine Learning. Lille, France:JMLR, 2015: 448-456.
[21]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Delving deep into rectifiers:surpassing human-level performance on imagenet classification[C]∥Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015: 1026-1034.
[22]	CLEVERT D A, UNTERTHINER T, HOCHREITER S. Fast and accurate deep network learning by exponential linear units (ELUs) [C]∥Proceedings of the International Conference on Learning Representations. San Diego, CA:Computational and Biological Learning Society, 2015.
[23]	EIGEN D, FERGUS R. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture [C]∥Proceedings of the IEEE International Conference on Computer Vision. Santiago, Chile:IEEE, 2015: 2650-2658.
[24]	MOSTAJABI M, YADOLLAHPOUR P, SHAKHNAROVICH G. Feedforward semantic segmentation with zoom-out features [C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA, USA:IEEE, 2015: 3376-3385.
[25]	ROTHER C, KOLMOGOROV V, BLAKE A. Grabcut:interactive foreground extraction using iterated graph cuts [C]∥Proceedings of the ACM SIGGRAPH 2004. Los Angeles, California:ACM, 2004: 309-314.
[26]	KOHLI P, LADICKY L, TORR P H S. Robust higher order potentials for enforcing label consistency [C]∥Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK, USA:IEEE, 2009.
[27]	KRÄHENBÜHL P, KOLTUN V. Efficient inference in fully connected CRFs with Gaussian edge potentials [C]∥Proceedings of the 25th annual conference on Neural Information Processing Systems. Granada:NIF, 2011.
[28]	ADAMS A, BAEK J, DAVIS M A . Fast high-dimensional filtering using the permutohedral lattice[J]. Computer Graphics Forum, 2010,29(2):753-762.
[29]	GERKE M, ROTTENSTEINER F, WEGNER J D, et al. ISPRS semantic labeling contest[C]∥ Proceedings of PCV-Photogrammetric Computer Vision.[S.l.]:ISPRS, 2014.
[30]	GERKE M. Use of the stair vision library within the ISPRS 2D semantic labeling benchmark (Vaihingen) [C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Boston, MA, USA:IEEE, 2015.
[31]	PAISITKRIANGKRAI S, SHERRAH J, JANNEY P, et al. Effective semantic pixel labelling with convolutional networks and conditional random fields[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Boston, MA, USA:IEEE, 2015: 36-43.
[32]	AUDEBERT N, LE SAUX B , LEFÈVRE S. Semantic seg-mentation of earth observation data using multimodal and multi-scale deep networks[M]∥LAI S H, LEPETIT V, NISHINO K, et al. Computer Vision-ACCV 2016. Cham:Springer, 2017.
[33]	ZHOU Hao, ZHANG Jun, LEI Jun, et al. Image semantic segmentation based on FCN-CRF model [C]∥Proceedings of International Conference on Image, Vision and Computing. Portsmouth, UK:IEEE, 2016: 9-14.

Methods	Segmentation accuracy					Global accuracy
Methods	Roads	Buildings	Vegetation	Trees	Vehicles	Global accuracy
CNNs^[32]	87.8	91.9	77.8	86.2	50.7	85.9
FCNs^[33]	89.2	92.5	81.6	86.9	57.3	87.3
SegNet	90.5	93.7	82.7	89.2	70.6	89.1
SegNet+CRF	90.3	92.3	82.5	89.5	82.5	88.5
SegNet+DC	92.2	94.3	83.4	89.9	76.3	89.4
SegNet+DC+CRF(Iter=1)	91.1	93.5	84.4	89.1	77.8	89.8
SegNet+DC+CRF(Iter=10)	91.5	94.7	85.1	89.3	85.7	90.7