A Simple Deep Learning Network for Classification of 3D Mobile LiDAR Point Clouds

doi:10.11947/j.JGGS.2021.0305

Abstract

Abstract:

Automatic and accurate classification is a fundamental problem to the analysis and modeling of LiDAR (Light Detection and Ranging) data. Recently, convolutional neural network (ConvNet or CNN) has achieved remarkable performance in image recognition and computer vision. While significant efforts have also been made to develop various deep networks for satellite image scene classification, it still needs to further investigate suitable deep learning network frameworks for 3D dense mobile laser scanning (MLS) data. In this paper, we present a simple deep CNN for multiple object classification based on multi-scale context representation. For the pointwise classification, we first extracted the neighboring points within spatial context and transformed them into a three-channel image for each point. Then, the classification task can be treated as the image recognition using CNN. The proposed CNN architecture adopted common convolution, maximum pooling and rectified linear unit (ReLU) layers, which combined multiple deeper network layers. After being trained and tested on approximately seven million labeled MLS points, the deep CNN model can classify accurately into nine classes. Comparing with the widely used ResNet algorithm, this model performs better precision and recall rates, and less processing time, which indicated the significant potential of deep-learning-based methods in MLS data classification.

Key words: deep learning; convolutional neural network (CNN); mobile laser scanning (MLS); LiDAR data classification; point-to-image transformation

Yanjun WANG,Shaochun LI,Mengjie WANG,Yunhao LIN. A Simple Deep Learning Network for Classification of 3D Mobile LiDAR Point Clouds[J]. Journal of Geodesy and Geoinformation Science, 2021, 4(3): 49-59.

Figures/Tables 9

Fig.1

Tab.1

Fig.2

Fig.3

Fig.4

Fig.5

Tab.2

Tab.3

Tab.4

References 48

[1]	PINGEL T J, CLARKE K C, MCBRIDE W A. An improved simple morphological filter for the terrain classification of airborne LIDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2013,77:21-30. doi: 10.1016/j.isprsjprs.2012.12.002
[2]	WANG Heng, WANG Bin, LIU Bingbing, et al. Pedestrian recognition and tracking using 3D LiDAR for autonomous vehicle[J]. Robotics and Autonomous Systems, 2017,88:71-78. doi: 10.1016/j.robot.2016.11.014
[3]	GÉZERO L, ANTUNES C. Road rutting measurement using mobile LiDAR systems point cloud[J]. ISPRS International Journal of Geo-Information, 2019,8(9):404. doi: 10.3390/ijgi8090404
[4]	YAN Li, LIU Hua, TAN Junxiang, et al. A multi-constraint combined method for ground surface point filtering from mobile LiDAR point clouds[J]. Remote Sensing, 2017,9(9):958. doi: 10.3390/rs9090958
[5]	YADAV M, SINGH A K. Rural road surface extraction using mobile LiDAR point cloud data[J]. Journal of the Indian Society of Remote Sensing, 2018,46(4):531-538. doi: 10.1007/s12524-017-0732-4
[6]	XIA Shaobo, WANG Ruisheng. Extraction of residential building instances in suburban areas from mobile LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,144:453-468. doi: 10.1016/j.isprsjprs.2018.08.009
[7]	GARGOUM S A, KARSTEN L, EL-BASYOUNY K, et al. Automated assessment of vertical clearance on highways scanned using mobile LiDAR technology[J]. Automation in Construction, 2018,95:260-274. doi: 10.1016/j.autcon.2018.08.015
[8]	DÍAZ-VILARIÑO L, GONZÁLEZ-JORGE H, BUENO M, et al. Automatic classification of urban pavements using mobile LiDAR data and roughness descriptors[J]. Construction and Building Materials, 2016,102:208-215. doi: 10.1016/j.conbuildmat.2015.10.199
[9]	GARGOUM S A, El-BASYOUNY K, FROESE K, et al. A fully automated approach to extract and assess road cross sections from mobile lidar data[J]. IEEE Transactions on Intelligent Transportation Systems, 2018,19(11):3507-3516. doi: 10.1109/TITS.2017.2784623
[10]	DU Jianli, CHEN Dong, WANG Ruisheng, et al. A novel framework for 2.5-D building contouring from large-scale residential scenes[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019,57(6):4121-4145. doi: 10.1109/TGRS.36
[11]	DONG Zhen, LIANG Fuxun, YANG Bisheng, et al. Registration of large-scale terrestrial laser scanner point clouds: a review and benchmark[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020,163:327-342. doi: 10.1016/j.isprsjprs.2020.03.013
[12]	YANG Bisheng, LIU Yuan, DONG Zhen, et al. 3D local feature BKD to extract road information from mobile laser scanning point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017,130:329-343. doi: 10.1016/j.isprsjprs.2017.06.007
[13]	HUANG Bo, ZHAO Bei, SONG Yimeng. Urban land-use mapping using a deep convolutional neural network with high spatial resolution multispectral remote sensing imagery[J]. Remote Sensing of Environment, 2018,214:73-86. doi: 10.1016/j.rse.2018.04.050
[14]	PAOLETTI M E, HAUT J M, PLAZA J, et al. A new deep convolutional neural network for fast hyperspectral image classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,145:120-147. doi: 10.1016/j.isprsjprs.2017.11.021
[15]	MARCOS D, VOLPI M, KELLENBERGER B, et al. Land cover mapping at very high resolution with rotation equivariant CNNs: towards small yet accurate models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,145:96-107. doi: 10.1016/j.isprsjprs.2018.01.021
[16]	LIU Tao, ABD-ELRAHMAN A. Deep convolutional neural network training enrichment using multi-view object-based analysis of Unmanned Aerial systems imagery for wetlands classification[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,139:154-170. doi: 10.1016/j.isprsjprs.2018.03.006
[17]	ZHAO Wenzhi, DU Shihong. Learning multiscale and deep representations for classifying remotely sensed imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016,113:155-165. doi: 10.1016/j.isprsjprs.2016.01.004
[18]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas: IEEE, 2016: 770-778.
[19]	RIZALDY A, PERSELLO C, GEVAERT C, et al. Ground and multi-class classification of airborne laser scanner point clouds using fully convolutional networks[J]. Remote Sensing, 2018,10(11):1723. doi: 10.3390/rs10111723
[20]	HU Xiangyun, YUAN Yi. Deep-learning-based classification for DTM extraction from ALS point cloud[J]. Remote Sensing, 2016,8(9):730. doi: 10.3390/rs8090730
[21]	ZHANG Liqiang, ZHANG Liang. Deep learning-based classification and reconstruction of residential scenes from large-scale point clouds[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018,56(4):1887-1897. doi: 10.1109/TGRS.2017.2769120
[22]	XIANG Binbin, YAO Jian, LU Xiaohu, et al. Segmentation-based classification for 3D point clouds in the road environment[J]. International Journal of Remote Sensing, 2018,39(19):6182-6212. doi: 10.1080/01431161.2018.1455235
[23]	WEN Chenglu, SUN Xiaotian, LI J, et al. A deep learning framework for road marking extraction, classification and completion from mobile laser scanning point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019,147:178-192. doi: 10.1016/j.isprsjprs.2018.10.007
[24]	QIN Nannan, HU Xiangyun, WANG Puzuo, et al. Semantic labeling of ALS point cloud via learning voxel and pixel representations[J]. IEEE Geoscience and Remote Sensing Letters, 2020,17(5):859-863. doi: 10.1109/LGRS.8859
[25]	ZHANG Yongjun, XIONG Xiaodong, WANG Mengqiu, et al. A fast aerial image matching method using airborne LiDAR point cloud and POS data[J]. Journal of Geodesy and Geoinformation Science, 2019,2(1):26-36. DOI: 10.11947/j.JGGS.2019.0104. doi: 10.11947/j.JGGS.2019.0104
[26]	POUX F, BILLEN R. Voxel-based 3D point cloud semantic segmentation: unsupervised geometric and relationship featuring vs deep learning methods[J]. ISPRS International Journal of Geo-Information, 2019,8(5):213. doi: 10.3390/ijgi8050213
[27]	CHARLES R Q, SU Hao, MO Kaichun, et al. Pointnet: deep learning on point sets for 3D classification and segmentation[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu: IEEE, 2017: 77-85.
[28]	YOUSEFHUSSIEN M, KELBE D J, IENTILUCCI E J, et al. A multi-scale fully convolutional network for semantic labeling of 3D point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,143:191-204. doi: 10.1016/j.isprsjprs.2018.03.018
[29]	QI C R, YI Li, SU Hao, et al. Pointnet++: deep hierarchical feature learning on point sets in a metric space[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach: ACM, 2017: 5105-5114.
[30]	WU Zhirong, SONG Shuran, KHOSLA A, et al. 3D shapeNets: a deep representation for volumetric shapes[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1912-1920.
[31]	YU Yongtao, LI J, GUAN Haiyan, et al. Automated detection of three-dimensional cars in mobile laser scanning point clouds using DBM-Hough-Forests[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016,54(7):4130-4142. doi: 10.1109/TGRS.2016.2537830
[32]	WANG Lei, HUANG Yuchun, SHAN Jie, et al. MSNet: multi-scale convolutional network for point cloud classification[J]. Remote Sensing, 2018,10(4):612. doi: 10.3390/rs10040612
[33]	BOBKOV D, CHEN Sili, JIAN Ruiqing, et al. Noise-resistant deep learning for object classification in three-dimensional point clouds using a point pair descriptor[J]. IEEE Robotics and Automation Letters, 2018,3(2):865-872. doi: 10.1109/LSP.2016.
[34]	ZHAO Ruibin, PANG Mingyong, WANG Jidong. Classifying airborne LiDAR point clouds via deep features learned by a multi-scale convolutional neural network[J]. International Journal of Geographical Information Science, 2018,32(5):960-979. doi: 10.1080/13658816.2018.1431840
[35]	ZHANG Rui, LI Guangyun, LI Minglei, et al. Fusion of images and point clouds for the semantic segmentation of large-scale 3D scenes based on deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,143:85-96. doi: 10.1016/j.isprsjprs.2018.04.022
[36]	VETRIVEL A, GERKE M, KERLE N, et al. Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018,140:45-59. doi: 10.1016/j.isprsjprs.2017.03.001
[37]	YU Yongtao, LI J, GUAN Haiyan, et al. Learning hierarchical features for automated extraction of road markings from 3-D mobile LiDAR point clouds[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015,8(2):709-726. doi: 10.1109/JSTARS.4609443
[38]	MALTEZOS E, DOULAMIS N, DOULAMIS A, et al. Deep convolutional neural networks for building extraction from orthoimages and dense image matching point clouds[J]. Journal of Applied Remote Sensing, 2017,11(4):042620.
[39]	BOULCH A, GUERRY J, LE SAUX B, et al. SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks[J]. Computers & Graphics, 2018,71:189-198. doi: 10.1016/j.cag.2017.11.010
[40]	YANG Zhishuang, JIANG Wanshou, XU Bo, et al. A convolutional neural network-based 3D semantic labeling method for ALS point clouds[J]. Remote Sensing, 2017,9(9):936. doi: 10.3390/rs9090936
[41]	RIZALDY A, PERSELLO C, GEVAERT C M, et al. Fully convolutional networks for ground classification from LiDAR point clouds[J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-2:231-238. doi: 10.5194/isprs-annals-IV-2-231-2018
[42]	ZOU Xinhuai, CHENG Ming, WANG Cheng, et al. Tree classification in complex forest point clouds based on deep learning[J]. IEEE Geoscience and Remote Sensing Letters, 2017,14(12):2360-2364. doi: 10.1109/LGRS.2017.2764938
[43]	KUMAR B, PANDEY G, LOHANI B, et al. A multi-faceted CNN architecture for automatic classification of mobile LiDAR data and an algorithm to reproduce point cloud samples for enhanced training[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019,147:80-89. doi: 10.1016/j.isprsjprs.2018.11.006
[44]	ZHANG Le, SUN Jian, ZHENG Qiang. 3D point cloud recognition based on a multi-view convolutional neural network[J]. Sensors, 2018,18(11):3681. doi: 10.3390/s18113681
[45]	WANG Zhen, ZHANG Liqiang, ZHANG Liang, et al. A deep neural network with spatial pooling (DNNSP) for 3-D point cloud classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018,56(8):4594-4604. doi: 10.1109/TGRS.2018.2829625
[46]	BÖRCS A, NAGY B, BENEDEK C. Instant object detection in lidar point clouds[J]. IEEE Geoscience and Remote Sensing Letters, 2017,14(7):992-996. doi: 10.1109/LGRS.2017.2674799
[47]	LEDOUX H, GOLD C. An efficient natural neighbour interpolation algorithm for geoscientific modelling[C]//Proceedings of the 11th International Symposium on Spatial Data Handling. Leicester: SDH, 2004: 97-108.
[48]	QI Kunlun, YANG Chao, GUAN Qiangfeng, et al. A multiscale deeply described correlatons-based model for land-use scene classification[J]. Remote Sensing, 2017,9(9):917. doi: 10.3390/rs9090917

Set	Total	Ground	Building	Car	Tree	Curb	Fence	Street light	Utility pole	Electrical wire
S₁	1050774	447822	257125	18527	260738	22368	34199	3596	3980	2419
S₂	1074792	561797	177267	13206	125464	1633	186343	1913	4778	2391
S₃	975256	497100	137812	8526	207427	31429	80787	2907	3301	5967
S₄	724598	377269	129863	9879	129543	29026	37582	1181	5712	4543
S₅	713367	309787	102062	10898	247539	36597	1070	1921	2962	531
S₆	1239388	595236	355448	18288	255966	1122	1274	1598	7083	3373
S₇	1452821	772333	353405	28944	241177	0	43630	2023	7886	3423

Set	Method	Ground	Building	Car	Tree	Curb	Utility pole	Others
S₁	Xiang et al.’s^[22]	90.4/97.5	79.7/99.2	75.1/52.1	97.2/85.1	72.2/78.8	83.6/35.0	80.4/54.8
	Our method	99.8/98.1	97.9/87.5	94.1/95.0	98.8/93.4	94.6/94.9	90.9/98.7	92.5/95.9
S₂	Xiang et al.’s^[22]	98.4/99.4	88.6/85.3	96.2/61.1	99.2/97.1	94.1/84.3	96.7/85.8	85.8/91.0
	Our method	99.2/97.8	99.3/75.3	97.6/96.9	91.2/81.3	83.8/96.8	88.4/96.8	96.9/94.9
S₃	Xiang et al.’s^[22]	94.8/99.6	86.8/100	84.0/74.2	99.2/93.4	88.9/97.3	97.8/87.4	98.1/83.8
	Our method	98.1/88.2	95.4/77.3	96.5/96.9	94.9/86.6	94.6/95.9	85.7/97.7	94.7/96.8

Set	Method	Ground	Building	Car	Tree	Curb	Fence	Street light	Utility pole	Electrical wire
S₁	Our method	99.8/98.1	97.9/87.5	94.1/95.0	98.8/93.4	94.6/94.9	93.4/98.1	86.2/94.4	90.9/98.7	91.1/93.3
	ResNet	96.1/94.0	93.7/87.5	91.4/94.4	96.6/93.3	90.4/93.1	89.2/97.4	83.6/91.2	87.5/96.5	87.4/91.8
S₂	Our method	99.2/97.8	99.3/75.3	97.6/96.9	91.2/81.3	83.8/96.8	97.6/94.9	76.6/97.7	88.4/96.8	75.1/95.8
	ResNet	97.4/93.5	94.3/74.5	96.2/98.3	89.9/80.1	80.5/95.8	95.5/92.5	76.2/95.6	85.2/91.4	64.6/89.1
S₃	Our method	98.1/88.2	95.4/77.3	96.5/96.9	94.9/86.6	94.6/95.9	95.5/95.9	83.2/97.4	85.7/97.7	91.0/96.7
	ResNet	95.3/87.3	91.4/76.9	93.5/96.2	90.8/85.9	84.7/94.4	93.6/90.3	75.8/95.1	82.9/93.2	86.4/90.5

Set	Our method		ResNet
	T₁	T₂	T₁	T₂
S₁	218min 24s	15s	397min 42s	16s
S₂		18s		17s
S₃		10s		12s