Fully Convolutional Networks for Street Furniture Identification in Panorama Images

doi:10.11947/j.JGGS.2022.0406

Abstract

Abstract:

Panoramic images are widely used in many scenes, especially in virtual reality and street view capture. However, they are new for street furniture identification which is usually based on mobile laser scanning point cloud data or conventional 2D images. This study proposes to perform semantic segmentation on panoramic images and transformed images to separate light poles and traffic signs from background implemented by pre-trained Fully Convolutional Networks (FCN). FCN is the most important model for deep learning applied on semantic segmentation for its end to end training process and pixel-wise prediction. In this study, we use FCN-8s model that pre-trained on cityscape dataset and finetune it by our own data. Then replace cross entropy loss function with focal loss function in the FCN model and train it again to produce the predictions. The results show that in all results from pre-trained model, fine-tuning, and FCN model with focal loss, the light poles and traffic signs are detected well and the transformed images have better performance than panoramic images in the prediction according to the Recall and IoU evaluation.

Key words: panoramic images; semantic segmentation; street furniture; object identification; fully convolutional networks

Ying AO, Penglong LI, Li WEN, Tao ZHANG, Yanwen WANG. Fully Convolutional Networks for Street Furniture Identification in Panorama Images[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(4): 59-71.

Figures/Tables 22

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Tab.1

Fig.13

Tab.2

Tab.3

Tab.4

Tab.5

Tab.6

Tab.7

Fig.14

Fig.15

References 33

[1]	CHEN Xiaozhi, MA Huimin, WAN Ji, et al. Multi-view 3D object detection network for autonomous driving[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI: IEEE, 2017: 6526-6534.
[2]	ESS A, SCHINDLER K, LEIBE B, et al. Object detection and tracking for autonomous navigation in dynamic environments[J]. The International Journal of Robotics Research, 2010, 29(14): 1707-1725. doi: 10.1177/0278364910365417
[3]	BENAVIDEZ P, JAMSHIDI M. Mobile robot navigation and target tracking system[C]// Proceedings of the 6th International Conference on System of Systems Engineering. Albuquerque, NM: IEEE, 2011: 299-304.
[4]	CREUSEN I M, HAZELHOFF L, DE WITH P H N. A semi-automatic traffic sign detection, classification, and positioning system[C]// Proceedings of SPIE 8305, Visual Information Processing and Communication III. Burlingame, CA: SPIE, 2012.
[5]	HAZELHOFF L, CREUSEN I M, DE WITH P H N. Exploiting street-level panoramic images for large-scale automated surveying of traffic signs[J]. Machine Vision and Applications, 2014, 25(7): 1893-1911. doi: 10.1007/s00138-014-0628-z
[6]	MARINHO L B, ALMEIDA J S, SOUZA J W M, et al. A novel mobile robot localization approach based on topological maps using classification with reject option in omnidirectional images[J]. Expert Systems with Applications, 2017, 72: 1-17. doi: 10.1016/j.eswa.2016.12.007
[7]	PINTORE G, GANOVELLI F, GOBBETTI E, et al. Mobile reconstruction and exploration of indoor structures exploiting omnidirectional images[C]// Proceedings of SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications. Macau, China: ACM, 2016: 1.
[8]	PAPARODITIS N, PAPELARD J P, CANNELLE B, et al. Stereopolis II: a multi-purpose and multi-sensor 3D mobile mapping system for street visualisation and 3D metrology[J]. Revue Francaise de Photogrammetrie et de Teledetection, 2012, 200: 69-79.
[9]	SAXENA A, SUN Min, NG A Y. Make3D: learning 3D scene structure from a single still image[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(5): 824-840. doi: 10.1109/TPAMI.2008.132 pmid: 19299858
[10]	HE Hu, UPCROFT B. Nonparametric semantic segmentation for 3D street scenes[C]// Proceedings of 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems. Tokyo: IEEE, 2013: 3697-3703.
[11]	ALHO P, VAAJA M, KUKKO A, et al. Mobile laser scanning in fluvial geomorphology: mapping and change detection of point bars[J]. Zeitschrift Für Geomorphologie, Supplementary Issues, 2011, 55(2): 31-50. doi: 10.1127/0372-8854/2011/0055S2-0044
[12]	PU Shi, RUTZINGER M, VOSSELMAN G, et al. Recognizing basic structures from mobile laser scanning data for road inventory studies[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2011, 66(6): S28-S39. doi: 10.1016/j.isprsjprs.2011.08.006
[13]	CABO C, ORDOÑEZ C, GARCÍA-CORTÉS S, et al. An algorithm for automatic detection of pole-like street furniture objects from Mobile Laser Scanner point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 87: 47-56. doi: 10.1016/j.isprsjprs.2013.10.008
[14]	WANG Jinhu, LINDENBERGH R, MENENTI M. SigVox-A 3D feature matching algorithm for automatic street object recognition in mobile laser scanning point clouds[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2017, 128: 111-129. doi: 10.1016/j.isprsjprs.2017.03.012
[15]	RODRÍGUEZ-CUENCA B, GARCÍA-CORTÉS S, ORDÓÑEZ C, et al. Automatic detection and classification of pole-like objects in urban point cloud data using an anomaly detection algorithm[J]. Remote Sensing, 2015, 7(10): 12680-12703. doi: 10.3390/rs71012680
[16]	KRYLOV V A, KENNY E, DAHYOT R. Automatic discovery and geotagging of objects from street view imagery[J]. Remote Sensing, 2018, 10(5): 661. doi: 10.3390/rs10050661
[17]	HANG Weixing, WITHARANA C, LI Weidong, et al. Using deep learning to identify utility poles with crossarms and estimate their locations from google street view images[J]. Sensors, 2018, 18(8): 2484. doi: 10.3390/s18082484
[18]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice:IEEE, 2017: 2999-3007.
[19]	CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes data-set for semantic urban scene understanding[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV: IEEE, 2016.
[20]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. doi: 10.1145/3065386
[21]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA: IEEE, 2015: 3431-3440.
[22]	GARCIA-GARCIA A, ORTS-ESCOLANO S, OPREA S, et al. A review on deep learning techniques applied to semantic segmentation[EB/OL]. [2022-03-09]. https://arxiv.org/abs/1704.06857.
[23]	ZENG A, YU K T, SONG Shuran, et al. Multi-view self-supervised deep learning for 6D pose estimation in the Amazon picking challenge[C]// Proceedings of 2017 IEEE International Conference on Robotics and Automation. Singapore: IEEE, 2017: 1386-1393.
[24]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848. doi: 10.1109/TPAMI.2017.2699184
[25]	YANG M Y, LIAO W, LI X, et al. Vehicle detection in aerial images[EB/OL]. [2022-03-09]. https://arxiv.org/abs/1801.07339.
[26]	LIU Wiyang, WEN Yandong, YU Zhiding, et al. Large-margin softmax loss for convolutional neural networks[C]// Proceedings of the 33rd International Conference on International Conference on Machine Learning. New York, NY: JMLR.org, 2016: 507-516.
[27]	WEN Yandong, ZHANG Kaipeng, LI Zhifeng, et al. A discriminative feature learning approach for deep face recognition[C]// Proceedings of the 14th European Conference on Computer Vision. Amsterdam: Springer, 2016: 499-515.
[28]	SU Yuchuan, GRAUMAN K. Learning spherical convolution for fast features from 360° imagery[C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, CA: Curran Associates Inc., 2017.
[29]	COORS B, CONDURACHE A P, GEIGER A. SphereNet: learning spherical representations for detection and classification in omnidirectional images[C]// Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018.
[30]	WEISSTEIN E W. Gnomonic projection. From mathworld—a wolfram web resource[EB/OL]. [2022-09-21]. http://mathworld.wolfram.com/GnomonicProjection.html.
[31]	JUNE CHOI. Projects-gnomonic_projection[EB/OL]. [2022-08-13]. https://github.com/june-choi/projects-gnomonic_projection.
[32]	SHRIVASTAVA A, GUPTA A, GIRSHICK R. Training region-based object detectors with online hard example mining[C]// Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV: IEEE, 2016.
[33]	LYU Y, VOSSELMAN G, XIA G, et al. The uavid dataset for video semantic segmentation[EB/OL]. [2022-03-09]. https://arxiv.org/abs/1810.10438.

Class	Panoramic images	Transformed images
Light poles	74.06%	68.30%
Traffic sign	71.24%	78.16%
Average	72.65%	73.23%

Class	Panoramic images	Transformed images
Light poles	96.72%	94.13%
Traffic sign	84.62%	88.47%
Average	90.68%	91.30%

Class	Panoramic images	Transformed images
Light poles	94.36%	94.67%
Traffic sign	87.58%	85.39%
Average	90.97%	90.03%

Class	Panoramic images	Transformed images
Light poles	2.33%	2.26%
Traffic sign	3.43%	3.88%
Average	2.88%	3.07%

Class	Panoramic images	Transformed images
Light poles	38.44%	39.66%
Traffic sign	20.62%	33.16%
Mean	29.53%	36.41%