Photogrammetry and Deep Learning

doi:10.11947/j.JGGS.2018.0101

Abstract

Abstract:

Deep learning has become popular and the mainstream technology in many researches related to learning, and has shown its impact on photogrammetry. According to the definition of photogrammetry, that is, a subject that researches shapes, locations, sizes, characteristics and inter-relationships of real objects from optical images, photogrammetry considers two aspects, geometry and semantics. From the two aspects, we review the history of deep learning and discuss its current applications on photogrammetry, and forecast the future development of photogrammetry. In geometry, the deep convolutional neural network (CNN) has been widely applied in stereo matching, SLAM and 3D reconstruction, and has made some effects but needs more improvement. In semantics, conventional methods that have to design empirical and handcrafted features have failed to extract the semantic information accurately and failed to produce types of “semantic thematic map” as 4D productions (DEM, DOM, DLG, DRG) of photogrammetry. This causes the semantic part of photogrammetry be ignored for a long time. The powerful generalization capacity, ability to fit any functions and stability under types of situations of deep leaning is making the automatic production of thematic maps possible. We review the achievements that have been obtained in road network extraction, building detection and crop classification, etc., and forecast that producing high-accuracy semantic thematic maps directly from optical images will become reality and these maps will become a type of standard products of photogrammetry. At last, we introduce our two current researches related to geometry and semantics respectively. One is stereo matching of aerial images based on deep learning and transfer learning; the other is precise crop classification from satellite spatio-temporal images based on 3D CNN.

Key words: deep learning; convolutional neural network; photogrammetry; stereo matching; thematic map

Jianya GONG,Shunping JI. Photogrammetry and Deep Learning[J]. Journal of Geodesy and Geoinformation Science, 2018, 1(1): 1-15.

Figures/Tables 8

Fig.1

Tab.1

Fig.2

Tab.2

Fig.3

Fig.4

Tab.3

References 48

[1]	GONG Jianya, JI Shunping . From Photogrammetry to Computer Vision[J]. Geomatics and Information Science of Wuhan University, 2017,42(11):1518-1522.
[2]	BOYLE W S, SMITH G E . Charge Coupled Semiconductor Devices[J]. The Bell System Technical Journal, 1970,49(4):587-593. DOI: 10.1002/bltj.1970.49.issue-4 doi: 10.1002/bltj.1970.49.issue-4
[3]	ASHBY W R . An Introduction to Cybernetics[M]. London: Chapman & Hall Ltd, 1961.
[4]	FODOR J A, PYLYSHYN Z W . Connectionism and Cognitive Architecture:A Critical Analysis[J]. Cognition, 1988,28(1-2):3-71. DOI: 10.1016/0010-0277(88)90031-5. doi: 10.1016/0010-0277(88)90031-5
[5]	HINTON G E, OSINDERO S, TEH Y W . A Fast Learning Algorithm for Deep Belief Nets[J]. Neural Computation, 2006,18(7):1527-1554. DOI: 10.1162/neco.2006.18.7.1527. doi: 10.1162/neco.2006.18.7.1527
[6]	SUYKENS J A K, VANDERWALLE J . Least Squares Support Vector Machine Classifiers[J]. Neural Processing Letters, 1999,9(3):293-300. DOI: 10.1023/A:1018628609742. doi: 10.1023/A:1018628609742
[7]	KOLLER D, FRIEDMAN N. Probabilistic Graphical Models:Principles and Techniques[M]. Cambridge: MIT Press, 2009.
[8]	BENGIO Y, LAMBLIN P, POPOVICI D , et al. Greedy Layer-wise Training of Deep Networks [C]//Proceedings of the 19th International Conference on Neural Information Processing Systems. Canada: ACM, 2006: 153-160.
[9]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E . Imagenet Classification with Deep Convolutional Neural Networks [C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe, Nevada: ACM, 2012: 1097-1105.
[10]	MEHTA P, SCHWAB D J . An Exact Mapping between the Variational Renormalization Group and Deep Learning[J]. arXiv Preprint arXiv: 1410. 3831, 2014.
[11]	TISHBY N, PEREIRA F C, BIALEK W . The Information Bottleneck Method[J]. arXiv Preprint arXiv: physics/0004057, 2000.
[12]	HINTON G, DENG Li, YU Dong , et al. Deep Neural Networks for Acoustic Modeling in Speech Recognition:The Shared Views of Four Research Groups[J]. IEEE Signal Processing Magazine, 2012,29(6):82-97. DOI: 10.1109/MSP.2012.2205597. doi: 10.1109/MSP.2012.2205597
[13]	LECUN Y, BOSER B, DENKER J S , et al. Backpropagation Applied to Handwritten Zip Code Recognition[J]. Neural Computation, 1989,1(4):541-551. DOI: 10.1162/neco.1989.1.4.541. doi: 10.1162/neco.1989.1.4.541
[14]	KENDALL A, GRIMES M, CIPOLLA R . Posenet: A Convolutional Network for Real-time 6-dof Camera Relocalization [C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 2938-2946.
[15]	KITTI. The KITTI Vision Benchmark Suite[DB/OL].[2018-03-01]. .
[16]	BENGIO Y, COURVILLE A, VINCENT P . Representation Learning:A Review and New Perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013,35(8):1798-1828. DOI: 10.1109/TPAMI.2013.50. doi: 10.1109/TPAMI.2013.50
[17]	NG A . Sparse Autoencoder[J]. CS294A Lecture Notes, 2011,72(2011):1-19.
[18]	SANGER T D . Optimal Unsupervised Learning in A Single-layer Linear Feedforward Neural Network[J]. Neural Networks, 1989,2(6):459-473. DOI: 10.1016/0893-6080(89)90044-0. doi: 10.1016/0893-6080(89)90044-0
[19]	RUCK D W, ROGERS S K, KABRISKY M , et al. The Multilayer Perceptron as an Approximation to A Bayes Optimal Discriminant Function[J]. IEEE Transactions on Neural Networks, 1990,1(4):296-298. DOI: 10.1109/72.80266.
[20]	MIKOLOV T, KARAFIÁT M, BURGET L , et al. Recurrent Neural Network Based Language Model [C]//Proceedings of the 11th Annual Conference of the International Speech Communication Association.Makuhari, Chiba, Japan: International Speech Communication Association, 2010,2:3.
[21]	MINSKY M L, PAPERT S A. Perceptrons[M]. Cambridge: MIT Press, 1969.
[22]	NAIR V, HINTON G E . Rectified Linear Units Improve Restricted Boltzmann Machines [C]//Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel: ACM, 2010: 807-814.
[23]	SHORE J, JOHNSON R . Axiomatic Derivation of the Principle of Maximum Entropy and the Principle of Minimum Cross-entropy[J]. IEEE Transactions on Information Theory, 1980,26(1):26-37.DOI: 10.1109/TIT.1980.1056144. doi: 10.1109/TIT.1980.1056144
[24]	MORÉ JJ . The Levenberg-Marquardt Algorithm: Implementation and Theory[M]//WATSON G A. Numerical Analysis. Berlin, Heidelberg: Springer, 1978: 105-116.
[25]	LE CUN Y, BOSER B E, DENKER J S , et al. Handwritten Digit Recognition with a Back-propagation Network[M]//TOURETZKY D S. Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann Publishers Inc., 1990: 396-404.
[26]	GOODFELLOW I, BENGIO Y, COURVILLE A. Deep Learning[M]. Cambridge, Massachusetts: MIT Press, 2016.
[27]	HORN B. Robot Vision[M]. Cambridge: MIT Press, 1986.
[28]	GRAHAM B . Fractional Max-pooling[J]. arXiv Preprint arXiv: 1412. 6071, 2014.
[29]	ZEILER M D, FERGUS R . Visualizing and UnderstandingConvolutional Networks [C]//European Conference on Computer Vision. Zurich, Switzerland: Springer, 2014: 818-833.
[30]	SZEGEDY C, LIU W, JIA Y , et al. Going Deeper with Convolutions[J]. arXiv Preprint arXiv: 1409. 4842, 2014.
[31]	SIMONYAN K, ZISSERMAN A . Very Deep Convolutional Networks for Large-scale Image Recognition[J]. arXiv Preprint arXiv: 1409. 1556, 2014.
[32]	HEKaiming, ZHANG Xianyu, REN Shaoqing , et al. Deep Residual Learning for Image Recognition [C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 2016: 770-778.
[33]	KENDALL A, CIPOLLA R . Modelling Uncertainty in Deep Learning for Camera Relocalization [C]//Proceedings of 2016 IEEE International Conference on Robotics and Automation. Stockholm, Sweden: IEEE, 2016: 4762-4769.
[34]	ŽBONTAR J, LECUN Y . Computing the Stereo Matching Cost with a Convolutional Neural Network [C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, MA: IEEE, 2015: 1592-1599.
[35]	KENDALL A, MARTIROSYAN H, DASGUPTA S , et al. End-to-end Learning of Geometry and Context for Deep Stereo Regression [C]//Proceedings of the IEEE Conference on Computer Vision. Venice, Italy: IEEE, 2017: 66-75.
[36]	SEKI A, POLLEFEYS M . SGM-Nets: Semi-global Matching with Neural Networks [C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, HI: IEEE, 2017: 21-26.
[37]	MAYER N, ILG E, HÄUSSER P , et al. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation [C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 2016: 4040-4048.
[38]	LUO Wenjie, SCHWING A G, URTASUN R . Efficient Deep Learning for Stereo Matching [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV: IEEE, 2016: 5695-5703.
[39]	MARR D . Vision:A Computational Investigation into the Human Representation and Processing of Visual Information[M]. San Francisco: W.H.Freeman and Company, 1982.
[40]	CHENG Guangliang, WANG Ying, XU Shibiao , et al. Automatic Road Detection and Centerline Extraction via Cascaded End-to-end Convolutional Neural Network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017,55(6):3322-3337. DOI: 10.1109/TGRS.2017.2669341. doi: 10.1109/TGRS.2017.2669341
[41]	LIPeikang, ZANG Yu, WANG Cheng , et al. Road Network Extraction via Deep Learning and Line Integral Convolution [C]//Proceedings of the IEEE Conference on Geoscience and Remote Sensing Symposium (IGARSS). Beijing, China: IEEE, 2016: 1599-1602.
[42]	MNIH V, HINTON G E . Learning to Detect Roads in High-resolution Aerial Images [C]//Proceedings of the 11th European Conference on Computer Vision.Heraklion, Crete, Greece: Springer, 2010: 210-223.
[43]	WANG Jun, SONG Jingwei, CHEN Mingquan , et al. Road Network Extraction:A Neural-dynamic Framework Based on Deep Learning and a Finite State Machine[J]. International Journal of Remote Sensing, 2015,36(12):3144-3169. DOI: 10.1080/01431161.2015.1054049. doi: 10.1080/01431161.2015.1054049
[44]	PANBOONYUEN T, JITKAJORNWANICH K, LAWAWIROJWONG S , et al. Road Segmentation of Remotely-sensed Images Using Deep Convolutional Neural Networks with Landscape Metrics and Conditional Random Fields[J]. Remote Sensing, 2017,9(7):680. DOI: 10.3390/rs9070680. doi: 10.3390/rs9070680
[45]	VAKALOPOULOU M, KARANTZALOS K, KOMODAKIS N , et al. Building Detection in Very High Resolution Multispectral Data with Deep Learning Features [C]//Proceedings of the IEEE Conference on Geoscience and Remote Sensing Symposium (IGARSS). Milan, Italy: IEEE, 2015: 1873-1876.
[46]	KUSSUL N, LAVRENIUK M, SKAKUN S , et al. Deep Learning Classification of Land Cover and Crop Types Using Remote Sensing Data[J]. IEEE Geoscience and Remote Sensing Letters, 2017,14(5):778-782. DOI: 10.1109/LGRS.2017.2681128. doi: 10.1109/LGRS.2017.2681128
[47]	CASTELLUCCIO M, POGGI G, SANSONE C , et al. Land Use Classification in Remote Sensing Images by Convolutional Neural Networks[J]. arXiv Preprint arXiv: 1508. 00092, 2015.
[48]	ZHANG Liangpei, ZHANG Lefei, DU Bo . Deep Learning for Remote Sensing Data:A Technical Tutorial on the State of the Art[J]. IEEE Geoscience and Remote Sensing Magazine, 2016,4(2):22-40. DOI: 10.1109/MGRS.2016.2540798. doi: 10.1109/MGRS.2016.2540798

method	SGM (A)	KITTI2012	KITTI2015	KITTI	A	K to A	K+A
training acc	\	0.0221	0.0263	0.0258	0.0137	\	0.0136
test acc	0.100	0.0564	0.0580	0.0580	0.0467	0.0560	0.0456

dataset	band	period	size	GSD(m)	type	training sample	test sample
Ⅰ	4	4	1417×2652	4	4	400	2000
Ⅱ	4	7	1163×2102	4	10	3180	890

Test acc Classification acc	DATASET Ⅰ
		2D CNN	3D CNN	SVM	KNN	PCA+KNN
	OA	0.938	0.943	0.935	0.924	0.9215
	OA	0.935	0.939	0.932	0.927	0.9277
Test acc Classification acc	DATASET Ⅱ
		2D CNN	3D CNN	SVM	KNN	PCA+KNN
	OA	0.961	0.973	0.785	0.764	0.745
	OA	0.923	0.932	0.896	0.889	0.845