An Adaptive and Image-guided Fusion for Stereo Satellite Image Derived Digital Surface Models

doi:10.11947/j.JGGS.2022.0401

Abstract

Abstract:

The accuracy of Digital Surface Models (DSMs) generated using stereo matching methods varies due to the varying acquisition conditions and configuration parameters of stereo images. It has been a good practice to fuse these DSMs generated from various stereo pairs to achieve enhanced, in which multiple DSMs are combined through computational approaches into a single, more accurate, and complete DSM. However, accurately characterizing detailed objects and their boundaries still present a challenge since most boundary-ware fusion methods still struggle to achieve sharpened depth discontinuities due to the averaging effects of different DSMs. Therefore, we propose a simple and efficient adaptive image-guided DSM fusion method that applies k-means clustering on small patches of the orthophoto to guide the pixel-level fusion adapted to the most consistent and relevant elevation points. The experiment results show that our proposed method has outperformed comparing methods in accuracy and the ability to preserve sharpened depth edges.

Key words: Digital Surface Model (DSM); DSM Fusion; adaptive fusion; satellite stereo images

Hessah ALBANWAN, Rongjun QIN. An Adaptive and Image-guided Fusion for Stereo Satellite Image Derived Digital Surface Models[J]. Journal of Geodesy and Geoinformation Science, 2022, 5(4): 1-9.

Figures/Tables 7

Fig.1

Fig.2

Tab.1

Fig.3

Fig.4

Fig.5

Tab.2

References 33

[1]	ALBANWAN H, QIN Rongjun, LU Xiaohu, et al. 3D iterative spatiotemporal filtering for classification of multitemporal satellite data sets[J]. Photogrammetric Engineering & Remote Sensing, 2020, 86(1): 23-31. DOI: 10.14358/PERS.86.1.23. doi: 10.14358/PERS.86.1.23
[2]	ALBANWAN H, QIN R. Enhancement of depth map by fusion using adaptive and semantic-guided spatiotemporal filtering[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2020V-3: 227-232. DOI: 10.5194/isprs-annals-V-3-2020-227-2020. doi: 10.5194/isprs-annals-V-3-2020-227-2020
[3]	FACCIOLO G, DE FRANCHIS C, MEINHARDT-LLOPIS E. Automatic 3D reconstruction from multi-date satellite images[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI: IEEE, 2017: 1542-1551. DOI: 10.1109/CVPRW.2017.198. doi: 10.1109/CVPRW.2017.198
[4]	QIN Rongjun, SONG Shuang, LING Xiao, et al. 3D reconstruction through fusion of cross-view images[EB/OL]. [2022-12-20].
[5]	RUMPLER M, WENDEL A, BISCHOF H. Probabilistic range image integration for DSM and true-orthophoto generation[C]// Proceedings of the 18th Scandinavian Conference on Image Analysis. Espoo: Springer, 2013: 533-544. DOI: 10.1007/978-3-642-38886-6_50. doi: 10.1007/978-3-642-38886-6_50
[6]	TOUTIN T. DSM generation and evaluation from quickbird stereo imagery with 3D physical modelling[J]. International Journal of Remote Sensing, 2004, 25(22): 5181-5192. DOI: 10.1080/01431160410001726030. doi: 10.1080/01431160410001726030
[7]	CHANG Jiaren, CHEN Yongsheng. Pyramid stereo matching network[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT: IEEE, 2018: 5410-5418. DOI: 10.1109/CVPR.2018.00567. doi: 10.1109/CVPR.2018.00567
[8]	CHENG Xuelian, ZHONG Yiran, HARANDI M, et al. Hierarchical neural architecture search for deep stereo matching[C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: Curran Associates Inc., 2020: 22158-22169.
[9]	HIRSCHMULLER H. Accurate and efficient stereo processing by semi-global matching and mutual information[C]// Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). San Diego, CA: IEEE, 2005: 807-814. DOI: 10.1109/CVPR.2005.56. doi: 10.1109/CVPR.2005.56
[10]	KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]// Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Venice:IEEE, 2017: 66-75. DOI: 10.1109/ICCV.2017.17. doi: 10.1109/ICCV.2017.17
[11]	ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA: IEEE, 2015: 1592-1599. DOI: 10.1109/CVPR.2015.7298767. doi: 10.1109/CVPR.2015.7298767
[12]	ALBANWAN H, QIN R. Adaptive and non-adaptive fusion algorithms analysis for digital surface model generated using census and convolutional neural networks[J]// The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. 2021- XLIII-B2: 283-288. DOI: 10.5194/isprs-archives-XLIII-B2-2021-283-2021. doi: 10.5194/isprs-archives-XLIII-B2-2021-283-2021
[13]	QIN Rongjun. Automated 3D recovery from very high resolution multi-view satellite images[C]// Proceedings of Presented at the ASPRS Conference (IGTF) 2017. Baltimore, MD: [s.n.], 2017.
[14]	CHEN Ruijin, GAO Wei. Color-guided depth map super-resolution using a dual-branch multi-scale residual network with channel interaction[J]. Sensors, 2020, 20(6): 1560. DOI: 10.3390/s20061560. doi: 10.3390/s20061560
[15]	DONG Weisheng, SHI Guangming, LI Xin, et al. Color-guided depth recovery via joint local structural and nonlocal low-rank regularization[J]. IEEE Transactions on Multimedia, 2017, 19(2): 293-301. DOI: 10.1109/TMM.2016.2613824. doi: 10.1109/TMM.2016.2613824
[16]	PO L M, XU Xuyuan, ZHU Yuesheng, et al. Automatic 2D-to-3D video conversion technique based on depth-from-motion and color segmentation[C]// Proceedings of IEEE 10th International Conference on Signal Processing Proceedings. Beijing: IEEE, 2010: 1000-1003. DOI: 10.1109/ICOSP.2010.5655850. doi: 10.1109/ICOSP.2010.5655850
[17]	QI Fei, HAN Junyu, WANG Pengjin, et al. Structure guided fusion for depth map inpainting[J]. Pattern Recognition Letters, 2013, 34(1): 70-76. DOI: 10.1016/j.patrec.2012.06.003. doi: 10.1016/j.patrec.2012.06.003
[18]	PAPPAS T N, JAYANT N S. An adaptive clustering algorithm for image segmentation[C]// Proceedings of International Conference on Acoustics, Speech, and Signal Processing. Glasgow: IEEE, 1989: 1667-1670. DOI: 10.1109/ICASSP.1989.266767. doi: 10.1109/ICASSP.1989.266767
[19]	BOSCH M, KURTZ Z, HAGSTROM S, et al. A multiple view stereo benchmark for satellite imagery[C]// Proceedings of 2016 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). Washington, WA: IEEE, 2016: 1-9. DOI: 10.1109/AIPR.2016.8010543. doi: 10.1109/AIPR.2016.8010543
[20]	LE SAUX B, YOKOYA N, HÄNSCH R, et al. Data fusion contest 2019 (DFC2019)[M]. [S.l.]:IEEE DataPort, 2019. DOI: 10.21227/C6TM-VW12. doi: 10.21227/C6TM-VW12
[21]	SCHARSTEIN D, SZELISKI R, ZABIH R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithm[C]// Proceedings of IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001). Kauai: IEEE, 2001: 131-140. DOI: 10.1109/SMBV.2001.988771. doi: 10.1109/SMBV.2001.988771
[22]	HAMID M S, MANAP N A, HAMZAH R A, et al. Stereo matching algorithm based on deep learning: a survey[J]. Journal of King Saud University - Computer and Information Sciences, 2022, 34(5): 1663-1673. DOI: 10.1016/j.jksuci.2020.08.011. doi: 10.1016/j.jksuci.2020.08.011
[23]	LAGA H. A survey on deep learning architectures for image-based depth reconstruction[EB/OL]. [2022-10-12]. https://arxiv.org/abs/1906.06113.
[24]	ZHOU Kun, MENG Xiangxi, CHENG Bo. Review of stereo matching algorithms based on deep learning[J]. Computational Intelligence and Neuroscience, 2020, 2020: 8562323. DOI: 10.1155/2020/8562323. doi: 10.1155/2020/8562323
[25]	HAN Yilong, LIU Wei, HUANG Xu, et al. Stereo dense image matching by adaptive fusion of multiple-window matching results[J]. Remote Sensing, 2020, 12(19): 3138. DOI: 10.3390/rs12193138. doi: 10.3390/rs12193138
[26]	HIRSCHMULLER H. Stereo processing by semiglobal matching and mutual information[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 328-341. DOI: 10.1109/TPAMI.2007.1166. doi: 10.1109/TPAMI.2007.1166 pmid: 18084062
[27]	MA Li, LI Jingjiao, MA Ji, et al. A modified census transform based on the neighborhood information for stereo matching algorithm[C]// Proceedings of 2013 Seventh International Conference on Image and Graphics. Qingdao: IEEE, 2013: 533-538. DOI: 10.1109/ICIG.2013.113. doi: 10.1109/ICIG.2013.113
[28]	QIN Rongjun. A critical analysis of satellite stereo pairs for digital surface model generation and a matching quality prediction model[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 154: 139-150. DOI: 10.1016/j.isprsjprs.2019.06.005. doi: 10.1016/j.isprsjprs.2019.06.005
[29]	PAPASAIKA H, KOKIOPOULOU E, BALTSAVIAS E, et al. Fusion of digital elevation models using sparse representations[C]// Proceedings of ISPRS Conference on Photogrammetric Image Analysis. Munich: Springer, 2011: 171-184.
[30]	GRUEN A, AKCA D. Least squares 3D surface and curve matching[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2005, 59(3): 151-174. DOI: 10.1016/j.isprsjprs.2005.02.006. doi: 10.1016/j.isprsjprs.2005.02.006
[31]	SHEN Huanfeng, LI Xinghua, CHENG Qing, et al. Missing information reconstruction of remote sensing data: a technical review[J]. IEEE Geoscience and Remote Sensing Magazine, 2015, 3(3): 61-85. DOI: 10.1109/MGRS.2015.2441912. doi: 10.1109/MGRS.2015.2441912
[32]	TOU J T, GONZALEZ R C. Pattern recognition principles[M]. London: Addison-Wesley, 1974.
[33]	QIN R. Rpc stereo processor (RSP)—a software package for digital surface model and orthophoto generation from satellite stereo imagery[C]// Proceedings of ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Prague: [s.n.], 2016: 77-82. DOI: 10.5194/isprs-annals-III-1-77-2016. doi: 10.5194/isprs-annals-III-1-77-2016

	OMA and JAX	ARG
Satellite sensor	WorldView-3	WorldView-3
Spatial resolution/cm	30	30
Acquisition times of the stereo pairs	September 2014— February 2016	November 2014— January 2016

Dataset ID	Fusion method
Dataset ID	Median fusion	k-median clustering fusion	Adaptive median fusion	Adaptive spatiotemporal fusion	Adaptive image- guided fusion
OMA I	6.39	6.72	6.38	6.36	6.08
OMA II	6.47	3.75	4.33	4.65	3.40
JAX I	5.76	5.41	5.60	5.60	5.34
JAX II	4.11	3.78	3.87	3.90	3.45
ARG I	3.69	3.49	3.30	3.68	3.02
ARG II	6.38	6.64	6.30	6.35	5.84
Average RMSE/m	5.47	4.97	4.96	5.09	4.52