The Review of Land Use/Land Cover Mapping AI Methodology and Application in the Era of Remote Sensing Big Data

doi:10.11947/j.JGGS.2024.0301

Abstract

Abstract:

With the increasing number of remote sensing satellites, the diversification of observation modals, and the continuous advancement of artificial intelligence algorithms, historically opportunities have been brought to the applications of earth observation and information retrieval, including climate change monitoring, natural resource investigation, ecological environment protection, and territorial space planning. Over the past decade, artificial intelligence technology represented by deep learning has made significant contributions to the field of Earth observation. Therefore, this review will focus on the bottlenecks and development process of using deep learning methods for land use/land cover mapping of the Earth's surface. Firstly, it introduces the basic framework of semantic segmentation network models for land use/land cover mapping. Then, we summarize the development of semantic segmentation models in geographical field, focusing on spatial and semantic feature extraction, context relationship perception, multi-scale effects modelling, and the transferability of models under geographical differences. Then, the application of semantic segmentation models in agricultural management, building boundary extraction, single tree segmentation and inter-species classification are reviewed. Finally, we discuss the future development prospects of deep learning technology in the context of remote sensing big data.

Key words: remote sensing big data; deep learning; semantic segmentation; land use/land cover mapping

ZHANG Xinchang, SHI Qian, SUN Ying, HUANG Jianfeng, HE Da. The Review of Land Use/Land Cover Mapping AI Methodology and Application in the Era of Remote Sensing Big Data[J]. Journal of Geodesy and Geoinformation Science, 2024, 7(3): 1-23.

Figures/Tables 15

Table 1

Figure 1

Figure 2

Table 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Table 3

Table 4

Figure 11

References 76

[1]	OVERPECK J T, MEEHL G A, BONY S, et al. Climate data challenges in the 21st century[J]. Science, 2011, 331(6018): 700-702. doi: 10.1126/science.1197869 pmid: 21311006
[2]	YANG Jun, GONG Peng, FU Rong, et al. The role of satellite remote sensing in climate change studies[J]. Nature Climate Change, 2013, 3(10): 875-883.
[3]	GONG Peng, LIU Han, ZHANG Meinan, et al. Stable classification with limited sample: transferring a 30-m resolution sample set collected in 2015 to mapping 10-m resolution global land cover in 2017[J]. Science Bulletin, 2019, 64(6): 370-373. doi: 10.1016/j.scib.2019.03.002 pmid: 36659725
[4]	KARRA K, KONTGIS C, STATMAN-WEIL Z, et al. Global land use / land cover with Sentinel 2 and deep learning[C]// Proceedings of 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. Brussels: IEEE, 2021: 4704-4707.
[5]	ZANAGA D, VAN DE KERCHOVE R, DAEMS D, et al. ESA WorldCover 10 m 2021 v200[EB/OL]. [2024-06-29]. https://developers.google.com/earth/engine/datasets/catalog/ESA_WorldCover_v200.
[6]	YANG J, HUANG X. The 30 m annual land cover dataset and its dynamics in China from 1990 to 2019[J]. Earth System Science Data, 2021, 13(8): 3907-3925.
[7]	ZHANG Ce, SARGENT I, PAN Xin, et al. An object-based convolutional neural network for urban land use classification[J]. Remote Sensing of Environment, 2018, 216: 57-70.
[8]	VOLPI M, TUIA D. Dense semantic labeling of subdecimeter resolution images with convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2): 881-893.
[9]	DENG Zhipeng, SUN Hao, ZHOU Shilin, et al. Multi-scale object detection in remote sensing imagery with convolutional neural networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145: 3-22.
[10]	BLASCHKE T. Object based image analysis for remote sensing[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2010, 65(1): 2-16.
[11]	DURO D C, FRANKLIN S E, DUBÉ M G. A comparison of pixel-based and object-based image analysis with selected machine learning algorithms for the classification of agricultural landscapes using SPOT-5 HRG imagery[J]. Remote Sensing of Environment, 2012, 118: 259-272.
[12]	FOODY G M. Land cover classification by an artificial neural network with ancillary information[J]. International Journal of Geographical Information Systems, 1995, 9(5): 527-542.
[13]	HU Ting, HUANG Xin, LI Jiayi, et al. A novel co-training approach for urban land cover mapping with unclear Landsat time series imagery[J]. Remote Sensing of Environment, 2018, 217: 144-157.
[14]	MYINT S W, GOBER P, BRAZEL A, et al. Per-pixel vs. object-based classification of urban land cover extraction using high spatial resolution imagery[J]. Remote Sensing of Environment, 2011, 115(5): 1145-1161.
[15]	YAN Gao, MAS J F, MAATHUIS B H P, et al. Comparison of pixel‐based and object‐oriented image classification approaches—a case study in a coal fire area, Wuda, Nei Mongol, China[J]. International Journal of Remote Sensing, 2006, 27(18): 4039-4055.
[16]	EL-MAGD I A, TANTON T W. Improvements in land use mapping for irrigated agriculture from satellite sensor data using a multi-stage maximum likelihood classification[J]. International Journal of Remote Sensing, 2003, 24(21): 4197-4206.
[17]	PEÑA J M, GUTIÉRREZ P A, HERVÁS-MARTÍNEZ C, et al. Object-based image classification of summer crops with machine learning methods[J]. Remote Sensing, 2014, 6(6): 5019-5041.
[18]	LÖW F, MICHEL U, DECH S, et al. Impact of feature selection on the accuracy and spatial uncertainty of per-field crop classification using support vector machines[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2013, 85: 102-119.
[19]	PEÑA-BARRAGÁN J M, NGUGI M K, PLANT R E, et al. Object-based crop identification using multiple vegetation indices, textural features and crop phenology[J]. Remote Sensing of Environment, 2011, 115(6): 1301-1316.
[20]	WATTS J D, LAWRENCE R L. Merging random forest classification with an object-oriented approach for analysis of agricultural lands[C]// Proceedings of International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences. Beijing: Chinese Society for Geodesy Photogrammetry and Cartography, 2008: 579-582.
[21]	CRUZ-RAMÍREZ M, HERVÁS-MARTÍNEZ C, JURADO-EXPÓSITO M, et al. A multi-objective neural network based method for cover crop identification from remote sensed data[J]. Expert Systems with Applications, 2012, 39(11): 10038-10048.
[22]	LIN C H, HUANG S H, LIN T H, et al. Metasurface-empowered snapshot hyperspectral imaging with convex/deep small-data learning theory[J]. Nature Communications, 2023, 14(1): 6979.
[23]	KEMKER R, SALVAGGIO C, KANAN C. Algorithms for semantic segmentation of multispectral remote sensing imagery using deep learning[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145: 60-77.
[24]	BALL J E, ANDERSON D T, CHAN C S. Comprehensive survey of deep learning in remote sensing: theories, tools, and challenges for the community[J]. Journal of Applied Remote Sensing, 2017, 11(4): 042609.
[25]	MA Lei, LIU Yu, ZHANG Xueliang, et al. Deep learning in remote sensing applications: a meta-analysis and review[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 152: 166-177. doi: 10.1016/j.isprsjprs.2019.04.015
[26]	ZHANG Liangpei, ZHANG Lefei, DU Bo. Deep learning for remote sensing data: a technical tutorial on the state of the art[J]. IEEE Geoscience and Remote Sensing Magazine, 2016, 4(2): 22-40.
[27]	ZHU Xiaoxiang, TUIA D, MOU Lichao, et al. Deep learning in remote sensing: a comprehensive review and list of resources[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4): 8-36.
[28]	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444.
[29]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe: Curran Associates Inc., 2012: 1097-1105.
[30]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[C]// Proceedings of the 9th International Conference on Learning Representations. Appleton: ICLR, 2021.
[31]	XIE Enze, WANG Wenhai, YU Zhiding, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]// Proceedings of the 34th Annual Conference on Red Hook. NY: Neural Information Processing Systems. 2021: 12077-12090.
[32]	WANG Libo, LI Rui, ZHANG Ce, et al. UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 196-214.
[33]	CAO Hu, WANG Yueyue, CHEN J, et al. Swin-unet: Unet-like pure transformer for medical image segmentation[C]// Proceedings of the European Conference on Computer Vision. Tel Aviv: Springer, 2022: 205-218.
[34]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3431-3440.
[35]	ZHANG Jing, LIN Shaofu, DING Lei, et al. Multi-scale context aggregation for semantic segmentation of remote sensing images[J]. Remote Sensing, 2020, 12(4): 701.
[36]	ZHU Qiqi, GUO Xi, DENG Weihuan, et al. Land-use/land-cover change detection based on a Siamese global learning framework for high spatial resolution remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 63-78.
[37]	KIRILLOV A, WU Yuxin, HE Kaiming, et al. PointRend: image segmentation as rendering[C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 9796-9805.
[38]	TUIA D, PERSELLO C, BRUZZONE L. Domain adaptation for the classification of remote sensing data: an overview of recent advances[J]. IEEE Geoscience and Remote Sensing Magazine, 2016, 4(2): 41-57.
[39]	KOUW W M, LOOG M. An introduction to domain adaptation and transfer learning[EB/OL]. [2024-06-29]. https://arxiv.org/abs/1812.11806.
[40]	JI Shunping, WEI Shiqing, LU Meng. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1): 574-586.
[41]	LIU Yongcheng, FAN Bin, WANG Lingfeng, et al. Semantic labeling in very high resolution images via a self-cascaded convolutional neural network[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 145: 78-95.
[42]	LIU Penghua, LIU Xiaoping, LIU Mengxi, et al. Building footprint extraction from high-resolution images via spatial residual inception convolutional neural network[J]. Remote Sensing, 2019, 11(7): 830.
[43]	JI Shunping, WEI Shiqing, LU Meng. A scale robust convolutional neural network for automatic building extraction from aerial and satellite imagery[J]. International Journal of Remote Sensing, 2019, 40(9): 3308-3322.
[44]	CHEN L C, ZHU Yukun, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]// Proceedings of the 15th European Conference on Computer Vision. Munich: Springer, 2018: 833-851.
[45]	MOU Lichao, ZHU Xiaoxiang. RiFCN: recurrent network in fully convolutional network for semantic segmentation of high resolution remote sensing images[EB/OL]. [2024-06-29]. https://arxiv.org/abs/1805.02091
[46]	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 936-944.
[47]	ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]// Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6230-6239.
[48]	WANG Jingdong, SUN Ke, CHENG Tianheng, et al. Deep high-resolution representation learning for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3349-3364.
[49]	SUN Ying, ZHANG Xinchang, XIN Qinchuan, et al. Developing a multi-filter convolutional neural network for semantic segmentation using high-resolution aerial imagery and LiDAR data[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 143: 3-14.
[50]	YUAN Jiangye. Learning building extraction in aerial scenes with convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(11): 2793-2798. doi: 10.1109/TPAMI.2017.2750680 pmid: 28910757
[51]	BISCHKE B, HELBER P, FOLZ J, et al. Multi-task learning for segmentation of building footprints with deep neural networks[C]// Proceedings of 2019 IEEE International Conference on Image Processing (ICIP). Taipei, China: IEEE, 2019: 1480-1484.
[52]	MARMANIS D, SCHINDLER K, WEGNER J D, et al. Classification with an edge: improving semantic image segmentation with boundary detection[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 135: 158-172.
[53]	SHI Qian, ZHU Jiajun, LIU Zhengyu, et al. The last puzzle of global building footprints—mapping 280 million buildings in east Asia based on VHR images[J]. Journal of Remote Sensing, 2024, 4: 0138.
[54]	HUANG Jianfeng, ZHANG Xinchang, SUN Ying, et al. Attention-guided label refinement network for semantic segmentation of very high resolution aerial orthoimages[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 4490-4503.
[55]	DAI Wenyuan, YANG Qiang, XUE Guirong, et al. Boosting for transfer learning[C]// Proceedings of the 24th International Conference on Machine Learning. Corvalis: ACM, 2007: 193-200.
[56]	PERSELLO C, BRUZZONE L. A novel active learning strategy for domain adaptation in the classification of remote sensing images[C]// Proceedings of 2011 IEEE International Geoscience and Remote Sensing Symposium. Vancouver: IEEE, 2011: 3720-3723.
[57]	MATASCI G, TUIA D, KANEVSKI M. SVM-based boosting of active learning strategies for efficient domain adaptation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2012, 5(5): 1335-1343.
[58]	MATASCI G, VOLPI M, KANEVSKI M, et al. Semisupervised transfer component analysis for domain adaptation in remote sensing image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(7): 3550-3564.
[59]	PAN S J, TSANG I W, KWOK J T, et al. Domain adaptation via transfer component analysis[J]. IEEE Transactions on Neural Networks, 2011, 22(2): 199-210. doi: 10.1109/TNN.2010.2091281 pmid: 21095864
[60]	BRUZZONE L, MARCONCINI M. Toward the automatic updating of land-cover maps by a domain-adaptation SVM classifier and a circular validation strategy[J]. IEEE Transactions on Geoscience and Remote Sensing, 2009, 47(4): 1108-1122.
[61]	LI Jingjing, JING Mengmeng, LU Ke, et al. Locality preserving joint transfer for domain adaptation[J]. IEEE Transactions on Image Processing, 2019, 28(12): 6103-6115, doi: 10.1109/TIP.2019.2924174. pmid: 31251190
[62]	HAN Chengxi, WU Chen, GUO Haonan, et al. HANet: a hierarchical attention network for change detection with bitemporal very-high-resolution remote sensing images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 3867-3878.
[63]	DONG Lijia, GENG Jie, JIANG Wen. Spectral-spatial enhancement and causal constraint for hyperspectral image cross-scene classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5507013.
[64]	LIU Mengxi, ZHANG Pengyuan, SHI Qian, et al. An adversarial domain adaptation framework with KL-constraint for remote sensing land cover classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 3002305.
[65]	MÜLLER H, RUFIN P, GRIFFITHS P, et al. Mining dense Landsat time series for separating cropland and pasture in a heterogeneous Brazilian savanna landscape[J]. Remote Sensing of Environment, 2015, 156: 490-499.
[66]	VUOLO F, NEUWIRTH M, IMMITZER M, et al. How much does multi-temporal Sentinel-2 data improve crop type classification?[J]. International Journal of Applied Earth Observation and Geoinformation, 2018, 72: 122-130.
[67]	WARDLOW B D, EGBERT S L. Large-area crop mapping using time-series MODIS 250 m NDVI data: an assessment for the U.S. Central Great Plains[J]. Remote Sensing of Environment, 2008, 112(3): 1096-1116.
[68]	JOSHI N, BAUMANN M, EHAMMER A, et al. A review of the application of optical and radar remote sensing data fusion to land use mapping and monitoring[J]. Remote Sensing, 2016, 8(1): 70.
[69]	BOUVET A, LE TOAN T. Use of ENVISAT/ASAR wide-swath data for timely rice fields mapping in the Mekong River Delta[J]. Remote Sensing of Environment, 2011, 115(4): 1090-1101.
[70]	ZHOU Lichen, ZHANG Chuang, WU Ming. D-LinkNet: LinkNet with pretrained encoder and dilated convolution for high resolution satellite imagery road extraction[C]// Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Salt Lake City: IEEE, 2018: 192-1924.
[71]	GRADOS D, REYNARFAJE X, SCHREVENS E. A methodological approach to assess canopy NDVI-based tomato dynamics under irrigation treatments[J]. Agricultural Water Management, 2020, 240: 106208.
[72]	LI Jintao, CHENG Xiaojun, XIAO Zhihua. A branch-trunk-constrained hierarchical clustering method for street trees individual extraction from mobile laser scanning point clouds[J]. Measurement, 2022, 189: 110440.
[73]	FAN Zhun, LU Jiewei, GONG Maoguo, et al. Automatic tobacco plant detection in UAV images via deep neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3): 876-887.
[74]	BAYRAKTAR E, BASARKAN M E, CELEBI N. A low-cost UAV framework towards ornamental plant detection and counting in the wild[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 167: 1-11.
[75]	VALENTE J, SARI B, KOOISTRA L, et al. Automated crop plant counting from very high-resolution aerial imagery[J]. Precision Agriculture, 2020, 21(6): 1366-1384.
[76]	HARTLING S, SAGAN V, SIDIKE P, et al. Urban tree species classification using a WorldView-2/3 and LiDAR data fusion approach and deep learning[J]. Sensors, 2019, 19(6): 1284.