LoD2 Building Reconstruction from Stereo Satellite Imagery using Deep Learning and Model-Driven Approach

Rojgar Qarani Ismael; Haval Sadeq

doi:10.21271/ZJPAS.37.2.10

Authors

Rojgar Qarani Ismael Department of Geomatics (Surveying) Engineering, College of Engineering, Salahaddin University-Erbil, Erbil, Kurdistan Region, Iraq. https://orcid.org/0000-0001-9618-0082
Haval Sadeq Salahaddin University-Erbil/College of Engineering/Department of Geomatics (Surveying) Engineering https://orcid.org/0000-0003-0636-473X

DOI:

https://doi.org/10.21271/ZJPAS.37.2.10

Keywords:

LoD2, Deep Learning, DSM, Model-Driven

Abstract

This study presents a Level of Detail 2 building reconstruction approach for open and occluded areas from stereo-satellite imagery. The approach combines deep learning techniques, and digital surface models with model-driven methodology. The best performance of deep learning algorithms (U-Net, FCN, and Mask R-CNN) for building boundary segmentation was selected and then integrated with model-driven technique for the purpose of accurate geometric building fitting employing digital surface model (DSM) generated by semi global matching. The Reconstructed model was refined by utilizing OpenStreetMap library and graph cut optimization method. The suggested methodology is tested on the GeoEye-1 satellite imagery dataset for Erbil City, which is validated with ground truth data. The proposed algorithm presented promising results, it is shown that the model can predict building heights for ridge and eave to a mean absolute error of 0.70 m, and in the occluded area was approximately 1.0 m. Meanwhile, the computed root mean square error are shown to be within 0.9 m for the ridge and eave, which is essentially small. While for occluded area it was approximately 1.2 m and 0.8 m for ridge and eave heights, respectively. This indicates that the predicted values are close to real values. Furthermore, most of the building’s roofs were correctly classified in both open and occluded areas. These findings underline the effectiveness of the model-driven deep learning approach in producing reliable and accurate LoD2 building reconstructions, a precondition for detailed urban analysis and 3D city modeling.

References

Akiki, R., Mari, R., De Franchis, C., Morel, J.-M. & Facciolo, G. Robust Rational Polynomial Camera Modelling for SAR and Pushbroom Imaging. 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, 2021/7// 2021 Brussels, Belgium. IEEE, 7908-7911.

Alidoost, F., Arefi, H. & Tombari, F. 2019. 2D image-to-3D model: Knowledge-based 3D building reconstruction (3DBR) using single aerial images and convolutional neural networks (CNNs). Remote Sensing, 11.

Alin, A. Y., Kusrini, K. & Yuana, K. A. 2023. The Effect of Data Augmentation in Deep Learning with Drone Object Detection. IJCCS (Indonesian Journal of Computing and Cybernetics Systems), 17, 237-248.

Alsabhan, W., Alotaiby, T. & Dudin, B. 2022. Detecting Buildings and Nonbuildings from Satellite Images Using U-Net. Computational Intelligence and Neuroscience, 2022.

Amarù, S., Marelli, D., Ciocca, G. & Schettini, R. 2023. DALib: A Curated Repository of Libraries for Data Augmentation in Computer Vision. J Imaging, 9.

Amo-Boateng, M., Ekow Nkwa Sey, N., Ampah Amproche, A. & Kyereh Domfeh, M. 2022. Instance segmentation scheme for roofs in rural areas based on Mask R-CNN. Egyptian Journal of Remote Sensing and Space Science, 25, 569-577.

Amro, I., Mateos, J., Vega, M., Molina, R. & Katsaggelos, A. K. 2011. A survey of classical methods and new trends in pansharpening of multispectral images. EURASIP Journal on Advances in Signal Processing, 2011.

Bittner, K., Adam, F., Cui, S., Körner, M. & Reinartz, P. 2018a. Building Footprint Extraction From VHR Remote Sensing Images Combined With Normalized DSMs Using Fused Fully Convolutional Networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11, 2615-2629.

Bittner, K., D'angelo, P., Körner, M. & Reinartz, P. 2018b. DSM-to-LoD2: Spaceborne stereo digital surface model refinement. Remote Sensing, 10.

Buyukdemircioglu, M., Kocaman, S. & Kada, M. DEEP LEARNING FOR 3D BUILDING RECONSTRUCTION: A REVIEW. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2, 2022. 359-366.

Dorninger, P. & Pfeifer, N. 2008. A comprehensive automated 3D approach for building extraction, reconstruction, and regularization from airborne laser scanning point clouds. Sensors, 8, 7323-7343.

Dos Santos, R. C., Galo, M. & Habib, A. F. 2020. Regularization of building roof boundaries from airborne LiDAR data using an iterative CD-spline. Remote Sensing, 12.

Douglas, D. H. & Peucker, T. K. 1973. ALGORITHMS FOR THE REDUCTION OF THE NUMBER OF POINTS REQUIRED TO REPRESENT A DIGITIZED LINE OR ITS CARICATURE. Cartographica, 10, 112-122.

Duan, L. & Lafarge, F. Towards large-scale city reconstruction from satellites. European Conference on Computer Vision (ECCV), 2016/9// 2016. Computer Vision – ECCV 2016, 89-104.

Dukai, B., Ledoux, H. & Stoter, J. E. A multi-height LOD1 model of all buildings in the Netherlands. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2019. Copernicus GmbH, 51-57.

Dukai, B., Peters, R., Vitalis, S., Liempt, J. V. & Stoter, J. QUALITY ASSESSMENT of A NATIONWIDE DATA SET CONTAINING AUTOMATICALLY RECONSTRUCTED 3D BUILDING MODELS. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 2021/9// 2021. International Society for Photogrammetry and Remote Sensing, 17-24.

Dutta, A. & Zisserman, A. The VIA Annotation Software for Images, Audio and Video. Proceedings of the 27th ACM International Conference on Multimedia, 2019/10// 2019 New York, NY, USA. 2276-2279.

Gao, W., Peters, R., Ledoux, H. & Stoter, J. Filling holes in LoD2 building models. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2024/6// 2024. Copernicus Publications, 171-177.

Grompone Von Gioi, R., Jakubowicz, J., Morel, J.-M. & Randall, G. 2010. LSD: A fast line segment detector with a false detection control. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32, 722-732.

Gui, S., Qin, R. & Tang, Y. 2022. Sat2Lod2: a Software for Automated Lod-2 Building Reconstruction From Satellite-Derived Orthophoto and Digital Surface Model. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 43, 379-386.

Han, Q., Yin, Q., Zheng, X. & Chen, Z. 2022. Remote sensing image building detection method based on Mask R-CNN. Complex and Intelligent Systems, 8, 1847-1855.

He, K., Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), 2017/10// 2017. IEEE, 2980-2988.

Huang, H., Michelini, M., Schmitz, M., Roth, L. & Mayer, H. LOD3 BUILDING RECONSTRUCTION from MULTI-SOURCE IMAGES. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, 2020/9// 2020. International Society for Photogrammetry and Remote Sensing, 427-434.

Huang, J., Stoter, J., Peters, R. & Nan, L. 2022. City3D: Large-Scale Building Reconstruction from Airborne LiDAR Point Clouds. Remote Sensing, 14.

Lai, F. & Yang, X. 2020. Integrating spectral and non-spectral data to improve urban settlement mapping in a large Latin-American city. GIScience & Remote Sensing, 57, 830-844.

Li, T., Jiang, C., Bian, Z., Wang, M. & Niu, X. 2020. A Review of True Orthophoto Rectification Algorithms. Materials Science and Engineering, 780, 22-35.

Long, J., Shelhamer, E. & Darrell, T. Fully Convolutional Networks for Semantic Segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. IEEE, 3431-3440.

Lv, B., Liu, J., Wang, P. & Yasir, M. 2022. DSM Generation from Multi-View High-Resolution Satellite Images Based on the Photometric Mesh Refinement Method. Remote Sensing, 14, 6259-6259.

Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N. & Terzopoulos, D. 2022. Image Segmentation Using Deep Learning: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 3523-3542.

Muftah, H., Rowan, T. S. L. & Butler, A. P. 2022. Towards open-source LOD2 modelling using convolutional neural networks. Modeling Earth Systems and Environment, 8, 1693-1709.

Noureldeen, A. & Wahed, M. E. 2024. Enhanced building footprint extraction from satellite imagery using Mask R-CNN and PointRend. Bulletin of Electrical Engineering and Informatics, 13, 3601-3608.

Nys, G. A., Poux, F. & Billen, R. 2020. City json building generation from airborne LiDAR 3D point clouds. ISPRS International Journal of Geo-Information, 9.

Partovi, T., Fraundorfer, F., Bahmanyar, R., Huang, H. & Reinartz, P. 2019. Automatic 3-D building model reconstruction from very high resolution stereo satellite imagery. Remote Sensing, 11.

Peters, R., Dukai, B., Vitalis, S., Van Liempt, J. & Stoter, J. 2022. Automated 3D Reconstruction of LoD2 and LoD1 Models for All 10 Million Buildings of the Netherlands. Photogrammetric Engineering and Remote Sensing, 88, 165-170.

Proulx-Bourque, J.-S., Mathieu, L., Papasodoro, C., Pilon, D., Sabo, N. & Pelchat, M. T. Experiment on the Impact of Spatial Resolution on Building Extraction Accuracy. 2019 IEEE International Geoscience and Remote Sensing Symposium, 2019/8// 2019 Yokohama, Japan. IEEE.

Ps, P. & Aithal, B. H. 2023. Building footprint extraction from very high-resolution satellite images using deep learning. Journal of Spatial Science, 68, 487-503.

Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation. ISBI challenge for segmentation of neuronal structures in electron microscopic stacks, 2015/9// 2015.

Stucker, C. & Schindler, K. 2022. ResDepth: A deep residual prior for 3D reconstruction from high-resolution satellite images. ISPRS Journal of Photogrammetry and Remote Sensing, 183, 560-580.

Sugihara, K., Murase, T. & Zhou, X. Automatic generation of 3D building models from building polygons on digital maps. 2015 International Conference on 3D Imaging (IC3D), 2015/12// 2015 Liege, Belgium. IEEE.

Wagner, F. H., Dalagnol, R., Tarabalka, Y., Segantine, T. Y. F., Thomé, R. & Hirye, M. C. M. 2020. U-net-id, an instance segmentation model for building extraction from satellite images-Case study in the Joanopolis City, Brazil. Remote Sensing, 12.

Wang, Y., Zorzi, S. & Bittner, K. Machine-learned 3D Building Vectorization from Satellite Imagery. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021/6// 2021. IEEE, 1072-1081.

Weng, Q. Essential Urban Variables from Satellite Observations: An Introduction. IEEE, 2018/9// 2018. 2018 IEEE International Geoscience and Remote Sensing Symposium.

Wysocki, O., Hoegner, L. & Stilla, U. MLS2LoD3: Refining low LoDs building models with MLS point clouds to reconstruct semantic LoD3 building models. In: THOMAS H. KOLBE, A. D., CHRISTOF BEIL, ed. 18th 3D GeoInfo Conference, 2024/9// 2024. Springer, 367-380.

Xu, B., Zhang, X., Li, Z., Leotta, M., Chang, S.-F. & Shan, J. 2020. Deep Learning Guided Building Reconstruction from Satellite Imagery-derived Point Clouds. ISPRS Journal of Photogrammetry and Remote Sensing.

Zhang, S., Han, F. & Bogus, S. M. Building Footprint and Height Information Extraction from Airborne LiDAR and Aerial Imagery. Construction Research Congress 2020, 2020/11// 2020 Reston, VA. American Society of Civil Engineers, 326-335.

Zhang, Y., Zhang،, Y., Mo،, D., Zhang, Y. & Li, X. 2017. Direct Digital Surface Model generation by semi-global vertical line locus matching. Remote Sensing, 9.

Zhao, K., Kang, J., Jung, J. & Sohn, G. Building Extraction from Satellite Images Using Mask R-CNN with Building Boundary Regularization. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018, pp. 247-251, 2018. 247-251.