Enhanced Directional Light Vector Estimation From a Virtual AR Object and its 2D Shadow Mask

Aleksander Yacovenco; Luiz Maurílio da Silva Maciel; Marcelo Bernardes Vieira; Rodrigo Luis de Souza da Silva

doi:10.5753/jbcs.2026.7529

Authors

Aleksander Yacovenco Federal University of Juiz de Fora https://orcid.org/0009-0003-1344-337X
Luiz Maurílio da Silva Maciel Federal University of Juiz de Fora https://orcid.org/0000-0001-9193-2302
Marcelo Bernardes Vieira Federal University of Juiz de Fora https://orcid.org/0000-0003-3356-6679
Rodrigo Luis de Souza da Silva Federal University of Juiz de Fora https://orcid.org/0000-0002-4187-8798

DOI:

https://doi.org/10.5753/jbcs.2026.7529

Keywords:

Directional Light Estimation, Shadow Estimation, 3D lighting, Augmented Reality

Abstract

This work presents a novel method for inferring the main directional light source in a 3D scene, given only 2D inputs, namely a camera image and a rough shadow mask. A two stage algorithm is proposed, in which the first stage handles the inputs and makes an initial estimation for the light source. The second stage refines this first estimate to find a new vector assumed to be closer to the real directional light vector. One virtual object is rendered into the real scene in both stages. In the first stage, an external shadow estimator produces a coarse shadow for the virtual object, enabling the computation of an initial directional light vector. Next, our proposal is to compare the coarse shadow with the virtual shadow computed from the object. Thus, in the second stage, we seek to maximize the intersection over union (IoU) between both shadows. We assume that the best directional light vector provides the best shadow matching. The experiments are made both in virtual and real environments, in scenes with different levels of control and known data. Results show that our method is capable of finding the 3D light vector from the 2D scene, enhancing the initial rough shadow input.

Downloads

Download data is not yet available.

References

Azuma, R. T. (1997). A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4):355-385. DOI: 10.1162/pres.1997.6.4.355.

Boom, B. J., Orts-Escolano, S., Ning, X. X., McDonagh, S., Sandilands, P., and Fisher, R. B. (2017). Interactive light source position estimation for augmented reality with an rgb-d camera. Computer Animation and Virtual Worlds, 28(1):e1686. DOI: 10.1002/cav.1686.

Cao, X. and Foroosh, H. (2007). Camera calibration and light source orientation from solar shadows. Computer Vision and Image Understanding, 105(1):60-72. DOI: 10.1016/j.cviu.2006.08.003.

Chen, Y., Wang, Q., Chen, H., Song, X., Tang, H., and Tian, M. (2019). An overview of augmented reality technology. In Journal of Physics: Conference Series, volume 1237, page 022082. IOP Publishing. DOI: 10.1088/1742-6596/1237/2/022082.

Chotikakamthorn, N. (2015). Near point light source location estimation from shadow edge correspondence. In 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), pages 30-35. IEEE. DOI: 10.1109/iccis.2015.7274543.

Frahm, J.-M., Koeser, K., Grest, D., and Koch, R. (2005). Markerless augmented reality with light source estimation for direct illumination. In Conference on Visual Media Production CVMP, London, pages 211-220. IET Stevenage. Avaialble at:[link].

Gruber, L., Richter-Trummer, T., and Schmalstieg, D. (2012). Real-time photometric registration from arbitrary geometry. In 2012 IEEE international symposium on mixed and augmented reality (ISMAR), pages 119-128. IEEE. DOI: 10.1109/ismar.2012.6402548.

Kán, P. and Kaufmann, H. (2019). Deeplight: light source estimation for augmented reality using deep learning. The Visual Computer, 35:873-883. DOI: 10.1007/s00371-019-01666-x.

LeGendre, C., Ma, W.-C., Fyffe, G., Flynn, J., Charbonnel, L., Busch, J., and Debevec, P. (2019). Deeplight: Learning illumination for unconstrained mobile mixed reality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5918-5928. DOI: 10.1109/cvpr.2019.00607.

Liu, C., Wang, L., Li, Z., Quan, S., and Xu, Y. (2022). Real-time lighting estimation for augmented reality via differentiable screen-space rendering. IEEE Transactions on Visualization and Computer Graphics, 29(4):2132-2145. DOI: 10.1109/tvcg.2022.3141943.

Liu, D., Long, C., Zhang, H., Yu, H., Dong, X., and Xiao, C. (2020). Arshadowgan: Shadow generative adversarial network for augmented reality in single light scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8139-8148. DOI: 10.1109/cvpr42600.2020.00816.

Liu, D. S.-M. and Wu, S.-J. (2022). Light direction estimation and hand touchable interaction for augmented reality. Virtual Reality, 26(3):1155-1172. DOI: 10.1007/s10055-022-00624-8.

Lopez-Moreno, J., Garces, E., Hadap, S., Reinhard, E., and Gutierrez, D. (2013). Multiple light source estimation in a single image. In Computer Graphics Forum, volume 32, pages 170-182. Wiley Online Library. DOI: 10.1111/cgf.12195.

Marques, B. A. D., Clua, E. W. G., Montenegro, A. A., and Vasconcelos, C. N. (2022). Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality. Computers & Graphics, 102:257-268. DOI: 10.1016/j.cag.2021.08.007.

Meilland, M., Barat, C., and Comport, A. (2013). 3d high dynamic range dense visual slam and its application to real-time object re-lighting. In 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 143-152. IEEE. DOI: 10.1109/ismar.2013.6671774.

Nguyen, R. M. and Le, M. N. (2012). Light source estimation from a single image. In 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV), pages 1358-1363. IEEE. DOI: 10.1109/icarcv.2012.6485343.

Otsu, N. (1979). A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics, 9(1):62-66. DOI: 10.1109/tsmc.1979.4310076.

Sun, Y.-k., Li, D., Liu, S., Cao, T.-C., and Hu, Y.-S. (2020). Learning illumination from a limited field-of-view image. In 2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE. DOI: 10.1109/icmew46912.2020.9105957.

Suzuki, S. et al. (1985). Topological structural analysis of digitized binary images by border following. Computer vision, graphics, and image processing, 30(1):32-46. DOI: 10.1016/0734-189x(85)90136-7.

Wang, Y. and Samaras, D. (2002). Estimation of multiple directional light sources for synthesis of mixed reality images. In 10th Pacific Conference on Computer Graphics and Applications, 2002. Proceedings., pages 38-47. IEEE. DOI: 10.1016/s1524-0703(03)00043-2.

Wang, Z., Chen, W., Acuna, D., Kautz, J., and Fidler, S. (2022). Neural light field estimation for street scenes with differentiable virtual object insertion. In European Conference on Computer Vision, pages 380-397. Springer. DOI: 10.1007/978-3-031-20086-1char`_22.

Whelan, T., Salas-Moreno, R. F., Glocker, B., Davison, A. J., and Leutenegger, S. (2016). Elasticfusion: Real-time dense slam and light source estimation. The International Journal of Robotics Research, 35(14):1697-1716. DOI: 10.1177/0278364916669237.

Yacovenco, A., da Silva, R., Vieira, M., and Maciel, L. (2025). Directional light vector estimation from a virtual ar object and its 2d shadow mask. In Anais do XXVII Simpósio de Realidade Virtual e Aumentada. SBC. DOI: 10.1109/SVR67689.2025.00025.