Comparing Perceptual Visual Quality of Hybrid Neural and Ray Tracing in Foveated Rendering

Horácio Henriques; Alan de Oliveira; Eder Oliveira; Esteban Clua; Daniela Trevisan

doi:10.5753/jis.2025.5593

Authors

Horácio Henriques Fluminense Federal University https://orcid.org/0000-0003-1147-4749
Alan de Oliveira Federal Institute of Education, Science and Technology of Rondonia https://orcid.org/0009-0006-5042-2326
Eder Oliveira Fluminense Federal University https://orcid.org/0000-0002-6437-4883
Esteban Clua Fluminense Federal University https://orcid.org/0000-0001-5650-1718
Daniela Trevisan Fluminense Federal University https://orcid.org/0000-0003-0061-4689

DOI:

https://doi.org/10.5753/jis.2025.5593

Keywords:

Virtual Reality, Foveated Rendering, Perception-based Rendering, Ray Tracing, NeRF, 3D Gaussian Splatting

Abstract

Enabling Foveated Rendering for VR devices displays is fundamental when dealing with real-time ray tracing. Combining traditional methods with Neural based strategies, such as NeRFs and 3D Gaussian Splatting, may impact on leveraging performance even more. In this work we enhance and validate how well our traditional Instant-NeRF reconstructs common ray-traced effects through user metrics and quality metrics. We also show that 3D Gaussian Splatting used in the periphery vision area presents better results than the perceptual quality achieved through NeRFs. We present a deep human perception experiment through different global illumination light effects.

Downloads

Download data is not yet available.

References

Albert, R., Patney, A., Luebke, D., and Kim, J. (2017). Latency requirements for foveated rendering in virtual reality. ACM Transactions on Applied Perception (TAP), 14(4):1–13. DOI: https://doi.org/10.1145/3127589.

Avidan, S. and Shashua, A. (1997). Novel view synthesis in tensor space. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1034–1040. IEEE.

Banks, M. S., Sekuler, A. B., and Anderson, S. J. (1991). Peripheral spatial vision: Limits imposed by optics, photoreceptors, and receptor pooling. JOSA A, 8(11):1775–1787. DOI: https://doi.org/10.1364/josaa.8.001775.

Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, P. (2022). Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479. DOI: https://doi.org/10.1109/CVPR52688.2022.00539.

Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, P. (2023). Zip-NeRF: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697–19705.

Bates, D., Mächler, M., Bolker, B. M., and Walker, S. C. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1). DOI: https://doi.org/10.18637/jss.v067.i01.

Buehler, C., Bosse, M., McMillan, L., Gortler, S., and Cohen, M. (2001). Unstructured lumigraph rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pages 425–432.

Caulfield, B. (2022). What is path tracing? NVIDIA Blog. [link] (accessed May 08, 2025).

Chen, X., Liu, J., Zhao, H., Zhou, G., and Zhang, Y.-Q. (2023a). NeRRF: 3D reconstruction and view synthesis for transparent and specular objects with neural refractive-reflective fields. arXiv preprint arXiv:2309.13039. DOI: https://doi.org/10.48550/arXiv.2309.13039.

Chen, Z., Funkhouser, T., Hedman, P., and Tagliasacchi, A. (2023b). MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16569–16578. DOI: https://doi.org/10.1109/cvpr52729.2023.01590.

Cui, Z., Gu, L., Sun, X., Ma, X., Qiao, Y., and Harada, T. (2024). Aleth-NeRF: Illumination adaptive NeRF with concealing field assumption. In Proceedings of the AAAI Conference on Artificial Intelligence, 38:1435–1444. DOI: https://doi.org/10.1609/aaai.v38i2.27908.

Davis, A., Levoy, M., and Durand, F. (2012). Unstructured light fields. In Computer Graphics Forum, 2:305–314.

Debevec, P. E., Taylor, C. J., and Malik, J. (2023). Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 465–474.

Deng, N., He, Z., Ye, J., Duinkharjav, B., Chakravarthula, P., Yang, X., and Sun, Q. (2022). FOV-NeRF: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics, 28(11):3854–3864. DOI: https://doi.org/10.1109/TVCG.2022.3203102.

Deng, W., Campbell, D., Sun, C., Kanitkar, S., Shaffer, M., and Gould, S. (2024). Ray deformation networks for novel view synthesis of refractive objects. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3118–3128. DOI: https://doi.org/10.1109/wacv57701.2024.00309.

Dix, A., Finlay, J., Abowd, G. D., and Beale, R. (2003). Human-Computer Interaction. Pearson Education.

Duchowski, A. T., Bate, D., Stringfellow, P., Thakur, K., Melloy, B. J., and Gramopadhye, A. K. (2009). On spatiochromatic visual sensitivity and peripheral color LOD management. ACM Transactions on Applied Perception (TAP), 6(2):1–18. DOI: https://doi.org/10.1145/1498700.1498703.

Einhorn, E. and Mawdsley, J. (2023). Generate groundbreaking ray-traced images with next-generation NVIDIA DLSS. [link] (accessed May 13, 2025).

Elkin, L., Kay, M., Higgins, J., and Wobbrock, J. (2021). An aligned rank transform procedure for multifactor contrast tests. In UIST 2021 – Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology, pages 754–768. Association for Computing Machinery. DOI: https://doi.org/10.1145/3472749.3474784.

Feng, Y., Feng, X., Shang, Y., Jiang, Y., Yu, C., Zong, Z., Shao, T., Wu, H., Zhou, K., Jiang, C., and Yang, Y. (2024). Gaussian splashing: Dynamic fluid synthesis with Gaussian splatting. CoRR, abs/2401.15318.

Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., and Kanazawa, A. (2022). Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501–5510. DOI: https://doi.org/10.1109/CVPR52688.2022.00542.

Gao, Q., Xu, Q., Cao, Z., Mildenhall, B., Ma, W., Chen, L., Tang, D., and Neumann, U. (2024). GaussianFlow: Splatting Gaussian dynamics for 4D content creation. arXiv preprint arXiv:2403.12365. DOI: https://doi.org/10.48550/arXiv.2403.12365.

Genova, K., Cole, F., Maschinot, A., Sarna, A., Vlasic, D., and Freeman, W. T. (2018). Unsupervised training for 3D morphable model regression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8377–8386.

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. (1996). The lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH ’96), pages 43–54. Association for Computing Machinery. DOI: https://doi.org/10.1145/237170.237200.

Guenter, B., Finch, M., Drucker, S., Tan, D., and Snyder, J. (2012). Foveated 3D graphics. ACM Transactions on Graphics (TOG), 31(6):1–10. DOI: https://doi.org/10.1145/2366145.2366183.

Guo, Z., Zhou, W., Li, L., Wang, M., and Li, H. (2025). Motion-aware 3D Gaussian splatting for efficient dynamic scene reconstruction. IEEE Transactions on Circuits and Systems for Video Technology, 35(4):3119–3133. DOI: https://doi.org/10.1109/TCSVT.2024.3502257.

Henriques, H., de Oliveira, A., Oliveira, E., Trevisan, D., and Clua, E. (2024a). Foveated path culling: A mixed path tracing and radiance field approach for optimizing rendering in XR displays. Journal on Interactive Systems, 15(1):576–590. DOI: https://doi.org/10.5753/jis.2024.4352.

Henriques, H., Oliveira, E., Clua, E., and Trevisan, D. (2024b). Analysing hybrid neural and ray tracing perception for foveated rendering. In Proceedings of the 26th Symposium on Virtual and Augmented Reality, pages 21–30. DOI: https://doi.org/10.1145/3691573.3691580.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, pages 65–70.

Hore, A. and Ziou, D. (2010). Image quality metrics: PSNR vs. SSIM. In 2010 20th International Conference on Pattern Recognition, pages 2366–2369. IEEE. DOI: https://doi.org/10.1109/ICPR.2010.579.

Jiang, Y., Tu, J., Liu, Y., Gao, X., Long, X., Wang, W., and Ma, Y. (2023). Gaussianshader: 3D Gaussian splatting with shading functions for reflective surfaces. DOI: https://doi.org/10.48550/arXiv.2311.17977.

Kajiya, J. T. (1986). The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pages 143–150. DOI: https://doi.org/10.1145/15922.15902.

Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. (2023). 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1–14. DOI: https://doi.org/10.1145/3592433.

Kilgariff, E., Moreton, H., Stam, N., and Bell, B. (2018). NVIDIA Turing Architecture In-Depth | NVIDIA Technical Blog. [link] (accessed May 13, 2025).

Kim, J., Kim, J., Jung, M., Kwon, T., and Kim, K. K. (2024). Individualized foveated rendering with eye-tracking head-mounted display. Virtual Reality, 28(1):25. DOI: https://doi.org/10.1007/s10055-023-00931-8.

Koskela, M. (2020). Foveated path tracing with fast reconstruction and efficient sample distribution.

Koskela, M., Lotvonen, A., Mäkitalo, M., Kivi, P., Viitanen, T., and Jääskeläinen, P. (2019). Foveated real-time path tracing in visual-polar space. In Boubekeur, T. and Sen, P., editors, Eurographics Symposium on Rendering - DL-only and Industry Track, pages 1–12. The Eurographics Association. DOI: https://doi.org/10.2312/sr.20191219.

Levoy, M. and Whitaker, R. (1990). Gaze-directed volume rendering. In Proceedings of the 1990 symposium on interactive 3D graphics, pages 217–223. DOI: https://doi.org/10.1145/91385.91449.

Li, T.-M., Aittala, M., Durand, F., and Lehtinen, J. (2018). Differentiable Monte Carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG), 37(6):1–11. DOI: https://doi.org/10.1145/3272127.3275109.

Liu, L., Gu, J., Lin, K. Z., Chua, T., and Theobalt, C. (2020). Neural sparse voxel fields. CoRR, abs/2007.11571.

Lu, C., Yin, F., Chen, X., Liu, W., Chen, T., Yu, G., and Fan, J. (2023). A large-scale outdoor multi-modal dataset and benchmark for novel view synthesis and implicit scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7557–7567. DOI: https://doi.org/10.48550/arXiv.2301.06782.

Mantiuk, R. K., Denes, G., Chapiro, A., Kaplanyan, A., Rufo, G., Bachy, R., Lian, T., and Patney, A. (2021). Fovvideovdp: A visible difference predictor for wide field-of-view video. ACM Transactions on Graphics (TOG), 40(4):1–19. DOI: https://doi.org/10.1145/3450626.3459831.

Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. (2021). NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106. DOI: https://doi.org/10.1145/3503250.

Mohanto, B., Islam, A. T., Gobbetti, E., and Staadt, O. (2022). An integrative view of foveated rendering. Computers & Graphics, 102:474–501. DOI: https://doi.org/10.1016/j.cag.2021.10.010.

Müller, T., Evans, A., Schied, C., and Keller, A. (2022). Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41(4):1–15. DOI: https://doi.org/10.1145/3528223.3530127.

Nilsson, J. and Akenine-Möller, T. (2020). Understanding SSIM. arXiv preprint arXiv:2006.13846. DOI: https://doi.org/10.48550/arXiv.2006.13846.

Ogboso, Y. U. and Bedell, H. E. (1987). Magnitude of lateral chromatic aberration across the retina of the human eye. JOSA A, 4(8):1666–1672.

Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C., Benty, N., Luebke, D., and Lefohn, A. (2016). Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG), 35(6):1–12. DOI: https://doi.org/10.1145/2980179.2980246.

Porcino, T., Trevisan, D., and Clua, E. (2020). Minimizing cybersickness in head-mounted display systems: causes and strategies review. In 2020 22nd Symposium on Virtual and Augmented Reality (SVR), pages 154–163. IEEE. DOI: https://doi.org/10.1109/SVR51698.2020.00035.

Qian, S., Kirschstein, T., Schoneveld, L., Davoli, D., Giebenhain, S., and Nießner, M. (2023). Gaussianavatars: Photorealistic head avatars with rigged 3D Gaussians. arXiv preprint arXiv:2312.02069. DOI: https://doi.org/10.48550/arXiv.2312.02069.

Reiser, C., Peng, S., Liao, Y., and Geiger, A. (2021). KiloNeRF: Speeding up neural radiance fields with thousands of tiny MLPs. arXiv e-prints, pages 1–11. DOI: https://doi.org/10.48550/arXiv.2103.13744.

Schütz, M., Krösl, K., and Wimmer, M. (2019). Real-time continuous level of detail rendering of point clouds. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 103–110. DOI: https://doi.org/10.1109/VR.2019.8798284.

Series, B. (2012). Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT, 500:1–46.

Setiadi, D. R. I. M. (2021). PSNR vs SSIM: Imperceptibility quality assessment for image steganography. Multimedia Tools and Applications, 80(6):8423–8444. DOI: https://doi.org/10.1007/s11042-020-10035-z.

Sorokin, A. and Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8. IEEE. DOI: https://doi.org/10.1109/CVPRW.2008.4562953.

Swafford, N. T., Iglesias-Guitian, J. A., Koniaris, C., Moon, B., Cosker, D., and Mitchell, K. (2016). User, metric, and computational evaluation of foveated rendering methods. In Proceedings of the ACM Symposium on Applied Perception, pages 7–14, Anaheim, California. ACM. DOI: https://doi.org/10.1145/2931002.2931011.

Turki, H., Ramanan, D., and Satyanarayanan, M. (2022). Mega-NeRF: Scalable construction of large-scale NeRFs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12922–12931. DOI: https://doi.org/10.1109/CVPR52688.2022.01258.

Tursun, O. T., Arabadzhiyska-Koleva, E., Wernikowski, M., Mantiuk, R., Seidel, H.-P., Myszkowski, K., and Didyk, P. (2019). Luminance-contrast-aware foveated rendering. ACM Transactions on Graphics (TOG), 38(4):1–14. DOI: https://doi.org/10.1145/3306346.332298.

Ubrani, J., Llamas, R., and Reith, R. (2024). AR VR headsets market insights. [link] (accessed Jan. 20, 2025).

Ujjainkar, N., Shahan, E., Chen, K., Duinkharjav, B., Sun, Q., and Zhu, Y. (2024). Exploiting human color discrimination for memory- and energy-efficient image encoding in virtual reality. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 1, pages 166–180, La Jolla, CA, USA. ACM. DOI: https://doi.org/10.1145/3617232.3624860.

Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J. T., and Srinivasan, P. P. (2022). Ref-NeRF: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE. DOI: https://doi.org/10.1109/CVPR52688.2022.00541.

Wandell, B. A. (1995). Foundations of Vision. Sinauer Associates.

Wang, L., Shi, X., and Liu, Y. (2023). Foveated rendering: A state-of-the-art survey. Computational Visual Media, 9(2):195–228. DOI: https://doi.org/10.1007/s41095-022-0306-4.

Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612. DOI: https://doi.org/10.1109/TIP.2003.819861.

Weier, M., Roth, T., Kruijff, E., Hinkenjann, A., Pérard-Gayot, A., Slusallek, P., and Li, Y. (2016). Foveated real-time ray tracing for head-mounted displays. Pages 289–298. The Eurographics Association and John Wiley & Sons Ltd. DOI: https://doi.org/10.1111/cgf.13026.

Whitted, T. (1979). An improved illumination model for shaded display. In Proceedings of the 6th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH ’79, page 14, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/800249.807419.

Wobbrock, J. O., Findlater, L., Gergle, D., and Higgins, J. J. (2011). The aligned rank transform for nonparametric factorial analyses using only ANOVA procedures. Page 143–146. Association for Computing Machinery. DOI: https://doi.org/10.1145/1978942.1978963.

Wu, Q., Martinez Esturo, J., Mirzaei, A., Moenne-Loccoz, N., and Gojcic, Z. (2024). 3DGUT: Enabling distorted cameras and secondary rays in Gaussian splatting. arXiv e-prints, arXiv:2412.12507. DOI: https://doi.org/10.48550/arXiv.2412.12507.

Yan, X., Xu, J., Huo, Y., and Bao, H. (2024). Neural rendering and its hardware acceleration: A review. arXiv preprint arXiv:2402.00028. DOI: https://doi.org/10.48550/arXiv.2402.00028.

Yang, W., Shi, J., Bai, S., Qian, Q., Ou, Z., Xu, D., Shu, X., and Sun, Y. (2024). Clear-plenoxels: Floaters free radiance fields without neural networks. Knowledge-Based Systems, 299:112096. DOI: https://doi.org/10.1016/j.knosys.2024.112096.

Yee, H., Pattanaik, S., and Greenberg, D. P. (2001). Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments. ACM Transactions on Graphics (TOG), 20(1):39–65. DOI: https://doi.org/10.1145/383745.383748.

Yu, A., Li, R., Tancik, M., Li, H., Ng, R., and Kanazawa, A. (2021). Plenoctrees for real-time rendering of neural radiance fields. arXiv preprint arXiv:2103.14024. DOI: https://doi.org/10.48550/arXiv.2103.14024.

Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. (2018). The unreasonable effectiveness of deep features as a perceptual metric. arXiv preprint arXiv:1801.03924. DOI: https://doi.org/10.48550/arXiv.1801.03924.