A Semantic Multimodal Architecture for Accessible Navigation in Immersive Virtual Environments

Luiza Cintra; Elisa Ayumi; Gustavo Webster; Andressa Bastos; Valdemar V. Graciano-Neto; Sofia Costa Paiva; Rafael Sousa; Arlindo Galvão

doi:10.5753/jis.2026.7402

Authors

Luiza Cintra Federal University of Goiás https://orcid.org/0009-0007-6287-5944
Elisa Ayumi Federal University of Goiás https://orcid.org/0009-0000-6638-1959
Gustavo Webster Federal University of Goiás https://orcid.org/0009-0000-5369-3005
Andressa Bastos Federal University of Goiás https://orcid.org/0009-0005-3707-8949
Valdemar V. Graciano-Neto Federal University of Goiás https://orcid.org/0000-0003-2190-5477
Sofia Costa Paiva Federal University of Goiás https://orcid.org/0000-0001-8309-1144
Rafael Sousa Federal University of Goiás https://orcid.org/0000-0001-5998-046X
Arlindo Galvão Federal University of Goiás https://orcid.org/0000-0003-2151-8039

DOI:

https://doi.org/10.5753/jis.2026.7402

Keywords:

Virtual Reality, Accessibility, Visual Impairment, Haptic Feedback, Vision-Language Models

Abstract

Accessible interaction in immersive virtual environments remains largely constrained by visually driven design paradigms, limiting meaningful engagement for users with visual impairments. Existing approaches to VR accessibility often rely on application-specific adaptations, low-level sensory cues, or vision-centered augmentations, resulting in fragmented and non-scalable solutions. This paper introduces a conceptual architecture for a semantic multimodal interface that mediates access to immersive virtual environments through non-visual representations. The architecture operates as a reusable interaction layer that translates visual information into coherent auditory and haptic feedback grounded in semantic understanding, supporting users across the full spectrum of visual impairment, from total blindness to low vision. Central to the architecture is the use of Vision Large Language Models as a semantic mediation layer, enabling dynamic interpretation of visual scenes and context-aware multimodal feedback independent of application logic. The interface is designed as a modular architecture that can be integrated into diverse VR applications, decoupling accessibility mechanisms from scenario-specific implementations. The architecture is instantiated in heterogeneous immersive environments to illustrate its generality and architectural feasibility. By treating accessibility as a foundational interface concern rather than an add-on feature, this work contributes a design-oriented perspective for scalable, inclusive interaction in immersive virtual environments.

Downloads

Download data is not yet available.

References

Anjos, F. E. V. d., Martins, A. d. O., Rodrigues, G. S., Sellitto, M. A., and Silva, D. O. d. (2024). Boosting engineering education with virtual reality: An experiment to enhance student knowledge retention. Applied System Innovation, 7(3):50. DOI: https://doi.org/10.3390/asi7030050.

Bendaly Hlaoui, Y., Zouhaier, L., and Ben Ayed, L. (2019). Model driven approach for adapting user interfaces to the context of accessibility: case of visually impaired users. Journal on Multimodal User Interfaces, 13(4):293–320. DOI: https://doi.org/10.1007/s12193-018-0277-z.

Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77–101. DOI: https://doi.org/10.1191/1478088706qp063oa.

Brooke, J. et al. (1996). Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4–7.

Brown, D. J. and Proulx, M. J. (2016). Audio–vision substitution for blind individuals: Addressing human information processing capacity limitations. IEEE Journal of Selected Topics in Signal Processing, 10(5):924–931. DOI: https://doi.org/10.1109/JSTSP.2016.2543678.

Creed, C., Al-Kalbani, M., Theil, A., Sarcar, S., and Williams, I. (2024). Inclusive ar/vr: accessibility barriers for immersive technologies. Universal Access in the Information Society, 23(1):59–73. DOI: https://doi.org/10.1007/s10209-023-00969-0.

Delaney, B. and Biocca, F. (1995). Immersive virtual reality technology. Communication in the age of virtual reality, pages 57–124.

Dudley, J., Yin, L., Garaj, V., and Kristensson, P. O. (2023). Inclusive immersion: a review of efforts to improve accessibility in virtual reality, augmented reality and the metaverse. Virtual Reality, 27(4):2989–3020. DOI: https://doi.org/10.1007/s10055-023-00850-8.

Fernandes, H., Costa, P., Filipe, V., Paredes, H., and Barroso, J. (2019). A review of assistive spatial orientation and navigation technologies for the visually impaired. Universal Access in the Information Society, 18(1):155–168. DOI: https://doi.org/10.1007/s10209-017-0570-8.

Fernandes, P. C., Rocha, G. H. M., Adachi, B. H., Silvas, F. P., Coelho, B. N., and Delabrida, S. (2024). Exploring human interaction in virtual reality: An experience report on users with and without visual impairment. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, pages 1–11. DOI: https://doi.org/10.1145/3702038.3702047.

Freire, A. P., Tavares, D. C., and Aranha, R. V. (2023). Pesquisa em interação humano-computador com pessoas com deficiência. In Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC), pages 13–14. SBC. DOI: https://doi.org/10.5753/ihc_estendido.2023.233768.

Fryer, L. (2013). Putting it into words: The impact of visual impairment on perception, experience and presence. PhD thesis, Goldsmiths, University of London.

Giudice, N. A. (2018). Navigating without vision: Principles of blind spatial cognition. In Handbook of behavioral and cognitive geography, pages 260–288. Edward Elgar Publishing. DOI: https://doi.org/10.4337/9781784717544.00024.

Heilemann, F., Zimmermann, G., and Münster, P. (2021). Accessibility guidelines for vr games-a comparison and synthesis of a comprehensive set. Frontiers in Virtual Reality, 2:697504. DOI: https://doi.org/10.3389/frvir.2021.697504.

Kane, S. K., Morris, M. R., Perkins, A. Z., Wigdor, D., Ladner, R. E., and Wobbrock, J. O. (2011). Access overlays: improving non-visual access to large touch screens for blind users. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 273–282. DOI: https://doi.org/10.1145/2047196.2047232.

Kuang, J., Shen, Y., Xie, J., Luo, H., Xu, Z., Li, R., Li, Y., Cheng, X., Lin, X., and Han, Y. (2025). Natural language understanding and inference with mllm in visual question answering: A survey. ACM Computing Surveys, 57(8):1–36. DOI: https://doi.org/10.1145/3711680.

Kudo, T. N., de Freitas Bulcão-Neto, R., Neto, V. V. G., and Vincenzi, A. M. R. (2022). Aligning requirements and testing through metamodeling and patterns: design and evaluation. Requirements Engineering, 28(1):97. DOI: https://doi.org/10.1007/s00766-022-00377-5.

Lahav, O. (2022). Virtual reality systems as an orientation aid for people who are blind to acquire new spatial information. Sensors, 22(4):1307. DOI: https://doi.org/10.3390/s22041307.

Leat, S. J., Legge, G. E., and Bullimore, M. A. (1999). What is low vision? a re-evaluation of definitions. Optometry and Vision Science, 76(4):198–211.

Li, Y., Lai, Z., Bao, W., Tan, Z., Dao, A., Sui, K., Shen, J., Liu, D., Liu, H., and Kong, Y. (2025a). Visual large language models for generalized and specialized applications. arXiv preprint arXiv:2501.02765. DOI: https://doi.org/10.48550/arXiv.2501.02765.

Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., and Shi, G. (2025b). A survey of state of the art large vision language models: Benchmark evaluations and challenges. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1587–1606. DOI: https://doi.org/10.1109/CVPRW67362.2025.00147.

Liang, C. X., Tian, P., Yin, C. H., Yua, Y., Wei, A.-H., Li, M., Song, X., Wang, T., Bi, Z., Liu, M., et al. (2026). A comprehensive survey and guide to multimodal large language models in vision–language tasks. Computation, 14(6):125. DOI: https://doi.org/10.3390/computation14060125.

Liu, M., Chen, C., and Gurari, D. (2023). An evaluation of gpt-4v and gemini in online vqa. arXiv preprint arXiv:2312.10637. DOI: https://doi.org/10.48550/arXiv.2312.10637.

Liu, T., Fazli, P., and Jeong, H. (2024). Artificial intelligence in virtual reality for blind and low vision individuals: Literature review. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume 68, pages 1333–1338. SAGE Publications Sage CA: Los Angeles, CA. DOI: https://doi.org/10.1177/10711813241266832.

Macaluso, E. and Driver, J. (2001). Spatial attention and crossmodal interactions between vision and touch. Neuropsychologia, 39(12):1304–1316. DOI: https://doi.org/10.1016/S0028-3932(01)00119-1.

Mack, K., McDonnell, E., Jain, D., Lu Wang, L., E. Froehlich, J., and Findlater, L. (2021). What do we mean by "accessibility research"? a literature survey of accessibility papers in chi and assets from 1994 to 2019. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–18. DOI: https://doi.org/10.1145/3411764.3445412.

Maidenbaum, S., Buchs, G., Abboud, S., Lavi-Rotbain, O., and Amedi, A. (2016). Perception of graphical virtual environments by blind users via sensory substitution. PloS one, 11(2):e0147501. DOI: https://doi.org/10.1371/journal.pone.0147501.

Masnadi, S., Williamson, B., González, A. N. V., and LaViola, J. J. (2020). Vriassist: An eye-tracked virtual reality low vision assistance tool. In 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), pages 808–809. IEEE. DOI: https://doi.org/10.1109/VRW50115.2020.00255.

Mott, M., Cutrell, E., Franco, M. G., Holz, C., Ofek, E., Stoakley, R., and Morris, M. R. (2019). Accessible by design: An opportunity for virtual reality. In 2019 IEEE international symposium on mixed and augmented reality adjunct (ISMAR-adjunct), pages 451–454. IEEE. DOI: https://doi.org/10.1109/ISMAR-Adjunct.2019.00122.

Oliveira, A. D., dos Santos, P. S., Júnior, W. E. M., Aljedaani, W., Eler, D. M., and Eler, M. M. (2023). Análise de avaliaçoes de acessibilidade digital associadas a deficiências visuais e condiçoes oculares. In Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais (IHC), pages 241–245. SBC. DOI: https://doi.org/10.5753/ihc_estendido.2023.233060.

Ortiz-Escobar, L. M., Chavarria, M. A., Schönenberger, K., Hurst, S., Stein, M. A., Mugeere, A., and Rivas Velarde, M. (2023). Assessing the implementation of user-centred design standards on assistive technology for persons with visual impairments: a systematic review. Frontiers in Rehabilitation Sciences, 4:1238158. DOI: https://doi.org/10.3389/fresc.2023.1238158.

Persson, H., Åhman, H., Yngling, A. A., and Gulliksen, J. (2015). Universal design, inclusive design, accessible design, design for all: different concepts—one goal? on the concept of accessibility—historical, methodological and philosophical aspects. Universal access in the information society, 14(4):505–526. DOI: https://doi.org/10.1007/s10209-014-0358-z.

Picot, S. J. F., Samonte, J., Tierney, J. A., Connor, J., and Powel, L. L. (2001). Effective sampling of rare population elements: Black female caregivers and noncaregivers. Research on Aging, 23(6):694–712. DOI: https://doi.org/10.1177/0164027501236004.

Pinheiro, M. C. and da Silva, F. M. (2010). Comunicação visual e design inclusivo, cor, legibilidade e visão envelhecida. Design Ergonômico – Estudos e Aplicações, 17033:62.

Pur, D. R., Lee-Wing, N., and Bona, M. D. (2023). The use of augmented reality and virtual reality for visual field expansion and visual acuity improvement in low vision rehabilitation: a systematic review. Graefe's Archive for Clinical and Experimental Ophthalmology, 261(6):1743–1755. DOI: https://doi.org/10.1007/s00417-022-05972-4.

Qiu, S., Hu, J., Han, T., and Rauterberg, M. (2024). Can blindfolded users replace blind ones in product testing? an empirical study. Behaviour & Information Technology, 43(8):1664–1682. DOI: https://doi.org/10.1080/0144929X.2023.2226768.

Rahimi, F., Sadeghi-Niaraki, A., and Choi, S.-M. (2025). Generative ai meets virtual reality: A comprehensive survey on applications, challenges, and future direction. IEEE Access. DOI: https://doi.org/10.1109/ACCESS.2025.3574779.

Raybourn, E. M., Stubblefield, W. A., Trumbo, M., Jones, A., Whetzel, J., and Fabian, N. (2019). Information design for xr immersive environments: Challenges and opportunities. In International Conference on Human-Computer Interaction, pages 153–164. Springer. DOI: https://doi.org/10.1007/978-3-030-21607-8_12.

Real, S. and Araujo, A. (2019). Navigation systems for the blind and visually impaired: Past work, challenges, and open problems. Sensors, 19(15):3404. DOI: https://doi.org/10.3390/s19153404.

Rodrigues, C. S. C., Nazareth, V., Azevedo, R. O., Barbosa, P., and Werner, C. (2025). Unseen: Advancing digital accessibility with binaural audio technology in an immersive gaming prototype. Journal on Interactive Systems, 16(1):98–108. DOI: https://doi.org/10.5753/jis.2025.4439.

Rubio-Tamayo, J. L., Gertrudix Barrio, M., and García García, F. (2017). Immersive environments and virtual reality: Systematic review and advances in communication, interaction and simulation. Multimodal technologies and interaction, 1(4):21. DOI: https://doi.org/10.3390/mti1040021.

Runeson, P. and Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering, 14(2):131–164. DOI: https://doi.org/10.1007/s10664-008-9102-8.

Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., and Wang, J. (2022). Towards natural language interfaces for data visualization: A survey. IEEE transactions on visualization and computer graphics, 29(6):3121–3144. DOI: https://doi.org/10.1109/TVCG.2022.3148007.

Silva, G. M. S., de C. Andrade, R. M., and de Gois R. Darin, T. (2019). Design and evaluation of mobile applications for people with visual impairments: A compilation of usable accessibility guidelines. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, pages 1–10. DOI: https://doi.org/10.1145/3357155.3358450.

Silveira, C. S., dos Santos, A. N., dos Santos Oliveira, C., and de Souza Arnaut, F. F. (2024). Development of accessible virtual reality technology for user inclusion. JOURNAL OF BIOENGINEERING, TECHNOLOGIES AND HEALTH, 7(Suppl2):15–22. DOI: https://doi.org/10.34178/jbth.v7iSuppl2.447.

Story, M. F. (2001). Principles of universal design. Universal design handbook, 2.

Wang, Z. and Ben-Arie, J. (1996). Conveying visual information with spatial auditory patterns. IEEE transactions on speech and audio processing, 4(6):446–455. DOI: https://doi.org/10.1109/89.544529.

Wen, L., Yang, X., Fu, D., Wang, X., Cai, P., Li, X., Ma, T., Li, Y., Xu, L., Shang, D., et al. (2024). On the road with gpt-4v(ision): Explorations of utilizing visual-language model as autonomous driving agent. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.

Whicher, D., Philbin, S., and Aronson, N. (2018). An overview of the impact of rare disease characteristics on research methodology. Orphanet journal of rare diseases, 13(1):14. DOI: https://doi.org/10.1186/s13023-017-0755-5.

Yuan, B., Folmer, E., and Harris Jr, F. C. (2011). Game accessibility: a survey. Universal Access in the information Society, 10(1):81–100. DOI: https://doi.org/10.1007/s10209-010-0189-5.

Zhao, Y., Bennett, C. L., Benko, H., Cutrell, E., Holz, C., Morris, M. R., and Sinclair, M. (2018). Enabling people with visual impairments to navigate virtual reality with a haptic and auditory cane simulation. In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–14. DOI: https://doi.org/10.1145/3173574.317369.

Zhao, Y., Cutrell, E., Holz, C., Morris, M. R., Ofek, E., and Wilson, A. D. (2019). Seeingvr: A set of tools to make virtual reality more accessible to people with low vision. In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–14. DOI: https://doi.org/10.1145/3290605.3300341.