User Experience Evaluation in Non-Immersive 3D Digital Environments Using Facial Emotion Recognition

Érico de Souza Veriscimo; João Luiz Bernardes Júnior; Luciano Antonio Digiampietri

doi:10.5753/jis.2026.7625

Authors

Érico de Souza Veriscimo Sao Paulo Federal Institute https://orcid.org/0000-0003-0401-295X
João Luiz Bernardes Júnior University of Sao Paulo https://orcid.org/0000-0002-8248-113X
Luciano Antonio Digiampietri University of Sao Paulo https://orcid.org/0000-0003-4890-1548

DOI:

https://doi.org/10.5753/jis.2026.7625

Keywords:

Emotion Recognition, User Experience, 3D Interaction, Facial Expression Analysis

Abstract

User experience (UX) is fundamental for the acceptance and use of information systems. Although there are well-known and widely used UX evaluation techniques for traditional interfaces, a literature review revealed several gaps regarding UX evaluation in non-immersive 3D interaction. One significant gap is the predominant use of pragmatic criteria in assessments, while another is the lack of an approach that evaluates hedonic aspects using facial emotion recognition. This work proposes an approach for automatically evaluating user experience in non-immersive three-dimensional environments, focusing on its hedonic aspects based on facial emotion recognition. An experimental protocol was developed and approved by the Ethics Committee. The experiment was conducted with 52 participants. Throughout the testing period (before, during, and after the interaction), participants' faces were recorded using a low-cost camera. The experiment involved participants playing a game and answering questionnaires, including categorization and mood profile instruments, as well as the UEQ-S and the PLEX Framework. The Face-api.js library was used for facial emotion recognition. The hypothesis that automatic facial emotion recognition can support user experience evaluation was confirmed. This method enabled the estimation of UEQ-S and PLEX questionnaire responses with an average error of approximately ±1 point using only emotion extraction through an artificial intelligence model. Given that UX evaluation is crucial for the acceptance of new software or functionality, this work contributes to improving system quality and acceptance.

Downloads

Download data is not yet available.

References

Amusement Vision (2001). Super monkey ball. Video game, Nintendo GameCube.

Aranha, R. V., Casaes, A. B., and Nunes, F. L. S. (2020). Influence of environmental conditions in the performance of open-source software for facial expression recognition. In Proceedings of the Brazilian Symposium on Human Factors in Computing Systems, pages 1–10. DOI: https://doi.org/10.1145/3424953.3426630.

Arrasvuori, J. et al. (2011). Applying the plex framework in designing for playfulness. In Proceedings of the Conference on Designing Pleasurable Products and Interfaces, pages 1–8. ACM. DOI: https://doi.org/10.1145/2347504.2347531.

Baltrusaitis, T., Zadeh, A., Lim, Y. C., and Morency, L.-P. (2018). Openface 2.0: Facial behavior analysis toolkit. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pages 59–66. DOI: https://doi.org/10.1109/FG.2018.00019.

Bevan, N. (2009). What is the difference between the purpose of usability and user experience evaluation methods. In Proceedings of the Workshop on UX Evaluation Methods, pages 1–4.

Den Uyl, M. J. and Van Kuilenburg, H. (2005). The facereader: Online facial expression recognition. In Proceedings of Measuring Behavior, pages 589–590.

Deshmukh, R. S. and Jagtap, V. (2017). A survey: Software API and database for emotion recognition. In Proceedings of the International Conference on Intelligent Computing and Control Systems (ICICCS), pages 284–289. DOI: https://doi.org/10.1109/ICCONS.2017.8250727.

International Organization for Standardization (2010). ISO 9241-210:2010 ergonomics of human-system interaction – human-centred design for interactive systems. [link]. Accessed on 26 May 2026.

Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In HCI and Usability for Education and Work, pages 63–76. Springer. DOI: https://doi.org/10.1007/978-3-540-89350-9_6.

Lucero, A., Holopainen, J., Ollila, E., Suomela, R., and Karapanos, E. (2013). The playful experiences (PLEX) framework as a guide for expert evaluation. In Proceedings of the International Conference on Designing Pleasurable Products and Interfaces, pages 221–230. ACM. DOI: https://doi.org/10.1145/2513506.2513530.

Magdin, M. and Prikler, F. (2018). Real-time facial expression recognition using webcam and sdk affectiva. International Journal of Interactive Multimedia and Artificial Intelligence. DOI: https://doi.org/10.9781/ijimai.2017.11.002.

Magix (2023). Sony vegas. [link]. Version 21.0.0. Accessed on 26 May 2026.

Martinelli, S., Lopes, L., and Zaina, L. (2022). Ux research in the software industry: An investigation of long-term ux practices. In Proceedings of the Brazilian Symposium on Human Factors in Computing Systems. SBC. DOI: https://doi.org/10.1145/3554364.3559126.

Mennig, P., Scherr, S. A., and Elberzhager, F. (2019). Supporting rapid product changes through emotional tracking. In Proceedings of the IEEE/ACM 4th International Workshop on Emotion Awareness in Software Engineering (SEmotion), pages 8–12. DOI: https://doi.org/10.1109/SEmotion.2019.00009.

Milgram, P., Takemura, H., Utsumi, A., and Kishino, F. (1995). Augmented reality: A class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies, 2351:282–292. DOI: https://doi.org/10.1117/12.197321.

Mühler, V. (2020). face-api.js – JavaScript API for face recognition in the browser with TensorFlow.js. [link]. Accessed on 26 May 2026.

Rajeshkumar, S., Omar, R., and Mahmud, M. (2013). Taxonomies of user experience (ux) evaluation methods. In Proceedings of the International Conference on Research and Innovation in Information Systems (ICRIIS), pages 533–538. DOI: https://doi.org/10.1109/ICRIIS.2013.6716765.

Rehman, A., Mujahid, M., Elyassih, A., Alghofaily, B., and Saeed, A. (2025). Comprehensive review and analysis on facial emotion recognition: Performance insights into deep and traditional learning with current updates and challenges. Computers, Materials, & Continua, 82(1):41–72. DOI: https://doi.org/10.32604/cmc.2024.058036.

Santos, B. and Digiampietri, L. (2024). User experience evaluation using machine learning and facial expressions: A systematic review. In Anais do XXI Encontro Nacional de Inteligência Artificial e Computacional, pages 930–941, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/eniac.2024.245150.

Schrepp, M., Hinderks, A., and Thomaschewski, J. (2017). Design and evaluation of a short version of the user experience questionnaire (ueq-s). International Journal of Interactive Multimedia and Artificial Intelligence, 4(6):103–108. DOI: https://doi.org/10.9781/ijimai.2017.09.001.

Veriscimo, E. S., Bernardes Junior, J. L., and Digiampietri, L. A. (2020). Evaluating user experience in 3D interaction: A systematic review. In Proceedings of the Brazilian Symposium on Information Systems, pages 1–8. DOI: https://doi.org/10.1145/3411564.3411640.

Veriscimo, E. S., Bernardes Junior, J. L., and Digiampietri, L. A. (2021). Facial emotion recognition in ux evaluation: A systematic review. In International Conference on Universal Access in Human-Computer Interaction. Springer. DOI: https://doi.org/10.1007/978-3-030-78462-1_40.

Viana, M. F., Almeida, P., and Santos, R. C. (2001). Adaptação portuguesa da versão reduzida do perfil de estados de humor – POMS. Análise Psicológica, 19(1):77–92. DOI: https://doi.org/10.14417/ap.345.