Usability and UX Evaluation in a Mixed Reality Puzzle Game Using Questionnaires

Thiago Prado de Campos; Saul Delabrida; Natasha Malveira Costa Valentim

doi:10.5753/jis.2025.6089

Authors

Thiago Prado de Campos Federal Technological University of Paraná https://orcid.org/0000-0003-1038-4004
Saul Delabrida Federal University of Ouro Preto https://orcid.org/0000-0002-8961-5313
Natasha Malveira Costa Valentim Federal University of Paraná https://orcid.org/0000-0002-6027-3452

DOI:

https://doi.org/10.5753/jis.2025.6089

Keywords:

Usability, User Experience, Puzzle Game, Mixed Reality, Questionnaire Comparison, Evaluation Methods

Abstract

Background: Mixed Reality (MR) technologies present unique usability and user experience (UX) challenges, particularly in gesture-based interaction and spatial perception. Puzzle games in MR environments rely heavily on intuitive control and immersive feedback to maintain user engagement. Purpose: This study aims to evaluate the usability and UX of the Cubism puzzle game in an MR context, comparing different standardized instruments to capture user perceptions and identify specific interaction issues. Methods: The evaluation was conducted during a usability and UX workshop, where participants used Meta Quest 2 and Meta Quest 3 headsets. Participants were divided into two groups: Group A completed the UUXE-ToH v4 questionnaire, while Group B answered the USE, Slater-Usoh-Steed (SUS), and UEQ questionnaires. In addition to quantitative assessments, participants reported issues, which were classified and analyzed thematically. Results: Cubism was positively evaluated in terms of learnability, memorability, and satisfaction. However, notable challenges were identified in object manipulation, gesture accuracy, and error recovery. Immersion and presence scores were comparatively lower, particularly in SUS, suggesting limited perceptual integration of virtual objects into the real environment. Interaction precision and environmental coherence were also affected by hardware limitations, especially on devices with monochrome passthrough. Conclusion: The findings highlight specific usability and UX limitations in MR puzzle games and illustrate how different instruments (applied to distinct groupings of participants) captured complementary aspects of the experience. Although no direct statistical comparison was performed, indirect analysis suggests varying diagnostic contributions across instruments. These findings suggest that multidimensional evaluation frameworks (either through a single comprehensive instrument or a combination of focused tools) can provide richer insights for guiding design improvements and identifying usability and UX issues in MR applications.

Downloads

Download data is not yet available.

References

Bai, Z. and Blackwell, A. F. (2012). Analytic review of usability evaluation in ISMAR. Interacting with Computers, 24(6):450–460. DOI: https://doi.org/10.1016/j.intcom.2012.07.004.

Borges, J. B., Juy, C. L., Matos, I. S. d. A., Silveira, P. V. A., and Darin, T. d. G. R. (2020). Player Experience Evaluation: a Brief Panorama of Instruments and Research Opportunities. Journal on Interactive Systems, 11(1):74–91. DOI: https://doi.org/10.5753/jis.2020.765.

Bouwel, T. V. (2024). Cubism Presskit. Retrieved from [link] on July 28, 2025.

Campos, T., Castello, M., Damasceno, E., and Valentim, N. (2025a). An Updated Systematic Mapping Study on Usability and User Experience Evaluation of Touchable Holographic Solutions. Journal on Interactive Systems, 16(1):172–198. DOI: https://doi.org/10.5753/jis.2025.4694.

Campos, T., Damasceno, E., and Valentim, N. (2024a). Usability and User Experience Questionnaire Evaluation and Evolution for Touchable Holography. In Proceedings of the 26th International Conference on Enterprise Information Systems, pages 449–460, Angers, France. SCITEPRESS - Science and Technology Publications. DOI: https://doi.org/10.5220/0012564100003690.

Campos, T., Delabrida, S., Damasceno, E., and Valentim, N. (2025b). Evaluating Performance and Acceptance of the UUXE-ToH Questionnaire for Touchable Holographic Solutions. pages 641–648.

Campos, T. P. d., Damasceno, E. F., and Valentim, N. M. C. (2023). Usability and User Experience Evaluation of Touchable Holographic solutions: A Systematic Mapping Study. In IHC ’23: Proceedings of the 22st Brazilian Symposium on Human Factors in Computing Systems, IHC ’23, pages 1–13, Maceio, Brazil. ACM. DOI: https://doi.org/10.1145/3638067.3638071.

Campos, T. P. D., Hounsell, M. D. S., Damasceno, E. F., and Valentim, N. M. C. (2024b). Evaluating Usability, User Experience, and Playability of a Puzzle Game in Mixed Reality. In Symposium on Virtual and Augmented Reality, pages 61–70, Manaus Brazil. ACM. DOI: https://doi.org/10.1145/3691573.3691584.

Desurvire, H. and Wiberg, C. (2009). Game Usability Heuristics (PLAY) for Evaluating and Designing Better Games: The Next Iteration. In Ozok, A. A. and Zaphiris, P., editors, Online Communities and Social Computing, pages 557–566, Berlin, Heidelberg. Springer. DOI: https://doi.org/10.1007/978-3-642-02774-1_60.

Dünser, A. and Billinghurst, M. (2011). Evaluating Augmented Reality Systems. In Furht, B., editor, Handbook of Augmented Reality, pages 289–307. Springer New York, New York, NY. DOI: https://doi.org/10.1007/978-1-4614-0064-6_13.

Dünser, A., Grasset, R., and Billinghurst, M. (2008). A survey of evaluation techniques used in augmented reality studies. In ACM SIGGRAPH ASIA 2008 courses on - SIGGRAPH Asia ’08, pages 1–27, Singapore. ACM Press. DOI: https://doi.org/10.1145/1508044.1508049.

Fast-Berglund, A., Gong, L., and Li, D. (2018). Testing and validating Extended Reality (xR) technologies in manufacturing. Procedia Manufacturing, 25:31–38. DOI: https://doi.org/10.1016/j.promfg.2018.06.054.

Frata Furlan Peres, F., Nunes, F., Teixeira, J. M., Maurício, C. R. M., Conceição, K. P., and Yoshida, L. (2024). Methods for Evaluating Immersive 3D Virtual Environments: a Systematic Literature Review. In Proceedings of the 26th Symposium on Virtual and Augmented Reality, SVR ’24, pages 140–151, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3691573.3691595.

Hartson, R. and Pyla, P. (2012). The UX Book: Process and Guidelines for Ensuring a Quality User Experience. Morgan Kaufmann, 1st edition.

Ivory, M. Y. and Hearst, M. A. (2001). The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4):470–516. DOI: https://doi.org/10.1145/503112.503114.

Johnson, D., Deterding, S., Kuhn, K.-A., Staneva, A., Stoyanov, S., and Hides, L. (2016). Gamification for health and wellbeing: A systematic review of the literature. Internet Interventions, 6:89–106. DOI: https://doi.org/10.1016/j.invent.2016.10.002.

Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and Evaluation of a User Experience Questionnaire. In Holzinger, A., editor, HCI and Usability for Education and Work, Lecture Notes in Computer Science, pages 63–76, Graz, Austria. Springer. DOI: https://doi.org/10.1007/978-3-540-89350-9_6.

LaViola, J. J. (2000). A discussion of cybersickness in virtual environments. SIGCHI Bull., 32(1):47–56. DOI: https://doi.org/10.1145/333329.333344.

Lee, B. G., Tang, H., and Wen, X. (2023). Exploring the Fusion of Mixed Reality and Digital Game-Based Learning: The Case of Puzzle Box Games for Education. In 2023 IEEE International Conference on Teaching, Assessment and Learning for Engineering (TALE), pages 1–8. DOI: https://doi.org/10.1109/TALE56641.2023.10398389.

Lund, A. M. (2001). Measuring usability with the USE questionnaire. Usability interface, 8(2):3–6.

Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., and Weiskopf, D. (2020). Evaluating Mixed and Augmented Reality: A Systematic Literature Review (2009-2019). In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 438–451, Porto de Galinhas, Brazil. IEEE. DOI: https://doi.org/10.1109/ISMAR50242.2020.00069.

Nielsen, J. (1993). Usability engineering. Academic Press, Boston.

Pagano, D. and Bruegge, B. (2013). User involvement in software evolution practice: A case study. In 2013 35th International Conference on Software Engineering (ICSE), pages 953–962. ISSN: 1558-1225. DOI: https://doi.org/10.1109/ICSE.2013.6606645.

Prado De Campos, T., Damasceno, E. F., and Valentim, N. M. C. (2024). Evaluating Usability and UX in Touchable Holographic Solutions: A Validation Study of the UUXE-ToH Questionnaire. International Journal of Human–Computer Interaction, pages 1–21. DOI: https://doi.org/10.1080/10447318.2024.2400755.

Pranoto, H., Tho, C., Warnars, H. L. H. S., Abdurachman, E., Gaol, F. L., and Soewito, B. (2017). Usability testing method in augmented reality application. In 2017 International Conference on Information Management and Technology (ICIMTech), pages 181–186. DOI: https://doi.org/10.1109/ICIMTech.2017.8273534.

Rhiu, I., Kim, Y. M., Kim, W., and Yun, M. H. (2020). The evaluation of user experience of a human walking and a driving simulation in the virtual reality. International Journal of Industrial Ergonomics, 79:103002. DOI: https://doi.org/10.1016/j.ergon.2020.103002.

Ritterfeld, U., Cody, M., and Vorderer, P., editors (2009). Serious Games: Mechanisms and Effects. Routledge, New York. DOI: https://doi.org/10.4324/9780203891650.

Roto, V., Obrist, M., and Väänänen-vainio mattila, K. (2009). User Experience Evaluation Methods in Academic and Industrial Contexts. In Proceedings of the Workshop UXEM’09, volume II, page 4 p, Uppsala, Sweden. Springer.

Sharples, S., Cobb, S., Moody, A., and Wilson, J. R. (2008). Virtual reality induced symptoms and effects (VRISE): Comparison of head mounted display (HMD), desktop and projection display systems. Displays, 29(2):58–69. DOI: https://doi.org/10.1016/j.displa.2007.09.005.

Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1535):3549–3557. DOI: https://doi.org/10.1098/rstb.2009.0138.

Slater, M., Usoh, M., and Steed, A. (1994). Depth of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments, 3(2):130–144. DOI: https://doi.org/10.1162/pres.1994.3.2.130.

Swan, J. and Gabbard, J. L. (2005). Survey of User-Based Experimentation in Augmented Reality. In Proceedings 1st International Conference on Virtual Reality, pages 1–9, Las Vegas, Nevada, USA. Mira Digital Publishing.

Veriscimo, E. D. S., Bernardes Junior, J. L., and Digiampietri, L. A. (2020). Evaluating User Experience in 3D Interaction: a Systematic Review. In XVI Brazilian Symposium on Information Systems, pages 1–8, São Bernardo do Campo Brazil. ACM. DOI: https://doi.org/10.1145/3411564.3411640.

Yáñez-Gómez, R., Cascado-Caballero, D., and Sevillano, J.-L. (2017). Academic methods for usability evaluation of serious games: a systematic review. Multimedia Tools and Applications, 76(4):5755–5784. DOI: https://doi.org/10.1007/s11042-016-3845-9.

Zhang, T., Booth, R., Jean-Louis, R., Chan, R., Yeung, A., Gratzer, D., and Strudwick, G. (2020). A Primer on Usability Assessment Approaches for Health-Related Applications of Virtual Reality. JMIR Serious Games, 8(4):e18153. DOI: https://doi.org/10.2196/18153.