Visualizing Multimodal UX and Human Motion data in 3D scenes
DOI:
https://doi.org/10.5753/jbcs.2026.8218Keywords:
Human-Centered Computing, Immersive Visualization, Visual Analytics, Human-Computer Interaction, Interaction Paradigms, Virtual RealityAbstract
Understanding user experience (UX) in immersive and interactive media increasingly involves rich embodied interactions, where human motion, encompassing both human position and human body kinematics, evolves alongside multimodal UX data streams over space and time. While visualization techniques play a central role in supporting the analysis of such data, existing approaches typically scatter these individual dimensions across detached views, failing to examine how physiological and experiential signals relate directly to their embodied context. Immersive analytics opens perspectives by allowing dimensions to be displayed in context directly within the 3D environment in which the experience takes place. In this paper, we address how the combined visualization of human motion and multimodal UX data streams in 3D scenes contribute to immersive analytics. We structure our investigation around three core elements: data synchronization, body-linked immersive representation, and analytical perspectives. Specifically, we present an approach that synchronizes body kinematics within a 3D scene while establishing a direct data-to-body mapping that binds physiological and contextual UX signals directly onto moving 3D skeleton. To support discovery of complex behavioral patterns, we combine complementary analytical perspectives, including exocentric viewpoints for spatial overviews, egocentric viewpoints for situated inspection, and abstract temporal views for focused trend analysis. We demonstrate the analytical potential of our approach through a systematic heuristic walkthrough and task-oriented scenarios based on an existing human motion dataset. By bridging the spatial-physiological correlation gap, this work contributes an exploratory framework for investigating the dimensions of immersive systems and informs future research on visualization designs for examining embodied interaction in XR.
Downloads
References
Alallah, F. S., Faleel, S. A., Sakamoto, Y., Rey, B., and Irani, P. (2022). Ssca: Situated space-time cube analytics. In EuroVis (Short Papers), pages 25-29, Rome, Italy. EuroVis. DOI: 10.2312/evs.20221088.
Arpatzoglou, V., Kardara, A. A., Diehl, A., Flueckiger, B., Helmer, S., and Pajarola, R. (2021). Dancemoves: A visual analytics tool for dance movement analysis. ZORA, 1. DOI: 10.2312/evs.20211057.
Astsatryan, H. V., Grogoryan, H., Gyulgyulyan, E., Hakobyan, A., Kocharyan, A., Narsisian, W., Sahakyan, V., Shoukourian, Y., Abrahamyan, R., Petrosyan, Z., et al. (2018). Weather data visualization and analytical platform. Scalable Computing: Practice and Experience, 19(2):79-86. DOI: 10.12694/scpe.v19i2.1351.
Aupetit, S., Escaich, S., Gillet, C., Herber, P., Grison, E., and Morgagni, S. (2023). Virtual reality study of the effects of railway crossing safety technologies on pedestrian behaviour. Transportation Research Procedia, 72:1264-1271. TRA Lisbon 2022 Conference Proceedings Transport Research Arena (TRA Lisbon 2022),14th-17th November 2022, Lisboa, Portugal. DOI: 10.1016/j.trpro.2023.11.586.
Browen, I., Camarillo-Abad, H. M., Cibrian, F. L., and Qi, T. D. (2024). Creative insights into motion: enhancing human activity understanding with 3d data visualization and annotation. In Proceedings of the 16th Conference on Creativity & Cognition, pages 482-487, Chicago IL USA. CC 24. DOI: 10.1145/3635636.3664256.
Brusi, P. (2021). Making a game character move: Animation and motion capture for video games. South-Eastern Finland University of Applied Sciences. Available at: [link].
Büschel, W., Lehmann, A., and Dachselt, R. (2021). Miria: A mixed reality toolkit for the in-situ visualization and analysis of spatio-temporal interaction data. In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1-15. DOI: 10.1145/3411764.3445651.
Cabouat, A.-F. (2024). A cognitive load approach to designing and evaluating data visualizations. Master's thesis, Universite Paris Cite. Available at: [link].
Chen, J., Shaw, S.-L., Yu, H., Lu, F., Chai, Y., and Jia, Q. (2011). Exploratory data analysis of activity diary data: a space-time gis approach. Journal of Transport Geography, 19(3):394-404. DOI: 10.1016/j.jtrangeo.2010.11.002.
Chen, W., Xia, J., Wang, X., Wang, Y., Chen, J., and Chang, L. (2018). Relationlines: Visual reasoning of egocentric relations from heterogeneous urban data. ACM Transactions on Intelligent Systems and Technology (TIST), 10(1):1-21. DOI: 10.1145/3200766.
Choi, M. G., Yang, K., Igarashi, T., Mitani, J., and Lee, J. (2012). Retrieval and visualization of human motion data via stick figures. In Computer Graphics Forum, volume 31, pages 2057-2065, . Wiley Online Library, Computer Graphics Forum. DOI: 10.1111/j.1467-8659.2012.03198.x.
Climavision (2025). How to read a weather forecast: Key symbols and terms explained. [link].
Contributors, W. (2025). Map symbol. Wikipedia, The Free Encyclopedia. Available at: [link].
Corona, E., Pumarola, A., Alenya, G., and Moreno-Noguer, F. (2020). Context-aware human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6992-7001, . IEEE CVF. DOI: 10.1109/CVPR42600.2020.00702.
Cowley, B. U. and Torniainen, J. (2016). A short review and primer on electrodermal activity in human computer interaction applications. arXiv preprint arXiv:1608.06986, 1(1). DOI: 10.48550/arXiv.1608.06986.
Gao, S. (2015). Spatio-temporal analytics for exploring human mobility patterns and urban dynamics in the mobile age. Spatial Cognition & Computation, 15(2):86-114. DOI: 10.1080/13875868.2014.984300.
Ginns, P. (2006). Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects. Learning and instruction, 16(6):511-525. DOI: 10.1016/j.learninstruc.2006.10.001.
Gkikas, S., Kyprakis, I., and Tsiknakis, M. (2025). Multi-representation diagrams for pain recognition: Integrating various electrodermal activity signals into a single image. arXiv preprint arXiv:2507.21881, 1(1). DOI: 10.1145/3747327.3764793.
Gonçalves, T., Afonso, A. P., and Martins, B. (2016). Why not both?-combining 2d maps and 3d space-time cubes for human trajectory data visualization. In Proceedings of the 30th International BCS Human Computer Interaction Conference 30, pages 1-10, Bournemouth, UK. HCI. DOI: 10.14236/ewic/HCI2016.22.
Guths, L. S., Negrão, M. D., Freitas, C. M., Maciel, A., Torchelsen, R., and Nedel, L. (2025). Debriefing tool for immersive analytics in medical training simulations. Symposium on Virtual and Augmented Reality SVR'2025), 1(1). DOI: 10.1109/SVR67689.2025.00072.
Hu, Y., Wu, S., Xia, S., Fu, J., and Chen, W. (2010). Motion track: Visualizing variations of human motion data. In 2010 IEEE Pacific Visualization Symposium (PacificVis), pages 153-160, Taipei, Taiwan. PacificVis. DOI: 10.1109/PACIFICVIS.2010.5429596.
Hubenschmid, S., Wieland, J., Fink, D. I., Batch, A., Zagermann, J., Elmqvist, N., and Reiterer, H. (2022). Relive: Bridging in-situ and ex-situ visual analytics for analyzing mixed reality user studies. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-20, New Orleans LA USA. CHI 2022. DOI: 10.1145/3491102.3517550.
Hägerstrand, T. (1970). What about people in regional science? Regional Science Association, 1(1). DOI: 10.1111/j.1435-5597.1970.tb01464.x.
Jansen, P., Britten, J., Häusele, A., Segschneider, T., Colley, M., and Rukzio, E. (2023). Autovis: Enabling mixed-immersive analysis of automotive user interface interaction studies. In Proceedings of the 2023 CHI conference on human factors in computing systems, pages 1-23. DOI: 10.1145/3544548.3580760.
Javerliat, C., Villenave, S., Raimbaud, P., and Lavoué, G. (2024). Plume: Record, replay, analyze and share user behavior in 6dof xr experiences. IEEE Transactions on Visualization and Computer Graphics, 30(5):2087-2097. DOI: 10.1109/TVCG.2024.3372107.
Johnson, C. I. and Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 18(2):178. DOI: 10.1037/a0026923.
Kim, Y., Aamir, Z., Singh, M., Boorboor, S., Mueller, K., and Kaufman, A. E. (2025). Explainable xr: Understanding user behaviors of xr environments using llm-assisted analytics framework. IEEE Transactions on Visualization and Computer Graphics. DOI: 10.1109/TVCG.2025.3549537.
Kloiber, S., Settgast, V., Schinko, C., Weinzerl, M., Fritz, J., Schreck, T., and Preiner, R. (2020). Immersive analysis of user motion in vr applications. The Visual Computer, 36(10):1937-1949. DOI: 10.1007/s00371-020-01942-1.
Kveladze, I., Kraak, M.-J., and Van Elzakker, C. P. (2015). The space-time cube as part of a geovisual analytics environment to support the understanding of movement data. International Journal of Geographical Information Science, 29(11):2001-2016. DOI: 10.1080/13658816.2015.1058386.
Lammert, A., Rendle, G., Immohr, F., Neidhardt, A., Brandenburg, K., Raake, A., and Froehlich, B. (2024). Immersive study analyzer: Collaborative immersive analysis of recorded social vr studies. IEEE transactions on visualization and computer graphics, 30(11):7214-7224. DOI: 10.1109/TVCG.2024.3456146.
League, C. and Kennelly, P. (2019). Cartographic symbol design considerations for the space-time cube. The Cartographic Journal, 56(2):117-133. DOI: 10.1080/00087041.2018.1533291.
Lee, G., Lozano-Pérez, T., and Kaelbling, L. P. (2015). Hierarchical planning for multi-contact non-prehensile manipulation. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 264-271, Hamburg, Germany. IEEE RSJ. DOI: 10.1109/IROS.2015.7353384.
Li, A., Liu, J., Cordeil, M., Topliss, J., Piumsomboon, T., and Ens, B. (2023). Gestureexplorer: Immersive visualisation and exploration of gesture data. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-16, Hamburg, Germany. CHI 2023. DOI: 10.1145/3544548.3580678.
Luo, W., Yu, Z., Rzayev, R., Satkowski, M., Gumhold, S., McGinity, M., and Dachselt, R. (2023). Pearl: Physical environment based augmented reality lenses for in-situ human movement analysis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-15. DOI: 10.1145/3544548.3580715.
Malta de La Plata, B. (2013). 3d modeling optimization for multimedia production. Helsinki Metropolia University of Applied Sciences. Available at: [link].
Mayer, R. E. and Fiorella, L. (2014). 12 principles for reducing extraneous processing in multimedia learning: Coherence, signaling, redundancy, spatial contiguity, and temporal contiguity principles. In The Cambridge handbook of multimedia learning, volume 279, pages 279-315. Cambridge University Press New York, NY. DOI: 10.1017/CBO9781139547369.015.
Miki'c, I., Trivedi, M., Hunter, E., and Cosman, P. (2003). Human body model acquisition and tracking using voxel data. International Journal of Computer Vision, 53(3):199-223. DOI: 10.1023/A:1023012723347.
Munzner, T. (2025). Visualization analysis and design. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Courses, pages 1-2, Vancouver, Canada. SIGGRAPH. DOI: 10.1145/3721241.3733989.
Oshita, M. (2019). Motion volume: Visualization of human motion manifolds. In Proceedings of the 17th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry, pages 1-7, Los Angeles, California, USA. SIGGRAPH. DOI: 10.1145/3359997.3365684.
Palmas, G., Bachynskyi, M., Oulasvirta, A., Seidel, H.-P., and Weinkauf, T. (2014). Movexp: A versatile visualization tool for human-computer interaction studies with 3d performance and biomechanical data. IEEE Transactions on Visualization and Computer Graphics, 20(12):2359-2368. DOI: 10.1109/TVCG.2014.2346311.
Pervin, L. A. and Lewis, M., editors (1978). Situational Analysis and the Study of Behavioral Development, pages 49-66. Springer US, Boston, MA. DOI: https://doi.org/10.1007/978-1-4613-3997-7_3.
Poirel, N., Vidal, M., Pineau, A., Lano"e, C., Leroux, G., Lubin, A., Turbelin, M.-R., Berthoz, A., and Houdé, O. (2011). Evidence of different developmental trajectories for length estimation according to egocentric and allocentric viewpoints in children and adults. Experimental Psychology, 1(1). DOI: 10.1027/1618-3169/a000079.
Quere, C., Menin, A., Wu, H.-Y., Moura, P., and Winck, M. (2025). Inspire: An incremental, spatio-temporal and immersive approach for representing and exploring mobility data. In CHItaly 2025-16th Biannual Conference of the Italian SIGCHI Chapter, Italy. CHItaly 2025. DOI: 10.1145/3750069.3750444.
Reipschläger, P., Brudy, F., Dachselt, R., Matejka, J., Fitzmaurice, G., and Anderson, F. (2022). Avatar: An immersive analysis environment for human motion data combining interactive 3d avatars and trajectories. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1-15, . CHI 2022. DOI: 10.1145/3491102.3517676.
Ren, X., Huang, J., Feng, S., Wei, Y., Gong, J., and Ma, S. (2026). Svata: A spatial visual attention tracking and analysis platform for embodied cognition research. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1-27. DOI: 10.1145/3772318.3791192.
Robert, F., Wu, H.-Y., Sassatelli, L., Ramanoel, S., Gros, A., and Winckler, M. (2023). An integrated framework for understanding multimodal embodied experiences in interactive virtual reality. In Proceedings of the 2023 ACM International Conference on Interactive Media Experiences, pages 14-26. DOI: 10.1145/3573381.3596150.
Robert, F., Wu, H.-Y., Sassatelli, L., and Winckler, M. (2024). Task-based methodology to characterise immersive user experience with multivariate data. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 722-731, Orlando, US. IEEE. DOI: 10.1109/VR58804.2024.00092.
Rokoko (2020). Modern mocap: An evolutionary overview. newblock [link].
Schmidt, B., Doeweling, S., and Mühlhäuser, M. (2012). Interaction history visualization. In Proceedings of the 30th ACM international conference on Design of communication, pages 261-270, Seattle Washington USA. SIGDOC. DOI: 10.1145/2379057.2379107.
Schroeder, N. L. and Cenkci, A. T. (2018). Spatial contiguity and spatial split-attention effects in multimedia learning environments: A meta-analysis. Educational Psychology Review, 30(3):679-701. DOI: 10.1007/s10648-018-9435-9.
Seraji, M. R., Piray, P., Zahednejad, V., and Stuerzlinger, W. (2024). Analyzing user behaviour patterns in a cross-virtuality immersive analytics system. IEEE Transactions on Visualization and Computer Graphics, 30(5):2613-2623. DOI: 10.1109/TVCG.2024.3372129.
Shen, C., Fan, C., Wu, W., Wang, R., Huang, G. Q., and Yu, S. (2023). Lidargait: Benchmarking 3d gait recognition with point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1054-1063, Vancouver, Canada. IEEE CVF. DOI: 10.1109/CVPR52729.2023.00108.
Shneiderman, B. (2003). The eyes have it: A task by data type taxonomy for information visualizations. In The craft of information visualization, pages 364-371. Elsevier. DOI: 10.1016/B978-155860915-0/50046-9.
Skulmowski, A. and Xu, K. M. (2022). Understanding cognitive load in digital and online learning: A new perspective on extraneous cognitive load. Educational psychology review, 34(1):171-196. DOI: 10.1007/s10648-021-09624-7.
Vrotsou, K., Forsell, C., and Cooper, M. (2010). 2d and 3d representations for feature recognition in time geographical diary data. Information Visualization, 9(4):263-276. DOI: 10.1057/ivs.2009.30.
Wagner, J., Silva, C. T., Stuerzlinger, W., and Nedel, L. (2024). Reimagining taxivis through an immersive space-time cube metaphor and reflecting on potential benefits of immersive analytics for urban data exploration. In 2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 827-838, Orlando, US. IEEE. DOI: 10.1109/VR58804.2024.00102.
Wei, C., Cheng, R., Ning, C., Wei, X., Peng, X., Lv, T., Sheng, F., Dong, K., and Wang, Z. L. (2023). A self-powered body motion sensing network integrated with multiple triboelectric fabrics for biometric gait recognition and auxiliary rehabilitation training. Advanced Functional Materials, 33(35):2303562. DOI: 10.1002/adfm.202303562.
Wijntjes, M., Cesar, P., and El Ali, A. (2021). Designing and evaluating avatar biosignal visualization techniques in social virtual reality. CWI. Available at: [link].
Wu, H.-Y., Robert, F., Gallo, F. F., Pirkovets, K., Quéré, C., Delachambre, J., Ramano"el, S., Gros, A., Winckler, M., Sassatelli, L., et al. (2025). Exploring, walking, and interacting in virtual reality with simulated low vision: a living contextual dataset. Scientific Data, 12(1):330. DOI: 10.1038/s41597-025-04560-5.
Xu, B. (2024). A review on application of motion sensing technology in human-computer interaction. In Third International Conference on Intelligent Mechanical and Human-Computer Interaction Technology (IHCIT 2024), volume 13284, pages 233-239, Bristol, UK. SPIE, IHCIT2024. DOI: 10.1117/12.3049765.
Yu, X., Lu, J., Liu, W., Cheng, Z., and Xiao, G. (2024). Exploring physiological stress response evoked by passive translational acceleration in healthy adults: a pilot study utilizing electrodermal activity and heart rate variability measurements. Scientific reports, 14(1):11349. DOI: 10.1038/s41598-024-61656-5.
Zhang, Y., Murat, E., Yu, L., Huang, H., Choi, M., Mousas, C., and Yu, L.-F. (2026). Hieravisvr: Hierarchical visual analytics for motion-centric vr playtesting. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, pages 1-19. DOI: 10.1145/3772318.3790591.
Zhang, Y., Wang, Z., Zhang, J., Shan, G., and Tian, D. (2023). A survey of immersive visualization: Focus on perception and interaction. Visual Informatics, 7(4):22-35. DOI: 10.1016/j.visinf.2023.10.003.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Florent Robert, Clément Quere, Aline Menin, Hui-Yin Wu, Marco Winckler
This work is licensed under a Creative Commons Attribution 4.0 International License.
