TeamBridge 2.0: an extensible and non-invasive middleware for control and adapting games for motor rehabilitation

Rummenigge Dantas; Alan K. S. Alves; Andre V. dos Santos

doi:10.5753/jis.2024.3516

Authors

Rummenigge Dantas Universidade Federal do Rio Grande do Norte https://orcid.org/0000-0001-9486-6574
Alan K. S. Alves Universidade Federal do Rio Grande do Norte https://orcid.org/0000-0003-2501-4751
Andre V. dos Santos Universidade Federal do Rio Grande do Norte https://orcid.org/0000-0002-3164-9166

DOI:

https://doi.org/10.5753/jis.2024.3516

Keywords:

Exergames, Virtual Reality, Middleware, Gesture Interaction, Motor Rehabilitation

Abstract

This work present the TeamBridge 2.0, a middleware able to perform the communication between digital games and hardware devices, like joysticks, mice and cameras (both rgbd and monocular). That communication does not require any modification to the game source code, allowing an old game to be adapted to work with a new hardware device. To prove this, tests were carried out with several games, including one of them being a commercial game. This middleware also allows the use of more than one device at the same time, so we can obtain more accurate information, one device can supply the deficiencies of the other. Finally, we performed tests to make sure that the middleware would not interfere with the user experience. Tests have shown that TeamBridge 2.0 can receive, interpret and send information quickly, the time varies according to the device used, getting 33ms when used with Kinect, 40ms with Leap Motion and 255ms with a DIY Data-Glove.

Downloads

Download data is not yet available.

References

Bambach, S., Lee, S., Crandall, D. J., and Yu, C. (2015). Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE international conference on computer vision, pages 1949–1957.

Bianor, F., Cavalcanti, A., and Dantas, R. R. (2017a). Evaluate leap motion control for multiple hand posture recognition. 19th Symposium on Virtual and Augmented Reality.

Bianor, F., Cavalcanti, A., and Dantas, R. R. (2017b). Physiohappy: A low cost device for virtual rehabilitation. 19th Symposium on Virtual and Augmented Reality.

Borja, E. F., Lara, D. A., Quevedo, W. X., and Andaluz, V. H. (2018). Haptic stimulation glove for fine motor rehabilitation in virtual reality environments. In International conference on augmented reality, Virtual Reality and Computer Graphics, pages 211–229. Springer.

Cano Porras, D., Sharon, H., Inzelberg, R., Ziv-Ner, Y., Zeilig, G., and Plotnik, M. (2019). Advanced virtual reality-based rehabilitation of balance and gait in clinical practice. Therapeutic advances in chronic disease, 10:2040622319868379.

Cassiano, L. (2013). Kayak supremo. Demos do SBGames 2013. [link] Accessed: 13 January 2024.

Conconi, A., Ganchev, T., Kocsis, O., Papadopoulos, G., Fernández-Aranda, F., and Jiménez-Murcia, S. (2008). Playmancer: A serious gaming 3d environment. In 2008 International Conference on Automated Solutions for Cross Media Content and Multi-Channel Distribution, pages 111–117. IEEE.

Corbetta, D., Imeri, F., and Gatti, R. (2015). Rehabilitation that incorporates virtual reality is more effective than standard rehabilitation for improving walking speed, balance and mobility after stroke: a systematic review. Journal of physiotherapy, 61(3):117–124.

Davies, B. R., McDonald Jr, M. J., and Harrigan, R. W. (1995). Virtual collaborative environments: programming and controlling robotic devices remotely. In Telemanipulator and Telepresence Technologies, volume 2351, pages 34–43. SPIE.

Drummond, D., Hadchouel, A., and Tesnière, A. (2017). Serious games for health: three steps forwards. Advances in Simulation, 2(1):1–8.

Fregnan, E., Baum, T., Palomba, F., and Bacchelli, A. (2019). A survey on software coupling relations and tools. Information and Software Technology, 107:159–178.

Garrett, B., Taverner, T., Gromala, D., Tao, G., Cordingley, E., Sun, C., et al. (2018). Virtual reality clinical research: promises and challenges. JMIR serious games, 6(4):e10839.

Huang, M.-C., Xu, W., Su, Y., Lange, B., Chang, C.-Y., and Sarrafzadeh, M. (2012). Smartglove for upper extremities rehabilitative gaming assessment. In Proceedings of the 5th International Conference on Pervasive Technologies Related to Assistive Environments, page 20. ACM.

Iqbal, J., Tsagarakis, N. G., Fiorilla, A. E., and Caldwell, D. G. (2010). A portable rehabilitation device for the hand. In Engineering in Medicine and Biology Society (EMBC), 2010 Annual International Conference of the IEEE, pages 3694–3697. IEEE.

Jones, S. E. and Thiruvathukal, G. K. (2012). Codename revolution: the Nintendo Wii platform. MIT Press.

Konstantinidis, E. I., Antoniou, P. E., Bamparopoulos, G., and Bamidis, P. D. (2015). A lightweight framework for transparent cross platform communication of controller data in ambient assisted living environments. Information Sciences, 300:124–139.

Lee, E.-S. and Shin, B.-S. (2021). A flexible input mapping system for next-generation virtual reality controllers. Electronics, 10(17):11.

Lu, A. S. and Kharrazi, H. (2018). A state-of-the-art systematic content analysis of games for health. Games for health journal, 7(1):1–15.

Lun, R. and Zhao, W. (2015). A survey of applications and human motion recognition with microsoft kinect. International Journal of Pattern Recognition and Artificial Intelligence, 29(05):1555008.

Luo, D., Du, S., and Ikenaga, T. (2019). Multi-task and multilevel detection neural network based real-time 3d pose estimation. In 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 1427–1434. IEEE.

Marshall, D. (1999). Further threads programming:synchronization. [link] Accessed: 13 January 2024.

Matamala-Gomez, M., Slater, M., and Sanchez-Vives, M. V. (2022). Impact of virtual embodiment and exercises on functional ability and range of motion in orthopedic rehabilitation. Scientific reports, 12(1):1–10.

Mathiowetz, V., Volland, G., Kashman, N., and Weber, K. (1985). Adult norms for the box and block test of manual dexterity. American Journal of Occupational Therapy, 39(6):386–391.

Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., Xu, W., Casas, D., and Theobalt, C. (2017). Vnect: Real-time 3d human pose estimation with a single rgb camera. Acm transactions on graphics (tog), 36(4):1–14.

Meier, J., Farre, C., Bansode, P., Barber, S., and Rea, D. (2007). Performance Testing Guidance for Web Applications: Patterns & Practices. Microsoft Press, Redmond, WA, USA.

Michotte, A. (2017). The perception of causality, volume 21. Routledge.

Milazzo, F., Gentile, V., Gentile, A., and Sorce, S. (2018). Kind-dama: A modular middleware for kinect-like device data management. Software: Practice and Experience, 48(1):141–160.

Mossel, A., Schönauer, C., Gerstweiler, G., and Kaufmann, H. (2013). Artifice-augmented reality framework for distributed collaboration. International Journal of Virtual Reality.

Mugueta-Aguinaga, I. and Garcia-Zapirain, B. (2017). Fred: Exergame to prevent dependence and functional deterioration associated with ageing. a pilot three-week randomized controlled clinical trial. International journal of environmental research and public health, 14(12):1439.

Oh, Y. and Yang, S. (2010). Defining exergames & exergaming. Proceedings of Meaningful Play, pages 1–17.

Pacheco, T. B. F. (2020). Desenvolvimento e usabilidade de um jogo digital para reabilitação do equilíbrio postural de idosos. In Tese de Doutorado em Fisioterapia, page 38. Universidade Federal do Rio Grande do Norte (UFRN).

Pandit, S., Tran, S., Gu, Y., Saraee, E., Jansen, F., Singh, S., Cao, S., Sadeghi, A., Shandelman, E., Ellis, T., et al. (2019). Exercisecheck: A scalable platform for remote physical therapy deployed as a hybrid desktop and web application. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pages 101–109.

Park, D.-S., Lee, D.-G., Lee, K., and Lee, G. (2017). Effects of virtual reality training using xbox kinect on motor function in stroke survivors: a preliminary study. Journal of Stroke and Cerebrovascular Diseases, 26(10):2313–2319.

Pirovano, M., Lanzi, P. L., Mainetti, R., and Borghese, N. A. (2013). The design of a comprehensive game engine for rehabilitation. In Games Innovation Conference (IGIC), 2013 IEEE International, pages 209–215. IEEE.

Rose, T., Nam, C. S., and Chen, K. B. (2018). Immersion of virtual reality for rehabilitation-review. Applied ergonomics, 69:153–161.

Rybarczyk, Y., Luis Pérez Medina, J., Leconte, L., Jimenes, K., González, M., and Esparza, D. (2019). Implementation and assessment of an intelligent motor tele-rehabilitation platform. Electronics, 8(1):58.

Santos, L. H. d. O. (2016). Arcabouço para construção de jogos ubíquos com foco em reabilitação. In Dissertação de Mestrado em Informática, page 60. Universidade de Brasília (UnB).

Sartori, F. (2020). An api for wearable environments development and its application to mhealth field. Sensors, 20(21):5970.

Shen, J. and Pantic, M. (2009). A software framework for multimodal humancomputer interaction systems. In Systems, Man and Cybernetics, 2009. SMC 2009. IEEE International Conference on, pages 2038–2045. IEEE.

Siddiqui, S. A., Snober, Y., Raza, S., Khan, F. M., and Syed, T. Q. (2015). Arm gesture recognition on microsoft kinectusinga hidden markov model-based representations of poses. In 2015 International Conference on Information and Communication Technologies (ICICT), pages 1–6. IEEE.

Silva, F. G. M., Bessa, N. P. O. S., Amaral, R. H. O., Pacheco, T. B. F., Medeiros, F. B. S., Nagem, D. A. P., Dantas, R. R., and Cavalcanti, F. A. d. C. (2020). Physiohappy interface in the treatment of wrist and hand of a child with cerebral palsy: case repot. Research, Society and Development, 9.

Silva, L., Dantas, R., Diniz, P., Jeronimo, V., Bueno, L., and Dutra, T. (2016). Phys. io: Wearable hand tracking device. In Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), 2016 IEEE International Conference on, pages 1–6. IEEE.

Silva, L., Dantas, R., Pantoja, A., and Pereira, A. (2013). Development of a low cost dataglove based on arduino for virtual reality applications. In 2013 IEEE International conference on computational intelligence and virtual environments for measurement systems and applications (CIVEMSA), pages 55–59. IEEE.

Souza, C. H. R., de Oliveira, D. M., do Nascimento, D. F., de Oliveira Berretta, L., and de Carvalho, S. T. (2022). A serious games and game elements based approach for patient telerehabilitation contexts. Journal on Interactive Systems, 13(1):179–191.

Suma, E. A., Lange, B., Rizzo, A. S., Krum, D. M., and Bolas, M. (2011). Faast: The flexible action and articulated skeleton toolkit. In Virtual Reality Conference (VR), 2011 IEEE, pages 247–248. IEEE.

Takaiwa, M., Noritsugu, T., Ito, N., and Sasaki, D. (2011). Wrist rehabilitation device using pneumatic parallel manipulator based on emg signal. Int. J. Autom. Technol., 5(4):472–477.

Taylor II, R. M., Hudson, T. C., Seeger, A., Weber, H., Juliano, J., and Helser, A. T. (2001). Vrpn: a deviceindependent, network-transparent vr peripheral system. In Proceedings of the ACM symposium on Virtual reality software and technology, pages 55–61. ACM.

Teófilo, L. F., Nogueira, P. A., and Silva, P. B. (2013). Gemini: A generic multi-modal natural interface framework for videogames. In Advances in Information Systems and Technologies, pages 873–884. Springer.

Tieri, G., Morone, G., Paolucci, S., and Iosa, M. (2018). Virtual reality in cognitive and motor rehabilitation: facts, fiction and fallacies. Expert review of medical devices, 15(2):107–117.

Wainer, J. et al. (2007). Métodos de pesquisa quantitativa e qualitativa para a ciência da computação. Atualização em informática, 1:221–262.

Wang, Y., Ijaz, K., Yuan, D., and Calvo, R. A. (2020). Vrrides: An object-oriented application framework for immersive virtual reality exergames. Software: Practice and Experience, 50(7):1305–1324.

Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D. (2013). Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13(5):6380–6393.

Weiss, P. L., Keshner, E. A., and Levin, M. F. (2014). Virtual reality for physical and motor rehabilitation. Springer.