Breaking Communication Barriers: Key Requirements for User Interaction with Oral-Sign Language Translation Systems

Natália Sales Santos; Lucas Almeida S. de Souza; Julia Manuela G.  Soares; Raniere Alislan A.  Cordeiro; Elidéa Lúcia Almeida Bernardino; Mario Fernando Montenegro Campos; Raquel Oliveira Prates

doi:10.5753/jbcs.2026.5745

Authors

Natália Sales Santos Universidade Federal de Minas Gerais https://orcid.org/0000-0002-6037-1378
Lucas Almeida S. de Souza Universidade Federal de Minas Gerais https://orcid.org/0009-0007-4038-1548
Julia Manuela G. Soares Universidade Federal de Minas Gerais https://orcid.org/0009-0009-7680-771X
Raniere Alislan A. Cordeiro Universidade Federal de Minas Gerais https://orcid.org/0009-0006-5588-6023
Elidéa Lúcia Almeida Bernardino Universidade Federal de Minas Gerais https://orcid.org/0000-0002-3718-9631
Mario Fernando Montenegro Campos Universidade Federal de Minas Gerais https://orcid.org/0000-0002-8336-9190
Raquel Oliveira Prates Universidade Federal de Minas Gerais https://orcid.org/0000-0002-7128-4974

DOI:

https://doi.org/10.5753/jbcs.2026.5745

Keywords:

Automatic translation, Brazilian sign language, Libras, Portuguese, deaf user, accessibility, healthcare

Abstract

Communication between deaf and hearing people is still a major challenge, especially in critical moments such as medical emergencies, where information needs to be transmitted clearly and quickly. In this context, for the development of efficient machine translation systems, it is essential that they meet the real needs of both deaf people and health professionals. This article identifies and defines fundamental requirements for the development of such systems, taking into account the experience of both groups. In addition to mapping these requirements, an evaluation was carried out to validate their relevance and ensure that they correspond to the real demands of users. The work is part of the project "Captar-Libras: Video communication system for the deaf applied to pre-medical care" (In Portuguese: "Captar-Libras: Sistema de Comunicação por vídeos para surdos aplicado ao pré-atendimento médico"), which develops a bidirectional translation solution between Portuguese and Brazilian Sign Language (Libras) using photorealistic avatars, with a focus on healthcare. The results of this article offer guidelines for the development of more accessible and efficient systems, promoting more inclusive communication between deaf and hearing people.

Downloads

Download data is not yet available.

References

Areeb, Q. M. and Nadeem, M. (2021). Deep learning based hand gesture recognition for emergency situation: A study on indian sign language. In 2021 International Conference on Data Analytics for Business and Industry (ICDABI), pages 33-36. IEEE. DOI: 10.1109/ICDABI53623.2021.9655842.

Barnet, H. (2021). Snapshot insight report: Deaf people’s healthcare challenges. Available at:[link]. Last access: November, 2025.

Beaver, S. and Carty, B. (2021). Viewing the healthcare system through a deaf lens. Public Health Research and Practice, 31(5):1-4. DOI: 10.17061/phrp3152127.

Bond, L. (2019). Surdos enfrentam dificuldade para atendimento em saúde. Available at:[link]. Last access: November, 2025.

da Silva, D. R., de Araújo, T. M. U., do Rêgo, T. G., Brandão, M. A. C., and Gonçalves, L. M. G. (2024). A multiple stream architecture for the recognition of signs in brazilian sign language in the context of health. Multimedia Tools and Applications, 83(7):19767-19785. DOI: 10.1007/s11042-023-16332-7.

da Silva, D. R. B., Araujo, T. M. U., Rêgo, T. G. d., and Brandão, M. A. C. (2020). A two-stream model based on 3d convolutional neural networks for the recognition of brazilian sign language in the health context. In Proceedings of the Brazilian Symposium on Multimedia and the Web, WebMedia '20, page 5–12, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3428658.3430980.

Das, H. V., Mohan, K., Paul, L., Kumaresan, S., and Nair, C. S. (2024). Transforming consulting atmosphere with indian sign language translation. Multimedia Tools and Applications, 83(5):13543-13555. DOI: 10.1007/s11042-023-15214-2.

David, D., Alamoodi, A., Albahri, O., Garfan, S., Albahri, A., Zaidan, B., and Chen, J. (2024). Sign language mobile apps: a systematic review of current app evaluation progress and solution framework. Evolving Systems, 15(2):669-686. DOI: 10.1007/s12530-023-09494-0.

de Menezes, J. E. S. A. and Feitosa, C. R. d. S. (2015). Língua Brasileira de Sinais (LIBRAS). Editora da Universidade Estadual do Ceará (EdUECE). Available at:[link].

De Meulder, M. (2021). Is "good enough" good enough? ethical and responsible development of sign language technologies. In Shterionov, D., editor, Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), pages 12-22, Virtual. Association for Machine Translation in the Americas. Available at:[link].

de Souza, V. D., Hoeckele, A. G., Borim, M. L. C., Christinelli, H. C. B., and Costa, M. A. R. (2020). Percepção de surdos sobre o atendimento nos serviços de saúde. Brazilian Journal of Development, 6(8):55347-55356. DOI: 10.34117/bjdv6n8-091.

Dere, M., Dere, R. O., Adesina, A., and Yauri, A. R. (2022). Smartcall: A real-time, sign language medical emergency communicator. In 2022 5th Information Technology for Education and Development (ITED), pages 1-6. DOI: 10.1109/ITED56637.2022.10051420.

Escudeiro, P., Escudeiro, N., Reis, R., Lopes, J., Norberto, M., Baltasar, A. B., Barbosa, M., and Bidarra, J. (2015). Virtual sign-a real time bidirectional translator of portuguese sign language. Procedia Computer Science, 67:252-262. DOI: 10.1016/j.procs.2015.09.269.

Farooq, U., Rahim, M. S. M., Sabir, N., Hussain, A., and Abid, A. (2021). Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Computing and Applications, 33(21):14357-14399. DOI: 10.1007/s00521-021-06079-3.

Francisco, G. d. S. A. M. and Mesquita, C. T. (2023). Access to health services among deaf people: An issue of inclusion and linguistic rights. International Journal of Cardiovascular Sciences, 36. DOI: 10.36660/ijcs.20220196.

Guo, Y., Zhao, J., Ding, B., Tan, C., Ling, W., Tan, Z., Miyaki, J., Du, H., and Lu, S. (2023). Sign-to-911: Emergency Call Service for Sign Language Users with Assistive AR Glasses. Association for Computing Machinery, New York, NY, USA. DOI: 10.1145/3570361.3613260.

Hisham, B. and Hamouda, A. (2019). Supervised learning classifiers for arabic gestures recognition using kinect v2. SN Applied Sciences, 1(7):768. DOI: 10.1007/s42452-019-0771-2.

Izquierdo-Condoy, J. S., Sánchez Abadiano, L. E., Sánchez, W., Rodríguez, I., De La Cruz Matías, K., Paz, C., and Ortiz-Prado, E. (2024). Exploring healthcare barriers and satisfaction levels among deaf individuals in ecuador: A video-based survey approach. Disability and Health Journal, 17(3):101622. DOI: 10.1016/j.dhjo.2024.101622.

Kipp, M., Nguyen, Q., Heloir, A., and Matthes, S. (2011). Assessing the deaf user perspective on sign language avatars. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility, pages 107-114. DOI: 10.1145/2049536.2049557.

Ministério da Saúde (2019). Pesquisa nacional em saúde. Available at:[link]. Last access: November, 2025.

Naranjo-Zeledón, L., Peral, J., Ferrández, A., and Chacón-Rivas, M. (2019). A systematic mapping of translation-enabling technologies for sign languages. Electronics, 8(9):1047. DOI: 10.3390/electronics8091047.

Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J. and Mack, R. L., editors, Usability Inspection Methods, pages 25-62. John Wiley & Sons, Inc., New York, NY, USA. Book.

Oliveira, T., Escudeiro, P., Escudeiro, N., Rocha, E., and Barbosa, F. M. (2019). Automatic sign language translation to improve communication. In 2019 IEEE Global Engineering Education Conference (EDUCON), pages 937-942. IEEE. DOI: 10.1109/educon.2019.8725244.

Parker, C., Scott, S., and Geddes, A. (2019). Snowball sampling. SAGE research methods foundations. DOI: 10.4135/9781526421036831710.

Prates, R. O., Santos, N. S., de Souza, L. A., Marcolino, M. S., Bernardino, E. L., and Campos, M. F. (2025). Hci challenges in the development of captar-libras: Providing accessibility for deaf patients in health context. In IFIP Conference on Human-Computer Interaction, pages 313-322. Springer. DOI: 10.1007/978-3-032-05005-2_16.

Rezende, R. F., Guerra, L. B., and Carvalho, S. A. d. S. (2021). The perspective of deaf patients on health care. Revista CEFAC, 23:e0620. DOI: 10.1590/1982-0216/20212320620.

Rocha, J., Lensk, J., Ferreira, T., and Ferreira, M. (2020). Towards a tool to translate brazilian sign language (libras) to brazilian portuguese and improve communication with deaf. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 1-4. DOI: 10.1109/VL/HCC50065.2020.9127257.

Rogers, K. D., Lovell, K., Bower, P., Armitage, C. J., and Young, A. (2025). What are deaf sign language users’ experiences as patients in healthcare services? a scoping review. PLOS Global Public Health, 5(2):e0003535. DOI: 10.37766/inplasy2022.1.0102.

Santos, N. S., Souza, L. A. S. d., Soares, J. M. G., and Prates, R. O. (2024). Connecting silent worlds: Requirements for automatic oral-sign language translation. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, pages 1-14. DOI: 10.1145/3702038.3702066.

Soares, J. M. G., Carvalho, I. F. d., Bernardino, E. L. A., Marcolino, M. S., and Prates, R. O. (2024). An evaluation of portuguese to libras translator apps applied to the medical context. In Antona, M. and Stephanidis, C., editors, Universal Access in Human-Computer Interaction, pages 290-304, Cham. Springer Nature Switzerland. DOI: 10.1007/978-3-031-60884-1_20.

Sosa-Jiménez, C. O., Ríos-Figueroa, H. V., and Solís-González-Cosío, A. L. (2022). A prototype for mexican sign language recognition and synthesis in support of a primary care physician. IEEE Access, 10:127620-127635. DOI: 10.1109/access.2022.3226696.

Steinberg, A. G., barnett, S., Meador, H. E., Wiggins, E. A., and Zazove, P. (2006). Health care system accessibility: experiences and perceptions of deaf people. Journal of general internal medicine, 21:260-266. DOI: 10.1111/j.1525-1497.2006.00340.x.

Süzgün, M., Özdemir, H., Camgöz, N., Kındıroğlu, A., Başaran, D., Togay, C., and Akarun, L. (2015). Hospisign: an interactive sign language platform for hearing impaired. Journal of Naval Sciences and Engineering, 11(3):75-92. Available at:[link].

Tran, N., Ladner, R. E., and Bragg, D. (2023). U.s. deaf community perspectives on automatic sign language translation. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS '23. Association for Computing Machinery. DOI: 10.1145/3597638.3614507.

Wheatley, M. (2021). Barriers faced by deaf people while accessing healthcare. Available at:[link]. Last access: November, 2025.

WHO (2021). World report on hearing. Available at:[link]. Licence: CC BY-NC-SA 3.0 IGO. Last access: November, 2025.

Xia, K., Lu, W., Fan, H., and Zhao, Q. (2022). A sign language recognition system applied to deaf-mute medical consultation. Sensors, 22(23):9107. DOI: 10.3390/s22239107.