Use of Neural Networks to Classify Real Time Signs From Brazilian Sign Language
DOI:
https://doi.org/10.5753/jis.2026.6369Keywords:
Neural Network, Sign Classification, Brazilian Sign Language, Real-Time DetectionAbstract
This study presents the use of convolutional neural networks aimed at detecting and simultaneously translating continuous signs of Brazilian Sign Language (LIBRAS) in videos captured by common computer cameras. The objective of this work was to propose a methodology that enabled the classification of signs performed by people of different genders, body types, and skin tones, with minimal interference from the video background while using cameras of medium or low quality, creating an accessible approach for various applications. For this purpose, the MediaPipe algorithm was used for video data extraction, the FastDTW algorithm was employed for data standardization, convolutional neural networks were implemented using the TensorFlow and Keras libraries for sign recognition, and a custom dataset was created to train the network. All these tools were integrated using the Python programming language to produce a model for real-time classification of continuous LIBRAS signals.
Downloads
References
Abdallah, E. E. and Fayyoumi, E. (2016). Assistive technology for deaf people based on android platform. Procedia Computer Science, 94:295–301. DOI: https://doi.org/10.1016/j.procs.2016.08.044.
Almeida, S. G. M., Rezende, T. M., Almeida, G. T. B., Toffolo, A. C. R., and Guimarães, F. G. (2019). Minds-libras dataset. Zenodo. DOI: https://doi.org/10.5281/zenodo.2667329.
Antunes, D. R., Guimarães, C., García, L. S., Oliveira, L. E. S., and Fernandes, S. (2011). A framework to support development of sign language humancomputer interaction: Building tools for effective information access and inclusion of the deaf. In 2011 Fifth International Conference on Research Challenges in Information Science, pages 1–12. DOI: https://doi.org/10.1016/10.1109/RCIS.2011.6006832.
Badhe, P. C. and Kulkarni, V. (2015). Indian sign language translator using gesture recognition algorithm. In 2015 IEEE International Conference on Computer Graphics, Vision and Information Security (CGVIS), pages 195–200. DOI: https://doi.org/10.1109/CGVIS.2015.7449921.
Bastos, I. L., Angelo, M. F., and Loula, A. C. (2015). Recognition of static gestures applied to brazilian sign language (libras). In 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, pages 305–312. DOI: https://doi.org/10.1109/SIBGRAPI.2015.26.
Bhamidipati, V. S. P., Saxena, I., Saisanthiya, D., and Retnadhas, M. (2023). Robust intelligent posture estimation for an ai gym trainer using mediapipe and opencv. In 2023 International Conference on Networking and Communications (ICNWC), pages 1–7. DOI: https://doi.org/10.1109/ICNWC57852.2023.10127264.
Brasil (2005). Decreto nº 5.626, de 22 de dezembro de 2005. Diário Oficial da União. Regulamenta a Lei nº 10.436, de 24 de abril de 2002, que dispõe sobre a Língua Brasileira de Sinais - Libras, e o art. 18 da Lei nº 10.098, de 19 de dezembro de 2000.
Brasil (2015). Lei nº 13.146, de 6 de julho de 2015. Diário Oficial da União. Institui a Lei Brasileira da Inclusão da Pessoa com Deficiência. [link]. Accessed on 07 July 2026.
Carvalho, G., Brandão, A., and Ferreira, F. (2021). Handarch: A deep learning architecture for libras hand configuration recognition. In Anais do XVII Workshop de Visão Computacional, pages 19–24, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/wvc.2021.18883.
Castro, S. S., Lefèvre, F., Lefèvre, A. M. C., and Cesar, C. L. G. (2011). Acessibilidade aos serviços de saúde por pessoas com deficiência. Revista de Saúde Pública, 45(1):99–105. DOI: https://doi.org/10.1590/S0034-89102010005000048.
Corrêa, Y., Maristela, C., Santarosa, L., and Biazus, M. C. (2014). Aplicativos de tradução para libras e a busca pela validade social da tecnologia assistiva. In XXV Simpósio Brasileiro de Informática na Educação, page 164. DOI: https://doi.org/10.5753/cbie.sbie.2014.164.
Costa, A. P., Diniz, E., Sales, E., Coelho, F., Costa, W. C., and Sousa, L. (2024). Usabilidade de um aplicativo de libras para educação e governança em uma cidade amazônica. In Anais do XXXV Simpósio Brasileiro de Informática na Educação, pages 1248–1262, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbie.2024.242459.
Courtin, C. (2000). The impact of sign language on the cognitive development of deaf children: The case of theories of mind. The Journal of Deaf Studies and Deaf Education, 5(3):266–276. DOI: https://doi.org/10.1093/deafed/5.3.266.
Córdova, G. D. (2018). Língua de herança: Língua brasileira de sinais – livro de ronice müller de quadros. Pensares em Revista, 13. DOI: https://doi.org/10.12957/pr.2018.33402.
De Oliveira Feliciano, F. D., De Andrade Briano, A., Prudencio, R. B. C., and Alves, E. C. (2023). Recognition of static or dynamic libras words in complex background environments. In 2023 18th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–4. DOI: https://doi.org/10.23919/CISTI58278.2023.10211498.
Dias, T. S., Junior, J. J. A. M., and Pichorim, S. F. (2024). Classification of brazilian sign language gestures based on recurrent neural networks models, with instrumented glove. In Marques, J. L. B., Rodrigues, C. R., Suzuki, D. O. H., Marino Neto, J., and García Ojeda, R., editors, IX Latin American Congress on Biomedical Engineering and XXVIII Brazilian Congress on Biomedical Engineering, pages 611–620, Cham. Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-031-49407-9_61.
Figueiredo, A., Guarisi, A., Valinho, C., Marcelino, G., and Souza, V. (2025). De sinais a texto: Um sistema inteligente para tradução automática de libras com foco na inclusão educacional. In Anais do XXXVI Simpósio Brasileiro de Informática na Educação, pages 1489–1496, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbie.2025.12170.
Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press, Cambridge, MA, USA. ISBN: 978-0262035613.
Hommes, R. E., Borash, A. I., Hartwig, K., and DeGracia, D. (2018). American sign language interpreters perceptions of barriers to healthcare communication in deaf and hard of hearing patients. Journal of Community Health, 43:956–961. DOI: https://doi.org/10.1007/s10900-018-0511-3.
Instituto Brasileiro de Geografia e Estatística (2019). Pesquisa nacional de saúde. [link]. Accessed on 07 July 2026.
Li, D., Opazo, C. R., Yu, X., and Li, H. (2020). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1448–1458. DOI: https://doi.org/10.1109/WACV45572.2020.9093512.
Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C., Yong, M. G., Lee, J., Chang, W., Hua, W., Georg, M., and Grundmann, M. (2019). Mediapipe: A framework for building perception pipelines. CoRR, abs/1906.08172. DOI: https://doi.org/10.48550/arXiv.1906.08172.
Palavissini, C. F. C., Lima, K. R. L. d., Castro, L. P. V. d., and Lima, D. F. d. (2021). Digital information and communication technologies on the acquisition of scientific knowledge to deaf students: an integrative literature review. Research, Society and Development, 10(16). DOI: https://doi.org/10.33448/rsd-v10i16.23998.
Peixoto, J. S., Cukla, A. R., Welfer, D., and Gamarra, D. F. T. (2022). Comparison of different processing methods of joint coordinates features for gesture recognition with a rnn in the msrc-12. In Abraham, A., Gandhi, N., Hanne, T., Hong, T.-P., Nogueira Rios, T., and Ding, W., editors, Intelligent Systems Design and Applications, pages 498–507, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-96308-8_46.
Peixoto, J. S., Pfitscher, M., de Souza Leite Cuadros, M. A., Welfer, D., and Gamarra, D. F. T. (2021). Comparison of different processing methods of joint coordinates features for gesture recognition with a cnn in the msrc-12 database. In Abraham, A., Piuri, V., Gandhi, N., Siarry, P., Kaklauskas, A., and Madureira, A., editors, Intelligent Systems Design and Applications, pages 590–599, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-71187-0_54.
Quadros, R. M. D. and Karnopp, L. B. (2007). Língua de sinais brasileira: estudos lingüísticos. Biblioteca Artmed. Lingüística. Artmed, São Paulo. ISBN: 9788536303086. [link]. Accessed on 11 July 2026.
Rastgoo, R., Kiani, K., and Escalera, S. (2021). Sign language recognition: A deep survey. Expert Systems with Applications, 164:113794. DOI: https://doi.org/10.1016/j.eswa.2020.113794.
Rocha, J., Lensk, J., Ferreira, T., and Ferreira, M. (2020). Towards a tool to translate brazilian sign language (libras) to brazilian portuguese and improve communication with deaf. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pages 1–4. DOI: https://doi.org/10.1109/VL/HCC50065.2020.9127257.
Rodrigues, A. and Zanchettin, C. (2024). V-librasil - a new dataset with signs in brazilian sign language (libras). IEEE Dataport. DOI: https://doi.org/10.21227/v589-hn68.
Rumjanek, V. and da Silva, W. (2019). Ciência para todos? Revista Brasileira de Pós-Graduação, 15:1–20. DOI: https://doi.org/10.21713/rbpg.v15i34.1606.
Salvador, S. and Chan, P. (2007). Toward accurate dynamic time warping in linear time and space. Intelligent Data Analysis, 11(5):561–580. DOI: https://doi.org/10.3233/IDA-2007-11508.
Sarmento, A. H. d. A. and Ponti, M. A. (2023). A cross-dataset study on the brazilian sign language translation. In IEEE/CVF International Conference on Computer Vision Workshops - ICCVW. IEEE. DOI: https://doi.org/10.1109/ICCVW60793.2023.00300.
Silva, D., Araujo, T., Rêgo, T., Brandão, M., and Gonçalves, L. (2023). A multiple stream architecture for the recognition of signs in brazilian sign language in the context of health. Multimedia Tools and Applications, 83. DOI: https://doi.org/10.1007/s11042-023-16332-7.
Stopa, S. R., Szwarcwald, C. L., Oliveira, M. M. d., Gouvea, E. d. C. D. P., Vieira, M. L. F. P., Freitas, M. P. S. d., Sardinha, L. M. V., and Macário, E. M. (2020). Pesquisa nacional de saúde 2019: histórico, métodos e perspectivas. Epidemiologia e Serviços de Saúde, 29(5):e2020315. DOI: https://doi.org/10.1590/S1679-49742020000500004.
Trotta, T., Rocha, L., de Andrade, T. R., de Paiva Guimarães, M., and Dias, D. R. C. (2022). C-libras: A gesture recognition app for the brazilian sign language. In Gervasi, O., Murgante, B., Hendrix, E. M. T., Taniar, D., and Apduhan, B. O., editors, Computational Science and Its Applications – ICCSA 2022, pages 603–618, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-031-10522-7_41.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Luciano Alves Cardona Junior, Anselmo Rafael Cukla, Solon Bevilacqua, Daniel Pinheiro Bernardon, Daniel Fernando Tello Gamarra
This work is licensed under a Creative Commons Attribution 4.0 International License.
JIS is free of charge for authors and readers, and all papers published by JIS follow the Creative Commons Attribution 4.0 International (CC BY 4.0) license.
