Semiotic Engineering Theory for Human-Computer Integration: An Applicability and Usefulness Evaluation

Glívia Angélica Rodrigues Barbosa; Raquel Oliveira Prates

doi:10.5753/jbcs.2026.5620

Authors

Glívia Angélica Rodrigues Barbosa Federal University of Minas Gerais & Federal Center for Technological Education of Minas Gerais https://orcid.org/0000-0002-1872-5291
Raquel Oliveira Prates Federal University of Minas Gerais https://orcid.org/0000-0002-7128-4974

DOI:

https://doi.org/10.5753/jbcs.2026.5620

Keywords:

Human-Computer Integration, HInt, Semiotic Engineering, Ontology, Evaluation

Abstract

The relationship between users and autonomous technologies is evolving towards integration (in the sense of partnership), transcending the stimulus-response interaction between these two agents. To follow this evolution, Human-Computer Interaction (HCI) researchers have defined and characterized a new interaction paradigm, Human-Computer Integration (HInt), which extends the focus of the HCI area to cover this new relationship of partnership between humans and autonomous technologies. As HInt is an emerging paradigm, the concepts and ontology of Semiotic Engineering Theory have been extended to address HInt as an extension of the traditional HCI interaction. Thus, this paper aims to evaluate and discuss the applicability and usefulness of the extension of Semiotic Engineering to define, explore, and explain the phenomena involved in HInt. Our findings provide useful insights and reflections on the benefits and limits of Semiotic Engineering for HInt to support the study, design, and evaluation of the partnership between humans and autonomous technologies.

Downloads

Download data is not yet available.

References

Alharbi, M. and Huang, S. (2020). A survey of incorporating affective computing for human-system co-adaptation. In Proceedings of the 2020 The 2nd World Symposium on Software Engineering, WSSE 2020, page 72–79, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3425329.3425343.

Andres, J., de Hoog, J., and Mueller, F. F. (2018). "i had super-powers when ebike riding" towards understanding the design of integrated exertion. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY '18, page 19–31, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3242671.3242688.

Andres, J., Kari, T., von Kaenel, J., and Mueller, F. F. (2019). Co-riding with my ebike to get green lights. In Proceedings of the 2019 on Designing Interactive Systems Conference, DIS '19, page 1251–1263, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3322276.3322307.

Andres, J., schraefel, m., Semertzidis, N., Dwivedi, B., Kulwe, Y. C., von Kaenel, J., and Mueller, F. F. (2020). Introducing peripheral awareness as a neurological state for human-computer integration. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, page 1–13, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3313831.3376128.

Andres, J., Semertzidis, N., Li, Z., Wang, Y., and Floyd Mueller, F. (2023). Integrated exertion—understanding the design of human–computer integration in an exertion context. ACM Trans. Comput.-Hum. Interact., 29(6). DOI: 10.1145/3528352.

Barbosa, G. A. R., da S. Fernandes, U., Santos, N. S., and Prates, R. O. (2023). Human-computer integration as an extension of interaction: Understanding its state-of-the-art and the next challenges. International Journal of Human–Computer Interaction, 40(11):2761–2780. DOI: 10.1080/10447318.2023.2177797.

Barbosa, G. A. R. and Prates, R. O. (2022). Extending the ontology, metacommunication and communicability of semiotic engineering to the emerging paradigm of human-computer integration (hint). In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, IHC'22, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3554364.3559109.

Barbosa, G. A. R. and Prates, R. O. (2023). Evoluir é preciso: Movendo da interação para a integração humano-computador. In Anais Estendidos do XXII Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais - Minicurso, pages 1-8, Porto Alegre, RS, Brasil. SBC. DOI: 10.5753/ihc_estendido.2023.232984.

Barbosa, G. A. R. and Prates, R. O. (2024a). Introdução ao paradigma da integração humano-computador à luz da engenharia semiótica. In Livro de Minicursos do XXII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, pages 1-23, Porto Alegre, RS, Brasil. SBC. DOI: 10.5753/sbc.15504.9.1.

Barbosa, G. A. R. and Prates, R. O. (2024b). Reflections on the applicability of the semiotic engineering ontology extended to the hint paradigm. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, IHC '24, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3702038.3702104.

Barbosa, G. A. R., Prates, R. O., da S. Fernandes, U., and Santos, N. S. (2021a). Extending interaction to human-computer integration: What do we already know and what do we need to explore? In Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems, IHC'21, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3472301.3484351.

Barbosa, G. D. J., Nunes, J. L., De Souza, C. S., and Barbosa, S. D. J. (2024). Investigating the extended metacommunication template: How a semiotic tool may encourage reflective ethical practice in the development of machine learning systems. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, IHC '23, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3638067.3638070.

Barbosa, S. D. J., Barbosa, G. D. J., Souza, C. S. d., and Leitão, C. F. (2021b). A semiotics-based epistemic tool to reason about ethical issues in digital technology design and development. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 363–374, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3442188.3445900.

Barbosa, S. D. J., Silva, B. S. d., Silveira, M. S., Gasparini, I., Darin, T., and Barbosa, G. D. J. (2021c). Interação humano-computador e experiência do usuario. Auto publicação. Book.

Boldu, R., Dancu, A., Matthies, D. J., Buddhika, T., Siriwardhana, S., and Nanayakkara, S. (2018). Fingerreader2.0: Designing and evaluating a wearable finger-worn camera to assist people with visual impairments while shopping. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 2(3). DOI: 10.1145/3264904.

Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77-101. DOI: 10.1191/1478088706qp063oa.

Britton, L. M. and Semaan, B. (2017). Manifesting the cyborg through techno-body modification: From human-computer interaction to integration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, page 2499–2510, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3025453.3025629.

Carroll, J. M. (2003). HCI models, theories, and frameworks: Toward a multidisciplinary science. Elsevier. Book.

Chagas, B. A. (2020). Metacommunication and appropriation in the design of the interactive internet of things. DOI: 10.17771/pucrio.acad.50190.

Chagas, B. A., Redmiles, D. F., and de Souza, C. S. (2018). Observed appropriation of iot technology: A semiotic account. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, IHC 2018, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3274192.3274225.

Chagas, B. A., Redmiles, D. F., and de Souza, C. S. (2019). Signs of appropriation: A semiotic account of breakdowns with iot technology. Journal on Interactive Systems, 10(2):3-19. DOI: 10.5753/jis.2019.550.

Cornelio, P., Haggard, P., Hornbaek, K., Georgiou, O., Bergström, J., Subramanian, S., and Obrist, M. (2022). The sense of agency in emerging technologies for human-computer integration: A review. Frontiers in Neuroscience, 16. DOI: 10.3389/fnins.2022.949138.

da Silva, B. S. and Barbosa, S. D. J. (2007). Designing human-computer interaction with MoLIC diagrams-a practical guide. (12/07):12. Available at:[link].

Danry, V., Pataranutaporn, P., Mueller, F., Maes, P., and Leigh, S.-w. (2022). On eliciting a sense of self when integrating with computers. In Augmented Humans 2022, AHs 2022, page 68–81, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3519391.3519414.

de Souza, C. S. (2005). The semiotic engineering of human-computer interaction. MIT press. DOI: 10.7551/mitpress/6175.001.0001.

de Souza, C. S. and Leitão, C. F. (2009). Semiotic engineering methods for scientific research in hci. Synthesis Lectures on Human-Centered Informatics, 2(1):1-122. DOI: 10.1007/978-3-031-02185-5.

de Souza, C. S., Leitão, C. F., Prates, R. O., Bim, S. A., and da Silva, E. J. (2010). Can inspection methods generate valid new knowledge in hci? the case of semiotic inspection. International Journal of Human-Computer Studies, 68(1-2):22-40. DOI: 10.1016/j.ijhcs.2009.08.006.

Dengel, A., Devillers, L., and Schaal, L. M. (2021). Augmented human and human-machine co-evolution - efficiency and ethics. Reflections on Artificial Intelligence for Humanity, pages 203-227. DOI: 10.1007/978-3-030-69128-8_13.

Farooq, U. and Grudin, J. (2016). Human-computer integration. Interactions, 23(6):26–32. DOI: 10.1145/3001896.

Gil, M., Albert, M., Fons, J., and Pelechano, V. (2019). Designing human-in-the-loop autonomous cyber-physical systems. International Journal of Human-Computer Studies, 130:21-39. DOI: 10.1016/j.ijhcs.2019.04.006.

Gil, M., Albert, M., Fons, J., and Pelechano, V. (2020). Engineering human-in-the-loop interactions in cyber-physical systems. Information and Software Technology, 126:106349. DOI: 10.1016/j.infsof.2020.106349.

Ho, J. C. F. (2018). Computer as partner: A critique perspective of interaction design for social sustainability. In Proceedings of the Sixth International Symposium of Chinese CHI, ChineseCHI '18, page 95–99, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3202667.3202680.

Leitão, C. F. and Prates, R. O. (2017). A aplicação de métodos qualitativos em computação. In Jornadas de Atualização em Informática 2017, pages 43-90. DOI: 10.5753/sbc.1174.1.2.

Maués, R. d. A. and Barbosa, S. D. J. (2013). Cross-communicability: Evaluating the meta-communication of cross-platform applications. In Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., and Winckler, M., editors, Human-Computer Interaction - INTERACT 2013, pages 241-258, Berlin, Heidelberg. Springer Berlin Heidelberg. DOI: 10.1007/978-3-642-40477-1_15.

Maués, R. d. A. and Barbosa, S. D. J. (2014). Reflections on the cross-platform semiotic inspection method. In Kurosu, M., editor, Human-Computer Interaction. Theories, Methods, and Tools, pages 533-544, Cham. Springer International Publishing. DOI: 10.1007/978-3-319-07233-3_49.

Mueller, F., Maes, P., and Grudin, J. (2019). Human-Computer Integration (Dagstuhl Seminar 18322). Dagstuhl Reports, 8(8):18-47. DOI: 10.4230/DagRep.8.8.18.

Mueller, F. F., Lopes, P., Andres, J., Byrne, R., Semertzidis, N., Li, Z., Knibbe, J., and Greuter, S. (2021). Towards understanding the design of bodily integration. International Journal of Human-Computer Studies, 152:102643. DOI: 10.1016/j.ijhcs.2021.102643.

Mueller, F. F., Lopes, P., Strohmeier, P., Ju, W., Seim, C., Weigel, M., Nanayakkara, S., Obrist, M., Li, Z., Delfa, J., Nishida, J., Gerber, E. M., Svanaes, D., Grudin, J., Greuter, S., Kunze, K., Erickson, T., Greenspan, S., Inami, M., Marshall, J., Reiterer, H., Wolf, K., Meyer, J., Schiphorst, T., Wang, D., and Maes, P. (2020). Next steps for human-computer integration. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, page 1–15, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3313831.3376242.

Mueller, F. F., Semertzidis, N., Andres, J., Weigel, M., Nanayakkara, S., Patibanda, R., Li, Z., Strohmeier, P., Knibbe, J., Greuter, S., Obrist, M., Maes, P., Wang, D., Wolf, K., Gerber, L., Marshall, J., Kunze, K., Grudin, J., Reiterer, H., and Byrne, R. (2022). Human–computer integration: Towards integrating the human body with the computational machine. Foundations and Trends® in Human-Computer Interaction, 16(1):1-64. DOI: 10.1561/1100000086.

Nicolaci-da Costa, A. M., Leitão, C. F., and Romão-Dias, D. (2004). Como conhecer usuários através do método de explicitação do discurso subjacente (meds). VI Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, IHC, pages 47-56. Available at:[link].

Niess, J. and Woźniak, P. W. (2020). Embracing companion technologies. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, NordiCHI '20, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3419249.3420134.

Oliveira, E. R., Luz, L. C. S., and Prates, R. O. (2008). Aplicação semi-estruturada do método de inspeção semiótica: Estudo de caso para o domínio educacional. In Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems, IHC '08, page 50–59, BRA. Sociedade Brasileira de Computação. Available at:[link].

Peirce, C. S. P. and Peirce, C. S. (1992). The essential Peirce, volume 1: Selected philosophical writings?(1867-1893), volume 1. Indiana University Press. Book.

Prates, R. O. (2017). An overview of semiotic engineering epistemic tools for the design of collaborative systems. Conversations Around Semiotic Engineering, pages 81-99. DOI: 10.1007/978-3-319-56291-9_9.

Prates, R. O. and Barbosa, S. D. J. (2007). Introdução à teoria e prática da interação humano computador fundamentada na engenharia semiótica. Jornadas de Atualizações em informática, pages 263-326. Available at:[link].

Prates, R. O. and de Souza, C. S. (1998). Towards a semiotic environment for supporting the development of multi-user interfaces. In CRIWG, volume 98, page 15pp. Available at:[link].

Prates, R. O., de Souza, C. S., and Barbosa, S. D. (2000). Methods and tools: a method for evaluating the communicability of user interfaces. interactions, 7(1):31-38. DOI: 10.1145/328595.328608.

Stephanidis, C. C., Salvendy, G., of the Group Margherita Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., Guo, Y., Harris, D., Ioannou, A., ah (Kate) Jeong, K., Konomi, S., Krömker, H., Kurosu, M., Lewis, J. R., Marcus, A., Meiselwitz, G., Moallem, A., Mori, H., Nah, F. F.-H., Ntoa, S., Rau, P.-L. P., Schmorrow, D., Siau, K., Streitz, N., Wang, W., Yamamoto, S., Zaphiris, P., and Zhou, J. (2019). Seven hci grand challenges. International Journal of Human–Computer Interaction, 35(14):1229-1269. DOI: 10.1080/10447318.2019.1619259.

Zaina, L., Prates, R. O., Delabrida Silva, S. E., Choma, J., Valentim, N. M. C., Frigo, L. B., and Bicho, A. d. L. (2024). Grandihc-br 2025-2035 - gc7: Interaction with emerging technologies: An ecosystem integrating humans technologies and contexts. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, IHC '24, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3702038.3702060.