Proposing and Evaluating a MALTU Extension for UbiComp and IoT Systems
DOI:
https://doi.org/10.5753/jbcs.2026.4471Keywords:
Ubiquitous systems, IoT, HCI, Evaluation, MALTUAbstract
Systems developed following the Ubiquitous Computing (UbiComp) and Internet of Things (IoT) paradigms have certain specific quality-of-use characteristics that traditional Human-Computer Interaction (HCI) evaluation methods may not be able to address, such as context-awareness, invisibility, attentiveness, calmness, mobility and synchronicity, since most HCI methods were traditionally created to evaluate desktop or web systems. The Model for Evaluating Interaction in Social Systems Based on the User’s Textual Language (MALTU) evaluates usability and user experience based on comments left by users on social systems (e.g. Amazon, Google Play Store, Facebook forums). This paper presents an extension in the model to address these UbiComp/IoT quality characteristics, describing the entire extension creation process and presenting the results of the first evaluations carried out using the extended MALTU. Before proposing the extension, three evaluations were performed with the original MALTU. Then, three evaluators performed two evaluations each on the same smart device: the first one was the application of the extended MALTU, and the second one was a heuristic evaluation using a specific set of heuristics for UbiComp systems. After that, a focus group was held in order to verify if the extended MALTU was capable of identifying typical problems of ubiquitous devices considering its quality characteristics. With the first version of the extension completed and published, a round of external evaluations was carried out with the extended MALTU, by authors not participating in the design of the extension. The results indicate that the extension was able to identify interaction problems in ubiquitous devices, such as Context Awareness, Synchronicity and Mobility, even with difficulties in understanding the concepts of quality characteristics of ubiquitous and IoT systems.
Downloads
References
Andrade, R. M. C., Carvalho, R. M., de Araújo, I. L., Oliveira, K. M., and Maia, M. E. F. (2017). What changes from ubiquitous computing to internet of things in interaction evaluation? page 3–21. DOI: 10.1007/978-3-319-58697-7_1.
Asghari, P., Rahmani, A. M., and Javadi, H. H. S. (2019). Internet of things applications: A systematic review. Computer Networks, 148:241-261. DOI: 10.1016/j.comnet.2018.12.008.
Barbosa, S. and Silva, B. (2010). Interação humano-computador. Elsevier Brasil. Book.
Brito, M. R. F. (2018). Explorando o modelo maltu na avaliação de usabilidade e experiência de uso em um dispositivo ubíquo. Available at:[link].
Carvalho, R. M. (2017). Dealing with conflicts between non-functional requirements of ubicomp and iot applications. In 2017 IEEE 25th International Requirements Engineering Conference (RE), pages 544-549. DOI: 10.1109/RE.2017.51.
Carvalho, R. M. (2019). Correlate & lead: process and catalog of non-functional requirements correlations in ubicomp and iot systems. Available at:[link].
Carvalho, R. M., de Castro Andrade, R. M., and de Oliveira, K. M. (2018). Aquarium-a suite of software measures for hci quality evaluation of ubiquitous mobile applications. Journal of Systems and Software, 136:101-136. DOI: 10.1016/j.jss.2017.11.022.
Carvalho, R. M., de Castro Andrade, R. M., de Oliveira, K. M., de Sousa Santos, I., and Bezerra, C. I. M. (2017). Quality characteristics and measures for human-computer interaction evaluation in ubiquitous systems. Software Quality Journal, 25:743-795. DOI: 10.1007/s11219-016-9320-z.
De Souza, C. S. (2005). The semiotic engineering of human-computer interaction. MIT press. DOI: 10.7551/mitpress/6175.001.0001.
de Souza, L. G., Barbosa, S. D. J., and Conte, T. (2015). Evaluating a molic extension for collaborative systems design. In Proceedings of the 14th Brazilian Symposium on Human Factors in Computing Systems, IHC '15, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3148456.3148471.
de Souza Filho, J. C., Brito, M. R. F., Mendonça, A. R. R., Martins, M. V., and Sampaio, A. L. (2018). Hidrate spark: Avaliando um sistema ubíquo para motivar a ingestão de Água. In Anais Estendidos do XVII Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, Porto Alegre, RS, Brasil. SBC. DOI: 10.5753/ihc.2018.4204.
de Souza Filho, J. C., Brito, M. R. F., and Sampaio, A. L. (2020). Comparing heuristic evaluation and maltu model in interaction evaluation of ubiquitous systems. In Proceedings of the 19th Brazilian Symposium on Human Factors in Computing Systems, pages 1-10. DOI: 10.1145/3424953.3426639.
de Souza Filho, J. C., Sampaio, A. L., and Feitosa, E. L. (2022). Hidden things and implicit interaction: Looking back on how invisibility evolved as a quality-in-use attribute. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, IHC '22. Association for Computing Machinery. DOI: 10.1145/3554364.3559136.
Diniz, L. D. N., Filho, J. C. D. S., and Carvalho, R. M. (2022). Can user reviews indicate usability heuristic issues? Conference on Human Factors in Computing Systems - Proceedings. DOI: 10.1145/3491101.3519659.
Freitas, L. M., da Silva, T. H. O., and Mendes, M. S. (2016). Evaluation of spotify: An evaluation textual experience using the maltu methodology. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, IHC '16, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3033701.3033752.
Karvonen, H. and Kujala, T. (2014). Designing and evaluating ubicomp characteristics of intelligent in-car systems. Advances in Ergonomics In Design, Usability & Special Populations: Part I, 16:27. DOI: 10.54941/ahfe100765.
Korhonen, H., Arrasvuori, J., and Väänänen-Vainio-Mattila, K. (2010). Let users tell the story: evaluating user experience with experience reports. In CHI'10 Extended Abstracts on Human Factors in Computing Systems, pages 4051-4056. DOI: 10.1145/1753846.1754101.
Kourouthanassis, P. E., Giaglis, G. M., and Karaiskos, D. C. (2008). Delineating the degree of'pervasiveness' in pervasive information systems: an assessment framework and design implications. In 2008 Panhellenic Conference on Informatics, pages 251-255. IEEE. DOI: 10.1109/pci.2008.28.
Melo, B., Andrade, R., and Darin, T. (2022). Longitudinal user experience studies in the iot domain: a brief panorama and challenges to overcome. Association for Computing Machinery. DOI: 10.1145/3554364.3559135.
Mendes, M. S. (2015). Maltu-um modelo para avaliação da interação em sistemas sociais a partir da linguagem textual do usuário. Available at:[link].
Mendes, M. S. and Furtado, E. (2018). An experience of textual evaluation using the maltu methodology. In Social Computing and Social Media. Technologies and Analytics: 10th International Conference, SCSM 2018, Held as Part of HCI International 2018, Las Vegas, NV, USA, July 15-20, 2018, Proceedings, Part II 10, pages 236-246. Springer. DOI: 10.1007/978-3-319-91485-5_18.
Metsis, V., Le, Z., Lei, Y., and Makedon, F. (2008). Towards an evaluation framework for assistive environments. In Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments, pages 1-8. DOI: 10.1145/1389586.1389601.
Moreira, F. and Coutinho, E. (2023). Evaluating the user experience of type 1 diabetes control applications. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, pages 1-11. DOI: 10.1145/3638067.3638134.
Nielsen, J. and Molich, R. (1990). rochas. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '90, page 249–256, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/97243.97281.
Ntoa, S., Margetis, G., Antona, M., and Stephanidis, C. (2021). User experience evaluation in intelligent environments: A comprehensive framework. Technologies, 9(2):41. DOI: 10.3390/technologies9020041.
Oliveira, E. R., Luz, L. C., and Prates, R. O. (2008). Aplicação semi-estruturada do método de inspeção semiótica: estudo de caso para o domínio educacional. In Proceedings of the VIII Brazilian Symposium on Human Factors in Computing Systems, pages 50-59. Available at:[link].
Olsson, T. and Salo, M. (2012). Narratives of satisfying and unsatisfying experiences of current mobile augmented reality applications. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 2779-2788. DOI: 10.1145/2207676.2208677.
Quiñones, D. and Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces, 53:89-122. DOI: 10.1016/j.csi.2017.03.009.
Rocha, L. C. (2017). Hubis: heurísticas de usabilidade para avaliar sistemas ubíquos. Available at:[link].
Rusu, C., Roncagliolo, S., Rusu, V., and Collazos, C. (2011). A methodology to establish usability heuristics.(2011). Available at:[link].
Santiago, M. T. and Marques, A. B. (2022). Are user reviews useful for identifying accessibility issues that autistic users face? an exploratory study. ACM International Conference Proceeding Series. DOI: 10.1145/3554364.3559114.
Santiago, M. T. and Marques, A. B. (2023). Exploring user reviews to identify accessibility problems in applications for autistic users. Journal on Interactive Systems, 14:317-330. DOI: 10.5753/jis.2023.3238.
Santos, P. S. H. D., Oliveira, A. D. A., Jesus, T. B. N. D., Aljedaani, W., and Eler, M. M. (2024). Evolution may come with a price: analyzing user reviews to understand the impact of updates on mobile apps accessibility. Association for Computing Machinery. DOI: 10.1145/3638067.3638081.
Santos, R. M. (2014). Características e medidas de software para avaliação da qualidade da interação humano-computador em sistemas ubíquos. Available at:[link].
Siewerdt, F. L., Carvalho, R. M., and Andrade, R. M. C. (2016). Recommendations for usability testing in ubiquitous applications. In Proceedings of the 15th Brazilian Symposium on Human Factors in Computing Systems, IHC '16, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3033701.3033749.
Tuch, A. N., Trusell, R., and Hornbæk, K. (2013). Analyzing users' narratives to understand experience with interactive products. In Proceedings of the SIGCHI conference on human factors in computing systems, pages 2079-2088. DOI: 10.1145/2470654.2481285.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Carlos E. Almeida Feitosa, Francisco Das C. Moreira Catunda Neto, Ingrid Teixeira Monteiro
This work is licensed under a Creative Commons Attribution 4.0 International License.
