Investigating usability pitfalls in Brazilian and Foreign governmental chatbots

Mateus de Souza Monteiro; Gabriely Oliveira da Silva Batista; Luciana Cardoso de Castro Salgado

doi:10.5753/jis.2023.3104

Authors

Mateus de Souza Monteiro Universidade Federal Fluminense https://orcid.org/0000-0002-4502-4319
Gabriely Oliveira da Silva Batista Universidade Federal Fluminense https://orcid.org/0000-0002-2457-4231
Luciana Cardoso de Castro Salgado Universidade Federal Fluminense https://orcid.org/0000-0003-1207-6021

DOI:

https://doi.org/10.5753/jis.2023.3104

Keywords:

Chatbot, Electronic Government, Usability, Service delivery

Abstract

The usability of chatbots is an essential aspect of e-government initiatives that aim to improve citizens’ access to public services. Despite significant efforts to enhance e-gov usability in Brazil, there is still a lack of research examining usability pitfalls in human-chatbot interaction. Therefore, this study aims to fill this research gap by presenting an analysis and comparison of ten electronic government chatbots, five national (Brazilian) and five foreign (1 Argentine, 1 Portuguese, 1 American, 1 British, and 1 Singaporean), using an adapted version of Heuristics for chatbots. This study examines the design issues and opportunities that can affect the usability and adoption of e-government chatbots in Brazil. These include enhancing the navigation mechanism for dialogues, improving response times to requests, clearly indicating the end of the dialogue, providing guidance on inputting utterances correctly and implementing standardized utterances for the same actions.

Downloads

Download data is not yet available.

References

Borsci, S., Malizia, A., Schmettow, M., Van Der Velde, F., Tariverdiyeva, G., Balaji, D., and Chamberlain, A. (2022). The chatbot usability scale: the design and pilot of a usability scale for interaction with AI-based conversational agents. Personal and Ubiquitous Computing, 26:95–119.

Brandtzaeg, P. B. and Følstad, A. (2018). Chatbots: changing user needs and motivations. Interactions, 25(5):38–43.

Brasil (2015). Lei nº 13.146, de 6 de julho de 2015. Institui a lei brasileira de inclusão da pessoa com deficiência (estatuto da pessoa com deficiência). Diário Oficial [da] República Federativa do Brasil.

Carvalho, V., Cagnin, M., and Paiva, D. (2017). Avaliação de acessibilidade de web sites de governos estaduais do Brasil. In Anais do XIII Simpósio Brasileiro de Sistemas de Informação, pages 116–123. SBC.

Chaves, A. P. and Gerosa, M. A. (2020). How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design. International Journal of Human–Computer Interaction, pages 1–30.

da Silva Batista, G. O., de Souza Monteiro, M., and de Castro Salgado, L. C. (2022). Investigando chatbots governamentais: um panorama sobre a usabilidade dentro e fora do Brasil. In Anais do X Workshop de Computação Aplicada em Governo Eletrônico, pages 73–84. SBC.

de Moura Regly, T. R. and Souza, R. F. (2022). Disponibilização de dados e democratização do acesso à informação pública: uma análise do portal da transparência do governo federal. Encontros Bibli: revista eletrônica de biblioteconomia e ciência da informação, 27.

de Oliveira, F. E. and Freire, A. (2020). Fatores que influenciam a aceitação e intenção de uso de governo eletrônico por pessoas com deficiência visual: um estudo sobre o site da receita federal. In Anais Estendidos do XVI Simpósio Brasileiro de Sistemas de Informação, pages 125–125. SBC.

Deibel, D. and Evanhoe, R. (2021). Conversations with Things: UX design for Chat and Voice. Rosenfeld Media.

Domingues, M. C. M., Melo, K. M., Miranda, A. F., de Freitas Cayres, A. Z., and Elias, R. (2021). Programa autoestima: Uma ferramenta web pública e amigável que integra formação de profissionais da saúde e acolhimento psicossocial da população. In Anais do IX Workshop de Computação Aplicada em Governo Eletrônico, pages 251–258. SBC.

Guimarães, C., Silva, L. D. R., Antunes, D. R., and Formighieri, R. (2014). E-governance transparency in Brazil – the lack of usability is detrimental to citizenship. In International Conference on Human Interface and the Management of Information, pages 590–601. Springer.

Höhn, S. and Bongard-Blanchy, K. (2021). Heuristic evaluation of covid-19 chatbots. In International Workshop on Chatbot Research and Design, pages 131–144. Springer.

Huai, J. (2011). Quality evaluation of e-government public service. In 2011 International Conference on Management and Service Science, pages 1–4. IEEE.

Jain, M., Kota, R., Kumar, P., and Patel, S. N. (2018). Convey: Exploring the use of a context view for chatbots. In Proceedings of the 2018 chi conference on human factors in computing systems, pages 1–6.

Langevin, R., Lordon, R. J., Avrahami, T., Cowan, B. R., Hirsch, T., and Hsieh, G. (2021). Heuristic evaluation of conversational agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15.

Luk, T. T., Lui, J. H. T., and Wang, M. P. (2022). Efficacy, usability, and acceptability of a chatbot for promoting covid-19 vaccination in unvaccinated or booster-hesitant young adults: Pre-post pilot study. Journal of Medical Internet Research, 24(10):e39063.

Monteiro, M., Vasconcelos, L., Viterbo, J., Salgado, L., and Bernardini, F. (2021). Assessing the quality of local e-government service through citizen-sourcing applications. In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 1178–1183. IEEE.

Mordor (2022). CHATBOT MARKET - GROWTH, TRENDS, COVID-19 IMPACT, AND FORECASTS (2022 - 2027). Mordor Intelligence. [link].

Nielsen, J. (1992). Finding usability problems through heuristic evaluation. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 373–380.

Nielsen, J. and Molich, R. (1990). Heuristic evaluation of user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 249–256.

Pedrosa, G. V., Judice, A., Judice, M., Araújo, L., Fleury, F., and Figueiredo, R. (2022). Applying user-centered design on digital transformation of public services: A case study in Brazil. In DG. O 2022: The 23rd Annual International Conference on Digital Government Research, pages 372–379.

Prates, R. O. and Barbosa, S. D. J. (2007). Introdução à teoria e prática da interação humano computador fundamentada na engenharia semiótica. Atualizações em informática, pages 263–326.

Quispe, F. E. M. and Eler, M. M. (2018). Recomendações de acessibilidade para aplicativos móveis: uma contribuição para os padrões do governo digital brasileiro. In Anais do XIV Simpósio Brasileiro de Sistemas de Informação, pages 528–535. SBC.

Radziwill, N. M. and Benton, M. C. (2017). Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579.

Ren, R., Castro, J. W., Acuña, S. T., and de Lara, J. (2019). Evaluation techniques for chatbot usability: A systematic mapping study. International Journal of Software Engineering and Knowledge Engineering, 29(11n12):1673–1702.

Saenz, J., Burgess, W., Gustitis, E., Mena, A., and Sasangohar, F. (2017). The usability analysis of chatbot technologies for internal personnel communications. In IIE Annual Conference. Proceedings, pages 1357–1362. Institute of Industrial and Systems Engineers (IISE).

Santos, S. K. S., Mendes, N. F. O., and da Silva, Q. P. (2021). Propostas para o atendimento às pessoas surdas em serviços de governo eletrônico. In Anais do IX Workshop de Computação Aplicada em Governo Eletrônico, pages 1–12. SBC.

Teo, T. S., Srivastava, S. C., and Jiang, L. (2008). Trust and electronic government success: An empirical study. Journal of management information systems, 25(3):99–132.

Tielman, M. L., Neerincx, M. A., Bidarra, R., Kybartas, B., and Brinkman, W.-P. (2017). A therapy system for post-traumatic stress disorder using a virtual agent and virtual storytelling to reconstruct traumatic memories. Journal of medical systems, 41(8):1–10.

Wang, Y., Zhang, N., and Zhao, X. (2022). Understanding the determinants in the different government AI adoption stages: Evidence of local government chatbots in China. Social Science Computer Review, 40(2):534–554.

Welchen, V., Matte, J., Giacomello, C. P., Dalle Molle, F., and Camargo, M. E. (2022). Usability perception of the health information systems in Brazil: The view of hospital health professionals on the electronic health record. RAUSP Management Journal, 57:264–279.

Zaidi, S. F. H. and Qteishat, M. K. (2012). Assessing e-government service delivery (government to citizen). International journal of e-business and e-government studies, 4(1):45–54.