A Systematic Mapping Study about Technologies for Hedonic Aspects Evaluation in Text-based Chatbots

Pamella A. de L. Mariano; Ana Carolina R. de Souza; Guilherme C. Guerino; Ana Paula Chaves; Natasha M. C. Valentim

doi:10.5753/jis.2024.4350

Authors

Pamella A. de L. Mariano Federal University of Paraná https://orcid.org/0009-0006-9919-3448
Ana Carolina R. de Souza Federal University of Paraná https://orcid.org/0000-0002-7932-2592
Guilherme C. Guerino State University of Paraná https://orcid.org/0000-0002-4979-5831
Ana Paula Chaves Northern Arizona University https://orcid.org/0000-0002-2307-3099
Natasha M. C. Valentim Federal University of Paraná https://orcid.org/0000-0002-6027-3452

DOI:

https://doi.org/10.5753/jis.2024.4350

Keywords:

Chatbots, User Experience, Systematic Mapping Study

Abstract

Many studies present and evaluate daily-use systems ranging from information to conversational systems. Chatbots, either text-based or voice-based, have attracted the attention of researchers. In particular, User eXperience (UX) has been pointed out as one of the chatbot's leading aspects of evaluation involving pragmatic and hedonic aspects. Pragmatic aspects deal with the usability and efficiency of the system, while hedonic aspects consider aspects related to the originality, innovation, beauty of the system, and the user's psychological well-being. Even with existing research on usability evaluation and human-computer interaction within conversational systems, there is a clear shortfall in studies specifically addressing the hedonic aspects of user experience in chatbots. Therefore, this paper presents a Systematic Mapping Study that investigates various UX evaluation technologies (questionnaires, methods, techniques, and models, among others), focusing on the hedonic aspect of chatbots. We focused on studies with chatbots that are activated by text, although they may be able to display click interactions, videos, and images in addition to the text modality. We discovered 69 technologies to evaluate hedonic aspects of UX in chatbots, and the most frequent aspect found is the General UX . Our study provides relevant data on the research topic, addressing the specific characteristics of human-chatbot interaction, such as identity and social interaction. Moreover, we highlight gaps in the hedonic aspect evaluation in chatbots, such as a few works investigating the assessment of user emotional state.

Downloads

Download data is not yet available.

References

Al-Emran, M., AlQudah, A. A., Abbasi, G. A., Al-Sharafi, M. A., and Iranmanesh, M. (2024). Determinants of using ai-based chatbots for knowledge sharing: Evidence from pls-sem and fuzzy sets (fsqca). IEEE Transactions on Engineering Management, 71:4985–4999. DOI: https://doi.org/10.1109/TEM.2023.3237789.

Alazraki, L., Ghachem, A., Polydorou, N., Khosmood, F., and Edalat, A. (2021). An empathetic ai coach for self-attachment therapy. In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI), pages 78–87. DOI: https://doi.org/10.1109/CogMI52975.2021.00019.

Altman, D. G. (1990). Practical Statistics for Medical Research. Chapman and Hall/CRC. DOI: http://dx.doi.org/10.1201/9780429258589.

Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., and Chatzigeorgiou, A. (2019). Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology, 106:201–230. DOI: http://dx.doi.org/10.1016/j.infsof.2018.10.006.

Ashktorab, Z., Jain, M., Liao, Q. V., and Weisz, J. D. (2019). Resilient chatbots: Repair strategy preferences for conversational breakdowns. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19. ACM. DOI: http://dx.doi.org/10.1145/3290605.3300484.

Bae Brandtzæg, P. B., Skjuve, M., Kristoffer Dysthe, K. K., and Følstad, A. (2021). When the social becomes nonhuman: Young people’s perception of social support in chatbots. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21. ACM. DOI: http://dx.doi.org/10.1145/3411764.3445318.

Basili, V. R. and Rombach, H. D. (1988). Towards a comprehensive framework for reuse: A reuse-enabling software evolution environment. In NASA, Goddard Space Flight Center, Proceedings of the Thirteenth Annual Software Engineering Workshop, number UMIACS-TR-88-92.

Bawa, A., Khadpe, P., Joshi, P., Bali, K., and Choudhury, M. (2020). Do multilingual users prefer chat-bots that codemix? let’s nudge and find out! Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–23. DOI: http://dx.doi.org/10.1145/3392846.

Benke, I., Knierim, M. T., and Maedche, A. (2020). Chatbot-based emotion management for distributed teams: A participatory design study. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW2):1–30. DOI: https://doi.org/10.1145/3415189.

Brooke, J. et al. (1996). Sus-a quick and dirty usability scale. Usability evaluation in industry, 189(194):4–7. DOI: http://dx.doi.org/10.1201/9781498710411-35.

Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165. DOI: https://doi.org/10.48550/arXiv.2005.14165.

Cai, W., Jin, Y., and Chen, L. (2022). Impacts of personal characteristics on user trust in conversational recommender systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491102.3517471.

Cai, W., Jin, Y., Zhao, X., and Chen, L. (2023). “listen to music, listen to yourself”: Design of a conversational agent to support self-awareness while listening to music. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3544548.3581427.

Campos, T. P. d., Damasceno, E. F., and Valentim, N. M. C. (2022). Proposal and evaluation of a collaborative is to support systematic reviews and mapping studies. In XVIII Brazilian Symposium on Information Systems, pages 1–8. DOI: https://doi.org/10.1145/3535511.3535531.

Candello, H. and Pinhanez, C. (2016). Designing conversational interfaces. Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais.

Ceha, J., Lee, K. J., Nilsen, E., Goh, J., and Law, E. (2021). Can a humorous conversational agent enhance learning experience and outcomes? In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14. DOI: https://doi.org/10.1145/3411764.3445068.

Chen, E. (2022). The effect of multiple replies for natural language generation chatbots. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491101.3516800.

Chen, J., Chen, C., B. Walther, J., and Sundar, S. S. (2021). Do you feel special when an ai doctor remembers you? individuation effects of ai vs. human doctors on user experience. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. DOI: https://doi.org/10.1145/3411763.3451735.

Chitturi, R., Raghunathan, R., and Mahajan, V. (2008). Delight by design: The role of hedonic versus utilitarian benefits. Journal of marketing, 72(3):48–63. DOI: https://doi.org/10.1509/jmkg.72.3.48.

Dahiya, M. (2017). A tool of conversation: Chatbot. International Journal of Computer Sciences and Engineering, 5(5):158–161.

Daniel, R., Purwarianti, A., and Lestari, D. P. (2022). Interaction design of indonesian anti hoax chatbot using user centered design. In 2022 Seventh International Conference on Informatics and Computing (ICIC), pages 1–6. DOI: https://doi.org/10.1109/ICIC56845.2022.10007024.

Day, M.-Y. and Shaw, S.-R. (2021). Ai customer service system with pre-trained language and response ranking models for university admissions. In 2021 IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI), pages 395–401. DOI: https://doi.org/10.1109/IRI51335.2021.00062.

De Nieva, J. O., Joaquin, J. A., Tan, C. B., Marc Te, R. K., and Ong, E. (2020). Investigating students’ use of a mental health chatbot to alleviate academic stress. In 6th International ACM InCooperation HCI and UX Conference, pages 1–10. DOI: https://doi.org/10.1145/3431656.3431657.

De Souza, A. C. R., Mariano, P. A. D. L., Guerino, G. C., Chaves, A. P., and Valentim, N. M. C. (2024). Technologies for hedonic aspects evaluation in text-based chatbots: A systematic mapping study. In Proceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems, IHC ’23, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3638067.3638089.

Denecke, K., Vaaheesan, S., and Arulnathan, A. (2020). A mental health chatbot for regulating emotions (sermo)-concept and usability test. IEEE Transactions on Emerging Topics in Computing, 9(3):1170–1182. DOI: https://doi.org/10.1109/tetc.2020.2974478.

Dopler, F. and Göschlberger, B. (2022). Assessing expectations and potential of domain independent corporate learning chatbots. In 2022 20th International Conference on Emerging eLearning Technologies and Applications (ICETA), pages 135–140. DOI: https://doi.org/10.1109/ICETA57911.2022.9974903.

El Hefny, W., El Bolock, A., Herbert, C., and Abdennadher, S. (2021). Chase away the virus: A character-based chatbot for covid-19. In 2021 IEEE 9th International Conference on Serious Games and Applications for Health(SeGAH), pages 1–8. DOI: https://doi.org/10.1109/SEGAH52098.2021.9551895.

El Kamali, M., Angelini, L., Lalanne, D., Abou Khaled, O., and Mugellini, E. (2020). Multimodal conversational agent for older adults’ behavioral change. In Companion Publication of the 2020 International Conference on Multimodal Interaction, pages 270–274. DOI: https://doi.org/10.1145/3395035.3425315.

Elsholz, E., Chamberlain, J., and Kruschwitz, U. (2019). Exploring language style in chatbots to increase perceived product value and user engagement. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pages 301–305. DOI: https://doi.org/10.1145/3295750.3298956.

Essop, L., Singh, A., and Wing, J. (2023). Developing a comprehensive evaluation questionnaire for university faq administration chatbots. In 2023 Conference on Information Communications Technology and Society (ICTAS), pages 1–7. DOI: https://doi.org/10.1109/ICTAS56421.2023.10082753.

Fadhil, A., Schiavo, G., Wang, Y., and Yilma, B. A. (2018). The effect of emojis when interacting with conversational interface assisted health coaching system. In Proceedings of the 12th EAI international conference on pervasive computing technologies for healthcare, pages 378–383. DOI: https://doi.org/10.1145/3240925.3240965.

Fahn, V. and Riener, A. (2021). Time to get conversational: Assessment of the potential of conversational user interfaces for mobile banking. In Proceedings of Mensch Und Computer 2021, MuC ’21, page 34–43, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3473856.3473872.

Fiore, D., Baldauf, M., and Thiel, C. (2019). ” forgot your password again?” acceptance and user experience of a chatbot for in-company it support. In Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, pages 1–11. DOI: https://doi.org/10.1145/3365610.3365617.

Flandrin, P., Hellemans, C., van der Linden, J., and Van de Leemput, C. (2022). Smart technologies in hospitality: effects on activity, work design and employment. a case study about chatbot usage. In Proceedings of the 17th “Ergonomie et Informatique Avancée” Conference, ErgoIA ’21, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3486812.3486838.

Flohr, L. A., Kalinke, S., Krüger, A., and Wallach, D. P. (2021). Chat or tap?–comparing chatbots with ‘classic’graphical user interfaces for mobile interaction with autonomous mobility-on-demand systems. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction, pages 1–13. DOI: https://doi.org/10.1145/3447526.3472036.

Følstad, A., Araujo, T., Law, E. L.-C., Brandtzaeg, P. B., Papadopoulos, S., Reis, L., Baez, M., Laban, G., McAllister, P., Ischen, C., Wald, R., Catania, F., von Wolff, R. M., Hobert, S., and Luger, E. (2021). Future directions for chatbot research: an interdisciplinary research agenda. Computing, 103:2915–2942. REGULAR PAPER. DOI: https://doi.org/10.1007/s00607-021-01016-7.

Gambetta, Z. A., Dessi Puji, L., and Ginar Santika, N. (2021). Calla beauty assistant: Beauty advisory chatbot. In 2021 8th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA), pages 1–6. DOI: https://doi.org/10.1109/ICAICTA53211.2021.9640281.

Guerino, G. C. and Valentim, N. M. C. (2020). Usability and user experience evaluation of conversational systems: A systematic mapping study. In Proceedings of the XXXIV Brazilian Symposiumon Software Engineering, pages 427–436. DOI: https://doi.org/10.1145/3422392.3422421.

Hassenzahl, M. (2004). The interplay of beauty, goodness, and usability in interactive products. Human–Computer Interaction, 19(4):319–349. DOI: https://doi.org/10.1207/s15327051hci1904_2.

Hassenzahl, M., Platz, A., Burmester, M., and Lehner, K. (2000). Hedonic and ergonomic quality aspects determine a software’s appeal. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 201–208. DOI: https://doi.org/10.1145/332040.332432.

Höhn, S. and Bongard-Blanchy, K. (2021). Heuristic evaluation of covid-19 chatbots. In Chatbot Research and Design: 4th International Workshop, CONVERSATIONS 2020, Virtual Event, November 23–24, 2020, Revised Selected Papers, pages 131–144. Springer. DOI: https://doi.org/10.1007/978-3-030-68288-0_9.

Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., and Mctear, M. (2019). Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? In Proceedings of the 31st European Conference on Cognitive Ergonomics, ECCE ’19, page 207–214, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3335082.3335094.

Iniesto, F., Coughlan, T., Lister, K., Devine, P., Freear, N., Greenwood, R., Holmes, W., Kenny, I., McLeod, K., and Tudor, R. (2023). Creating ‘a simple conversation’: Designing a conversational user interface to improve the experience of accessing support for study. ACM Trans. Access. Comput., 16(1). DOI: https://doi.org/10.1145/3568166.

ISO (2019). Ergonomics of human-system interaction — part 210: Human-centred design for interactive systems.

Jain, M., Kumar, P., Kota, R., and Patel, S. N. (2018). Evaluating and informing the design of chatbots. In Proceedings of the 2018 Designing Interactive Systems Conference, DIS ’18, page 895–906, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3196709.3196735.

Jia, M. and Jyou, L. (2021). The study of the application of a keywords-based chatbot system on the teaching of foreign languages. Journal of Intelligent & Fuzzy Systems, pages 1–10. DOI: https://doi.org/10.48550/arXiv.cs/0310018.

Jin, Y., Chen, L., Cai, W., and Pu, P. (2021). Key qualities of conversational recommender systems: From users’ perspective. In Proceedings of the 9th International Conference on Human-Agent Interaction, HAI ’21, page 93–102, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3472307.3484164.

Jin, Y., Zhang, X., and Wang, W. (2019). Musicbot: Evaluating critiquing-based music recommenders with conversational interaction. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–22. DOI: https://doi.org/10.1145/3357384.3357923.

Joshi, A., Kale, S., Chandel, S., and Pal, D. (2015). Likert scale: Explored and explained. British Journal of Applied Science amp; Technology, 7(4):396–403. DOI: http://dx.doi.org/10.9734/bjast/2015/14975.

Jung, J.-Y., Qiu, S., Bozzon, A., and Gadiraju, U. (2022). Great chain of agents: The role of metaphorical representation of agents in conversational crowdsourcing. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491102.3517653.

Kattenbeck, M., Kilian, M. A., Ferstl, M., Alt, F., and Ludwig, B. (2018). Airbot: using a work flow model for proactive assistance in public spaces. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pages 213–220. DOI: https://doi.org/10.1145/3236112.3236142.

Kernan Freire, S., Niforatos, E., Wang, C., Ruiz-Arenas, S., Foosherian, M., Wellsandt, S., and Bozzon, A. (2023). Lessons learned from designing and evaluating claica: A continuously learning ai cognitive assistant. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, page 553–568, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3581641.3584042.

Kim, S., Lee, J., and Gweon, G. (2019). Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–12. DOI: https://doi.org/10.1145/3290605.3300316.

Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report 2.3, EBSE, Ver. 2.3 EBSE Technical Report.

Lankton, N., McKnight, D. H., and Tripp, J. (2015). Technology, humanness, and trust: Rethinking trust in technology. Journal of the Association for Information Systems, 16:880–918. DOI: https://doi.org/10.17705/1jais.00411.

Laugwitz, B., Held, T., and Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. In Holzinger, A., editor, HCI and Usability for Education and Work, pages 63–76, Berlin, Heidelberg. Springer Berlin Heidelberg. DOI: https://doi.org/10.1007/978-3-540-89350-9_6.

Law, E. L.-C., FØLstad, A., and Van As, N. (2022). Effects of humanlikeness and conversational breakdown on trust in chatbots for customer service. In Nordic Human-Computer Interaction Conference, NordiCHI ’22, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3546155.3546665.

Lee, Y.-C., Yamashita, N., and Huang, Y. (2021). Exploring the effects of incorporating human experts to deliver journaling guidance through a chatbot. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–27. DOI: https://doi.org/10.1145/3449196.

Liu, C., Zhou, S., Zhang, Y., Liu, D., Peng, Z., and Ma, X. (2022). Exploring the effects of self-mockery to improve task-oriented chatbot’s social intelligence. In Proceedings of the 2022 ACM Designing Interactive Systems Conference, DIS ’22, page 1315–1329, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3532106.3533461.

Liu, Y., Kim, D.-j., Miao, T., and Chuang, Y. (2020). Slumberbot: An interactive agent for helping users investigate disturbance factors of sleep quality. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society, pages 1–4. DOI: https://doi.org/10.1145/3419249.3420091.

Mafra, M. G. S., Nunes, K., Rocha, S., Braz Junior, G., Silva, A., Viana, D., Silva, W., and Rivero, L. (2024). Proposing usability-ux technologies for the design and evaluation of text-based chatbots. Journal on Interactive Systems, 15(1):234–251. DOI: https://doi.org/10.5753/jis.2024.3856.

Mariano, P., Chaves, A. P., and Valentim, N. (2024). Technical report. Technical report, Federal University of Paraná. DOI: http://doi.org/10.6084/m9.figshare.25493947.v2.

mobiletime (2022). Mapa do ecossistema brasileiro de bots 2022. [link]. Access: 14 August 2024.

Moilanen, J., Visuri, A., Suryanarayana, S. A., Alorwu, A., Yatani, K., and Hosio, S. (2022). Measuring the effect of mental health chatbot personality on user engagement. In Proceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia, MUM ’22, page 138–150, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3568444.3568464.

Mudofi, L. N. H. and Yuspin, W. (2022). Evaluating quality of chatbots and intelligent conversational agents of bca (vira) line. Interdisciplinary Social Studies, 1(5):532–542. DOI: https://doi.org/10.55324/iss.v1i5.122.

Park, S., Thieme, A., Han, J., Lee, S., Rhee, W., and Suh, B. (2021). “i wrote as if i were telling a story to someone i knew.”: Designing chatbot interactions for expressive writing in mental health. In Designing Interactive Systems Conference 2021, pages 926–941. DOI: https://doi.org/10.1145/3461778.3462143.

Portela, M. and Granell-Canut, C. (2017). A new friend in our smartphone? observing interactions with chatbots in the search of emotional engagement. In Proceedings of the XVIII International Conference on Human Computer Interaction, pages 1–7. DOI: https://doi.org/10.1145/3123818.3123826.

Przegalinska, A., Ciechanowski, L., Stroz, A., Gloor, P., and Mazurek, G. (2019). In bot we trust: A new methodology of chatbot performance measures. Business Horizons, 62(6):785–797. DOI: https://doi.org/10.1016/j.bushor.2019.08.005.

Rapp, A., Curti, L., and Boldi, A. (2021). The human side of human-chatbot interaction: A systematic literature review of ten years of research on text-based chatbots. International Journal of Human-Computer Studies, 151:102630. DOI: https://doi.org/10.1016/j.ijhcs.2021.102630.

Ren, R., Castro, J. W., Acuña, S. T., and Lara, J. d. (2019). Usability of chatbots: A systematic mapping study. In Proceedings of the 31st International Conference on Software Engineering and Knowledge Engineering, SEKE2019. KSI Research Inc. and Knowledge Systems Institute Graduate School. DOI: http://dx.doi.org/10.18293/seke2019-029.

Ruane, E., Farrell, S., and Ventresque, A. (2021). User perception of text-based chatbot personality. In Følstad, A., Araujo, T., Papadopoulos, S., Law, E. L.-C., Luger, E., Goodwin, M., and Brandtzaeg, P. B., editors, Chatbot Research and Design, pages 32–47, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-68288-0_3.

Santos, G., Rocha, A. R., Conte, T., Barcellos, M. P., and Prikladnicki, R. (2012). Strategic alignment between academy and industry: A virtuous cycle to promote innovation in technology. In 2012 26th Brazilian Symposium on Software Engineering, pages 196–200. DOI: https://doi.org/10.1109/SBES.2012.31.

Schmitt, A., Wambsganss, T., and Leimeister, J. M. (2022). Conversational agents for information retrieval in the education domain: A user-centered design investigation. Proc. ACM Hum.-Comput. Interact., 6(CSCW2). DOI: https://doi.org/10.1145/3555587.

Schrepp, M., Hinderks, A., and Thomaschewski, J. (2017). Design and evaluation of a short version of the user experience questionnaire (ueq-s). International Journal of Interactive Multimedia and Artificial Intelligence, 4 (6), 103-108.. DOI: https://doi.org/10.9781/ijimai.2017.09.001.

Sharma, M., Yadav, S., Kaushik, A., and Sharma, S. (2021). Examining usability on atreya bot: A chatbot designed for chemical scientists. In 2021 International Conference on Computational Performance Evaluation (ComPE), pages 729–733. DOI: https://doi.org/10.1109/ComPE53109.2021.9752288.

Shawar, B. A. and Atwell, E. (2007). Chatbots: are they really useful? Journal for Language Technology and Computational Linguistics, 22(1):29–49. DOI: https://doi.org/10.21248/jlcl.22.2007.88.

Shull, F., Carver, J., and Travassos, G. H. (2001). An empirical methodology for introducing software processes. ACM SIGSOFT Software Engineering Notes, 26(5):288–296. DOI: https://doi.org/10.1145/503271.503248.

Skjuve, M., Haugstveit, I. M., Følstad, A., and Brandtzaeg, P. B. (2019). Help! is my chatbot falling into the uncanny valley? : An empirical study of user experience in human-chatbot interaction. Human Technology, 15(1):30–54. DOI: https://doi.org/10.17011/ht/urn.201902201607.

Souza, A., Guerino, G., and Valentim, N. (2023). Technical report. Technical report, Federal University of Paraná. DOI: http://doi.org/10.6084/m9.figshare.23145257.v2

Telner, J. (2021). Chatbot user experience: Speed and content are king. In Advances in Artificial Intelligence, Software and Systems Engineering: Proceedings of the AHFE 2021 Virtual Conferences on Human Factors in Software and Systems Engineering, Artificial Intelligence and Social Computing, and Energy, July 25-29, 2021, USA, pages 47–54. Springer. DOI: https://doi.org/10.1007/978-3-030-80624-8_6.

Torkamaan, H. (2023). Mood measurement on smartphones: Which measure, which design? Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 7(1). DOI: https://doi.org/10.1145/3580864.

Tubin, C., Mazuco Rodriguez, J. P., and de Marchi, A. C. B. (2022). User experience with conversational agent: A systematic review of assessment methods. Behaviour & Information Technology, 41(16):3519–3529. DOI: https://doi.org/10.1080/0144929x.2021.2001047.

Veglis, A., Maniou, T. A., et al. (2019). Chatbots on the rise: A new narrative in journalism. Studies in Media and Communication, 7(1):1–6. DOI: https://doi.org/10.11114/smc.v7i1.3986.

Völkel, S. T. and Kaya, L. (2021). Examining user preference for agreeableness in chatbots. In Proceedings of the 3rd Conference on Conversational User Interfaces, pages 1–6. DOI: https://doi.org/10.1145/3469595.3469633.

Wald, R., Heijselaar, E., and Bosse, T. (2021). Make your own: The potential of chatbot customization for the development of user trust. In Adjunct Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pages 382–387. DOI: https://doi.org/10.1145/3450614.3463600.

Wambsganss, T., Kueng, T., Soellner, M., and Leimeister, J. M. (2021). Arguetutor: An adaptive dialogbased learning system for argumentation skills. In Proceedings of the 2021 CHI conference on human factors in computing systems, pages 1–13. DOI: https://doi.org/10.1145/3411764.3445781.

Wambsganss, T., Zierau, N., Söllner, M., Käser, T., Koedinger, K. R., and Leimeister, J. M. (2022). Designing conversational evaluation tools: A comparison of text and voice modalities to improve response quality in course evaluations. Proc. ACM Hum.-Comput. Interact., 6(CSCW2). DOI: https://doi.org/10.1145/3555619.

Watson, K. (2018). Establishing psychological wellbeing metrics for the built environment. Building Services Engineering Research and Technology, 39:014362441875449. DOI: https://doi.org/10.1177/0143624418754497.

Xiao, Z., Zhou, M. X., and Fu, W.-T. (2019). Who should be my teammates: Using a conversational agent to understand individuals and help teaming. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 437–447. DOI: https://doi.org/10.1145/3301275.3302264.

Xu, H., Dinev, T., Smith, H., and Hart, P. (2008). Examining the formation of individual’s privacy concerns: Toward an integrative view. ICIS 2008 Proceedings - Twenty Ninth International Conference on Information Systems.

Yu, D., Tian, J., Su, T., Tu, Z., Xu, X., and Wang, Z. (2021). Incorporating multimodal sentiments into conversational bots for service requirement elicitation. In 2021 IEEE International Conference on ServiceOriented System Engineering (SOSE), pages 81–90. DOI: https://doi.org/10.1109/SOSE52839.2021.00014.

Yuen, M. (2022). Chatbot market in 2022: Stats, trends, and companies in the growing ai chatbot industry. [link]. Access: 14 August 2024.

Yun, H., Ham, A., Kim, J., Kim, T., Kim, J., Lee, H., Park, J., and Jang, J. (2020). Chatbot with touch and graphics: An interaction of users for emotional expression and turn-taking. In Proceedings of the 2nd Conference on Conversational User Interfaces, pages 1–5. DOI: https://doi.org/10.1145/3405755.3406147.

Zhang, C., Li, G., Hashimoto, H., and Zhang, Z. (2022). Digital transformation (dx) for skill learners: The design methodology and implementation of educational chatbot using knowledge connection and emotional expression. In 2022 IEEE Global Engineering Education Conference (EDUCON), pages 998–1003. DOI: https://doi.org/10.1109/EDUCON52537.2022.9766384.

Zorrilla, A. L. and Torres, M. I. (2022). A multilingual neural coaching model with enhanced long-term dialogue structure. ACM Trans. Interact. Intell. Syst., 12(2). DOI: https://doi.org/10.1145/3487066.