Exploring ChatGPT's Performance in Supporting the Design of Human-Computer Interaction

Arisa Sakaguti; Vania Neris

doi:10.5753/jis.2026.6947

Authors

Arisa Sakaguti São Carlos Federal University https://orcid.org/0009-0006-9402-6237
Vania Neris São Carlos Federal University https://orcid.org/0000-0002-4059-8700

DOI:

https://doi.org/10.5753/jis.2026.6947

Keywords:

LLM, ChatGPT, HCI, Personas, Questionary, Interview, Scenario

Abstract

Background: Large Language Models (LLMs) have significant potential for Human-Computer Interaction (HCI), presenting opportunities to assist several stages of design. However, challenges remain regarding contextual understanding, user-aligned suggestions, and the reliability and transparency of AI-driven decision-making. Purpose: This study examines ChatGPT’s potential in supporting user interface design during requirements gathering, focusing on idea generation and its implications for HCI. Methods: An exploratory study was conducted within the context of a chatbot for detecting depressive signs in university students. ChatGPT-4 generated design artifacts for requirements elicitation using four HCI techniques: Questionnaires, Interviews, Personas, and Scenarios. Neutral prompts were used, without prompt-engineering strategies. The outputs were evaluated through a literature-based comparison with established HCI references Preece et al. [2013], Cooper [2004], Barbosa et al. [2021], and by four academic experts using a 5 point Likert scale across six criteria: pertinence, comprehensiveness, clarity, depth, applicability, and coverage. Results: Findings indicate that generative AI produces well-structured outputs that assist in identifying user needs. Pertinence achieved the highest ratings across all techniques, and experts emphasized ChatGPT’s utility as a “starting point” for design activities. However, limitations were observed regarding depth, comprehensiveness, and coverage. In complex domains such as mental health, outputs tended to be generic, while human-led focus groups captured these aspects more effectively. Conclusion: The results show that generative AI tools like ChatGPT are valuable preliminary resources for supporting early HCI design. However, their limitations point to the continued necessity of human expertise to review AI-generated content in context-sensitive domains. This calls for hybrid workflows where human creativity and judgment are complemented, but not replaced, by AI capabilities.

Downloads

Download data is not yet available.

References

Abbas, A. M. H., Ghauth, K. I., and Ting, C.-Y. (2022). User experience design using machine learning: A systematic review. IEEE Access, 10:51501–51514. DOI: https://doi.org/10.1109/ACCESS.2022.3173289.

Alencar, T., Ciscon, L., Pereira, G., Macedo, M., Martinelli, S., Choma, J., and Zaina, L. (2025). Genai embedded in activities of academic research: Experiences and lessons from hci studies. In Anais do XXIV Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, pages 58–79, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/ihc.2025.10800.

Alsaqer, S., Alajmi, S., Ahmad, I., and Alfailakawi, M. (2025). The potential of llms in hardware design. Journal of Engineering Research, 13(3):2392–2404. DOI: https://doi.org/10.1016/j.jer.2024.08.001.

Barbosa, S. et al. (2021). Interação Humano Computador e Experiência do Usuário. Elsevier, São Paulo.

Borges, M. G., Correa, C. M., Rosa, D. M. d., Gnecco, A., and Silveira, M. S. (2025). Can (a)i help you? Comparing human and genai analysis of hci qualitative research results. Journal on Interactive Systems, 16(1):962–975. DOI: https://doi.org/10.5753/jis.2025.5416.

Caseli, H. M. and Nunes, M. G. V. (2024). ChatGPT, Maritalk e outros agentes de conversação - um retrato de 2023. In Processamento de Linguagem Natural: Conceitos, Técnicas e Aplicações em Português. BPLN, 3ª edição edition.

Conselho Nacional de Saúde (2016). Resolução nº 510, de 7 de abril de 2016. Acessado em 2025-02-11.

Cooper, A. (2004). The inmates are running the asylum: why high-tech products drive us crazy and how to restore the sanity. Sams Publishing, Indianapolis.

de Cássia Alves, V., Garcia, F. E., Saud, C., Mendes, A., Caseli, H. M., Motti, V. G., de Oliveira Neris, L., Blecher, T., and Neris, V. P. A. (2023). College students-in-the-loop for their mental health: a case of ai and humans working together to support well-being. Interaction Design and Architecture(s) Journal - IxD&A, 59:79–94. DOI: https://doi.org/10.55612/s-5002-059-003.

Dix, A., Finlay, J., Abowd, G. D., and Beale, R. (2003). Human-Computer Interaction. Pearson Education, 3rd edition.

Duarte, E. F., Toledo Palomino, P., Pontual Falcão, T., Porto, G. L. P. M. B., Portela, C. d. S., Ribeiro, D. F., Nascimento, A., Costa Aguiar, Y. P., Souza, M., Moutin Segoria Gasparotto, A., and Maciel Toda, A. (2024). Grandihc-br 2025-2035 - gc6: Implications of artificial intelligence in hci: A discussion on paradigms ethics and diversity equity and inclusion. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, IHC '24, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3702038.3702059.

Garcia, F. E., Brandão, R. P., Mendes, G. C. d. P., and Neris, V. P. d. A. (2019). Able to create, able to (self-)improve: How an inclusive game framework fostered self-improvement through creation and play in alcohol and drugs rehabilitation. In Human-Computer Interaction – INTERACT 2019: 17th IFIP TC 13 International Conference, Paphos, Cyprus, September 2–6, 2019, Proceedings, Part I, page 337–358, Berlin, Heidelberg. Springer-Verlag. DOI: https://doi.org/10.1007/978-3-030-29381-9_21.

Goodman, L. A. (1961). Snowball sampling. Annals of Mathematical Statistics, 32(1):148–170. DOI: https://doi.org/10.1214/aoms/1177705148.

Kim, N. W., Ahn, Y., Myers, G., and Bach, B. (2025). How good is chatgpt in giving advice on your visualization design? ACM Trans. Comput.-Hum. Interact., 32(5). DOI: https://doi.org/10.1145/3745768.

Korteling, J. E. H., van de Boer-Visschedijk, G. C., Blankendaal, R. A. M., Boonekamp, R. C., and Eikelboom, A. R. (2021). Human- versus artificial intelligence. Frontiers in Artificial Intelligence, 4. DOI: https://doi.org/10.3389/frai.2021.622364.

Liu, M. X., Kittur, A., and Myers, B. A. (2022). Crystalline: Lowering the cost for developers to collect and organize information for decision making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22), pages 1–16, New Orleans, LA, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491102.3501968.

Preece, J., Rogers, Y., and Sharp, H. (2013). Design da interação: além da interação homem-computador. Pearson, São Paulo.

Sakaguti, A. (2026). Supplementary materials: Exploratory study on the use of ChatGPT as support in Human-Computer Interaction design. DOI: https://doi.org/10.6084/m9.figshare.32087091.

Sampaio, S. S., Lima, M. S., de Souza, E. R., Meireles, M. A., Pessoa, M. S., and Conte, T. U. (2024). Exploring the use of large language models in requirements engineering education: An experience report with chatgpt 3.5. In Proceedings of the XXIII Brazilian Symposium on Software Quality, SBQS '24, page 624–634, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3701625.3701687.

Shneiderman, B. (2020). Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy human-centered ai systems. ACM Trans. Interact. Intell. Syst., 10(4). DOI: https://doi.org/10.1145/3419764.

Subramonyam, H., Thakkar, D., Ku, A., Dieber, J., and Sinha, A. K. (2025). Prototyping with prompts: Emerging approaches and challenges in generative ai design for collaborative software teams. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, CHI '25, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3706598.3713166.

Suh, S., Lai, M., Pu, K., Dow, S. P., and Grossman, T. (2025). Storyensemble: Enabling dynamic exploration & iteration in the design process with ai and forward-backward propagation. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25), pages 1–36, Busan, Republic of Korea. Association for Computing Machinery. DOI: https://doi.org/10.1145/3746059.3747772.

Takaffoli, M., Li, S., and Mäkelä, V. (2024). Generative ai in user experience design and research: How do ux practitioners, teams, and companies use genai in industry? pages 1579–1593. DOI: https://doi.org/10.1145/3643834.3660720.

Yildirim, N., Kass, A., Tung, T., Upton, C., Costello, D., Giusti, R., Lacin, S., Lovic, S., O'Neill, J., Meehan, R. O., Loideáin, E. , Pini, A., Corcoran, M., Hayes, J., Cahalane, D., Shivhare, G., Castoro, L., Caruso, G., Oh, C., McCann, J., Forlizzi, J., and Zimmerman, J. (2022). How experienced designers of enterprise applications engage ai as a design material. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22), pages 1–13, New Orleans, LA, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491102.3517491.