Can (A)I help you? Comparing human and GenAI analysis of HCI qualitative research results

Mariana Gomes Borges; Claiton Marques Correa; Diego Moreira da Rosa; Andrea Gnecco; Milene Selbach Silveira

doi:10.5753/jis.2025.5416

Authors

Mariana Gomes Borges PUCRS https://orcid.org/0000-0002-0085-5696
Claiton Marques Correa IFFar https://orcid.org/0000-0003-2339-1844
Diego Moreira da Rosa IFRS https://orcid.org/0000-0002-3008-5100
Andrea Gnecco PUCRS https://orcid.org/0000-0002-3103-3361
Milene Selbach Silveira PUCRS https://orcid.org/0000-0003-2159-551X

DOI:

https://doi.org/10.5753/jis.2025.5416

Keywords:

Generative AI, Qualitative Research, User Research, Text Analysis, ChatGPT, Human-AI Collaboration

Abstract

Generative AI (GenAI) is experiencing rapid growth, particularly in its application as a tool for qualitative text analysis—a key element of Human-Computer Interaction (HCI) research. This study examines the potential of GenAI, specifically ChatGPT, to assist in the analysis of qualitative research data. Four qualitative HCI studies, previously conducted and analyzed by our research group, were selected for this investigation. ChatGPT was employed to perform AI-assisted analyses on the raw data from these studies, and the AI-generated insights were then compared with the human-led analyses already completed. The results reveal significant alignment between the human and AI-assisted analyses, indicating that GenAI can serve as an effective support tool in qualitative research. However, while GenAI offers considerable advantages in enhancing research efficiency, human oversight remains crucial to ensuring accurate interpretation and contextual alignment. This study also provides practical recommendations for researchers interested in incorporating GenAI into their qualitative analysis processes.

Downloads

Download data is not yet available.

References

Barambones, J., Moral, C., de Antonio, A., Imbert, R., Martínez, L., and Villalba-Mora, E. (2024). Chatgpt for learning hci techniques: A case study on interviews for personas. IEEE Transactions on Learning Technologies. DOI: https://doi.org/10.1109/TLT.2024.3386095.

Borges, M. G. (2022). Apoio à construção de visualizações narrativas de dados com ênfase em gêneros narrativos. Master's thesis, Pontifical Catholic University of Rio Grande do Sul.

Correa, C. M. (2023). An approach for designing and exploring customizable narrative data visualizations. PhD thesis, Pontifical Catholic University of Rio Grande do Sul.

Correa, C. M. and Silveira, M. S. (2024). An approach for designing and exploring customizable narrative data visualizations. In Proceedings of the XXIII Brazilian Symposium on Human Factors in Computing Systems, pages 1–13. ACM. DOI: https://doi.org/10.1145/3702038.3702080.

da Rosa, D. and Silveira, M. (2025). Human-computer interaction patterns of design for multilingualism. In Anais do XXIV Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais, pages 862–880, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/ihc.2025.12228.

da Rosa, D. M. (2025). Human-Computer Interaction Patterns of Design for Multilingualism. PhD thesis, Pontifical Catholic University of Rio Grande do Sul.

dos Anjos, J. R., de Souza, M. G., de Andrade Neto, A. S., and de Souza, B. C. (2024). An analysis of the generative ai use as analyst in qualitative research in science education. Revista Pesquisa Qualitativa, 12:1–29. DOI: https://doi.org/10.33361/RPQ.2024.v.12.n.30.724.

Feuston, J. L. and Brubaker, J. R. (2021). Putting tools in their place: The role of time and perspective in human-ai collaboration for qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW2):1–25. DOI: https://doi.org/10.1145/3479856.

Gnecco, A. (2022). O uso dos padrões de design de interação em design systems. Master's thesis, Pontifical Catholic University of Rio Grande do Sul.

Hamilton, L., Elliott, D., Quick, A., Smith, S., and Choplin, V. (2023). Exploring the use of ai in qualitative analysis: A comparative study of guaranteed income data. International Journal of Qualitative Methods, 22:1–13. DOI: https://doi.org/10.1177/16094069231201504.

Jiang, J. A., Wade, K., Fiesler, C., and Brubaker, J. R. (2021). Supporting serendipity: Opportunities and challenges for human-ai collaboration in qualitative analysis. Proceedings of the ACM on Human-Computer Interaction, 5:1–23. DOI: https://doi.org/10.1145/3449168.

Liu, Z., Dai, J., Conati, C., and McGrenere, J. (2025). Envisioning AI support during semi-structured interviews across the expertise spectrum. Proceedings of the ACM on Human-Computer Interaction, 9:1–29. DOI: https://doi.org/10.1145/3710909.

Lopez-Fierro, S. and Nguyen, H. (2024). Making Human-AI contributions transparent in qualitative coding. In Proceedings of the 17th International Conference on Computer-Supported Collaborative Learning-CSCL 2024, pp. 3-10. International Society of the Learning Sciences. DOI: https://doi.org/10.22318/cscl2024.352932.

Morgan, D. L. (2023). Exploring the use of artificial intelligence for qualitative data analysis: The case of chatgpt. International journal of qualitative methods, 22:1–10. DOI: https://doi.org/10.1177/1609406923121124.

Perkins, M. and Roe, J. (2024). Generative AI tools in academic research: Applications and implications for qualitative and quantitative research methodologies. arXiv preprint arXiv:2408.06872. DOI: https://doi.org/10.48550/arXiv.2408.06872.

Saare, K. (2024). Natural language based AI tools in interaction design research: Using chatgpt for qualitative user research insight analysis.

Sinha, R., Solola, I., Nguyen, H., Swanson, H., and Lawrence, L. (2024). The role of generative AI in qualitative research: Gpt-4's contributions to a grounded theory analysis. In Proceedings of the Symposium on Learning, Design and Technology, pages 17–25. DOI: https://doi.org/10.1145/3663433.3663456.

Skjuve, M., Følstad, A., and Brandtzaeg, P. B. (2023). The user experience of ChatGPT: findings from a questionnaire study of early users. In Proceedings of the 5th international conference on conversational user interfaces, pages 1–10. DOI: https://doi.org/10.1145/3571884.3597144.

Wachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., and McMahon, S. A. (2024). Prompts, pearls, imperfections: Comparing chatgpt and a human researcher in qualitative data analysis. Qualitative Health Research, 0:1–16. DOI: https://doi.org/10.1177/10497323241244669.

Wang, D., Churchill, E., Maes, P., Fan, X., Shneiderman, B., Shi, Y., and Wang, Q. (2020). From Human-Human collaboration to Human-AI collaboration: Designing AI systems that can work together with people. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI EA '20, page 1–6, New York, NY. Association for Computing Machinery. DOI: https://doi.org/10.1145/3334480.3381069.

Yan, L., Echeverria, V., Fernandez-Nieto, G. M., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., and Martinez-Maldonado, R. (2024). Human-AI collaboration in thematic analysis using ChatGPT: A user study and design recommendations. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–7. DOI: https://doi.org/10.1145/3613905.3650732.

Yang, Q., Steinfeld, A., Rosé, C., and Zimmerman, J. (2020). Re-examining whether, why, and how human-ai interaction is uniquely difficult to design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI '20, page 1–13, New York, NY. Association for Computing Machinery. DOI: https://doi.org/10.1145/3313831.3376301.