Which UX Research method or technique should I take? Assisting software startup professionals in making their choices

Luana de Parolis Bianchini; Luciana Zaina

doi:10.5753/jis.2024.4028

Authors

Luana de Parolis Bianchini Federal University of São Carlos https://orcid.org/0009-0000-2185-2905
Luciana Zaina Federal University of São Carlos https://orcid.org/0000-0002-1736-544X

DOI:

https://doi.org/10.5753/jis.2024.4028

Keywords:

UX, UX Research, Software Startups, Software Development Professionals, Thematic Analysis

Abstract

User Experience (UX) is a field that aids in comprehending the impact of system usage on the user during the product development process. Software startups are entities that deal with cutting-edge products and experience significant growth, even in the face of unpredictability and potential market volatility. UX methods and techniques serve as tools for software start-up professionals to improve their products, establish connections with their customers, and maintain superior software quality. The goal of this paper is to present a thematic map filled with recommendations for the use of approaches, methods, techniques, and tools designed to assist software startup professionals in conducting research with users. We begin by conducting a gray literature review (GLR) on expert websites and blogs to uncover the approaches, methods, techniques, and tools most frequently referenced by industry professionals and their respective objectives. A thematic map was suggested, based on the thematic examination of the GLR findings. We created a web-based catalog named CatUX to facilitate the map’s assessment. We evaluated the catalog’s utilization by professionals from two angles: the use of the catalog and the catalog’s acceptance. The findings indicated that the catalog helped the participants select the most suitable approach, method, technique, and tool for each given scenario.

Downloads

Download data is not yet available.

References

Ageieva, A. (2021). A survival guide to user interviews. Available in: Supplementary Material.

Almaraz, V. (2021). Usando o obs studio para testes de usabilidade remotos. Available in: Supplementary Material.

Almeida, M. (2019). Melhorando a eficiência do acompanhamento legislativo — um estudo de caso de ux. Available in: Supplementary Material.

Amarante, M. (2021). 5 coisas que aprendi conduzindo testes de usabilidade remotos. Available in: Supplementary Material.

Amorim, L. (2019). Entrevista motivacional é a ferramenta de ux research que você nunca ouviu falar sobre. Available in: Supplementary Material.

Anderson, N. (2020). Which ux research methodology should you use? [chart included]. Available in: Supplementary Material.

Araújo, B. S. (2019). Melhorando os cuidados com a saúde das crianças brasileiras — estudo de caso de ux. Available in: Supplementary Material.

Banus, O. (2020a). Basic guide for ux research planning. Available in: Supplementary Material.

Banus, O. (2020b). Basic guide for ux research planning. Available in: Supplementary Material.

Banus, O. (2020c). A guide to top ux research methods. Available in: Supplementary Material.

Barbosa, S. D. J., da Silva, B. S., Silveira, M. S., Gasparini, I., Darin, T., and Barbosa, G. D. J. (2021). Interação Humano-Computador. Editora Campus, 2th edition.

Bastos, G. (2020). Pesquisa quantitativa ou qualitativa: Qual eu devo fazer? Available in: Supplementary Material.

Batista, L. M. (2018). Como tornar a experiência da organização financeira pessoal e familiar mais rápida e prática? — ux case study. Available in: Supplementary Material.

Benyon, D. (2011). Interação Humano-Computador. Editora Pearson.

Bernstein, J. (2021). Doing rite wrong. Available in: Supplementary Material.

Biduski, D., Bellei, E. A., Rodriguez, J. P. M., Zaina, L. A. M., and De Marchi, A. C. B. (2020). Assessing long-term user experience on a mobile health application through an in-app embedded conversation-based questionnaire. Computers in Human Behavior, 104:106169. DOI: https://doi.org/10.1016/j.chb.2019.106169.

Bittner, M. (2020). The ultimate ux research cheat sheet. Available in: Supplementary Material.

Bowers, M. (2020). Ux research methods and the path to user empathy. Available in: Supplementary Material.

Bowman, J. (2021a). 17 tools that will streamline your ux research. Available in: Supplementary Material.

Bowman, J. (2021b). The best ux research methods in a pinch. Available in: Supplementary Material.

Bowman, J. (2021c). Fast and cheap ways to find ux research participants. Available in: Supplementary Material.

Boylan, L. (2020). My ux research toolbox as a team of one. Available in: Supplementary Material.

Bozzi, C. (2019). Melhorando a saúde dos brasileiros que precisam de ajuda psicológica ou psiquiátrica — estudo de caso de ux. Available in: Supplementary Material.

Braum, M. (2019). Descobertas de um aplicativo de saúde — estudo de caso de ux. Available in: Supplementary Material.

Braum, M. (2020). Guia: Como medir a usabilidade de produtos com system usability scale (sus). Available in: Supplementary Material.

Braun, V. and Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2):77–101. DOI: https://doi.org/10.1191/1478088706qp063oa.

Brito, D. (2019). Worcoffee, um app para trabalhadores remotos — estudo de caso de ux. Available in: Supplementary Material.

Budiu, R. (2021). Why 5 participants are okay in a qualitative study, but not in a quantitative one. Available in: Supplementary Material.

Calheiros, R. (2020). Estourando a bolha: Um relato sobre uma pesquisa imersiva em cidades remotas do brasil. Available in: Supplementary Material.

Camillo, R. (2019). Criando um solução de gestão financeira pessoal para o banco do brasil — estudo de caso de ux. Available in: Supplementary Material.

Cavalcanti, V. (2020). A aplicação do ux research para melhoria no setor do turismo em período de covid-19. Available in: Supplementary Material.

Conrick, K. (2020). A breakdown of ux research types. Available in: Supplementary Material.

Cybis, W., Betiol, A. H., and Faust, R. (2015). Ergonomia e Usabilidade – Conhecimentos, Métodos e Aplicações. Novatec, 3th edition.

Dana, H. (2018). Ux case study: Dramafever. Available in: Supplementary Material.

Darin, T., Coelho, B., and Borges, B. (2019). Which instrument should i use? supporting decision-making about the evaluation of user experience. In Design, User Experience, and Usability. Practice and Case Studies: 8th International Conference, DUXU 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part IV, page 49–67. Springer-Verlag. DOI: https://doi.org/10.1007/978-3-030-23535-2_4.

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3):319–340. DOI: https://doi.org/10.2307/249008.

Dawud, M. (2021). Ux research e design thinking: o match perfeito. Available in: Supplementary Material.

Dexter, S. (2021). Figma continues to skyrocket — 63Available in: Supplementary Material.

Dzekman, T. (2019). 4 methods for analysing user interviews. Available in: Supplementary Material.

Fard, A. (2019). User research methods: how to choose the right one? Available in: Supplementary Material.

Farrell, S. (2017). Ux research cheat sheet. Available in: [link]. Access on 10 January 2024.

Fathallah, S. (2018). Researching family planning through proverbial hausa — a ux case study. Available in: Supplementary Material.

Fátima, I. d. (2020). Resultados da pesquisa sobre os pesquisadores de ux no brasil. Available in: Supplementary Material.

Fátima, I. d. (2021). Panorama ux research: Atividades e ferramentas – parte 3. Available in: Supplementary Material.

Garousi, V., Felderer, M., and Mäntylä, M. V. (2019). Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology, 106:101–121. DOI: https://doi.org/10.1016/j.infsof.2018.09.006.

Giardino, C., Paternoster, N., Unterkalmsteiner, M., Gorschek, T., and Abrahamsson, P. (2016). Software development in startup companies: The greenfield startup model. IEEE Transactions on Software Engineering, 42(6):585–604. DOI: https://doi.org/10.1109/TSE.2015.2509970.

Gibbons, S. (2020). Ux roadmaps: Definition and components. Available in: Supplementary Material.

Grilo, A. (2022). Bibliografia brasileira sobre experiência do usuário - brasil ux design. Available in: [link]. Access on 10 January 2024.

Guerino, G. C., Dias, N. S. B. C., Chanin, R., Prikladnicki, R., Balancieri, R., and Leal, G. C. L. (2021). User experience practices in early-stage software startups - an exploratory study. In Wang, X., Martini, A., NguyenDuc, A., and Stray, V., editors, Software Business, pages 122–136, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-91983-2_10.

Harley, A. (2018). Multivariate vs. a/b testing: Incremental vs. radical changes. Available in: Supplementary Material.

Harper, C. (2018). Best practices for ethnographic research, lessons learned in the wild. Available in: Supplementary Material.

Hassenzahl, M. (2018). The Thing and I (Summer of ’17 Remix), pages 17–31. Springer International Publishing, Cham. DOI: https://doi.org/10.1007/978-3-319-68213-6_2.

Henriques, C., Pilar, D., and Ignácio, E. (2022). UX Research com Sotaque Brasileiro. Casa do Código, 1th edition.

Ho, E. (2020). 5 things ux researchers can do differently: a reflection after uber’s layoffs. Available in: Supplementary Material.

Hokkanen, L., Kuusinen, K., and Väänänen, K. (2015). Early product design in startups: Towards a ux strategy. In Abrahamsson, P., Corral, L., Oivo, M., and Russo, B., editors, Product-Focused Software Process Improvement, pages 217–224, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-319-26844-6_16.

Hokkanen, L. and Leppänen, M. (2015). Three patterns for user involvement in startups. In Proceedings of the 20th European Conference on Pattern Languages of Programs, pages 1–8. Association for Computing Machinery. DOI: https://doi.org/10.1145/2855321.2855373.

Hokkanen, L. and Väänänen-Vainio-Mattila, K. (2015). Ux work in startups: Current practices and future needs. In Lassenius, C., Dingsøyr, T., and Paasivaara, M., editors, Agile Processes in Software Engineering and Extreme Programming, pages 81–92. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-319-18612-2_7.

Hott, K. (2020). Perfecting the art of the ux research report. Available in: Supplementary Material.

Johns, R. (2005). One size doesn’t fit all: Selecting response scales for attitude items. Journal of Elections, Public Opinion Parties, 15:237–264. DOI: https://doi.org/10.1080/13689880500178849.

Jones, S. (2020). Advanced ux lab techniques: Don’t just listen to users. Available in: Supplementary Material.

Joyce, A. (2019). Formative vs. summative evaluations. Available in: Supplementary Material.

Júnior, E. (2021). O que é e quais as funções do ux research? Available in: Supplementary Material.

Kaley, A. (2021). Structuring intranet discovery design research. Available in: Supplementary Material.

Kaplan, K. (2019a). How to conduct research for customer journey-mapping. Available in: Supplementary Material.

Kaplan, K. (2019b). The science of silence: Intentional silence as a moderation technique. Available in: Supplementary Material.

Kaplan, K. (2020). Remote ux work: Guidelines and resources. Available in: Supplementary Material.

Kaplan, K. (2021). 10 usability heuristics applied to complex applications. Available in: Supplementary Material.

Kato, M. (2019). Amaar: Criação de um website para uma ong de animais. Available in: Supplementary Material.

Kawakami, G. (2021). Testes de usabilidade na pandemia. Available in: Supplementary Material.

Klotins, E. (2018). Software start-ups through an empirical lens: are start-ups snowflakes? In Workshop on Softwareintensive Business.

Krause, R. (2019). Tracking research questions, assumptions, and facts in agile. Available in: Supplementary Material.

Krause, R. (2021). Remote design work: Top challenges. Available in: Supplementary Material.

Kuusinen, K., Sørensen, M., Frederiksen, N., Laugesen, N., and Juul, S. (2019). From startup to scaleup: An interview study of the development of user experience work in a data-intensive company. In Lárusdóttir, M., Winckler, M., Kuusinen, K., Palanque, P., and Bogdan, C., editors, Human-Centered Software Engineering - 7th IFIP WG 13.2 International Working Conference, HCSE 2018, Revised Selected Papers, pages 3–14. Springer. DOI: https://doi.org/10.1007/978-3-030-05909-5_1.

Lacey, A. and Luff, D. (2001). Qualitative data analysis. Trent Focus Sheffield.

Laubheimer, P. (2021). Data is more than numbers: Why qualitative data isn’t just opinions. Available in: Supplementary Material.

Leslie, C. (2021). O mercado de ux em um ano pandêmico: Atividades e ferramentas. Available in: Supplementary Material.

Library, C. U. (2022). Science immersion: Power research: Finding gray literature. Available in: [link]. Access on 10 January 2024.

Lima, L. (2019). Usando o percurso cognitivo para avaliar uma interface. Available in: Supplementary Material.

Lima, S. (2020a). Como criar bons questionários de pesquisa. Available in: Supplementary Material.

Lima, S. (2020b). Como planejar uma pesquisa de ux em 8 passos simples. Available in: Supplementary Material.

Lima, S. (2020c). Como tabular testes de usabilidade. Available in: Supplementary Material.

Lima, S. (2020d). Covid-19 e ux research: como pesquisar em tempos de corona. Available in: Supplementary Material.

Lima, S. (2020e). Personas: como criamos conexão com os clientes seguro auto youse. Available in: Supplementary Material.

Liu, F. (2019). Iterative design of a survey question: A case study. Available in: Supplementary Material.

Lowdermilk, T. (2013). Design Centrado no Usuário: Um guia para o desenvolvimento de aplicativos amigáveis. Novatec Editora.

Machado, F. (2021). 3 dicas para conhecer seus usuários sem precisar entrevistá-los. Available in: Supplementary Material.

Marti, P. and Iacono, I. (2016). Anticipated, momentary, episodic, remembered: the many facets of user experience. In Conference on Computer Science and Intelligence Systems, pages 1647–1655. DOI: https://doi.org/10.15439/2016F302.

Martinelli, S., Lopes, L., and Zaina, L. (2022). Ux research in the software industry: An investigation of longterm ux practices. In Proceedings of the 21st Brazilian Symposium on Human Factors in Computing Systems, pages 1–13. Association for Computing Machinery. DOI: https://doi.org/10.1145/3554364.3559126.

Matt, A. (2021). Ux research — o guia do investigador. Available in: Supplementary Material.

Mattei, L. (2020). A ux research framework to speed up your design process. Available in: Supplementary Material.

May, B. (2012). Applying lean startup: An experience report – lean lean ux by a ux veteran: Lessons learned in creating launching a complex consumer app. In 2012 Agile Conference, pages 141–147. DOI: https://doi.org/10.1109/Agile.2012.18.

MICHAELIS (2024a). Dicionário brasileiro da língua portuguesa: Definição de estratégia. Available in: [link]. Access on 10 January 2024.

MICHAELIS (2024b). Dicionário brasileiro da língua portuguesa: Definição de ferramenta. Available in: [link]. Access on 10 January 2024.

Michelan, F. (2020). Do que os designers brasileiros se alimentam? Available in: Supplementary Material.

Miyake, A. (2019). Pão de açúcar mais — estudo de caso de ux. Available in: Supplementary Material.

Moran, K. (2018a). Quantitative user-research methodologies: An overview. Available in: Supplementary Material.

Moran, K. (2018b). Quantitative ux research in practice. Available in: Supplementary Material.

Moran, K. (2019a). Remote ux work: The nn/g case study. Available in: Supplementary Material.

Moran, K. (2019b). Usability testing 101. Available in: Supplementary Material.

Moran, K. (2021a). Collecting metrics during qualitative studies. Available in: Supplementary Material.

Moran, K. (2021b). Context methods: Study guide. Available in: Supplementary Material.

Moran, K. (2021c). Quantitative research: Study guide. Available in: Supplementary Material.

Moran, K. and Pernice, K. (2020a). Remote moderated usability tests: How to do them. Available in: Supplementary Material.

Moran, K. and Pernice, K. (2020b). Remote moderated usability tests: Why to do them. Available in: Supplementary Material.

Moran, K. and Rosala, M. (2021). When to use context methods: Field and diary studies. Available in: Supplementary Material.

Neto, W. D. (2019). Como solucionamos o fluxo de compra de hospitais usando o design sprint 2.0 — estudo de caso. Available in: Supplementary Material.

Ngai, J. (2020). 13 helpful ux research tools. Available in: Supplementary Material.

Nielsen, J. (2017). A 100-year view of user experience. Available in: [link]. Access on 10 January 2024.

Norman, D. (2018). Principles of human-centered design. Available in: [link]. Access on 10 January 2024.

Norman, D. and Nielsen, J. (2016). The definition of user experience (ux). Available in: [link]. Access on 10 January 2024.

Oliveira, M. d. (2019). Ajudando pessoas que utilizam o metrô pela primeira vez — estudo de caso de ux. Available in: Supplementary Material.

Paternoster, N., Giardino, C., Unterkalmsteiner, M., Gorschek, T., and Abrahamsson, P. (2014). Software development in startup companies: A systematic mapping study. Information and Software Technology, 56(10):1200–1218. DOI: https://doi.org/10.1016/j.infsof.2014.04.014.

Pernice, K. (2018). User interviews 101. Available in: Supplementary Material.

Pernice, K. (2020). Research repositories for tracking ux research and growing your researchops. Available in: Supplementary Material.

Philips, M. (2020). The complete guide to ux research methods. Available in: Supplementary Material.

Preece, J., Sharp, H., and Rogers, Y. (2013). Design de Interação: Além da Interação Humano-Computador. Bookman, 3th edition.

Prithyani, K. (2020). 65 ux methods and when to use them. Available in: Supplementary Material.

Rahardjo, J. (2021). How research builds the designer. Available in: Supplementary Material.

Ramos, A. L. P. (2019). Como melhorar a qualidade de vida de entregadores de bicicleta? — estudo de caso de ux. Available in: Supplementary Material.

Reciputti, G. (2020). Dicas práticas sobre ux research: recrutamento. Available in: Supplementary Material. Ríos, A. (2019). Interviewing: more than a user research method. Available in: Supplementary Material.

Rogers, A. (2021). How to share your ux research. Available in: Supplementary Material.

Rogers, D. (2019). Qualitative and quantitative user research. Available in: Supplementary Material.

Rosala, M. (2019). How to analyze qualitative data from ux research: Thematic analysis. Available in: Supplementary Material.

Rosala, M. (2020a). The critical incident technique in ux. Available in: Supplementary Material.

Rosala, M. (2020b). The discovery phase in ux projects. Available in: Supplementary Material.

Rosala, M. (2020c). Rating scales in ux research: Likert or semantic differential? Available in: Supplementary Material.

Rosala, M. (2021). Writing an effective guide for a ux interview. Available in: Supplementary Material.

Saad, J., Martinelli, S., Machado, L. S., de Souza, C. R., Alvaro, A., and Zaina, L. (2021). Ux work in software startups: A thematic analysis of the literature. Information and Software Technology, 140:106688. DOI: https://doi.org/10.1016/j.infsof.2021.106688.

Sabino, A. (2020). Métodos criativos de user research para startups. Available in: Supplementary Material.

Salvi, V. (2021). How can you make ux research insights visible, traceable, and fun? Available in: Supplementary Material.

Samsyka, M. (2019). Understanding methods and tools in ux research. Available in: Supplementary Material.

Sertório, W. (2020). Pesquisa qualitativa em tempos de quarentena. Available in: Supplementary Material.

Shen, S. (2018). Best practices for on-video user testing. Available in: Supplementary Material.

Sherwin, K. (2018). Card sorting: Uncover users’ mental models for better information architecture. Available in: Supplementary Material.

Silveira, S. A. M., Choma, J., Pereira, R., Guerra, E. M., and Zaina, L. A. M. (2021). Ux work in software startups: Challenges from the current state of practice. In Gregory, P., Lassenius, C., Wang, X., and Kruchten, P., editors, Agile Processes in Software Engineering and Extreme Programming, pages 19–35, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-78098-2_2.

Soares, E. (2020). Teste de usabilidade à distância (remoto). Available in: Supplementary Material.

Someren, M., Barnard, Y., and Sandberg, J. (1994). The Think Aloud Method - A Practical Guide to Modelling Cognitive Processes. Academic Press.

Süner-Pla-Cerdà, Sedefand Töre Yargın, G. Ş. H. D. S. (2021). Examining the impact of covid-19 pandemic on ux research practice through ux blogs. In Soares, M. M., Rosenzweig, E., and Marcus, A., editors, Design, User Experience, and Usability: UX Research and Design, pages 579–592. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-78221-4_40.

Sutton, S. (2000). The role of process in software start-up. IEEE Software, 17(4):33–39. DOI: https://doi.org/10.1109/52.854066.

Teixeira, F. (2020). Os testes a/b enquanto válvulas de escape. Available in: Supplementary Material.

Topps, R. (2020). Key considerations when planning a remote usability study for accessibility. Available in: Supplementary Material.

Unterkalmsteiner, M., Abrahamsson, P., Wang, X., NguyenDuc, A., Shah, S., Bajwa, S. S., Baltes, G., Conboy, K., Cullina, E., Dennehy, D., Edison, H., Fernandez-Sanchez, C., Garbajosa, J., Gorschek, T., Klotins, E., Hokkanen, L., Kon, F., Lunesu, I., Marchesi, M., Morgan, L., Oivo, M., Selig, C., Seppänen, P., Sweetman, R., Tyrväinen, P., Ungerer, C., and Yagüe, A. (2016). Software startups - a research agenda. e-Informatica Software Engineering Journal, 10(1):89 – 123. DOI: https://doi.org/10.5277/e-Inf160105.

Utesch, B. (2021). Ux research characteristics: An analysis of 100 open jobs. Available in: Supplementary Material.

Varsaluoma, J. and Sahar, F. (2014). Usefulness of long-term user experience evaluation to product development: Practitioners’ views from three case studies. In Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, page 79–88. Association for Computing Machinery. DOI: https://doi.org/10.1145/2639189.2641218.

Wärnestål, P. (2021). How to design ai-powered services — lessons from the emergency room. Available in: Supplementary Material.

Weber, M. (2019). Ux research: por onde começar uma pesquisa? Available in: Supplementary Material.

Wenzel, M. (2020). How to select appropriate ux research methods. Available in: Supplementary Material.

Whitenton, K. (2021). Triangulation: Get better research results by using multiple ux methods. Available in: Supplementary Material.

Zaghi, R. (2019). Como usei os processos de ux para facilitar a matrícula de crianças nas creches públicas. Available in: Supplementary Material.

Zaina, L. A., Fortes, R. P., Casadei, V., Nozaki, L. S., and Paiva, D. M. B. (2022). Preventing accessibility barriers: Guidelines for using user interface design patterns in mobile applications. Journal of Systems and Software, 186:111213. DOI: https://doi.org/10.1016/j.jss.2021.111213.

Zukerman, B. (2021). Writer in tech? 7 reasons you should run a/b tests. Available in: Supplementary Material.