Investigating the Adoption of Research Software: From Success Factors to Challenges Faced by Brazilian Academic Researchers
DOI:
https://doi.org/10.5753/jserd.2025.3696Keywords:
Software Engineering Research, Empirical Software Engineering, Research Software, Software Adoption, Challenges, Academia, Universities, Survey Research, Brazilian Academic ResearchersAbstract
Modern research strongly depends on software that is essential in various domains, including engineering, sciences, and other fields, and most often developed within universities and used by the researchers themselves. This software, specifically referred to as research software, presents a unique set of challenges and constraints in its life cycle, methodology, practices, disclosure, and incentives, thereby distinguishing it from conventional software. However, while certain research software developed and used in universities have achieved widespread use and been adopted by the academic community, thereby considered “successful”, others have failed to secure the same level of adoption, resulting in non-adoption and abandonment software leading to an unnecessary expenditure of development time and resources. Gaining a comprehensive understanding of the factors influencing the adoption can assist in mitigating these issues. In this context, the primary aim of this study is to investigate factors influencing the adoption of research software by users and developers, to discern their relative importance, and to explore the challenges associated with fostering the adoption of such software. A survey involving 173 Brazilian academic researchers was executed to gather insights regarding the significance of the factors contributing to the adoption of research software and the obstacles encountered in its adoption. The collected data underwent rigorous statistical analysis, utilizing Independent Sample T-test and Factor Analysis to thoroughly examine the information and provide a ranking of factors. The survey responses were used to rank factors contributing to the adoption of research software, with results stratified by respondents’ profiles. This study suggests that academic researchers can optimize their use of research software by recognizing and prioritizing adoption factors, preventing non-adoption, and overcoming the identified challenges. Understanding the relative importance of each factor can enhance support for research software developers and users and, consequently, promote the broader adoption of research software.
Downloads
References
AlNoamany, Y. and Borghi, J. A. (2018). Towards computational reproducibility: researcher perspectives on the use and sharing of software. PeerJ Computer Science, 4:e163.
Barker, M., Chue Hong, N. P., Katz, D. S., Lamprecht, A.-L., Martinez-Ortiz, C., Psomopoulos, F., Harrow, J., Castro, L. J., Gruenpeter, M., Martinez, P. A., et al. (2022). Introducing the fair principles for research software. Scientific Data, 9(1):622.
Basili, V. R. (1993). Applying the goal/question/metric paradigm in the experience factory. Software quality assurance and measurement: A worldwide perspective, 7(4):21–44.
Basili, V. R., Caldiera, G., and Rombach, H. D. (1994). Goal, question metric paradigm. Encyclopedia of Software Engineering, vol. 1.
Basili, V. R. and Rombach, H. D. (1988). The tame project: Towards improvement-oriented software environments. IEEE Transactions on Software Engineering, 14(6).
Baxter, R., Hong, N. C., Gorissen, D., Hetherington, J., and Todorov, I. (2012). The research software engineer. In Digital Research Conference, Oxford, pages 1–3.
Brasil (2013). Resolução nº 466, de 12 de dezembro de 2012. Diário Oficial da União, 12:59–59. [link].
Brasil (2016). Resolução nº 510, de 7 abril de 2016. [link].
Brett, A., Croucher, M., Haines, R., Hettrick, S., Hetherington, J., Stillwell, M., and Wyatt, C. (2017). Research software engineers: State of the nation report 2017.
Campanelli, A. S., Camilo, R. D., and Parreiras, F. S. (2018). The impact of tailoring criteria on agile practices adoption: A survey with novice agile practitioners in Brazil. Journal of Systems and Software, 137:366–379.
Carver, J., Heaton, D., Hochstein, L., and Bartlett, R. (2013). Self-perceptions about software engineering: A survey of scientists and engineers. Computing in Science & Engineering, 15(1):7–11.
Carver, J. C., Hong, N. P. C., and Thiruvathukal, G. K. (2016). Software Engineering for Science. CRC Press.
Carver, J. C., Weber, N., Ram, K., Gesing, S., and Katz, D. S. (2022). A survey of the state of the practice for research software in the United States. PeerJ Computer Science, 8:e963.
Charmaz, K. (2014). Constructing Grounded Theory. Sage.
Coccia, M. (2019). Why do nations produce science advances and new technology? Technology in Society, 59:101124.
Cohen, J., Katz, D. S., Barker, M., Chue Hong, N., Haines, R., and Jay, C. (2021). The four pillars of research software engineering. IEEE Software, 38(1):97–105.
Corbin, J. and Strauss, A. (2014). Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications.
Creswell, J. W. (2014). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Sage Publications, Thousand Oaks, CA.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297–334.
Cruz, M., Mourão, É., Bernardini, F., Viterbo, J., and Trevisan, D. (2022). Uso do TAM—Technology Acceptance Model—no ciclo de design de aplicações computacionais. In Anais Estendidos do XXI Simpósio Brasileiro de Fatores Humanos em Sistemas Computacionais, pages 3–4. SBC.
da Silva, M., Viterbo, J., Bernardini, F., and Maciel, C. (2018). Identifying privacy functional requirements for crowdsourcing applications in smart cities. In 2018 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 106–111. IEEE.
De Sá, J. P. M. (2007). Applied Statistics Using SPSS, Statistica, Matlab and R. Springer Science & Business Media.
de Souza, M. R., Haines, R., Vigo, M., and Jay, C. (2019). What makes research software sustainable? An interview study with research software engineers. In 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pages 135–138. IEEE.
Denzin, N. K. (2012). Triangulation 2.0. Journal of Mixed Methods Research, 6(2):80–88.
Detofeno, T., Malucelli, A., and Reinehr, S. (2023). Technical debt guild: Managing technical debt from code up to build. Journal of Software Engineering Research and Development, pages 1–1.
Dwivedi, A. K., Mallawaarachchi, I., and Alvarado, L. A. (2017). Analysis of small sample size studies using nonparametric bootstrap test with pooled resampling method. Statistics in Medicine, 36(14):2187–2205.
Eisty, N. U. and Carver, J. C. (2022). Developers perception of peer code review in research software development. Empirical Software Engineering, 27(1):1–26.
Eisty, N. U., Thiruvathukal, G. K., and Carver, J. C. (2018). A survey of software metric use in research software development. In 2018 IEEE 14th International Conference on e-Science (e-Science), pages 212–222. IEEE.
Eisty, N. U., Thiruvathukal, G. K., and Carver, J. C. (2019). Use of software process in research software development: A survey. In Proceedings of the Evaluation and Assessment on Software Engineering, pages 276–282.
Fávero, L. P. and Belfiore, P. (2017). Manual de Análise de Dados: Estatística e Modelagem Multivariada com Excel®, SPSS® e Stata®. Elsevier Brasil.
Fink, A. (2003). The Survey Handbook. Sage, Thousand Oaks, California.
Flick, U. (2004). Triangulation in qualitative research. A Companion to Qualitative Research, 3:178–183.
Flick, U. (2020). Triangulation. Springer.
Gruenpeter, M., Katz, D. S., Lamprecht, A.-L., Honeyman, T., Garijo, D., Struck, A., Niehues, A., Martinez, P. A., Castro, L. J., Rabemanantsoa, T., et al. (2021). Defining research software: A controversial discussion. Zenodo, https://doi.org/10.5281/zenodo.5504016.
Hannay, J. E., MacLeod, C., Singer, J., Langtangen, H. P., Pfahl, D., and Wilson, G. (2009). How do scientists develop and use scientific software? In 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, pages 1–8. IEEE.
Hasselbring, W., Carr, L., Hettrick, S., Packer, H., and Tiropanis, T. (2020). From FAIR research data toward FAIR and open research software. it - Information Technology, 62(1):39–47.
Hauge, Ø., Ayala, C., and Conradi, R. (2010). Adoption of open source software in software-intensive organizations – a systematic literature review. Information and Software Technology, 52(11):1133–1154.
Heaton, D. and Carver, J. C. (2015). Claims about the use of software engineering practices in science: A systematic literature review. Information and Software Technology, 67:207–219.
Hettrick, S., Antonioletti, M., Carr, L., Chue Hong, N., Crouch, S., De Roure, D. C., Emsley, I., Goble, C., Hay, A., Inupakutika, D., et al. (2014). UK Research Software Survey 2014.
IEEE (1990). IEEE Std 610.12-1990: IEEE Standard Glossary of Software Engineering Terminology. IEEE.
Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24(4):602–611.
Johnson, R. A., Wichern, D. W., et al. (2002). Applied Multivariate Statistical Analysis, volume 5. Prentice Hall, Upper Saddle River, NJ.
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23(3):187–200.
Karita, L., Mourão, B. C., Martins, L. A., Soares, L. R., and Machado, I. (2021). Software industry awareness on sustainable software engineering: A Brazilian perspective. Journal of Software Engineering Research and Development, 9:2–1.
Katz, D. S., Gruenpeter, M., and Honeyman, T. (2021). Taking a fresh look at FAIR for research software. Patterns, 2(3):100222.
Katz, D. S. and Hettrick, S. (2023). Research software engineering in 2030. In 2023 IEEE 19th International Conference on e-Science (e-Science), pages 1–2. IEEE.
Katz, D. S., McHenry, K., Reinking, C., and Haines, R. (2019). Research software development & management in universities: Case studies from Manchester’s RSDS group, Illinois’ NCSA, and Notre Dame’s CRC. In 2019 IEEE/ACM 14th International Workshop on Software Engineering for Science (SE4Science), pages 17–24. IEEE.
Kitchenham, B. A. and Pfleeger, S. L. (2008). Personal opinion surveys. In Shull, F., Singer, J., and Sjøberg, D. I. K., editors, Guide to Advanced Empirical Software Engineering, pages 63–92. Springer London, London.
Lahorgue, M. A. (2006). Institutional autonomy and impact of research funding: The case of Brazilian public universities. Knowledge, Power and Dissent, page 255.
Lusk, K. (2018). Lessons learned in effective community–university–industry collaboration models for smart and connected communities research.
McIlroy, M. D., Buxton, J., Naur, P., and Randell, B. (1968). Mass-produced software components. In Proceedings of the 1st International Conference on Software Engineering, Garmisch Partenkirchen, Germany, pages 88–98.
McMANUS, C. and Nobre, C. A. (2017). Brazilian scientific mobility program Science Without Borders: Preliminary results and perspectives. Anais da Academia Brasileira de Ciências, 89:773–786.
Mourão, E., Kalinowski, M., Murta, L., Mendes, E., and Wohlin, C. (2017). Investigating the use of a hybrid search strategy for systematic reviews. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 193–198. IEEE.
Mourão, E., Pimentel, J. F., Murta, L., Kalinowski, M., Mendes, E., and Wohlin, C. (2020). On the performance of hybrid search strategies for systematic literature reviews in software engineering. Information and Software Technology, 123:106294.
Mourão, E., Trevisan, D., and Viterbo, J. (2023a). Understanding the success factors of research software: Interviews with Brazilian computer science academic researchers. In ICITS’23 - 6th International Conference on Information Technology & Systems. Springer.
Mourão, E., Trevisan, D., Viterbo, J., and Pantoja, C. E. (2023b). Investigating developers’ perception on success factors for research software development. In Accepted for publication in 22nd IFIP Conference e-Business, e-Services, and e-Society I3E2023 – IFIP WG. 6.11 (IFIP I3E 2023). Springer.
Nangia, U. and Katz, D. S. (2017). Track 1 paper: Surveying the U.S. National Postdoctoral Association regarding software use and training in research.
Nangia, U., Katz, D. S., et al. (2017). Track 1 paper: Surveying the US National Postdoctoral Association regarding software use and training in research. In Workshop on Sustainable Software for Science: Practice and Experiences (WSSSPE 5.1).
Naur, P., Randell, B., and Buxton, J. N. (1976). Software Engineering: Concepts and Techniques.
Neave, G. R. et al. (2006). Knowledge, Power and Dissent: Critical Perspectives on Higher Education and Research in Knowledge Society. UNESCO.
Nguyen-Hoan, L., Flint, S., and Sankaranarayana, R. (2010). A survey of scientific software development. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, pages 1–10.
Noble, H. and Smith, J. (2015). Issues of validity and reliability in qualitative research. Evidence-Based Nursing, 18(2):34–35.
Nunnally, J. C. (1994). Psychometric Theory 3E. Tata McGraw-Hill Education.
Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, gy, 64:1–18.
Pinto, G., Wiese, I., and Dias, L. F. (2018). How do scientists develop scientific software? An external replication. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 582–591. IEEE.
Pituch, K. A. and Stevens, J. P. (2015). Applied Multivariate Statistics for the Social Sciences: Analyses with SAS and IBM’s SPSS. Routledge.
Rahman, A. A. U. and Farhana, E. (2021). An empirical study of bugs in COVID-19 software projects. J. Softw. Eng. Res. Dev., 9:3–1.
Santos, I., M. Melo, S., S. L. Souza, P., and R. S. Souza, S. (2022). A survey on the practices of software testing: A look into Brazilian companies. Journal of Software Engineering Research and Development, 10:11:1–11:15.
Segal, J. (2009). Some challenges facing software engineers developing software for scientists. In 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering, pages 9–14. IEEE.
Sjøberg, D. I., Hannay, J. E., Hansen, O., Kampenes, V. B., Karahasanovic, A., Liborg, N.-K., and Rekdal, A. C. (2005). A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733–753.
Smith, A. M., Katz, D. S., and Niemeyer, K. E. (2016). Software citation principles. PeerJ Computer Science, 2:e86.
Sochat, V., May, N., Cosden, I., Martinez-Ortiz, C., and Bartholomew, S. (2022). The research software encyclopedia: A community framework to define research software. Journal of Open Research Software.
Sousa, A. M. d. M. B. (2002). Guia Prático de Utilização do SPSS: Análise de Dados para Ciências Sociais e Psicologia.
Stol, K.-J., Ralph, P., and Fitzgerald, B. (2016). Grounded theory in software engineering research: A critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering, pages 120–131.
Sue, V. M. and Ritter, L. A. (2012). Conducting Online Surveys. Sage.
Tilley, S., Huang, S., and Payne, T. (2003). On the challenges of adopting ROTS software. In 3rd International Workshop on Adoption-Centric Software Engineering.
Van Vliet, H., Van Vliet, H., and Van Vliet, J. (2008). Software Engineering: Principles and Practice, volume 13. John Wiley & Sons, Hoboken, NJ.
Velicer, E. G. W. (1988). Relation of sample size to the stability of component patterns. Psychological Bulletin, 103(2):265.
Wagner, S., Mendez, D., Felderer, M., Graziotin, D., and Kalinowski, M. (2020). Challenges in survey research. In Contemporary Empirical Methods in Software Engineering, pages 93–125. Springer.
Wiese, I., Polato, I., and Pinto, G. (2019). Naming the pain in developing scientific software. IEEE Software, 37(4):75–82.
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3(1):1–9.
Wohlin, C., Kalinowski, M., Felizardo, K. R., and Mendes, E. (2022). Successful combination of database search and snowballing for identification of primary studies in systematic literature studies. Information and Software Technology, 147:106908.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering. Springer Science & Business Media.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Erica Mourão, Daniela Trevisan, José Viterbo, Ana Paula Barbosa Sobral, Mônica da Silva, Carlos Eduardo Pantoja
This work is licensed under a Creative Commons Attribution 4.0 International License.
