Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil: An Extended Analysis
DOI:
https://doi.org/10.5753/jserd.2025.5689Keywords:
Requirements Engineering, Machine Learning, Survey, BrazilAbstract
[Context] In Brazil, 41% of companies use machine learning (ML) to some extent. However, several challenges have been reported when engineering ML-enabled systems, including unrealistic customer expectations and vagueness in ML problem specifications. Literature suggests that Requirements Engineering (RE) practices and tools may help to alleviate these issues, yet there is insufficient understanding of RE’s practical application and its perception among practitioners. [Goal] This study aims to investigate the application of RE in developing ML-enabled systems in Brazil, creating an overview of current practices, perceptions, and problems in the Brazilian industry. [Method] To this end, we extracted and analyzed data from an international survey focused on ML-enabled systems, concentrating specifically on responses from practitioners based in Brazil. We analyzed the cluster of RE-related answers gathered from 72 practitioners involved in data-driven projects. We conducted quantitative statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative studies on the reported problems involving open and axial coding procedures. [Results] Our findings highlight distinct RE implementation aspects in Brazil’s ML projects. For instance, (i) RE-related tasks are predominantly conducted by data scientists; (ii) the most common techniques for eliciting requirements are interviews and workshop meetings; (iii) there is a prevalence of interactive notebooks in requirements documentation; (iv) practitioners report problems that include a poor understanding of the problem to solve and the business domain, low customer engagement, and difficulties managing stakeholders expectations. Our analysis suggests that development methodology plays a role in these challenges. Agile methods appear to facilitate the management of customer expectations compared to traditional approaches; however, they also appear to introduce greater difficulties in problem understanding and customer involvement. [Conclusion] These results provide an understanding of RE-related practices and challenges in the Brazilian ML industry, helping to guide research and initiatives toward improving the maturity of RE for ML-enabled system projects.
Downloads
References
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., and Grundy, J. (2021). What’s up with requirements engineering for artificial intelligence systems? In 2021 IEEE 29th International Requirements Engineering Conference, pages 1–12. IEEE.
Aho, T., Sievi-Korte, O., Kilamo, T., Yaman, S., and Mikkonen, T. (2020). Demystifying data science projects: A look on the people and process of data science today. In Product-Focused Software Process Improvement: 21st International Conference, PROFES 2020, Turin, Italy, November 25–27, 2020, Proceedings 21, pages 153–167. Springer.
Alves, A. P. S., Kalinowski, M., Giray, G., Mendez, D., Lavesson, N., Azevedo, K., Villamizar, H., Escovedo, T., Lopes, H., Biffl, S., et al. (2023). Status quo and problems of requirements engineering for machine learning: Results from an international survey. In Product-Focused Software Process Improvement: 24st International Conference, PROFES 2023, Dornbirn, Austria, December 10–13, pages 153–167. Springer.
Alves, A. P. S., Kalinowski, M., Mendez, D., Villamizar, H., Azevedo, K., Escovedo, T., and Lopes, H. (2024). Industrial practices of requirements engineering for ml-enabled systems in brazil. In Anais do XXXVIII Simpósio Brasileiro de Engenharia de Software, pages 224–233, Porto Alegre, RS, Brasil. SBC.
Alves, A. P. S., Kalinowski, M., Méndez, D., Villamizar, H., Escovedo, T., and Lopes, H. (2025). Artifacts: Industrial practices of requirements engineering for ml-enabled systems in brazil: An extended analysis. available at: https://doi.org/10.5281/zenodo.15029764.
Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., and Zimmermann, T. (2019). Software engineering for machine learning: A case study. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice, pages 291–300. IEEE.
Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., and Vardoulakis, L. M. (2020). A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–12.
Challa, H., Niu, N., and Johnson, R. (2020). Faulty requirements made valuable: On the role of data quality in deep learning. In 2020 IEEE 7th International Workshop on Artificial Intelligence for Requirements Engineering, pages 61–69. IEEE.
Correia, J. a. L., Pereira, J. A., Mello, R., Garcia, A., Fonseca, B., Ribeiro, M., Gheyi, R., Kalinowski, M., Cerqueira, R., and Tiengo, W. (2021). Brazilian data scientists: Revealing their challenges and practices on machine learning model development. In Proceedings of the XIX Brazilian Symposium on Software Quality, SBQS ’20, New York, NY, USA. Association for Computing Machinery.
Dalpiaz, F. and Niu, N. (2020). Requirements engineering in the days of artificial intelligence. IEEE Software, 37(4):7–10.
Damian, D. (2007). Stakeholders in global requirements engineering: Lessons learned from practice. IEEE software, 24(2):21–27.
Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC.
Fernández, D. M., Wagner, S., Kalinowski, M., Felderer, M., Mafra, P., Vetrò, A., Conte, T., Christiansson, M.-T., Greer, D., Lassenius, C., et al. (2017). Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice. Empirical Software Engineering, 22:2298–2338.
Fry, H. (2018). Hello World: How to be Human in the Age of the Machine. Random House.
Gartner (2020). Gartner identifies the top strategic technology trends for 2021.
Giray, G. (2021). A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software, 180:111031.
Habibullah, K. M., Gay, G., and Horkoff, J. (2023). Non-functional requirements for machine learning: Understanding current use and challenges among practitioners. Requirements Engineering, 28(2):283–316.
Herrmann, A. (2013). Requirements engineering in practice: There is no requirements engineer position. In Doerr, J. and Opdahl, A. L., editors, Requirements Engineering: Foundation for Software Quality, pages 347–361, Berlin, Heidelberg. Springer Berlin Heidelberg.
Ishikawa, F. and Matsuno, Y. (2020). Evidence-driven requirements engineering for uncertainty of machine learning-based systems. In 2020 IEEE 28th International Requirements Engineering Conference, pages 346–351.
Ishikawa, F. and Yoshioka, N. (2019). How do engineers perceive difficulties in engineering of machine-learning systems? - questionnaire survey. 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP), pages 2–9.
Jordan, M. I. and Mitchell, T. M. (2015). Machine learning: Trends, perspectives, and prospects. Science, 349(6245):255–260.
Kalinowski, M., Card, D. N., and Travassos, G. H. (2012). Evidence-based guidelines to defect causal analysis. IEEE Software, 29(4):16–18.
Kalinowski, M., Escovedo, T., Villamizar, H., and Lopes, H. (2023). Engenharia de Software para Ciência de Dados: Um guia de boas práticas com ênfase na construção de sistemas de Machine Learning em Python. Casa do Código.
Kalinowski, M., Mendes, E., Card, D. N., and Travassos, G. H. (2010). Applying dppi: A defect causal analysis approach using bayesian networks. In Product-Focused Software Process Improvement: 11th International Conference, PROFES 2010, pages 92–106. Springer.
Kalinowski, M., Mendes, E., and Travassos, G. H. (2011). Automating and evaluating probabilistic cause-effect diagrams to improve defect causal analysis. In Product-Focused Software Process Improvement: 12th International Conference, PROFES 2011, pages 232–246. Springer.
Kalinowski, M., Mendez, D., Giray, G., Alves, A. P. S., Azevedo, K., Escovedo, T., Villamizar, H., Lopes, H., Baldassarre, T., Wagner, S., et al. (2025). Naming the pain in machine learning-enabled systems engineering. Information and Software Technology, 187:107866.
Khalajzadeh, H., Abdelrazek, M., Grundy, J., Hosking, J., and He, Q. (2018). A survey of current end-user data analytics tool support. In 2018 IEEE International Congress on Big Data, pages 41–48. IEEE.
Kim, M., Zimmermann, T., DeLine, R., and Begel, A. (2017). Data scientists in software teams: State of the art and challenges. IEEE Transactions on Software Engineering, 44(11):1024–1038.
Kuwajima, H., Yasuoka, H., and Nakae, T. (2020). Engineering problems in ml systems. Machine Learning, 109(5):1103–1126.
Lei, S. and Smith, M. (2003). Evaluation of several nonparametric bootstrap methods to estimate confidence intervals for software metrics. IEEE Transactions on Software Engineering, 29(11):996–1004.
Lewis, G. A., Bellomo, S., and Ozkaya, I. (2021a). Characterizing and detecting mismatch in machine-learning-enabled systems. In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), pages 133–140. IEEE.
Lewis, G. A., Ozkaya, I., and Xu, X. (2021b). Software architecture challenges for ml systems. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 634–638.
Linaker, J., Sulaman, S. M., Höst, M., and de Mello, R. M. (2015). Guidelines for conducting surveys in software engineering v. 1.1. Lund University, 50.
Lunneborg, C. E. (2001). Bootstrap inference for local populations. Therapeutic Innovation & Regulatory Science, 35(4):1327–1342.
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., and Wagner, S. (2022). Software engineering for ai-based systems: a survey. ACM Transactions on Software Engineering and Methodology, 31(2):1–59.
Mitchell, T. M. (1997). Machine learning.
Nahar, N., Zhang, H., Lewis, G., Zhou, S., and Kästner, C. (2023). A meta-summary of challenges in building products with ml components – collecting experiences from 4758+ practitioners. In 2023 IEEE/ACM 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), pages 171–183.
Nahar, N., Zhou, S., Lewis, G., and Kästner, C. (2022). Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process. In Proceedings of the 44th International Conference on Software Engineering, pages 413–425.
Perkel, J. (2018). Why jupyter is data scientists’ computational notebook of choice. Nature, 563:145–146.
Schröer, C., Kruse, F., and Gómez, J. M. (2021). A systematic literature review on applying crisp-dm process model. Procedia Computer Science, 181:526–534.
Sharma, L. and Garg, P. K. (2021). Artificial intelligence: technologies, applications, and challenges.
Stol, K.-J., Ralph, P., and Fitzgerald, B. (2016). Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International Conference on Software Engineering, pages 120–131.
Villamizar, H., Escovedo, T., and Kalinowski, M. (2021). Requirements engineering for machine learning: A systematic mapping study. In 2021 47th Euromicro Conference on Software Engineering and Advanced Applications, pages 29–36.
Villamizar, H., Kalinowski, M., Lopes, H., and Mendez, D. (2024). Identifying concerns when specifying machine learning-enabled systems: A perspective-based approach. Journal of Systems and Software, 213:112053.
Vogelsang, A. and Borg, M. (2019). Requirements engineering for machine learning: Perspectives from data scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops, pages 245–251.
Wagner, S., Fernández, D. M., Felderer, M., and Kalinowski, M. (2017). Requirements engineering practice and problems in agile projects: Results from an international survey. In Proceedings of the XX Iberoamerican Conference on Software Engineering, Buenos Aires, Argentina, May 22-23, 2017, pages 389–402. Curran Associates.
Wagner, S., Fernández, D. M., Felderer, M., Vetrò, A., Kalinowski, M., Wieringa, R., Pfahl, D., Conte, T., Christiansson, M.-T., Greer, D., Lassenius, C., Männistö, T., Nayebi, M., Oivo, M., Penzenstadler, B., Prikladnicki, R., Ruhe, G., Schekelmann, A., Sen, S., Spínola, R., Tuzcu, A., Vara, J. L. D. L., and Winkler, D. (2019). Status quo in requirements engineering: A theory and a global family of surveys. ACM Transactions on Software Engineering and Methodology (TOSEM), 28(2).
Wagner, S., Fernández, D. M., Kalinowski, M., and Felderer, M. (2018). Agile requirements engineering in practice: Status quo and critical problems. CLEI Electronic Journal, 21(1):6–1.
Wagner, S., Mendez, D., Felderer, M., Graziotin, D., and Kalinowski, M. (2020). Challenges in survey research. Contemporary Empirical Methods in Software Engineering, pages 93–125.
Wang, C., Cui, P., Daneva, M., and Kassab, M. (2018). Understanding what industry wants from requirements engineers: an exploration of re jobs in canada. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’18, New York, NY, USA. Association for Computing Machinery.
Zimelewicz, E., Kalinowski, M., Mendez, D., Giray, G., Santos Alves, A. P., Lavesson, N., Azevedo, K., Villamizar, H., Escovedo, T., Lopes, H., Biffl, S., Musil, J., Felderer, M., Wagner, S., Baldassarre, T., and Gorschek, T. (2024). Ml-enabled systems model deployment and monitoring: Status quo and problems. In Software Quality as a Foundation for Security, pages 112–131, Cham. Springer Nature Switzerland.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Antonio Pedro Santos Alves, Marcos Kalinowski, Daniel Mendez, Hugo Villamizar, Tatiana Escovedo, Helio Lopes
This work is licensed under a Creative Commons Attribution 4.0 International License.
