Requirements Engineering for Machine Learning-Based AI Systems: A Tertiary Study
DOI:
https://doi.org/10.5753/jserd.2025.4892Keywords:
Requirements Engineering, Machine Learning, AI Systems, Tertiary StudyAbstract
Context: In the last decade, machine learning (ML) components have become more and more present in contemporary software systems. A number of secondary literature studies reports challenges impacting on the development of ML-based systems, including those for requirements engineering (RE) activities. Motivation/Problem: Synthesizing secondary literature contributes to building knowledge and reaching conclusions about the existing RE approaches for ML-based systems (RE4ML), besides the novelty of a tertiary study on that subject. Objective: Through a tertiary study protocol we elaborated on, this paper synthesizes the body of evidence present in secondary studies on RE4ML systems. Method: We followed well-accepted guidelines about tertiary study protocols, including automatic search, the snowballing technique, selection and quality criteria, and data extraction and synthesis. Results: Nine secondary studies on RE4ML systems were aligned to our tertiary study's goal. We extracted and summarized the requirements elicitation, analysis, specification, validation, and management techniques for ML-based systems as well as the great challenges identified. Finally, we contribute with a nine-item research agenda to direct current and future searches to fill the gaps found. Conclusions: We conclude that RE has not been left aside in ML research, however, there are still challenges to be overcome, such as dealing with non-functional requirements, collaboration between stakeholders, and research in an industrial environment.
Downloads
References
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., and Grundy, J. (2023). Requirements engineering for artificial intelligence systems: A systematic mapping study. Information and Software Technology, 159:107176.
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., and Grundy, J. (2021). What’s up with requirements engineering for artificial intelligence systems? In Proceedings of the 29th IEEE International Requirements Engineering Conference (RE 2021), pages 1–12. IEEE.
Alves, A., Kalinowski, M., Giray, G., Méndez Fernández, D., Lavesson, N., Azevedo, K., Villamizar, H., Escovedo, T., Lopes, H., Biffl, S., Musil, J., Felderer, M., Wagner, S., Baldassarre, T., and Gorschek, T. (2023). Status quo and problems of requirements engineering for machine learning: Results from an international survey. In Product-Focused Software Process Improvement: 24th International Conference, PROFES 2023, Dornbirn, Austria, December 10–13, 2023, Proceedings, Part I, page 159–174, Berlin, Heidelberg. Springer-Verlag.
Ampatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., and Chatzigeorgiou, A. (2019). Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Information and Software Technology, 106:201–230.
Centre for Reviews and Dissemination (2002). The database of abstracts of reviews of effects (dare). Effectiveness Matters, 6(2):1–4.
Clement, T., Kemmerzell, N., Abdelaal, M., and Amberg, M. (2023). XAIR: A systematic metareview of explainable AI (XAI) aligned to the software development process. Machine Learning and Knowledge Extraction, 5(1):78–108.
Committee, I. C. S. P. P. et al. (2022). Swebok: Guide to the software engineering body of knowledge, 2022 version beta. IEEE Computer Society.
Costal, D., Farré, C., Franch, X., and Quer, C. (2021). How tertiary studies perform quality assessment of secondary studies in software engineering. In Proceedings of the XXIV Iberoamerican Conference on Software Engineering (ESELAW@CIbSE), pages 1–14.
Cruzes, D. S. and Dybå, T. (2011). Research synthesis in software engineering: A tertiary study. Information and Software Technology, 53(5):440–455.
Dalpiaz, F. and Niu, N. (2020). Requirements engineering in the days of artificial intelligence. IEEE Software, 37(4):7–10.
Dey, S. and Lee, S.-W. (2021). Multilayered review of safety approaches for machine learning-based systems in the days of AI. Journal of Systems and Software, 176:110941.
Fabbri, S., Felizardo, K., Ferrari, F., Hernandes, E., Octaviano, F., Nakagawa, E., and Maldonado, J. (2013). Externalising tacit knowledge of the systematic review process. Software, IET, 7:298–307.
Giray, G. (2021). A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software, 180:111031.
Goodell, J. W., Kumar, S., Lim, W. M., and Pattnaik, D. (2021). Artificial intelligence and machine learning in finance: Identifying foundations, themes, and research clusters from bibliometric analysis. Journal of Behavioral and Experimental Finance, 32:100577.
Hu, B. C., Salay, R., Czarnecki, K., Rahimi, M., Selim, G. M. K., and Chechik, M. (2020). Towards requirements specification for machine-learned perception based on human performance. In 2020 IEEE Seventh International Workshop on Artificial Intelligence for Requirements Engineering (AIRE), pages 48–51. IEEE.
Ishikawa, F. and Yoshioka, N. (2019). How do engineers perceive difficulties in engineering of machine-learning systems?-questionnaire survey. In 2019 IEEE/ACM Joint 7th International Workshop on Conducting Empirical Studies in Industry (CESI) and 6th International Workshop on Software Engineering Research and Industrial Practice (SER&IP), pages 2–9. IEEE.
Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., and Wang, Y. (2017). Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology, 2(4):230–243.
Kitchenham, B. (2004). Procedures for performing systematic reviews. Technical Report TR/SE-0401, Keele University, UK.
Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., and Linkman, S. (2010). Systematic literature reviews in software engineering – a tertiary study. Information and Software Technology, 52(8):792–805.
Kucak, D., Juricic, V., and Dambic, G. (2018). Machine learning in education: A survey of current research trends. In Proceedings of the 29th DAAAM International Symposium on Intelligente Manufacturing and Automation, volume 29, pages 406–410. DAAAM International Vienna.
Kudo, T. N., Bulcão-Neto, R. F., and Vincenzi, A. M. (2020a). Requirement patterns: a tertiary study and a research agenda. IET Software, 14(1):18–26.
Kudo, T. N., Bulcão-Neto, R. F., and Vincenzi, A. M. (2020b). Requirement patterns: a tertiary study and a research agenda. IET Software, 14(1):18–26.
Kumeno, F. (2020). Software engineering challenges for machine learning applications: A literature review. Intelligent Decision Technologies, 13(4):463–476.
Kuwajima, H., Yasuoka, H., and Nakae, T. (2020). Engineering problems in machine learning systems. Machine Learning, 109(5):1103–1126.
Lakha, B., Bhetwal, K., and Eisty, N. U. (2023). Analysis of software engineering practices in general software and machine learning startups. In 2023 IEEE/ACIS 21st International Conference on Software Engineering Research, Management and Applications (SERA). IEEE.
Lewis, G. A., Bellomo, S., and Ozkaya, I. (2021). Characterizing and detecting mismatch in machine-learning-enabled systems. In 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Engineering for AI (WAIN), pages 133–140. IEEE.
Martínez-Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., and Wagner, S. (2022). Software engineering for ai-based systems: A survey. ACM Trans. Softw. Eng. Methodol., 31(2).
Nahar, N., Zhou, S., Lewis, G., and Kästner, C. (2022). Collaboration challenges in building ML-enabled systems: communication, documentation, engineering, and process. In Proceedings of the 44th International Conference on Software Engineering, ICSE ’22, page 413–425, New York, NY, USA. Association for Computing Machinery.
Villamizar, H., Escovedo, T., and Kalinowski, M. (2021). Requirements engineering for machine learning: A systematic mapping study. In 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pages 29–36. IEEE.
Vogelsang, A. and Borg, M. (2019). Requirements engineering for machine learning: Perspectives from data scientists. In 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), pages 245–251.
Washizaki, H. (2024). Guide to the software engineering body of knowledge (swebok guide), version 4.0. IEEE Computer Society, Waseda University, Japan.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, New York, NY, USA. Association for Computing Machinery.
Yoshioka, N., Husen, J. H., Tun, H. T., Chen, Z., Washizaki, H., and Fukazawa, Y. (2021). Landscape of requirements engineering for machine learning-based AI systems. In Proceedings of the 28th Asia-Pacific Software Engineering Conference Workshops (APSEC Workshops 2021), pages 5–8. IEEE.
Zaharia, M., Chen, A., Davidson, A., Ghodsi, A., Hong, S. A., Konwinski, A., Murching, S., Nykodym, T., Ogilvie, P., Parkhe, M., et al. (2018). Accelerating the machine learning lifecycle with MLflow. IEEE Data Engineering Bulletin, 41(4):39–45.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Mariana Crisostomo Martins, Livia Mancine C. Campos, João Lucas R. Soares, Taciana Novo Kudo, Renato F. Bulcão-Neto
This work is licensed under a Creative Commons Attribution 4.0 International License.
