On the “fairness” of search-based software engineering: Investigating the capability of a scalarizing-based function to control the software metrics’ influence on optimization process
DOI:
https://doi.org/10.5753/jserd.2025.3906Keywords:
Scalarizing Function, Software Engineering Metrics, Search-Based Software EngineeringAbstract
Search-Based Software Engineering (SBSE) aims to transform Software Engineering (SE) problems into search problems by defining a fitness function that guides the search for an optimal or sub-optimal solution. However, designing a fitness function that provides equitable relevance (or weight) to every SE metric associated with an SBSE problem is an assumption that may be challenging. This issue derives from the several properties related to SE metric value domains that can induce the search process to privilege specific metrics over others, misleading to suboptimal outcomes. To deal with this problem, this work proposes a mathematical model based on the scalarization function concept to better control each metric’s relevance in the search process. Our empirical study comprises two computational experiments. The first experiment aimed to evaluate the proposed scalarizing-based approach’s control capability over the SE metrics in a scenario where all metrics should have the same relevance, while the second experiment covers the scenario where metrics do not necessarily should have the same relevance. The results demonstrate the importance of properly considering the impact, nature, and value range of SE metrics in the search process and the effectiveness of the proposed model in controlling SE metric relevance in different scenarios. This research makes three significant contributions. Firstly, we empirically highlight the importance of properly considering the relevance of individual SE metrics in the search process. Secondly, we propose a generic mathematical model based on scalarizing functions to cope with the normalization process and can be applied to a wide range of SBSE problems. Finally, we show that the our scalarizing approach is capable of guiding search-based process not only in the scenario where all metrics relevance must be equal, but also in the variation of the relevance alongside the optimization process, which is quite important for the design of fitness functions in SBSE.
Downloads
References
Alizadeh, V., Fehri, H., and Kessentini, M. (2019). Less is more: From multi-objective to mono-objective refactoring via developer’s knowledge extraction. In 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 181–192. IEEE.
Almeida Neto, A. D., Pereira, R. L., and De Oliveira, R. C. L. (2022). Application of a genetic algorithm with social interaction to search based testing. In 2022 IEEE Latin American Conference on Computational Intelligence (LACCI), pages 1–6. IEEE.
Arcuri, A. and Briand, L. (2014). A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 24(3):219–250.
Arcuri, A. and Fraser, G. (2011). On parameter tuning in search based software engineering. In International Symposium on Search Based Software Engineering, pages 33–47. Springer.
Arcuri, A. and Fraser, G. (2013). Parameter tuning or default values? an empirical investigation in search-based software engineering. Empirical Software Engineering, 18:594–623.
Augusto, O. B., Bennis, F., and Caro, S. (2012). A new method for decision making in multi-objective optimization problems. Pesquisa Operacional, 32(2):331–369.
Bagnall, A., Rayward-Smith, V., and Whittley, I. (2001). The next release problem. Information and Software Technology, 43(14):883 – 890.
Bansiya, J. and Davis, C. G. (2002). A hierarchical model for object-oriented design quality assessment. IEEE Transactions on software engineering, 28(1):4–17.
Barros, M. and Dias-Neto, A. (2011). Threats to validity in search-based software engineering empirical studies. Relatórios Técnicos do DIA/UNIRIO, (0006).
Bleiweiss, A. (2008). Gpu accelerated pathfinding. In Proceedings of the 23rd ACM SIGGRAPH/EUROGRAPHICS Symposium on Graphics Hardware, GH ’08, pages 65–74, Aire-la-Ville, Switzerland, Switzerland. Eurographics Association.
Chang, C. K., Christensen, M. J., and Zhang, T. (2001). Genetic algorithms for project management. Annals of Software Engineering, 11(1):107–139.
Chen, T. and Li, M. (2022). The weights can be harmful: Pareto search versus weighted search in multi-objective search-based software engineering. ACM Transactions on Software Engineering and Methodology.
Chisalita-Cretu, C. (2014). First results on the evolutionary solution for the strategy-based refactoring set selection problem. International Journal of Emerging Technology and Advanced Engineering.
Colanzi, T. E., Assunção, W. K., Vergilio, S. R., Farah, P. R., and Guizzo, G. (2020). The symposium on search-based software engineering: Past, present and future. Information and Software Technology, 127:106372.
Colanzi, T. E., Vergilio, S. R., Gimenes, I. M., and Oizumi, W. N. (2014). A search-based approach for software product line design. In Proceedings of the 18th International Software Product Line Conference-Volume 1, pages 237–241.
Dantas, A., Yeltsin, I., Araújo, A. A., and Souza, J. (2015). Interactive software release planning with preferences base. In International Symposium on Search Based Software Engineering, pages 341–346. Springer.
Deb, K. and Kalyanmoy, D. (2001). Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley & Sons, Inc., New York, NY, USA.
del Sagrado, J., ÁAguila, I. M., and Orellana, F. J. (2011). Requirements interaction in the next release problem. In Proceedings of the 13th Annual Conference Companion on Genetic and Evolutionary Computation, GECCO ’11, pages 241–242, New York, NY, USA. ACM.
Delahaye, D., Chaimatanan, S., and Mongeau, M. (2019). Simulated annealing: From basics to applications. Handbook of metaheuristics, pages 1–35.
Dreyton, D., Araújo, A. A., Dantas, A., Freitas, Á., and Souza, J. (2015). Search-based bug report prioritization for kate editor bugs repository. In International Symposium on Search Based Software Engineering, pages 295–300. Springer.
Fenton, N. and Bieman, J. (2014). Software metrics: a rigorous and practical approach. CRC press.
Fenton, N. E. (1991). Software Metrics: A Rigorous Approach. Chapman & Hall, Ltd., London, UK, UK.
Fenton, N. E. and Neil, M. (2000). Software metrics: roadmap. In Proceedings of the Conference on the Future of Software Engineering, pages 357–370.
Ferrucci, F., Harman, M., and Sarro, F. (2014). Search-based software project management. Software project management in a changing world, pages 373–399.
Harman, M. (2007a). Automated test data generation using search based software engineering. In Second International Workshop on Automation of Software Test (AST’07), pages 2–2. IEEE.
Harman, M. (2007b). The current state and future of search based software engineering. In 2007 Future of Software Engineering, pages 342–357. IEEE Computer Society.
Harman, M. (2011). Software engineering meets evolutionary computation. Computer, 44(10):31–39.
Harman, M. and Clark, J. (2004). Metrics are fitness functions too. In Software Metrics, 2004. Proceedings. 10th International Symposium on, pages 58–69. IEEE.
Harman, M. and Jones, B. F. (2001). Search-based software engineering. Information and Software Technology, 43(14):833 – 839.
Harman, M., Mansouri, S. A., and Zhang, Y. (2009). Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Department of Computer Science, King’s College London, Tech. Rep. TR-09-03.
Harman, M., Mansouri, S. A., and Zhang, Y. (2012). Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR), 45(1):1–61.
Hughes, E. J. (2005). Evolutionary many-objective optimisation: many once or one many? In 2005 IEEE congress on evolutionary computation, volume 1, pages 222–227. IEEE.
Ishibuchi, H. and Nojima, Y. (2007). Optimization of scalarizing functions through evolutionary multiobjective optimization. In International Conference on Evolutionary Multi-Criterion Optimization, pages 51–65. Springer.
Ishibuchi, H., Sakane, Y., Tsukamoto, N., and Nojima, Y. (2009a). Adaptation of scalarizing functions in moea/d: An adaptive scalarizing function-based multiobjective evolutionary algorithm. In International Conference on Evolutionary Multi-Criterion Optimization, pages 438–452. Springer.
Ishibuchi, H., Tsukamoto, N., and Nojima, Y. (2009b). Empirical analysis of using weighted sum fitness functions in nsga-ii for many-objective 0/1 knapsack problems. In 2009 11th International Conference on Computer Modelling and Simulation, pages 71–76. IEEE.
Jiang, S., Ong, Y.-S., Zhang, J., and Feng, L. (2014). Consistencies and contradictions of performance metrics in multiobjective optimization. IEEE transactions on cybernetics, 44(12):2391–2404.
Karim, M. R. and Ruhe, G. (2014). Bi-objective genetic search for release planning in support of themes. In Search-Based Software Engineering: 6th International Symposium, SSBSE 2014, Fortaleza, Brazil, August 26-29, 2014. Proceedings 6, pages 123–137. Springer.
Kasimbeyli, R., Ozturk, Z. K., Kasimbeyli, N., Yalcin, G. D., and Erdem, B. I. (2019). Comparison of some scalarization methods in multiobjective optimization. Bulletin of the Malaysian Mathematical Sciences Society, 42(5):1875–1905.
Lightner, M. and Director, S. (1981). Multiple criterion optimization for the design of electronic circuits. IEEE Transactions on Circuits and Systems, 28(3):169–179.
Maashi, M. S. (2022). A comprehensive review of software testing methodologies based on search-based software engineering. Webology (ISSN: 1735-188X), 19(1).
Malhotra, R. (2015). Empirical research in software engineering : concepts, analysis, and applications. CRC Press.
Mann, H. B. and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60.
Miettinen, K. (1999). Nonlinear multiobjective optimization, volume 12. Springer Science & Business Media.
Miettinen, K. and Mäkelä, M. M. (2002a). On scalarizing functions in multiobjective optimization. OR Spectrum, 24(2):193–213.
Miettinen, K. and Mäkelä, M. M. (2002b). On scalarizing functions in multiobjective optimization. OR spectrum, 24:193–213.
Mohan, M. and Greer, D. (2108). A survey of search-based refactoring for software maintenance. Journal of Software Engineering Research and Development, 6:1–52.
Nuh, J. A., Koh, T. W., Baharom, S., Osman, M. H., and Kew, S. N. (2021). Performance evaluation metrics for multi-objective evolutionary algorithms in search-based software engineering: Systematic literature review. Applied Sciences, 11(7):3117.
O’Keeffe, M. and Cinnéide, M. . (2008). Search-based refactoring for software maintenance. Journal of Systems and Software, 81(4):502 – 516. Selected papers from the 10th Conference on Software Maintenance and Reengineering (CSMR 2006).
Ouni, A. (2020). Search based software engineering: challenges, opportunities and recent applications. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pages 1114–1146.
Pérez-Piqueras, V., Bermejo, P., and Gámez, J. A. (2023). Feda-nrp: a fixed-structure multivariate estimation of distribution algorithm to solve the multi-objective next release problem with requirements interactions. Engineering Applications of Artificial Intelligence, 124:106555.
Purshouse, R. C. and Fleming, P. J. (2003). Evolutionary many-objective optimisation: An exploratory analysis. In The 2003 Congress on Evolutionary Computation, 2003. CEC’03., volume 3, pages 2066–2073. IEEE.
Rahimi, I., Gandomi, A. H., Nikoo, M. R., and Chen, F. (2023). A comparative study on evolutionary multiobjective algorithms for next release problem. Applied Soft Computing, 144:110472.
Rodriguez, D. V. and Carver, D. L. (2020). Multi-objective information retrieval-based nsga-ii optimization for requirements traceability recovery. In 2020 IEEE International Conference on Electro Information Technology (EIT), pages 271–280. IEEE.
Roque, L., Araújo, A. A., Dantas, A., Saraiva, R., and Souza, J. (2016). Human resource allocation in agile software projects based on task similarities. In Search Based Software Engineering: 8th International Symposium, SSBSE 2016, Raleigh, NC, USA, October 8-10, 2016, Proceedings 8, pages 291–297. Springer.
Ruiz, A. B., Saborido, R., and Luque, M. (2015). A preference-based evolutionary algorithm for multiobjective optimization: the weighting achievement scalarizing function genetic algorithm. Journal of Global Optimization, 62(1):101–129.
Saidani, I., Ouni, A., Chouchen, M., and Mkaouer, M. W. (2020). On the prediction of continuous integration build failures using search-based software engineering. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pages 313–314.
Samli, R., Aydın, Z. B. G., and Yücel, U. O. (2020). Measurement in software engineering: The importance of software metrics. In Applications and Approaches to ObjectOriented Software Design: Emerging Research and Opportunities, pages 166–182. IGI Global.
Saraiva, R., Araújo, A. A., Dantas, A., Yeltsin, I., and Souza, J. (2017). Incorporating decision maker’s preferences in a multi-objective approach for the software release planning. Journal of the Brazilian Computer Society, 23(1):11.
Sharma, D. and Sharma, G. (2023). Systematic literature review of search-based software engineering techniques for code modularization/remodularization. Computational Intelligence Applications for Software Engineering Problems, pages 241–266.
Simons, C., Singer, J., and White, D. R. (2015). Search-based refactoring: Metrics are not enough. In Search-Based Software Engineering: 7th International Symposium, SSBSE 2015, Bergamo, Italy, September 5-7, 2015, Proceedings 7, pages 47–61. 1 Springer.
Singh, G., Singh, D., and Singh, V. (2011). A study of software metrics. IJCEM International Journal of Computational Engineering & Management, 11:22–27.
Sommerville, I. (2015). Software Engineering. Pearson, 10th edition.
Thiele, L., Miettinen, K., Korhonen, P. J., and Molina, J. (2009). 2 A preference-based evolutionary algorithm for multi-objective optimization. Evolutionary computation, 17(3):411–436.
van Laarhoven, P. J. M. and Aarts, E. H. L. (1987). chapter Simulated annealing, pages 7–15. Springer Netherlands, Dordrecht.
Van Veldhuizen, D. A. (1999). Multiobjective evolutionary algorithms: classifications, analyses, and new innovations. 3 Air Force Institute of Technology.
Vargha, A. and Delaney, H. D. (2000). A critique and improvement of the cl common language effect size statistics of mcgraw and wong. Journal of Educational and Behavioral Statistics, 25(2):101–132.
Wang, S., Ali, S., Yue, T., Li, Y., and Liaaen, M. (2016). A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering. In Proceedings of the 38th international conference on software engineering, pages 631–642.
Wang, Y. (2003). The measurement theory for software engineering. In CCECE 2003 - Canadian Conference on Electrical and Computer Engineering. Toward a Caring and Humane Technology (Cat. No.03CH37436), volume 2, pages 1321–1324 vol.2.
Wierzbicki, A. P. (1980). The use of reference objectives in multiobjective optimization. In Fandel, G. and Gal, T., editors, Multiple Criteria Decision Making Theory and Application, pages 468–486, Berlin, Heidelberg. Springer Berlin Heidelberg.
Zhang, Q. and Li, H. (2007). Moea/d: A multiobjective evolutionary algorithm based on decomposition. 4 IEEE Transactions on evolutionary computation, 11(6):712–731.
Zhang, Y. (2012). Repository of publications on search-based software engineering.
Zhang, Y., Harman, M., and Mansouri, S. A. (2007). 5 The multi-objective next release problem. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation, GECCO ’07, pages 1129–1137, New York, NY, USA. ACM.
Zhang, Y., Harman, M., Ochoa, G., Ruhe, G., and Brinkkemper, S. (2014). 6 An empirical study of meta-and hyperheuristic search for multi-objective release planning. RN, 14(07).
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Italo Yeltsin, Allysson Allex Araújo, Altino Dantas, Pamella Soares, Raphael Saraiva, Jerffeson Souza
This work is licensed under a Creative Commons Attribution 4.0 International License.
