Uniqueness of suspiciousness scores: towards boosting evolutionary fault localization
DOI:
https://doi.org/10.5753/jserd.2024.3651Keywords:
Fault Localization, SBSE, Search Based Software Engineering, Software Debugging, Genetic ProgrammingAbstract
Context. Software is subject to the presence of faults, which impacts its quality as well as production and maintenance costs. Evolutionary fault localization uses data from the test activity (test spectra) as a source of information about defects, and its automation aims to obtain better accuracy and lower software repair cost. Motivation. Our analysis identified that the test spectra commonly used in research exhibit a high ratio of sample repetition, which impairs the training and evolution of Genetic Programming (GP) evolved heuristics. Problem. We investigate whether evolutionary training based on the uniqueness of suspiciousness scores can increase the ability to find software faults (defects), even in repeat-sample scenarios. Specifically, we examine whether the GP-evolved heuristic, which is based on the distinguishability of program elements in terms of faults, is really competitive. Methodology. The investigation formalized hypotheses, introduced three training strategies to guide the research and carried out an experimental evaluation, aiming to reach conclusions regarding the assessment of research questions and hypotheses. Analysis. The results have shown the competitiveness of all the proposed training strategies through evaluation metrics commonly used in the research field. Conclusion. Statistical analyses confirmed that the uniqueness of suspiciousness scores guides the generation of superior heuristics for fault localization.
Downloads
References
Abreu, R., Zoeteweij, P., and c. Van Gemund, A. J. (2006). An evaluation of similarity coefficients for software fault localization. In 2006 12th Pacific Rim International Symposium on Dependable Computing (PRDC’06), pages 39–46.
Abreu, R., Zoeteweij, P., and Gemund, A. J. C. v. (2009). Spectrum-based multiple fault localization. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering, ASE ’09, pages 88–99, Washington, DC, USA. IEEE Computer Society. [link]
Abreu, R., Zoeteweij, P., and van Gemund, A. J. C. (2007). On the accuracy of spectrum-based fault localization. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007), pages 89–98.
Arcuri, A. and Briand, L. (2014). A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliab., 24(3):219–250.
Barros, M. d. O. and Dias-Neto, A. C. (2011). 0006/2011 - threats to validity in search-based software engineering empirical studies. RelaTe-DIA, 5(1).
Campos, J., Abreu, R., Fraser, G., and d’Amorim, M. (2013). Entropy-based test generation for improved fault localization. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 257–267. [link]
Choi, K., Sohn, J., and Yoo, S. (2018). Learning fault localisation for both humans and machines using multi-objective gp. In Colanzi, T. E. and McMinn, P., editors, Search-Based Software Engineering, pages 349–355, Cham. Springer International Publishing.
De-Freitas, D. M., Leitao-Junior, P. S., Camilo-Junior, C. G., and Harrison, R. (2018). Mutation-Based Evolutionary Fault Localisation. In 2018 IEEE Congress on Evolutionary Computation (CEC), pages 1–8. [link]
Hailpern, B. and Santhanam, P. (2002). Software debugging, testing, and verification. IBM Systems Journal, 41(1):4–12. [link]
Hirsch, T. and Hofer, B. (2022). A systematic literature review on benchmarks for evaluating debugging approaches. Journal of Systems and Software, 192. Cited by: 0; All Open Access, Hybrid Gold Open Access. https://doi.org/10.1016/j.jss.2022.111362
Janssen, T., Abreu, R., and van Gemund, A. J. (2009). Zoltar: A spectrum-based fault localization tool. In Proceedings of the 2009 ESEC/FSE Workshop on Software Integration and Evolution @ Runtime, SINTER ’09, pages 23–30, New York, NY, USA. ACM. [link]
Jones, J. A., Harrold, M. J., and Stasko, J. T. (2009). Visualization for Fault Localization. In in Proceedings of ICSE 2001 Workshop on Software Visualization, pages 71–75.
Just, R., Jalali, D., and Ernst, M. D. (2014). Defects4j: A database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, page 437–440, New York, NY, USA. Association for Computing Machinery. https://doi.org/10.1145/2610384.2628055
Koza, J. R. (1992). Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge, MA, USA.
Leitao-Junior, P. S., Freitas, D. M., Vergilio, S. R., Camilo-Junior, C. G., and Harrison, R. (2020). Search-based fault localisation: A systematic mapping study. Information and Software Technology, 123:106295. https://doi.org/10.1016/j.infsof.2020.106295
Lu, S., Li, Z., Qin, F., Tan, L., Zhou, P., and Zhou, Y. (2005). Bugbench: Benchmarks for evaluating bug detection tools. In Workshop on the Evaluation of Software Defect Detection Tools. [link]
Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., and Keller, B. (2017). Evaluating and improving fault localization. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pages 609–620. https://doi.org/10.1109/ICSE.2017.64
Silva-Junior, D., Leitao-Junior, P., Dantas, A., Camilo-Junior, C., and Harrison, R. (2020). Data-flow-based evolutionary fault localization. In Proceedings of the ACM Symposium on Applied Computing, pages 1963–1970. https://doi.org/10.1145/3341105.3373941
Sohn, J. and Yoo, S. (2017). FLUCCS: Using Code and Change Metrics to Improve Fault Localization. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2017, pages 273–283, New York, NY, USA. Association for Computing Machinery. event-place: Santa Barbara, CA, USA. https://doi.org/10.1145/3092703.3092726
Wang, S., Lo, D., Jiang, L., Lucia, and Lau, H. (2011). Search-based fault localization. In 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings, pages 556–559. [link]
Wong, W., Gao, R., Li, Y., Abreu, R., and Wotawa, F. (2016). A survey on software fault localization. IEEE Transactions on Software Engineering, 42(8):707–740.
Yoo, S. (2012). Evolving human competitive spectra-based fault localisation techniques. In Fraser, G. and Teixeira de Souza, J., editors, Search Based Software Engineering, pages 244–258, Berlin, Heidelberg. Springer Berlin Heidelberg.
Yoo, S., Xie, X., Kuo, F.-C., Chen, T. Y., and Harman, M. (2017). Human competitiveness of genetic programming in spectrum-based fault localisation: Theoretical and empirical analysis. ACM Trans. Softw. Eng. Methodol., 26(1). https://doi.org/10.1145/3031672
Zheng, Y., Wang, Z., Fan, X., Chen, X., and Yang, Z. (2018). Localizing multiple software faults based on evolution algorithm. Journal of Systems and Software, 139:107–123. https://doi.org/10.1016/j.jss.2018.01.034
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Willian de Jesus Ferreira, Plinio S. Leitao-Junior, Diogo Machado de Freitas, Deuslirio Silva-Junior, Rachel Harrison
This work is licensed under a Creative Commons Attribution 4.0 International License.
