Test smell refactoring revisited: What can internal quality attributes and developers’ experience tell us?
DOI:
https://doi.org/10.5753/jserd.2023.3195Keywords:
Test Smells, Test Refactoring, Software Quality, Developers’ PerceptionAbstract
Test smells represent a set of poorly designed tests, which can harm test code maintainability. Although fundamental steps to understand test smells have been taken, there is still an evident lack of studies evaluating the impact of test smell refactoring from the perspective of internal quality attributes, such as size, cohesion, coupling, and complexity. In addition, the literature still lacks research that addresses the difficulties developers face during test smell refactoring. This article investigates the impact of test smell refactoring from a developer’s perspective, considering the internal quality attributes and the background experience. We investigated the perceptions and difficulties encountered by 20 developers while removing five types of test smells in four open-source projects over two months. Through this study, we analyzed: (i) the impact that test smell refactoring has on internal quality attributes; (ii) the developers’ perception of test smells as actual problems within software systems; (iii) the main difficulties developers face during test smell refactoring; (iv) the influence of developers’ experience on assertiveness and refactoring time of test smells, and (v) the effects of refactoring on the test smell density. Our findings can help developers design a prioritization scheme for test smell refactoring and make them aware of the real benefits of test smell refactoring.
Downloads
References
Alenezi, M. and Almustafa, K. (2015). Empirical analysis of the complexity evolution in open-source software systems. International Journal of Hybrid Information Technology, 8(2):257–266.
Alizadeh, V., Kessentini, M., Mkaouer, M. W., Ó Cinnéide, M., Ouni, A., and Cai, Y. (2020). An interactive and dynamic search-based approach to software refactoring recommendations. IEEE Transactions on Software Engineering, 46(9):932–961.
Aljedaani, W., Peruma, A., Aljohani, A., Alotaibi, M., Mkaouer, M. W., Ouni, A., Newman, C. D., Ghallab, A., and Ludi, S. (2021). Test smell detection tools: A systematic mapping study. In Evaluation and Assessment in Software Engineering, EASE 2021, page 170–180, New York, NY, USA. Association for Computing Machinery.
AlOmar, E. A., Peruma, A., Mkaouer, M. W., Newman, C. D., and Ouni, A. (2021). Behind the scenes: On the relationship between developer experience and refactoring. CoRR, abs/2109.11089.
AlOmar, E. A., Peruma, A., Newman, C. D., Mkaouer, M. W., and Ouni, A. (2020). On the relationship between developer experience and refactoring: An exploratory study and preliminary results. In IEEE/ACM 42nd International Conference on Software Engineering Workshops, ICSEW’20, page 342–349, New York, NY, USA. ACM.
Bavota, G., Qusef, A., Oliveto, R., De Lucia, A., and Binkley, D. (2015). Are test smells really harmful? an empirical study. Empirical Software Engineering, 20(4):1052–1094.
Bavota, G., Qusef, A., Oliveto, R., Lucia, A. D., and Binkley, D. W. (2012). An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 28th IEEE International Conference on Software Maintenance, ICSM 2012, Trento, Italy, September 23-28, 2012, pages 56–65. IEEE Computer Society.
Beller, M., Gousios, G., Panichella, A., Proksch, S., Amann, S., and Zaidman, A. (2019). Developer testing in the IDE: patterns, beliefs, and behavior. IEEE Trans. Software Eng., 45(3):261–284.
Beller, M., Gousios, G., Panichella, A., and Zaidman, A. (2015). When, how, and why developers (do not) test in their IDEs. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, page 179–190, New York, NY, USA. Association for Computing Machinery.
Berner, S., Weber, R., and Keller, R. K. (2005). Observations and lessons learned from automated testing. In Proceedings of the 27th International Conference on Software Engineering, ICSE ’05, page 571–579, New York, NY, USA. Association for Computing Machinery.
Bertolino, A. (2007). Software testing research: Achievements, challenges, dreams. In Briand, L. C. and Wolf, A. L., editors, International Conference on Software Engineering, ISCE 2007, Workshop on the Future of Software Engineering, FOSE 2007, May 23-25, 2007, Minneapolis, MN, USA, pages 85–103. IEEE Computer Society.
Bieman, J. M. and Kang, B.-K. (1995). Cohesion and reuse in an object-oriented system. SIGSOFT Softw. Eng. Notes, 20(SI):259–262.
Campos, D., Martins, L., and Machado, I. (2022). An empirical study on the influence of developers’ experience on software test code quality. In Proceedings of the XXI Brazilian Symposium on Software Quality, SBQS ’22, New York, NY, USA. Association for Computing Machinery.
Campos, D., Rocha, L., and Machado, I. (2021). Developers’ perception of the severity of test smells: an empirical study. pages 192–205.
Candea, G., Bucur, S., and Zamfir, C. (2010). Automated software testing as a service. SoCC ’10, page 155–160, New York, NY, USA. Association for Computing Machinery.
Chávez, A., Ferreira, I., Fernandes, E., Cedrim, D., and Garcia, A. (2017). How does refactoring affect internal quality attributes? a multi-project study. In Proceedings of the 31st Brazilian Symposium on Software Engineering, SBES’17, page 74–83, New York, NY, USA. Association for Computing Machinery.
Chidamber, S. R. and Kemerer, C. F. (1991). Towards a metrics suite for object oriented design. SIGPLAN Not., 26(11):197–211.
Damasceno, H., Bezerra, C., Coutinho, E., and Machado, I. (2022). Analyzing test smells refactoring from a developer's perspective. In Proceedings of the XXI Brazilian Symposium on Software Quality, SBQS ’22, New York, NY, USA. Association for Computing Machinery.
Darcy, D. P., Kemerer, C. F., Slaughter, S., and Tomayko, J. E. (2005). The structural complexity of software: An experimental test. IEEE Trans. Software Eng., 31(11):982–995.
Dyer, R., Rajan, H., and Cai, Y. (2012). An exploratory study of the design impact of language features for aspect-oriented interfaces. In Proceedings of the 11th Annual International Conference on Aspect-Oriented Software Development, AOSD ’12, page 143–154, New York, NY, USA. Association for Computing Machinery.
Fernandes, E., Chávez, A., Garcia, A., Ferreira, I., Cedrim, D., da Silva Sousa, L., and Oizumi, W. N. (2020). Refactoring effect on internal quality attributes: What haven’t they told you yet? Inf. Softw. Technol., 126:106347.
Fowler, M. (1999). Refactoring - Improving the Design of Existing Code. Addison Wesley object technology series. Addison-Wesley.
França, A. C. C., da Silva, F. Q. B., and Sharp, H. (2020). Motivation and satisfaction of software engineers. IEEE Trans. Software Eng., 46(2):118–140.
Gall, H., Devanbu, P., Murphy, B., Bird, C., and Nagappan, N. (2010). An analysis of the effect of code ownership on software quality across Windows, Eclipse, and Firefox.
Garousi, V. and Küçük, B. (2018). Smells in software test code: A survey of knowledge in industry and academia. Journal of Systems and Software, 138:52–81.
ISO (2011). IEC 25010: 2011 systems and software engineering–systems and software quality requirements and evaluation (SQuaRE)–system and software quality models. International Organization for Standardization, 34:2910.
Kim, D. J. (2020). An empirical study on the evolution of test smell. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings, ICSE ’20, page 149–151, New York, NY, USA. Association for Computing Machinery.
Kim, D. J., Chen, T.-H. P., and Yang, J. (2021). The secret life of test smells - an empirical study on test smell evolution and maintenance. Empirical Softw. Engg., 26(5).
Lincke, R., Lundberg, J., and Löwe, W. (2008). Comparing software metrics tools. In Ryder, B. G. and Zeller, A., editors, Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2008, Seattle, WA, USA, July 20-24, 2008, pages 131–142. ACM.
Lorenz, M. and Kidd, J. (1994). Object-oriented software metrics - a practical guide. Prentice Hall.
Malhotra, R. and Chug, A. (2016). An empirical study to assess the effects of refactoring on software maintainability. In 2016 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2016, Jaipur, India, September 21-24, 2016, pages 110–117. IEEE.
Martins, J., Bezerra, C., Uchôa, A., and Garcia, A. (2021). How do code smell co-occurrences removal impact internal quality attributes? a developers’ perspective. In Brazilian Symposium on Software Engineering, SBES ’21, page 54–63, New York, NY, USA. Association for Computing Machinery.
McCabe, T. (1976). A complexity measure. IEEE Transactions on Software Engineering, SE-2(4):308–320.
Morasca, S. (2009). A probability-based approach for measuring external attributes of software artifacts. In Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, October 15-16, 2009, Lake Buena Vista, Florida, USA, pages 44–55. IEEE Computer Society.
Myers, G. J., Sandler, C., and Badgett, T. (2011). The art of software testing. John Wiley & Sons.
Nilson, M., Antinyan, V., and Gren, L. (2019). Do internal software quality tools measure validated metrics? In Franch, X., Männistö, T., and Martínez-Fernández, S., editors, Product-Focused Software Process Improvement, pages 637–648, Cham. Springer International Publishing.
Orso, A. and Rothermel, G. (2014). Software testing: A research travelogue (2000–2014). In Future of Software Engineering Proceedings, FOSE 2014, page 117–132, New York, NY, USA. Association for Computing Machinery.
Paixão, M., Uchôa, A., Bibiano, A. C., Oliveira, D., Garcia, A., Krinke, J., and Arvonio, E. (2020). Behind the intents: An in-depth empirical study on software refactoring in modern code review. In Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 125–136, New York, NY, USA. Association for Computing Machinery.
Palomba, F., Bavota, G., Penta, M. D., Oliveto, R., and Lucia, A. D. (2014). Do they really smell bad? A study on developers’ perception of bad code smells. In 30th IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014, pages 101–110. IEEE Computer Society.
Palomba, F., Zaidman, A., and Lucia, A. D. (2018). Automatic test smell detection using information retrieval techniques. In 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018, Madrid, Spain, September 23-29, 2018, pages 311–322. IEEE Computer Society.
Pecorelli, F., Palomba, F., and Lucia, A. D. (2021). The relation of test-related factors to software quality: A case study on Apache systems. Empir. Softw. Eng., 26(2):18.
Peruma, A., Almalki, K., Newman, C. D., Mkaouer, M. W., Ouni, A., and Palomba, F. (2019). On the distribution of test smells in open source Android applications: an exploratory study. pages 193–202.
Santana, R., Fernandes, D., Campos, D., Soares, L., Maciel, R., and Machado, I. (2021). Understanding practitioners’ strategies to handle test smells: A multi-method study. In Proceedings of the XXXV Brazilian Symposium on Software Engineering, SBES ’21, page 49–53, New York, NY, USA. Association for Computing Machinery.
Santana, R., Martins, L., Virgínio, T., Soares, L., Costa, H., and Machado, I. (2022). Refactoring assertion roulette and duplicate assert test smells: a controlled experiment. arXiv preprint arXiv:2207.05539.
Sjøberg, D. I., Yamashita, A., Anda, B. C., Mockus, A., and Dybå, T. (2013). Quantifying the effect of code smells on maintenance effort. IEEE Transactions on Software Engineering, 39(8):1144–1156.
Soares, E., Ribeiro, M., Amaral, G., Gheyi, R., Fernandes, L., Garcia, A., Fonseca, B., and Santos, A. (2020). Refactoring test smells: A perspective from open-source developers. In Proceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing, SAST 20, page 50–59, New York, NY, USA. Association for Computing Machinery.
Soares, E., Ribeiro, M., Gheyi, R., Amaral, G., and Santos, A. L. M. (2022). Refactoring test smells with JUnit 5: Why should developers keep up-to-date? IEEE Trans. Software Eng., 49(3):1152–1170.
Spadini, D., Palomba, F., Zaidman, A., Bruntink, M., and Bacchelli, A. (2018). On the relation of test smells to software code quality. In 2018 IEEE International Conference on Software Maintenance and Evolution, ICSME 2018, Madrid, Spain, September 23-29, 2018, pages 1–12. IEEE Computer Society.
Spadini, D., Schvarcbacher, M., Oprescu, A.-M., Bruntink, M., and Bacchelli, A. (2020). Investigating severity thresholds for test smells. In Proceedings of the 17th International Conference on Mining Software Repositories, MSR ’20, page 311–321, New York, NY, USA. Association for Computing Machinery.
Tahir, A., Counsell, S., and MacDonell, S. G. (2016). An empirical study into the relationship between class features and test smells. In Potanin, A., Murphy, G. C., Reeves, S., and Dietrich, J., editors, 23rd Asia-Pacific Software Engineering Conference, APSEC 2016, Hamilton, New Zealand, December 6-9, 2016, pages 137–144. IEEE Computer Society.
Tarwani, S. and Chug, A. (2016). Sequencing of refactoring techniques by greedy algorithm for maximizing maintainability. In 2016 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2016, Jaipur, India, September 21-24, 2016, pages 1397–1403. IEEE.
Van Deursen, A., Moonen, L., Van Den Bergh, A., and Kok, G. (2001). Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001), pages 92–95. Citeseer.
Van Rompaey, B., Du Bois, B., Demeyer, S., and Rieger, M. (2007). On the detection of test smells: A metrics-based approach for general fixture and eager test. IEEE Transactions on Software Engineering, 33(12):800–817.
Virgínio, T., Martins, L., Rocha, L., Santana, R., Cruz, A., Costa, H., and Machado, I. (2020). Jnose: Java test smell detector. In Proceedings of the 34th Brazilian Symposium on Software Engineering, SBES ’20, page 564–569, New York, NY, USA. Association for Computing Machinery.
Virgínio, T., Martins, L., Santana, R., Cruz, A., Rocha, L., Costa, H., and Machado, I. (2021). On the test smells detection: an empirical study on the Jnose test accuracy. Journal of Software Engineering Research and Development, 9:8–1.
Virgínio, T. G. A. and Machado, I. (2021). Avaliação empírica da geração automatizada de testes de software sob a perspectiva de test smells. In Anais Estendidos do XII Congresso Brasileiro de Software: Teoria e Prática, pages 112–126. SBC.
Xie, G., Chen, J., and Neamtiu, I. (2009). Towards a better understanding of software evolution: An empirical study on open source software. In 25th IEEE International Conference on Software Maintenance (ICSM 2009), September 20-26, 2009, Edmonton, Alberta, Canada, pages 51–60. IEEE Computer Society.
Yusifoğlu, V. G., Amannejad, Y., and Can, A. B. (2015). Software test-code engineering: A systematic mapping. Information and Software Technology, 58:123–147.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Humberto Damasceno, Carla Bezerra, Denivan Campos, Ivan Machado, Emanuel Coutinho
This work is licensed under a Creative Commons Attribution 4.0 International License.
