How are test smells treated in the wild? A tale of two empirical studies
DOI:
https://doi.org/10.5753/jserd.2021.1802Keywords:
Test Smells, Survey Study, Interview Study, Mixed-Methods ResearchAbstract
Developing test code may be a time-consuming process that requires much effort and cost, especially when done manually. In addition, during this process, developers and testers are likely to adopt bad design choices, which may lead to introducing the so-called test smells in the test code. As the test code with test smells size increases, these tests might become more complex, and as a consequence, much more challenging to understand and evolve them correctly. Therefore, test smells may harm the test code quality and maintenance and break the whole software testing activities. In this context, this study aims to understand whether software testing practitioners unintentionally insert test smells when they implement test code. We first carried out an expert survey to analyze the usage frequency of a set of test smells and then interviews to reach a deeper understanding of how practitioners deal with test smells. Sixty professionals participated in the survey, and fifty professionals participated in the interviews. The yielded results indicate that experienced professionals introduce test smells during their daily programming tasks, even when using their companies’ standardized practices. Additionally, tools support test development and quality improvement, but most interviewees are not aware of test smells’ concepts.
Downloads
References
Bavota, G., Qusef, A., Oliveto, R., Lucia, A., and Binkley, D. (2012). An empirical analysis of the distribution of unit test smells and their impact on software maintenance. In 28th IEEE International Conference on Software Maintenance (ICSM).
Bavota, G., Qusef, A., Oliveto, R., Lucia, A., and Binkley, D. (2015). Are test smells really harmful? An empirical study. Empirical Software Engineering, 20(4).
Corbin, J. and Strauss, A. (2014). Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
Creswell, J. W. and Clark, V. L. P. (2018). Designing and Conducting Mixed Methods Research. SAGE Publications, third edition.
Fraser, G. and Arcuri, A. (2011). Evosuite: Automatic test suite generation for object-oriented software. In 13th European Conference on Foundations of Software Engineering, ESEC/FSE, New York, NY, USA. ACM.
Garousi, V. and Felderer, M. (2016). Developing, verifying, and maintaining high-quality automated test scripts. IEEE Software, 33(3).
Garousi, V. and Küçük, B. (2018). Smells in software test code: A survey of knowledge in industry and academia. Journal of systems and software, 138.
Greiler, M., van Deursen, A., and Storey, M. (2013). Automated detection of test fixture strategies and smells. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.
Gubrium, J. F., Holstein, J. A., Marvasti, A. B., and McKinney, K. D., editors (2012). The SAGE Handbook of Interview Research: The Complexity of the Craft. SAGE Publications, 2nd edition.
Junior, N. S., Martins, L., Rocha, L., Costa, H., and Machado, I. (2021). How are test smells treated in the wild? A tale of two empirical studies [Dataset]. Available at: https://doi.org/10.5281/zenodo.4548406.
Junior, N. S., Rocha, L., Martins, L. A., and Machado, I. (2020). A survey on test practitioners’ awareness of test smells. In Proceedings of the XXIII Iberoamerican Conference on Software Engineering, CIbSE 2020, pages 462–475. Curran Associates.
Kitchenham, B. A., Budgen, D., and Brereton, P. (2015). Evidence-based software engineering and systematic reviews, volume 4. CRC press.
Melegati, J. and Wang, X. (2020). Case survey studies in software engineering research. In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), ESEM ’20, New York, NY, USA. Association for Computing Machinery.
Meszaros, G., Smith, S. M., and Andrea, J. (2003). The test automation manifesto. In Maurer, F. and Wells, D., editors, Extreme Programming and Agile Methods XP/Agile Universe 2003. Springer Berlin Heidelberg.
Miles, M. B., Huberman, A. M., and Saldaña, J. (2014). Qualitative Data Analysis. SAGE Publications, fourth edition.
Palomba, F., Di Nucci, D., Panichella, A., Oliveto, R., and De Lucia, A. (2016). On the diffusion of test smells in automatically generated test code: An empirical study. In 9th International Workshop on Search-based Software Testing. ACM.
Palomba, F. and Zaidman, A. (2019). The smell of fear: On the relation between test smells and flaky tests. Empirical Software Engineering.
Peruma, A. S. A. (2018). What the Smell? An Empirical Investigation on the Distribution and Severity of Test Smells in Open Source Android Applications. PhD Thesis, Rochester Institute of Technology.
Pfleeger, S. L. and Kitchenham, B. A. (2001). Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Software Engineering Notes, 26(6):16–18.
Singer, J., Sim, S. E., and Lethbridge, T. C. (2008). Software Engineering Data Collection for Field Studies, pages 9–34. Springer London, London.
Smeets, N. and Simons, A. J. (2011). Automated unit testing with Randoop, JWalk and μJava versus manual JUnit testing. Research report, Department of Computer Science, University of Sheffield/University of Antwerp, Sheffield, Antwerp.
Spadini, D., Schvarcbacher, M., Oprescu, A.M., Bruntink, M., and Bacchelli, A. (2020). Investigating severity thresholds for test smells. In Proceedings of the 17th International Conference on Mining Software Repositories, MSR.
Tufano, M., Palomba, F., Bavota, G., Di Penta, M., Oliveto, R., De Lucia, A., and Poshyvanyk, D. (2016). An empirical investigation into the nature of test smells. In 31st International Conference on Automated Software Engineering. IEEE.
Van Deursen, A., Moonen, L., Van Den Bergh, A., and Kok, G. (2001). Refactoring test code. In Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP).
Van Rompaey, B., Du Bois, B., and Demeyer, S. (2006). Characterizing the relative significance of a test smell. In 22nd International Conference on Software Maintenance, ICSM’06. IEEE Computer Society.
Virgínio, T., Martins, L. A., Soares, L. R., Santana, R., Costa, H., and Machado, I. (2020). An empirical study of automatically generated
tests from the perspective of test smells. In SBES ’20: 34th Brazilian Symposium on Software Engineering, pages 92–96. ACM.
Virgínio, T., Santana, R., Martins, L. A., Soares, L. R., Costa, H., and Machado, I. (2019). On the influence of test smells on test coverage. In Proceedings of the XXXIII Brazilian Symposium on Software Engineering. ACM.
Wiederseiner, C., Jolly, S. A., Garousi, V., and Eskandar, M. M. (2010). An open-source tool for automated generation of black-box xunit test code and its industrial evaluation. In Bottaci, L. and Fraser, G., editors, Testing – Practice and Research Techniques. Springer Berlin Heidelberg.
Yusifoğlu, V. G., Amannejad, Y., and Can, A. B. (2015). Software test-code engineering: A systematic mapping. Information and Software Technology, 58.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Nildo Silva Junior, Luana Martins, Larissa Rocha, Heitor Costa, Ivan Machado
This work is licensed under a Creative Commons Attribution 4.0 International License.
