Technical Debt Tools: a Survey and an Empirical Evaluation
DOI:
https://doi.org/10.5753/jserd.2024.3591Keywords:
Empirical Evaluation, Survey, Technical debt, ToolsAbstract
Background: The life cycle of a technical debt from its identification to its payment is long and may include several activities, such as identification and management. There is a lot of research in the literature to address different sets of these activities by different means. Specifically, several tools have already tackled such technical debt identification problems. However, only a few studies empirically assessed those tools. Method: In this article, we carried a multi-method research. We first surveyed the literature for the technical debt tools available and then we evaluated two of them, which aim at identification of self-admitted technical debt. They are named eXcomment e DebtHunter. Results: We found 97 tools employing different approaches to support technical debt life cycle management. Most of them (59%) address only the high level task of management, instead of actually identify and pay the debt. Additionally, as for our empirical evaluation of tools, our results show that DebtHunter found only 7% of debt identified by eXcomment. In the other way around, eXcomment found 19.9% the debt found by DebtHunter. Besides, both tools have low levels of precision and recall. Conclusion: It is hard to find technical debt through comments. Both tools can find indicators of debt items, however they struggle on the precision and recall. In fact, although eXcomment and DebtHunter diverge on the amount of debt identified, they seem to converge with regard to the type of debt present in the system under evaluation.
Downloads
References
Alves, N. S., Mendes, T. S., de Mendonça, M. G., Spínola, R. O., Shull, F., and Seaman, C. (2016). Identification and management of technical debt: A systematic mapping study. Information and Software Technology, 70:100–121.
Ampatzoglou, A., Chatzigeorgiou, A., Arvanitou, E. M., and Bibi, S. (2022). Sdk4ed: A platform for technical debt management. Software: Practice and Experience, 52(8):1879–1902.
Avgeriou, P. C., Taibi, D., Ampatzoglou, A., Arcelli Fontana, F., Besker, T., Chatzigeorgiou, A., Lenarduzzi, V., Martini, A., Moschou, A., Pigazzini, I., Saarimaki, N., Sas, D. D., de Toledo, S. S., and Tsintzira, A. A. (2021). An overview and comparison of technical debt measurement tools. IEEE Software, 38(3):61–71.
Bavota, G. and Russo, B. (2016). A large-scale empirical study on self-admitted technical debt. In Proceedings of the 13th International Conference on Mining Software Repositories, MSR ’16, page 315–326, New York, NY, USA. Association for Computing Machinery.
Chandramouli, P., Codabux, Z., and Vidoni, M. (2022). analyzer: A sonarqube plugin for analyzing object-oriented r packages. SoftwareX, 19:101113.
Cunningham, W. (1992). The wycash portfolio management system. ACM SIGPLAN OOPS Messenger, 4(2):29–30.
Farias, M., Mendes, T. S., Mendonça, M. G., and Spínola, R. O. (2021). On comment patterns that are good indicators of the presence of self-admitted technical debt and those that lead to false positive items. In Proceedings of the 27th Annual Americas Conference on Information Systems (AMCIS), pages 1–10.
Farias, M. A. F., de Mendonça Neto, M. G., Kalinowski, M.,and Spínola, R. O. (2020). Identifying self-admitted technical debt through code comment analysis with a contextualized vocabulary. Information Software Technologies, 121:106270.
Fontana, F. A., Roveda, R., and Zanoni, M. (2016). Technical debt indexes provided by tools: a preliminary discussion. In 2016 IEEE 8th International Workshop on Managing Technical Debt (MTD), pages 28–31. IEEE.
Fowler, M. (2018). Refactoring: improving the design of existing code. Addison Wesley Professional, United States.
Freire, S., Rios, N., Mendonça, M., Falessi, D., Seaman, C., Izurieta, C., and Spínola, R. O. (2020). Actions and impediments for technical debt prevention: results from a global family of industrial surveys. In Proceedings of the 35th Annual ACM Symposium on Applied Computing, pages 1548–1555.
Freitas, G., Reboucas, R., and Coelho, R. (2022). Exploring technical debt tools: A systematic mapping study. In Enterprise Information Systems: 23rd International Conference, ICEIS 2021, Virtual Event, April 26–28, 2021, Revised Selected Papers, volume 455, page 280. Springer Nature.
Gomes, F., Santos, E., Freire, S., Mendes, T. S., Mendonça, M., and Spínola, R. (2023). Investigating the point of view of project management practitioners on technical debt - a study on stack exchange. Journal of Software Engineering Research and Development, 11(1):12:1 – 12:15.
Gomes, F. G. S., Mendes, T. S., Spínola, R. O., Mendonça, M., and Farias, M. (2019). Uma análise da relação entre code smells e dívida técnica auto-admitida. In Anais do VII Workshop on Software Visualization, Evolution and Maintenance (VEM), pages 37–44, Porto Alegre, RS, Brasil. SBC.
Gomes, T., Loiola, D., and Santos, A. R. (2022). Avaliação de ferramentas de identificação de dívida técnica auto-admitida. In Anais do X Workshop de Visualização, Evolução e Manutenção de Software, pages 16–20, Porto Alegre, RS, Brasil. SBC.
Gomes, T., Loiola, D., and Santos, A. R. (2024). Technical debt tools: a survey and an empirical evaluation. Online at http://www.doi.org/10.5281/zenodo.8198082. Creative Commons 4.0 Licensed Dataset.
Guo, Z., Liu, S., Liu, J., Li, Y., Chen, L., Lu, H., Zhou, Y., and Xu, B. (2019). Mat: A simple yet strong baseline for identifying self-admitted technical debt.
Landis, J. R. and Koch, G. G. (1977). The measurement of observer agreement for categorical data. biometrics, pages 159–174.
Lefever, J., Cai, Y., Cervantes, H., Kazman, R., and Fang, H. (2021). On the lack of consensus among technical debt detection tools. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 121–130. IEEE.
Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., and Fontana, F. A. (2021a). A systematic literature review on Technical Debt prioritization: Strategies, processes, factors, and tools. Journal of Systems and Software, 171:110827.
Lenarduzzi, V., Besker, T., Taibi, D., Martini, A., and Fontana, F. A. (2021b). A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools. Journal of Systems and Software, 171:110827.
Lenarduzzi, V., Pecorelli, F., Saarimaki, N., Lujan, S., and Palomba, F. (2022). A critical comparison on six static analysis tools: Detection, agreement, and precision. Journal of Systems and Software, 198:111575.
Lenarduzzi, V., Sillitti, A., and Taibi, D. (2020). A survey on code analysis tools for software maintenance prediction. In Proceedings of 6th International Conference in Software Engineering for Defence Applications: SEDA 2018 6, pages 165–175. Springer.
Li, Y., Soliman, M., and Avgeriou, P. (2023). Automatic identification of self-admitted technical debt from four different sources. Empirical Software Engineering, 28(3):1–38.
Li, Z., Avgeriou, P., and Liang, P. (2015). A systematic mapping study on technical debt and its management. Journal of Systems and Software, 101:193–220.
Loiola, D. A. M., Gomes, T. B. S., Santos, A. R., and Farias, M. A. F. (2023). Avaliação do uso de heurísticas de contexto na mineração de dívida técnica com a eXComment. Revista de Sistemas e Computação., 13.
Maldonado, E. D. S. and Shihab, E. (2015). Detecting and quantifying different types of self-admitted technical debt. In Proceedings of the 7th Workshop on Managing Technical Debt, MTD ’11, pages 9–15, New York, NY, USA. IEEE.
Mortara, J., Bavota, G., Lanza, M., and Panichella, S. (2022). Customizable visualization of quality metrics for object-oriented variability implementations. In 26th ACM International Systems and Software Product Line Conference (SPLC), pages 1–12. ACM.
Murillo, M. I., López, G., Spínola, R., Guzmán, J., Rios, N., and Pacheco, A. (2023). Identification and management of technical debt: A systematic mapping study update. Journal of Software Engineering Research and Development, 11(1):8:1 – 8:20.
Nikolaidis, N., Ampatzoglou, A., Chatzigeorgiou, A., Tsekeridou, S., and Piperidis, A. (2022). Technical debt in service-oriented software systems. In International Conference on Product-Focused Software Process Improvement, pages 265–281. Springer.
Oliveira, I., Marques-Neto, H., and Xavier, L. (2020). Analisando estratégias para identificação de dívidas técnicas. In Anais do VIII Workshop de Visualização, Evolução e Manutenção de Software, pages 9–16, Porto Alegre, RS, Brasil. SBC.
Potdar, A. and Shihab, E. (2014). An exploratory study on self-admitted technical debt. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 91–100, Victoria, BC, Canada. IEEE.
Rios, N., de Mendonça Neto, M. G., and Spínola, R. O. (2018). A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners. Information and Software Technology, 102:117–145.
Sala, I., Tommasel, A., and Arcelli Fontana, F. (2021). Debthunter: A machine learning-based approach for detecting self-admitted technical debt. In Evaluation and Assessment in Software Engineering, EASE 2021, page 278–283, New York, NY, USA. Association for Computing Machinery.
Tom, E., Aurum, A., and Vidgen, R. (2013). An exploration of technical debt. Journal of Systems and Software, 86(6):1498–1516.
Tsoukalas, D., Chatzigeorgiou, A., Ampatzoglou, A., Mittas, N., and Kehagias, D. (2022). Td classifier: Automatic identification of java classes with high technical debt. In Proceedings of the International Conference on Technical Debt, pages 76–80. ACM.
Venters, C. C., Capilla, R., Betz, S., Penzenstadler, B., Crick, T., Crouch, S., Nakagawa, E. Y., Becker, C., and Carrillo, C. (2018). Software sustainability: Research and practice from a software architecture viewpoint. Journal of Systems and Software, 138:174–188.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Tchalisson Brenne S. Gomes, Diogo Alves de Moura Loiola, Alcemir Rodrigues Santos
This work is licensed under a Creative Commons Attribution 4.0 International License.
