Design of Empirically-Grounded Mutation Operators for Terraform IaC
DOI:
https://doi.org/10.5753/reic.2026.8471Keywords:
Mutation Testing, Infrastructure as Code, Terraform, Mining Software Repositories, Defect Taxonomy, Software TestingAbstract
Defects in Infrastructure as Code (IaC) scripts, particularly Terraform configurations, can cause severe operational failures, yet systematic criteria for designing effective test suites remain scarce. Mutation testing addresses this need by seeding representative faults into code and measuring whether tests detect them. Its effectiveness, however, depends on mutation operators that faithfully model real developer mistakes and no such operators currently exist for Terraform. This paper fills that gap with an empirically grounded catalog of mutation operators derived from a quantitative analysis of 6,749 corrective commits across 54 open-source Terraform repositories. Commit messages were encoded with Sentence-RoBERTa, projected via UMAP, and clustered with HDBSCAN to identify recurring defect patterns. The resulting clusters were synthesized into a defect taxonomy from which mutation operators were formally defined. The contributions are twofold: (1) a catalog of mutation operators for Terraform grounded in real-world defect data, and (2) a replicable multi-stage pipeline for mining defect patterns from version-control histories. The technical feasibility of these operators was validated through a proof-of-concept tool, paving the way for future large-scale automated evaluations.
Downloads
References
Al-Fraihat, D., Sharrab, Y., Al-Ghuwairi, A.-R., Sbaih, N., and Qahmash, A. (2024). Detecting refactoring type of software commit messages based on ensemble machine learning algorithms. Scientific Reports, 14(1):21367. DOI: 10.1038/s41598-024-72178-6.
Amit, I. and Feitelson, D. G. (2021). Corrective commit probability: a measure of the effort invested in bug fixing. Software Quality Journal, 29(4):817–861. DOI: 10.1007/s11219-021-09564-z.
Aqua Security (2025). tfsec: static analysis security scanner for terraform code. Software. Disponível em: [link].
Bridgecrew (2025). Checkov: policy-as-code scanner for infrastructure as code. Software. Disponível em: [link].
Campello, R. J. G. B., Moulavi, D., and Sander, J. (2013). Density-based clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining, PAKDD 2013, volume 7819 of Lecture Notes in Computer Science, pages 160–172. Springer Berlin Heidelberg. DOI: 10.1007/978-3-642-37456-2_14.
Carnegie Mellon University (2019). Infrastructure as code: Final report. Technical report, Software Engineering Institute, Carnegie Mellon University. Disponível em: [link]. Accessed: February 1, 2025.
Chittibala, D. R. (2023). Infrastructure as Code (IaC) and its role in achieving DevOps goals. International Journal of Science and Research (IJSR), 12(1):1258–1262. DOI: 10.21275/SR24304170702.
Dalla Palma, S., Di Nucci, D., Palomba, F., and Tamburri, D. A. (2022). Within-project defect prediction of infrastructure-as-code using product and process metrics. IEEE Transactions on Software Engineering, 48(6):2086–2104. DOI: 10.1109/TSE.2021.3051492.
Hassan and Rahman (2022). As code testing: Characterizing test quality in open source ansible development. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST), pages 208–219. DOI: 10.1109/ICST53961.2022.00031.
Hassan, A. E. (2008). The road ahead for mining software repositories. In 2008 Frontiers of Software Maintenance, pages 48–57. IEEE. DOI: 10.1109/FOSM.2008.4659248.
Howard, M. (2022). Terraform: Automating infrastructure as a service. arXiv preprint arXiv:2205.10676. Disponível em: [link].
IEEE Computer Society (2010). Ieee standard classification for software anomalies. Revision of IEEE Std 1044-1993. Approved 9 November 2009; Published 7 January 2010. DOI: 10.1109/IEEESTD.2010.5399061.
Jia, Y. and Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649–678. DOI: 10.1109/TSE.2010.62.
Kalliamvakou, E., Gousios, G., Blincoe, K., Singer, L., German, D. M., and Damian, D. (2014). The promises and perils of mining github. In Proceedings of the 11th Working Conference on Mining Software Repositories, pages 92–101. DOI: 10.1145/2597073.2597074.
Lee, J. Y. D. and Chieu, H. L. (2021). Co-training for commit classification. In Xu, W., Ritter, A., Baldwin, T., and Rahimi, A., editors, Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021), pages 389–395, Online. Association for Computational Linguistics. DOI: 10.18653/v1/2021.wnut-1.43.
Malzer, C. and Baum, M. (2020). A hybrid approach to hierarchical density-based cluster selection. In 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pages 223–228. IEEE. DOI: 10.1109/MFI49285.2020.9235263.
McInnes, L., Healy, J., and Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426. Disponível em: [link].
Offutt, A. J. and Untch, R. H. (2001). Mutation Testing for the New Century. Springer.
Ozdoğan, E., Ceran, O., and Üstündağ, M. (2023). Systematic analysis of infrastructure as code technologies. Gazi University Journal of Science Part A: Engineering and Innovation, 10. DOI: 10.54287/gujsa.1373305.
Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y. L., and Harman, M. (2019). Section six - mutation testing advances: An analysis and survey. In Memon, A. M., editor, Advances in Computers, volume 112 of Advances in Computers, pages 275–378. Elsevier. DOI: 10.1016/bs.adcom.2018.03.015.
Rahman, A. (2018). Characteristics of defective infrastructure as code scripts in devops. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceedings, ICSE ’18, pages 476–479, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3183440.3183452.
Rahman, A., Elder, S., Shezan, F. H., Frost, V., Stallings, J., and Williams, L. A. (2018). Categorizing defects in infrastructure as code. CoRR, abs/1809.07937. Disponível em: [link].
Rahman, A. and Williams, L. (2019). Source code properties of defective infrastructure as code scripts. Information and Software Technology, 112:148–163. DOI: 10.1016/j.infsof.2019.04.013.
Ramler, R., Wetzlmaier, T., and Klammer, C. (2017). An empirical study on the application of mutation testing for a safety-critical industrial software system. In Proceedings of the Symposium on Applied Computing, SAC ’17, pages 1401–1408, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3019612.3019830.
Rebai, S., Kessentini, M., Alizadeh, V., Sghaier, O. B., and Kazman, R. (2020). Recommending refactorings via commit message analysis. Information and Software Technology, 126:106332. DOI: 10.1016/j.infsof.2020.106332.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084. Disponível em: [link].
Shimizu, R. and Kanuka, H. (2020). Test-based least privilege discovery on cloud infrastructure as code. In 2020 IEEE International Conference on Cloud Computing Technology and Science (CloudCom), pages 1–8. IEEE. DOI: 10.1109/CloudCom49646.2020.00017.
Shimizu, R., Nunomura, Y., and Kanuka, H. (2024). Test-suite-guided discovery of least privilege for cloud infrastructure as code. Automated Software Engineering, 31(1):25. DOI: 10.1007/s10515-024-00423-3.
Sokolowski, D., Spielmann, D., and Salvaneschi, G. (2024). Automated infrastructure as code program testing. IEEE Transactions on Software Engineering, 50(6):1585–1599. DOI: 10.1109/TSE.2024.3393070.
Tong, J., Wang, Z., and Rui, X. (2023). Boosting commit classification with contrastive learning. arXiv preprint arXiv:2308.08263. Disponível em: [link].
Wang, B., Chen, M., Lin, Y., Papadakis, M., and Zhang, J. M. (2024). An exploratory study on using large language models for mutation testing. Disponível em: [link].
Yang, Y., Ronchieri, E., and Canaparo, M. (2022). Natural language processing application on commit messages: a case study on hep software. Applied Sciences, 12(21):10773. DOI: 10.3390/app122110773.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 The authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
