An Experimental Evaluation of the Use of BDD in Security Test Automation
DOI:
https://doi.org/10.5753/reic.2026.7750Keywords:
Security testing, Behavior-Driven Development, PyTest, Cognitive Load, Controlled ExperimentAbstract
Currently, security test automation has become essential in software development, especially in agile environments that demand continuous quality assurance and rapid vulnerability detection. However, differences in the structure and specification of automated tests may affect their comprehension and application by novice testers. This study investigates and compares the use of Pytest and Robot Framework with Behavior-Driven Development (BDD) regarding their impact on cognitive workload and security test comprehension. To this end, a controlled experiment was conducted with novice participants, evaluating cognitive workload, perceived temporal demand, and accuracy in the interpretation of security tests. The results indicate that Robot Framework with BDD presented lower levels of mental demand, effort, and frustration when compared to Pytest, with statistically significant differences observed in these dimensions of the NASA-TLX scale. Regarding test comprehension, descriptive results suggested a favorable trend for the BDD-based approach, particularly in vulnerability identification and interpretation of test success criteria, although this difference was not statistically significant. Overall, the findings suggest that the use of BDD may help reduce the perceived cognitive workload associated with analyzing security tests performed by novice testers.
Downloads
References
Abdiukov, T. (2024). Automated security testing in devsecops pipelines: Integrating ai-based vulnerability discovery and compliance validation. World Journal of Advanced Research and Reviews, 22:2083–2093. DOI: 10.30574/wjarr.2024.22.1.1083.
Aloraini, B., Nagappan, M., German, D. M., Hayashi, S., and Higo, Y. (2019). An empirical study of security warnings from static application security testing tools. Journal of Systems and Software, 158:110427. DOI: 10.1016/j.jss.2019.110427.
Binamungu, L. P. and Maro, S. (2023). Behaviour driven development: A systematic mapping study. Journal of Systems and Software, 203:111749. DOI: 10.1016/j.jss.2023.111749.
Hart, L. E. S. S. G. (1988). Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research, pages 139–183. DOI: 10.1016/S0166-4115(08)62386-9.
Lenka, R. K., Kumar, S., and Mamgain, S. (2018). Behavior driven development: Tools and challenges. In 2018 International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), pages 1032–1037. IEEE. DOI: 10.1109/ICACCCN.2018.8748595.
Lima, M. Y., Silva, C., and Souza, R. (2022). Explorando a especificação de cenários de teste de segurança no desenvolvimento orientado por comportamento (behavior driven development). In Anais do Workshop em Engenharia de Requisitos. Even3. DOI: 10.29327/1298262.25-23.
Murazvu, G., Parkinson, S., Khan, S., Liu, N., and Allen, G. (2024). A survey on factors preventing the adoption of automated software testing: A principal component analysis approach. Software, 3:1–27. DOI: 10.3390/software3010001.
Okubo, T., Kakizaki, Y., Kobashi, T., Washizaki, H., Ogata, S., Kaiya, H., and Yoshioka, N. (2014). Security and Privacy Behavior Definition for Behavior Driven Development, pages 306–309. DOI: 10.1007/978-3-319-13835-0_28.
Purkayastha, S., Goyal, S., Phillips, T., Wu, H., Haakenson, B., and Zou, X. (2020). Continuous security through integration testing in an electronic health records system. In 2020 International Conference on Software Security and Assurance (ICSSA), pages 26–31. DOI: 10.1109/ICSSA51305.2020.00012.
Qusef, A., Elish, M. O., and Binkley, D. (2019). An exploratory study of the relationship between software test smells and fault-proneness. IEEE Access, 7:139526–139536. DOI: 10.1109/ACCESS.2019.2943488.
Rajapakse, R. N., Zahedi, M., Babar, M. A., and Shen, H. (2022). Challenges and solutions when adopting devsecops: A systematic review. Information and Software Technology, 141:106700. DOI: 10.1016/j.infsof.2021.106700.
Rubatino, V., Nogueira, A. B., de Souza, F. G. R., and Pagliares, R. M. (2024). A Method Based on Behavior Driven Development (BDD) and System-Theoretic Process Analysis (STPA) for Verifying Security Requirements in Critical Software Systems, pages 373–380. DOI: 10.1007/978-3-031-56599-1_48.
Santos, S., Fernandes, R., Santos, M., Soares, M., Rocha, F., and Marczak, S. (2024). Increasing test coverage by automating bdd tests in proofs of concepts (pocs) using llm. In Anais do XXIII Simpósio Brasileiro de Qualidade de Software, page 519–525, Porto Alegre, RS, Brasil. SBC. Disponível em: [link].
Wang, Y. and Wagner, S. (2018). Combining stpa and bdd for safety analysis and verification in agile development. In Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings, pages 286–287. ACM. DOI: 10.1145/3183440.3194973.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2024). Experimentation in Software Engineering. Springer Berlin Heidelberg, Berlin, Heidelberg. DOI: 10.1007/978-3-662-69306-3.
Zhang, X., Shen, W., Liang, Z., Cui, L., and Wang, Y. (2024). Research and application of automated testing technology for data security vulnerabilities. In 2024 4th International Conference on Mobile Networks and Wireless Communications (ICMNWC), pages 01–05. IEEE. DOI: 10.1109/ICMNWC63764.2024.10872029.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 The authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
