Evaluating the effectiveness of vulnerability analyzers in blockchain smart contracts
DOI:
https://doi.org/10.5753/reic.2026.6719Keywords:
Blockchain, Smart Contracts, security, vulnerabilities, DASP Top TenAbstract
The security of smart contracts is a recurring issue not only in the Ethereum blockchain but also in other EVM-based networks such as Hyperledger Besu. This work presents an empirical analysis of the vulnerability detection capabilities of smart contract analysis tools, focusing on the evolution and effectiveness of the tools integrated into the SmartBugs framework. In the first experiment, 215 real contract samples collected from Etherscan were analyzed, revealing that 98% of the alerts generated by the tools were classified as "other", which indicates that the DASP Top 10 taxonomy, used in previous studies, is outdated when compared to the current development landscape. In other experiments, we evaluated the actual detection rate on a dataset of intentionally vulnerable contracts, using versions 2.0.10 and 2.0.15 of SmartBugs. In addition to the original tools, new static and dynamic analyzers were incorporated, and a more refined validation methodology was adopted, based on the exact location of the vulnerability in the source code, rather than solely on nominal matching of the vulnerability type. The results show that, despite the evolution between versions, significant discrepancies still exist among the tools included in SmartBugs, with some showing substantial improvements in precision while others maintain performance below expectations. The findings indicate that the vulnerability classification used in the initial studies no longer reflects the current state of the ecosystem, and that the lack of standardization in the validation process still compromises comparative analyses.
Downloads
References
Atzei, N., Bartoletti, M., and Cimoli, T. (2017). A survey of attacks on ethereum smart contracts (sok). Principles of Security and Trust (POST), 10204:164–186. DOI: 10.1007/978-3-662-54455-6_8.
Bennour, I. E., Wannes, M. H., Ghiss, M., Braham, M., Lahbib, A., Habib, N., and Ribeiro, H. (2024). Enhancing dapp supply chain with verified smart contracts: A case study on the olive-oil industry. In 2024 IEEE/ACS 21st International Conference on Computer Systems and Applications (AICCSA), pages 1–8. DOI: 10.1109/AICCSA63423.2024.10912547.
Casale-Brunet, S., Ribeca, P., Doyle, P., and Mattavelli, M. (2021). Networks of ethereum non-fungible tokens: A graph-based analysis of the erc-721 ecosystem. In 2021 IEEE International Conference on Blockchain (Blockchain), pages 188–195. DOI: 10.1109/Blockchain53845.2021.00033.
Chen, W., Zhang, T., Chen, Z., Zheng, Z., and Lu, Y. (2020). Traveling the token world: A graph analysis of ethereum erc20 token ecosystem. In Proceedings of The Web Conference 2020, WWW ’20, page 1411–1421, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3366423.3380215.
Durieux, T., Ferreira, H., Abreu, R., and State, R. (2020a). SmartBugs-curated: Dataset of vulnerable ethereum smart contracts. Disponível em: [link].
Durieux, T., Ferreira, J. F., Abreu, R., and Cruz, P. (2020b). Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pages 530–541.
Eshghie, M., Artho, C., and Gurov, D. (2021). Dynamic vulnerability detection on smart contracts using machine learning.
Etherscan (2025). Verified Contracts - Etherscan. Disponível em: [link].
Feist, J., Grieco, G., and Groce, A. (2019). Slither: A static analysis framework for smart contracts. In 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), pages 8–15. DOI: 10.1109/WETSEB.2019.00008.
Ferreira, J. F., Cruz, P., Durieux, T., and Abreu, R. (2020). SmartBugs: A framework to analyze Solidity smart contracts. In Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pages 1349–1352.
Grieco, G., Song, W., Cygan, A., Feist, J., and Groce, A. (2020). Echidna: effective, usable, and fast fuzzing for smart contracts. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2020, page 557–560, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3395363.3404366.
Grishchenko, I., Maffei, M., and Schneidewind, C. (2018). Foundations and tools for the static analysis of ethereum smart contracts. In Chockler, H. and Weissenbacher, G., editors, Computer Aided Verification, pages 51–78, Cham. Springer International Publishing.
JJ, L. and Singh, K. (2024). Enhancing oyente: four new vulnerability detections for improved smart contract security analysis. International Journal of Information Technology, 16(6):3389–3399.
Kushwaha, S. S., Joshi, S., Singh, D., Kaur, M., and Lee, H.-N. (2022). Ethereum smart contract analysis tools: A systematic review. IEEE Access, 10:57037–57062. DOI: 10.1109/ACCESS.2022.3169902.
Mehar, M., Shier, C., Giambattista, A., Gong, E., Fletcher, G., Sanayhie, R., Kim, H. M., and Laskowski, M. (2017). Understanding a revolutionary and flawed grand experiment in blockchain: The dao attack. Journal of Cases on Information Technology, 21(1):19–32.
Mossberg, M., Manzano, F., Hennenfent, E., Groce, A., Grieco, G., Feist, J., Brunson, T., and Dinaburg, A. (2019). Manticore: A user-friendly symbolic execution framework for binaries and smart contracts.
Mueller, B. (2018). Smashing ethereum smart contracts for fun and real profit. HITB SECCONF Amsterdam, 9(54):4–17.
NCC Group (2018). Decentralized application security project (dasp) top 10. Disponível em: [link].
Pinna, A., Ibba, S., Baralla, G., Tonelli, R., and Marchesi, M. (2019). A massive analysis of ethereum smart contracts empirical study and code metrics. IEEE Access. DOI: 10.1109/ACCESS.2019.2921936.
Salzer, G. and Di Angelo, M. (2019). A survey of tools for analyzing ethereum smart contracts. DOI: 10.1109/DAPPCON.2019.00018.
Staderini, M., Palli, C., and Bondavalli, A. (2020). Classification of ethereum vulnerabilities and their propagations. In 2020 Second International Conference on Blockchain Computing and Applications (BCCA), pages 44–51. DOI: 10.1109/BCCA50787.2020.9274458.
Vidal, F. R., Ivaki, N., and Laranjeiro, N. (2024). Openscv: An open hierarchical taxonomy for smart contract vulnerabilities. Empirical Software Engineering, 29(4):101.
Wang, Y., Lahiri, S. K., Chen, S., Pan, R., and Dillig, I. (2020). Formal verification of workflow policies for smart contracts in azure blockchain. In International Conference on Verified Software: Theories, Tools, and Experiments, pages 230–250. Springer. DOI: 10.1007/978-3-030-41600-3_7.
Wang, Y., Sheng, S., and Wang, Y. (2023). A Systematic Literature Review on Smart Contract Vulnerability Detection by Symbolic Execution, pages 226–241. DOI: 10.1007/978-981-99-8101-4_16.
Wood, G. et al. (2014). Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper, 151(2014):1–32.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 The authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
