Software Tool Qualification: A Systematic Literature Review
DOI:
https://doi.org/10.5753/jserd.2025.6011Keywords:
Software, Tools, Qualification, Toolchain, Systematic Literature ReviewAbstract
Safety-critical systems are subject to rigorous regulation, demanding compliance with specific criteria before it is made available. In safety-critical domains, this is accomplished through the fulfillment of recognized safety standards requirements or objectives. This SLR has the purpose of identifying approaches or methods to comply with safety standards requirements or objectives related to tool qualification aspects. It also pursued the identification of methods used to demonstrate tool confidence, as well as the activities and artifacts generated to demonstrate that. This study also identified challenges practitioners face to qualify software tools. This SLR was conducted per guidelines proposed by Kitchenham and Biolchini. A search string was developed, and its application returned more than 1,180 papers. For this study, 41 papers were selected, as per the criteria defined in the protocol. These SLR results indicate that there are many studies describing approaches based on established safety standards, for ensuring confidence in tool functionality. The results indicated that research on different proposed methods for tool qualification exists, especially in automotive context. The results also point to the necessity of additional research on toolchain integration. The SLR main findings were the identification of methods for tool qualification, concerns, approaches to address toolchain integration.
Downloads
References
Aiello, M., Comar, C., & Ruiz, J. (2020). An Assurance Case based on Overarching Properties for a TQL1 Code Generator. Embedded Real Time Systems ERTS 2020. IEEE Xplore.
Asplund, F., Biehl, M., & Loiret, F. (2012). Towards the Automated Qualification of Tool Chain Design. Comput-er Safety, Reliability, and Security. SAFECOMP 2012.
Asplund, F., El-khoury, J., & Törngren, M. (2012). Qualify-ing Software Tools, a Systems Approach. Computer Safe-ty, Reliability, and Security. SAFECOMP 2012.
Batista, L., & Monsuez, B. (2020). The conception of a large-scale Systems Engineering environment. 2020 AI-AA/IEEE 39th Digital Avionics Systems Conference (DASC). San Antonio, TX, USA: IEEE Xplore.
Baumgart, S., Chen, Y., Hamrén, R., & Punnekkat, S. (2021). A Model-Based Approach to Document Software Toolchains for Supporting a Safety Analysis. 2021 IEEE International Systems Conference (SysCon). Vancouver, BC, Canada: IEEE XPlore.
Blackburn, M., & Busser, R. (1996). T-VEC: a tool for de-veloping critical systems. Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96. Gaithersburg, MD, USA: IEEE Xplore.
Blackburn, M., & Busser, R. (1996). T-VEC: a tool for de-veloping critical systems. Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96. Gaithersburg, MD, USA: IEEE Xplore.
Borgh, A., & Towhidnejad, M. (2019). Guidelines for De-velopment of Operational Requirements for Model Checking Tools. 2019 Integrated Communications, Nav-igation and Surveillance Conference (ICNS). Herndon, VA, USA: IEEE Xplore.
Brauer, J., Peleska, J., & Schulze, U. (2012). Efficient and Trustworthy Tool Qualification for Model-Based Testing Tools. Testing Software and Systems. ICTSS 2012.
Cofer, D. (2015). You keep using that word. ACM SIGLOG News 2, 4 (October 2015), 17-25.
da Silva, F., Bagbaba, A., Hamdioui, S., & Sauer, C. (2019). Efficient Methodology for ISO26262 Functional Safety Verification. 2019 IEEE 25th International Sym-posium on On-Line Testing and Robust System Design (IOLTS). Rhodes, Greece: IEEE Xplore.
de la Vara, J., Ruiz, A., & Blondelle, G. (2021). Assurance and certification of cyber–physical systems: The AMASS open source ecosystem. Journal of Systems and Soft-ware.
Durak, U., D’Ambrogio, A., & Bocciarelli, P. (2020). Safety-critical simulation engineering. In Proceedings of the 2020 Summer Simulation Conference (SummerSim '20). San Diego, CA, USA: ACM Digital Library.
Gallina, B., Kashiyarandi, S., Zugsbratl, K., & Geven, A. (2014). Enabling Cross-Domain Reuse of Tool Qualifica-tion Certification Artefacts. Computer Safety, Reliability, and Security. SAFECOMP 2014.
Gleirscher, M., Sachtleben, R., & Peleska, J. (2023). Qual-ification of proof assistants, checkers, and generators: Where are we and what next? Science of Computer Programming.
Guo, Y., & Slay, J. (2010). A Function Oriented Method-ology to Validate and Verify Forensic Copy Function of Digital Forensic Tools. 2010 International Confer-ence on Availability, Reliability and Security. Krakow, Poland: IEEE Xplore.
Guo, Y., Slay, J., & Beckett, J. (2009). Validation and verification of computer forensic software tools— Searching Function. Digital Investigation, S12-S22.
Horsman, G. (2019). Tool testing and reliability issues in the field of digital forensics. Digital Investigation, 163-175.
Hughes, N., Rabieh, K., & Aydogan, A. (2019). Employ-ing a Continuous Measurement Process During Digital Tool Validation. 2019 International Symposium on Networks, Computers and Communications (ISNCC). Istanbul, Türkiye: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2003). Design tool assess-ment for safety-critical software development. 28th Annual NASA Goddard Software Engineering Work-shop, 2003. Proceedings. Greenbelt, MD, USA: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2008). Software certifica-tion for safety-critical systems: A status report. 2008 International Multiconference on Computer Science and Information Technology. Wisla, Poland: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2010). Hardware certifica-tion for real-time safety-critical systems: State of the art. Annual Reviews in Control, 163-174.
Kornecki, A., Hall, K., Hearn, D., Lau, H., & Zalewski, J. (2004). Evaluation of software development tools for high assurance safety critical systems. Eighth IEEE In-ternational Symposium on High Assurance Systems Engineering, 2004. Proceedings. Tampa, FL, USA: IEEE Xplore.
Krauss, S., Rejzek, M., & Hilbes, C. (2015). Tool Qualifi-cation Considerations for Tools Supporting STPA. Pro-cedia Engineering, 15-24.
Marques, J., & da Cunha, A. (2015). Use of the RTCA DO-330 in aeronautical databases. 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC). Prague, Czech Republic.
Marques, J., & da Cunha, A. (2017). COTS tool qualifi-cation using RTCA DO-330: Common pitfalls. 2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC). St. Petersburg, FL, USA: IEEE Xplore.
Marques, J., & da Cunha, A. (2017). Verification scenar-ios of onboard databases under the RTCA DO- 178C and the RTCA DO-200B. 2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC). St. Petersburg, FL, USA: IEEE Xplore.
Mishra, A., Rao, M., CU, C., Rao, V., Jeppu, Y., & Murthy, N. (2013). An auto-review tool for model- based testing of safety-critical systems. In Proceedings of the 2013 International Workshop on Joining Aca-deMiA and Industry Contributions to testing Automa-tion (JAMAICA 2013). New York, NY, USA: ACM Dig-ital Library.
Prabhu, S., Kapil, H., & Lakshmaiah, S. (2018). Safety Critical Embedded Software: Significance and Ap-proach to Reliability. 2018 International Conference on Advances in Computing, Communications and In-formatics (ICACCI). Bengaluru, India: IEEE Xplore.
Schuster, G. (2022). Certification of software tools used in safety-critical software development. IEEE 5th In-ternational Conference and Workshop Óbuda on Elec-trical and Power Engineering (CANDO-EPE). Budapest, Hungary: IEEE Xplore.
Slotosch, O. (2014). Model-Based Tool Qualification. Information Technology and Open Source: Applica-tions for Education, Innovation, and Sustainability.
Taft, S., Richa, E., & Toom, A. (2017). Building Trust in a Model-Based Automatic Code Generator. ACM Digi-tal Library.
Taft, T., & Bordin, M. (2014). Towards a lean tool quali-fication process: Digital avionics systems conference. 2014 IEEE/AIAA 33rd Digital Avionics Systems Con-ference (DASC). Colorado Springs, CO, USA: IEEE Xplore.
Tian, Y., & Wang, P. (2011). Research of software tools for DO-254 projects. 2011 International Conference on Computer Science and Service System (CSSS). Nan-jing: IEEE Xplore.
Tietz, V., Frey, C., Schoepf, J., & Annighoefer, B. (2022). Why the use of domain-specific modeling in airworthy software requires new methods and how these might look like? In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (MODELS '22). ACM Digital Library.
Tietz, V., Schoepf, J., Waldvogel, A., & Annighoefer, B. (2021). A Concept for a Qualifiable (Meta)- Modeling Framework Deployable in Systems and Tools of Safe-ty-Critical and Cyber-Physical Environments. 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MOD-ELS). Fukuoka, Japan: IEEE Xplore.
Trompouki, M., & Kosmidis, L. (2019). BRASIL: A High-Integrity GPGPU Toolchain for Automotive Sys-tems. 2019 IEEE 37th International Conference on Computer Design (ICCD). Abu Dhabi, United Arab Emirates: IEEE Xplore.
Wagner, L., Mebsout, A., Tinelli, C., Cofer, D., & Slind, K. (2017). Qualification of a Model Checker for Avionics Software Verification. NASA Formal Methods. NFM 2017.
Waldvogel, A. (2022). Towards qualifiable graphical editing of complex domain-specific models in safety- critical avionics. In Proceedings of the 25th Interna-tional Conference on Model Driven Engineering Lan-guages and Systems: Companion Proceedings (MOD-ELS '22). New York, NY, USA: ACM Digital Library.
Wang, Q., Wallin, A., Izosimov, V., Ingelsson, U., & Peng, Z. (2012). Test tool qualification through fault injec-tion. 2012 17th IEEE European Test Symposium (ETS). Annecy, France: IEEE Xplore.
Wildmoser, M., Philipps, J., & Slotosch, O. (2012). De-termining Potential Errors in Tool Chains. Computer Safety, Reliability, and Security. SAFECOMP 2012. Springer Link.
Biolchini, J., Mian, P., Natali, A., & Travassos, G. (2005). Systematic Review in Software Engineering. Tech. Rep. RT-ES 679/05, COOPE/UFRJ,, vol. 107, no. 1273 SPEC. ISS., pp. 32–37.
Boulanger, J. (2015). CENELEC 50128 and IEC 62279 Standards. Wiley, 2015. Wiley.
Camus, J.-L.E. (2014). Tool qualification in multiple domains: Status and perspectives. Embedded real-time software and systems (ERTS² 2014).
Federal Aviation Administration. (2017). Advisory Circu-lar 20-115D - Airborne Software Development Assur-ance Using EUROCAE ED-12( ) and RTCA DO-178( ). 2017. FAA.
Federal Aviation Administration. (2018). Order 8110.49A - Software Approval Guidelines. 2018. FAA.
Food and Drugs Administration. (2024, March 23). “Recognized Consensus Standards,” Date of Entry 01/14/2019. (accessed March 2024). Retrieved from [link].
International Electrotechnical Commission. (2015a). IEC 62279:2015 Railway Applications - Communica-tion, Signalling And Processing Systems - Software For Railway Control And Protection Systems. IEC, 2015. IEC.
International Electrotechnical Commission. (2015b). IEC 62304:2006/AMD 1:2015 Medical Device soft-ware — Software life cycle processes — Amendment 1. IEC, 2015. IEC.
International Standardization Organization. (2018). ISO 26262-6:2018 Road vehicles — Functional safety— Part 6: Product development at the software level. ISO, 2018. ISO.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3 EBSE Tech. Rep. EBSE- 2007-01, 2007, doi: 10.1109/ACCESS.2016.2603219.
Knight, J. (2002). Safety critical systems: challenges and directions. Proceedings of the 24th International Con-ference on Software Engineering. ICSE 2002, (pp. 547-550).
Martins, L., & Gorschek, T. (2016). Requirements engi-neering for safety-critical systems: A systematic litera-ture review,” Inf. Softw. Technol., vol. 75, pp. 71–89, 2016, doi: 10.1016/j.infsof.2016.04.002. Inf. Softw. Technol., vol. 75, 71-89.
RTCA Inc. (2011a). DO-331, Model-Based Develop-ment and Verification Supplement to DO- 178C and DO-278A. RTCA.
RTCA Inc. (2011b). DO-332 - Object Oriented Technol-ogy and Related Techniques Supplement to DO-178C and DO-278A. RTCA.
RTCA Inc. (2011c). DO-333 - Formal Methods Supple-ment to DO-178C and DO278A. RTCA.
RTCA, Inc. (1992). DO-178B - Software Considera-tions in Airborne Systems and Equipment Certification. 1992. RTCA.
RTCA, Inc. (2011d). DO-178C – Software Consid-erations in Airborne Systems and Equipment Certifica-tion. 2011. RTCA.
RTCA, Inc. (2011e). DO-248C, Supporting Information for DO-178C and DO-278A. RTCA Inc., 2011. RTCA.
RTCA, Inc. (2011f). DO-330 - Software Tool Qualifica-tion Guidelines. 2011. RTCA.
RTCA, Inc. (2016). DO-200B - Standards for Processing Aeronautical Data RTCA, 2016. RTCA.
Rushby, J. (1994). Critical system properties: survey and taxonomy. Reliability engineering & system safety, Vol.43(2), pp.189-219, doi: https://doi.org/10.1016/0951-8320(94)90065-5.
SAE International. (2012). ARP-4754 A -- Guidelines for Development of Civil Aircraft and Systems. SAE Inter-national, 2012. SAE International.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Giovanni César Borges da Costa, Luiz Eduardo Galvão Martins
This work is licensed under a Creative Commons Attribution 4.0 International License.
