Software Tool Qualification: A Systematic Literature Review
DOI:
https://doi.org/10.5753/jserd.2025.6011Keywords:
Software, Tools, Qualification, Toolchain, Systematic Literature ReviewAbstract
Safety-critical systems are subject to rigorous regulation, demanding compliance with specific criteria before it is made available. In safety-critical domains, this is accomplished through the fulfillment of recognized safety standards requirements or objectives. This SLR has the purpose of identifying approaches or methods to comply with safety standards requirements or objectives related to tool qualification aspects. It also pursued the identification of methods used to demonstrate tool confidence, as well as the activities and artifacts generated to demonstrate that. This study also identified challenges practitioners face to qualify software tools. This SLR was conducted per guidelines proposed by Kitchenham and Biolchini. A search string was developed, and its application returned more than 1,180 papers. For this study, 41 papers were selected, as per the criteria defined in the protocol. These SLR results indicate that there are many studies describing approaches based on established safety standards, for ensuring confidence in tool functionality. The results indicated that research on different proposed methods for tool qualification exists, especially in automotive context. The results also point to the necessity of additional research on toolchain integration. The SLR main findings were the identification of methods for tool qualification, concerns, approaches to address toolchain integration.
Downloads
References
Aiello, M., Comar, C., & Ruiz, J. (2020). An Assurance Case based on Overarching Properties for a TQL1 Code Generator. Embedded Real Time Systems ERTS 2020. IEEE Xplore.
Asplund, F., Biehl, M., & Loiret, F. (2012). Towards the Automated Qualification of Tool Chain Design. Computer Safety, Reliability, and Security. SAFECOMP 2012.
Asplund, F., El-khoury, J., & Törngren, M. (2012). Qualifying Software Tools, a Systems Approach. Computer Safety, Reliability, and Security. SAFECOMP 2012.
Batista, L., & Monsuez, B. (2020). The conception of a large-scale Systems Engineering environment. 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC). San Antonio, TX, USA: IEEE Xplore.
Baumgart, S., Chen, Y., Hamrén, R., & Punnekkat, S. (2021). A Model-Based Approach to Document Software Toolchains for Supporting a Safety Analysis. 2021 IEEE International Systems Conference (SysCon). Vancouver, BC, Canada: IEEE XPlore.
Blackburn, M., & Busser, R. (1996). T-VEC: a toolfor developing critical systems. Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96. Gaithersburg, MD, USA: IEEE Xplore.
Blackburn, M., & Busser, R. (1996). T-VEC: a toolfor developing critical systems. Proceedings of 11th Annual Conference on Computer Assurance. COMPASS '96. Gaithersburg, MD, USA: IEEE Xplore.
Borgh, A., & Towhidnejad, M. (2019). Guidelines for Development of Operational Requirements for Model Checking Tools. 2019 Integrated Communications, Navigation and Surveillance Conference (ICNS). Herndon, VA, USA: IEEE Xplore.
Brauer, J., Peleska, J., & Schulze, U. (2012). Efficient and Trustworthy Tool Qualification for Model-Based Testing Tools. Testing Software and Systems. ICTSS 2012.
Cofer, D. (2015). You keep using that word. ACM SIGLOG News 2, 4 (October 2015), 17-25.
da Silva, F., Bagbaba, A., Hamdioui, S., & Sauer, C. (2019). Efficient Methodology for ISO26262 Functional Safety Verification. 2019 IEEE 25th International Symposium on On-Line Testing and Robust System Design (IOLTS). Rhodes, Greece: IEEE Xplore.
de la Vara, J., Ruiz, A., & Blondelle, G. (2021). Assurance and certification of cyber–physical systems: The AMASS open source ecosystem. Journal of Systems and Software.
Durak, U., D’Ambrogio, A., & Bocciarelli, P. (2020). Safety-criticalsimulation engineering. In Proceedings of the 2020 Summer Simulation Conference (SummerSim '20). San Diego, CA, USA: ACM Digital Library.
Gallina, B., Kashiyarandi, S., Zugsbratl, K., & Geven, A. (2014). Enabling Cross-Domain Reuse of Tool Qualification Certification Artefacts. Computer Safety, Reliability, and Security. SAFECOMP 2014.
Gleirscher, M., Sachtleben, R., & Peleska, J. (2023). Qualification of proof assistants, checkers, and generators: Where are we and what next? Science of Computer Programming.
Guo, Y., & Slay, J. (2010). A Function Oriented Methodology to Validate and Verify Forensic Copy Function of Digital Forensic Tools. 2010 International Conference on Availability, Reliability and Security. Krakow, Poland: IEEE Xplore.
Guo, Y., Slay, J., & Beckett, J. (2009). Validation and verification of computer forensic software tools— Searching Function. Digital Investigation, S12-S22.
Horsman, G. (2019). Tool testing and reliability issues in the field of digital forensics. Digital Investigation, 163-175.
Hughes, N., Rabieh, K., & Aydogan, A. (2019). Employing a Continuous Measurement Process During Digital Tool Validation. 2019 International Symposium on Networks, Computers and Communications (ISNCC). Istanbul, Türkiye: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2003). Design tool assessment for safety-critical software development. 28th Annual NASA Goddard Software Engineering Workshop, 2003. Proceedings. Greenbelt, MD, USA: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2008). Software certification for safety-critical systems: A status report. 2008 International Multiconference on Computer Science and Information Technology. Wisla, Poland: IEEE Xplore.
Kornecki, A., & Zalewski, J. (2010). Hardware certification for real-time safety-criticalsystems: State of the art. Annual Reviews in Control, 163-174.
Kornecki, A., Hall, K., Hearn, D., Lau, H., & Zalewski, J. (2004). Evaluation of software development tools for high assurance safety critical systems. Eighth IEEE International Symposium on High Assurance Systems Engineering, 2004. Proceedings. Tampa, FL, USA: IEEE Xplore.
Krauss, S., Rejzek, M., & Hilbes, C. (2015). Tool Qualification Considerations for Tools Supporting STPA. Procedia Engineering, 15-24.
Marques, J., & da Cunha, A. (2015). Use of the RTCA DO330 in aeronautical databases. 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC). Prague, Czech Republic.
Marques, J., & da Cunha, A. (2017). COTS tool qualification using RTCA DO-330: Common pitfalls. 2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC). St. Petersburg, FL, USA: IEEE Xplore.
Marques, J., & da Cunha, A. (2017). Verification scenarios of onboard databases under the RTCA DO-178C and the RTCA DO-200B. 2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC). St. Petersburg, FL, USA: IEEE Xplore.
Mishra, A., Rao, M., CU, C., Rao, V., Jeppu, Y., & Murthy, N. (2013). An auto-review tool for modelbased testing of safety-critical systems. In Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation (JAMAICA 2013). New York, NY, USA: ACM Digital Library.
Prabhu, S., Kapil, H., & Lakshmaiah, S. (2018). Safety Critical Embedded Software: Significance and Approach to Reliability. 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI). Bengaluru, India: IEEE Xplore.
Schuster, G. (2022). Certification of software tools used in safety-critical software development. IEEE 5th International Conference and Workshop Óbuda on Electrical and Power Engineering (CANDO-EPE). Budapest, Hungary: IEEE Xplore.
Slotosch, O. (2014). Model-Based Tool Qualification. Information Technology and Open Source: Applications for Education, Innovation, and Sustainability.
Taft, S., Richa, E., & Toom, A. (2017). Building Trust in a Model-Based Automatic Code Generator. ACM Digital Library.
Taft, T., & Bordin, M. (2014). Towards a lean tool qualification process: Digital avionics systems conference. 2014 IEEE/AIAA 33rd Digital Avionics Systems Conference (DASC). Colorado Springs, CO, USA: IEEE Xplore.
Tian, Y., & Wang, P. (2011). Research of software tools for DO-254 projects. 2011 International Conference on Computer Science and Service System (CSSS). Nanjing: IEEE Xplore.
Tietz, V., Frey, C., Schoepf, J., & Annighoefer, B. (2022). Why the use of domain-specific modeling in airworthy software requires new methods and how these might look like? In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (MODELS '22). ACM Digital Library.
Tietz, V., Schoepf, J., Waldvogel, A., & Annighoefer, B. (2021). A Concept for a Qualifiable (Meta)- Modeling Framework Deployable in Systems and Tools of SafetyCritical and Cyber-Physical Environments. 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS). Fukuoka, Japan: IEEE Xplore.
Trompouki, M., & Kosmidis, L. (2019). BRASIL: A High-Integrity GPGPU Toolchain for Automotive Systems. 2019 IEEE 37th International Conference on Computer Design (ICCD). Abu Dhabi, United Arab Emirates: IEEE Xplore.
Wagner, L., Mebsout, A., Tinelli, C., Cofer, D., & Slind, K. (2017). Qualification of a Model Checker for Avionics Software Verification. NASA Formal Methods. NFM 2017.
Waldvogel, A. (2022). Towards qualifiable graphical editing of complex domain-specific models in safety- critical avionics. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings (MODELS '22). New York, NY, USA: ACM Digital Library.
Wang, Q., Wallin, A., Izosimov, V., Ingelsson, U., & Peng, Z. (2012). Test tool qualification through fault injection. 2012 17th IEEE European Test Symposium (ETS). Annecy, France: IEEE Xplore.
Wildmoser, M., Philipps, J., & Slotosch, O. (2012). Determining Potential Errors in Tool Chains. Computer Safety, Reliability, and Security. SAFECOMP 2012. Springer Link.
Biolchini, J., Mian, P., Natali, A., & Travassos, G. (2005). Systematic Review in Software Engineering. Tech. Rep. RT-ES 679/05, COOPE/UFRJ,, vol. 107, no. 1273 SPEC. ISS., pp. 32–37.
Boulanger, J. (2015). CENELEC 50128 and IEC 62279 Standards. Wiley, 2015. Wiley.
Camus, J.-L.E. (2014). Tool qualification in multiple domains: Status and perspectives. Embedded real-time software and systems (ERTS² 2014).
Federal Aviation Administration. (2017). Advisory Circular 20-115D - Airborne Software Development Assurance Using EUROCAE ED-12( ) and RTCA DO-178( ). 2017. FAA.
Federal Aviation Administration. (2018). Order 8110.49A - Software Approval Guidelines. 2018. FAA.
Food and Drugs Administration. (2024, March 23). “Recognized Consensus Standards,” Date of Entry 01/14/2019. (accessed March 2024). Retrieved from [link].
International Electrotechnical Commission. (2015a). IEC 62279:2015 Railway Applications - Communication, Signalling And Processing Systems - Software For Railway Control And Protection Systems. IEC, 2015. IEC.
International Electrotechnical Commission. (2015b). IEC 62304:2006/AMD 1:2015 Medical Device software — Software life cycle processes — Amendment 1. IEC, 2015. IEC.
International Standardization Organization. (2018). ISO 26262-6:2018 Road vehicles — Functional safety— Part 6: Product development at the software level. ISO, 2018. ISO.
Kitchenham, B., & Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Version 2.3 EBSE Tech. Rep. EBSE-2007-01, 2007, doi: 10.1109/ACCESS.2016.2603219.
Knight, J. (2002). Safety critical systems: challenges and directions. Proceedings of the 24th International Conference on Software Engineering. ICSE 2002, (pp. 547-550).
Martins, L., & Gorschek, T. (2016). Requirements engineering for safety-critical systems: A systematic literature review,” Inf. Softw. Technol., vol. 75, pp. 71–89, 2016, doi: 10.1016/j.infsof.2016.04.002. Inf. Softw. Technol., vol. 75, 71-89.
RTCA Inc. (2011a). DO-331, Model-Based Development and Verification Supplement to DO-178C and DO-278A. RTCA.
RTCA Inc. (2011b). DO-332 - Object Oriented Technology and Related Techniques Supplement to DO-178C and DO-278A. RTCA.
RTCA Inc. (2011c). DO-333 - Formal Methods Supplement to DO-178C and DO-278A. RTCA.
RTCA, Inc. (1992). DO-178B - Software Considerations in Airborne Systems and Equipment Certification. 1992. RTCA.
RTCA, Inc. (2011d). DO-178C – Software Considerations in Airborne Systems and Equipment Certification. 2011. RTCA.
RTCA, Inc. (2011e). DO-248C, Supporting Information for DO-178C and DO-278A. RTCA Inc., 2011. RTCA.
RTCA, Inc. (2011f). DO-330 - Software Tool Qualification Guidelines. 2011. RTCA.
RTCA, Inc. (2016). DO-200B - Standards for Processing Aeronautical Data. RTCA, 2016. RTCA.
Rushby, J. (1994). Critical system properties: survey and taxonomy. Reliability engineering & system safety, Vol. 43(2), pp. 189-219, doi: https://doi.org/10.1016/0951-8320(94)90065-5.
SAE International. (2012). ARP-4754 A -- Guidelines for Development of Civil Aircraft and Systems. SAE International, 2012. SAE International.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Giovanni César Borges da Costa, Luiz Eduardo Galvão Martins
This work is licensed under a Creative Commons Attribution 4.0 International License.
