Test case prioritization methods: A systematic literature review
DOI:
https://doi.org/10.5753/jserd.2025.4504Keywords:
Test case prioritization, Software testing, Software quality, Methods for prioritizing test casesAbstract
Software testing is a process that assists in achieving software quality. However, software testing used to be a repetitive activity that takes a long time to be performed. Several methods for test case prioritization have been created with the purpose to reduce the time of software testing activity without compromising its quality. The main idea of this review is to examine and classify test case prioritization methods based on the research questions created. In order to achieve the outlined objective, seventy-one primary studies that propose, analyze, or compare test case prioritization methods were used as a basis for the analysis. Sixty-eight of them were found by search string and four studies were manually included by the author. The main methods of each group were mapped, analyzed and described. Despite the existence of many methods for prioritizing test cases, none of them were designed especially to be applied in agile environments, therefore there are opportunities for improvement in this topic.
Downloads
References
Alazzam, I. and Nahar, K. M. O. (2018). Combined source code approach for test case prioritization. In Proceedings of the 2018 International Conference on Information Sci- ence and System - ICISS ’18, New York, New York, USA. ACM Press.
Alspaugh, S., Walcott, K. R., Belanich, M., Kapfhammer, G. M., and Soffa, M. L. (2007). Efficient time-aware prioritization with knapsack solvers. In Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007 - WEASELTech ’07, New York, New York, USA. ACM Press.
Arafeen, M. J. and Do, H. (2013). Test case prioritization using requirements-based clustering. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation.
Ashima, G. Shaheamlung and K. Rote. (2020). A comprehensive review for test case prioritization in Software Engineering, 2020 International Conference on Intelligent Engineering and Management (ICIEM), London, UK, 2020, pp. 331-336, doi: 10.1109/ICIEM48762.2020.9160217.
Biswas, S., Rathi, R., Dutta, A., Mitra, P., and Mall, R. (2022). A regression test case prioritization technique targeting ‘hard to detect’ faults. International Journal of System Assurance Engineering and Management, pages 1066–1081.
Black, R. (2019). Foundations of software testing: ISTQB certification (4th ed.). Cengage Learning.
Brooks, Jr, F. (2021). Mythical man-month, anniversary edition, the. Addison Wesley, Boston, MA, 2 edition.
Bryce, R. C., Sampath, S., Pedersen, J. B., and Manchester, S. (2011). Test suite prioritization by cost-based combinatorial interaction coverage. Int. J. Syst. Assur. Eng. Manag., 2(2):126–134.
Campos Junior, H. S., Araújo, M. A. P., David, J. M. N., Braga R., Campos, F. and Ströele, V. (2017). Test case prioritization: a systematic review and mapping of the literature. In Proceedings of the XXXI Brazilian Symposium on Software Engineering (SBES '17). Association for Computing Machinery, New York, NY, USA, 34–43. https://doi.org/10.1145/3131151.3131170
Carlson, R., Do, H., and Denton, A. (2011). A clustering approach to improving test case prioritization: An industrial case study. In 2011 27th IEEE International Conference on Software Maintenance (ICSM). IEEE.
Catal, C. and Mishra, D. (2013). Test case prioritization: a systematic mapping study. Softw. Qual. J., 21(3):445–478.
Chen, T. Y., Leung, H., and Mak, I. K. (2004). Adaptive random testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, Lecture notes in computer science, pages 320–329. Springer Berlin Heidel- berg, Berlin, Heidelberg.
Dobuneh, M. R. N., Jawawi, D. N., Ghazali, M., and Malakooti, M. V. (2014). Development test case prioritization technique in regression testing based on hybrid criteria. In 8th. Malaysian Software Engineering Conference (MySEC), pages 301–305.
Eghbali, S. and Tahvildari, L. (2016). Test case prioritization using lexicographical ordering. IEEE Trans. Softw. Eng., 42(12):1178–1195.
Elbaum, S., Malishevsky, A., and Rothermel, G. (2001). Incorporating varying test costs and fault severities into test case prioritization. In Proceedings of the 23rd Interna- tional Conference on Software Engineering. ICSE 2001, pages 329–338.
Elbaum, S., Malishevsky, A. G., and Rothermel, G. (2000). Prioritizing test cases for regression testing. Softw. Eng. Notes, 25(5):102–112.
Fairley, R. E. (1985). Software engineering concepts. In McGraw-Hill series in software engineering and technology.
Fang, C., Chen, Z., Wu, K., and Zhao, Z. (2014). Similarity- based test case prioritization using ordered sequences of program entities. Softw. Qual. J., 22(2):335–361.
Fazlalizadeh, Y., Khalilian, A., Abdollahi Azgomi, M., and Parsa, S. (2009). Incorporating historical test case perfor- mance data and resource constraints into test case priori- tization. In Tests and Proofs, Lecture notes in computer science, pages 43–57. Springer Berlin Heidelberg, Berlin, Heidelberg.
Ganjkhani, E. and Afsharchi, M. (2019). An effective test case prioritization by combination of strategies. SN Appl. Sci., 1(9).
Gokilavani, N. and Bharathi, B. (2021). Test case prioriti- zation to examine software for fault detection using PCA extraction and k-means clustering with ranking. Soft Com- put., 25(7):5163–5172.
Hao, D., Zhang, L., Zhang, L., Rothermel, G., and Mei, H. (2014). A unified test case prioritization approach. ACM Trans. Softw. Eng. Methodol., 24(2):1–31.
Hasnain, M., Pasha, M.F., Ghani, I. et al. Functional Requirement-Based Test Case Prioritization in Regression Testing: A Systematic Literature Review. SN COMPUT. SCI. 2, 421 (2021). https://doi.org/10.1007/s42979-021-00821-3
Hettiarachchi, C., Do, H., and Choi, B. (2014). Effective regression testing using requirements and risks. In 2014, the Eighth International Conference on Software Security and Reliability.
Hettiarachchi, C., Do, H., and Choi, B. (2016). Risk-based test case prioritization using a fuzzy expert system. Inf. Softw. Technol., 69:1–15.
Huang, R., Chen, J., Li, Z., Wang, R., and Lu, Y. (2014). Adaptive random prioritization for interaction test suits. In Proceedings of the 29th Annual ACM Symposium on Applied Computing, New York, NY, USA. ACM.
Huang, R., Zhang, Q., Towey, D., Sun, W., and Chen, J. (2020). Regression test case prioritization by code combinations coverage.
Huang, Y.-C., Peng, K.-L., and Huang, C.-Y. (2012). A history-based cost-cognizant test case prioritization technique in regression testing. J. Syst. Softw., 85(3):626–637.
Jahan, H., Feng, Z., and Mahmud, S. M. H. (2020). Risk- based test case prioritization by correlating system methods and their associated risks. Arab. J. Sci. Eng., 45(8):6125–6138.
Jiang, B., Zhang, Z., Chan, W. K., and Tse, T. H. (2009). Adaptive random test case prioritization. In 2009 IEEE/ACM International Conference on Automated Soft- ware Engineering, pages 233–244.
Jones, J. A. and Harrold, M. J. (2003). Test-suite reduction and prioritization for modified condition/decision cover- age. IEEE Trans. Softw. Eng., 29(3):195–209.
Kavitha, R., Kavitha, V. R., and Suresh Kumar, N. (2010). Requirement based test case prioritization. In 2010 IEEE International Conference on Communication Control and Computing Technologies.
Khalid, Z. and Qamar, U. (2019). Weight and cluster based test case prioritization technique. In 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON).
Khalilian, A., Abdollahi Azgomi, M., and Fazlalizadeh, Y. (2012). An improved method for test case prioritization by incorporating historical test case data. Sci. Comput. Pro- gram., 78(1):93–116.
Khatibsyarbini, M., Isa, M. A., Jawawi, D. N. A., and Tu- meng, R. (2018). Test case prioritization approaches in regression testing: A systematic literature review. Inf. Softw. Technol., 93:74–93.
Kim, J., Jeong, H., and Lee, E. (2017). Failure history data-based test case prioritization for effective regression test. In Proceedings of the Symposium on Applied Computing, New York, NY, USA.
Kim, J.-M. and Porter, A. (2003). A history-based test prioritization technique for regression testing in resource con- strained environments. In Proceedings of the 24th International Conference on Software Engineering. ICSE 2002.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering.
Korel, B., Koutsogiannakis, G., and Tahat, L. H. (2008). Application of system models in regression test suite prioritization. In 2008 IEEE International Conference on Software Maintenance.
Korel, B., Tahat, L., and Harman, M. (2005). Test prioritization using system models. In 21st IEEE International Conference on Software Maintenance (ICSM’05), pages 559–568.
Krishnamoorthi, R. and Sahaaya Arul Mary, S. A. (2009). Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf. Softw. Technol., 51(4):799–808.
Kwon, J.-H., Ko, I.-Y., Rothermel, G., and Staats, M. (2014). Test case prioritization based on information retrieval concepts. In 2014 21st IEEE Asia-Pacific Software Engineering Conference.
Leon, D. and Podgurski, A. (2003). A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In 14th IEEE International Symposium on Software Reliability Engineering, 2003. ISSRE 2003.
Lin, C.-T., Tang, K.-W., and Kapfhammer, G. M. (2014). Test suite reduction methods that decrease regression testing costs by identifying irreplaceable tests. Inf. Softw. Technol., 56(10):1322–1344.
Liu, W., Wu, X., Zhang, W., and Xu, Y. (2014). The re- search of the test case prioritization algorithm for black box testing. In 2014 IEEE 5th International Conference on Software Engineering and Service Science.
Ma, T., Zeng, H., and Wang, X. (2016). Test case prioritization based on requirement correlations. In 2016 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD).
Magalhães, C., Mota, A., and Momente, L. (2021). UI test case prioritization on an industrial setting: A search for the best criteria. Softw. Qual. J., 29(2):381–403.
Mahali, P. and Mohapatra, D. P. (2018). Model based test case prioritization using UML behavioural diagrams and association rule mining. Int. J. Syst. Assur. Eng. Manag., 9(5):1063–1079.
Malz, C. and Göhner, P. (2011). Agent-Based test case prioritization. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops.
Marijan, D., Gotlieb, A., and Sen, S. (2013). Test case prioritization for continuous regression testing: An industrial case study. In 2013 IEEE International Conference on Software Maintenance.
Masri, W. and El-Ghali, M. (2009). Test case filtering and prioritization based on coverage of combinations of pro- gram elements. In Proceedings of the Seventh International Workshop on Dynamic Analysis, New York, NY, USA.
Mohd-Shafie, M.L., Kadir, W.M.N.W., Lichter, H. et al. Model-based test case generation and prioritization: a systematic literature review. Softw Syst Model 21, 717–753 (2022). https://doi.org/10.1007/s10270-021-00924-8
Nie, C. and Leung, H. (2011). A survey of combinatorial testing. ACM Comput. Surv., 43(2):1–29.
Panda, N., Acharya, A. A., and Mohapatra, D. P. (2019). Test scenario prioritization for object-oriented systems using UML diagram. Int. J. Syst. Assur. Eng. Manag., 10(3):316–325.
Panda, N. and Mohapatra, D. P. (2021). Test scenario prioritization from user requirements for web-based software. Int. J. Syst. Assur. Eng. Manag., 12(3):361–376.
Panigrahi, C. R. and Mall, R. (2014). A heuristic-based regression test case prioritization approach for object- oriented programs. Innov. Syst. Softw. Eng., 10(3):155– 163.
Parashar, P., Kalia, A., and Bhatia, R. (2012). Pair-wise time- aware test case prioritization for regression testing. In Information Systems, Technology and Management, Communications in computer and information science, pages 176–186. Springer Berlin Heidelberg, Berlin, Heidelberg.
Park, H., Ryu, H., and Baik, J. (2008). Historical value-based approach for cost-cognizant test case prioritization to improve the effectiveness of regression testing. In 2008, the Second International Conference on Secure System Integration and Reliability Improvement.
Pressman, R. and Maxim, B. (2019). Loose leaf for software engineering: A practitioner’s approach. McGraw-Hill Education, 9th edition.
Rahmani, A., Ahmad, S., Jalil, I. E. A. and Herawan, A. P. (2021). A Systematic Literature Review on Regression Test Case Prioritization. International Journal of Advanced Computer Science and Applications(IJACSA), 12(9), 2021. http://dx.doi.org/10.14569/IJACSA.2021.0120929
Romano, S., Scanniello, G., Antoniol, G., and Marchetto, A. (2018). Spiritus: A simple information retrieval regression test selection approach. information and software technol- ogy, v. 99. pages 62–80.
Rothermel, G., Untch, R. H., Chu, C., and Harrold, M. J. (1999). Test case prioritization: an empirical study. In Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM’99). ’Software Maintenance for Business Change’ (Cat. No.99CB36360).
Sapna, P. G. and Mohanty, H. (2009). Prioritization of scenarios based on uml activity diagrams. In 2009 First International Conference on Computational Intelligence, Communication Systems and Networks, pages 271–276.
Siddik, M. S. and Sakib, K. (2014). RDCC: An effective test case prioritization framework using software requirements, design and source code collaboration. In 2014 17th International Conference on Computer and Information Technology (ICCIT).
Singh, Y., Kaur, A., Suri, B., and Singhal, S. (2012). Systematic literature review on regression test prioritization techniques. Informatica (Ljubl.), 36(4).
Solanki, K., Singh, Y., and Dalal, S. (2015). Test case prioritization: An approach based on modified ant colony optimization (m-ACO). In 2015 International Conference on Computer, Communication and Control (IC4).
Sommerville, I. (2011). Software Engineering. Pearson, Up- per Saddle River, NJ, 9th edition.
Srikanth, H., Cashman, M., and Cohen, M. B. (2016a). Test case prioritization of build acceptance tests for an enterprise cloud application: An industrial case study. J. Syst. Softw., 119:122–135.
Srikanth, H., Hettiarachchi, C., and Do, H. (2016b). Requirements based test prioritization using risk factors: An industrial study. Inf. Softw. Technol., 69:71–83.
Srikanth, H., Williams, L., and Osborne, J. (2005). System test case prioritization of new and regression test cases. In 2005 International Symposium on Empirical Software Engineering, 2005.
Srivastva, P. R., Kumar, K., and Raghurama, G. (2008). Test case prioritization based on requirements and risk factors. Softw. Eng. Notes, 33(4):1–5.
Su, W., Li, Z., Wang, Z., and Yang, D. (2020). A meta- heuristic test case prioritization method based on hybrid model. In 2020 International Conference on Computer Engineering and Application (ICCEA).
Thomas, S. W., Hemmati, H., Hassan, A. E., and Blostein, D. (2014). Static test case prioritization using topic models. Empir. Softw. Eng., 19(1):182–212.
Tonella, P., Avesani, P., and Susi, A. (2006). Using the case- based ranking methodology for test case prioritization. In 2006 22nd IEEE International Conference on Software Maintenance.
Vedpal, Chauhan, N., and Kumar, H. (2014). A hierarchical test case prioritization technique for object oriented soft- ware. In 2014 International Conference on Contemporary Computing and Informatics (IC3I).
Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., and Liaaen, M. (2014). Multi-objective test prioritization in software product line testing: an industrial case study. Proceedings of the 18th International Software Product Line Conference, 1:32–41.
Wang, X., Jiang, X., and Shi, H. (2015a). Prioritization of test scenarios using hybrid genetic algorithm based on UML activity diagram. In 2015 6th IEEE International Conference on Software Engineering and Service Science (IC- SESS).
Wang, X. and Zeng, H. (2016). History-based dynamic test case prioritization for requirement properties in regression testing. In 2016 IEEE/ACM International Workshop on Continuous Software Evolution and Delivery (CSED), pages 41–47.
Wang, Y., Zhao, X., and Ding, X. (2015b). An effective test case prioritization method based on fault severity. In 2015 6th IEEE International Conference on Software Engineering and Service Science (ICSESS).
Wu, H., Nie, C., and Kuo, F.-C. (2014). Test suite prioritization by switching cost. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation Workshops.
Yadav, D. K. and Dutta, S. (2016). Test case prioritization technique based on early fault detection using fuzzy logic. In 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), pages 1033–1036.
Yadav, D. K. and Dutta, S. (2020). Regression test case se- lection and prioritization for object oriented software. Mi- crosyst. Technol., 26(5):1463–1477.
Yoo, S. and Harman, M. (2012). Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab., 22(2):67–120.
Zhang, L., Hao, D., Zhang, L., Rothermel, G., and Mei, H. (2013). Bridging the gap between the total and additional test-case prioritization strategies. In 2013 35th International Conference on Software Engineering (ICSE).
Zhou, Z. Q. (2010). Using coverage information to guide test case selection in adaptive random testing. In 2010 IEEE 34th Annual Computer Software and Applications Conference Workshops.
Zhu, H., Hall, P. A. V., and May, J. H. R. (1997). Software unit test coverage and adequacy. ACM Comput. Surv., 29(4):366–427.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Icaro Prado Fernandes, Luiz Eduardo Galvão Martins
This work is licensed under a Creative Commons Attribution 4.0 International License.
