Software Operational Profile vs. Test Profile: Towards a Better Software Testing Strategy
DOI:
https://doi.org/10.5753/jserd.2020.546Keywords:
Software Quality, Software Testing, Operational Profile, Test ProfileAbstract
Software Operational Profile (SOP) is a software specification based on how users use the software. This specification corresponds to a quantitative representation of software that identifies the most used software parts. As software reliability depends on the context in which users operate the software, the SOP is used in software reliability engineering. However, there are evidences of a misalignment between the software tested parts and SOP. Therefore, this paper investigates a possible misalignment between SOP and the tested software parts to obtain, based on experimental data, more evidence of this misalignment. We performed an exploratory study composed of four activities to verify: a) whether there are significant variations in how users operate the software; b) whether there is a misalignment between SOP and the tested software parts; c) if failures occur in untested SOP parts in case of misalignment; d) in case of misalignment between SOP and untested software parts, whether a test strategy based on the amplification of the existent test set with additional test data generated automatically, can contribute to reduce the misalignment. We collected data form four software while users were operating them. We analyzed this collected data in an attempt to reach the goals of this work. To evaluate the originality of this research, we performed a Literature Systematic Review (SLR) and presented its conclusions. The obtained results evidence that there are significant variations in how users operate the software and also that there is a misalignment between SOP and the tested software parts when we evaluated the four software mentioned above. There are also indications of the occurrence of failures in the untested SOP parts. Although the test strategy mentioned above has reduced the possible misalignment, the test strategy is not enough to avoid it, thus denoting the need of specifics test strategies using SOP as a test criterion. These results indicate that SOP becomes relevant not only to software reliability engineering but also to contribute to testing activities, regardless of the adopted strategy.
Downloads
References
Amrita and Yadav, D. K. (2015). A novel method for allocating software test cases. In 3rd International Conference on Recent Trends in Computing 2015 (ICRTC-2015), volume 57, pages 131 – 138, Delhi, India. Elsevier.
Assesc, F. (2012). Propriedade intelectual e software - cursos de mídia eletrônica e sistema de informação.
Bach, T., Andrzejak, A., Pannemans, R., and Lo, D. (2017). The impact of coverage on bug density in a large industrial software project. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 307–313.
Begel, A. and Zimmermann, T. (2014). Analyze this! 145 questions for data scientists in software engineering. ICSE 2014, pages 12–23.
Bertolino, A., Miranda, B., Pietrantuono, R., and Russo, S. (2017b). Adaptive coverage and operational profile-based testing for reliability improvement. In Proceedings of the 39th International Conference on Software Engineering, ICSE ’17, pages 541–551, Piscataway, NJ, USA. IEEE Press.
Bittanti, S., Bolzern, P., and Scattolini, R. (1988). An introduction to software reliability modelling, chapter 12, pages 43–67. Springer Berlin Heidelberg, Berlin, Heidelberg.
Cavamura Júnior, L. (2019). Operational profile and software testing: Aligning user interest and test strategy. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pages 492–494.
Chen, M. ., Lyu, M. R., and Wong, W. E. (2001). Effect of code coverage on software reliability measurement. IEEE Transactions on Reliability, 50(2):165–170.
Cukic, B. and Bastani, F. B. (1996). On reducing the sensitivity of software reliability to variations in the operational profile. pages 45–54, White Plains, NY, USA. IEEE, Los Alamitos, CA, United States.
Falbo, R. A. (2005). Engenharia de software. Ferrari, F. C., Rashid, A., and Maldonado, J. C. (2013). Towards the practical mutation testing of aspectj programs. Science of Computer Programming, 78(9):1639 – 1662.
Fraser, G. and Arcuri, A. (2011). Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE ’11, pages 416–419, New York, NY, USA. ACM.
Fukutake, H., Xu, L., Takagi, T., Watanabe, R., and Yaegashi, R. (2015). The method to create test suite based on operational profiles for combination test of status. pages 1–4, White Plains, NY, USA. Institute of Electrical and Electronics Engineers Inc.
Gittens, M., Lutfiyya, H., and Bauer, M. (2004). An extended operational profile model. pages 314 – 325, Saint-Malo, France.
Kashyap, A. (2013). A Markov Chain and Likelihood-Based Model Approach for Automated Test Case Generation, Validation and Prioritization: Theory and Application. PhD thesis.
Laddad, R. (2009). AspectJ in Action: Enterprise AOP with Spring Applications. Manning Publications Co., Greenwich, CT, USA, 2nd edition.
Leung, Y.-W. (1997). Software reliability allocation under an uncertain operational profile. Journal of the Operational Research Society, 48(4):401 – 411.
Musa, J. (1993). Operational profiles in software-reliability engineering. IEEE Software, 10(2):14–32. cited By 396.
Musa, J. and Ehrlich, W. (1996). Advances in software reliability engineering. Advances in Computers, 42(C):77–117.
Musa, J. D. (1979). Software reliability measures applied to systems engineering. In Managing Requirements Knowledge, International Workshop on(AFIPS), volume 00, page 941, S.I. IEEE.
Musa, J. D. (1994). Adjusting measured field failure intensity for operational profile variation. In Proceedings of the International Symposium on Software ReliabilityEngineering, ISSRE, pages 330–333, Monterey, CA, USA. IEEE, Los Alamitos, CA, United States.
Namba, Y., Akimoto, S., and Takagi, T. (2015). Overview of graphical operational profiles for generating test cases of gui software. pages 1–3, White Plains, NY, USA. Institute of Electrical and Electronics Engineers Inc.
Poore, J., Walton, G., and Whittaker, J. (2000). A constraintbased approach to the representation of software usage models. Information and Software Technology, 42(12):825 – 833.
Pressman, R. S. (2010). Software Engineering A Practitioner’s Approach. McGraw-Hill, New York, NY, 7rd edition.
Rincon, A. M. (2011). Qualidade de conjuntos de teste de software de código aberto, uma análise baseada em critérios estruturais.
Rocha, A. D. (2005). Uma ferramenta baseada em aspectos para apoio ao teste funcional de programas java.
Shukla, R. (2009). Deriving parameter characteristics. pages 57 – 63, New York, NY, USA. ACM.
Sommerville, I. (1995). Software Engineering. AddisonWesley, Wokingham, England, fifth edition edition.
Sommerville, I. (2011). Software Engineering. AddisonWesley, Harlow, England, 9 edition.
Spínola, R. O., Dias-neto, A. C., and Travassos, G. H. (2008). Abordagem para desenvolver tecnologia de software com apoio de estudos secundários e primários. In 5th Experimental Software Engineering Latin American Workshop, São Carlos, SP, Brasil. ICMC/USP.
Takagi, T., Furukawa, Z., and Yamasaki, T. (2007). An overview and case study of a statistical regression testing method for software maintenance. Electronics and Communications in Japan Part II Electronics, 90(12):23–34.
van Solingen, R., Basili, V., Caldiera, G., and Rombach, H. D. (2002). Goal Question Metric (GQM) Approach. American Cancer Society.
