ADoTe: Approach to teaching and learning functional testing technique criteria supported by Testing Dojo
DOI:
https://doi.org/10.5753/jserd.2025.4103Keywords:
Software Engineering Education, Software Testing, Functional Testing Criteria, Testing DojoAbstract
Context: Students of computer science and related courses (including those who have already graduated) often have knowledge gaps on software testing and low levels of motivation and interest in learning this subject. Functional testing technique criteria, for example, are usually required and used in industry, but their teaching still needs support. Faced with the problems caused by such gaps and a historical shortage of qualified testing professionals, improvements in testing education are demanded by the industry for the academy. Dojos may contribute in this context since positive results are usually reported regarding their impact on the motivation and perception of learning of students. However, their use is still little explored in the testing area. Objective: In this study we aimed to define and evaluate an approach supported by testing dojo to teaching and learning functional testing technique criteria in higher education. Methods: We defined the approach (named ADoTe) iteratively, supported by literature data and by quali-quantitative analysis of the results of two executions of a controlled experiment that evaluated the impact of ADoTe (compared to a traditional teaching approach) on the learning (measured by knowledge tests) and motivation (measured by a questionnaire named Intrinsic Motivation Inventory (IMI)) of students. Results: From the analysis of data collected from 44 participants we noticed that, although the averages for learning and general motivation of the groups submitted to ADoTe were greater than those of the groups submitted to a traditional teaching approach, it was not possible to state that, for these variables, there was a statistically significant difference between the approaches evaluated. However, the results of the IMI subscale related to intrinsic motivation were statistically significant, indicating that students feel more interest/enjoyment in learning functional testing technique criteria through ADoTe (the averages of the groups submitted to ADoTe were 15.80% and 19.71% higher than those of the control groups). Additionally, the results of thematic analyses carried out on the answers to a retrospective questionnaire showed that ADoTe was well accepted by students and reinforced the importance of its steps and guidelines. Conclusion: ADoTe positively impacts the learning and motivation of students and, compared to traditional teaching, tends to lead to greater levels of interest/enjoyment in learning functional testing technique criteria.
Downloads
References
Aniche, M., Hermans, F., and van Deursen, A. (2019). Pragmatic software testing education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education, pages 414–420, New York. ACM.
Aniche, M. F. and Silveira, G. d. A. (2011). Increasing learning in an agile environment: Lessons learned in an agile team. In 2011 Agile Conference, pages 289–295.
Bache, E. (2013). The Coding Dojo Handbook: a practical guide to creating a space where good programmers can become great programmers. Leanpub, 1st edition.
Barbetta, P. A., Reis, M. M., and Bornia, A. C. (2010). Estatística Para Cursos de Engenharia e Informática. Atlas, São Paulo, 3rd edition.
Bossavit, L. and Gaillot, E. (2005). The coder’s dojo – A different way to teach and learn programming. In Baumeister, H., Marchesi, M., and Holcombe, M., editors, Extreme Programming and Agile Processes in Software Engineering, pages 290–291, Berlin, Heidelberg. Springer.
Bowman, N. A., Logel, C., LaCosse, J., Jarratt, L., Canning, E. A., Emerson, K. T. U., and Murphy, M. C. (2022). Gender representation and academic achievement among STEM-interested students in college STEM courses. Journal of Research in Science Teaching, 59(10):1876–1900.
Bravo, M. and Goldman, A. (2010). Reinforcing the learning of agile practices using coding dojos. In Sillitti, A., Martin, A., Wang, X., and Whitworth, E., editors, Agile Processes in Software Engineering and Extreme Programming, pages 379–380, Berlin, Heidelberg. Springer.
Carpinetti, L. C. R. and Gerolamo, M. C. (2016). Gestão da qualidade ISO 9001: 2015: requisitos e integração com a ISO 14001:2015. Atlas, Rio de Janeiro, 1st edition.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum Associates, New York, 2nd edition.
Costa, I. E. F. and Oliveira, S. R. B. (2019). A systematic strategy to teaching of exploratory testing using gamification. In Proceedings of the 14th International Conference on Evaluation of Novel Approaches to Software Engineering, pages 307–314. SciTePress.
Costa, I. E. F. and Oliveira, S. R. B. (2022). Development of a teaching plan to support learning activities of exploratory test design and execution. In 2022 IEEE Frontiers in Education Conference (FIE), pages 1–9.
Costa, I. E. F., Oliveira, S. R. B., Elgrably, I. S., Guerra, A. d. S., Soares, E. M., and Costa, I. V. F. (2023). Using active methodologies for teaching and learning of exploratory test design and execution. Education Sciences, 13(2).
CSDT (2023). Intrinsic Motivation Inventory (IMI). Center for Self-Determination Theory (CSDT).
de Oliveira, C. M. C., Canedo, E. D., Faria, H., Amaral, L. H. V., and Bonifácio, R. (2018). Improving student’s learning and cooperation skills using coding dojos (in the wild!). In 2018 IEEE Frontiers in Education Conference (FIE), pages 1–8.
Delamaro, M. E., Vincenzi, A. M. R., and Maldonado, J. C. (2006). A strategy to perform coverage testing of mobile applications. In Proceedings of the 2006 International Workshop on Automation of Software Test, pages 118–124. ACM.
Dybå, T., Dingsoyr, T., and Hanssen, G. K. (2007). Applying systematic reviews to diverse study types: An experience report. In First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007), pages 225–234.
Elgrably, I. S. and de Oliveira, S. R. B. (2021). A diagnosis on software testing education in the Brazilian universities. In 2021 IEEE Frontiers in Education Conference (FIE), pages 1–8. IEEE Press.
Elgrably, I. S. and Oliveira, S. R. B. (2018). Gamification and evaluation of the use the agile tests in software quality subjects: The application of case studies. In Proceedings of the 13th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE), pages 416–423. SciTePress.
Elgrably, I. S. and Oliveira, S. R. B. (2020). Model for teaching and training software testing in an agile context. In 2020 IEEE Frontiers in Education Conference (FIE), pages 1–9. IEEE Press.
Elgrably, I. S. and Oliveira, S. R. B. (2022a). Perception from the professors’ point of view in the remote teaching of software testing using active methodologies during the covid-19 pandemic. In 2022 IEEE Frontiers in Education Conference (FIE), pages 1–9.
Elgrably, I. S. and Oliveira, S. R. B. (2022b). A quasi-experimental evaluation of teaching software testing in software quality assurance subject during a post-graduate computer science course. International Journal of Emerging Technologies in Learning, 17(5):57–86.
Estácio, B., Kroll, J., and Prikladnicki, R. (2016a). Distributed coding dojo randori: An overview and research opportunities. In Workshop on Distributed Software Development, Software Ecosystems and Systems-of-Systems.
Estácio, B., Oliveira, R., Marczak, S., Kalinowski, M., Garcia, A., Prikladnicki, R., and Lucena, C. (2015a). Evaluating collaborative practices in acquiring programming skills: Findings of a controlled experiment. In 2015 29th Brazilian Symposium on Software Engineering, pages 150–159.
Estácio, B., Valentim, N., Rivero, L., Conte, T., and Prikladnicki, R. (2015b). Evaluating the use of pair programming and coding dojo in teaching mockups development: An empirical study. In 2015 48th Hawaii International Conference on System Sciences, pages 5084–5093.
Estácio, B., Zieris, F., Prechelt, L., and Prikladnicki, R. (2016b). On the randori training dynamics. In Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering, pages 44–47, New York, NY, USA. ACM.
Felizardo, K. R., Nakagawa, E. Y., Fabbri, S. C. P. F., and Ferrari, F. C. (2017). Revisão sistemática da literatura em engenharia de software: teoria e prática. Elsevier, Rio de Janeiro, 1 edition.
Filho, A. F. G. and de Toledo, R. (2015). Visual management and blind software developers. In 2015 Agile Conference, pages 31–39.
Fonseca, F. M., Silva, E. B. d., and Mendonça, D. S. (2019). Designing dojo: A collaborative method for teaching design patterns. In 2019 IEEE/ACM 12th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE), pages 39–40.
Furtado, L. S. and Oliveira, S. R. B. (2020). A teaching proposal for the software measurement process using gamification: an experimental study. In 2020 IEEE Frontiers in Education Conference (FIE), pages 1–8.
Gaertner, M. (2010). Testing dojos. Methods & Tools, 18(4):8–14.
Garcia, F. W. d. S. G., Oliveira, S. R. B., and Carvalho, E. d. C. (2022). Application of a teaching plan for algorithm subjects using active methodologies: An experimental report. International Journal of Emerging Technologies in Learning, 17(7):175–207.
Garcia, F. W. d. S. G., Oliveira, S. R. B., and Carvalho, E. d. C. (2023). A second experimental study the application of a teaching plan for the algorithms subject in an undergraduate course in computing using active methodologies. Informatics in Education, 22(2):233–255.
Garousi, V., Felderer, M., Kuhrmann, M., Herkiloğlu, K., and Eldh, S. (2020a). Exploring the industry’s challenges in software testing: An empirical study. Journal of Software Evolution and Process, 32(8).
Garousi, V., Rainer, A., Lauvås, P., and Arcuri, A. (2020b). Software-testing education: A systematic literature mapping. Journal of Systems and Software, 165.
Gehringer, E. F. (2007). Active and collaborative learning strategies for teaching computing. In Proceedings of the 2007 ASEE Annual Conference & Exposition, pages 12.167.1–12.167.13.
Gomes, R. F. and Lelli, V. (2021). GAMUT: GAMe-Based Learning Approach for Teaching Unit Testing. In XX Brazilian Symposium on Software Quality, pages 1–11, New York. ACM.
Gregory, J. and Crispin, L. (2014). More Agile Testing: Learning Journeys For The Whole Team. Addison-Wesley Professional, New Jersey, 1 edition.
Heinonen, K., Hirvikoski, K., Luukkainen, M., and Vihavainen, A. (2013). Learning agile software engineering practices using coding dojo. In Proceedings of the 14th Annual ACM SIGITE Conference on Information Technology Education, pages 97––102, New York, NY, USA. ACM.
Jesus, G. M., Paschoal, L. N., Ferrari, F. C., and Souza, S. R. S. (2019). Is it worth using gamification on software testing education? An experience report. In Proceedings of the XVIII Brazilian Symposium on Software Quality, pages 178–187, New York. ACM.
Kassab, M., DeFranco, J. F., and Laplante, P. A. (2017). Software testing: The state of the practice. IEEE Software, 34(5):46–52.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. Technical Report version 2.3, Keele University and University of Durham Joint Technical Report, UK. EBSE Technical Report EBSE-2007-01.
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4(1):1–12.
Lee, Y., Marepalli, D. B., and Yang, J. (2017). Teaching test-drive development using dojo. Journal of Computing Sciences in Colleges, 32(4):106––112.
Linkman, S., Vincenzi, A. M. R., and Maldonado, J. C. (2003). An evaluation of systematic functional testing using mutation testing. In Proceedings of 7th International Conference on Empirical Assessment in Software Engineering, pages 1–15.
Luz, R. B. d., Neto, A. G. S. S., and Noronha, R. V. (2013). Teaching TDD, the coding dojo style. In 2013 IEEE 13th International Conference on Advanced Learning Technologies, pages 371–375.
McAuley, E., Duncan, T., & Tammen, V. V. (1989). Psychometric properties of the intrinsic motivation inventory in a competitive sport setting: A confirmatory factor analysis. Research Quarterly for Exercise and Sport, 60(1), 48–58.
McConnell, J. J. (2005). Active and cooperative learning. SIGCSE Bulletin, 37(2), 27–30.
Meireles, M., de Souza, C., de Barros, F. C., Chaves, L., de Castro, R., & Giuntini, F. (2022a). The employment of testing dojo as a collaborative learning methodology for teaching failure analysis: An experience report. In Proceedings of the 4th International Conference on Computer Science and Technologies in Education (CSTE) (pp. 47–54).
Meireles, M. A. C., Filho, A. R. L., Lima, K. R. S., Ferraz, L. G. C., Batista, F. A., Barros, F. C. P., Chaves, L. C., Souza, C., & Roque, L. F. N. M. (2022b). Use of testing dojo as a methodology of collaborative learning in teaching testcase writing: An experience report. In Proceedings of the 13th International Conference on Society and Information Technologies (ICSIT 2022) (pp. 41–44).
Melo, S. M., Moreira, V. X. S., Paschoal, L. N., & Souza, S. R. S. (2020). Testing education: A survey on a global scale. In Proceedings of the 34th Brazilian Symposium on Software Engineering (pp. 554–563). New York: ACM.
Melo, S. M., Santos, I., Souza, P. S. L., & Souza, S. R. S. (2022). A survey on the practices of software testing: A look into Brazilian companies. Journal of Software Engineering Research and Development, 10(8), 11:1–11:15.
Myers, G. J., Badgett, T., & Sandler, C. (2012). The Art of Software Testing (3rd ed.). New Jersey: John Wiley & Sons, Inc.
Oliveira, R., Estácio, B., Garcia, A., Marczak, S., Prikladnicki, R., Kalinowski, M., & Lucena, C. (2016). Identifying code smells with collaborative practices: A controlled experiment. In Proceedings of the X Brazilian Symposium on Software Components, Architectures and Reuse (SBCARS) (pp. 61–70).
Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64, 1–18.
Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223–231.
Robson, C., & McCartan, K. (2016). Real World Research (4th ed.). John Wiley & Sons Ltd.
Rodrigues, P. L. R., Franz, L. P., Cheiran, J. F. P., da Silva, J. P. S., & Bordin, A. S. (2017). Coding dojo as a transforming practice in collaborative learning of programming: An experience report. In Proceedings of the XXXI Brazilian Symposium on Software Engineering (SBES) (pp. 348–357). New York: ACM.
Rooksby, J., Hunt, J., & Wang, X. (2014). The theory and practice of randori coding dojos. In G. Cantone & M. Marchesi (Eds.), Agile Processes in Software Engineering and Extreme Programming (pp. 251–259). Cham: Springer International Publishing.
Santos, E. A., & Oliveira, S. (2019). The use of game elements and scenarios for teaching and learning the function point analysis technique: An experimental study. In Proceedings of the 14th International Conference on Software Technologies (ICSOFT) (pp. 162–169). SciTePress.
Santos, V., Goldman, A., & de Souza, C. R. B. (2015). Fostering effective inter-team knowledge sharing in agile software development. Empirical Software Engineering, 20, 1006–1051.
Santos, V. A., Goldman, A., & Santos, C. D. (2012). Uncovering steady advances for an extreme programming course. CLEI Electronic Journal, 15.
Sato, D. T., Corbucci, H., & Bravo, M. V. (2008). Coding dojo: An environment for learning and sharing agile practices. In Agile 2008 Conference (pp. 459–464). IEEE Press.
Scatalon, L. P., Fioravanti, M. L., Prates, J. M., Garcia, R. E., & Barbosa, E. F. (2018). A survey on graduates’ curriculum-based knowledge gaps in software testing. In 2018 IEEE Frontiers in Education Conference (FIE) (pp. 1–8). IEEE Press.
Seaman, C. B. (2008). Qualitative Methods, Chapter 2, pages 35–62. Springer-Verlag, London.
Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4), 591–611.
Sherif, E., Liu, A., Nguyen, B., Lerner, S., & Griswold, W. G. (2020). Gamification to aid the learning of test coverage concepts. In 2020 IEEE 32nd Conference on Software Engineering Education and Training (pp. 1–5). IEEE Press.
Soomlek, C. (2015). Applying randori-style kata and agile practices to an undergraduate-level programming class. In XP 2015: 16th International Conference on Agile Processes in Software Engineering and Extreme Programming (pp. 369–370).
Valle, P. H. D., Barbosa, E. F., & Maldonado, J. C. (2015a). CS curricula of the most relevant universities in Brazil and abroad: Perspective of software testing education. In 2015 International Symposium on Computers in Education (SIIE) (pp. 62–68). IEEE Press.
Valle, P. H. D., Barbosa, E. F., & Maldonado, J. C. (2015b). Um mapeamento sistemático sobre ensino de teste de software. In Anais do XXVI Simpósio Brasileiro de Informática na Educação (SBIE 2015) (pp. 71–80).
Weyuker, E., & Jeng, B. (1991). Analyzing partition testing strategies. IEEE Transactions on Software Engineering, 17(7), 703–711.
Weyuker, E., & Ostrand, T. (1980). Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, SE6(3), 236–2461.
Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In EASE’14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (pp. 1–10). New York: ACM.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering. Berlin: Springer-Verlag, 1st edition.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Vladimir Belinski, Adolfo Gustavo Serra Seca Neto, Maria Claudia Emer
This work is licensed under a Creative Commons Attribution 4.0 International License.
