A Model for the Evaluation of Educational Games for Computing Education
DOI:
https://doi.org/10.5753/rbie.2019.27.03.52Keywords:
Educational game, Evaluation, Model, Measurement instrumentAbstract
Educational games are being used as an innovative instructional strategy for teaching and learning in different knowledge areas. However, it is essential to systematically evaluate such games in order to obtain sound evidence on their contributions. Currently, there are a few models/methods that provide a systematic support for the evaluation of educational games. In this context, a prominent evaluation model is MEEGA (Model for the Evaluation of Educational Games). However, analyses of its initial version have identified limitations in terms of its validity. Thus, the objective of this article is to present its evolution, the MEEGA+ model. The MEEGA+ model has been systematically developed by decomposing evaluation goals into measures and defining a measurement instrument to evaluate the quality of educational games. The analysis of the reliability and validity based on data collected from 48 case studies involving a population of 843 students, evaluating 18 different games for computing education, indicates that the MEEGA+ measurement instrument can be considered reliable (Cronbach’s alpha α=.928). Results of an exploratory factor analysis indicate that the quality of educational games is evaluated through two quality factors (player experience and usability). In addition, the results demonstrate convergent validity through a satisfactory degree of correlation found between most of the items for all quality factors. Thus, the MEEGA+ model provides a reliable and valid measurement instrument for game creators, instructors and researchers in order to evaluate the quality of educational games as a basis for their improvement and adoption in practice.
Downloads
References
Abdellatif, A. J., McCollum, B., & McMullan, P. (2018). Serious Games: Quality Characteristics Evaluation Framework and Case Study. Proc. of IEEE Integrated STEM Education Conference, (pp. 112-119). Princeton, USA. doi: 10.1109/ISECon.2018.8340460.
Abt, C. C. (2002). Serious Games. Lanhan: University Press of America.
ACM/IEEE-CS. (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science, 2013. Disponível em: [Link]. Access: 06 Nov. 2018.
Ak, O. (2012). A Game Scale to Evaluate Educational Computer Games. Procedia – Social and Behavioral Sciences, 46, 2477-2481. doi: 10.1016/j.sbspro.2012.05.506.[GS Search]
All, A., Castellar, E. P. N., & Looy, J. V. (2016). Assessing the effectiveness of digital game-based learning: Best practices. Computers & Education, 92–93, 90-103. doi: 10.1016/j.compedu.2015.10.007. [GS Search]
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman. [GS Search]
Andrade, H. & Valtcheva, A. (2009). Promoting learning and achievement through self-assessment. Theory into Practice, 48, 12-19. doi: 10.1080/00405840802577544. [GS Search]
Backlund, P. & Hendrix, M. (2013). Educational games - Are they worth the effort? A literature survey of the effectiveness of serious games. Proc. of the 5th Int. Conf. on Games and Virtual Worlds for Serious Applications, Poole, GB. doi: 10.1109/VS-GAMES.2013.6624226. [GS Search]
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J. Marciniak, Encyclopedia of Software Engineering, (pp. 528-532). New York: Wiley-Interscience. [GS Search]
Battistella, P. & Gresse von Wangenheim, C. (2016). Games for teaching computing in higher education – a systematic review. IEEE Technology and Engineering Education Journal, 9(1), 8-30. [GS Search]
Battistella, P. E., Gresse von Wangenheim, C., von Wangenheim, A., & Martina, J. E. (2017). Design and Large-scale Evaluation of Educational Games for Teaching Sorting Algorithms. Informatics in Education, 17(2), 141-164. doi: 10.15388/infedu.2017.08. [GS Search]
Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. Journal of Systems and Software, 76(3), 251-275. doi: 10.1016/j.jss.2004.06.004. [GS Search]
Bittencourt, I. I. & Isotani, S. (2018). Informática na Educação baseada em Evidências: Um Manifesto. Revista Brasileira de Informática na Educação - RBIE, 26(3), 108-119. doi: 10.5753/RBIE.2018.26.03.108. [GS Search]
Bowman, D. D. (2018). Declining Talent in Computer Related Careers. Journal of Academic Administration in Higher Education, 14(1), 1-4. [GS Search]
Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, (94), 178-192. doi: 10.1016/j.compedu.2015.11.003. [GS Search]
Branch, R. M. (2010). Instructional Design: The ADDIE Approach. New York: Springer New York Dordrecht Heidelberg London. [GS Search]
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4-7. [GS Search]
Brown, G., Andrade, H., & Chen, F. (2015). Accuracy in student self-assessment: Directions and cautions for research. Assessment in Education Principles Policy and Practice, 22(4), 1-26. doi: 10.1080/0969594X.2014.996523. [GS Search]
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press. [GS Search]
Budgen, D., Turner, M., Brereton, P., & Kitchenham, B. (2008). Using mapping studies in Software Engineering. Evidence-based Software Engineering. Lancaster University, 195-204.[GS Search]
Calderón, A. & Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396-422. doi: 10.1016/j.compedu.2015.07.011 . [GS Search]
Calderón, A., Ruiz M., & O'Connor, R. (2018). A multivocal literature review on serious games for software process standards education. Computer Standards & Interfaces, 57, 36-48. doi: 10.1016/j.csi.2017.11.003 . [GS Search]
Caulfield, C., Xia, J., Veal, D., & Maj, S. P. (2011). A systematic survey of games used for software engineering education. Modern Applied Science, 5(6), 28-43. doi: 10.5539/mas.v5n6p00. [GS Search]
Carmines, E. G., & Zeller, R. A. (1982). Reliability and validity assessment (5th ed.). Beverly Hills: Sage Publications Inc. [GS Search]
Carvalho, C. V. (2012). Is game-based learning suitable for engineering education? Proc. of the Global Engineering Education Conf., (pp.1-8). Marrakech, Morocco. doi: 10.1109/EDUCON.2012.6201140. [GS Search]
Cattell, R. B. (1978). The Scientific Use of Factor Analysis in Behavioral and Life Sciences. Nova York: Plenum.
Cattell. R. B. (1966). The Scree Test for the Number of Factors. Multivariate Behavioral Research, 1(2), 245-276. [GS Search]
Chen, P. Y. & Popovich, P. M. (2002). Correlation: Parametric and nonparametric measures. Thousand Oaks: Sage Publications. [GS Search]
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York: Routledge Academic. [GS Search]
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi: 10.1016/j.compedu.2012.03.004 . [GS Search]
Connolly, T. M., Stansfield, M. H., & Hainey, T. (2009). Towards the development of a games-based learning evaluation framework. In T. M. Connolly, M. H. Stansfield, & E. Boyle (Eds.), Games-based learning advancement for multisensory human computer interfaces: Techniques and effective practices. Hershey: Idea-Group Publishing. [GS Search]
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. [GS Search]
Cuesta, M. (1996). Unidimensionalidade. In J. Muñiz (Ed.), Psicometría. Madrid: Editorial Universitas.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319-340. doi: 10.2307/249008. [GS Search]
Dawes, J. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50(1), 61-77. [GS Search]
Dempsey, J. V., Lucassen, B. A., Haynes, L. L., & Casey, M. S. (1996). Instructional applications of computer games. Paper presented at the American Education Research Association, NY. [GS Search]
DeVellis, R. F. (2016). Scale development: theory and applications (4th. ed.). Thousand Oaks: SAGE Publications. [GS Search]
Djaouti, D., Alvarez J., Jessel J. P., & Rampnoux O. (2011). Origins of Serious Games. In: Ma M., Oikonomou A., Jain L. (Eds). Serious Games and Edutainment Applications. London: Springer. [GS Search]
Dziuban, C. D. & Shirkey, E. C. (1974). When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychological Bulletin, 81, 358-361. doi: 10.1037/h0036316. [GS Search]
Freitas, S. D., & Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers & Education, 46(3), 249-264. doi: 10.1016/j.compedu.2005.11.007 . [GS Search]
Freire M., Serrano-Laguna Á., Iglesias B. M., Martínez-Ortiz I., Moreno-Ger P., & Fernández-Manjón B. (2016). Game Learning Analytics: Learning Analytics for Serious Games. In: Spector M., Lockee B., Childress M. (Eds) Learning, Design, and Technology. Cham: Springer. [GS Search]
Fu, F., Su, R., & Yu, S. (2009). EGameFlow: A scale to measure learners' enjoyment of e-learning games. Computers & Education, 52(1), 101-112. doi: 10.1016/j.compedu.2008.07.004 . [GS Search]
Gámez, E. H. (2009). On the Core Elements of the Experience of Playing Video Games (Dissertation). UCL Interaction Centre, Department of Computer Science, London.
Garson, D. (2013). Factor Analysis. Raleigh: Statistical Associates Publishing.
Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55(3), 377-393. doi: 10.1177/0013164495055003002. [GS Search]
Gomes, M. N. (2016). Desenvolvimento de uma Atividade Gamificada Voltada ao Turismo de Caxias Do Sul. Trabalho de Conclusão de Curso (Curso de Ciência da Computação), Universidade de Caxias do Sul. [GS Search]
Gresse von Wangenheim, C., Savi, R., & Borgatto, A. F. (2013). SCRUMIA - An educational game for teaching SCRUM in computing courses. Journal of Systems and Software, 86(10), 2675-2687. doi: 10.1016/j.jss.2013.05.030. [GS Search]
Herpich, F., Nunes, F. B., Voss, G. B., Sindeaux, P., Tarouco, L. M. R., & Lima, J. V. (2017). Realidade Aumentada em Geografia: uma atividade de orientação no ensino fundamental. Revista Novas Tecnologias na Educação, 15(2), 1–10. [GS Search]
International Standard Organization (ISO). (2014). ISO/IEC 25010: Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, Technical Report.
Johnson, R. B. & Christensen, L. (2014). Educational Research – Quantitative, Qualitative, and Mixed Approaches (5th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Kasunic, M. (2005). Designing an effective survey. Handbook CMU/SEI-2005-HB-004, Software Engineering Institute. Pittsburgh: Carnegie Mellon University. [GS Search]
Keller, J. (1987). Development and Use of the ARCS Model of motivational Design. Journal of Instructional Development, 10(3), 2-10. doi: 10.1007/BF02905780. [GS Search]
Kosa, M., Yilmaz, M., O'Connor, R., & Clarke, P. (2016). Software engineering education and games: a systematic literature review. Journal of Universal Computer Science, 22(12), 1558-1574. doi: 10.3217/jucs-022-12-1558. [GS Search]
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. Philadelphia: David McKay Company, Inc.
Malhotra, N. K., & Birks, D. F. (2008). Marketing Research: An Applied Approach (3rd ed.). Philadelphia: Trans-Atlantic Publications. [GS Search]
Mayer, I. (2012). Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games. Procedia Computer Science, 15, 233-247. doi: 10.1016/j.procs.2012.10.075. [GS Search]
Mohamed, H. & Jaafar, A. (2010). Development and Potential Analysis of Heuristic Evaluation for Educational Computer Game (PHEG). Proc. of the 5th Int. Conf. on Computer Sciences and Convergence Information Technology, (pp. 222-227). Seoul, South Korea. doi: 10.1109/ICCIT.2010.5711061. [GS Search]
Moosa, A. M., Al-Maadeed, N., & Alja'am, J. M. (2018). A Simple Health-Based Game for Children. Proc. of the International Conference on Computer and Applications. Beirut, Líbano, p. 309-312. doi: 10.1109/COMAPP.2018.8460213. [GS Search]
O’Brien, H. L. & Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User Engagement. Journal of the American Society for Information Science and Technology, 61(1), 50–69. doi: 10.1002/asi.21229. [GS Search]
Pereira, C. X., Hoffmann, Z., Castro, C. P., Santos, G. S., Aires, T. A., & Francisco, R. E. (2017). CHEMIS3: A game for learning chemical concepts through elements of nature. Proc. of the 12nd Iberian Conference on Information Systems and Technologies, Lisboa, Portugal, p. 1-5. doi: 10.23919/CISTI.2017.7975801. [GS Search]
Petri, G. & Chiavegatti, N. C. (2015). Um Role-Playing Game para o Ensino de Elicitação e Análise de Requisitos. Revista Novas Tecnologias na Educação, 13(1), 1-10. doi: 10.22456/1679-1916.57599. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2016). How to evaluate educational games: a systematic literature review. Journal of Universal Computers Science, 22(7), 992-1021. doi: 10.3217/jucs-022-07-0992. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, 107, 68-90. doi: 10.1016/j.compedu.2017.01.004. [GS Search]
Petri, G., Battistella, P., Cassettari, F., Gresse von Wangenheim, C., & Hauck, J. (2016). Um Quiz Game para a revisão de conhecimento em Gerenciamento de Projetos. Anais do 27º Simpósio Brasileiro de Informática na Educação, pp. 320-329. Uberlândia/MG, Brasil. doi: 10.5753/cbie.sbie.2016.320. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2017). A Large-scale Evaluation of a Model for the Evaluation of Games for Teaching Software Engineering. Proc. of the IEEE/ACM 39th Int. Conf. on Software Engineering: Software Engineering Education and Training Track, (pp. 180-189). Buenos Aires/Argentina. doi: 10.1109/ICSE-SEET.2017.11. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2018). MEEGA+, Systematic Model to Evaluate Educational Games. In Newton Lee (Eds) Encyclopedia of Computer Graphics and Games, (pp. 1-7). Cham: Springer. doi: 10.1007/978-3-319-08234-9. [GS Search]
Poels, K., Kort, Y. D., & Ijsselsteijn, W. (2007). It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. Proc. of Conf. on Future Play, (pp. 83-89). Toronto, Canada. doi: 10.1145/1328202.1328218. [GS Search]
Prensky, M. (2007). Digital Game-Based Learning. New York: Paragon House. [GS Search]
Raiche, G., Riopel, M., & Blais, J.-G. (2006). Nongraphical solutions for the Cattell’s scree test. Proc. of the Annual meeting of the Psychometric Society. Montreal, Canada. [GS Search]
Ritterfeld, U., Cody, M., & Vorderer, P. (Eds.). (2010). Serious Games. New York: Routledge.
Ross, J. A. (2006). The reliability, validity, and utility of self-assessment. Practical Assessment, Research & Evaluation, 11(10), 1-13. [GS Search]
Savi, R., Gresse von Wangenheim, C., & Borgatto, A. F. (2011). A model for the evaluation of educational games for teaching software engineering. Anais do XXV Simpósio Brasileiro de Engenharia de Software, pp. 194-203. São Paulo, Brasil. doi: 10.1109/SBES.2011.27 [GS Search]
Schanzenbach, D. W. (2012). Limitations of Experiments in Education Research. Education Finance and Policy, 7(2), 219-232. doi: 10.1162/EDFP_a_00063. [GS Search]
Sharma, R., Jain, A., Gupta, N., Garg, S., Batta, M., & Dhir, S. K. (2016). Impact of self-assessment by students on their learning. International Journal of Applied and Basic Medical Research, 6(3), 226–229. doi: 10.4103/2229-516X.186961. [GS Search]
Serrano-Laguna, A., Manero, B., Freire, M., & Fernández-Manjón, B. (2018). A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes. Multimedia Tools and Applications, 77(2), 2849-2871. [GS Search]
Silva, J. P., Carvalho, F. C., Silvestre, A., Luiz, M. F., Alencar, R. L., & Silveira, I. F. (2017) Forca dos Sinônimos: Um Jogo da Forca Multiplayer para o ensino de Tesauros em Língua Portuguesa como Ferramenta de busca na Web. Anais do Simpósio Brasileiro de Jogos e Entretenimento Digital, Curitiba/PR, Brasil, p. 1128-1131.[GS Search]
Silva, J. P., Jorge, A. A., Gaia, C. C., & Costa, G. C. (2017). Quais as chances? Um jogo de dados e cartas para o ensino do cálculo de probabilidades. Anais do 8º Congresso de Inovação, Ciência e Tecnologia do IFSP, Cubatão/SP, Brasil, p. 1-3. [GS Search]
Simpson, E. J. (1972). The classification of educational objectives, psychomotor domain. Washington: Gryphon House. [GS Search]
Sindre, G. & Moody, D. (2003). Evaluating the Effectiveness of Learning Interventions: an Information Systems Case Study. Proc. of the 11th European Conf. on Information Systems, Paper 80. Naples, Italy. [GS Search]
Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure? Academy of Management Learning & Education, 9(2), 169-191. [GS Search]
Sweetser, P. & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), 1-24. [GS Search]
Tabachnick, B. G. & Fidell, L. S. (2007). Using Multivariate Statistics (5th ed.). Boston: Allyn and Bacon. [GS Search]
Takatalo, J., Häkkinen, J., Kaistinen, J., & Nyman, G. (2010). Presence, Involvement, and Flow in Digital Games. In: Bernhaupt, R. (Ed.). Evaluating User Experience in Games: Concepts and Methods, (pp. 23-46). London: Springer.[GS Search]
Thomas, G., Martin, D., & Pleasants, K. (2011). Using self- and peer-assessment to enhance students’ future-learning in higher education. Journal of University Teaching & Learning Practice, 8(1), 1-17. [GS Search]
Tonussi, L. P. & Hauck, J. C. R. (2017). Um Módulo de Jogo de Perguntas e Respostas para apoio ao Ensino de Gerência de Projetos Integrado ao Moodle. Proc. of the Computer on the Beach. Florianópolis, Brazil (in Portuguese). [GS Search]
Trochim, W. M., & Donnelly, J. P. (2008). Research methods knowledge base (3rd ed.). Mason: Atomic Dog Publishing. [GS Search]
Tullis, T. & Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington: Morgan Kaufmann. [GS Search]
Wiebe, E. N., Lamb, A., Hardy, M., & Sharek, D. (2014). Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, 32, 123-132. doi: 10.1016/j.chb.2013.12.001 . [GS Search]
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering. New York: Springer-Verlag Berlin Heidelberg. [GS Search]
Yin, R. K. (2017). Case study research: design and methods (6th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Zaibon, S. B. & Shiratuddin, N. (2010). Heuristics Evaluation Strategy for Mobile Game-Based Learning. Proc of the 6th IEEE Int. Conf. on Wireless, Mobile and Ubiquitous Technologies in Education, (pp. 127-131). Kaohsiung, Taiwan. doi: 10.1109/WMUTE.2010.27. [GS Search]
Zaibon, S. B. (2015). User Testing on Game Usability, Mobility, Playability, and Learning Content of Mobile Game-Based Learning. Journal Teknologi, 77(29), 131-139. [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 Giani Petri, Christiane Gresse von Wangenheim, Adriano Ferret Borgatto
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
