MEEGA+: Um Modelo para a Avaliação de Jogos Educacionais para o ensino de Computação
DOI:
https://doi.org/10.5753/rbie.2019.27.03.52Keywords:
Jogo Educacional, Avaliação, Modelo, Instrumento de mediçãoAbstract
Jogos educacionais estão sendo usados como uma estratégia instrucional inovadora para o ensino e aprendizagem em diversas áreas de conhecimento. No entanto, é essencial avaliar sistematicamente esses jogos a fim de obter evidências de suas contribuições. Atualmente, existem poucos modelos/métodos que forneçam um suporte sistemático para a avaliação de jogos educacionais. Neste contexto, um modelo que têm se destacado é o MEEGA (Model for the Evaluation of Educational GAmes). Porém, resultados de uma análise de sua versão inicial apontam limitações em termos de sua validade. Assim, o objetivo deste artigo é apresentar o modelo MEEGA+, evoluindo a versão inicial do modelo MEEGA. O modelo MEEGA+ é sistematicamente desenvolvido por meio da decomposição de um objetivo de avaliação em medidas e definindo um instrumento de medição para avaliar a qualidade de jogos educacionais. A análise da confiabilidade e validade, com base em dados coletados em 48 estudos de caso envolvendo uma população de 843 alunos, avaliando 18 diferentes jogos para o ensino de computação, indica que o instrumento de medição do modelo MEEGA+ é confiável (alfa de Cronbach α=,928). Os resultados de uma análise fatorial exploratória indicam que a qualidade de jogos educacionais é avaliada por meio de dois fatores de qualidade (experiência do jogador e usabilidade). Os resultados também demonstram validade convergente por meio de um grau de correlação satisfatório encontrado entre a maioria dos itens do instrumento de medição. Assim, o modelo MEEGA+ fornece um instrumento de medição confiável e válido para desenvolvedores de jogos, instrutores e pesquisadores, a fim de avaliar a qualidade de jogos educacionais como base para sua melhoria e adoção na prática.
Downloads
Referências
Abdellatif, A. J., McCollum, B., & McMullan, P. (2018). Serious Games: Quality Characteristics Evaluation Framework and Case Study. Proc. of IEEE Integrated STEM Education Conference, (pp. 112-119). Princeton, USA. doi: 10.1109/ISECon.2018.8340460.
Abt, C. C. (2002). Serious Games. Lanhan: University Press of America.
ACM/IEEE-CS. (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science, 2013. Disponível em: [Link]. Access: 06 Nov. 2018.
Ak, O. (2012). A Game Scale to Evaluate Educational Computer Games. Procedia – Social and Behavioral Sciences, 46, 2477-2481. doi: 10.1016/j.sbspro.2012.05.506.[GS Search]
All, A., Castellar, E. P. N., & Looy, J. V. (2016). Assessing the effectiveness of digital game-based learning: Best practices. Computers & Education, 92–93, 90-103. doi: 10.1016/j.compedu.2015.10.007. [GS Search]
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman. [GS Search]
Andrade, H. & Valtcheva, A. (2009). Promoting learning and achievement through self-assessment. Theory into Practice, 48, 12-19. doi: 10.1080/00405840802577544. [GS Search]
Backlund, P. & Hendrix, M. (2013). Educational games - Are they worth the effort? A literature survey of the effectiveness of serious games. Proc. of the 5th Int. Conf. on Games and Virtual Worlds for Serious Applications, Poole, GB. doi: 10.1109/VS-GAMES.2013.6624226. [GS Search]
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J. Marciniak, Encyclopedia of Software Engineering, (pp. 528-532). New York: Wiley-Interscience. [GS Search]
Battistella, P. & Gresse von Wangenheim, C. (2016). Games for teaching computing in higher education – a systematic review. IEEE Technology and Engineering Education Journal, 9(1), 8-30. [GS Search]
Battistella, P. E., Gresse von Wangenheim, C., von Wangenheim, A., & Martina, J. E. (2017). Design and Large-scale Evaluation of Educational Games for Teaching Sorting Algorithms. Informatics in Education, 17(2), 141-164. doi: 10.15388/infedu.2017.08. [GS Search]
Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. Journal of Systems and Software, 76(3), 251-275. doi: 10.1016/j.jss.2004.06.004. [GS Search]
Bittencourt, I. I. & Isotani, S. (2018). Informática na Educação baseada em Evidências: Um Manifesto. Revista Brasileira de Informática na Educação - RBIE, 26(3), 108-119. doi: 10.5753/RBIE.2018.26.03.108. [GS Search]
Bowman, D. D. (2018). Declining Talent in Computer Related Careers. Journal of Academic Administration in Higher Education, 14(1), 1-4. [GS Search]
Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, (94), 178-192. doi: 10.1016/j.compedu.2015.11.003. [GS Search]
Branch, R. M. (2010). Instructional Design: The ADDIE Approach. New York: Springer New York Dordrecht Heidelberg London. [GS Search]
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4-7. [GS Search]
Brown, G., Andrade, H., & Chen, F. (2015). Accuracy in student self-assessment: Directions and cautions for research. Assessment in Education Principles Policy and Practice, 22(4), 1-26. doi: 10.1080/0969594X.2014.996523. [GS Search]
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press. [GS Search]
Budgen, D., Turner, M., Brereton, P., & Kitchenham, B. (2008). Using mapping studies in Software Engineering. Evidence-based Software Engineering. Lancaster University, 195-204.[GS Search]
Calderón, A. & Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396-422. doi: 10.1016/j.compedu.2015.07.011 . [GS Search]
Calderón, A., Ruiz M., & O'Connor, R. (2018). A multivocal literature review on serious games for software process standards education. Computer Standards & Interfaces, 57, 36-48. doi: 10.1016/j.csi.2017.11.003 . [GS Search]
Caulfield, C., Xia, J., Veal, D., & Maj, S. P. (2011). A systematic survey of games used for software engineering education. Modern Applied Science, 5(6), 28-43. doi: 10.5539/mas.v5n6p00. [GS Search]
Carmines, E. G., & Zeller, R. A. (1982). Reliability and validity assessment (5th ed.). Beverly Hills: Sage Publications Inc. [GS Search]
Carvalho, C. V. (2012). Is game-based learning suitable for engineering education? Proc. of the Global Engineering Education Conf., (pp.1-8). Marrakech, Morocco. doi: 10.1109/EDUCON.2012.6201140. [GS Search]
Cattell, R. B. (1978). The Scientific Use of Factor Analysis in Behavioral and Life Sciences. Nova York: Plenum.
Cattell. R. B. (1966). The Scree Test for the Number of Factors. Multivariate Behavioral Research, 1(2), 245-276. [GS Search]
Chen, P. Y. & Popovich, P. M. (2002). Correlation: Parametric and nonparametric measures. Thousand Oaks: Sage Publications. [GS Search]
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York: Routledge Academic. [GS Search]
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi: 10.1016/j.compedu.2012.03.004 . [GS Search]
Connolly, T. M., Stansfield, M. H., & Hainey, T. (2009). Towards the development of a games-based learning evaluation framework. In T. M. Connolly, M. H. Stansfield, & E. Boyle (Eds.), Games-based learning advancement for multisensory human computer interfaces: Techniques and effective practices. Hershey: Idea-Group Publishing. [GS Search]
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. [GS Search]
Cuesta, M. (1996). Unidimensionalidade. In J. Muñiz (Ed.), Psicometría. Madrid: Editorial Universitas.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319-340. doi: 10.2307/249008. [GS Search]
Dawes, J. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50(1), 61-77. [GS Search]
Dempsey, J. V., Lucassen, B. A., Haynes, L. L., & Casey, M. S. (1996). Instructional applications of computer games. Paper presented at the American Education Research Association, NY. [GS Search]
DeVellis, R. F. (2016). Scale development: theory and applications (4th. ed.). Thousand Oaks: SAGE Publications. [GS Search]
Djaouti, D., Alvarez J., Jessel J. P., & Rampnoux O. (2011). Origins of Serious Games. In: Ma M., Oikonomou A., Jain L. (Eds). Serious Games and Edutainment Applications. London: Springer. [GS Search]
Dziuban, C. D. & Shirkey, E. C. (1974). When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychological Bulletin, 81, 358-361. doi: 10.1037/h0036316. [GS Search]
Freitas, S. D., & Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers & Education, 46(3), 249-264. doi: 10.1016/j.compedu.2005.11.007 . [GS Search]
Freire M., Serrano-Laguna Á., Iglesias B. M., Martínez-Ortiz I., Moreno-Ger P., & Fernández-Manjón B. (2016). Game Learning Analytics: Learning Analytics for Serious Games. In: Spector M., Lockee B., Childress M. (Eds) Learning, Design, and Technology. Cham: Springer. [GS Search]
Fu, F., Su, R., & Yu, S. (2009). EGameFlow: A scale to measure learners' enjoyment of e-learning games. Computers & Education, 52(1), 101-112. doi: 10.1016/j.compedu.2008.07.004 . [GS Search]
Gámez, E. H. (2009). On the Core Elements of the Experience of Playing Video Games (Dissertation). UCL Interaction Centre, Department of Computer Science, London.
Garson, D. (2013). Factor Analysis. Raleigh: Statistical Associates Publishing.
Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55(3), 377-393. doi: 10.1177/0013164495055003002. [GS Search]
Gomes, M. N. (2016). Desenvolvimento de uma Atividade Gamificada Voltada ao Turismo de Caxias Do Sul. Trabalho de Conclusão de Curso (Curso de Ciência da Computação), Universidade de Caxias do Sul. [GS Search]
Gresse von Wangenheim, C., Savi, R., & Borgatto, A. F. (2013). SCRUMIA - An educational game for teaching SCRUM in computing courses. Journal of Systems and Software, 86(10), 2675-2687. doi: 10.1016/j.jss.2013.05.030. [GS Search]
Herpich, F., Nunes, F. B., Voss, G. B., Sindeaux, P., Tarouco, L. M. R., & Lima, J. V. (2017). Realidade Aumentada em Geografia: uma atividade de orientação no ensino fundamental. Revista Novas Tecnologias na Educação, 15(2), 1–10. [GS Search]
International Standard Organization (ISO). (2014). ISO/IEC 25010: Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, Technical Report.
Johnson, R. B. & Christensen, L. (2014). Educational Research – Quantitative, Qualitative, and Mixed Approaches (5th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Kasunic, M. (2005). Designing an effective survey. Handbook CMU/SEI-2005-HB-004, Software Engineering Institute. Pittsburgh: Carnegie Mellon University. [GS Search]
Keller, J. (1987). Development and Use of the ARCS Model of motivational Design. Journal of Instructional Development, 10(3), 2-10. doi: 10.1007/BF02905780. [GS Search]
Kosa, M., Yilmaz, M., O'Connor, R., & Clarke, P. (2016). Software engineering education and games: a systematic literature review. Journal of Universal Computer Science, 22(12), 1558-1574. doi: 10.3217/jucs-022-12-1558. [GS Search]
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. Philadelphia: David McKay Company, Inc.
Malhotra, N. K., & Birks, D. F. (2008). Marketing Research: An Applied Approach (3rd ed.). Philadelphia: Trans-Atlantic Publications. [GS Search]
Mayer, I. (2012). Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games. Procedia Computer Science, 15, 233-247. doi: 10.1016/j.procs.2012.10.075. [GS Search]
Mohamed, H. & Jaafar, A. (2010). Development and Potential Analysis of Heuristic Evaluation for Educational Computer Game (PHEG). Proc. of the 5th Int. Conf. on Computer Sciences and Convergence Information Technology, (pp. 222-227). Seoul, South Korea. doi: 10.1109/ICCIT.2010.5711061. [GS Search]
Moosa, A. M., Al-Maadeed, N., & Alja'am, J. M. (2018). A Simple Health-Based Game for Children. Proc. of the International Conference on Computer and Applications. Beirut, Líbano, p. 309-312. doi: 10.1109/COMAPP.2018.8460213. [GS Search]
O’Brien, H. L. & Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User Engagement. Journal of the American Society for Information Science and Technology, 61(1), 50–69. doi: 10.1002/asi.21229. [GS Search]
Pereira, C. X., Hoffmann, Z., Castro, C. P., Santos, G. S., Aires, T. A., & Francisco, R. E. (2017). CHEMIS3: A game for learning chemical concepts through elements of nature. Proc. of the 12nd Iberian Conference on Information Systems and Technologies, Lisboa, Portugal, p. 1-5. doi: 10.23919/CISTI.2017.7975801. [GS Search]
Petri, G. & Chiavegatti, N. C. (2015). Um Role-Playing Game para o Ensino de Elicitação e Análise de Requisitos. Revista Novas Tecnologias na Educação, 13(1), 1-10. doi: 10.22456/1679-1916.57599. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2016). How to evaluate educational games: a systematic literature review. Journal of Universal Computers Science, 22(7), 992-1021. doi: 10.3217/jucs-022-07-0992. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, 107, 68-90. doi: 10.1016/j.compedu.2017.01.004. [GS Search]
Petri, G., Battistella, P., Cassettari, F., Gresse von Wangenheim, C., & Hauck, J. (2016). Um Quiz Game para a revisão de conhecimento em Gerenciamento de Projetos. Anais do 27º Simpósio Brasileiro de Informática na Educação, pp. 320-329. Uberlândia/MG, Brasil. doi: 10.5753/cbie.sbie.2016.320. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2017). A Large-scale Evaluation of a Model for the Evaluation of Games for Teaching Software Engineering. Proc. of the IEEE/ACM 39th Int. Conf. on Software Engineering: Software Engineering Education and Training Track, (pp. 180-189). Buenos Aires/Argentina. doi: 10.1109/ICSE-SEET.2017.11. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2018). MEEGA+, Systematic Model to Evaluate Educational Games. In Newton Lee (Eds) Encyclopedia of Computer Graphics and Games, (pp. 1-7). Cham: Springer. doi: 10.1007/978-3-319-08234-9. [GS Search]
Poels, K., Kort, Y. D., & Ijsselsteijn, W. (2007). It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. Proc. of Conf. on Future Play, (pp. 83-89). Toronto, Canada. doi: 10.1145/1328202.1328218. [GS Search]
Prensky, M. (2007). Digital Game-Based Learning. New York: Paragon House. [GS Search]
Raiche, G., Riopel, M., & Blais, J.-G. (2006). Nongraphical solutions for the Cattell’s scree test. Proc. of the Annual meeting of the Psychometric Society. Montreal, Canada. [GS Search]
Ritterfeld, U., Cody, M., & Vorderer, P. (Eds.). (2010). Serious Games. New York: Routledge.
Ross, J. A. (2006). The reliability, validity, and utility of self-assessment. Practical Assessment, Research & Evaluation, 11(10), 1-13. [GS Search]
Savi, R., Gresse von Wangenheim, C., & Borgatto, A. F. (2011). A model for the evaluation of educational games for teaching software engineering. Anais do XXV Simpósio Brasileiro de Engenharia de Software, pp. 194-203. São Paulo, Brasil. doi: 10.1109/SBES.2011.27 [GS Search]
Schanzenbach, D. W. (2012). Limitations of Experiments in Education Research. Education Finance and Policy, 7(2), 219-232. doi: 10.1162/EDFP_a_00063. [GS Search]
Sharma, R., Jain, A., Gupta, N., Garg, S., Batta, M., & Dhir, S. K. (2016). Impact of self-assessment by students on their learning. International Journal of Applied and Basic Medical Research, 6(3), 226–229. doi: 10.4103/2229-516X.186961. [GS Search]
Serrano-Laguna, A., Manero, B., Freire, M., & Fernández-Manjón, B. (2018). A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes. Multimedia Tools and Applications, 77(2), 2849-2871. [GS Search]
Silva, J. P., Carvalho, F. C., Silvestre, A., Luiz, M. F., Alencar, R. L., & Silveira, I. F. (2017) Forca dos Sinônimos: Um Jogo da Forca Multiplayer para o ensino de Tesauros em Língua Portuguesa como Ferramenta de busca na Web. Anais do Simpósio Brasileiro de Jogos e Entretenimento Digital, Curitiba/PR, Brasil, p. 1128-1131.[GS Search]
Silva, J. P., Jorge, A. A., Gaia, C. C., & Costa, G. C. (2017). Quais as chances? Um jogo de dados e cartas para o ensino do cálculo de probabilidades. Anais do 8º Congresso de Inovação, Ciência e Tecnologia do IFSP, Cubatão/SP, Brasil, p. 1-3. [GS Search]
Simpson, E. J. (1972). The classification of educational objectives, psychomotor domain. Washington: Gryphon House. [GS Search]
Sindre, G. & Moody, D. (2003). Evaluating the Effectiveness of Learning Interventions: an Information Systems Case Study. Proc. of the 11th European Conf. on Information Systems, Paper 80. Naples, Italy. [GS Search]
Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure? Academy of Management Learning & Education, 9(2), 169-191. [GS Search]
Sweetser, P. & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), 1-24. [GS Search]
Tabachnick, B. G. & Fidell, L. S. (2007). Using Multivariate Statistics (5th ed.). Boston: Allyn and Bacon. [GS Search]
Takatalo, J., Häkkinen, J., Kaistinen, J., & Nyman, G. (2010). Presence, Involvement, and Flow in Digital Games. In: Bernhaupt, R. (Ed.). Evaluating User Experience in Games: Concepts and Methods, (pp. 23-46). London: Springer.[GS Search]
Thomas, G., Martin, D., & Pleasants, K. (2011). Using self- and peer-assessment to enhance students’ future-learning in higher education. Journal of University Teaching & Learning Practice, 8(1), 1-17. [GS Search]
Tonussi, L. P. & Hauck, J. C. R. (2017). Um Módulo de Jogo de Perguntas e Respostas para apoio ao Ensino de Gerência de Projetos Integrado ao Moodle. Proc. of the Computer on the Beach. Florianópolis, Brazil (in Portuguese). [GS Search]
Trochim, W. M., & Donnelly, J. P. (2008). Research methods knowledge base (3rd ed.). Mason: Atomic Dog Publishing. [GS Search]
Tullis, T. & Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington: Morgan Kaufmann. [GS Search]
Wiebe, E. N., Lamb, A., Hardy, M., & Sharek, D. (2014). Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, 32, 123-132. doi: 10.1016/j.chb.2013.12.001 . [GS Search]
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering. New York: Springer-Verlag Berlin Heidelberg. [GS Search]
Yin, R. K. (2017). Case study research: design and methods (6th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Zaibon, S. B. & Shiratuddin, N. (2010). Heuristics Evaluation Strategy for Mobile Game-Based Learning. Proc of the 6th IEEE Int. Conf. on Wireless, Mobile and Ubiquitous Technologies in Education, (pp. 127-131). Kaohsiung, Taiwan. doi: 10.1109/WMUTE.2010.27. [GS Search]
Zaibon, S. B. (2015). User Testing on Game Usability, Mobility, Playability, and Learning Content of Mobile Game-Based Learning. Journal Teknologi, 77(29), 131-139. [GS Search]
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2019 Giani Petri, Christiane Gresse von Wangenheim, Adriano Ferret Borgatto
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
