MEEGA+: Um Modelo para a Avaliação de Jogos Educacionais para o ensino de Computação
DOI:
https://doi.org/10.5753/rbie.2019.27.03.52Keywords:
Jogo Educacional, Avaliação, Modelo, Instrumento de mediçãoAbstract
Jogos educacionais estão sendo usados como uma estratégia instrucional inovadora para o ensino e aprendizagem em diversas áreas de conhecimento. No entanto, é essencial avaliar sistematicamente esses jogos a fim de obter evidências de suas contribuições. Atualmente, existem poucos modelos/métodos que forneçam um suporte sistemático para a avaliação de jogos educacionais. Neste contexto, um modelo que têm se destacado é o MEEGA (Model for the Evaluation of Educational GAmes). Porém, resultados de uma análise de sua versão inicial apontam limitações em termos de sua validade. Assim, o objetivo deste artigo é apresentar o modelo MEEGA+, evoluindo a versão inicial do modelo MEEGA. O modelo MEEGA+ é sistematicamente desenvolvido por meio da decomposição de um objetivo de avaliação em medidas e definindo um instrumento de medição para avaliar a qualidade de jogos educacionais. A análise da confiabilidade e validade, com base em dados coletados em 48 estudos de caso envolvendo uma população de 843 alunos, avaliando 18 diferentes jogos para o ensino de computação, indica que o instrumento de medição do modelo MEEGA+ é confiável (alfa de Cronbach α=,928). Os resultados de uma análise fatorial exploratória indicam que a qualidade de jogos educacionais é avaliada por meio de dois fatores de qualidade (experiência do jogador e usabilidade). Os resultados também demonstram validade convergente por meio de um grau de correlação satisfatório encontrado entre a maioria dos itens do instrumento de medição. Assim, o modelo MEEGA+ fornece um instrumento de medição confiável e válido para desenvolvedores de jogos, instrutores e pesquisadores, a fim de avaliar a qualidade de jogos educacionais como base para sua melhoria e adoção na prática.
Descargas
Citas
Abdellatif, A. J., McCollum, B., & McMullan, P. (2018). Serious Games: Quality Characteristics Evaluation Framework and Case Study. Proc. of IEEE Integrated STEM Education Conference, (pp. 112-119). Princeton, USA. doi: 10.1109/ISECon.2018.8340460.
Abt, C. C. (2002). Serious Games. Lanhan: University Press of America.
ACM/IEEE-CS. (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Science, 2013. Disponível em: [Link]. Access: 06 Nov. 2018.
Ak, O. (2012). A Game Scale to Evaluate Educational Computer Games. Procedia – Social and Behavioral Sciences, 46, 2477-2481. doi: 10.1016/j.sbspro.2012.05.506.[GS Search]
All, A., Castellar, E. P. N., & Looy, J. V. (2016). Assessing the effectiveness of digital game-based learning: Best practices. Computers & Education, 92–93, 90-103. doi: 10.1016/j.compedu.2015.10.007. [GS Search]
Anderson, L. W., Krathwohl, D. R., & Bloom, B. S. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives. Longman. [GS Search]
Andrade, H. & Valtcheva, A. (2009). Promoting learning and achievement through self-assessment. Theory into Practice, 48, 12-19. doi: 10.1080/00405840802577544. [GS Search]
Backlund, P. & Hendrix, M. (2013). Educational games - Are they worth the effort? A literature survey of the effectiveness of serious games. Proc. of the 5th Int. Conf. on Games and Virtual Worlds for Serious Applications, Poole, GB. doi: 10.1109/VS-GAMES.2013.6624226. [GS Search]
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). Goal, Question Metric Paradigm. In J. J. Marciniak, Encyclopedia of Software Engineering, (pp. 528-532). New York: Wiley-Interscience. [GS Search]
Battistella, P. & Gresse von Wangenheim, C. (2016). Games for teaching computing in higher education – a systematic review. IEEE Technology and Engineering Education Journal, 9(1), 8-30. [GS Search]
Battistella, P. E., Gresse von Wangenheim, C., von Wangenheim, A., & Martina, J. E. (2017). Design and Large-scale Evaluation of Educational Games for Teaching Sorting Algorithms. Informatics in Education, 17(2), 141-164. doi: 10.15388/infedu.2017.08. [GS Search]
Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. Journal of Systems and Software, 76(3), 251-275. doi: 10.1016/j.jss.2004.06.004. [GS Search]
Bittencourt, I. I. & Isotani, S. (2018). Informática na Educação baseada em Evidências: Um Manifesto. Revista Brasileira de Informática na Educação - RBIE, 26(3), 108-119. doi: 10.5753/RBIE.2018.26.03.108. [GS Search]
Bowman, D. D. (2018). Declining Talent in Computer Related Careers. Journal of Academic Administration in Higher Education, 14(1), 1-4. [GS Search]
Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, (94), 178-192. doi: 10.1016/j.compedu.2015.11.003. [GS Search]
Branch, R. M. (2010). Instructional Design: The ADDIE Approach. New York: Springer New York Dordrecht Heidelberg London. [GS Search]
Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4-7. [GS Search]
Brown, G., Andrade, H., & Chen, F. (2015). Accuracy in student self-assessment: Directions and cautions for research. Assessment in Education Principles Policy and Practice, 22(4), 1-26. doi: 10.1080/0969594X.2014.996523. [GS Search]
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press. [GS Search]
Budgen, D., Turner, M., Brereton, P., & Kitchenham, B. (2008). Using mapping studies in Software Engineering. Evidence-based Software Engineering. Lancaster University, 195-204.[GS Search]
Calderón, A. & Ruiz M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396-422. doi: 10.1016/j.compedu.2015.07.011 . [GS Search]
Calderón, A., Ruiz M., & O'Connor, R. (2018). A multivocal literature review on serious games for software process standards education. Computer Standards & Interfaces, 57, 36-48. doi: 10.1016/j.csi.2017.11.003 . [GS Search]
Caulfield, C., Xia, J., Veal, D., & Maj, S. P. (2011). A systematic survey of games used for software engineering education. Modern Applied Science, 5(6), 28-43. doi: 10.5539/mas.v5n6p00. [GS Search]
Carmines, E. G., & Zeller, R. A. (1982). Reliability and validity assessment (5th ed.). Beverly Hills: Sage Publications Inc. [GS Search]
Carvalho, C. V. (2012). Is game-based learning suitable for engineering education? Proc. of the Global Engineering Education Conf., (pp.1-8). Marrakech, Morocco. doi: 10.1109/EDUCON.2012.6201140. [GS Search]
Cattell, R. B. (1978). The Scientific Use of Factor Analysis in Behavioral and Life Sciences. Nova York: Plenum.
Cattell. R. B. (1966). The Scree Test for the Number of Factors. Multivariate Behavioral Research, 1(2), 245-276. [GS Search]
Chen, P. Y. & Popovich, P. M. (2002). Correlation: Parametric and nonparametric measures. Thousand Oaks: Sage Publications. [GS Search]
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New York: Routledge Academic. [GS Search]
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi: 10.1016/j.compedu.2012.03.004 . [GS Search]
Connolly, T. M., Stansfield, M. H., & Hainey, T. (2009). Towards the development of a games-based learning evaluation framework. In T. M. Connolly, M. H. Stansfield, & E. Boyle (Eds.), Games-based learning advancement for multisensory human computer interfaces: Techniques and effective practices. Hershey: Idea-Group Publishing. [GS Search]
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. [GS Search]
Cuesta, M. (1996). Unidimensionalidade. In J. Muñiz (Ed.), Psicometría. Madrid: Editorial Universitas.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319-340. doi: 10.2307/249008. [GS Search]
Dawes, J. (2008). Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 50(1), 61-77. [GS Search]
Dempsey, J. V., Lucassen, B. A., Haynes, L. L., & Casey, M. S. (1996). Instructional applications of computer games. Paper presented at the American Education Research Association, NY. [GS Search]
DeVellis, R. F. (2016). Scale development: theory and applications (4th. ed.). Thousand Oaks: SAGE Publications. [GS Search]
Djaouti, D., Alvarez J., Jessel J. P., & Rampnoux O. (2011). Origins of Serious Games. In: Ma M., Oikonomou A., Jain L. (Eds). Serious Games and Edutainment Applications. London: Springer. [GS Search]
Dziuban, C. D. & Shirkey, E. C. (1974). When is a correlation matrix appropriate for factor analysis? Some decision rules. Psychological Bulletin, 81, 358-361. doi: 10.1037/h0036316. [GS Search]
Freitas, S. D., & Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers & Education, 46(3), 249-264. doi: 10.1016/j.compedu.2005.11.007 . [GS Search]
Freire M., Serrano-Laguna Á., Iglesias B. M., Martínez-Ortiz I., Moreno-Ger P., & Fernández-Manjón B. (2016). Game Learning Analytics: Learning Analytics for Serious Games. In: Spector M., Lockee B., Childress M. (Eds) Learning, Design, and Technology. Cham: Springer. [GS Search]
Fu, F., Su, R., & Yu, S. (2009). EGameFlow: A scale to measure learners' enjoyment of e-learning games. Computers & Education, 52(1), 101-112. doi: 10.1016/j.compedu.2008.07.004 . [GS Search]
Gámez, E. H. (2009). On the Core Elements of the Experience of Playing Video Games (Dissertation). UCL Interaction Centre, Department of Computer Science, London.
Garson, D. (2013). Factor Analysis. Raleigh: Statistical Associates Publishing.
Glorfeld, L. W. (1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55(3), 377-393. doi: 10.1177/0013164495055003002. [GS Search]
Gomes, M. N. (2016). Desenvolvimento de uma Atividade Gamificada Voltada ao Turismo de Caxias Do Sul. Trabalho de Conclusão de Curso (Curso de Ciência da Computação), Universidade de Caxias do Sul. [GS Search]
Gresse von Wangenheim, C., Savi, R., & Borgatto, A. F. (2013). SCRUMIA - An educational game for teaching SCRUM in computing courses. Journal of Systems and Software, 86(10), 2675-2687. doi: 10.1016/j.jss.2013.05.030. [GS Search]
Herpich, F., Nunes, F. B., Voss, G. B., Sindeaux, P., Tarouco, L. M. R., & Lima, J. V. (2017). Realidade Aumentada em Geografia: uma atividade de orientação no ensino fundamental. Revista Novas Tecnologias na Educação, 15(2), 1–10. [GS Search]
International Standard Organization (ISO). (2014). ISO/IEC 25010: Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models, Technical Report.
Johnson, R. B. & Christensen, L. (2014). Educational Research – Quantitative, Qualitative, and Mixed Approaches (5th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Kasunic, M. (2005). Designing an effective survey. Handbook CMU/SEI-2005-HB-004, Software Engineering Institute. Pittsburgh: Carnegie Mellon University. [GS Search]
Keller, J. (1987). Development and Use of the ARCS Model of motivational Design. Journal of Instructional Development, 10(3), 2-10. doi: 10.1007/BF02905780. [GS Search]
Kosa, M., Yilmaz, M., O'Connor, R., & Clarke, P. (2016). Software engineering education and games: a systematic literature review. Journal of Universal Computer Science, 22(12), 1558-1574. doi: 10.3217/jucs-022-12-1558. [GS Search]
Krathwohl, D. R., Bloom, B. S., & Masia, B. B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. Philadelphia: David McKay Company, Inc.
Malhotra, N. K., & Birks, D. F. (2008). Marketing Research: An Applied Approach (3rd ed.). Philadelphia: Trans-Atlantic Publications. [GS Search]
Mayer, I. (2012). Towards a Comprehensive Methodology for the Research and Evaluation of Serious Games. Procedia Computer Science, 15, 233-247. doi: 10.1016/j.procs.2012.10.075. [GS Search]
Mohamed, H. & Jaafar, A. (2010). Development and Potential Analysis of Heuristic Evaluation for Educational Computer Game (PHEG). Proc. of the 5th Int. Conf. on Computer Sciences and Convergence Information Technology, (pp. 222-227). Seoul, South Korea. doi: 10.1109/ICCIT.2010.5711061. [GS Search]
Moosa, A. M., Al-Maadeed, N., & Alja'am, J. M. (2018). A Simple Health-Based Game for Children. Proc. of the International Conference on Computer and Applications. Beirut, Líbano, p. 309-312. doi: 10.1109/COMAPP.2018.8460213. [GS Search]
O’Brien, H. L. & Toms, E. G. (2010). The Development and Evaluation of a Survey to Measure User Engagement. Journal of the American Society for Information Science and Technology, 61(1), 50–69. doi: 10.1002/asi.21229. [GS Search]
Pereira, C. X., Hoffmann, Z., Castro, C. P., Santos, G. S., Aires, T. A., & Francisco, R. E. (2017). CHEMIS3: A game for learning chemical concepts through elements of nature. Proc. of the 12nd Iberian Conference on Information Systems and Technologies, Lisboa, Portugal, p. 1-5. doi: 10.23919/CISTI.2017.7975801. [GS Search]
Petri, G. & Chiavegatti, N. C. (2015). Um Role-Playing Game para o Ensino de Elicitação e Análise de Requisitos. Revista Novas Tecnologias na Educação, 13(1), 1-10. doi: 10.22456/1679-1916.57599. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2016). How to evaluate educational games: a systematic literature review. Journal of Universal Computers Science, 22(7), 992-1021. doi: 10.3217/jucs-022-07-0992. [GS Search]
Petri, G. & Gresse von Wangenheim, C. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, 107, 68-90. doi: 10.1016/j.compedu.2017.01.004. [GS Search]
Petri, G., Battistella, P., Cassettari, F., Gresse von Wangenheim, C., & Hauck, J. (2016). Um Quiz Game para a revisão de conhecimento em Gerenciamento de Projetos. Anais do 27º Simpósio Brasileiro de Informática na Educação, pp. 320-329. Uberlândia/MG, Brasil. doi: 10.5753/cbie.sbie.2016.320. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2017). A Large-scale Evaluation of a Model for the Evaluation of Games for Teaching Software Engineering. Proc. of the IEEE/ACM 39th Int. Conf. on Software Engineering: Software Engineering Education and Training Track, (pp. 180-189). Buenos Aires/Argentina. doi: 10.1109/ICSE-SEET.2017.11. [GS Search]
Petri, G., Gresse von Wangenheim, C., & Borgatto, A. F. (2018). MEEGA+, Systematic Model to Evaluate Educational Games. In Newton Lee (Eds) Encyclopedia of Computer Graphics and Games, (pp. 1-7). Cham: Springer. doi: 10.1007/978-3-319-08234-9. [GS Search]
Poels, K., Kort, Y. D., & Ijsselsteijn, W. (2007). It is always a lot of fun!: exploring dimensions of digital game experience using focus group methodology. Proc. of Conf. on Future Play, (pp. 83-89). Toronto, Canada. doi: 10.1145/1328202.1328218. [GS Search]
Prensky, M. (2007). Digital Game-Based Learning. New York: Paragon House. [GS Search]
Raiche, G., Riopel, M., & Blais, J.-G. (2006). Nongraphical solutions for the Cattell’s scree test. Proc. of the Annual meeting of the Psychometric Society. Montreal, Canada. [GS Search]
Ritterfeld, U., Cody, M., & Vorderer, P. (Eds.). (2010). Serious Games. New York: Routledge.
Ross, J. A. (2006). The reliability, validity, and utility of self-assessment. Practical Assessment, Research & Evaluation, 11(10), 1-13. [GS Search]
Savi, R., Gresse von Wangenheim, C., & Borgatto, A. F. (2011). A model for the evaluation of educational games for teaching software engineering. Anais do XXV Simpósio Brasileiro de Engenharia de Software, pp. 194-203. São Paulo, Brasil. doi: 10.1109/SBES.2011.27 [GS Search]
Schanzenbach, D. W. (2012). Limitations of Experiments in Education Research. Education Finance and Policy, 7(2), 219-232. doi: 10.1162/EDFP_a_00063. [GS Search]
Sharma, R., Jain, A., Gupta, N., Garg, S., Batta, M., & Dhir, S. K. (2016). Impact of self-assessment by students on their learning. International Journal of Applied and Basic Medical Research, 6(3), 226–229. doi: 10.4103/2229-516X.186961. [GS Search]
Serrano-Laguna, A., Manero, B., Freire, M., & Fernández-Manjón, B. (2018). A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes. Multimedia Tools and Applications, 77(2), 2849-2871. [GS Search]
Silva, J. P., Carvalho, F. C., Silvestre, A., Luiz, M. F., Alencar, R. L., & Silveira, I. F. (2017) Forca dos Sinônimos: Um Jogo da Forca Multiplayer para o ensino de Tesauros em Língua Portuguesa como Ferramenta de busca na Web. Anais do Simpósio Brasileiro de Jogos e Entretenimento Digital, Curitiba/PR, Brasil, p. 1128-1131.[GS Search]
Silva, J. P., Jorge, A. A., Gaia, C. C., & Costa, G. C. (2017). Quais as chances? Um jogo de dados e cartas para o ensino do cálculo de probabilidades. Anais do 8º Congresso de Inovação, Ciência e Tecnologia do IFSP, Cubatão/SP, Brasil, p. 1-3. [GS Search]
Simpson, E. J. (1972). The classification of educational objectives, psychomotor domain. Washington: Gryphon House. [GS Search]
Sindre, G. & Moody, D. (2003). Evaluating the Effectiveness of Learning Interventions: an Information Systems Case Study. Proc. of the 11th European Conf. on Information Systems, Paper 80. Naples, Italy. [GS Search]
Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-Assessment of Knowledge: A Cognitive Learning or Affective Measure? Academy of Management Learning & Education, 9(2), 169-191. [GS Search]
Sweetser, P. & Wyeth, P. (2005). GameFlow: a model for evaluating player enjoyment in games. Computers in Entertainment, 3(3), 1-24. [GS Search]
Tabachnick, B. G. & Fidell, L. S. (2007). Using Multivariate Statistics (5th ed.). Boston: Allyn and Bacon. [GS Search]
Takatalo, J., Häkkinen, J., Kaistinen, J., & Nyman, G. (2010). Presence, Involvement, and Flow in Digital Games. In: Bernhaupt, R. (Ed.). Evaluating User Experience in Games: Concepts and Methods, (pp. 23-46). London: Springer.[GS Search]
Thomas, G., Martin, D., & Pleasants, K. (2011). Using self- and peer-assessment to enhance students’ future-learning in higher education. Journal of University Teaching & Learning Practice, 8(1), 1-17. [GS Search]
Tonussi, L. P. & Hauck, J. C. R. (2017). Um Módulo de Jogo de Perguntas e Respostas para apoio ao Ensino de Gerência de Projetos Integrado ao Moodle. Proc. of the Computer on the Beach. Florianópolis, Brazil (in Portuguese). [GS Search]
Trochim, W. M., & Donnelly, J. P. (2008). Research methods knowledge base (3rd ed.). Mason: Atomic Dog Publishing. [GS Search]
Tullis, T. & Albert, W. (2008). Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Burlington: Morgan Kaufmann. [GS Search]
Wiebe, E. N., Lamb, A., Hardy, M., & Sharek, D. (2014). Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, 32, 123-132. doi: 10.1016/j.chb.2013.12.001 . [GS Search]
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012). Experimentation in Software Engineering. New York: Springer-Verlag Berlin Heidelberg. [GS Search]
Yin, R. K. (2017). Case study research: design and methods (6th ed.). Thousand Oaks: Sage Publications, Inc. [GS Search]
Zaibon, S. B. & Shiratuddin, N. (2010). Heuristics Evaluation Strategy for Mobile Game-Based Learning. Proc of the 6th IEEE Int. Conf. on Wireless, Mobile and Ubiquitous Technologies in Education, (pp. 127-131). Kaohsiung, Taiwan. doi: 10.1109/WMUTE.2010.27. [GS Search]
Zaibon, S. B. (2015). User Testing on Game Usability, Mobility, Playability, and Learning Content of Mobile Game-Based Learning. Journal Teknologi, 77(29), 131-139. [GS Search]
Archivos adicionales
Published
Cómo citar
Issue
Section
Licencia
Derechos de autor 2019 Giani Petri, Christiane Gresse von Wangenheim, Adriano Ferret Borgatto
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
