Planning the Design and Execution of Student Performance Assessment in Serious Games

Rháleff N. R. Oliveira; Rafaela V. Rocha; Denise H. Goya

doi:10.5753/jis.2021.1907

Authors

Rháleff N. R. Oliveira Universidade Federal do ABC https://orcid.org/0000-0001-6983-4896
Rafaela V. Rocha Universidade Federal do ABC https://orcid.org/0000-0003-4573-3016
Denise H. Goya Universidade Federal do ABC https://orcid.org/0000-0003-0852-6456

DOI:

https://doi.org/10.5753/jis.2021.1907

Keywords:

Serious Games, Conceptual Model, Human Performance Assessment, Evaluation

Abstract

Serious Games (SGs) are used to support knowledge acquisition and skill development. For this, there is a need to measure the results achieved (both during and after students play) to ensure the game effectiveness. In this context, the aim is to develop and evaluate the AvaliaJS, a conceptual model to structure, guide and support the planning of the design and execution of the student's performance assessment in SGs. AvaliaJS has two artifacts: a canvas model, for high-level planning, and an assessment project document, for more detailed specifications of the canvas. To analyze and exemplify the use of the model, the artifacts were applied to three ready-made games as a proof of concept. In addition, the quality of AvaliaJS was evaluated by experts in SGs development and assessment using a questionnaire. The results of experts' answers confirm a good internal consistency (Cronbach's alpha α = 0.87) which indicates that AvaliaJS is correct, authentic, consistent, clear, unambiguous and flexible. However, the model will need to be validated during the process of creating a new game to ensure its usability and efficiency. In general, AvaliaJS can be used to support the team in the planning, documentation and development of artifacts and data collection in SGs, as well as in the execution of the assessment, learning measurement and constant and personalized feedback for students.

Downloads

Download data is not yet available.

References

Abdulmajed, H.; Park, Y. S.; Tekian, A. (2015). Assessment of educational games for health professions: A systematic review of trends and outcomes. Journal Medical Teacher, v. 37(1), p. 27-32.

Abt, C. C. (1987). Serious Games. Lanham. MD: University Press of America.

Akilli, G.K.; Cagiltay, K. (2006). An Instructional Design/Development Model for the Creation of Game-Like Learning Environments: the FIDGE model. Pivec, M. (Ed.), Affective and Emotional Aspects of Human-Computer Interaction: Game-Based and Innovative Learning. IOS Press, p. 93-112.

Alfarah, Z.; Schünemann, H. J.; Akl, E. A. (2010). Educational games in geriatric medicine education: a systematic review. BMC Geriatrics, p. 1-5.

Alvarez, J.; Djaouti, D. (2012). Introduction au serious game. 2 ed. Paris: Questions théoriques.

Ariffin, M. M.; Oxley, A.; Sulaiman, S. (2014). Evaluating Game-based Learning Effectiveness in Higher Education. Procedia - Social and Behavioral Sciences, v. 123, p. 20-27.

Aslan, S.; Balci, O. (2015). GAMED: digital educational game development methodology. Simulation, v. 9(4), p. 307-319.

Baehr, M. (2010). 4.1.2 Distinction between assessment and evaluation. Faculty guidebook (p. 7-10). (4th ed.). Lisle, IL: Pacific Crest.

Basili, V.; Caldiera, G.; Rombach, H. (1994). Goal Question Metric Paradigm. Encyclopedia of Software Engineering, v. 1, John Wiley & Sons, 1994. p. 528-532.

Battistella, P. E.; Wangenheim, C. G. V.; Fernandes, J. M. (2014). Como jogos educacionais são desenvolvidos? Uma revisão sistemática da literatura. In XXII Workshop sobre Educação em Computação, Brasília, DF, p. 1-10.

Beecham, S.; Hall, T.; Britton, C.; Cottee, M.; Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. The Journal of Systems and Software, v. 76.

Bellotti, F.; Kapralos, B.; Lee, K.; Moreno-ger, P.; Berta, R. (2013). Assessment in and of serious games: an overview. Advances in Human-Computer Interaction, v. 2013, p. 1–11.

Bente, G.; Breuer, J. (2009). Making the implicit explicit: embedded measurement in serious games. Ritterfield, U. et al. (eds.), Serious Games: Mechanisms and Effects, Routledge, NY, p. 322–343.

Bloom, B. S. (1956). Taxonomy of Educational Objectives: the classification of educational goals – Handbook I: Cognitive Domain. New York, NY, USA: McKay.

Boyle, E. A.; Connolly, T. M.; Hainey, T.; Boyle, J. M. (2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, v. 28(3), p. 771–780.

Calderón, A.; Ruiz, M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, v. 87, p. 396–422.

Carey, R. (2015). Game Design Canvas. In: Serious Play Conference, Pittsburgh, PA.

Chaudy, Y; Connolly, T. (2019). Specification and evaluation of an assessment engine for educational games: Integrating learning analytics and providing an assessment authoring tool. Entertainment Comput., v. 30, p. 1-16.

Coelho, J. A. P. M.; Souza, G. H. S.; Albuquerque, J. (2020). Desenvolvimento de questionários e aplicação na pesquisa em Informática na Educação. Jaques, P. A. et al., (Org.) Metodologia de Pesquisa Científica em Informática na Educação: Abordagem Quantitativa. Porto Alegre, RS: SBC.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, v. 16(3), p. 297–334.

Csikszentmihalyi, M. (1990). Flow: the psychology of optimal experience. NY: Harper & Row.

Daoudi, I.; Tranvouez, E.; Chebil, R.; Espinasse, B.; Chaari, W. (2017). Learners’ Assessment and Evaluation in Serious Games: Approaches and Techniques Review. International Conference on Information Systems for Crisis Response and Management in Mediterranean Countries, Xanthi, Greece.

Dempsey, J., Rasmussen, K.; Lucassen, B. (1996). The instructional gaming literature: Implications and sources. Technical Report 96-1. College of Education, University of South Alabama, AL.

Devellis, R. F. (2016). Scale development: theory and applications. 4th ed. Thousand Oaks: SAGE Publications.

Djaouti, D.; Alvarez, J.; Jessel, J. P. (2011). Classifying serious games: the G/P/S model. Felicia, P. (ed.), Handbook of research on improving learning and motivation through educational games: Multidisciplinary Approaches, p. 118-136.

Durand, T. (2000). L’alchimie de la compétence. Revue Française de Gestion, n. 127, jan./fev. 2000. p. 84-102.

Emmerich, K.; Bockholt, M. (2016). Serious Games Evaluation: Processes, Models, and Concepts. Dörner R., et al. (eds.), Entertainment Computing and Serious Games. Lecture Notes in Computer Science, Springer, Cham, v. 9970, p. 265-283.

Emmerich, K.; Bogacheva, N.; Bockholt, M.; Wendel, V. (2016). Operationalization and Measurement of Evaluation Constructs. In: Dörner R. et al. (eds.), Entertainment Computing and Serious Games. Lecture Notes in Computer Science, Springer, Cham, v. 9970, p. 306-331.

Eseryel, D.; Ifenthaler, D.; Ge, X. (2011). Alternative Assessment Strategies for Complex Problem Solving in Game-Based Learning Environments. Ifenthaler D. et al. (eds) Multiple Perspectives on Problem Solving and Learning in the Digital Age. Springer, p. 159-178.

Falchikov, N. (2005). Improving Assessment Through Student Involvement: Practical Solutions For Aiding Learning In Higher And Further Education. New York: RoutledgeFalmer.

Hattie, J.; Timperley, H. (2007). The power of feedback. Review of Educational Research, v. 77(1), p. 81‐112.

Hays, R. T. (2005). The Effectiveness of Instructional Games: a literature review and discussion. Naval Air Warfare Center Training Systems Division (Technical Report 2005 – 004).

Hettiarachchi, Enosha; Huertas, M. Antonia; Mor, Enric (2013). Skill and Knowledge E-Assessment: A Review of the State of the Art. In IN3 of the UOC Working Paper Series, p. 1-46

Ibrahim, R.; Jaafar, A. (2009). Educational games (EG) design framework: Combination of game design, pedagogy and content modeling. International Conference on Electrical Engineering and Informatics, IEEE, Selangor, Malaysia, p. 293-298.

IEEE. (2010). Systems and Software Engineering – Vocabulary. International Standard, ISO/IEC/IEEE 24765:2010(E), p. 1-418.

Ifenthaler, D.; Eseryel, D.; Ge, X. (2012). Assessment for Game-Based Learning. Springer, Verlag New York, p. 1-464.

Jappur, R. F.; Forcellini, F. A.; Spanhol, F. J. (2014). Modelo conceitual para jogos educativos digitais. AtoZ: novas práticas em informação e conhecimento, v. 3(2), p. 116-127.

Keller, J. M. (1987). Development and Use of the ARCS Model of Instructional Design. Journal of Instructional Development, v. 10(3), p. 2-10.

Keller, J. M. (2009). Motivational Design for Learning and Performance: The ARCS model approach. Springer.

Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. Internet and Higher Education, v. 8, p. 13-24.

Kirkpatrick, D. L.; Kirkpatrick, J. D. (2006). Evaluating Training Programs: The Four Levels. San Francisco, CA: Berrett-Koehler, 3th ed.

Kirkpatrick, J. D.; Kirkpatrick, W. K. (2016). Kirkpatrick's Four Levels of Training Evaluation. ATD Press.

Kolb, A.Y.; Kolb, D.A. (2005). Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education. Academy of Management Learning and Education, v. 4(2), p. 193-212.

Korhonen, T.; Halonen, R.; Ravelin, T.; Kemppainem, J.; Kokela, K. (2017). A Multidisciplinary Approach to Serious Game Development in the Health Sector. In 11th Mediterranean Conference on Information Systems (MCIS), Genoa, Italy, p. 1-14.

Lopes, M. C.; Fialho, F. A. P.; Cunha, C. J. C. A.; Niveiros, S. I. (2013). Business Games for Leadership Development. Simulation & Gaming, v. 44(4), p. 523–543.

Mourão, L.; Meneses, P. P. M. (2012). Construção de Medidas em TD&E. Abbad, G. S.; et al. Medidas de Avaliação em Treinamento, Desenvolvimento e Educação: ferramentas para gestão de pessoas. Porto Alegre, RS: Artmed, p. 50-63.

Oliveira, R. N. R. (2020). Planejamento do design e da execução da avaliação do desempenho de alunos em Jogos Sérios. 2020. Dissertation (Computer Science) - Federal University of ABC, Santo André, SP. p. 1-128.

Oliveira, R. N. R.; Belarmino, G.; Rodriguez, C.; Goya, D.; Venero, M. F.; Oliveira Junior, A.; Rocha, R. V. (2019). Avaliações em Jogos Educacionais: instrumentos de avaliação da reação, aprendizagem e comparação de jogos. In XXX Simpósio Brasileiro de Informática na Educação), p. 972-981.

Oliveira, R. N. R.; Cardoso, R. P.; Braga, J. C. B.; Rocha, R. V. (2018). Frameworks para Desenvolvimento de Jogos Educacionais: uma revisão e comparação de pesquisas recentes. In Anais XXIX Simpósio Brasileiro de Informática na Educação, p. 854-863.

Oliveira, R. N. R.; Rocha, R.V. (2019). Guerra em Alto Mar: um Jogo de Tabuleiro com Quiz Personalizável para Engajar e Motivar Estudantes. In XVIII Brazilian Symposium on Computer Games and Digital Entertainment, p. 1-4.

Oliveira, R. N. R.; Rocha, R. V. (2020a). AvaliaJS: Modelo Conceitual de Planejamento da Avaliação do Desempenho de Alunos em Jogos Sérios. Technical Report (in Portuguese). Federal University of ABC. Available at: https://bit.ly/RelatorioJS. Accessed: May 2020.

Oliveira, R. N. R.; Rocha, R. V. (2020b). Modelo Conceitual de Planejamento da Avaliação do Desempenho de Alunos em Jogos Sérios. In XVIV Brazilian Symposium on Computer Games and Digital Entertainment, p. 1-10.

Osterwalder, A. (2004). The Business Model Ontology – A Proposition in a Design Science Approach. Thesis (Business Information Systems) - University of Lausanne, Switzerland, p. 1-172.

Pereira Junior, H. A.; Menezes, C. S. (2015). Modelo para um Framework Computacional para Avaliação Formativa da aprendizagem em jogos digitais. In XIV Simpósio Brasileiro de Games e Entretenimento Digital, p. 819-828.

Petri, G.; Wangenheim, C. G. V. (2016). How to Evaluate Educational Games: a Systematic Literature Review. Journal of Universal Computer Science, v. 22(7), p. 992-1021.

Petri, G.; Wangenheim, C. G. V. (2017). How games for computing education are evaluated? A systematic literature review. Computers & Education, p. 68-90.

Petri, G. (2018). A method for the evaluation of the quality of games for computing education. Dissertation (Computer Science) - Federal University of Santa Catarina. Florianópolis, SC, p. 1-335.

Raabe, A. L. A.; Bombasar, J. R. (2020). Mensuração e testes em Informática na Educação. JAQUES, P. A. et al. (Org.) Metodologia de Pesquisa Científica em Informática na Educação: Abordagem Quantitativa. Porto Alegre, RS: SBC.

Rocha, R. V.; Bittencourt, I. I.; Isotani, S. (2015). Avaliação de Jogos Sérios: questionário para autoavaliação e avaliação da reação do aprendiz. In XIV Simpósio Brasileiro de Jogos e Entretenimento Digital, p. 1-10.

Rocha, R. V.; Valle, P. H. D.; Maldonado, J. C.; Bittencourt, I. I.; Isotani, S. (2017). AIMED: agile, integrative and open method for open educational resources development. In XVII IEEE International Conference on Advanced Learning Technologies, p. 1-6.

Rocha, R. V. (2014). Metodologia iterativa e modelos integradores para desenvolvimento de jogos sérios de treinamento e avaliação de desempenho humano. 2014. Thesis (Computer Science) – Federal University of São Carlos. São Carlos. p. 1 -237.

Rossato, I. F. (1996). Uma metodologia para a análise e solução de problema. Thesis (Production and System Engineering) - Federal University of Santa Catarina. Florianópolis.

Salas, E.; Rosen, M. A.; Held, J. D.; Weissmuller, J. J. (2009). Performance measurement in simulation-based training: a review and best practices. Simulation & Gaming, v. 40(3), p. 328–376.

Salen, K.; Zimmerman, R. (2003). Rules of Play: Game Design Fundamentals. MIT Press.

Sarinho, V. T. (2017). Uma Proposta de Game Design Canvas Unificado. In Anais XVI Simpósio Brasileiro de Jogos e Entretenimento Digital, p. 141-148.

Savi, R.; Gresse Von Wangenheim, C.; Ulbricht, V. R.; Vanzin, T. (2010). Proposta de um modelo para avaliação de jogos educacionais. RENOTE, v. 8(3), p. 1-10.

Shell, J. (2008). The Art of Game Design: A Book of Lenses. 2nd ed. AK Peters.

Slussareff, M.; Braad, E.; Wilkinson, P.; Strååt, B. (2016). Games for Learning. DÖRNER, R. et al. (Eds.), Entertainment Computing and Serious Games, LNCS, v. 9970, p. 189–211.

Sousa, T. C. (2014). Manual do Game Design Canvas. Available at: http://bit.ly/gdcanvas. Accessed: Jan 2019.

Star, K., Vuillier, L.; Deterding, S. (2016). D2. 6 Prosocial Game design methodology. Gamification of Prosocial Learning for Increased Youth Inclusion and Academic Achievement, p. 1-60.

Trybus, J. (2010). Game-Based Learning: What it is, Why it Works, and Where it's Going. NMI White Paper. New Media Institute, New York.

Victal, E. R. N; Menezes, C. S. (2015). Avaliação para Aprendizagem baseada em Jogos: Proposta de um Framework. In Anais XIV Simpósio Brasileiro de Jogos e Entretenimento Digital, p. 970-977.

Walker, E. (2015). Game Canvas Design. Ideas at Play Workshop 2015. Available at: https://drive.google.com/file/d/0B66VbdUcz8k9TVotdEFJcFZQeE0/view. Accessed: May 2019.

Wang, R.; De Maria, S. J.; Goldberg, A.; Katz, D. (2016). A Systematic Review of Serious Games in Training Health Care Professionals. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, v. 11(1), p. 41-51.

Wangenheim, C. G. V.; Kochanski, D.; Savi, R. (2009). Revisão Sistemática sobre Avaliação de Jogos Voltados para Aprendizagem de Engenharia de Software no Brasil. In Anais Fórum de Educação em Engenharia de Software, Fortaleza, CE, p. 1-8.

Westera, W.; Nadolski, R.; Hummel, H.; Wopereis, I. (2008). Serious games for higher education: a framework for reducing design complexity. Journal of Computer Assisted Learning, v. 24, p. 420–432.

Wohlin, C.; Runeson, P.; Höst, M.; Ohlsson, MC.; Regnell, B.; Wesslén, A.. (2000). Experimentation in Software Engineering-An Introduction. (1st ed.). Kluwer Academic Publishers, Norwell, MA, USA.

Yedri, O. B.; Aachak, E. A.; Bouhorma, M. (2018). Assessment-driven Learning through Serious Games: Guidance and Effective Outcomes. International Journal of Electrical and Computer Engineering, v. 8(5), p. 3304-3316.

Zerbini, T.; Borges-ferreira, M. F.; Abbad, G. S. (2012). Medidas de Reação a Cursos a Distância. Abbad, G. S. Medidas de Avaliação em Treinamento, Desenvolvimento e Educação: ferramentas para gestão de pessoas. Porto Alegre, RS: Artmed, p. 91-107.

Zinovieff, M. A.; Rotem, A. (2008). Review and analysis of training impact evaluation methods, and proposed measures to support a United Nations system fellowships evaluation framework prepared. WHO’s Department of Human Resources for Health, p. 1-46.

Zyda, M. (2005) From visual simulation to virtual reality to games. Computer, v. 38(9), p. 25–32.