bASES21 - A Model for the Auto-Assessment of 21th Century Skills in the Context of Teaching Computing in Basic Education
DOI:
https://doi.org/10.5753/rbie.2019.27.01.26Keywords:
21st Century Skill, Model, Assessment, Computing Education, K-12Abstract
In today's knowledge society, knowing how to interpret, search, communicate and share new information are key skills for a successful citizen. Such skills, known as 21st century skills, include critical thinking, teamwork, communication, among others. Currently, there are several proposals on how to integrate the learning of these skills in basic education, including through the teaching of computing. Yet, based on the hypothesis that teaching computing can contribute to the learning of 21st century skills, there is few systematically evidence available to support this claim. One reason for this absence is the lack of 21st century skills assessment models. Thus, the objective of this article is to develop and evaluate a model for the assessment of 21st century skills in the context of teaching computing in basic education. Based on the results of a mapping study, the bASES21 model is systematically developed as well as its measurement instrument, a self-assessment questionnaire. An initial assessment in terms of reliability and validity of the measurement instrument, based on 148 responses from students in basic education, indicate a high internal reliability (Cronbach's alpha = 0.958). Results regarding the validity of the instrument were inconclusive. Even although some skills demonstrate internal correlation, an improvement opportunity is the possible re-classification of items based on a more detailed analysis with a larger sample. Thus, the bASES21 model represents a first step supporting the assessment of 21st century skills, aiming at the systematic improvement of teaching computing in basic education.
Downloads
References
ACM (Association for Computing Machinery). (2005). Computing Curricula 2005. Disponível em: [Link].
Aesaert, K., Nijlen, D. V., Vanderlinde, R. & Braak, J. V. (2014). Direct measures of digital information processing and communication skills in primary education: Using item response theory for the development and validation of an ICT competence scale. Computers & Education, 76, p. 168-181. doi: 10.1016/j.compedu.2014.03.013. [GS Search].
Anderson, R. (2008). Implications of the information and knowledge society for education. In J. Voogt & G. Knezek (Eds.), International Handbook of Information Technology in Primary and Secondary Education. New York: Springer. doi: 10.1007/978-0-387-73315-9_1. [GS Search].
Base Nacional Comum Curricular (BNCC). Disponível em: [Link].
Basili, V. R.; Caldeira, G.; Rombach, H. D. (1994). Goal Question Metric Paradigm. In: Encyclopedia of Software Engineering, John Wiley & Sons.
Beecham, S., Hall, T., Britton, C., Cottee, M., & Rainer, A. (2005). Using an Expert Panel to Validate a Requirements Process Improvement Model. Journal of Systems and Software, 76(3), 251-275. doi: 10.1016/j.jss.2004.06.004. [GS Search].
Binkley, M., Ola, E., Herman, J. Raizen, S., Ripley, M. & Rumble, M. (2011). Defining 21st Century Skills. In: Assessment and teaching of 21st century skills. Nova York: Springer, p. 17-66.
Black, P., William, D. (1998). Assessment and classroom learning. Assessment in Education: Principles. Policy & Practice, 5, (1), p. 7–74. doi: 10.1080/0969595980050102.
Branch, R. M. (2010). Instructional Design: The ADDIE Approach. New York: Springer.
Brennan, K., Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In: Proceedings of the Annual Meeting of the American Educational Research Association, Vancouver, Canada. [GS Search].
Chai, C. S., Deng, F., Tsai, P. Koh, J. H. L. & Tsai, C. (2015). Assessing multidimensional students’ perceptions of twenty-first-century learning practices. Asia Pacific Education Review, 16(3), 389-398. doi: 10.1007/s12564-015-9379-4. [GS Search].
Claro, M., Preiss, D. D., Martín, E. S., Jara, I., Hinostroza, E., Valenzuela, S., Cortes, F. & Nussbaum, M. (2012). Assessment of 21st century ICT skills in Chile: Test design and results from high school level students. Computers & Education, 59(3), 1042-1053. doi: 10.1016/j.compedu.2012.04.004. [GS Search].
Cohen, J. (1998). Statistical Power Analysis for the Behavioral Sciences. New York: Routledge Academic.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, p. 297–334. doi: 10.1007/BF02310555. [GS Search].
CSTA (The CSTA Standards Task Force). (2017). CSTA K–12 Computer Science Standards – Revised 2017, New York: ACM.
Dede, C. (2010). Comparing frameworks for 21St century skills. In J. Bellanca & R. Brandt (Eds.), 21st century skills (51-76). Bloomington: Solution Tree Press. [GS Search].
Delors, J., Mufti, I. A., Amagi, I., Carneiro, R., Chung, F., Geremek, B., Gorham, W., Kornhauser, A., Manley, M., Quero, M. P., Savané, M., Singh, K., Stavenhagen, R., Suhr, M. W. & Nanzhao, Z. (1996). Learning: The treasure within. Paris: UNESCO. Disponível em: [Link].
Devellis, R. F. (2016). Scale Development: Theory and Applications. Beverly Hills: SAGE Publications. [GS Search].
Drasgow, F. (1986). Polychoric and polyserial correlations. In: Kotz, S., Johnson, N. L. Encyclopedia of Statistical Sciences, New York: John Wiley, 68-74.
Duckworth, A. L., Peterson, C., Matthews, M. D., Kelly, D. R. (2007). Grit: perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087-1101. doi: 10.1037/0022-3514.92.6.1087. [GS Search].
EC (European Commission). (2017). Coding - the 21st century skill. Disponível em: [Link].
Griffin, P. & Care, E. (2014). Developing learners’ collaborative problem solving skills. Melbourne: Melbourne Graduate School of Education. Disponível em: [Link].
Grover S. & Pea R. (2013). Computational Thinking in K–12 a Review of the State of the Field. Educational Researcher, 42(1), 38-43. doi: 10.3102%2F0013189X12463051. [GS Search].
Hattie, J. & Timperley, H. (2007) The power of feedback. Review of educational research, 77(1), 81-112. doi: 10.3102%2F003465430298487.
IFL (Institute for the Future of Learning) Assessing the Learning that Matters Most. 2015. Disponível em: [Link].
Ihantola, P., Ahoniemi, T., Karavirta, V., Seppälä, O. (2010) Review of recent systems for automatic assessment of programming assignments. In: Proceedings of the 10th Koli Calling International Conference on Computing Education Research, Koli, Finlandia. doi: 10.1145/1930464.1930480. [GS Search].
K-12 Computer Science Framework. 2016. Disponível em: http://www.k12cs.org
Kang, M., Heo, H., Jo, I., Shin, J. & Seo, J. (2010). Developing an Educational Performance Indicator for New Millennium Learners. Journal of Research on Technology in Education, 43(2). doi: 10.1080/15391523.2010.10782567. [GS Search].
Kasunic, M. (2005). Designing an effective survey. Handbook CMU/SEI-2005-HB-004, Software Engineering Institute. Pittsburgh: Carnegie Mellon University.
Keuning, H., Jeuring, J., Heeren, B. (2016) Towards a Systematic Review of Automated Feedback Generation for Programming Exercises. In: Proceedings of the ACM Conference on Innovation and Technology in Computer Science Education, Arequipa, Peru. doi: 10.1145/2899415.2899422. [GS Search].
Kivunja, C. (2014). Do You Want Your Students to Be Job-Ready with 21st Century Skills? Change Pedagogies: A Pedagogical Paradigm Shift from Vygotskyian Social Constructivism to Critical Thinking, Problem Solving and Siemens’ Digital Connectivism. International Journal of Higher Education, 3(3), 81-91. [GS Search].
Kyllonen, P. C. (2012). Measurement of 21st Century Skills Within the Common Core State Standards. In: Proceedings of Invitational Research Symposium on Technology Enhanced Assessments, 7-8. [GS Search].
Lau, W. W. F. & Yuen, A. H. K. (2014). Developing and validating of a perceived ICT literacy scale for junior secondary school students: Pedagogical and educational contributions. Computers & Education, 78, 1-9. doi: 10.1016/j.compedu.2014.04.016. [GS Search].
Losby, J., Wetmore, A. (2012). Jan Losby and Anne Wetmore on Likert Scales – Odd or Even?, Disponível em: [Link]
Masethe, M. A. Masethe, H. D. Odunaike, S. A. (2017). Scoping Review of Learning Theories in the 21th Century. In: Proceedings of the World Congress on Engineering and Computer Science, San Francisco, CA, EUA.
Melhem, A. (2002). Modelos de avaliação escolar utilizados em sala de aula – Uma análise nos cursos de administração na universidade federal do espírito santo (UFES) e de uma faculdade particular: Insumos para o aperfeiçoamento da gestão educacional. Dissertação (Mestrado em Administração), Fundação Getúlio Vargas, Rio de Janeiro, Brasil. [GS Search].
Merril, D. C., Reiser, B. J., Ranney, M., Trafton, J. G. (1992) Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. Journal of the Learning Sciences, 2(3), 277-305. doi: 10.1207/s15327809jls0203_2. [GS Search].
Morelli R., Lanerolle, T., Lake, P., Limardo, N., Tamostu, E. & Uche, C. (2010). Can Android App Inventor Bring Computational Thinking to K-12? Hartford, CT: Trinity College Hartford. [GS Search].
O’Neil, H. F., Schacter, J. (1997) Testing Specifications for Problem Solving Assessments. Relatório técnico, Los Angeles: University of California. 463. Disponível em: [Link]
OECD (Organization for Economic Co-operation and Development), Collaborative problem-solving framework. Paris: OECD Publishing, 2015.
Partnership for 21st Century Skills (P21). (2015). P21 framework definitions. Disponível em: [Link]
Partnership for 21st Century Skills (P21). (2017). Computer Science: A playground for 21st century skills. Disponível em: [Link]
Petway, K. T., Rikoon, S. H., Brenneman, M. W., Burrus, J., Roberts, R. D. (2016). Development of the Mission Skills Assessment and Evidence of Its Reliability and Internal Structure. Relatório de pesquisa, ETS (Educational testing service). doi: 10.1002/ets2.12107. Disponível em: [Link]. [GS Search].
Rosen, Y. & Tager, M. (2014). Computer-based Performance Assessment of Creativity Skills: A Pilot Study. In: Proceedings of International Association for Educational Assessment Conference, Singapore. [GS Search].
Rosen, Y. (2015). Computer-based Assessment of Collaborative Problem Solving: Exploring the Feasibility of Human-to-Agent Approach. International Journal of Artificial Intelligence in Education, 25(3), 380-406. doi: 10.1007/s40593-015-0042-3. [GS Search].
Ross, J. A. (2006) The reliability, validity, and utility of self-assessment. Practical Assessment, Research & Evaluation, 11(10), 1-13. [GS Search].
Rusman, E., Boon, J., Martínez-Monés, A., Rodríguez-Triana, M. J., Simeos, R. (2003) Towards the Use of New Methods for Formative e-Assessments of 21st Century Skills in Schools. In: Proc. of Technology Enhanced Formative Assessment Workshop at EC-TEL. Paphos, Chipre. [GS Search].
Sadler, D. R. (1989) Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. doi: 10.1007/BF00117714. [GS Search].
Shute, V. J. (2008) Focus on formative feedback. Review of Educational Research, 78(1), 153-189. doi: 10.3102%2F0034654307313795. [GS Search].
Siddiq, F., Gochyyev, P. & Wilson, M. (2017). Learning in Digital Networks - ICT Literacy: A novel assessment of students’ 21st century skills. Computer & Education, 109, 11-37. doi: 10.1016/j.compedu.2017.01.014. [GS Search].
Sitzman, T., Ely, K., Brown, K. G., Bauer, K. N. (2010) Self-assessment of knowledge: A cognitive learning or affective measure? Academy of Management Learning & Education, 9(2), 169-191. doi: 10.5465/amle.9.2.zqr169. [GS Search].
Sociedade Brasileira de Computação (SCB). (2017). Referenciais de Formação em Computação: Educação Básica. Disponível em: [Link]
Soland, J., Hamilton, L. S. & Stecher, B. M. (2013) Measuring 21st Century Competencies, Guidance for Educators. Global Cities Education Network Report.
Stegeman, M., Barendsen, E., Smeters, S. (2016) Designing a rubric for feedback on code quality in programming courses. In: Proceedings of the 16th Koli Calling International Conference on Computing Education Research, Koli: Finlandia. doi: 10.1145/2999541.2999555. [GS Search].
Susnea, I. & Vasiliu, G. (2016). A Fuzzy Logic Software Tool and a New Scale for the Assessment of Creativity. International Journal of Computers Communications & Control, 11(3), 441-449. doi: 10.15837/ijccc.2016.3.2192. [GS Search].
Topping, K. (2003) Self and Peer Assessment in School and University: Reliability, Validity and Utility. In M. Segers, F. Dochy, & E. Cascallar (Eds.), Optimising New Modes of Assessment: In Search of Qualities and Standards, Dordrecht: Kluwer Academic Publishers. [GS Search].
Torrance, H. (1995) Evaluating authentic assessment: Problems and possibilities in new approaches to assessment. Buckingham: Open University Press.
Trochim, W. M. & Donnelly, J. P. (2008). Research Methods Knowledge Base. Mason: Atomic Dog Publishing.
Tsai, M., Wang, C., Hsu, P. (2018). Developing the Computer Programming Self-Efficacy Scale for Computer Literacy Education. Journal of Educational Computing Research, 1-16. doi: 10.1177%2F0735633117746747. [GS Search].
Voogt, J., Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44(3), 299-321. doi: 10.1080/00220272.2012.668938. [GS Search].
Wangenheim, C. G., Alves, N. C., Weber, A. R. (2017). Resumo do K-12 Computer Science Standards (Versão 2017). Relatório Técnico do INCoD/INE/UFSC, Florianópolis/SC.
Ward, J. D., Lee, C. L. (2002) A review of problem-based learning. Journal of Family and Consumer Sciences Education, 20(1), 16-26.
Wiggins, G. P. (1993) Assessing student performance: Exploring the purpose and limits of testing. San Francisco: Jossey-Bass.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2019 Fernanda Mioto, Giani Petri, Christiane Gresse von Wangenheim, Adriano F. Borgatto, Lúcia H. M. Pacheco
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
