Elaboration and validation of an instrument to evaluate the impact of computing interventions in midlle school final years
DOI:
https://doi.org/10.5753/rbie.2023.2932Keywords:
21st Century Skills, Computer Education, Assessment, ValidationAbstract
Due to the current and constant transformation of the world, the job market is increasingly selective. Companies are looking for creative professionals with cognitive skills adapted to reality. In the context of this work, these are the so-called 21st century skills. One of the ways to develop these skills is through computer education. However, since the area of computing education can be considered recent, there is a need for more research related to the identification of the effects of computing interventions on the development of 21st century skills. In addition, most of the existing studies adopt instruments that have not been subjected to a systematic validation process, that is, according to the context in which the studies are being applied. Thus, the objective of this work is to develop and validate a 21st century skills assessment instrument through the teaching of computing. The instrument is developed in Portuguese and is suitable for the reality of middle school. The initial assessment in terms of reliability and validity of the instrument, based on 580 responses from middle school students, indicates high internal reliability (Cronbach's alpha = 0.819 and 0.772), after a reclassification process. Thus, the instrument represents an important step towards the assessment of 21st century skills, and the growing systematic of teaching computing in education.
Downloads
References
Balbinotti, M. L. L., Marcos Alencar Abaide e Barbosa (2008, 08). Análise da consistência interna e fatorial confirmatório do imprafe-126 com praticantes de atividades físicas gaúchos. PsicoUSF, 13, 1 - 12. [GS Search]
Beeckman, D., Vanderwee, K., Demarré, L., Paquay, L., Hecke, A., & Defloor, T. (2009, 09). Pressure ulcer prevention: Development and psychometric validation of a knowledge assessment instrument. International journal of nursing studies, 47, 399-410. doi: 10.1016/j.ijnurstu.2009.08.010. [GS Search]
Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. , 17-66. doi: 10.1007/978-94-007-2324-5. [GS Search]
BNCC (2018). Base nacional comum curricular (bncc). [Link]. [acessado em 07/07/2022]
Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. In aera 2012. [GS Search]
Castle, A. (2006, 05). Assessment of the critical thinking skills of student radiographers. Radiography, 12, 88-95. doi: 10.1016/j.radi.2005.03.004. [GS Search]
Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245-276. doi: 10.1207/s15327906mbr0102_10. [GS Search]
Chai, C., Deng, F., Tsai, P., Koh, J., & Tsai, C. (2015, sep 13). Assessing multidimensional students’ perceptions of twenty-first-century learning practices. Asia Pacific Education Review, 16(3), 389–398. doi: 10.1007/s12564-015-9379-4. [GS Search]
Chen, G., Shen, J., Barth-Cohen, L., Jiang, S., Huang, X., & Eltoukhy, M. (2017, 03). Assessing elementary students’ computational thinking in everyday reasoning and robotics programming. Computers & Education, 109, 162-175. doi: 10.1016/j.compedu.2017.03.001. [GS Search]
Claro, M., Preiss, D., San Martin, E., Jara, I., Hinostroza, J., Valenzuela, S., . . . Nussbaum, M. (2012, 11). Assessment of 21st century ict skills in chile: Test design and results from high school level students. Computers & Education, 59, 1042–1053. doi: 10.1016/j.compedu.2012.04.004. [GS Search]
Connolly, T., Boyle, E., Macarthur, E., Hainey, T., & Boyle, J. (2012, 09). A systemic literature review of empirical evidence on computer games and serious games. Computers and Education, 59, 661 – 686. doi: 10.1016/j.compedu.2012.03.004. [GS Search]
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. [GS Search]
CSTA (2016). In Csta-12 computer science standards: Revised 2016. New York, NY, USA: Association for Computing Machinery. [GS Search]
CSTA (2017). In Csta k–12 computer science standards: Revised 2017. New York, NY, USA: Association for Computing Machinery.[GS Search]
Curzon, P., Bell, T., Waite, J., Dorling, M., Fincher, S., & Robins, A. (2019, 02). Computational thinking. In The cambridge handbook of computing education research (p. 513-546). Cambridge University Press. doi: 10.1017/9781108654555.018. [GS Search]
Denning, P. (2009, 06). The profession of it beyond computational thinking. Commun. ACM, 52, 28-30. doi: 10.1145/1516046.1516054. [GS Search]
Denning, P. J. (2017). Remaining trouble spots with computational thinking. Commun. ACM, 60, 33–39. doi: 10.1145/2998438. [GS Search]
DeWilde, J. (2020). 21st-century skills in the art classroom. Disponível em: [Link] Acesso em: 17 fevereiro 2022. [GS Search]
Dziuban, C. D., & Shirkey, E. C. (1974). When is a correlation matrix appropriate for factor analysis? some decision rules. Psychological Bulletin, 81, 358-361. [GS Search]
Figueiredo, D., & Júnior, J. (2010, 06). Visão além do alcance: Uma introdução à análise fatorial. Opinião Pública, 16, 160-185. doi: 10.1590/S0104-62762010000100007. [GS Search]
Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299. [GS Search]
Greve, B. (2019). The digital economy and the future of european welfare states. International Social Security Review, 72(3), 79–94. doi: 10.1111/issr.12214. [GS Search]
Heintz, F., Mannila, L., & Färnqvist, T. (2016). A review of models for introducing computational thinking, computer science and computing in k-12 education. FIE. [GS Search]
Hodge, K. A., & Lear, J. L. (2011). Employment skills for 21st century workplace: The gap between faculty and student perceptions. Journal of Career and Technical Education, 26, 28- 41. [GS Search]
Hoegh, A., & Moskal, B. M. (2009). Examining science and engineering students’ attitudes toward computer science. In 2009 39th ieee frontiers in education conference (p. 1-6). doi: 10.1109/FIE.2009.5350836. [GS Search]
Hutcheson, G. D., & Sofroniou, N. (1999). The multivariate social scientist: Introductory statistics using generalized linear models. United Kingdom: Sage Publications Ltd. [GS Search]
Jaquith, D. B., & Hathaway, N. E. (2013). The learner-directed classroom: Developing creative thinking skills through art. [GS Search]
Johnson, R., & Wichern, D. (2007). Applied multivariate statistical analysis. Pearson Prentice Hall. Retrieved from [GS Search]
Joseph Hair, W. C., J, B., Anderson, & Tatham, R. (1998). Multivariate data analysis. New Jersey: Prentice Hall. [GS Search]
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in factor analysis. Psychometrika, 23, 187-200. [GS Search]
Kanti Mardia, J. B., J. Kent (1979). Multivariate analysis. London: Academic Press. [GS Search]
Korkmaz, O., Cakir, R., & Ozden, M. (2017). A validity and reliability study of the computational thinking scales (cts). Computers in Human Behavior, 72. doi: 10.1016/j.chb.2017.01.005. [GS Search]
Kucuk, S., & Sisman, B. (2017, 04). Behavioral patterns of elementary students and teachers in one-to-one robotics instruction. Computers & Education, 111. doi: 10.1016/j.compedu.2017.04.002. [GS Search]
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33 1, 159-74. [GS Search]
Lau, W. W., & Yuen, A. H. (2014). Developing and validating of a perceived ict literacy scale for junior secondary school students: Pedagogical and educational contributions. Computers & Education, 78, 1-9. doi: 10.1016/j.compedu.2014.04.016. [GS Search]
Lee, K., Tsai, P.-S., Chai, C., & Koh, J. (2014, October). Students’ perceptions of self-directed learning and collaborative learning with and without technology. J. Comp. Assist. Learn., 30(5), 425–437. Retrieved from doi: 10.1111/jcal.12055. [GS Search]
Li, M.-C., & Tsai, C.-C. (2013, December). Game-based learning in science education: A review of relevant research. Journal of Science Education and Technology, 22(6), 877–898. [GS Search]
Lorenzo-Seva, U., Timmerman, M., & Kiers, H. (2011). The hull method for selecting the number of common factors. Multivariate Behavioral Research, 46(2), 340–364. doi: 10.1080/00273171.2011.564527. [GS Search]
Malmi, L., Utting, I., & Ko, A. J. (2019). Tools and environments. In The cambridge handbook of computing education research (p. 639–662). Cambridge University Press. doi: 10.1017/9781108654555.022. [GS Search]
Matos, D., & Castilho Rodrigues, E. (2019). Análise fatorial.[GS Search]
Mioto, F., Petri, G., Gresse von Wangenheim, C., Borgatto, A., & Pacheco, L. (2019, 01). bases21 - um modelo para a autoavaliação de habilidades do século xxi no contexto do ensino de computação na educação básica. Revista Brasileira de Informática na Educação, 27, 26. doi: 10.5753/rbie.2019.27.01.26. [GS Search]
P21 (2017). Framework for 21st century learning definitions.. Disponível em: [Link]. Acesso em: 17 fevereiro 2022. [GS Search]
Pacheco, L., Degering, L., Mioto, F., Gresse von Wangenheim, C., Borgato, A., & Petri, G. (2020, 01). Improvements in bases21: 21st-century skills assessment model to k12. In (p. 297-307). doi: 10.5220/0009581702970307. [GS Search]
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. USA: Basic Books, Inc. [GS Search]
Pasquali, L. (1999). Análise fatorial: um manual teórico-prático. Editora UnB. [GS Search]
Pasquali, L. (2009, Dec.). Psychometrics. Revista da Escola de Enfermagem da USP, 43(spe), 992-999. Retirado de: [Link] doi: 10.1590/S0080-62342009000500002. [GS Search]
Pereira, J. C. R. (1999). Análise de dados qualitativos: estratégias metodológicas para as ciências da saúde, humanas e sociais. São Paulo: EDUSP. [GS Search]
Piniuta, I. (2019). Technology based activities to develop 21st century skills in the foreign language classroom. In Proceedings of the 2019 8th international conference on educational and information technology (p. 79–85). New York, NY, USA: Association for Computing Machinery. doi: 10.1145/3318396.3318404. [GS Search]
Psycharis, S., & Kallia, M. (2017, 10). New paper "the effects of computer programming on high school students’ reasoning skills, and self-efficacy and problem solving in mathematics". Instructional Science, 45, 583–602. doi: 10.1007/s11251-017-9421-5. [GS Search]
Reise, S. P., Waller, N. G., & Comrey, A. L. (2000). Factor analysis and scale revision. Psychological assessment, 12 3, 287-97. [GS Search]
Robles, M. M. (2012). Executive perceptions of the top 10 soft skills needed in today’s workplace. Business Communication Quarterly, 75(4), 453-465. doi: 10.1177/1080569912460400. [GS Search]
Rodríguez-Martínez, J. A., González-Calero, J. A., & Sáez-López, J. M. (2020). Computational thinking and mathematics using scratch: an experiment with sixth-grade students. Interactive Learning Environments, 28(3), 316-327.[GS Search]
Rotherham, A., & Willingham, D. (2011). 21st-century” skillls. not new, but a worthy challenge. American Educator, 17-20. Retrieved from [Link]. [GS Search]
Sáez-López, J.-M., Sevillano-García, M., & Vázquez-Cano, E. (2019). The effect of programming on primary school students’ mathematical and scientific understanding: educational use of mbot. Educational Technology Research and Development, 1-21. [GS Search]
Scott, C. L. (2015). The futures of learning 3 : what kind of pedagogies for the 21st century? In Education research and foresight working papers. UNESDOC Digital Library.[GS Search]
Selby, C., & Woollard, J. (2013). Computational thinking: the developing definition. University of Southampton (E-prints). [GS Search]
Shackelford, R., McGettrick, A., Sloan, R., Topi, H., Davies, G., Kamali, R., . . . Lunt, B. (2006). Computing curricula 2005: The overview report. SIGCSE Bull., 38, 456–457. doi: 10.1145/1124706.1121482. [GS Search]
Sime, M., Arblaster, A. T., & Green, T. (1977). Structuring the programmer’s task. Journal of occupational psychology, 50, 205-216. [GS Search]
Spolaôr, N., & Benitti, F. B. V. (2017). Robotics applications grounded in learning theories on tertiary education: A systematic review. Comput. Educ., 112, 97-107. [GS Search]
Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 103 - 99. [GS Search]
Tabachnick, L., B.; Fidell (2007). c. United States: Allyn & Bacon. [GS Search]
Tan, J. P.-L., Choo, S. S., Kang, T., & Liem, G. A. D. (2017). Educating for twenty-first century competencies and future-ready learners: research perspectives from singapore. Asia Pacific Journal of Education, 37(4), 425-436. doi: 10.1080/02188791.2017.1405475. [GS Search]
Terwee, C., Bot, S., Boer, M., van der Windt, D., Knol, D., Dekker, J., . . . De Vet, H. (2007,02). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of clinical epidemiology, 60, 34-42. [GS Search]
Thorndike, R. M. (1995). Book review : Psychometric theory (3rd ed.) by jum nunnally and ira bernstein new york: Mcgraw-hill, 1994, xxiv + 752 pp. Applied Psychological Measurement, 19(3), 303-305. [GS Search]
Torres, G., & da Hora, H. (2014). Pesquisa em saúde pública: Como desenvolver e validar instrumentos de coleta de dados. Appris. [GS Search]
van Laar, E., van Deursen, A. J., van Dijk, J. A., & de Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72, 577-588. Retrieved from [Link]. doi: 10.1016/j.chb.2017.03.010. [GS Search]
Vasil, M., Weiss, L., & Powell, B. (2019). Popular music pedagogies: An approach to teaching 21st-century skills. Journal of Music Teacher Education, 28(3), 85-95. doi: 10.1177/1057083718814454. [GS Search]
Voogt, J., & Roblin, N. P. (2012). A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies. Journal of Curriculum Studies, 44(3), 299-321. doi: 10.1080/00220272.2012.668938. [GS Search]
Weinberg, S. (1971). Entropy Generation and the Survival of Protogalaxies in an Expanding Universe. The Astrophysical Journal, 168, 175. doi: 10.1086/151073. [GS Search]
Welch, D., & McDowall, J. (2010). A comparison of creative strategies in teaching undergraduate students in the visual arts and design. ACUADS. [GS Search]
Werner, L., Denner, J., & Campe, S. (2014). Using computer game programming to teach computational thinking skills. In Learning, education and games (p. 37–53). ETC Press. [GS Search]
Werner, L., Denner, J., Campe, S., & Kawamoto, D. C. (2012). The fairy performance assessment: Measuring computational thinking in middle school. In Proceedings of the 43rd acm technical symposium on computer science education (p. 215–220). New York, NY, USA: Association fo Computing Machinery. doi: 10.1145/2157136.2157200. [GS Search]
Wing, J. (2008). Computational thinking and thinking about computing. Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, 366, 3717-25. doi: 10.1098/rsta.2008.0118. [GS Search]
Wing, J. (2017, November). Computational thinking’s influence on research and education for all. Italian Journal of Educational Technology, 25(2), 7–14. Retrieved from [Link]. [GS Search]
Wing, J. M. (2006). Computational thinking. Commun. ACM, 49(3), 33–35. [GS Search]
Yağcı, M. (2019). A valid and reliable tool for examining computational thinking skills. Education and Information Technologies, 24, 929–951. [GS Search]
Yusoff, M. S. B. (2019, 06). Abc of content validation and content validity index calculation. Education in Medicine Journal, 11, 49-54. doi: 10.21315/eimj2019.11.2.6. [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Leandro Santos da Cruz, José Amancio Macedo Santos
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
