Detailed Item Response Theory Analysis of Algorithms and Programming Concepts in App Inventor Projects
DOI:
https://doi.org/10.5753/rbie.2021.2097Keywords:
Algorithms and Programming, App Inventor, Item Response Theory, Sequencing, RubricAbstract
Teaching computing in K-12 is often introduced focusing on algorithms and programming concepts using block-based programming environments, such as App Inventor. Yet, learning programming is a complex process and novices struggle with several difficulties. Thus, to be effective, instructional units need to be designed regarding not only the content but also its sequencing taking into consideration difficulties related to the concepts and the idiosyncrasies of programming environments. Such systematic sequencing can be based on large-scale project analyses by regarding the volition, incentive, and opportunity of students to apply the relevant program constructs as latent psychometric constructs using Item Response Theory to obtain quantitative ‘difficulty’ estimates for each concept. Therefore, this article presents the results of a large-scale data-driven analysis of the demonstrated use in practice of algorithms and programming concepts in App Inventor. Based on a dataset of more than 88,000 App Inventor projects assessed automatically with the CodeMaster rubric, we perform an analysis using Item Response Theory. The results demonstrate that the easiness of some concepts can be explained by their inherent characteristics, but also due to the characteristics of App Inventor as a programming environment. These results can help teachers, instructional and curriculum designers in the sequencing, scaffolding, and assessment design of programming education in K-12.
Descargas
Citas
Aivaloglou, E., Hermans, F. (2016). How kids code and how we know: An exploratory study on the Scratch repository. In Proceedings of the ACM Conference on International Computing Education Research, Melbourne, Australia, 8–12 September 2016; pp. 53–61. doi: 10.1145/2960310.2960325. [GS Search]
Alves, N. da C., Gresse von Wangenheim, C., & Hauck, J. C. R. (2019). Approaches to assess computational thinking competences based on code analysis in K-12 education: A systematic mapping study. Informatics in Education, 18(1), 17-39. doi: 10.15388/infedu.2019.02. [GS Search]
Alves, N. da C., von Wangenheim, C. G., Hauck, J., Borgatto, A., & Andrade, D. (2019, November). Uma análise do sequenciamento pedagógico no ensino de computação na educação básica. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE) (Vol. 30, No. 1, p. 1). doi:10.5753/cbie.sbie.2019.1. [GS Search]
Alves, N. da C., von Wangenheim, C. G., Hauck, J. C. R., & Borgatto, A. F. (2021, April). An Item Response Theory Analysis of Algorithms and Programming Concepts in App Inventor Projects. In Anais do Simpósio Brasileiro de Educação em Computação (pp. 01-11). Jataí, Goiás. SBC. doi:10.5753/educomp.2021.14466. [GS Search]
Alves, N. da C., Gresse von Wangenheim, C., Hauck, J. C. R., & Borgatto, A. F. (2020a, February).A large-scale evaluation of a rubric for the automatic assessment of algorithms and programming concepts. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 556-562). Association for Computing Machinery, New York, NY, USA, 556–562. doi:10.1145/3328778.3366840. [GS Search]
Alves, N. da C., von Wangenheim, C. G., Hauck, J. C. R., Borgatto, A. F., & Andrade, D. F. (2020b).An Item Response Theory Analysis of the Sequencing of Algorithms & Programming Concepts. Proceedings of International Conference on Computational Thinking Education (CoolThink@). Hong Kong: The Education University of Hong Kong. [GS Search]
De Ayala, R. J. (2009). The theory and practice of item response theory. Guilford Press. [GS Search]
Basili, V. R., Caldiera, G., & Rombach, H. D. (1994). The goal question metric approach. In Encyclopedia of Software Engineering, 528-532, John Wiley & Sons. [GS Search]
Bennedsen, J., & Caspersen, M. E. (2007). Failure Rates in Introductory Programming, ACM SIGCSE Bulletin, 39(2), 32-36. doi:10.1145/1272848.1272879. [GS Search]
Bennedsen, J. & Caspersen, M. E. (2019). Failure rates in introductory programming: 12 years later. ACM Inroads 10(2), 30–36. doi:10.1145/3324888. [GS Search]
Berges, M., & Hubwieser, P. (2015, June). Evaluation of Source Code with Item Response Theory. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (pp. 51-56), Association for Computing Machinery, New York, NY, USA. doi:10.1145/2729094.2742619. [GS Search]
Branch, R. M. (2010). Instructional Design: The ADDIE Approach. Springer. [GS Search]
Brennan, K. & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking. In Proceedings of the Annual Meeting of the American Educational Research Association, Vancouver, Canada (Vol. 1, p. 25). [GS Search]
Bruner, J. S. (1966). Toward a theory of instruction (Vol. 59). Harvard University Press. [GS Search]
Carlson, J. E., & von Davier, M. (2017). Item Response Theory. In Advancing Human Assessment (pp. 133-178), eds. Bennet & van Davier, Springer, Cham. [GS Search]
CAS. (2015). Computing at School. Retrieved October 29, 2021, from https://www.computingatschool.org.uk/
Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R Environment. Journal of Statistical Software, 48(6), 1–29. [GS Search]
CSTA. (2016). K-12 Computer Science Framework. Retrieved October 29, 2021, from https://k12cs.org/ [GS Search]
Dede, C. (1986). A review and synthesis of recent research in intelligent computer-assisted instruction. International Journal on Man-Machine Studies, 24(4), 329-353. doi:10.1016/S0020-7373(86)80050-5. [GS Search]
Fee, S. B. & Holland-Minkley, A. M. (2010). Teaching computer science through problems, not solutions. Computer Science Education, 20(2), 129-144. doi:10.1080/08993408.2010.486271. [GS Search]
Ferreira, M. N. F.; da Cruz Pinheiro, F.; Gresse Von Wangenheim,C.; Missfeldt Filho, R.; Hauck, J. C. R. Ensinando Design de Interface de Usuário de Aplicativos Móveis no Ensino Fundamental. Revista Brasileira de Informática na Educação, vol. 28, 2020. doi:10.5753/rbie.2020.28.0.48. [GS Search]
Franklin, D., Skifstad, G., Rolock, R., Mehrotra, I., Ding, V., Hansen, A., Weintrop, D., & Harlow, D. (2017, March). Using Upper-Elementary Student Performance to Understand Conceptual Sequencing in a Blocks-based Curriculum. In Proceedings of the ACM SIGCSE Technical Symposium on Computer Science Education (pp. 231-236). Association for Computing Machinery, New York, NY, USA, 231–236. doi:10.1145/3017680.3017760. [GS Search]
Goodrich, H. (1997). Understanding Rubrics. Educational Leadership, 54(4), 14–17. [GS Search]
Green, T. R. G. (1989) Cognitive dimensions of notations. In A. Sutcliffe and L. Macaulay (Eds.) People and Computers V. Cambridge, UK: Cambridge University Press, pp 443-460. [GS Search]
Gresse von Wangenheim, C., Hauck, J. C. R., Demetrio, M. F., Pelle, R., da C. Alves, N. da C., Barbosa, H., Azevedo, L. F .(2018). CodeMaster – Automatic Assessment and Grading of App Inventor and Snap! Programs. Informatics in Education, 17(1), 117-150. doi:10.15388/infedu.2018.08. [GS Search]
Grover, S., & Basu, S. (2017, March). Measuring student learning in introductory block-based programming: Examining misconceptions of loops, variables, and Boolean Logic. In Proceedings of the ACM SIGCSE Technical Symposium on Computer Science Education (pp. 267–272). Association for Computing Machinery. New York, NY, USA. doi:10.1145/3017680.3017723. [GS Search]
Grover, S., Basu, S., & Schank, P. (2018, February). What We Can Learn About Student Learning From Open-Ended Programming Projects in Middle School Computer Science. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education (pp. 999-1004). Association for Computing Machinery, NY, USA. doi:10.1145/3159450.3159522. [GS Search]
Grover, S. & R. Pea. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. doi:10.3102/0013189X12463051. [GS Search]
Hubwieser, P., Giannakos, M. N., Berges, M., Brinda, T., Diethelm, I., Magenheim, J., Pal, Y., Jackova, J., & Jasute, E. (2015). A Global Snapshot of Computer Science Education in K-12 Schools. In Proceedings of the ITiCSE on Working Group Reports(pp. 65-83). Association for Computing Machinery, New York, NY, USA, 65–83. doi:10.1145/2858796.2858799. [GS Search]
Khosravi, H., Sadiq, S., & Gasevic, D. (2020, February). Development and Adoption of an Adaptive Learning System. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education (pp. 58-64), Association for Computing Machinery, New York, NY, USA, 58–64. doi:10.1145/3328778.3366900. [GS Search]
Ko, J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., Rosson, M. B., Rothermel, G., Shaw, M., & Wiedenbeck, S. (2011). The state of the art in end-user software engineering. ACM Computing Surveys, 43(3), Article 21. doi:10.1145/1922649.1922658. [GS Search]
Kramer, M., Tobinski, D. A., & Brinda, T. (2016, November). On the way to a test instrument for object-oriented programming competencies. In Proceedings of the 16th Koli Calling International Conference on Computing Education Research (pp. 145-149). Association for Computing Machinery, New York, NY, USA. doi:10.1145/2999541.2999544. [GS Search]
Krugel, J., Hubwieser, P., Goedicke, M., Striewe, M., Talbot, M., Olbricht, C., Schypula, M. & Zettler, S. (2020, April). Automated Measurement of Competencies and Generation of Feedback in Object-Oriented Programming Courses. In Proceedings of the IEEE Global Engineering Education Conference (EDUCON) (pp. 329-338), Porto, Portugal. doi:10.1109/EDUCON45650.2020.9125323. [GS Search]
LEGO. (2018). Lego Education Documentation. Retrieved October 29, 2021, from: https://makecode.mindstorms.com/types/string
Li, I., Turbak, F., & Mustafaraj, E. (2017, October). Calls of the Wild:Exploring Procedural Abstraction in App Inventor. In Proceedings of the IEEE Blocks and Beyond Workshop (pp. 79-86). Raleigh, NC, USA. doi:10.1109/BLOCKS.2017.8120417. [GS Search]
Lye, S. Y. & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Computers in Human Behavior, 41, C, 51–61. doi:10.1016/j.chb.2014.09.012. [GS Search]
Lytle, N., Cateté, V., Boulden, D., Dong, Y., Houchins, J., Milliken, A., Isvik, A., Bounajim, D., Wiebe, E., & Barnes, T. (2019, July). Use, Modify, Create: Comparing Computational Thinking Lesson Progressions for STEM Classes. In Proceedings of the ACM Conference on Innovation and Technology in Computer Science Education (pp. 395-401). Association for Computing Machinery, New York, NY, USA. doi:10.1145/3304221.3319786. [GS Search]
MIT. (2020). MIT App Inventor. About us. Retrieved October 29, 2021, from http://appinventor.mit.edu/explore/about-us.html
Moreno-León, J., Robles, G., & Román-González, M. (2020). Towards data-driven learning paths to develop computational thinking with Scratch. IEEE Transactions on Emerging Topics in Computing, 8(1), 193-205. doi:10.1109/TETC.2017.2734818. [GS Search]
Morrison, G. R., Ross, S. M., & Kemp, J. E. (2010). Designing Effective Instruction, 6th ed. John Wiley & Sons. [GS Search]
Mustafaraj, E., Turbak, F., & Svanberg, M. (2017, May). Identifying Original Projects in App Inventor. In Proceedings of the 30th International Florida Artificial Intelligence Research Society Conference, Marco Island, FL, USA. 567-572. [GS Search]
Papadakis, S., Kalogiannakis, M., Orfanakis, V., & Zaranis, N. (2017). The appropriateness of Scratch and App Inventor as educational environments for teaching introductory programming in primary and secondary education. International Journal of Web-Based Learning and Teaching Technologies, 12(4), 58-77. doi:10.4018/IJWLTT.2017100106. [GS Search]
Park, Y. & Shin, Y. (2019). Comparing the Effectiveness of Scratch and App Inventor with Regard to Learning Computational Thinking Concepts. Electronics, 8(11), 1269. doi:10.3390/electronics8111269. [GS Search]
van Patten, J., Chao, C. I., & Reigeluth, C. M. (1986). A review of strategies for sequencing and synthesizing instruction. Review of Educational Research, 56(4), 437-471. doi:10.3102/00346543056004437. [GS Search]
Patton, E. W., Tissenbaum, M., & Harunani, F. (2019). MIT App Inventor: Objectives, Design, and Development. Kong SC., Abelson H. (eds), Computational Thinking Education(pp. 31-49). Springer, Singapore. doi:10.1007/978-981-13-6528-7_3. [GS Search]
Pea, R. D. & Kurland, D. M. (1984). On the cognitive effects of learning computer programming. New Ideas in Psychology, 2(2), 137–168. doi:10.1016/0732-118X(84)90018-7. [GS Search]
Piech, C., Sahami, M., Koller, D., Cooper, S., & Blikstein, P. (2012, February). Modeling how students learn to program. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (pp. 153-160). Association for Computing Machinery, New York, NY, USA, 153–160. doi:10.1145/2157136.2157182. [GS Search]
Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of educational statistics, Sage Publications CA: Thousand Oaks, 4(3), 207–230. [GS Search]
Reigeluth, C. M. (1999). The Elaboration theory: Guidance for scope and sequence decision. In C. Reigeluth (ed.) Instructional-Design Theories and Models (vol.II), Erlbaum Associates. [GS Search]
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., & Kafai, Y. (2009). Scratch: programming for all. Communications of the ACM 52(11), 60–67. doi:10.1145/1592761.1592779. [GS Search]
Rich, K. M., Strickland, C. T., Binkowski, T. A., Moran, T. A., & Franklin, D. (2017). K-8 learning trajectories derived from research literature: Sequence, repetition, conditionals. In Proceedings of the ACM Conference on International Computing Education Research (pp. 182-190), Association for Computing Machinery, New York, NY, USA. doi:10.1145/3105726.3106166. [GS Search]
Rich, K. M., Strickland, C. T., Binkowski, T. A., & Franklin, D. (2018). Decomposition: A K-8 computational thinking learning trajectory. In Proceedings of the 2018 ACM Conference on International Computing Education Research (pp. 124-132), Association for Computing Machinery, New York, NY, USA. doi:10.1145/3230977.3230979. [GS Search]
Rich, K. M., Strickland, C. T., Binkowski, T. A., & Franklin, D. (2019). A K-8 debugging learning trajectory derived from research literature. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education (pp. 745-751), Association for Computing Machinery, New York, NY, USA. doi:10.1145/3287324.3287396. [GS Search]
Rogalski, J. & Samurçay, R. (1990). Acquisition of programming knowledge and skills. In J.M.Hoc, T.R.G. Green, R. Samurçay, & D.J. Gillmore (eds.), Psychology of programming, Academic Press. doi:10.1016/B978-0-12-350772-3.50015-X. [GS Search]
Samejima, F. A. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, 34(4), 2-17. [GS Search]
Santos, J. S., Andrade, W. L., Brunet, J., & Araujo Melo, M. R. (2020, October). A Systematic Literature Review of Methodology of Learning Evaluation Based on Item Response Theory in the Context of Programming Teaching. In Proceedings of the IEEE Frontiers in Education Conference (FIE) (pp. 1-9). Uppsala, Sweden. doi:10.1109/FIE44824.2020.9274068. [GS Search]
SBC. (2018). Brazilian Computer Society Guidelines for Computing Education in K-12 (Diretrizes para ensino de Computação na Educação Básica).Retrieved October 29, 2021, from https://www.sbc.org.br/educacao/diretoria-de-educacao-basica
Seiter, K. & Foreman, B. (2013, August). Modeling the learning progressions of computational thinking of primary grade students. In Proceedings of the 9th Annual International ACM Conference on International Computing Education Research (pp. 59-66), Association for Computing Machinery, New York, NY, USA, 59–66. doi: 10.1145/2493394.2493403. [GS Search]
Sherman, M., & Martin, F. (2015). The assessment of mobile computational thinking. Journal of Computing Sciences in Colleges, 30(6), 53–59. [GS Search]
Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251–296. doi:10.1023/A:1022193728205. [GS Search]
Tissenbaum, M., Sheldon, J., & Abelson, H. (2019). From Computational Thinking to Computational Action. Communications of the ACM, 62(3), 34-36. doi:10.1145/3265747. [GS Search]
Turbak, F., Sherman, M., Martin, F., Wolber, D., & Pokress, S. C. (2014). Events First Programming in App Inventor. Journal of Computing Sciences in Colleges, 29(6), 81-89. [GS Search]
Vainas, O., Ben-David, Y., Gilad-Bachrach, R., Ronen, M., Bar-Ilan, O., Shillo, R., Lukin, G., Sitton, D. (2019). Staying in The Zone: Sequencing Content in Classrooms Based on The Zone of Proximal Development. In Proceedings of the 12th International Conference on Educational Data Mining (pp. 659-662), Montreal, Canada. [GS Search]
Webb, M., Davis, N., Bell, T., Katz, Y. J., Reynolds, N., Chambers, D. P., & Sysło, M. M. (2017). Computer science in K-12 school curricula of the 2lst century: Why, what and when?. Education and Information Technologies, 22(2), 445–468. doi:10.1007/s10639-016-9493-x. [GS Search]
Weintrop, D. (2019). Block-based Programming in Computer Science Education. Communications of the ACM, 62(8), 22-25. doi: 10.1145/3341221. [GS Search]
Wolber, D., Abelson, H., & Friedman, M. (2014). Democratizing Computing with App Inventor. GetMobile: Mobile Computing and Communications. 18(4), 53–58. doi:10.1145/2721914.2721935. [GS Search]
Xie, B. & Abelson, H. (2016, September). Skill progression in MIT app inventor. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (pp. 213-217). Cambridge, GB. doi:10.1109/VLHCC.2016.7739687. [GS Search]
Xie, B., Shabir, I., & Abelson, H. (2015). Measuring the programmatic sophistication of app inventor projects grouped by functionality. Retrieved October 29, 2021 from https://dspace.mit.edu/bitstream/handle/1721.1/98913/measuring%20usability.pdf [GS Search]
Archivos adicionales
Published
Cómo citar
Issue
Section
Licencia
Derechos de autor 2021 Nathalia da Cruz Alves, Christiane Gresse von Wangenheim, Jean Carlo Rossa Hauck, Adriano Ferreti Borgatto
Esta obra está bajo una licencia internacional Creative Commons Atribución-NoComercial-SinDerivadas 4.0.
