Deep learning for early performance prediction of introductory programming students: a comparative and explanatory study
DOI:
https://doi.org/10.5753/rbie.2020.28.0.723Keywords:
Online judges, Deep Learning, CS1, introductory programming, predictionAbstract
Introductory programming may be complex for many students. Moreover, there is a high failure and dropout rate in these courses. A potential way to tackle this problem is to predict student performance at an early stage, as it facilitates human-AI collaboration towards prescriptive analytics, where the instructors/monitors will be told how to intervene and support students - where early intervention is crucial. However, the literature states that there is no reliable predictor yet for programming students’ performance, since even large-scale analysis of multiple features have resulted in only limited predictive power. Notice that Deep Learning (DL) can provide high-quality results for huge amount of data and complex problems. In this sense, we employed DL for early prediction of students’ performance using data collected in the very first two weeks from introductory programming courses offered for a total of 2058 students during 6 semesters (longitudinal study). We compared our results with the state-of-the-art, an Evolutionary Algorithm (EA) that automatic creates and optimises machine learning pipelines. Our DL model achieved an average accuracy of 82.5%, which is statistically superior to the model constructed and optimised by the EA (p-value << 0.05 even with Bonferroni correction). In addition, we also adapted the DL model in a stacking ensemble for continuous prediction purposes. As a result, our regression model explained ~62% of the final grade variance. In closing, we also provide results on the interpretation of our regression model to understand the leading factors of success and failure in introductory programming.
Downloads
Referências
Aguiar, G. C. L., Pereira, F. D (2018). Uma Abordagem Data-Driven Para Predição Precoce Da Evasão Em Turmas De Programação Que Utilizam Juízes Online. Graduação em Ciência da Computação, Universidade Federal de Roraima, Boa Vista. doi: 10.13140/RG.2.2.26694.50248. [GS Search]
Ahadi, A., Vihavainen, V., Lister, R. (2016). On the Number of Attempts Students Made on Some Online Programming Exercises During Semester and their Subsequent Performance on Final Exam Questions. ACM conference on Innovation and Technology in Computer Science Education, 218–223. doi: 10.1145/2899415.2899452. [GS Search]
Alamri, A., Alshehri, M., Cristea, A., Pereira, F. D., Oliveira, E., Shi, L., & Stewart, C. (2019, June). Predicting MOOCs dropout using only two easily obtainable features from the first week’s activities. In International Conference on Intelligent Tutoring Systems (pp. 163-173). Springer, Cham. doi: 10.1007/978-3-030-22244-4_20. [GS Search]
Aljohani, T., Pereira, F. D., Cristea, A. I., & Oliveira, E. (2020, June). Prediction of Users’ Professional Profile in MOOCs Only by Utilising Learners’ Written Texts. In International Conference on Intelligent Tutoring Systems (pp. 163-173). Springer, Cham. doi: 10.1007/978-3-030-49663-0_20. [GS Search]
Amra, I. A. A., & Maghari, A. Y. (2017, May). Students performance prediction using KNN and Naïve Bayesian. In 2017 8th International Conference on Information Technology (ICIT) (pp. 909-913). IEEE. doi: 10.1109/ICITECH.2017.8079967. [GS Search]
Auvinen, T. (2015). Harmful study habits in online learning environments with automatic assessment. In Learning and Teaching in Computing and Engineering (LaTiCE), International Conference on. IEEE, 50–57. doi: 10.1109/LaTiCE.2015.31. . [GS Search]
Baker, R. S. J., & Siemens, G. (2014). Educational data mining and learning analytics. In R. K. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences, 2nd edn. (pp. 253– 274). Cambridge, UK: Cambridge University Press. doi: 10.1007/978-1-4614-3305-7_4. [GS Search]
Batista, G. E., Prati, R. C., & Monard, M. C. (2004). A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD explorations newsletter, 6(1), 20-29. doi: 10.1145/1007730.1007735
Blikstein, P.; Worsley, M.; Piech, C.; Sahami, M.; Cooper, S.; Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, v. 23, n. 4, p. 561–599. ISSN 1050-8406. doi: 10.1080/10508406.2014.954750. [GS Search]
Carter, A., Hundhausen, C., Olivares, D. (2019). Leveraging the Integrated Development Environment for Learning Analytics. In S. Fincher & A. Robins (Eds.), The Cambridge Handbook of Computing Education Research (Cambridge Handbooks in Psychology, pp. 679-706). Cambridge: Cambridge University Press. doi: 10.1017/9781108654555. [GS Search]
Castro-Wunsch, K., Ahadi, A., Petersen, A. (2017). Evaluating Neural Networks as a Method for Identifying Students in Need of Assistance. In Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education. ACM, 111–116. doi: 10.1145/3017680.3017792. [GS Search]
Costa, E. B., Fonseca, B., Santana, M. A., de Araújo, F. F., Rego, J. (2017). Evaluating the effectiveness of educational data mining techniques for early prediction of students' academic failure in introductory programming courses. Computers in Human Behavior, 73, 247-256. doi: 10.1016/j.chb.2017.01.047. [GS Search]
Dwan, F., Oliveira, E., Fernandes, D. (2017). Predição de zona de aprendizagem de alunos de introdução à programação em ambientes de correção automática de código. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). Vol. 28. No. 1. doi: 10.5753/cbie.sbie.2017.1507. [GS Search]
Edwards, S.H., Snyder, J., Pérez-Quiñones, M.A., Allevato, A., Kim, D., Tretola, B. (2009). Comparing effective and ineffective behaviors of student programmers. in: Proceedings of the fifth international workshop on Computing education research workshop, ACM. pp. 3– 14. doi: 10.1145/1584322.1584325. [GS Search]
Elarde, J. (2016). Toward improving introductory programming student course success rates: experiences with a modified cohort model to student success sessions. Journal of Computing Sciences in Colleges, 32(2), 113– 119. doi: 10.5555/3015063.3015081. [GS Search]
Estey, A. and Coady, Y. (2016). Can Interaction Patterns with Supplemental Study Tools Predict Outcomes in CS1? Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education – ITiCSE ’16, 236–241. doi: 10.1145/2899415.2899428. [GS Search]
Fonseca, S., Oliveira, E., Pereira, F., Fernandes, D., & de Carvalho, L. S. G. (2019). Adaptação de um método preditivo para inferir o desempenho de alunos de programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). Vol. 30, No. 1, p. 1651. doi: 10.5753/cbie.sbie.2019.1651. [GS Search]
Fonseca, S. C., Pereira, F. D., Oliveira, E. H., Oliveira, D. B. F., Carvalho, L. S. G., Cristea, A. I. (2020). Automatic Subject-based Contextualisation of Programming Assignment Lists. In the Proceedings of The 13th International Conference on Educational Data Mining (EDM 2020). Found on: http://dro.dur.ac.uk/31164/1/31164.pdf. [GS Search]
Galvão, L., Fernandes, D., and Gadelha, B. (2016). Juiz online como ferramenta de apoio a uma metodologia de ensino híbrido em programação. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). Volume 27, page 140. doi: 10.5753/cbie.sbie.2016.140. [GS Search]
Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. O'Reilly Media. Found on: https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/. [GS Search]
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Liao, S. N. (2018, July). Predicting academic performance: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (pp. 175-199). doi: 10.1145/3293881.3295783. [GS Search]
Hernández-Blanco, A., Herrera-Flores, B., Tomás, D., & Navarro-Colorado, B. (2019). A systematic review of deep learning approaches to educational data mining. Complexity, 2019. ISSN: 1076-2787. doi: 10.1155/2019/1306039. [GS Search]
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R. R. (2012). Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580. Found on: https://arxiv.org/abs/1207.0580. [GS Search]
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S., Isohanni, E., Korhonen, A., Petersen, A., Rivers, K., Rubio, M., Sheard, J., Skupas, B., Spacco, J., Szabo, C., Toll, D. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. In Proceedings of the 2015 ITiCSE on working group reports. In Proceedings of the 2015 ITiCSE on Working Group Reports (pp. 41-63). doi: 10.1145/2858796.2858798. [GS Search]
Jadud, M. C. (2006) Methods and tools for exploring novice compilation behaviour. In Proceedings of the Second International Workshop on Computing Education Research. ACM, 73–84. doi: 10.1145/1151588.1151600. [GS Search]
Freitas, H. B. & Pereira, F. D. (2019). Recomendação de Problemas em Juízes Onlines Utilizando Técnicas de Processamento de Linguagem Natural e Análise Dirigida aos Dados. Graduação em Ciência da Computação, Universidade Federal de Roraima, Boa Vista. doi: 10.13140/RG.2.2.24254.92486. [GS Search]
Junior, F. P. & Pereira, F. D. (2019). Identificação Automática de Categorias de Problemas em Juízes Online. Graduação em Ciência da Computação, Universidade Federal de Roraima, Boa Vista. doi: 10.13140/RG.2.2.36760.83201. [GS Search]
Kazerouni, A. M., Edwards, S. H., Hall, T. S., & Shaffer, C. A. (2017, June). DevEventTracker: Tracking development events to assess incremental development and procrastination. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (pp. 104-109). doi: 10.1145/3059009.3059050. [GS Search]
Kingma, D. P., & Ba, J. Adam (2014). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), arXiv preprint arXiv (Vol. 1412). Found on: https://arxiv.org/abs/1412.6980. [GS Search]
Leinonen, J., Longi, K., Klami, A., & Vihavainen, A. (2016). Automatic inference of programming performance and experience from typing patterns. Proceedings of the 47th ACM Technical Symposium on Computing Science Education. ACM. doi: 10.1145/2839509.2844612. [GS Search]
Luxton-Reilly, A., Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M.J., Sheard, J., Szabo, C. 2018. Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE 2018 Companion). Association for Computing Machinery, New York, NY, USA, 55–106. doi: 10.1145/3293881.3295779. [GS Search]
Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Baju, B., Shahrzad, H, Navruzyan, A., Duffy, N, Hodjat, B. (2019). Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain Computing (pp. 293-312). Academic Press. doi: 10.1016/B978-0-12-815480-9.00015-3. [GS Search]
Montgomery, D. C., Peck, E. A., & Vining, G. G. (2012). Introduction to linear regression analysis (Vol. 821). John Wiley & Sons. Found on: [Link]. [GS Search]
Newman, M. E. (2005). Power laws, Pareto distributions and Zipf's law. Contemporary physics, 46(5), 323-351. doi: 10.1080/00107510500052444. [GS Search]
Otero, J., Junco, L., Suarez, R., Palacios, A., Couso, I. Sanchez, L. (2016). Finding informative code metrics under uncertainty for predicting the pass rate of online courses. In Information Sciences. pp. 373, 42–56. doi: 10.1016/j.ins.2016.08.090. [GS Search]
Pedregosa, F., Gaël V., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of machine learning research 12, no.: 2825-2830. Found on: http://www.jmlr.org/papers/volume12/pedregosa11a/pedregosa11a.pdf. [GS Search]
Pereira, F. D., Oliveira, E. H. T., e Oliveira, D. F. B. (2018). Uso de um método preditivo para inferir a zona de aprendizagem de alunos de programação em um ambiente de correção automática de código. Mestrado em informática, Universidade Federal do Amazonas, Manaus. Found on: [Link]. [GS Search]
Pereira, F. D., Oliveira, E. H., Fernandes, D., & Cristea, A. (2019a). Early performance prediction for CS1 course students using a combination of machine learning and an evolutionary algorithm. In 2019 IEEE 19th International Conference on Advanced Learning Technologies (ICALT) (Vol. 2161, pp. 183-184). IEEE. doi: 10.1109/ICALT.2019.00066. [GS Search]
Pereira, F., Oliveira, E., Fernandes, D., Junior, H., & de Carvalho, L. S. G. (2019b, November). Otimização e automação da predição precoce do desempenho de alunos que utilizam juízes online: uma abordagem com algoritmo genético. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE). Vol. 30, No. 1, p. 1451. doi: 10.5753/cbie.sbie.2019.1451. [GS Search]
Pereira, F. D., Oliveira, E., Cristea, A., Fernandes, D., Silva, L., Aguiar, G., ... & Alshehri, M. (2019c, June). Early dropout prediction for programming courses supported by online judges. In International Conference on Artificial Intelligence in Education (pp. 67-72). Springer, Cham. doi: 10.1007/978-3-030-23207-8_13. [GS Search]
Pereira, F. D., Oliveira, E. H., Oliveira, D. B. F., Cristea, A. I., Carvalho, L. S. G., Fonseca, S. C., Toda, A., Isotani, S. (2020a). Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British journal of educational technology. ISSN:1467-8535. doi: 10.1111/bjet.12953. [GS Search]
Pereira, F. D., Toda, A., Oliveira, H. T., Cristea, A. I., Isotani, I., Laranjeira, D., Almeida, A. P., Mendonça, J. (2020b). Can we use Gamification to predict students' performance? A case study supported by an online judge. In International Conference on Intelligent Tutoring Systems (pp. 259-269). Springer, Cham. doi: 10.1007/978-3-030-49663-0_30. [GS Search]
Quille, K., and Bergin, S. (2018). Programming: predicting student success early in CS1. a re-validation and replication study. Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education. doi: 10.1145/3197091.3197101. [GS Search]
Robins, A. V. (2019). Novice programmers and introductory programming. In S. Fincher & A. Robins (Eds.), The Cambridge Handbook of Computing Education Research (Cambridge Handbooks in Psychology, pp. 327-376). Cambridge: Cambridge University Press. doi: 10.1017/9781108654555. [GS Search]
Romero, C., Ventura, S. (2019). Guest editorial: Special issue on early prediction and supporting of learning performance. IEEE Transactions on Learning Technologies, 12(2), 145-147. doi: 10.1109/TLT.2019.2908106. [GS Search]
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1), 1929-1958. doi: 10.1109/TLT.2019.2908106. [GS Search]
Tomek, I. (1976). Two Modifications of CNN. IEEE Transactions on Systems Man and Communications SMC-6, 769–772. Found on: [Link]. [GS Search]
Vihavainen, A., Luukkainen, M., & Ihantola, P. (2014). Analysis of source code snapshot granularity levels. In Proceedings of the 15th annual conference on information technology education (pp. 21–26). doi: 10.1145/2656450.2656473. [GS Search]
Vihavainen, A. (2013). Predicting students’ performance in an introductory programming course using data from students’ own programming process. Proceedings - 2013 IEEE 13th International Conference on Advanced Learning Technologies, ICALT 2013, 498–499. doi: 10.1109/ICALT.2013.161. [GS Search]
Wasik, S., Antczak, M., Badura, J., Laskowski, A., & Sternal, T. (2018). A survey on online judge systems and their applications. ACM Computing Surveys (CSUR), 51(1), 1-34. doi: 10.1145/3143560. [GS Search]
Watson, C. and Li, F. WB. (2014). Failure rates in introductory programming revisited. In Proceedings of the 2014 conference on Innovation & technology in computer science education. ACM, 39–44. doi: 10.1145/2591708.2591749. [GS Search]
Watson, C., Li, F. W., Godwin, J. L. (2013). Predicting performance in an introductory programming course by logging and analyzing student programming behavior. In 2013 IEEE 13th international conference on advanced learning technologies (pp. 319-323). IEEE. doi: 10.1109/ICALT.2013.99. [GS Search]
Wolpert, D. H. (1992). Stacked generalization. Neural networks, 5(2), 241-259. doi: 10.1016/S0893-6080(05)80023-1. [GS Search]
Yadin, A. (2013). Using unique assignments for reducing the bimodal grade distribution. ACM Inroads, 4(1), 38– 42. doi: 10.1145/2432596.2432612. [GS Search]
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2020 Filipe Dwan Pereira, Samuel C. Fonseca, Elaine H. T. Oliveira, David B. F. Oliveira, Alexandra I. Cristea, Leandro S. G. Carvalho
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
