Multistage Adaptive Testing in ENEM
DOI:
https://doi.org/10.5753/rbie.2023.2529Keywords:
Multistage Adaptive Testing, Item Response Theory, Computerized Adaptive TestingAbstract
International student assessment tests in recent years have changed their structures to implement the adaptive format. The digital ENEM makes it possible to restructure the test by adhering to adaptive tests. This article proposes an adaptive multistage test (MST) for the ENEM test based on the 2019 edition in the area of Mathematics. The ENEM items were analyzed through the Item Response Theory, and those items were used to build the MST modules and stages. The routing between the modules was defined by a study of the optimal cutoff point for the estimated latent trait, exhaustively tested to find the one that brought the best result compared to the complete exam. Ultimately, a multistage test architecture with specific cutoff values for routing was proposed. It was found that the adaptive test reduced the number of test items by 44.4%, and the estimation of skills was kept close to the estimation with the complete test. It was also noted that the ENEM is aimed at evaluating higher levels on the skill scale, making the estimation of skills impaired for less proficient individuals.
Downloads
References
Andrade, D. F., Tavares, H. R., & Valle, R. C. (2000). Teoria da resposta ao item: conceitos e aplicações. ABE, São Paulo. [GS Search]
Angoff, W.H. (1957). The “Equating” of non-parallel tests. Journal of Experimental Education, 25(3), pp. 241–247. doi: 10.1080/00220973.1957.11010574. [GS Search]
Avellar, Simone. Enem pode ser aplicado via computador a partir de 2016. O Globo, Rio de Janeiro, 16/12/2012. Seção Educação. Disponível em [Link].
Barbosa, M. E. F., & Fernandes, C. (2001). A escola brasileira faz diferença? Uma investigação dos efeitos da escola na proficiência em Matemática dos alunos da 4ª série. Em C. Franco (org), Promoção, ciclos e avaliação educacional. ArtMed, Curitiba. [GS Search]
Bennett, R. E. (2015). The Changing Nature of Educational Assessment. Review of Research in Education, 39. doi: 10.3102/0091732X14554179. [GS Search]
Birnbaum, A. (1968). Some Latent Trait Models and Their Use in Inferring an Examinee’s Ability. In: Lord, F.M. and Novick, M.R., Eds., Statistical Theories of Mental Test Scores, Addison-Wesley, Reading, pp. 397-479. [GS Search]
Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), pp. 1–29. doi: 10.18637/jss.v048.i06. [GS Search]
Chalmers, R. P. (2016). Generating Adaptive and Non-Adaptive Test Interfaces for Multidimensional Item Response Theory Applications. Journal of Statistical Software, 71(5), pp. 1-38. doi: 10.18637/jss.v071.i05. [GS Search]
de Macedo, E. P. N. (2021). As diferentes fases do Enem: olhar o passado para pensar o futuro. Em Aberto, 34(112). doi: 10.24109/2176-6673.emaberto.34i112.4999. [GS Search]
Dias, A. C. F. (2019). Uma solução Bayesiana para se considerar a incerteza associada à calibração de itens na teoria de resposta ao item. Dissertação de Mestrado. Programa de Pós-Graduação em Estatística. Universidade Federal de Minas Gerais. Belo Horizonte. Defesa: 28/06/2019. [GS Search]
Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), pp. 1–35.
Fritts, B. E., & Marszalek, J. M. (2010). Computerized adaptive testing, anxiety levels, and gender differences. Social Psychology of Education: An International Journal, 13(3), 441–458. [GS Search]
Gierl, M.J., & Lai, H. (2013). Instructional Topics in Educational Measurement (ITEMS) Module: Using Automated Processes to Generate Test Items. Educational Measurement: Issues and Practice, 32(3), pp. 36–50. doi: 10.1111/emip.12018. [GS Search]
Grégoire, J., & Laveault, D. (2002). Introdução às Teorias dos Testes em Ciências Humanas. Porto, Portugal: Porto.
Hendrickson, A. (2007). An NCME Instructional Module on Multistage Testing. Educ. Meas. Issues Pract. 26 (2), 44–52. doi: 10.1111/j.1745-3992.2007.00093.x. [GS Search]
INEP. (2019). Edital ENEM 2019. Diário Oficial da União – Seção 3. Disponível em [Link].
INEP. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (n.d.). Exame Nacional do Ensino Médio (ENEM). Disponível em [Link].
Jatobá, V. M. G., Delgado, K. V., Farias, J. S., & Freire, V. (2018). Comparação de Regras de Seleção de Itens em Testes Adaptativos Computadorizados: um estudo de caso no ENEM. Anais do XXIX Simpósio Brasileiro de Informática na Educação. Pp. 1453-1462. doi: 10.5753/cbie.sbie.2018.1453. [GS Search]
Jatobá, V. M. G. (2019). Uma abordagem personalizada no processo de seleção de itens em Testes Adaptativos Computadorizados. Dissertação de Mestrado. Programa de Pós-Graduação em Sistemas de Informação. Universidade de São Paulo. São Paulo. Defesa: 08/10/2018. [GS Search]
Kirsch, I., & Lennon, M.L. (2017). PIAAC: a new design for a new era. Large-Scale Assessments in Education, 5(1), 11. doi: 10.1186/s40536-017-0046-6. [GS Search]
Lord, F.M. (1953). The relation of test score to the trait underlying the test. Educational and Psychological Measurement, 13(4), pp. 517–549. doi: 10.1177/001316445301300401. [GS Search]
Palermo, G. A., Silva, D. B. N., & Novellino, M. S. F. (2014). Fatores associados ao desempenho escolar: uma análise da proficiência em matemática dos alunos do 5º ano do ensino fundamental da rede municipal do Rio de Janeiro. Revista Brasileira de Estudos de População, 31(2). Pp. 367-394. doi: 10.1590/S0102-30982014000200007. [GS Search]
Piton-Gonçalves, J. (2020). Testes Adaptativos para o Enade: uma aplicação metodológica. Revista Meta: Avaliação. 12(36). doi: 10.22347/2175-2753v12i36.2735. [GS Search]
Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. Danish Institute for Educational Research. [GS Search]
Reckase, M.D. (1974). An interactive computer program for tailored testing based on the one-parameter logistic model. Behavior Research Methods & Instrumentation, 6(2), pp. 208–212. doi: 10.3758/BF03200330. [GS Search]
Ricarte, T. A. M., Curi, M., & von Davier, A. (2018). Modeling Accidental Mistakes in Multistage Testing: A Simulation Study. Springer Proceedings in Mathematics & Statistics, 233, pp. 55-65. doi: 10.1007/978-3-319-77249-3_5. [GS Search]
Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen Ways to Look at the Correlation Coefficient. The American Statistician, 42(1), pp. 59-66. doi: 10.1080/00031305.1988.10475524. [GS Search]
Silva, V. R., Curi, M. (2019). Academic English proficiency assessment using a computerized adaptive test. TEMA: Tendências em Matemática Aplicada e Computacional, 20(2). doi: 10.5540/tema.2019.020.02.0381. [GS Search]
Spenassato, D., Trierweiller, A. C., Andrade, D. F., & Bornia, A. C. (2016). Testes Adaptativos Computadorizados Aplicados em Avaliações Educacionais. Revista Brasileira de Informática na Educação, 24(2), pp. 1. ISSN 2317-6121. doi: 10.5753/rbie.2016.24.02.1. [GS Search]
Van Der Linden, W. J., & Glas, C. A. W. (2010). Elements of Adaptative Testing. Springer. Capítulo 18. Disponível em [Link].
Von Davier, A. (2017). Computational Psychometrics in Support of Collaborative Educational. Assessments Journal of Educational Measurement, 54(1), pp. 3–11. [GS Search]
Wainer, H., Kaplan, B., & Lewis, C. (1992). A Comparison of the Performance of Simulated Hierarchical and Linear Testlets. Journal of Educational Measurement, 1992, 29(3), pp. 243–251. doi: 10.1111/j.1745-3984.1992.tb00376.x. [GS Search]
Weiss, D.J. (1985). Adaptive Testing by Computer. Journal of Consulting and Clinical Psychology, 53(6), pp. 774–789. doi: 10.1037/0022-006X.53.6.774
Wickham, H. (2016). ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York. doi: 10.1080/15366367.2019.1565254
Willse, J. T., & Shu, Z. (2008). CTT: Classical Test Theory Functions. R package version 1.0.
Yamamoto, K., Shin, H. J., & Khorramdel, L. (2019). Introduction of multistage adaptive testing design in PISA 2018. OECD Education Working Paper Número 209. doi: 10.1787/19939019. [GS Search]
Yan, D., Lewis, C., & von Davier, A. (2014). Computerized multistage testing: Theory and applications. Capítulo 1. CRC. [GS Search]
Zheng, Y., Nozawa, Y., Gao, X., & Chang, H. (2012). Multistage adaptive testing for a large-scale classification test: The designs, automated heuristic assembly, and comparison with other testing modes. Report number: ACT Research Reports 2012-6Affiliation: ACT, Inc. [GS Search]
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Gabriel Couto Tabak, Jean Piton-Gonçalves, Thales Ricarte, Mariana Curi
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
