Evaluating LLM-Assisted Multiple-Choice Question Generation in Requirements Engineering: Prompting Strategies and Professor Validation
DOI:
https://doi.org/10.5753/rbie.2026.8196Keywords:
Large Language Models, Multiple-Choice Questions, Prompt Engineering, Automated Question Generation, Requirements Engineering, Educational Assessment, Human EvaluationAbstract
Large Language Models (LLMs) are increasingly being explored for their potential to automate the generation of multiple-choice questions (MCQs). However, the quality of these questions is still influenced by various factors, including pedagogical, conceptual, and linguistic considerations. This article presents the outcomes of a new experimental round conducted in 2026, focusing on MCQ generation within the Requirements Engineering domain, and builds upon a previous study published in 2025. The current study uses the same domain, Prompt Engineering techniques, evaluation criteria, and validation protocol, and merges automatic evaluations by LLMs with assessments from professors. In total, 200 questions were generated and analyzed. The results indicate that automatic evaluations and human assessments often produce differing judgments, with LLMs generally assigning higher scores than their human counterparts. Additionally, the findings reveal that Prompt Engineering techniques and generator models exhibit different behaviors across evaluators. Challenges related to the criteria of diversity and complexity persist in automated question generation. This study highlights the considerable potential of LLMs to enhance assessment design, while also emphasizing the essential role of human curation and the need for continuous validation in the rapidly evolving landscape of LLM ecosystems.
Downloads
Referências
Al Shuraiqi, S., Alzaabi, A., & Aal Abdulsalam, A. K. (2026). Prompt engineering strategies for generating medical case-based MCQs with large language models: A multi-model comparative study. Machine Learning and Knowledge Extraction, 8(2), 41. https://doi.org/10.3390/make8020041 [GS Search].
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
Angermeir, F., Amougou, M., Kreitz, M., Bauer, A., Linhuber, M., Fucci, D., Moyón Constante, F., Méndez, D., & Gorschek, T. (2025). Reflections on the reproducibility of commercial LLM performance in empirical software engineering studies. https://doi.org/10.48550/arXiv.2510.25506 [GS Search].
Baltes, S., Angermeir, F., Arora, C., Muñoz Barón, M., Chen, C., Böhme, L., Calefato, F., Ernst, N. A., Falessi, D., Fitzgerald, B., Fucci, D., Kalinowski, M., Lambiase, S., Russo, D., Lungu, M., Prechelt, L., Ralph, P., Treude, C., & Wagner, S. (2025). Guidelines for empirical studies in software engineering involving large language models. https://doi.org/10.48550/arXiv.2508.15503 [GS Search].
Bhowmick, A., Jagmohan, A., Vempaty, A., Dey, P., Hall, L., Hartman, J., Kokku, R., & Maheshwari, H. (2023). Automating question generation from educational text. https://doi.org/10.48550/arXiv.2309.15004 [GS Search].
Biancini, G., Ferrato, A., & Limongelli, C. (2024). Multiple-choice question generation using large language models: Methodology and educator insights. Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization. https://doi.org/10.1145/3631700.3665233 [GS Search].
Bitew, S. K., Deleu, J., Develder, C., & Demeester, T. (2023). Distractor generation for multiple-choice questions with predictive prompting and large language models. https://doi.org/10.48550/arXiv.2307.16338 [GS Search].
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., ... Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. [GS Search].
Case, S. M., & Swanson, D. B. (2016). Constructing written test questions for the basic and clinical sciences. National Board of Medical Examiners. [Link] [GS Search].
Chico, V. J. S., Regino, A. G., Bonacin, R., & Reis, J. (2025). LLM-based automatic generation of multiple-choice questions with meaningful distractors. Anais do XXXVI Simpósio Brasileiro de Informática na Educação, 813–827. https://doi.org/10.5753/sbie.2025.12675 [GS Search].
Cho, Y., AlMamlook, R. E., & Gharaibeh, T. (2024). A systematic review of knowledge tracing and large language models in education: Opportunities, issues, and future research. arXiv preprint arXiv:2412.09248. https://doi.org/10.48550/arXiv.2412.09248 [GS Search].
Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods (2nd ed.). McGraw-Hill. [GS Search].
Elkins, S., Kochmar, E., Cheung, J. C. K., & Serban, I. (2024). How teachers can use large language models and Bloom's taxonomy to create educational quizzes. https://doi.org/10.48550/arXiv.2401.05914 [GS Search].
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333. https://doi.org/10.1207/S15324818AME1503_5 [GS Search].
Hang, C. N., Tan, C. W., & Yu, P.-D. (2024). MCQGen: A large language model-driven MCQ generator for personalized learning. IEEE Access, 12, 102261–102273. https://doi.org/10.1109/ACCESS.2024.3420709 [GS Search].
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Computing Surveys. https://doi.org/10.1145/3703155 [GS Search].
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., ... Kasneci, G. (2023). ChatGPT for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274 [GS Search].
Li, K., & Zhang, Y. (2024). Planning first, question second: An LLM-guided method for controllable question generation. Findings of the Association for Computational Linguistics: ACL 2024, 4715–4729. https://doi.org/10.18653/v1/2024.findings-acl.280 [GS Search].
Maity, S., Deroy, A., & Sarkar, S. (2023). Harnessing the power of prompt-based techniques for generating school-level questions using large language models. Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation, 30–39. https://doi.org/10.1145/3632754.3632755 [GS Search].
Mucciaccia, S. S., Paixão, T. M., Mutz, F. W., Badue, C. S., Souza, A. F., & Oliveira-Santos, T. (2025). Automatic multiple-choice question generation and evaluation systems based on LLM: A study case with university resolutions. Proceedings of the 31st International Conference on Computational Linguistics, 2246–2260. [Link] [GS Search].
Quincozes, C. B., Molinos, D., Araújo, R. D., Quincozes, S., & Guedes, G. T. A. (2025). Engenharia de prompt para a geração automatizada de questões assistida por LLMs: Uma análise comparativa. Anais do XXXVI Simpósio Brasileiro de Informática na Educação, 1348–1361. https://doi.org/10.5753/sbie.2025.12891 [GS Search].
Scaria, N., Chenna, S. D., & Subramani, D. (2024a). Automated educational question generation at different Bloom's skill levels using large language models: Strategies and evaluation. Artificial Intelligence in Education, 14830, 165–179. https://doi.org/10.1007/978-3-031-64299-9_12 [GS Search].
Scaria, N., Chenna, S. D., & Subramani, D. (2024b). How good are modern LLMs in generating relevant and high-quality questions at different Bloom's skill levels for Indian high school social science curriculum? Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), 1–10. [Link] [GS Search].
Shu, C., Yao, N., Chen, Y., Wijeratne, V., Ma, L., Loo, J., Chai, K. K., Alam, A. S., & Abuelmaatti, A. (2025). AI-assisted multiple-choice questions generation with multimodal large language models in engineering higher education. 2025 IEEE Global Engineering Education Conference (EDUCON), 1–9. https://doi.org/10.1109/EDUCON62633.2025.11016449 [GS Search].
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023). Generating multiple choice questions for computing courses using large language models. 2023 IEEE Frontiers in Education Conference (FIE), 1–8. https://doi.org/10.1109/FIE58773.2023.10342898 [GS Search].
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824–24837. [GS Search].
White, J., Fu, Q., Hays, S., Sandborn, S., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023). A prompt pattern catalog to enhance prompt engineering with ChatGPT. arXiv preprint arXiv:2302.11382. https://doi.org/10.48550/arXiv.2302.11382 [GS Search].
Xu, H., Gan, W., Qi, Z., Wu, J., & Yu, P. S. (2024). Large language models for education: A survey. arXiv preprint arXiv:2405.13001. https://doi.org/10.48550/arXiv.2405.13001 [GS Search].
Yao, Z., Parashar, A., Zhou, H., Jang, W. S., Ouyang, F., Yang, Z., & Yu, H. (2025). MCQG-SRefine: Multiple choice question generation and evaluation with iterative self-critique, correction, and comparison feedback. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 10728–10777. https://doi.org/10.18653/v1/2025.naacl-long.538 [GS Search].
Arquivos adicionais
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2026 Camilla Borchhardt Quincozes, Diego Nunes Molinos, Rafael Dias Araújo, Silvio Ereno Quincozes, Gillianes Thorwald Araujo Guedes
Este trabalho está licenciado sob uma licença Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
