Calibration of Recommendation Systems with LLMs: Prompt Optimization to Balance Accuracy, Diversity, and Fairness
DOI:
https://doi.org/10.5753/reic.2026.8494Keywords:
Recommender Systems, LLM, Prompt Engineering, Popularity BiasAbstract
Recommender Systems (RSs) are essential in digital platforms but face challenges such as popularity bias, which reduces diversity and fairness. Calibration techniques aim to better align recommendations with user preferences, typically through post-processing. With the emergence of Large Language Models (LLMs), such as Llama, prompt engineering has become an alternative approach to personalization. This study investigates LLM-based calibration and compares it with traditional methods. We also evaluate different prompting strategies using metrics that encompass accuracy, diversity, and fairness. The results indicate that LLM-based prompting strategies can simultaneously improve personalization and fairness in recommender systems.
Downloads
References
Abdollahpouri, H., Mansoury, M., Burke, R., Mobasher, B., and Malthouse, E. C. (2021). User-centered evaluation of popularity bias in recommender systems. In Proceedings of the 29th ACM Conference UMAP 2021, Utrecht, pages 119–129. ACM. DOI: 10.1145/3450613.3456821.
Atauchi, P. D. F., Zanon, A. L., Rocha, L. C. D. d., and Manzato, M. G. (2025). Do calibrated recommendations affect explanations? a study on post-hoc adjustments. 16:441–460. DOI: 10.5753/jis.2025.5563.
Carvalho, R. and Rocha, L. (2020). Estratégias para aprimorar a diversidade categórica e geográfica de sistemas de recomendação de pois. In Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia), pages 23–26. SBC.
da Silva, D. C., Manzato, M. G., and Durão, F. A. (2021). Exploiting personalized calibration and metrics for fairness recommendation. Expert Systems with Applications, 181:115112. DOI: 10.1016/j.eswa.2021.115112.
de Souza, R. F. and Manzato, M. G. (2024). Uma abordagem em etapa de processamento para redução do viés de popularidade. In Brazilian Symposium on Multimedia and the Web (WebMedia), pages 310–317. SBC.
Fonseca, G., Cunha, W., Prenassi, G., Gonçalves, M. A., and Rocha, L. C. D. D. (2025). Instance-selection-inspired undersampling strategies for bias reduction in small and large language models for binary text classification. In Proceedings of the 63rd ACL, pages 9323–9340. Association for Computational Linguistics.
Gao, J., Chen, B., Zhao, X., Liu, W., Li, X., Wang, Y., Wang, W., Guo, H., and Tang, R. (2025). Llm4rerank: Llm-based auto-reranking framework for recommendations. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 228–239. DOI: 10.1145/3696410.3714922.
Harper, F. M. and Konstan, J. A. (2015). The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst., 5(4). DOI: 10.1145/2827872.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). Lora: Low-rank adaptation of large language models.
Klimashevskaia, A., Jannach, D., Elahi, M., and Trattner, C. (2024). A survey on popularity bias in recommender systems. User Modeling and User-Adapted Interaction, 34(5):1777–1834.
Lichtenberg, J. M., Buchholz, A., and Schwöbel, P. (2024). Large language models as recommender systems: A study of popularity bias. In Proceedings of the SIGIR 2024 Workshop on Generative Information Retrieval.
Liu, D., Yang, B., Du, H., Greene, D., Lawlor, A., Dong, R., and Li, I. (2023). Recprompt: A prompt tuning framework for news recommendation using large language models. CoRR.
Ortega, G. M., de Souza, R. F., and Manzato, M. G. (2024). Evaluating zero-shot large language models recommenders on popularity bias and unfairness: A comparative approach to traditional algorithms. In Simpósio Brasileiro de Sistemas Multimídia e Web (WebMedia), pages 45–48. SBC.
Prenassi, G., Souza, R. F. d., Sekido, H. Y., Fonseca, G., Manzato, M. G., and Rocha, L. C. D. d. (2025). Calibração de sistemas de recomendação com llms: otimização de prompts para balancear precisão, diversidade e justiça. Anais.
Quadrana, M., Cremonesi, P., and Jannach, D. (2018). Sequence-aware recommender systems. ACM computing surveys (CSUR), 51(4):1–36.
Rendle, S., Freudenthaler, C., Gantner, Z., and Schmidt-Thieme, L. (2012). Bpr: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
Rey, D. and Neuhäuser, M. (2011). Wilcoxon-signed-rank test. In International encyclopedia of statistical science, pages 1658–1659. Springer.
Sacilotti, A., Souza, R. F. d., and Manzato, M. G. (2023). Counteracting popularity-bias and improving diversity through calibrated recommendations. In In Proceedings of the 25th International Conference on Enterprise Information Systems, volume 1, Prague, Czech Republic. Scitepress.
Souza, R. and Manzato, M. (2024). A two-stage calibration approach for mitigating bias and fairness in recommender systems. In Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, pages 1659–1661, New York, NY, USA. ACM.
Steck, H. (2018). Calibrated recommendations. In Proceedings of the 12th ACM conference on recommender systems, pages 154–162.
Wang, J., Chen, X., Lee, K.-C., Ghosh, D., Rao, N., and Hu, H. (2025). Automating personalization: Prompt optimization for recommendation reranking.
Werneck, H., Silva, N., Viana, M. C., Mourão, F., Pereira, A. C., and Rocha, L. (2020). A survey on point-of-interest recommendation in location-based social networks. In Proceedings of the Brazilian Symposium on Multimedia and the Web, pages 185–192.
Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q., et al. (2024). A survey on large language models for recommendation. World Wide Web, 27(5):60.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. (2023). Large language models as optimizers. In The Twelfth International Conference on Learning Representations.
Zanon, A. L., da Rocha, L. C. D., and Manzato, M. G. (2022). Balancing the trade-off between accuracy and diversity in recommender systems with personalized explanations based on linked open data. Knowl. Based Syst., 252:109333. DOI: 10.1016/J.KNOSYS.2022.109333.
Zhao, Y., Wang, Y., Liu, Y., Cheng, X., Aggarwal, C. C., and Derr, T. (2025). Fairness and diversity in recommender systems: A survey. ACM Trans. Intell. Syst. Technol., 16(1). DOI: 10.1145/3664928.
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 The authors
This work is licensed under a Creative Commons Attribution 4.0 International License.
