Evaluating a Procedural Content Orchestrator Gameplay Data and Classifying User Profiles

Leonardo Tórtoro Pereira; T. Yuji Teoi; Claudio Fabiano Motta Toledo

doi:10.5753/jis.2025.5630

Authors

Leonardo Tórtoro Pereira São Paulo State University (UNESP) https://orcid.org/0000-0003-3032-6653
T. Yuji Teoi University of São Paulo https://orcid.org/0009-0002-6430-6693
Claudio Fabiano Motta Toledo University of São Paulo https://orcid.org/0000-0003-4776-8052

DOI:

https://doi.org/10.5753/jis.2025.5630

Keywords:

Procedural Content Generation, Player Profiling, Content Orchestration, Gameplay Metrics Evaluation, Machine Learning

Abstract

Background: Content Orchestration is a novel research field focused on coordinating distinct types of algorithmically generated game content. Purpose: Thus, the lack of research in this area hinders the analysis of gameplay data and player profiling in games with orchestrated content. Methods: This paper is an extension of a work that collected and analyzed gameplay logs of 15 players who played 119 game sections of 12 different dungeons of a top-down action game. The game’s Levels, Rules, and Narrative content were orchestrated and adapted to player profiles defined from a pre-test questionnaire. PCA and clustering techniques were used to highlight relevant gameplay metrics for distinguishing play styles. In this extension, we used the gameplay data alone to train classifiers with and without data augmentation to predict a user’s profile, measuring the accuracy, precision, recall and f1-score with a train-test split and a 5-fold cross-validation for a more robust accuracy. We also implemented data augmentation on our gameplay metrics sample. Results: We identified, through the previous work, two components of PCA explaining a total of 65% of data variability, containing data such as Lock Usage Rate, Enemy Kill Rate, Map Completion, and Completed Immersion Quests. We also found game difficulty as an important level component for impact clustering. Through data augmentation, we achieved novel results, such as a mean accuracy
of almost 95%, measured with a 5-fold cross-validation, for the Histogram-based Gradient Boosting classifier when predicting a player’s profile based on their gameplay data, even with our small sample size. Conclusion: Our work guides developers and researchers to choose relevant gameplay metrics to determine players’ play styles. Our extended results suggest that we can predict player’s profiles through gameplay metrics and data augmentation, even for small samples. More studies are needed to validate our findings, with a larger and more diverse player-base.

Downloads

Download data is not yet available.

References

Althnian, A., AlSaeed, D., Al-Baity, H., Samha, A., Dris, A. B., Alzakari, N., Abou Elwafa, A., and Kurdi, H. (2021). Impact of dataset size on classification performance: An empirical evaluation in the medical domain. Applied Sciences, 11(2). DOI: https://doi.org/10.3390/app11020796.

Bernhaupt, R. (2015). Game User Experience Evaluation. Springer Publishing Company, Incorporated. DOI: https://doi.org/10.5555/2807325.

Borges, J. B., Juy, C. L., de Andrade Matos, I. S., Angelo Silveira, P. V., and Darin, T. d. G. R. (2020). Player experience evaluation: a brief panorama of instruments and research opportunities. Journal on Interactive Systems, 11(1):74–91. DOI: https://doi.org/10.5753/jis.2020.765.

Carneiro, N., Miranda, D., Pereira, G., Mendonça, G., and Darin, T. (2022). A systematic mapping on player’s profiles: Motivations, behavior, and personality characteristics. Journal on Interactive Systems, 13(1):257–273. DOI: https://doi.org/10.5753/jis.2022.2572.

Cowley, B., Charles, D., Black, M., and Hickey, R. (2008). Toward an understanding of flow in video games. Comput. Entertain., 6(2). DOI: https://doi.org/10.1145/1371216.1371223.

de Lima, E. S., Feijó, B., and Furtado, A. L. (2021). Adaptive branching quests based on automated planning and story arcs. In 2021 20th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pages 9–18. DOI: https://doi.org/10.1109/SBGames54170.2021.00012.

Denisova, A. and Cairns, P. (2019). Player experience and deceptive expectations of difficulty adaptation in digital games. Entertainment Computing, 29:56–68. DOI: https://doi.org/10.1016/j.entcom.2018.12.001.

Dworkin, J. D., Linn, K. A., Teich, E. G., Zurn, P., Shinohara, R. T., and Bassett, D. S. (2020). The extent and drivers of gender imbalance in neuroscience reference lists. Nature Neuroscience, 23(8):918–926. DOI: https://doi.org/10.1038/s41593-020-0658-y.

Ekwaro-Osire, H., Ponugupati, S. L., Al Noman, A., Bode, D., and Thoben, K.-D. (2025). Data augmentation for numerical data from manufacturing processes: an overview of techniques and assessment of when which techniques work. Industrial Artificial Intelligence, 3(1):1–19. DOI: https://doi.org/10.1007/s44244-024-00021-x.

Eppmann, R., Bekk, M., and Klein, K. (2018). Gameful experience in gamification: Construction and validation of a gameful experience scale [gamex]. Journal of Interactive Marketing, 43:98–115. DOI: https://doi.org/10.1016/j.intmar.2018.03.002.

Heijne, N. and Bakkes, S. (2017). Procedural zelda: a pcg environment for player experience research. In Proceedings of the 12th International Conference on the Foundations of Digital Games, FDG ’17, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3102071.3102091.

Isbister, K. (2016a). How Games Move Us - Emotion by Design. MIT Press. DOI: https://doi.org/10.7551/mitpress/9267.001.0001.

Isbister, K. (2016b). How games move us: Emotion by design. MIT Press. DOI: https://doi.org/10.7551/mitpress/9267.001.0001.

Jacko, J. A. (2012). Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, Third Edition. CRC Press, Inc., Boca Raton, FL, USA, 3rd edition. DOI: https://doi.org/10.5555/2378709.

Kirchner-Krath, J., Altmeyer, M., Schürmann, L., Kordyaka, B., Morschheuser, B., Klock, A. C. T., Nacke, L., Hamari, J., and von Korflesch, H. F. (2024). Uncovering the theoretical basis of user types: An empirical analysis and critical discussion of user typologies in research on tailored gameful design. International Journal of Human-Computer Studies, 190:103314. DOI: https://doi.org/10.1016/j.ijhcs.2024.103314.

Koster, R. (2005). A Theory of Fun for Game Design. Paraglyph Press, Arizona. DOI: https://doi.org/10.5555/1207478.

Liapis, A., Yannakakis, G. N., Nelson, M. J., Preuss, M., and Bidarra, R. (2019). Orchestrating game generation. IEEE Transactions on Games, 11(1):48–68. DOI: https://doi.org/10.1109/TG.2018.2870876.

Liapis, A., Yannakakis, G. N., and Togelius, J. (2014). Computational game creativity. In Proceedings of the Fifth International Conference on Computational Creativity.

Loria, E. and Marconi, A. (2018). Player types and player behaviors: Analyzing correlations in an on-the-field gamified system. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, CHI PLAY 2018, Melbourne, VIC, Australia, October 28–31, 2018, pages 531–538. DOI: https://doi.org/10.1145/3270316.3271526.

Machado, M. C., Fantini, E. P. C., and Chaimowicz, L. (2011). Player modeling: What is it? how to do it? In SBGames 2011 - Tutorials Track.

Magy Seif El-Nasr, Anders Drachen, A. C. (2013). Game Analytics: Maximizing the Value of Player Data. Springer London. DOI: https://doi.org/10.1007/978-1-4471-4769-5.

Melhart, D., Azadvar, A., Canossa, A., Liapis, A., and Yannakakis, G. N. (2019). Your gameplay says it all: Modelling motivation in Tom Clancy’s The Division. In 2019 IEEE Conference on Games (CoG), pages 1–8. DOI: https://doi.org/10.1109/CIG.2019.8848123.

Microsoft (2020). Microsoft purchases ‘minecraft’. Accessed in: 2025-01-13.

Nunes, C. and Darin, T. (2024a). Echoes of player experience: A literature review on audio assessment and player experience in games. Proc. ACM Hum.-Comput. Interact., 8(CHI PLAY). DOI: https://doi.org/10.1145/3677069.

Nunes, C. and Darin, T. (2024b). Echoes of player experience: A literature review on audio assessment and player experience in games. Proc. ACM Hum.-Comput. Interact., 8(CHI PLAY). DOI: https://doi.org/10.1145/3677069.

Pereira, L., Rodrigues, K., Toledo, C., and Teoi, T. (2024a). Understanding players to enhance their fun: how to extract player data and motivation factors for procedural content generation. In Anais Estendidos do XXIII Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 37–42, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbgames_estendido.2024.241181.

Pereira, L., Teoi, T., and Toledo, C. (2024b). Towards evaluating a procedural content orchestrator gameplay data to differentiate user profiles. In Anais do III Workshop sobre Interação e Pesquisa de Usuários no Desenvolvimento de Jogos, pages 75–86, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/wiplay.2024.245483.

Pereira, L. T. (2022). Arquitetura e desenvolvimento de um sistema de geração procedural de múltiplos conteúdos para jogos eletrônicos em tempo real. PhD thesis, Universidade de São Paulo, São Carlos. Tese de Doutorado, Instituto de Ciências Matemáticas e de Computação.

Pereira, L. T., Viana, B. M. F., and Toledo, C. F. M. (2022). A system for orchestrating multiple procedurally generated content for different player profiles. IEEE Transactions on Games, pages 1–11. DOI: https://doi.org/10.1109/TG.2022.3213781.

Salen, K. and Zimmerman, E. (2003). Rules of Play: Game Design Fundamentals. The MIT Press. DOI: https://doi.org/10.5555/1215723.

Teoi, T., Pereira, L., and Toledo, C. (2024). Towards adapting a content orchestrator to a different game genre: Generating levels, rules, and narrative for diverse player profiles from a top-down adventure to a 2D platformer. In Anais do XXIII Simpósio Brasileiro de Jogos e Entretenimento Digital, pages 396–415, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbgames.2024.240282.

Togelius, J., Shaker, N., and Nelson, M. J. (2016). Introduction, pages 1–15. Springer International Publishing, Cham. DOI: https://doi.org/10.1007/978-3-319-42716-4_1.

Togelius, J., Yannakakis, G. N., Stanley, K. O., and Browne, C. (2011). Search-based procedural content generation: A taxonomy and survey. IEEE Transactions on Computational Intelligence and AI in Games, 3(3):172–186. DOI: https://doi.org/10.1109/TCIAIG.2011.2148116.

Tondello, G. F., Arrambide, K., Ribeiro, G., Cen, A. J.-l., and Nacke, L. E. (2019). “I don’t fit into a single type”: A trait model and scale of game playing preferences. In Lamas, D., Loizides, F., Nacke, L., Petrie, H., Winckler, M., and Zaphiris, P., editors, Human-Computer Interaction – INTERACT 2019, pages 375–395, Cham. Springer International Publishing. DOI: https://doi.org/10.1007/978-3-030-29384-0_23.

Tondello, G. F., Wehbe, R. R., Diamond, L., Busch, M., Marczewski, A., and Nacke, L. E. (2016). The gamification user types hexad scale. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’16, page 229–243, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/2967934.2968082.

Yannakakis, G. N., Spronck, P., Loiacono, D., and André, E. (2013). Player Modeling. In Lucas, S. M., Mateas, M., Preuss, M., Spronck, P., and Togelius, J., editors, Artificial and Computational Intelligence in Games, volume 6 of Dagstuhl Follow-Ups, pages 45–59. Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany. DOI: https://doi.org/10.4230/DFU.Vol6.12191.45.

Yannakakis, G. N. and Togelius, J. (2018). Modeling Players, pages 203–255. Springer International Publishing, Cham. DOI: https://doi.org/10.1007/978-3-319-63519-4_5.

Yee, N. (2006). Motivations for play in online games. CyberPsychology & Behavior, 9(6):772–775. PMID: 17201605. DOI: https://doi.org/10.1089/cpb.2006.9.772.

Yee, N. and Ducheneaut, N. (2018). Gamer motivation profiling: Uses and applications. Games User Research, pages 485–490. DOI: https://doi.org/10.1093/oso/9780198794844.003.0028.

Zaki, M. J., Meira Jr, W., and Meira, W. (2020). Data mining and machine learning: Fundamental concepts and algorithms. Cambridge University Press.

Zhou, D., Stiso, J., Bertolero, M., Cornblath, E., Teich, E., Blevins, A. S., Murphy, K., Oudyk, K., Michael, C., Urai, A., Matelsky, J., Virtualmario, Camp, C., Castillo, R. A., Saxe, R., Dworkin, J., and Bassett, D. (2022). dalejn/cleanbib: v1.1.3.

Zurn, P., Bassett, D. S., and Rust, N. C. (2020). The citation diversity statement: a practice of transparency, a way of life. Trends in Cognitive Sciences, 24(9):669–672. DOI: https://doi.org/10.1016/j.tics.2020.06.009.