Using Interviews to Evaluate Location-Based Games: Lessons and Challenges

Nayana Carneiro; Ticianne Darin; Mateus Pinheiro; Windson Viana

doi:10.5753/jis.2020.754

Authors

Nayana Carneiro Universidade Federal do Ceará (UFC) http://orcid.org/0000-0001-8253-6375
Ticianne Darin Universidade Federal do Ceará (UFC) http://orcid.org/0000-0003-3617-5462
Mateus Pinheiro Universidade Federal do Ceará (UFC) http://orcid.org/0000-0001-5614-3658
Windson Viana Universidade Federal do Ceará (UFC)

DOI:

https://doi.org/10.5753/jis.2020.754

Keywords:

Location-based games, Evaluation methods, Interview, Interaction, Player Experience

Abstract

Games User Research (GUR) is an interdisciplinary field of study that aims to measure, analyze, and understand the interaction and experience of players with digital games. Joining efforts to the advances in GUR, this work focuses on the evaluation of player interaction with Location-Based Games (LBGs), seeking to understand which aspects of this interaction can be explored through interviews and how researchers and practitioners have been using this method. We analyzed 23 studies that applied interviews to this end and conducted an expert opinion survey with the authors of these studies. As a result, we presented lessons and research challenges for the use of interviews in this type of evaluation, in hopes to encourage the conscious and systematic application of this method, in addition to guiding practitioners and researchers, especially beginners.

Downloads

Download data is not yet available.

References

Abeele, V. V., Spiel, K., Nacke, L., Johnson, D., and Gerling, K. (2020). Development and validation of the player experience inventory: A scale to measure player experiences at the level of functional and psychosocial consequences. International Journal of Human-Computer Studies, 135:102370.

Ahlqvist, O. (2018). Location-based games. International Encyclopedia of Geography: People, the Earth, Environment and Technology, pages 1–6.

Alha, K., Koskinen, E., Paavilainen, J., and Hamari, J. (2019). Why do people play location-based augmented reality games: A study on Pokémon Go. Computers in Human Behavior, 93:114–122.

Baillie, L., Morton, L., Uzor, S., and Moffatt, D. C. (2010). An investigation of user responses to specifically designed activities in a multimodal location-based game. Journal on Multimodal User Interfaces, 3(3):179–188.

Ballagas, R., Kuntze, A., and Walz, S. P. (2008). Gaming tourism: Lessons from evaluating Rexplorer, a pervasive game for tourists. In International Conference on Pervasive Computing, pages 244–261. Springer.

Benford, S., Crabtree, A., Flintham, M., Drozd, A., Anastasi, R., Paxton, M., Tandavanitj, N., Adams, M., and Row-Farr, J. (2006). Can you see me now? ACM Transactions on Computer-Human Interaction (TOCHI), 13(1):100–133.

Blandford, A., Furniss, D., and Makri, S. (2016). Qualitative HCI Research: Going Behind the Scenes. Morgan & Claypool Publishers.

Blum, L., Wetzel, R., McCall, R., Oppermann, L., and Broll, W. (2012). The final TimeWarp: using form and content to support player experience and presence when designing location-aware mobile augmented reality games. In Proceedings of the designing interactive systems conference, pages 711–720. ACM.

Boyce, C. and Neale, P. (2006). Conducting in-depth interviews: A guide for designing and conducting in-depth interviews for evaluation input.

Calvillo-Gámez, E., Crisp, J., and Romero, P. (2011). Empirical validation of the involvement component of the pervasive gameflow model. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, page 48. ACM.

Carneiro, N., Darin, T., and Viana, W. (2019a). Uso de entrevistas na avaliação de jogos baseados em localização: Lições e desafios. In Anais do I Workshop sobre Interação e Pesquisa de Usuários no Desenvolvimento de Jogos, pages 59–68. SBC.

Carneiro, N., Darin, T., and Viana, W. (2019b). What are we talking about when we talk about location-based games evaluation? a systematic mapping study. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, pages 1–13.

Chatzidimitris, T., Gavalas, D., and Michael, D. (2016). Soundpacman: Audio augmented reality in location-based games. In 18th Mediterranean Electrotechnical Conference (MELECON), pages 1–6. IEEE.

Chauncey, W. (2014). Interview techniques for UX practitioners. A User-Centered Design Method.

Chittaro, L. and Sioni, R. (2012). Turning the classic snake mobile game into a location–based exergame that encourages walking. In International Conference on Persuasive Technology, pages 43–54. Springer.

Crawford, S. B., Monks, S. M., and Wells, R. N. (2018). Virtual reality as an interview technique in evaluation of emergency medicine applicants. AEM education and training, 2(4):328–333.

Creswell, J. W. and Poth, C. N. (2018). Qualitative inquiry and research design: Choosing among five approaches. Sage publications.

Darin, T., Andrade, R., and Sánchez, J. (2019). Principles for evaluating usability in multimodal games for people who are blind. In International Conference on Human-Computer Interaction, pages 209–223. Springer.

De Souza e Silva, A. and Sutko, D. M. (2009). Digital cityscapes: Merging digital and urban play spaces.

De Souza e Silva, A. and Sutko, D. M. (2011). Theorizing locative technologies through philosophies of the virtual. Communication Theory, 21(1):23–42.

Diamantaki, K., Rizopoulos, C., Charitos, D., and Tsianos, N. (2011). Theoretical and methodological implications of designing and implementing multiuser location-based games. Personal and Ubiquitous Computing, 15(1):37–49.

Drachen, A., Mirza-Babaei, P., and Nacke, L. E. (2018). Games user research. Oxford University Press.

Ekman, I. (2007). Sound-based gaming for sighted audiences–experiences from a mobile multiplayer location aware game. In Proceedings of the 2nd audio mostly conference, pages 148–153.

El-Nasr, M. S., Durga, S., Shiyko, M., and Sceppa, C. (2015). Data-driven retrospective interviewing (DDRI): a proposed methodology for formative evaluation of pervasive games. Entertainment Computing, 11:1–19.

Fonteyn, M. E., Kuipers, B., and Grobe, S. J. (1993). A description of think aloud method and protocol analysis. Qualitative health research, 3(4):430–441.

Holmes, S. (2019). Play-based 7 interview techniques with young children. Using Innovative Methods in Early Years Research: Beyond the Conventional, page 92.

Isbister, K. and Schaffer, N. (2008). Game usability: Advancing the player experience. CRC press.

Jegers, K. (2007). Pervasive game flow: understanding player enjoyment in pervasive gaming. Computers in Entertainment (CIE), 5(1):9–es.

Kasapakis, V. and Gavalas, D. (2015). Pervasive gaming: Status, trends and design principles. Journal of Network and Computer Applications, 55:213–236.

Kasapakis, V. and Gavalas, D. (2017). User-generated content in pervasive games. Computers in Entertainment (CIE), 16(1):3.

Kiefer, P., Matyas, S., and Schlieder, C. (2006). Systematically exploring the design space of location-based games. In 4th International Conference on Pervasive Computing, pages 183–190.

Kitchenham, B. A., Budgen, D., and Brereton, O. P. (2010). The value of mapping studies-a participant-observer case study. In EASE, volume 10, pages 25–33.

Kurczak, J., Graham, T., Joly, C., and Mandryk, R. L. (2011). Hearing is believing: evaluating ambient audio for location-based games. In Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, page 32. ACM.

Kvale, S. (2008). Doing interviews. Sage.

Lamont, M. and Swidler, A. (2014). Methodological pluralism and the possibilities and limits of interviewing. Qualitative Sociology, 37(2):153–171.

Lankes, M. and Stiglbauer, B. (2016). Gazear: Mobile gaze-based interaction in the context of augmented reality games. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics, pages 397–406. Springer.

Lazar, J., Feng, J. H., and Hochheiser, H. (2017). Research methods in human-computer interaction. Morgan Kaufmann.

Leorke, D. (2018). Location-based gaming: play in public space. Springer.

Lindt, I., Ohlenburg, J., Pankoke-Babatz, U., and Ghellal, S. (2007). A report on the crossmedia game epidemic menace. Computers in Entertainment (CIE), 5(1):8.

Linehan, C., Bull, N., and Kirman, B. (2013). Bollocks!! designing pervasive games that play with the social rules of built environments. In International Conference on Advances in Computer Entertainment Technology, pages 123–137. Springer.

McCall, R. and Braun, A.-K. (2008). Experiences of evaluating presence in augmented realities. Psychnology, 6(2):157–163.

McCall, R., Wetzel, R., Löschner, J., and Braun, A.-K. (2011). Using presence to evaluate an augmented reality location aware game. Personal and Ubiquitous Computing, 15(1):25–35.

Merritt, T., Nielsen, C. L., Jakobsen, F. L., and Grønbæk, J. E. (2017). Glowphones: designing for proxemics play with low-resolution displays in location-based games. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pages 69–81. ACM.

Myers, M. D. and Newman, M. (2007). The qualitative interview in is research: Examining the craft. Information and organization, 17(1):2–26.

Nacke, L. E., Drachen, A., and Göbel, S. (2010). Methods for evaluating gameplay experience in a serious gaming context. International Journal of Computer Science in Sport, 9(2):1–12.

Nacke, L. E., Moser, C., Drachen, A., Mirza-Babaei, P., Abney, A., and Zhenyu, Z. C. (2016). Lightweight games user research for indies and non-profit organizations. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 3597–3603. ACM.

Nilsson, T., Hogsden, C., Perera, C., Aghaee, S., Scruton, D. J., Lund, A., and Blackwell, A. F. (2016). Applying seamful design in location-based mobile museum applications. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 12(4):56.

Oppermann, L., Schaal, S., Eisenhardt, M., Brosda, C., Müller, H., and Bartsch, S. (2017). Move, interact, learn, eat–a toolbox for educational location-based games. In International Conference on Advances in Computer Entertainment, pages 774–794. Springer.

Ozok, A. A. (2009). Survey design and implementation in HCI. Human-Computer Interaction: Development Process, 253.

Paavilainen, J., Korhonen, H., Alha, K., Stenros, J., Koskinen, E., and Mayra, F. (2017). The pokémon go experience: A location-based augmented reality mobile game goes mainstream. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 2493–2498. ACM.

Petersen, K., Vakkalanka, S., and Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64:1–18.

Prasetio, Y. L., Wijaya, R., Sjah, M. P., Christian, M. R., and Chowanda, A. (2017). Location-based game to enhance player’s experience in survival horror game. Procedia computer science, 116:206–213.

Preece, J., Sharp, H., and Rogers, Y. (2019). Interaction design: beyond human-computer interaction. John Wiley & Sons.

Pyae, A., Mika, L., and Smed, J. (2017). Understanding players’ experiences in location-based augmented reality mobile games: A case of Pokémon Go. In Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play, pages 535–541. ACM.

Quek, A. and See, J. (2015). Obscura: A mobile game with camera-based mechanics. In 2015 Game Physics and Mechanics International Conference (GAMEPEC), pages 21–25. IEEE.

Rowe, G. and Wright, G. (2001). Expert opinions in forecasting: the role of the Delphi technique. In Principles of forecasting, pages 125–144. Springer.

Rowley, J. (2012). Conducting research interviews. Management research review, 35(3/4):260–271.

Salen, K., Tekinbaş, K. S., and Zimmerman, E. (2004). Rules of play: Game design fundamentals. MIT press.

Sánchez, J. L. G., Vela, F. L. G., Simarro, F. M., and Padilla-Zea, N. (2012). Playability: analysing user experience in video games. Behaviour & Information Technology, 31(10):1033–1054.

Sandham, A., Ormerod, T., Dando, C., Bull, R., Jackson, M., and Goulding, J. (2011). Scent trails: Countering terrorism through informed surveillance. In International Conference on Engineering Psychology and Cognitive Ergonomics, pages 452–460. Springer.

Schell, J. (2014). The Art of Game Design: A book of lenses. AK Peters/CRC Press.

Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S., Elmqvist, N., and Diakopoulos, N. (2016). Designing the user interface: strategies for effective human-computer interaction. Pearson.

Spradley, J. P. (2016). The ethnographic interview. Waveland Press.

Sra, M. and Schmandt, C. (2015). Expanding social mobile games beyond the device screen. Personal and Ubiquitous Computing, 19(3-4):495–508.

Turner III, D. W. (2010). Qualitative interview design: A practical guide for novice investigators. The qualitative report, 15(3):754–760.

Verdejo, A. J., De Moor, K., Ketyko, I., Nielsen, K. T., Vanattenhoven, J., De Pessemier, T., Joseph, W., Martens, L., and De Marez, L. (2010). QoE estimation of a location-based mobile game using on-body sensors and qos-related data. In 2010 IFIP Wireless Days, pages 1–5. IEEE.

Wang, A. I., Forberg, S., and Øye, J. K. (2016). Knowledge war-a pervasive multiplayer role-playing learning game. In European Conference on Game Based Learning (ECGBL 2016). ACPI, Glasgow, Scotland.

Wood, L. E. (1997). Semi-structured interviewing for user-centered design. interactions, 4(2):48–61.