Gamification Techniques and Contribution Filtering in Crowdsourcing Micro-Task Applications

Ana Maria Amorim; Ailton Ribeiro; Murilo Guerreiro Arouca; Iago Meijon; Victor Cavalheiro; Maria Clara Pestana; Vaninha Vieira

doi:10.5753/jis.2024.3727

Authors

Ana Maria Amorim Federal University of Bahia, UFBA https://orcid.org/0000-0002-6454-5216
Ailton Ribeiro Federal University of Bahia, UFBA https://orcid.org/0000-0003-4524-2188
Murilo Guerreiro Arouca Federal University of Bahia, UFBA https://orcid.org/0000-0002-2909-2688
Iago Meijon Federal University of Bahia, UFBA https://orcid.org/0009-0002-3577-2185
Victor Cavalheiro Federal University of Bahia, UFBA https://orcid.org/0009-0001-2168-2704
Maria Clara Pestana Federal University of Bahia, UFBA https://orcid.org/0000-0001-8382-7844
Vaninha Vieira Institute of Computing, Federal University of Bahia, UFBA https://orcid.org/0000-0002-3424-8413

DOI:

https://doi.org/10.5753/jis.2024.3727

Keywords:

Gamification, Crowdsourcing, Micro-Tasks, Content Filtering, Motivation

Abstract

The rapid expansion of the internet has led to a growing worldwide user base, with Brazil alone having approximately 83% of households connected to the network, equating to around 61.8 million households. Crowdsourcing, a production model that harnesses collective wisdom for problem-solving, has gained prominence in this digital landscape. Challenges in crowdsourcing include improving people's motivation and engagement and verifying the quality of a high number of contributions. In our research, we investigated the hypothesis that using gamification techniques, including recognition badges, feedback mechanisms, and user rankings, improves users' engagement and motivation in crowdsourcing micro-tasks applications. This paper presents ConTask, a crowdsourcing micro-task application, and two studies conducted to investigate the impact of using gamification techniques and contribution filtering as motivational factors in crowdsourcing. The first was a case study comparing two versions of ConTask: the original version and a gamified one. The second was an experimental study to evaluate the developed contribution filtering mechanism. Findings suggest that the use of gamification and contribution filtering can improve user participation in crowdsourcing micro-task applications.

Downloads

Download data is not yet available.

References

Alam, S. L. and Sun, R. (2023). The role of system use practices for sustaining motivation in crowdsourcing: A technology-in-practice perspective. Information Systems Journal, 33(4):758–789. DOI: https://doi.org/10.1111/isj.12423.

Amorim, A. M. and Vieira, V. (2019). Exploratory study on the motivation of brazilian elderly people in crowdsourcing systems. In Proceedings of the 18th Brazilian Symposium on Human Factors in Computing Systems, IHC ’19, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3357155.3360478.

Amorim, A. M. and Vieira, V. (2023). Participation in crowdsourcing micro-tasks: what motivates Brazilian older adults? Universal Access in the Information Society, pages 1–25. DOI: https://doi.org/10.1007/s10209-023-01023-9.

Arouca, M., Ribeiro, A., Amorim, A., Neves, I., Vieira, V., Barreto, M., Costa, F., and Brito, R. (2024). Gamification to support crowdsourcing and participatory mapping for signaling and spatialization of covid19 transmission predictors. In Proceedings of the 19th Brazilian Symposium on Collaborative Systems, pages 42–55, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbsc.2024.238059.

Bastanfard, A., Shahabipour, M., and Amirkhani, D. (2023). Crowdsourcing of labeling image objects: An online gamification application for data collection. Multimedia Tools and Applications, pages 1–34. DOI: https://doi.org/10.1007/s11042-023-16325-6.

Boyer, R. S. and Moore, J. S. (1977). A fast string searching algorithm. Commun. ACM, 20(10):762–772. DOI: https://doi.org/10.1145/359842.359859.

Brabham, D. (2013). Crowdsourcing. The MIT Press Essential Knowledge series. MIT Press.

Brabham, D. C. (2008). Crowdsourcing as a model for problem solving: An introduction and cases. Convergence, 14(1):75–90. DOI: https://doi.org/10.1177/1354856507084420.

Brewer, R., Morris, M., and Piper, A. (2016). Why would anybody do this?: Older adults’ understanding of and experiences with crowd work. In CHI 2016 - 34th Annual CHI Conference on Human Factors in Computing Systems, Conference on Human Factors in Computing Systems, pages 2246–2257. Association for Computing Machinery. DOI: https://doi.org/10.1145/2858036.2858198.

CETIC.BR-NIC.BR (2021). Executive summary - survey on the use of information and communication technologies in brazilian households - ict households 2020. Regional Center for Studies on the Development of the Information Society – Cetic.br; Brazilian Network Information Center – NIC.br.

Chandler, D. and Kapelner, A. (2013). Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior Organization, 90:123–133. DOI: https://doi.org/10.1016/j.jebo.2013.03.003.

Charmaz, K. (2014). Constructing Grounded Theory. Introducing Qualitative Methods series. SAGE Publications.

Chen, K.-W., Chang, Y.-J., and Chan, L. (2022a). Predicting opportune moments to deliver notifications in virtual reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3491102.3517529.

Chen, L., Xu, P., and Liu, D. (2020). Effect of crowd voting on participation in crowdsourcing contests. Journal of Management Information Systems, 37(2):510–535. DOI: https://doi.org/10.1080/07421222.2020.1759342.

Chen, Z., Jiang, L., and Li, C. (2022b). Label augmented and weighted majority voting for crowdsourcing. Information Sciences, 606:397–409. DOI: https://doi.org/10.1016/j.ins.2022.05.066.

Chi, P.-Y. P., Batra, A., and Hsu, M. (2018). Mobile crowdsourcing in the wild: challenges from a global community. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, MobileHCI ’18, page 410–415, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3236112.3236176.

Cole, R. (1994). Tight bounds on the complexity of the boyer–moore string matching algorithm. SIAM Journal on Computing, 23(5):1075–1091. DOI: https://doi.org/10.1137/S0097539791195543.

Congcong Yang, H. J. Y. and Feng, Y. (2021). Using gamification elements for competitive crowdsourcing: exploring the underlying mechanism. Behaviour & Information Technology, 40(9):837–854. DOI: https://doi.org/10.1080/0144929X.2020.1733088.

da Silva, A. V. D. and Vieira, V. (2018). Towards an api for user attention prediction in mobile notification overload. In Anais Estendidos do XXIV Simpósio Brasileiro de Sistemas Multimídia e Web, pages 13–17, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/webmedia.2018.4552.

Deci, E. L. and Ryan, R. M. (1980). Self-determination theory: When mind mediates behavior. The Journal of mind and Behavior, pages 33–43.

Deci, E. L. and Ryan, R. M. (1985). Cognitive Evaluation Theory, pages 43–85. Springer US, Boston, MA. DOI: https://doi.org/10.1007/978-1-4899-2271-7_3.

Deci, E. L. and Ryan, R. M. (2010). Intrinsic Motivation, pages 1–2. John Wiley Sons, Ltd. DOI: https://doi.org/10.1002/9780470479216.corpsy0467.

Deterding, S., Sicart, M., Nacke, L., O’Hara, K., and Dixon, D. (2011). Gamification. using game-design elements in non-gaming contexts. In CHI ’11 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’11, page 2425–2428, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/1979742.1979575.

Esteves, B., Fraser, K., Kulkarni, S., Conlan, O., and Rodríguez-Doncel, V. (2022). Now, later, never: A study of urgency in mobile push-notifications. In Delir Haghighi, P., Khalil, I., and Kotsis, G., editors, Advances in Mobile Computing and Multimedia Intelligence, pages 38–44, Cham. Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-031-20436-4_4.

Feng, Y., Jonathan Ye, H., Yu, Y., Yang, C., and Cui, T. (2018). Gamification artifacts and crowdsourcing participation: Examining the mediating role of intrinsic motivations. Computers in Human Behavior, 81:124–136. DOI: https://doi.org/10.1016/j.chb.2017.12.018.

Feng, Y., Yi, Z., Yang, C., Chen, R., and Feng, Y. (2022). How do gamification mechanics drive solvers’ knowledge contribution? a study of collaborative knowledge crowdsourcing. Technological Forecasting and Social Change, 177:121520. DOI: https://doi.org/10.1016/j.techfore.2022.121520.

Fitz, N., Kushlev, K., Jagannathan, R., Lewis, T., Paliwal, D., and Ariely, D. (2019). Batching smartphone notifications can improve well-being. Computers in Human Behavior, 101:84–94. DOI: https://doi.org/10.1016/j.chb.2019.07.016.

Gadiraju, U., Kawase, R., and Dietze, S. (2014). A taxonomy of microtasks on the web. In Proceedings of the 25th ACM Conference on Hypertext and Social Media, HT ’14, page 218–223, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/2631775.2631819.

Günther, H. (2003). Como elaborar um questionário. Série: Planejamento de pesquisa nas ciências sociais, 1:1–15.

Howe, J. (2006). The rise of crowdsourcing. Wired magazine, 14(6):1–4.

Kobayashi, M., Arita, S., Itoko, T., Saito, S., and Takagi, H. (2015). Motivating multi-generational crowd workers in social-purpose work. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, CSCW ’15, page 1813–1824, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/2675133.2675255.

Law, E., Yin, M., Goh, J., Chen, K., Terry, M. A., and Gajos, K. Z. (2016). Curiosity killed the cat, but makes crowdwork better. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI ’16, page 4098–4110, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/2858036.2858144.

Lazar, J., Feng, J., and Hochheiser, H. (2017). Research Methods in Human-Computer Interaction. Elsevier Science.

Lee, T. Y., Dugan, C., Geyer, W., Ratchford, T., Rasmussen, J., Shami, N. S., and Lupushor, S. (2013). Experiments on motivational feedback for crowdsourced workers. DOI: https://doi.org/10.1609/icwsm.v7i1.14428.

Lopes, W., Augusto, P., Fernandes, I., and Madeira, C. (2024). Proposal for a gamification strategy applied to remote learning. Journal on Interactive Systems, 15(1):92–103. DOI: https://doi.org/10.5753/jis.2024.2700.

Meijon, I., Amorim, A. M., Ribeiro, A., Pestana, M. C., and Vieira, V. (2023). A study on applying gamification techniques to a crowdsourcing app for micro-tasks. In Proceedings of the 18th Brazilian Symposium on Collaborative Systems, pages 57–71, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbsc.2023.229082.

Meliande, R., Ribeiro, A., Arouca, M., Amorim, A., Pestana, M., and Vieira, V. (2024). Meta-education: A case study in academic events in the metaverse. In Proceedings of the 19th Brazilian Symposium on Collaborative Systems, pages 28–41, Porto Alegre, RS, Brasil. SBC. DOI: https://doi.org/10.5753/sbsc.2024.238057.

Morschheuser, B., Hamari, J., and Koivisto, J. (2016). Gamification in crowdsourcing: A review. In 2016 49th Hawaii International Conference on System Sciences (HICSS), pages 4375–4384. DOI: 10.1109/HICSS.2016.543.

Navarro, G. and Tarhio, J. (2000). Boyer—moore string matching over ziv-lempel compressed text. In Giancarlo, R. and Sankoff, D., editors, Combinatorial Pattern Matching, pages 166–180, Berlin, Heidelberg. Springer Berlin Heidelberg. DOI: https://doi.org/10.1007/3-540-45123-4_16.

Ooge, J., De Croon, R., Verbert, K., and Vanden Abeele, V. (2020). Tailoring gamification for adolescents: a validation study of big five and hexad in dutch. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play, CHI PLAY ’20, page 206–218, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3410404.3414267.

Park, S., Kwon, S., and Lee, U. (2018). Campuswatch: Exploring communitysourced patrolling with pervasive mobile technology. Proc. ACM Hum.-Comput. Interact., 2(CSCW). DOI: https://doi.org/10.1145/3274403.

Pestana, M. C. and Vieira, V. (2018a). Context-aware task distribution for mobile crowdsourcing. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, IHC ’18, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3274192.3274206.

Pestana, M. C. and Vieira, V. (2018b). Context-aware task distribution for mobile crowdsourcing. In Proceedings of the 17th Brazilian Symposium on Human Factors in Computing Systems, IHC 2018, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3274192.3274206.

Ribeiro, A., Vieira, V., Alves, L., and Maciel, C. (2024). Vishnu: An approach to support the personalization of self-expressive avatars using context-awareness. International Journal of Human-Computer Studies, 185:103243. DOI: https://doi.org/10.1016/j.ijhcs.2024.103243.

Rodrigues, L., Palomino, P. T., Toda, A. M., Klock, A. C. T., Oliveira, W., Avila-Santos, A. P., Gasparini, I., and Isotani, S. (2021). Personalization improves gamification: Evidence from a mixed-methods study. Proc. ACM Hum.-Comput. Interact., 5(CHI PLAY). DOI: https://doi.org/10.1145/3474714.

Sailer, M., Hense, J., Mandl, H., and Klevers, M. (2013). Psychological perspectives on motivation through gamification. Interaction Design and Architecture(s) - IxD&A, (19):28 – 37. DOI: https://doi.org/10.55612/s-5002-019-002.

Stol, K.-J. and Fitzgerald, B. (2014). Two’s company, three’s a crowd: A case study of crowdsourcing software development. In Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, page 187–198, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/2568225.2568249.

Tanalol, H., Hashim, H., Turumogan, P., Noor, N. A. M., Baharum, A., and Deris, F. D. (2023). Identifying gamified teaching elements in computer science course. In 2023 International Conference on Platform Technology and Service (PlatCon), pages 18–23. DOI: https://doi.org/10.1109/PlatCon60102.2023.10255180.

Tao, F., Jiang, L., and Li, C. (2020). Label similarity-based weighted soft majority voting and pairing for crowdsourcing. Knowledge and Information Systems, 62:2521–2538. DOI: https://doi.org/10.1007/s10115-020-01475-y.

Toda, A., Klock, A., Pereira, F. D., Rodrigues, L. A., Palomino, P. T., Lopes, V., Stewart, C., Oliveira, E. H. T., Gasparini, I., Isotani, S., and Cristea, A. (2022a). Towards the understanding of cultural differences in between gamification preferences: A data-driven comparison between the US and Brazil. In Mitrovic, A. and Bosch, N., editors, Proceedings of the 15th International Conference on Educational Data Mining, pages 560–564, Durham, United Kingdom. International Educational Data Mining Society. DOI: https://doi.org/10.5281/zenodo.6853030.

Toda, A., Palomino, P. T., Rodrigues, L., Klock, A. C. T., Pereira, F., Borges, S., Gasparini, I., Teixeira, E. H., Isotani, S., and Cristea, A. I. (2022b). Gamification through the looking glass - perceived biases and ethical concerns of brazilian teachers. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II, page 259–262, Berlin, Heidelberg. Springer-Verlag. DOI: https://doi.org/10.1007/978-3-031-11647-6_47.

Tondello, G. F. and Nacke, L. E. (2020). Validation of user preferences and effects of personalized gamification on task performance. Frontiers in Computer Science, 2:29. DOI: https://doi.org/10.3389/fcomp.2020.00029.

Tsvetkova, M., Müller, S., Vuculescu, O., Ham, H., and Sergeev, R. A. (2022). Relative feedback increases disparities in effort and performance in crowdsourcing contests: Evidence from a quasi-experiment on topcoder. Proc. ACM Hum.-Comput. Interact., 6(CSCW2). DOI: https://doi.org/10.1145/3555649.

Vaughan, J. W. (2018). Making better use of the crowd: How crowdsourcing can advance machine learning research. Journal of Machine Learning Research, 18(193):1–46.

Vieira, V., Tedesco, P., and Salgado, A. C. (2011). Designing context-sensitive systems: An integrated approach. Expert Systems with Applications, 38(2):1119–1138. Intelligent Collaboration and Design. DOI: https://doi.org/10.1016/j.eswa.2010.05.006.

Walter, V., Kölle, M., and Collmar, D. (2022). A gamification approach for the improvement of paid crowd-based labelling of geospatial data. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 5(4):113–120. DOI: https://doi.org/10.5194/isprs-AnnalsV-4-2022-113-2022.

Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering. Computer Science. Springer Berlin Heidelberg.

Wu, W. and Gong, X. (2021). Motivation and sustained participation in the online crowdsourcing community: the moderating role of community commitment. Internet Research, 31(1):287–314. DOI: https://doi.org/10.1108/INTR-01-2020-0008.

Zhang, X., Xia, E., Shen, C., and Su, J. (2022). Factors influencing solvers’ behaviors in knowledge-intensive crowdsourcing: A systematic literature review. Journal of Theoretical and Applied Electronic Commerce Research, 17(4):1297–1319. DOI: https://doi.org/10.3390/jtaer17040066.