Establishing the Awareness Assessment Process
DOI:
https://doi.org/10.5753/jbcs.2026.6273Keywords:
Awareness, Collaboration, Assessment, ProcessAbstract
Different strategies in the literature seek to help support awareness mechanisms in collaborative applications. However, the most common approaches are often applied in a specific context and do not focus on evaluating these mechanisms or the support provided from the user's perspective. Few studies present methods or processes that provide awareness aspects in collaborative systems; thus, finding a good starting point in the literature can be challenging for beginners in awareness design. We build upon the findings presented by Mantau and Benitti [2023a], providing contributions towards developing an awareness assessment process that enables accessing awareness and, consequently, collaboration support by measuring awareness mechanisms from the participant's viewpoint. Then, we expose the model’s artifacts to HCI and collaborative system examiners to gauge their appreciation and verify the suitability of the process based on reliability and usefulness criteria. The case study scenario demonstrated suitable indicators from the perspective of demographic data and IRT parameterization. As a result, the applicability of the awareness scale from the participant's perspective was achieved.
Downloads
References
Agogo, D. and Hess, T. J. (2018). "how does tech make you feel?" a review and examination of negative affective responses to technology use. European Journal of Information Systems, 27(5):570-599. DOI: 10.1080/0960085X.2018.1435230.
Altenburger, T., Guerriero, A., Vagner, A., and Martin, B. (2012). Groupware requirements modelling for adaptive user interface design. eWork and eBusiness in Architecture, Engineering and Construction - Proceedings of the European Conference on Product and Process Modelling 2012, ECPPM 2012, 1(1):825-832. DOI: 10.1201/b12516-131.
Antunes, P., Herskovic, V., Ochoa, S. F., and Pino, J. A. (2012). Structuring dimensions for collaborative systems evaluation. ACM Computing Surveys, 44(2):1-28. DOI: 10.1145/2089125.2089128.
Antunes, P., Herskovic, V., Ochoa, S. F., and Pino, J. A. (2014). Reviewing the quality of awareness support in collaborative applications. Journal of Systems and Software, 89(1):146-169. DOI: 10.1016/j.jss.2013.11.1078.
Baker, F. B. and Kim, S.-H. (2017). The basics of item response theory using R. Springer, Madision, WI, USA. DOI: 10.1007/978-3-319-54205-8.
Barbosa, S. and Silva, B. (2010). Interação humano-computador. Elsevier Brasil, Rio de Janeiro, RJ. Book.
BASILI, V. (1992). Software modeling and measurement: The goal/question/metric paradigm. Technical Report. Available at:[link].
Beecham, S., Hall, T., Britton, C., Cottee, M., and Rainer, A. (2005). Using an expert panel to validate a requirements process improvement model. Journal of Systems and Software, 76(3):251-275. DOI: 10.1016/j.jss.2004.06.004.
Belkadi, F., Bonjour, E., Camargo, M., Troussier, N., and Eynard, B. (2013). A situation model to support awareness in collaborative design. International Journal of Human Computer Studies, 71(1):110-129. DOI: 10.1016/j.ijhcs.2012.03.002.
Berkman, M. l., Karahoca, D., and Karahoca, A. (2018). A measurement and structural model for usability evaluation of shared workspace groupware. International Journal of Human-Computer Interaction, 34(1):35-56. DOI: 10.1080/10447318.2017.1326578.
Bravo, C., Duque, R., Molina, A. I., and Gallardo, J. (2023). Modeling awareness requirements in groupware: From cards to diagrams. IEEE Transactions on Human-Machine Systems, page 1–12. DOI: 10.1109/thms.2023.3332592.
Brezillon, P., Borges, M., Pino, J., and Pomerol, J. (2004). Context-based awareness in group work. In Proceeding of the 17th International FLAIRS Conference, Miami, FL, USA. AAAI Press. Available at:[link].
Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for therenvironment. Journal of Statistical Software, 48(6). DOI: 10.18637/jss.v048.i06.
Collazos, C. A., Gutierrez, F. L., Gallardo, J., Ortega, M., Fardoun, H. M., and Molina, A. I. (2019). Descriptive theory of awareness for groupware development. Journal of Ambient Intelligence and Humanized Computing, 10(12):4789-4818. DOI: 10.1007/s12652-018-1165-9.
Cruz, A., Correia, A., Paredes, H., Fonseca, B., Morgado, L., and Martins, P. (2012). Towards an overarching classification model of cscw and groupware: A socio-technical perspective. In Lecture Notes in Computer Science, volume 7493, pages 41-56, Raesfeld, Germany. Elsevier. DOI: 10.1007/978-3-642-33284-5-4.
De Araújo, L. P., Mantau, M. J., Citadin, J. R., Berkenbrock, C. D. M., Kemczinski, A., Berkenbrock, G. R., and Mattos, M. M. (2014). Heuristic evaluation for mobile groupware: Evaluating two audience response systems. Journal of Applied Computing Research, 3(2):64-77. DOI: 10.4013/jacr.2013.32.01.
Dourish, P. (2004). Where the action is: the foundations of embodied interaction. MIT Press, Cambridge, MA, USA. DOI: 10.7551/mitpress/7221.001.0001.
Dourish, P. and Bellotti, V. (1992). Awareness and coordination in shared workspaces. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work, CSCW '92, pages 107-114, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/143457.143468.
Dyba, T. (2000). An instrument for measuring the key factors of success in software process improvement. Empirical software engineering, 5:357-390. DOI: 10.1023/a:1009800404137.
El Emam, K. and Birk, A. (2000). Validating the iso/iec 15504 measure of software requirements analysis process capability. IEEE Transactions on Software Engineering, 26(6):541–566. DOI: 10.1109/32.852742.
Espirito Santo, D. A., De Oliveira Rodrigues, B. R., Hadad Zaidan, F., and Silva Parreiras, F. (2018). Quality assessment of awareness support in agile collaborative tools. In Proceedings - 2018 44th Latin American Computing Conference, CLEI 2018, pages 21-30, São Paulo, SP, BR. Elsevier. DOI: 10.1109/clei.2018.00013.
Gallardo, J., Bravo, C., and Molina, A. I. (2018). A framework for the descriptive specification of awareness support in multimodal user interfaces for collaborative activities. Journal on Multimodal User Interfaces, 12(2):145-159. DOI: 10.1007/s12193-017-0255-x.
Gallardo, J., Molina, A. I., Bravo, C., Redondo, M., and Collazos, C. (2011). An ontological conceptualization approach for awareness in domain-independent collaborative modeling systems: Application to a model-driven development method. volume 38, pages 1099-1118. Elsevier BV. DOI: 10.1016/j.eswa.2010.05.005.
George, J. F. (2003). Groupware. In Bidgoli, H., editor, Encyclopedia of Information Systems, pages 509-518. Elsevier, New York, NY, USA. DOI: 10.1016/B0-12-227240-4/00084-8.
Greenberg, S. (1997). Collaborative interfaces for the web. Human factors and web development, 18:241-254. Available at:[link].
Gross, T. (2013). Supporting effortless coordination: 25 years of awareness research. Computer Supported Cooperative Work, 22(4-6):425-474. DOI: 10.1007/s10606-013-9190-x.
Gutwin, C., Bateman, S., Arora, G., and Coveney, A. (2017). Looking away and catching up: Dealing with brief attentional disconnection in synchronous groupware. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW '17, pages 2221-2235, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2998181.2998226.
Gutwin, C. and Greenberg, S. (2002). A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work, 11(3-4):411-446. DOI: 10.1023/a:1021271517844.
Gutwin, C., Greenberg, S., and Roseman, M. (1996). Workspace awareness in real-time distributed groupware: Framework, widgets, and evaluation. In Sasse, M. A., Cunningham, R. J., and Winder, R. L., editors, People and Computers XI, HCI'96, pages 281-298, London. Springer London. DOI: 10.1007/978-1-4471-3588-3-18.
Hakim, C. (1987). Research design: Strategies and choices in the design of social research. Number 13. Allen and Unwin. Bokk.
Herskovic, V., Ochoa, S. F., Pino, J. A., and Neyem, H. A. (2011). The iceberg effect: Behind the user interface of mobile collaborative systems. JUCS - Journal of Universal Computer Science, 17(2):183-201. DOI: 10.3217/jucs-017-02-0183.
Hincapie-Ramos, J. D., Voida, S., and Mark, G. (2011). A design space analysis of availability-sharing systems. In UIST'11 - Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, pages 85-95, Santa Barbara, CA, United states. Elsevier. DOI: 10.1145/2047196.2047207.
Hummel, D., Schacht, S., and Maedche, A. (2016). Determinants of multi-channel behavior: Exploring avenues for future research in the services industry. In ICIS, Dublin, Ireland. Fraunhofer-Gesellschaft. Avaialble at:[link].
ISO/IEC 25.040 (2011). Systems and software engineering - systems and software quality requirements and evaluation (square) - evaluation process. 1. DOI: 10.3403/30205731.
Jonasson, N. (2010). Designing for collaboration: Interface guidelines for improving co-located collaboration on tabletop displays. In Proceedings of Umea's 14th Student Conference in Computing Science - USCCS 2010, pages 61-71, Umea, Sweeden. Umea University. Available at:[link].
Khalilijafarabad, A., Helfert, M., and Ge, M. (2016). Developing a data quality research taxonomy-an organizational perspective. In ICIQ: International Conference on Information Quality, pages 176-186, Ciudad Real, Spain. Universidad de Castilla-La Mancha. Available at:[link].
Kim, K. T., Javed, W., Williams, C., Elmqvist, N., and Irani, P. (2010). Hugin: A framework for awareness and coordination in mixed-presence collaborative information visualization. In ACM International Conference on Interactive Tabletops and Surfaces, ITS 2010, pages 231-240, New York, NY, USA. ACM. DOI: 10.1145/1936652.1936694.
Kirsch-Pinheiro, M., De Lima, J. V., and Borges, M. R. (2003). A framework for awareness support in groupware systems. Computers in Industry, 52(1):47-57. DOI: 10.1016/s0166-3615(03)00068-x.
Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., and Rosenberg, J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering, 28(8):721–734. DOI: 10.1109/tse.2002.1027796.
Lauesen, S. and Vinter, O. (2001). Preventing requirement defects: An experiment in process improvement. Requirements Engineering, 6(1):37–50. DOI: 10.1007/pl00010355.
Lopez, G. and Guerrero, L. A. (2017). Awareness supporting technologies used in collaborative systems: A systematic literature review. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing, CSCW '17, pages 808-820, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2998181.2998281.
Mantau, M. J. and Benitti, F. B. V. (2022a). Awareness support in collaborative system: Reviewing last 10 years of cscw research. In 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 564-569, Hangzhou, China. IEEE. DOI: 10.1109/cscwd54268.2022.9776091.
Mantau, M. J. and Benitti, F. B. V. (2022b). Towards an awareness taxonomy. In 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), pages 495-500, Hangzhou, China. IEEE. DOI: 10.1109/cscwd54268.2022.9776129.
Mantau, M. J. and Benitti, F. B. V. (2023a). The awareness assessment model: measuring the awareness and collaboration support over the participant’s perspective. In Anais do XVIII Simpósio Brasileiro de Sistemas Colaborativos, pages 30-43. SBC. DOI: 10.5753/sbsc.2023.229064.
Mantau, M. J. and Benitti, F. B. V. (2023b). The awareness assessment model repository. Data repository/Internal Documentation. Zenodo. DOI: 10.5281/zenodo.8298950.
Mantau, M. J. and Benitti, F. B. V. (2024). The awareness assessment model: measuring awareness and collaboration support over participant’s perspective. Universal Access in the Information Society. DOI: 10.1007/s10209-024-01110-5.
Mantau, M. J., Berkenbrock, C. D. M., and Berkenbrock, G. R. (2014). Visualization and filtering awareness information in mobile groupware: An action research approach. In Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD), volume 1 of CSCWD'14, pages 563-568, National Tsing Hua University, Hsinchu, Taiwan. IEEE. DOI: 10.1109/cscwd.2014.6846906.
Mantau, M. J., Berkenbrock, C. D. M., and Berkenbrock, G. R. (2017). Análise de requisitos de percepção em um groupware móvel síncrono. Revista de Sistemas de Informação da FSMA, 1(19):16-33. Avaialble at:[link].
MATOOK, S. and INDULSKA, M. (2009). Improving the quality of process reference models: A quality function deployment-based approach. Decision support systems, 47(1):60-71. DOI: 10.1016/j.dss.2008.12.006.
Molina, A. I., Gallardo, J., Redondo, M. Á., and Bravo, C. (2015). Assessing the awareness mechanisms of a collaborative programming support system. Dyna, 82(193):212-222. DOI: 10.15446/dyna.v82n193.53497.
Neyem, A., Ochoa, S. F., Pino, J. A., and Franco, R. D. (2012). A reusable structural design for mobile collaborative applications. Journal of Systems and Software, 85(3):511-524. DOI: 10.1016/j.jss.2011.05.046.
Nickerson, R. C., Varshney, U., and Muntermann, J. (2013). A method for taxonomy development and its application in information systems. European Journal of Information Systems, 22(3):336-359. DOI: 10.1057/ejis.2012.26.
Niemantsverdriet, K., Essen, H. V., Pakanen, M., and Eggen, B. (2019). Designing for awareness in interactions with shared systems: The dass framework. ACM Transaction on Computer-Human Interaction, 26(6):1-41. DOI: 10.1145/3338845.
Pasquali, L. (2020). TRI-Teoria de resposta ao item: Teoria, procedimentos e aplicacoes. Editora Appris, Curitiba, PR, BR. Book.
Poulovassilis, A. and Xhafa, F. (2013). Building event-based services for awareness in p2p groupware systems. In Proceedings - 2013 8th International Conference on P2P, Parallel, Grid, Cloud and Internet Computing, 3PGCIC 2013, pages 200-207, Compiegne, France. ACM. DOI: 10.1109/3pgcic.2013.36.
Poulovassilis, A., Xhafa, F., and O'Hagan, T. (2015). Event-based awareness services for p2p groupware systems. Informatica (Netherlands), 26(1):135-157. DOI: 10.15388/informatica.2015.42.
Prouzeau, A., Bezerianos, A., and Chapuis, O. (2018). Awareness techniques to aid transitions between personal and shared workspaces in multi-display environments. In ISS 2018 - Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, pages 291-304, Tokyo, Japan. ACM. DOI: 10.1145/3279778.3279780.
Richardson, R. J. (2017). Pesquisa social: Métodos e técnicas. Editora Atlas SA, São Paulo, SP, BR. Book.
Rittgen, P. (2010). Quality and perceived usefulness of process models. SAC '10, page 65–72, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/1774088.1774105.
Rocker, C. (2012). Universal access to awareness information: Using smart artefacts to mediate awareness in distributed teams. Universal Access in the Information Society, 11(3):259-271. DOI: 10.1007/s10209-011-0237-9.
Rogers, Y., Sharp, H., and Preece, J. (2013). Design de interação. Bookman Editora, Porto Alegre, RS, BR. Book.
Rosqvist, T. (2003). Software Quality Journal, 11(1):39–55. DOI: 10.1023/a:1023741528816.
Santos, N. S., Ferreira, L. S., and Prates, R. O. (2012). An overview of evaluation methods for collaborative systems. In 2012 Brazilian Symposium on Collaborative Systems, São Paulo, SP, BR. IEEE. DOI: 10.1109/sbsc.2012.29.
Schmidt, K. and Randall, D. (2016). Preface to the special issue on 'reconsidering "awareness" in cscw'. Computer Supported Cooperative Work (CSCW), 25(4):229-233. DOI: 10.1007/s10606-016-9257-6.
Seebach, C., Beck, R., and Pahlke, I. (2011). Situation awareness through social collaboration platforms in distributed work environments. In International Conference on Information Systems 2011, ICIS 2011, volume 1, pages 1-22, Shanghai, China. Association for Information Systems. Available at:[link].
Souza, L. G. and Barbosa, S. D. J. (2015). Evaluating the molicc notation using the cognitive dimensions of notations framework. In Proceedings of the 14th Brazilian Symposium on Human Factors in Computing Systems, IHC '15, pages 1-10, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3148456.3148474.
Steinmacher, I., Chaves, A. P., and Gerosa, M. A. (2013). Awareness support in distributed software development: A systematic review and mapping of the literature. Computer Supported Cooperative Work (CSCW), 22(2-3). DOI: 10.1007/s10606-012-9164-4.
Steinmacher, I., Gerosa, M., and Chaves, A. P. (2010). Awareness support in global software development: A systematic review based on the 3c collaboration model. In Collaboration and Technology, pages 185-201, Berlin, Heidelberg. Springer Berlin Heidelberg. DOI: 10.1007/978-3-642-15714-1-15.
Sternberg, R. J., Sternberg, K., and Mio, J. (2012). Cognitive Psychology. Wadsworth: Cengage Learning, Belmont, California, USA. Book.
Strasser, A. (2017). Delphi method variants in information systems research: Taxonomy development and application. Electronic Journal of Business Research Methods, 15(2):120-133. DOI: 10.25968/opus-1164.
Szopinski, D., Schoormann, T., and Kundisch, D. (2019). Because your taxonomy is worth it: towards a framework for taxonomy evaluation. In 27th European Conference on Information Systems, Stockholm University, Kista, Sweden. Association for Information Systems. Available at:[link].
Talaei-Khoei, A., Ray, P., Parameshwaran, N., and Lewis, L. (2012). A framework for awareness maintenance. Journal of Network and Computer Applications, 35(1):199-210. DOI: 10.1016/j.jnca.2011.06.011.
Talaei-Khoei, A., Vichitvanichphong, S., Solvoll, T., Ray, P., and Ghapanchi, A. H. (2014). A methodology to develop awareness in computer supported collaborative work using policies. Journal of Computer and System Sciences, 80(7):1323-1338. DOI: 10.1016/j.jcss.2014.03.003.
Tenenberg, J., Roth, W.-M., and Socha, D. (2016). From i-awareness to we-awareness in cscw. Computer Supported Cooperative Work (CSCW), 25(4):235-278. DOI: 10.1007/s10606-014-9215-0.
Teruel, M. A., Navarro, E., Lopez-Jaquero, V., Montero, F., and Gonzalez, P. (2017). A comprehensive framework for modeling requirements of cscw systems. Journal of Software: Evolution and Process, 29(5):1-19. DOI: 10.1002/smr.1858.
Van Solingen, R. and Berghout, E. W. (1999). The Goal/Question/Metric Method: a practical guide for quality improvement of software development. McGraw-Hill International (UK) Limited, Berkshire, UK. Book.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in software engineering: an introduction. Springer Science & Business Media, New York, NY, USA. DOI: 10.1007/978-3-642-29044-2.
Xhafa, F., Palau, D., Barolli, L., Spaho, E., and Takizawa, M. (2013). Coordination in android mobile teams. In Proceedings - 16th International Conference on Network-Based Information Systems, NBiS 2013, pages 482-487, Gwangju, Republic of Korea. IEEE. DOI: 10.1109/nbis.2013.80.
Yang, C.-L., Yuan, C. W. T., Wang, T.-Y., and Wang, H.-C. (2018). Knowing that you know what i know helps? understanding the effects of knowledge transparency in online knowledge transfer. ACM Transaction on Computer-Human Interaction, 2(CSCW):1-21. DOI: 10.1145/3274458.
Yin, R. K. (2009). Case study research: Design and methods, volume 5. Sage Publications, Inc, London, UK. Book.
Yuill, N. and Rogers, Y. (2012). Mechanisms for collaboration: A design and evaluation framework for multi-user interfaces. ACM Transactions on Computer-Human Interaction, 19(1):1-25. DOI: 10.1145/2147783.2147784.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Márcio J. Mantau, Fabiane B.V. Benitti
This work is licensed under a Creative Commons Attribution 4.0 International License.
