Effects of Nonsensical Responses in Virtual Human Simulations on Clinicians’ Empathic Communication and Emotional Responses

Alexandre Gomes de Siqueira; Heng Yao; Sarah Bloch-Elkouby; Megan L. Rogers; Olivia C. Lawrence; Devon Peterkin; Sifan Zheng; Kathleen Feeney; Erica D. Musser; Igor Galynker; Benjamin Lok

doi:10.5753/jbcs.2024.4665

Authors

Alexandre Gomes de Siqueira University of Florida https://orcid.org/0000-0002-9213-9602
Heng Yao University of Florida https://orcid.org/0000-0002-7830-4114
Sarah Bloch-Elkouby Icahn School of Medicine https://orcid.org/0000-0003-1555-3137
Megan L. Rogers Texas State University https://orcid.org/0000-0002-4969-7035
Olivia C. Lawrence Columbia University https://orcid.org/0009-0008-0241-0616
Devon Peterkin Columbia University https://orcid.org/0000-0002-1517-5992
Sifan Zheng University of Pittsburgh https://orcid.org/0000-0002-5231-4787
Kathleen Feeney Florida International University https://orcid.org/0000-0002-8442-7866
Erica D. Musser Barnard College https://orcid.org/0000-0003-0966-4068
Igor Galynker Icahn School of Medicine https://orcid.org/0000-0001-8083-9479
Benjamin Lok University of Florida https://orcid.org/0000-0002-1190-3729

DOI:

https://doi.org/10.5753/jbcs.2024.4665

Keywords:

virtual human, virtual patient, empathy, empathy skills, nonsensical responses, suicide crises syndrome

Abstract

In this manuscript, we report on research that explores the application of virtual human patients to train clinicians on empathic communication skills. During training, clinicians received empathy scores as they interacted with two virtual humans portraying suicidal ideation, who at times provided nonsensical responses. We video-recorded clinicians' interactions with virtual humans and analyzed their facial expressions, as well as their verbal responses. In phase I of our study, we analyzed clinicians' facial expressions during three key moments: after a sensical response from a virtual human (baseline), following the last nonsensical response of the interaction, and after a sensical response that followed the last nonsensical response. In phase I, facial expressions were grouped into Negative (anger, disgust, sadness, and fear) and Positive (happiness, neutral, and surprise) facial affective behaviors. We observed that nonsensical responses from virtual humans can negatively affect clinicians' positive and negative facial affective behaviors. We found a significant increase in the percentage of time clinicians express negative facial affective behaviors immediately following nonsensical responses. In phase II, we recruited additional clinician-participants and investigated how different proportions of nonsensical responses affect clinicians' facial expressions of individual basic emotions (instead of groups of positive and negative facial expressions), as well as whether nonsensical responses moderate the association between expressions of basic emotions and empathy scores obtained by clinicians during training. We observed a statistically significant positive interaction between proportions of nonsensical responses and angry facial expressions in predicting average empathy scores. That is, the relationship between anger and empathy scores was significant at low and mean levels of nonsensical responses, but not at high levels. These results suggest that at low and mean levels nonsensical responses negatively impact clinicians' performance, hindering their ability to acquire empathy skills. We discuss the impacts of technological limitations during virtual human interactions, particularly nonsensical responses, and the importance of controlling for such issues.

Downloads

Download data is not yet available.

References

Assor, A., Kaplan, H., Kanat-Maymon, Y., and Roth, G. (2005). Directly controlling teacher behaviors as predictors of poor motivation and engagement in girls and boys: The role of anger and anxiety. Learning and instruction, 15(5):397-413. DOI: 10.1016/j.learninstruc.2005.07.008.

Bates, D., Mächler, M., Bolker, B., and Walker, S. (2014). Fitting linear mixed-effects models using lme4. arXiv preprint arXiv:1406.5823. DOI: 10.18637/jss.v067.i01.

Bloch-Elkouby, S., Barzilay, S., Gorman, B. S., Lawrence, O. C., Rogers, M. L., Richards, J., Cohen, L. J., Johnson, B. N., and Galynker, I. (2021). The revised suicide crisis inventory (sci-2): Validation and assessment of prospective suicidal outcomes at one month follow-up. Journal of affective disorders, 295:1280-1291. DOI: 10.1016/j.jad.2021.08.048.

Bloch-Elkouby, S., Gorman, B., Schuck, A., Barzilay, S., Calati, R., Cohen, L. J., Begum, F., and Galynker, I. (2020). The suicide crisis syndrome: A network analysis. Journal of counseling psychology, 67(5):595. DOI: 10.1037/cou0000423.

Boekaerts, M. (1994). Anger in relation to school learning. Learning and instruction, 3(4):269-280. DOI: 10.1016/0959-4752(93)90019-V.

Borish, M., Cordar, A., Foster, A., Kim, T., Murphy, J., and Lok, B. (2014). Utilizing real-time human-assisted virtual humans to increase real-world interaction empathy. Kansei Engineering & Emotion Research (KEER14), 15. DOI: 10.1109/IM.2003.1240281.

Bryk, A. S. and Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Sage Publications, Inc. DOI: 10.1080/00401706.1994.10485413.

Bylund, C. L. and Makoul, G. (2002). Empathic communication and gender in the physician-patient encounter. Patient education and counseling, 48(3):207-216. DOI: 10.1016/s0738-3991(02)00173-8.

Bylund, C. L. and Makoul, G. (2005). Examining empathy in medical encounters: an observational study using the empathic communication coding system. Health communication, 18(2):123-140. DOI: 10.1207/s15327027hc1802_2.

Cohen, A. S., Morrison, S. C., and Callaway, D. A. (2013). Computerized facial analysis for understanding constricted/blunted affect: initial feasibility, reliability, and validity data. Schizophrenia research, 148(1-3):111-116. DOI: 10.1016/j.schres.2013.05.003.

Cohen, P., West, S. G., and Aiken, L. S. (2014). Applied multiple regression/correlation analysis for the behavioral sciences. Psychology press. DOI: 10.4324/9780203774441.

Cootes, T. and Taylor, C. (2004). Statistical models of appearance for computer vision, imaging science and biomedical engineering. University of Manchester, 17:2011. DOI: 10.1007/3-540-44935-3_47.

Davis, S. and Wiedenbeck, S. (2001). The mediating effects of intrinsic motivation, ease of use and usefulness perceptions on performance in first-time and subsequent computer users. Interacting with computers, 13(5):549-580. DOI: 10.1016/s0953-5438(01)00034-0.

Desai, M., Kaniarasu, P., Medvedev, M., Steinfeld, A., and Yanco, H. (2013). Impact of robot failures and feedback on real-time trust. In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 251-258. IEEE. DOI: 10.1109/hri.2013.6483596.

Desai, M., Medvedev, M., Vázquez, M., McSheehy, S., Gadea-Omelchenko, S., Bruggeman, C., Steinfeld, A., and Yanco, H. (2012). Effects of changing reliability on trust of robot systems. In 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 73-80. IEEE. DOI: 10.1145/2157689.2157702.

DeVault, D., Artstein, R., Benn, G., Dey, T., Fast, E., Gainer, A., Georgila, K., Gratch, J., Hartholt, A., Lhommet, M., et al. (2014). Simsensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 1061-1068. DOI: 10.1609/aaai.v29i1.9777.

Dixon, W. J. and Mood, A. M. (1946). The statistical sign test. Journal of the American Statistical Association, 41(236):557-566. DOI: 10.1037/e417432004-001.

Ekman, P. and Cordaro, D. (2011). What is meant by calling emotions basic. Emotion review, 3(4):364-370. DOI: 10.1177/1754073911410740.

Ekman, P., Sorenson, E. R., and Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164(3875):86-88. DOI: 10.1126/science.164.3875.86.

Foster, A., Chaudhary, N., Kim, T., Waller, J. L., Wong, J., Borish, M., Cordar, A., Lok, B., and Buckley, P. F. (2016). Using virtual patients to teach empathy: a randomized controlled study to enhance medical students empathic communication. Simulation in Healthcare, 11(3):181-189. DOI: 10.1097/sih.0000000000000142.

Galynker, I. (2017). The suicidal crisis: Clinical guide to the assessment of imminent suicide risk. Oxford University Press. DOI: 10.1093/med/9780197582718.002.0001.

Gomes de Siqueira, A., Yao, H., Bafna, A., Bloch-Elkouby, S., Richards, J., Lloveras, L. B., Feeney, K., Morris, S., Musser, E. D., Lok, B., et al. (2021). Investigating the effects of virtual patients nonsensical responses on users facial expressions in mental health training scenarios. In Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, pages 1-10. DOI: 10.1145/3489849.3489864.

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in quantitative methods for psychology, 8(1):23. Available online [link].

Holland, N. N. (1967). The" willing suspension of disbelief" revisited. Centennial Review, pages 1-23. DOI: 10.5040/9781474218627.ch-001.

Kim, S. S., Kaplowitz, S., and Johnston, M. V. (2004). The effects of physician empathy on patient satisfaction and compliance. Evaluation & the health professions, 27(3):237-251. DOI: 10.1177/0163278704267037.

Kim, Y. H., Kim, D. J., and Wachter, K. (2013). A study of mobile user engagement (moen): Engagement motivations, perceived value, satisfaction, and continued engagement intention. Decision support systems, 56:361-370. DOI: 10.1016/j.dss.2013.07.002.

Kleinsmith, A., Rivera-Gutierrez, D., Finney, G., Cendan, J., and Lok, B. (2015). Understanding empathy training with virtual patients. Computers in human behavior, 52:151-158. DOI: 10.1016/j.chb.2015.05.033.

Krieger, J. L., Neil, J. M., Duke, K. A., Zalake, M. S., Tavassoli, F., Vilaro, M. J., Wilson-Howard, D. S., Chavez, S. Y., Laber, E. B., Davidian, M., et al. (2021). A pilot study examining the efficacy of delivering colorectal cancer screening messages via virtual health assistants. American journal of preventive medicine. DOI: 10.1016/j.amepre.2021.01.014.

Kuznetsova, A., Brockhoff, P. B., and Christensen, R. H. (2017). lmertest package: tests in linear mixed effects models. Journal of statistical software, 82:1-26. DOI: 10.18637/jss.v082.i13.

Lewinski, P., den Uyl, T. M., and Butler, C. (2014). Automated facial coding: validation of basic emotions and facs aus in facereader. Journal of Neuroscience, Psychology, and Economics, 7(4):227. DOI: 10.1037/npe0000028.supp.

Lucas, G. M., Boberg, J., Traum, D., Artstein, R., Gratch, J., Gainer, A., Johnson, E., Leuski, A., and Nakano, M. (2018). Culture, errors, and rapport-building dialogue in social agents. In Proceedings of the 18th International Conference on intelligent virtual agents, pages 51-58. DOI: 10.1145/3267851.3267887.

Muckler, V. C. (2017). Exploring suspension of disbelief during simulation-based learning. Clinical Simulation in Nursing, 13(1):3-9. DOI: 10.1016/j.ecns.2016.09.004.

Nass, C., Steuer, J., and Tauber, E. R. (1994). Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 72-78. DOI: 10.1016/b978-155860643-2/50007-x.

Pinto, M. D., Greenblatt, A. M., Hickman, R. L., Rice, H. M., Thomas, T. L., and Clochesy, J. M. (2015). Assessing the critical parameters of esmart-mh: A promising avatar-based digital therapeutic intervention to reduce depressive symptoms. Perspectives in psychiatric care, 52(3):157-168. DOI: 10.1111/ppc.12112.

Rizzo, A., Reger, G., Perlman, K., Rothbaum, B., Difede, J., McLay, R., Graap, K., Gahm, G., Johnston, S., Deal, R., et al. (2011). Virtual reality posttraumatic stress disorder (ptsd) exposure therapy results with active duty oif/oef service members. DOI: 10.1515/ijdhd.2011.060.

Ross, J. and Watling, C. (2017). Use of empathy in psychiatric practice: constructivist grounded theory study. BJPsych open, 3(1):26-33. DOI: 10.1192/bjpo.bp.116.004242.

Rossen, B. and Lok, B. (2012). A crowdsourcing method to develop virtual human conversational agents. International Journal of Human-Computer Studies, 70(4):301-319. DOI: 10.1016/j.ijhcs.2011.11.004.

Schuck, A., Calati, R., Barzilay, S., Bloch-Elkouby, S., and Galynker, I. (2019). Suicide crisis syndrome: A review of supporting evidence for a new suicide-specific diagnosis. Behavioral sciences & the law, 37(3):223-239. DOI: 10.1002/bsl.2397.

Skarbez, R., Kotranza, A., Brooks, F. P., Lok, B., and Whitton, M. C. (2011). An initial exploration of conversational errors as a novel method for evaluating virtual human experiences. In 2011 IEEE Virtual Reality Conference, pages 243-244. IEEE. DOI: 10.1109/vr.2011.5759489.

Stevens, A., Hernandez, J., Johnsen, K., Dickerson, R., Raij, A., Harrison, C., DiPietro, M., Allen, B., Ferdig, R., Foti, S., et al. (2006). The use of virtual patients to teach medical students history taking and communication skills. The American Journal of Surgery, 191(6):806-811. DOI: 10.5005/jp/books/12827_18.

Stuart, J., Akinnola, I., Guido-Sanz, F., Anderson, M., Diaz, D., Welch, G., and Lok, B. (2020). Applying stress management techniques in augmented reality: Stress induction and reduction in healthcare providers during virtual triage simulation. In 2020 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW), pages 171-172. IEEE. DOI: 10.1109/vrw50115.2020.00037.

Stuart, J., Aul, K., Bumbach, M. D., Stephen, A., de Siqueira, A. G., and Lok, B. (2022). The effect of virtual humans making verbal communication mistakes on learners perspectives of their credibility, reliability, and trustworthiness. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 455-463. IEEE. DOI: 10.1109/vr51125.2022.00065.

Wang, Y., Khooshabeh, P., and Gratch, J. (2013). Looking real and making mistakes. In International Workshop on Intelligent Virtual Agents, pages 339-348. Springer. DOI: 10.1007/978-3-642-40415-3_30.

Wind, L. A., Van Dalen, J., Muijtjens, A. M., and Rethans, J.-J. (2004). Assessing simulated patients in an educational setting: the masp (maastricht assessment of simulated patients). Medical Education, 38(1):39-44. DOI: 10.1111/j.1365-2923.2004.01686.x.

Yao, H., de Siqueira, A. G., Foster, A., Galynker, I., and Lok, B. (2020). Toward automated evaluation of empathetic responses in virtual human interaction systems for mental health scenarios. In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pages 1-8. DOI: 10.1145/3383652.3423916.