A benchmark for Portuguese zero-shot stance detection
DOI:
https://doi.org/10.5753/jbcs.2024.3932Keywords:
Natural Language Processing, Stance detection, Zero-shotAbstract
Stance detection is the task of inferring for/against attitudes towards a particular target from text. As targets are in principle unlimited, however, research in the field has moved from so-called in-domain classification (which assume the availability of a sufficient number of stances towards the intended target for training purposes) to more realistic zero-shot scenarios. However, regardless of which - or how much - training data is taken into account, most existing zero-shot approaches are devoted to the English language, in stark opposition to alternatives devoted to Portuguese. As a means to overcome some of these difficulties, this article presents a benchmark (hereby understood as the combination of a dataset, baseline systems and their results) for zero-shot Portuguese stance detection that is, to the best of our knowledge, the first of it kind. More specifically, we adapt a number of existing models available for the English language to Portuguese, and introduce novel approaches to the task based on more recent prompt engineering methods and off-task labelling, achieving SOTA results that are, in some cases, even superior to in-domain classification.
Downloads
References
Aldayel, A. and Magdy, W. (2021). Stance detection on social media: State of the art and trends. Information Processing & Management, 58(4):102597. DOI: 10.1016/j.ipm.2021.102597.
Allaway, E. and McKeown, K. R. (2020). Zero-shot stance detection: A dataset and model using generalized topic representations. In EMNLP-2020 proceedings, pages 8913-8931, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-main.717.
Allaway, E. and McKeown, K. R. (2022). Zero-shot stance detection: Paradigms and challenges. Frontiers in Artificial Intelligence, 5:1070429. DOI: 10.3389/frai.2022.1070429.
Allaway, E., Srikanth, M., and McKeown, K. (2021). Adversarial learning for zero-shot stance detection on social media. In ACL-2021 proceedings, pages 4756-4767, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2021.naacl-main.379.
Alturayeif, N., Luqman, H., and Ahmed, M. (2023). A systematic review of machine learning techniques for stance detection and its applications. Neural Comput & Applic, 35:5113-5144. DOI: 10.1007/s00521-023-08285-7.
Augenstein, I., Rocktäschel, T., Vlachos, A., and Bontcheva, K. (2016). Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas. Association for Computational Linguistics. DOI: 10.18653/v1/D16-1084.
Chunling, W., Yijia, Z., Xingyu, Y., Guantong, L., Fei, C., and Hongfei, L. (2023). Adversarial network with external knowledge for zero-shot stance detection. In Proceedings of the 22nd Chinese National Conference on Computational Linguistics, pages 824-835, Harbin, China. Chinese Information Processing Society of China. DOI: 10.1007/978-981-99-6207-5_26.
Cignarella, A., Lai, M., Bosco, C., Patti, V., and Rosso, P. (2020). SardiStance @ EVALITA2020: Overview of the task on stance detection in italian tweets. In CEURS Proceedings vol. 2765, pages 177-186, online. CEUR-WS.org. Available online [link].
da Costa, P. B., Pavan, M. C., dos Santos, W. R., da Silva, S. C., and Paraboni, I. (2023). BERTabaporu: assessing a genre-specific language model for Portuguese NLP. In Recents Advances in Natural Language Processing (RANLP-2023), pages 217-223, Varna, Bulgaria. Available online [link].
da Silva, S. C., Ferreira, T. C., Ramos, R. M. S., and Paraboni, I. (2020). Data driven and psycholinguistics motivated approaches to hate speech detection. Computación y Systemas, 24(3):1179-1188. DOI: 10.13053/cys-24-3-3478.
da Silva, S. C. and Paraboni, I. (2023). Politically-oriented information inference from text. Journal of Universal Computer Science, 29(6):570-595. DOI: 10.3897/jucs.96652.
de Sousa, A. and Becker, K. (2023). SSSD: Leveraging pre-trained models and semantic search for semi-supervised stance detection. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 264-273, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. Available online [link].
Delmondes Neto, J. P. and Paraboni, I. (2021). Multi-source BERT stack ensemble for cross-domain author profiling. Expert Systems, pages -. DOI: 10.1111/exsy.12869.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-2019 proceedings, pages 4171-4186, Minneapolis, USA. Assoc. for Computational Linguistics. DOI: 10.18653/v1/N19-1423.
Flores, A. M., Pavan, M. C., and Paraboni, I. (2021). User profiling and satisfaction inference in public information access services. Journal of Intelligent Information Systems, pages -. DOI: 10.1007/s10844-021-00661-w.
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop.. DOI: 10.48550/arXiv.1503.02531.
Hosseinia, M., Dragut, E., and Mukherjee, A. (2020). Stance prediction for contemporary issues: Data and experiments. In 8th International Workshop on Natural Language Processing for Social Media, pages 32-40, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2020.socialnlp-1.5.
Jaziriyan, M. M., Akbari, A., and Karbasi, H. (2021). ExaASC: A General Target-Based Stance Detection Corpus in Arabic Language. In 11th International Conference on Computer Engineering and Knowledge (ICCKE), pages 424-429, Mashhad, Iran. IEEE. DOI: 10.1109/ICCKE54056.2021.9721486.
Kochkina, E., Liakata, M., and Augenstein, I. (2017). Turing at SemEval-2017 task 8: Sequential approach to rumour stance classification with branch-LSTM. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 475-480, Vancouver, Canada. Association for Computational Linguistics. DOI: 10.18653/v1/S17-2083.
Kucuk, D. and Can, F. (2020). Stance detection: A survey. ACM Computing Surveys, 53(1):1-37. DOI: 10.1145/3369026.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2020). BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics. DOI: 10.48550/arXiv.1910.13461.
Li, Y., Sosea, T., Sawant, A., Nair, A. J., Inkpen, D., and Caragea, C. (2021). P-stance: A large dataset for stance detection in political domain. In Findings of ACL-IJCNLP-2021, pages 2355-2365, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2021.findings-acl.208.
Liang, B., Zhu, Q., Li, X., Yang, M., Gui, L., He, Y., and Xu, R. (2022). JointCL: A joint contrastive learning framework for zero-shot stance detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 81-91, Dublin, Ireland. Association for Computational Linguistics. DOI: 10.18653/v1/2022.acl-long.7.
Liu, R., Lin, Z., Tan, Y., and Wang, W. (2021). Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph. In Findings of ACL-IJCNLP-2021, pages 3152-3157, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2021.findings-acl.278.
Luo, Y., Liu, Z., Shi, Y., Li, S. Z., and Zhang, Y. (2022). Exploiting sentiment and common sense for zero-shot stance detection. In Proceedings of the 29th International Conference on Computational Linguistics, pages 7112-7123, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. DOI: 10.48550/arXiv.2208.08797.
Mikolov, T., Wen-tau, S., and Zweig, G. (2013). Linguistic regularities in continuous space word representations. In Proc. of NAACL-HLT-2013, pages 746-751, Atlanta, USA. Assoc. for Computational Linguistics. Available online [link].
Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., and Cherry, C. (2016). SemEval-2016 task 6: Detecting stance in tweets. In 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31-41, San Diego, California. Assoc. for Computational Linguistics. DOI: 10.18653/v1/S16-1003.
Oshikawa, R., Qian, J., and Wang, W. Y. (2020). A survey on natural language processing for fake news detection. In 12th Language Resources and Evaluation Conference, pages 6086-6093, Marseille, France. European Language Resources Association. Available online [link].
Pavan, M. C., dos Santos, V. G., Lan, A. G. J., ao Trevisan Martins, J., dos Santos, W. R., Deutsch, C., da Costa, P. B., Hsieh, F. C., and Paraboni, I. (2023). Morality classification in natural language text. IEEE transactions on Affective Computing, 14(1):857-863. DOI: 10.1109/TAFFC.2020.3034050.
Pavan, M. C., dos Santos, W. R., and Paraboni, I. (2020). Twitter Moral Stance Classification using Long Short-Term Memory Networks. In BRACIS-2020 proceedings LNAI 12319, pages 636-647. Springer. DOI: 10.1007/978-3-030-61377-8_28.
Pavan, M. C. and Paraboni, I. (2022). Cross-target stance classification as domain adaptation. In Pichardo Lagunas, O., Martínez-Miranda, J., and Martínez Seis, B., editors, Advances in Computational Intelligence - MICAI 2022 - Lecture Notes in Artificial Intelligence vol 13612, pages 15-25, Cham. Springer Nature Switzerland. DOI: 10.1007/978-3-031-19493-1.
Pennington, J., Socher, R., and Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. In EMNLP-2014 proceedings, pages 1532-1543. DOI: 10.3115/v1/D14-1162.
Pereira, C., Pavan, M., Yoon, S., Ramos, R., Costa, P., Cavalheiro, L., and Paraboni, I. (2023). UstanceBR: a multimodal language resource for stance prediction. arXiv:2312.06374. DOI: 10.48550/arXiv.2312.06374.
Ryu, M. and Lee, K. (2022). Knowledge distillation for BERT unsupervised domain adaptation. Knowledge Information Systems, 64:3113-3128. DOI: 10.1007/s10115-022-01736-y.
Schlichtkrull, M., Kipf, T. N., Bloem, P., van den Berg, R., Titov, I., and Welling, M. (2018). Modeling relational data with graph convolutional networks. In The Semantic Web, pages 593-607, Cham. Springer International Publishing. DOI: 10.1007/978-3-319-93417-4_38.
Schmidt, A. and Wiegand, M. (2017). A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media, pages 1-10, Valencia, Spain. Association for Computational Linguistics. DOI: 10.18653/v1/W17-1101.
Speer, R., Chin, J., and Havasi, C. (2017). ConceptNet 5.5: An Open Multilingual Graph of General Knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, pages 4444-4451. AAAI Press. DOI: 10.1609/aaai.v31i1.11164.
Taulé, M., Martí, M. A., Pardo, F. M. R., Rosso, P., Bosco, C., and Patti, V. (2017). Overview of the task on stance and gender detection in tweets on catalan independence at IberEval 2017. In IberEval-2017 proceedings, pages 157-177, Murcia, Spain. CEUR-WS.org. Available online [link].
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv. DOI: 10.48550/arXiv.2302.13971.
Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. (2017). Adversarial discriminative domain adaptation. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2962-2971. DOI: 10.1109/CVPR.2017.316.
Vashishth, S., Sanyal, S., Nitin, V., and Talukdar, P. P. (2020). Composition-based multirelational graph convolutional networks. In 8th International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia. DOI: 10.48550/arXiv.1911.03082.
Wen, H. and Hauptmann, A. (2023). Zero-shot and few-shot stance detection on varied topics via conditional generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1491-1499, Toronto, Canada. Association for Computational Linguistics. DOI: 10.18653/v1/2023.acl-short.127.
Xu, C., Paris, C., Nepal, S., and Sparks, R. (2018). Cross-target stance classification with self-attention networks. In ACL-2018 proceedings, pages 778-783, Melbourne, Australia. Assoc. for Computational Linguistics. DOI: 10.18653/v1/P18-2123.
Xu, H., Vucetic, S., and Yin, W. (2022). OpenStance: Real-world zero-shot stance detection. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 314-324, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. DOI: 10.18653/v1/2022.conll-1.21.
Yang, R., Xie, W., Liu, C., and Yu, D. (2019). BLCU_NLP at SemEval-2019 task 7: An inference chain-based GPT model for rumour evaluation. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1090-1096, Minneapolis, Minnesota, USA. Association for Computational Linguistics. DOI: 10.18653/v1/S19-2191.
Zarrella, G. and Marsh, A. (2016). MITRE at SemEval-2016 task 6: Transfer learning for stance detection. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 458-463, San Diego, California. Association for Computational Linguistics. DOI: 10.18653/v1/S16-1074.
Zhang, B., Ding, D., and Jing, L. (2023). How would stance detection techniques evolve after the launch of chatgpt?
Zhang, B., Yang, M., Li, X., Ye, Y., Xu, X., and Dai, K. (2020). Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In ACL-2020 proceedings, pages 3188-3197, Online. Assoc. for Computational Linguistics. DOI: 10.18653/v1/2020.acl-main.291.
Zhang, Y., Barzilay, R., and Jaakkola, T. S. (2017). Aspect-augmented adversarial networks for domain adaptation. Trans. Assoc. Comput. Linguistics, 5:515-528. DOI: 10.48550/arXiv.1701.00188.
Zhao, X., Zou, J., Zhang, Z., Xie, F., Zhou, B., and Tian, L. (2022). Zero-shot stance detection based on cross-domain feature enhancement by contrastive learning. arXiv:2210.03380. DOI: 10.48550/arXiv.2210.03380.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Matheus Camasmie Pavan, Ivandré Paraboni
This work is licensed under a Creative Commons Attribution 4.0 International License.
