Domain Learning from Data for Large Language Model Translation and Adaptation
DOI:
https://doi.org/10.5753/jbcs.2025.5795Keywords:
Translation, Domain Adaptation, Portuguese, Large Language Model, LLM, PromptAbstract
Large Language Models (LLMs) have improved multilingual translation and adaptation, particularly for languages like Portuguese; however, they often fail to produce outputs that accurately reflect the linguistic, stylistic, and topical characteristics expected in real-world scenarios. This paper addresses the challenge of adapting texts to specific domains and audiences by moving beyond direct translation to include variations in genre and topic. We propose a method for learning domain representation vectors through prompt tuning, allowing LLMs to generate text that matches the communicative norms of a target domain or user profile (e.g., legal discourse, informal speech, or social media posts) or even topics. In contrast to most domain adaptation approaches that focus solely on translation, our method supports broader text adaptation and can be applied to multiple tasks. We demonstrate the effectiveness of our approach using two Portuguese datasets—a newly compiled corpus of video game discussions and a financial tweet corpus—and evaluate the results with respect to linguistic variation. Our main contributions include: (i) a method for learning reusable domain vectors to support prompt-based adaptation; (ii) application to translation and broader text adaptation tasks; and (iii) the release of a new domain-specific dataset in Portuguese.
Downloads
References
Banerjee, S. and Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Goldstein, J., Lavie, A., Lin, C.-Y., and Voss, C., editors, Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, Michigan. Association for Computational Linguistics. Available online [link].
Cai, D., Wang, Y., Li, H., Lam, W., and Liu, L. (2021). Neural machine translation with monolingual translation memory. In Zong, C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7307-7318, Online. Association for Computational Linguistics. DOI: 10.18653/v1/2021.acl-long.567.
Chu, C. and Wang, R. (2018). A survey of domain adaptation for neural machine translation. In Bender, E. M., Derczynski, L., and Isabelle, P., editors, Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, USA. Association for Computational Linguistics. Available online [link].
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics. DOI: 10.18653/v1/N19-1423.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024). A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. DOI: 10.48550/arxiv.2411.15594.
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the knowledge in a neural network. DOI: 10.48550/arXiv.1503.02531.
Koehn, P. and Knowles, R. (2017). Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28-39. DOI: 10.18653/v1/w17-3204.
Koehn, P. and Senellart, J. (2010). Convergence of translation memory and statistical machine translation. In Zhechev, V., editor, Proceedings of the Second Joint EM+/CNGL Workshop: Bringing MT to the User: Research on Integrating MT in the Translation Industry, Denver, Colorado, USA. Association for Machine Translation in the Americas. Available online [link].
Lester, B., Al-Rfou, R., and Constant, N. (2021). The power of scale for parameter-efficient prompt tuning. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. DOI: 10.48550/arXiv.2104.08691.
Li, X., Zhang, J., and Zong, C. (2018). One sentence one model for neural machine translation. In Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S., and Tokunaga, T., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). DOI: 10.48550/arXiv.1609.06490.
Liu, J., Zhang, X., Tian, X., Wang, J., and Sangaiah, A. K. (2020). A novel domain adaption approach for neural machine translation. Int. J. Comput. Sci. Eng., 22(4):445–453. DOI: 10.1504/ijcse.2020.109404.
Liu, X., Zeng, J., Wang, X., Wang, Z., and Su, J. (2024). Exploring iterative dual domain adaptation for neural machine translation. Knowledge-Based Systems, 283:111182. DOI: 10.1016/j.knosys.2023.111182.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. (2020). Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc. Available online [link].
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. (2022). Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316. DOI: 10.18653/v1/2023.eacl-main.148.
OpenAI (2024). Gpt-4.1. Available online [link].
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2024). Gpt-4 technical report. DOI: 10.48550/arxiv.2303.08774.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA. Association for Computational Linguistics. DOI: 10.3115/1073083.1073135.
Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-Smadi, M., Al-Ayyoub, M., Zhao, Y., Qin, B., De Clercq, O., Hoste, V., Apidianaki, M., Tannier, X., Loukachevitch, N., Kotelnikov, E., Bel, N., Jiménez-Zafra, S. M., and Eryiğit, G. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), SemEval'16, pages 19-30, San Diego, California. Association for Computational Linguistics. DOI: 10.18653/v1/S16-1002.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. Available online [link].
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2023). Exploring the limits of transfer learning with a unified text-to-text transformer. DOI: 10.48550/arxiv.1910.10683.
Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Inui, K., Jiang, J., Ng, V., and Wan, X., editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics. DOI: 10.18653/v1/D19-1410.
Ríos-Toledo, G., Posadas-Durán, J. P. F., Sidorov, G., and Castro-Sánchez, N. A. (2022). Detection of changes in literary writing style using n-grams as style markers and supervised machine learning. Plos one, 17(7):e0267590. DOI: 10.1371/journal.pone.0267590.
Santini, M. (2004). State-of-the-art on automatic genre identification. Technical report, Technical Report No. ITRI-04-03). Information Technology Research Institute. Available online [link].
Saunders, D. (2022). Domain adaptation and multi-domain adaptation for neural machine translation: A survey. DOI: 10.1613/jair.1.13566.
Sennrich, R., Haddow, B., and Birch, A. (2016). Improving neural machine translation models with monolingual data. In Erk, K. and Smith, N. A., editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany. Association for Computational Linguistics. DOI: 10.48550/arXiv.1511.06709.
Sinha, A., Kedas, S., Kumar, R., and Malo, P. (2022). Sentfin 1.0: Entity-aware sentiment analysis for financial news. Journal of the Association for Information Science and Technology, 73(9):1314-1335. DOI: 10.1002/asi.24634.
Stergiadis, E., Kumar, S., Kovalev, F., and Levin, P. (2021). Multi-domain adaptation in neural machine translation through multidimensional tagging. In Campbell, J., Huyck, B., Larocca, S., Marciano, J., Savenkov, K., and Yanishevsky, A., editors, Proceedings of Machine Translation Summit XVIII: Users and Providers Track, Virtual. Association for Machine Translation in the Americas. DOI: 10.48550/arXiv.2102.10160.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the inception architecture for computer vision. Available online [link].
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. (2023). Llama: Open and efficient foundation language models. DOI: 10.48550/arxiv.2302.13971.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2023). Attention is all you need. Available online [link].
You, W., Guo, P., Li, J., Chen, K., and Zhang, M. (2024). Efficient domain adaptation for non-autoregressive machine translation. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 13657-13670, Bangkok, Thailand. Association for Computational Linguistics. DOI: 10.18653/v1/2024.findings-acl.810.
Yu, J., Zhao, Q., and Xia, R. (2023). Cross-domain data augmentation with domain-adaptive language modeling for aspect-based sentiment analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, volume 1 of ACL'23, pages 1456-1470, Toronto, Canada. Association for Computational Linguistics. DOI: 10.18653/v1/2023.acl-long.81.
Zerbinati, M. M., Roman, N. T., and Felippo, A. D. (2024). A corpus of stock market tweets annotated with named entities. In Gamallo, P., Claro, D., Teixeira, A., Real, L., Garcia, M., Oliveira, H. G., and Amaro, R., editors, Proceedings of the 16th International Conference on Computational Processing of Portuguese - Vol. 1, Santiago de Compostela, Galicia/Spain. Association for Computational Lingustics. Available online [link].
Zhang, B., Haddow, B., and Birch, A. (2023a). Prompting large language model for machine translation: A case study. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 41092-41110. PMLR. Available online [link].
Zhang, K., Liu, Q., Qian, H., Xiang, B., Cui, Q., Zhou, J., and Chen, E. (2023b). Eatn: An efficient adaptive transfer network for aspect-level sentiment analysis. volume 35, pages 377-389. DOI: 10.1109/TKDE.2021.3075238.
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. DOI: 10.48550/arxiv.1904.09675.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. (2023). Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623. DOI: 10.48550/arxiv.2306.05685.
Zhou, H., Hou, Y., Li, Z., Wang, X., Wang, Z., Duan, X., and Zhang, M. (2023). How well do large language models understand syntax? an evaluation by asking natural language questions. arXiv preprint arXiv:2311.08287. DOI: 10.48550/arxiv.2311.08287.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 René Vieira Santin, Ricardo Marcondes Marcacini, Solange Oliveira Rezende
This work is licensed under a Creative Commons Attribution 4.0 International License.
