Designing Intelligent Multimodal Assistants for Digital Humanities: A Comparative Study of Models, Modalities, and Domains

Alexander Sergeev; Valeriya Buzmakova; Evgeny Kotelnikov

doi:10.5753/jis.2026.7338

Authors

Alexander Sergeev European University at Saint Petersburg https://orcid.org/0009-0002-4103-842X
Valeriya Buzmakova European University at Saint Petersburg https://orcid.org/0000-0003-1167-2606
Evgeny Kotelnikov European University at Saint Petersburg https://orcid.org/0000-0001-9745-1489

DOI:

https://doi.org/10.5753/jis.2026.7338

Keywords:

Multimodal Large Language Models, Intelligent Assistants, Digital Humanities, Retrieval-Augmented Generation, Hybrid Search, LLM-as-Judge

Abstract

The study addresses the problem of underutilization of the capabilities of modern Multimodal Large Language Models in the Digital Humanities. We propose an architecture for a scalable and adaptive multimodal virtual intelligent assistant based on the Retrieval-Augmented Generation approach. The architecture was tested on heterogeneous data in Russian from three subject domains: history, literature, and botany. Three modality processing strategies were used: text-only, images-only, and combined (text and images). In the retrieval experiments, full-text search, semantic text search, image search, and hybrid variants were compared. The most effective combination proved to be full-text search, semantic text search using the KaLM-Embedding model, and image search using the MetaCLIP 2 model. In the generation experiments, six open-weight Multimodal LLMs were compared using the LLM-as-Judge approach. The most performative model for the combined modality was Gemma 3, while for the text-only modality it was Qwen-3-VL Instruct. When working with images all models demonstrated relatively low performance. The study confirms the practical applicability of the proposed architecture and identifies candidate models for developing virtual intelligent assistants capable of working with complex multimodal databases in digital humanities projects.

Downloads

Download data is not yet available.

References

Chen, J., Xiao, S., Zhang, P., Luo, K., Lian, D., and Liu, Z. (2024). M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bangkok, Thailand. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2024.findings-acl.137.

Chuang, Y.-S., Li, Y., Wang, D., Yeh, C.-F., Lyu, K., Raghavendra, R., Glass, J., Huang, L., Weston, J., Zettlemoyer, L., Chen, X., Liu, Z., Xie, S., tau Yih, W., Li, S.-W., and Xu, H. (2025). Meta clip 2: A worldwide scaling recipe. DOI: https://doi.org/10.48550/arXiv.2507.22062.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. DOI: https://doi.org/10.48550/arXiv.2010.11929.

Hong, Z., Yuan, Z., Zhang, Q., Chen, H., Dong, J., Huang, F., and Huang, X. (2025). Next-generation database interfaces: A survey of llm-based text-to-sql. IEEE Transactions on Knowledge and Data Engineering, 37(12):7328–7345. DOI: https://doi.org/10.1109/TKDE.2025.3609486.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.. DOI: https://doi.org/10.48550/arXiv.2005.11401.

Li, D., Jiang, D., Huang, L., Beigi, A., Zhao, C., Tan, Z., Bhattacharjee, A., Jiang, Y., Chen, C., Wu, T., Shu, K., Cheng, L., and Liu, H. (2025). From generation to judgment: Opportunities and challenges of llm-as-a-judge. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2757–2791, Suzhou, China. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2025.emnlp-main.138.

Lima, B. M. d., Cardoso, F. E. A., Melo, R. F. d., and Carneiro, L. d. A. (2026). Development of a rag-based chatbot to support multiprofessional services in public education. REMUNOM, 13(11):1–37. DOI: https://doi.org/10.66104/14p87804.

Miftah, K., Matrane, Y., Ellaky, Z., and Benabbou, F. (2026). Chatbot for the preservation of palestine's history. In Idrissi, N., Hair, A., Lazaar, M., Saadi, Y., Chakib, H., Erritali, M., and El Kafhali, S., editors, Artificial Intelligence and Green Computing, pages 402–413, Cham. Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-032-02312-4_32.

Murugadoss, B., Poelitz, C., Drosos, I., Le, V., McKenna, N., Negreanu, C. S., Parnin, C., and Sarkar, A. (2025). Evaluating the evaluator: Measuring LLMs' adherence to task evaluation instructions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39 of AAAI-25. DOI: https://doi.org/10.1609/aaai.v39i18.34157.

Nagare, Pravin Vishnu and Shirke, Prajakta (2025). Advances in multimodal ai-powered chatbots: A comprehensive review and proposed efficient architecture. EPJ Web Conf., 341:01049. DOI: https://doi.org/10.1051/epjconf/202534101049.

Neupane, S., Hossain, E., Keith, J., Tripathi, H., Ghiasi, F., Golilarz, N. A., Amirlatifi, A., Mittal, S., and Rahimi, S. (2024). From questions to insightful answers: Building an informed chatbot for university resources. Accessed: 20 June 2026.. DOI: https://doi.org/10.48550/arXiv.2405.08120.

Quidwai, M. A. and Lagana, A. (2024). A rag chatbot for precision medicine of multiple myeloma. medRxiv. DOI: https://doi.org/10.1101/2024.03.14.24304293.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR. DOI: https://doi.org/10.48550/arXiv.2103.00020.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-betworks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992. DOI: https://doi.org/10.18653/v1/D19-1410.

Robertson, S. and Zaragoza, H. (2009). The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. DOI: https://doi.org/10.1561/1500000019.

Sergeev, A., Goloviznina, V., Melnichenko, M., and Kotelnikov, E. (2026). Talking to data: Designing smart assistants for humanities databases. In Bakaev, M., Bolgov, R., Chizhik, A., Chugunov, A., Demareva, V., Kabanov, Y., Pereira, R., R., E., and Zhang, W., editors, Internet and Modern Society, pages 166–180, Cham. Springer Nature Switzerland. DOI: https://doi.org/10.1007/978-3-032-05144-8_13.

Sokol, A., Sisk, M. L., Alvarez, J. E., and Chawla, N. (2025). Ventana a la verdad (window to the truth): A chatbot application for navigating the colombian truth commission's archives. In Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining, WSDM '25, page 1052–1055, New York, NY, USA. Association for Computing Machinery. DOI: https://doi.org/10.1145/3701551.3704123.

Spärck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11–21. DOI: https://www.doi.org/10.1108/eb026526.

Tai, C.-Y., Chen, Z., Zhang, T., Deng, X., and Sun, H. (2023). Exploring chain of thought style prompting for text-to-SQL. In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5376–5393, Singapore. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/2023.emnlp-main.327.

Tomar, M., Tiwari, A., Saha, T., Jha, P., and Saha, S. (2024). An ecosage assistant: Towards building a multimodal plant care dialogue assistant. In Advances in Information Retrieval, page 318–332, Berlin, Heidelberg. Springer-Verlag. DOI: https://doi.org/10.1007/978-3-031-56060-6_21.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.. DOI: https://doi.org/10.48550/arXiv.1706.03762.

Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., and Wei, F. (2024). Text embeddings by weakly-supervised contrastive pre-training. Accessed: 20 June 2026.. DOI: https://doi.org/10.48550/arXiv.2212.03533.

Yu, T., Zhang, Y.-F., Fu, C., Wu, J., Lu, J., Wang, K., Lu, X., Shen, Y., Zhang, G., Song, D., Yan, Y., Xu, T., Wen, Q., Zhang, Z., Huang, Y., Wang, L., and Tan, T. (2025). Aligning multimodal llm with human preference: A survey. Accessed: 20 June 2026.. DOI: https://doi.org/10.48550/arXiv.2503.14504.

Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. (2023). Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986. DOI: http://doi.org/10.1109/ICCV51070.2023.01100.

Zhou, Y., Zhang, Z., Wang, X., Sheng, Q., and Zhao, R. (2024). Multimodal archive resources organization based on deep learning: a prospective framework. Aslib Journal of Information Management, 77(3):530–553. DOI: https://doi.org/10.1108/AJIM-07-2023-0239.