Multimodal RAG with Knowledge Graphs for Portuguese Maintenance Manuals in the Context of Industry 4.0
DOI:
https://doi.org/10.5753/reic.2026.8503Keywords:
Retrieval-Augmented Generation, Knowledge Graphs, Multimodal RAG, Industry 4.0, Portuguese NLPAbstract
Navigating dense, multimodal technical documentation remains a critical bottleneck for industrial maintenance in non-English contexts. This paper presents a multimodal Retrieval-Augmented Generation (RAG) framework for Portuguese maintenance manuals, combining dense vector search over text and tables with knowledge graph traversal. We evaluate a local multilingual transformer (paraphrase-multilingual-MiniLM-L12-v2) and OpenAI’s text-embedding-3-small across 8 configurations (2 models × 4 retrieval modes) using a ground-truth set of 50 domain-specific queries. The best configurations (openai/text_only and openai/text_table) achieve a BERTScore-F1 of 0.70, ROUGE-L of 0.36, and an MRR of 0.65, answering 45 of 50 queries. Analysis shows that while high-dimensional proprietary embeddings excel at capturing specialized technical jargon, standard Reciprocal Rank Fusion (RRF) introduces rank contamination when fusing sparse graph signals with dense text vectors. Source code is publicly available.
Downloads
Referências
Arivazhagan, M. G., Liu, L., Qi, P., Chen, X., Wang, W. Y., and Huang, Z. (2023). Hybrid hierarchical retrieval for open-domain question answering. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10680–10689. Association for Computational Linguistics. DOI: 10.18653/v1/2023.findings-acl.679.
Bousdekis, A., Apostolou, D., and Mentzas, G. (2019). Predictive maintenance in the 4th industrial revolution: Benefits, business opportunities, and managerial implications. IEEE Engineering Management Review, 48(1):57–62. DOI: 10.1109/EMR.2019.2958037.
Bu, C., Chang, G., Chen, Z., Dang, C., Wu, Z., He, Y., and Wu, X. (2025). Query-driven multimodal GraphRAG: Dynamic local knowledge graph construction for online reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21360–21380. Association for Computational Linguistics. DOI: 10.18653/v1/2025.findings-acl.1100.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, J. (2023). Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829. IEEE. DOI: 10.1109/CVPR52729.2023.00276.
Cormack, G. V., Clarke, C. L. A., and Buettcher, S. (2009). Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 758–759. ACM. DOI: 10.1145/1571941.1572114.
Freitas, C., Rabonato, R. T., and Berton, L. (2025a). Enhancing epidemiological insights with RAG for SIREVA-SUS reports. In Anais do Encontro Nacional de Inteligência Artificial e Computacional (ENIAC), pages 1364–1375. SBC. DOI: 10.5753/eniac.2025.11796.
Freitas, C., Real, L., Berton, L., and de Paiva, V. (2026). Towards a universal dependencies corpus for Portuguese epidemiological reports. In Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) – Vol. 2, pages 228–237. Association for Computational Linguistics. DOI: 10.18653/v1/2026.propor-2.31.
Freitas, C., Vega-Oliveros, D. A., and Berton, L. (2025b). Enhancing industrial data access with text-to-SQL using Portuguese LLMs and LangGraph. In 2025 12th International Conference on Soft Computing & Machine Intelligence (ISCMI), pages 278–282. IEEE. DOI: 10.1109/ISCMI67495.2025.11358511.
Freitas, C., Vega-Oliveros, D. A., and Berton, L. (2025c). Orchestrating industrial data retrieval: A RAG-enhanced text-to-SQL pipeline for Brazilian Portuguese. Enterprise Information Systems. Under review.
Ghobakhloo, M. (2020). Industry 4.0, digitization, and opportunities for sustainability. Journal of Cleaner Production, 252:119869. DOI: 10.1016/j.jclepro.2019.119869.
Kagaya, T., Yuan, T. J., Lou, Y., Karlekar, J., Pranata, S., Kinose, A., Oguri, K., Wick, F., and You, Y. (2024). RAP: Retrieval-augmented planning with contextual memory for multimodal LLM agents. arXiv preprint arXiv:2402.03610. DOI: 10.48550/arXiv.2402.03610.
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. (2020). Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. Association for Computational Linguistics. DOI: 10.18653/v1/2020.emnlp-main.550.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., Riedel, S., and Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc.. DOI: 10.5555/3495724.3496517.
Liu, J., Tao, Y., Wang, F., Li, H., and Qin, X. (2025). SiQA: A large multi-modal question answering model for structured images based on RAG. In 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. DOI: 10.1109/ICASSP49660.2025.10888359.
Mei, L., Mo, S., Yang, Z., and Chen, C. (2025). A survey of multimodal retrieval-augmented generation. arXiv preprint arXiv:2504.08748. DOI: 10.48550/arXiv.2504.08748.
OpenAI (2024). Embeddings — OpenAI API documentation. Disponível em: [link]. Accessed: 2026-03-29.
Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. Association for Computational Linguistics. DOI: 10.18653/v1/D19-1410.
United Nations (2015). Transforming our world: The 2030 agenda for sustainable development. General Assembly 70th Session. Accessed: 2026-03-29.
Zhang, H., Schmidt, W. J., Shen, X., Cao, Q., Monka, S., and Paschke, A. (2025). Knowledge graph construction towards a graph RAG-enhanced intelligent maintenance chatbot. In International Workshop on Scaling Knowledge Graphs for Industry 2025.
Downloads
Published
Como Citar
Issue
Section
Licença
Copyright (c) 2026 Os autores
Este trabalho está licenciado sob uma licença Creative Commons Attribution 4.0 International License.
