Artificial intelligence

Abstract: Problem. Retrieval-Augmented Generation (RAG) systems optimized for English exhibit significant quality degradation when processing Ukrainian texts due to the rich inflectional morphology of the Ukrainian language. Seven grammatical cases, two numbers, verbal aspects, and free word order produce dozens of word forms for a single concept. Standard lexical search algorithms BM25 and TF-IDF treat these forms as different tokens, critically reducing search recall. Solution. The UA-RAG system is proposed, integrating three components: (1) a morphological normalization algorithm based on suffix stemming with 15+ rules for Ukrainian suffixes and a minimum stem length of 3 characters; (2) hybrid search combining lemmatized BM25 with n-gram TF-IDF (bigrams, trigrams) via Reciprocal Rank Fusion (RRF); (3) a fact verification module based on query keyword coverage analysis with a filtering threshold of 0.3. Results. Experimental evaluation was conducted on a corpus of 40 Ukrainian text chunks and 12 test queries. The full UA-RAG system achieves F1=0.633, NDCG@5=0.724, exceeding naive TF-IDF (F1=0.550) by 15.1%. Morphological normalization provides a +3% F1 improvement, fact verification adds +8.6% F1. This study presents the first Ukrainian RAG benchmark with morphological processing.

Keywords: retrieval-augmented generation, morphological normalization, Ukrainian language, fact verification, hybrid search, BM25, natural language processing, large language models.

References:

1. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Kiela, D. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33, 9459-9474. https://doi.org/10.48550/arXiv.2005.11401
2. Gao, Y., Xiong, Y., Dibia, V., Chi, L., Shu, D., Pham, H., ... & Neville, J. (2024). Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997. https://doi.org/10.48550/arXiv.2312.10997
3. Sawarkar, K., Mangal, A., & Nanduri, S. R. (2024). Blended RAG: Improving RAG Accuracy with Semantic Search and Hybrid Query-Based Retrievers. arXiv preprint arXiv:2404.07220. https://doi.org/10.48550/arXiv.2404.07220
4. Chan, C., Xu, C., Yuan, R., Luo, S., Zhu, W., & Miao, Y. (2024). RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation. arXiv preprint arXiv:2404.00610. https://doi.org/10.48550/arXiv.2404.00610
5. Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv preprint arXiv:2310.11511. https://doi.org/10.48550/arXiv.2310.11511
6. Yan, S., Gu, J., Zhu, Y., & Ling, Z. (2024). Corrective Retrieval Augmented Generation. arXiv preprint arXiv:2401.15884. https://doi.org/10.48550/arXiv.2401.15884
7. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 3982-3992. https://doi.org/10.18653/v1/D19-1410
8. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., ... & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. Proceedings of EMNLP, 6769-6781. https://doi.org/10.18653/v1/2020.emnlp-main.550
9. Korobov, M. (2015). Morphological Analyzer and Generator for Russian and Ukrainian Languages. Analysis of Images, Social Networks and Texts, 320-332. https://doi.org/10.1007/978-3-319-26123-2_31
10. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., & Zaharia, M. (2022). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv preprint arXiv:2112.01488. https://doi.org/10.48550/arXiv.2112.01488
11. Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389. https://doi.org/10.1561/1500000019
12. Morozov, O. V., & Donets, V. V. (2024). Application of Retrieval-Augmented Generation technology for building intelligent query-response systems. Visnyk NTU "KhPI". Series: Informatics and Modeling, (1), 88-97. [in Ukrainian]. https://doi.org/10.20998/2411-0558.2024.01.08
13. Nahold, O. (2024). RAG-based Approach for Intelligent Document Processing in Low-Code Platforms. Artificial Intelligence, (2), 45-56. https://doi.org/10.15407/jai2024.02.045
14. Prytula, M. I., Voitko, V. V., & Semeniuk, A. O. (2024). Comparative analysis of zero-shot and RAG approaches for Ukrainian-language question-answering tasks. System Research and Information Technologies, (3), 112-125. [in Ukrainian]. https://doi.org/10.20535/SRIT.2308-8893.2024.3.10
15. Syromiatnikov, D., & Ruvinskaya, V. (2024). UA-LLM: Ukrainian Language Model Pre-trained on Large Text Corpora. Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP), 44-52. https://doi.org/10.18653/v1/2024.unlp-1.6
16. Haltiuk, M., & Smywinski-Pohl, A. (2024). LiBERTa: A Ukrainian Language Model Based on DeBERTa. Proceedings of the 2024 Joint International Conference on Computational Linguistics (LREC-COLING), 5765-5774. https://doi.org/10.18653/v1/2024.lrec-main.510
17. Kiulian, A., Osaulenko, V., Kalpakchi, D., & Boström, J. (2024). Adapting Gemma and Mistral Models for Ukrainian Language Tasks. arXiv preprint arXiv:2404.10453. https://doi.org/10.48550/arXiv.2404.10453
18. Rohushyna, Yu. V. (2023). Semantic search in Ukrainian-language text corpora. Problems in Programming, (3-4), 175-186. [in Ukrainian]. https://doi.org/10.15407/pp2023.03-04.175
19. Malyha, S. V., & Shmatkov, S. I. (2023). Vector representations of Ukrainian words: a comparative analysis of models. Radioelectronic and Computer Systems, (2), 64-75. [in Ukrainian]. https://doi.org/10.32620/reks.2023.2.06
20. Prytula, N. (2024). Fine-tuning BERT for Ukrainian Natural Language Processing Tasks. Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP), 28-36. https://doi.org/10.18653/v1/2024.unlp-1.4
21. Nych, L. Ya., Krochak, D. M., & Bazylevych, R. P. (2024). Metrics for evaluating information retrieval quality: a comparative analysis. Bulletin of Lviv Polytechnic National University. Series: Information Systems and Networks, (15), 88-100. [in Ukrainian]. https://doi.org/10.23939/sisn2024.15.088
22. Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv preprint arXiv:2309.15217. https://doi.org/10.48550/arXiv.2309.15217
23. Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal Rank Fusion (RRF) Outperforms Condorcet and Individual Rank Learning Methods. Proceedings of the 32nd International ACM SIGIR Conference, 758-759. https://doi.org/10.1145/1571941.1572114
24. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., ... & Liu, T. (2023). A Survey on Hallucination in Large Language Models. arXiv preprint arXiv:2311.05232. https://doi.org/10.48550/arXiv.2311.05232
25. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., ... & Fung, P. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 1-38. https://doi.org/10.1145/3571730