Learning to Rank for Query Auto-completion in the Legal Domain
DOI:
https://doi.org/10.5753/jbcs.2025.4279Keywords:
Query auto-completion, Learning to rank, Contextual and additional features, LambdaMART, RankSVM, XGBoost, Genetic Programming, BERT, ColBERTAbstract
Most modern Web search engines implement query auto-completion (QAC) to facilitate faster user query input by predicting users' intended query. This is the case of Jusbrasil, Brazil’s most prominent and widely used legal search engine platform. Query auto-completion is typically performed in two steps: matching and ranking. Matching refers to the selection of candidate query from a suggestions dataset. Ranking sorts the matching results according to a score function that attempts to select the top most relevant suggestions for the user. In this paper, our main goal is to explore the effectiveness of learning to rank algorithms on the ranking step for query auto-completion in the legal domain. In particular, we explore four learning to rank algorithms: LambdaMART, XGBoost, RankSVM and Genetic Programming. LambdaMART is widely used in query auto-completion. On the other hand, as far as we know, this is the first time that the RankSVM and XGBoost are used for this task. Additionally, we propose the use of Genetic Programming as a lightweight and viable alternative for query auto-completion. One difficulty for exploring learning to rank algorithms in query auto-completion is the lack of fine-grained training and test datasets, since learning to rank algorithms rely on a large number of features. To bridge this gap, and also to foster research on this area, we propose two datasets with different types of features for query auto-completion in the legal domain. The datasets were created by collecting data from several data sources from Jusbrasil, including contextual features from search query logs, enriched with additional features extracted from other data sources like auto-completion log, document content and metadata available at Jusbrasil. Then, we show that learning to rank is effective for query auto-completion in the legal domain by answering four main research questions: 1) How each feature, specially the novel ones proposed in our work, impact the rankings in query auto-completion?; 2) How effective is learning to rank with respect to the Most Popular Completion (MPC), a ranking algorithm widely adopted as baseline in the literature?; 3) Among the four alternatives experimented, which learning to rank algorithm is more effective in the legal domain?; and 4) How effective is learning to rank with respect to ranking models based on BERT and ColBERT? Finally, we conduct an online A/B test at Jusbrasil.
Downloads
References
Al-Maskari, A., Sanderson, M., and Clough, P. (2007). The relationship between ir effectiveness measures and user satisfaction. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 773-774. DOI: 10.1145/1277741.1277902.
Bar-Yossef, Z. and Kraus, N. (2011). Context-sensitive query auto-completion. In Proceedings of the 20th International Conference on World Wide Web, WWW '11, page 107–116, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/1963405.1963424.
Block, A., Kidambi, R., Hill, D. N., Joachims, T., and Dhillon, I. S. (2022). Counterfactual learning to rank for utility-maximizing query autocompletion. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 791–802, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3477495.3531958.
Burges, C. J. (2010). From ranknet to lambdarank to lambdamart: An overview. Technical Report MSR-TR-2010-82, Microsoft Research. Available online [link].
Cai, F. and Chen, H. (2017). Term-level semantic similarity helps time-aware term popularity based query completion. J. Intell. Fuzzy Syst., 32(6):3999–4008. DOI: 10.3233/JIFS-151404.
Cai, F. and de Rijke, M. (2016). Learning from homologous queries and semantically related terms for query auto completion. Information Processing & Management, 52(4):628-643. DOI: 10.1016/j.ipm.2015.12.008.
Cai, F. and de Rijke, M. (2016). A survey of query auto completion in information retrieval. Found. Trends Inf. Retr., 10(4):273–363. DOI: 10.1561/1500000055.
Cai, F., Liang, S., and de Rijke, M. (2016). Prefix-adaptive and time-sensitive personalized query auto completion. IEEE Transactions on Knowledge and Data Engineering, 28(9):2452-2466. DOI: 10.1109/TKDE.2016.2568179.
Chen, T. and Guestrin, C. (2016). Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794. DOI: 10.1145/2939672.2939785.
da Costa Carvalho, A. L., Rossi, C., de Moura, E. S., da Silva, A. S., and Fernandes, D. (2012). Lepref: Learn to precompute evidence fusion for efficient query evaluation. Journal of the American Society for Information Science and Technology, 63(7):1383-1397. DOI: 10.1002/asi.22665.
Di Santo, G., McCreadie, R., Macdonald, C., and Ounis, I. (2015). Comparing approaches for query autocompletion. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, page 775–778, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2766462.2767829.
Fan, W., Gordon, M. D., and Pathak, P. (2004). A generic ranking function discovery framework by genetic programming for information retrieval. Information Processing & Management, 40(4):587-602. DOI: 10.1016/j.ipm.2003.08.001.
Ferreira, B., de Moura, E. S., and Silva, A. d. (2022). Applying burst-tries for error-tolerant prefix search. Information Retrieval Journal, 25(4):481-518. DOI: 10.1007/s10791-022-09416-9.
Fiorini, N. and Lu, Z. (2018). Personalized neural language models for real-world query auto completion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 208-215, New Orleans - Louisiana. Association for Computational Linguistics. DOI: 10.18653/v1/N18-3026.
Freund, Y. and Schapire, R. E. (1999). A short introduction to boosting. Journal of Japanese Society of Artificial Intelligence, 14(5):771 - 780. Available online [link].
Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189 - 1232. DOI: 10.1214/aos/1013203451.
Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., and Cheng, X. (2020). A deep look into neural ranking models for information retrieval. Information Processing & Management, 57(6):102067. DOI: 10.1016/j.ipm.2019.102067.
Hu, Y. H., Xiao, C., and Ishikawa, Y. (2018). Context-sensitive query auto-completion with knowledge base. In The 10th Forum on Data Engineering and Information Management (the 16th Annual Meeting of Database Society of Japan), pages 1-5. Available online [link].
Jaech, A. and Ostendorf, M. (2018). Personalized language model for query auto-completion. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 700-705, Melbourne, Australia. Association for Computational Linguistics. DOI: 10.18653/v1/P18-2111.
Järvelin, K. and Kekäläinen, J. (2002). Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422–446. DOI: 10.1145/582415.582418.
Jiang, D., Cai, F., and Chen, H. (2018a). Location-sensitive personalized query auto-completion. In 2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), volume 01, pages 15-19. DOI: 10.1109/IHMSC.2018.00012.
Jiang, D., Cai, F., and Chen, H. (2018b). Personalizing query auto-completion for multi-session tasks. In 2018 IEEE International Conference on Computer and Communication Engineering Technology (CCET), pages 203-207. DOI: 10.1109/CCET.2018.8542201.
Jiang, D., Chen, W., Cai, F., and Chen, H. (2018c). Neural attentive personalization model for query auto-completion. In 2018 IEEE 3rd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), pages 725-730. DOI: 10.1109/IAEAC.2018.8577694.
Jiang, J.-Y. and Cheng, P.-J. (2016). Classifying user search intents for query auto-completion. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, ICTIR '16, page 49–58, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2970398.2970400.
Jiang, J.-Y., Ke, Y.-Y., Chien, P.-Y., and Cheng, P.-J. (2014). Learning user reformulation behavior for query auto-completion. In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '14, page 445–454, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/2600428.2609614.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Marcos Aurélio Domingues, Lucas Rocha, Edleno Silva de Moura, Altigran Soares da Silva
This work is licensed under a Creative Commons Attribution 4.0 International License.
