Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages (2024)

Abstract: Large language models (LLMs) as listwise rerankers have shown impressive zero-shot capabilities in various passage ranking tasks. Despite their success, there is still a gap in existing literature on their effectiveness in reranking low-resource languages. To address this, we investigate how LLMs function as listwise rerankers in cross-lingual information retrieval (CLIR) systems with queries in English and passages in four African languages: Hausa, Somali, Swahili, and Yoruba. We analyze and compare the effectiveness of monolingual reranking using either query or document translations. We also evaluate the effectiveness of LLMs when leveraging their own generated translations. To grasp the general picture, we examine the effectiveness of multiple LLMs — the proprietary models RankGPT-4 and RankGPT-3.5, along with the open-source model RankZephyr. While the document translation setting, i.e., both queries and documents are in English, leads to the best reranking effectiveness, our results indicate that for specific LLMs, reranking in the African language setting achieves competitive effectiveness with the cross-lingual setting, and even performs better when using the LLM’s own translations.

## introduction several studies have shown that large language models (llms) excel in various nlp tasks @xcite @xcite . in text ranking, llms have been used effectively as retrievers @xcite and in both pointwise and listwise reranking. in reranking, models may generate an ordered list directly @xcite @xcite or sort based on token probabilities @xcite . the large context size of llms makes listwise approaches particularly attractive because the model attends to multiple documents to produce a relative ordering. cross-lingual retrieval aims to provide information in a language different from that of the search query. this is especially relevant when the required information is not available or prevalent in the query's language, as is the case for most lowresource languages. previous work has examined sparse and multilingual dense retrieval models in cross-lingual settings for these languages @xcite . however, studies on the effectiveness of llms as crosslingual retrievers or rerankers for low-resource languages are few to non-existent. in this study, we examine the effectiveness of proprietary and open-source models for listwise reranking in low-resource african languages. our investigation is guided by the following research questions: (1) how well do llms fare as listwise rerankers for low-resource languages? (2) how effectively do llms perform listwise reranking in cross-lingual scenarios compared to monolingual (english or low-resource language) scenarios? (3) when we leverage translation, is reranking more effective when translation uses the same llm used for zero-shot reranking? we answer these questions through an extensive investigation of the effectiveness of rankgpt @xcite and rankzephyr @xcite in cross-lingual and monolingual retrieval settings. we use ciral @xcite , a cross-lingual information retrieval dataset covering four african languages with queries in english and passages in african languages, and construct monolingual retrieval scenarios through document and query translations. our results show that cross-lingual reranking with these llms is generally more effective compared to reranking in the african languages, underscoring that they are better tuned to english than low-resource languages. across all languages, we achieve our best results when reranking entirely in english using retrieval results obtained by doc-ument translation. in this setting, we see up to 7 points improvement in ndcg@20 over crosslingual reranking using rankgpt 4 , and up to 9 points over reranking in african languages. we specifically notice improvements with rankgpt 4 when using its query translations for reranking in african languages. ## background and related work given a corpus @xmath0, d 2 , ..., d n } and a query q, information retrieval (ir) systems aim to return the k most relevant documents. modern ir pipelines typically feature a multi-stage architecture in which a first-stage retriever returns a list of candidate documents that a reranker reorders for improved quality @xcite @xcite . more recently, the effectiveness of decoder models as rerankers (dubbed "prompt decoders") has been explored in some depth. researchers have fine-tuned gpt-like models in the standard contrastive learning framework @xcite @xcite and studied different approaches to reranking using both open-source llms and proprietary gpt models. @xcite evaluated the effectiveness of openai models on multiple ir benchmarks using permutation generation approaches, while @xcite demonstrate the effectiveness of gpt-3 as a zero-shot listwise reranker and the superiority of listwise over pointwise approaches. while these papers focus on reranking with llms, they only cover two african languages-swahili and yoruba. for both languages, gpt-3 improves over bm25 significantly but still falls behind supervised reranking baselines. in this work, we examine the effectiveness of these llms as components of ir systems for african languages. specifically, we study the effectiveness of opensource and proprietary llms as listwise rerankers for four african languages (hausa, somali, swahili, and yoruba) using the ciral cross-lingual ir test collection @xcite . to be more precise, cross-lingual information retrieval (clir) is a variant of the standard retrieval task in which the queries q i are in a different language from the documents in the corpus c. popular approaches to clir include query translation, document translation, and language-independent representations @xcite . as the focus of this work is on the effectiveness of llms as listwise input prompt: system you are rankgpt, an intelligent assistant that can rank passages based on their relevancy to the query. user i will provide you with {num} passages, each indicated by number identifier []. rank the passages based on their relevance to the query: {query}. [1] {passage 1} [2] {passage 2} ... [num] {passage num} search query: {query} rank the {num} passages above based on their relevance to the search query. the passages should be listed in descending order using identifiers. the most relevant passages should be listed first. the output format should be [] > [], e.g., [1] > [2]. only respond with the ranking results, do not say any word or explain. rerankers in cross-lingual settings, we primarily explore document and query translation approaches in this study. ## experimental setup models. we implement zero-shot reranking for african languages with three models. these include proprietary reranking llms: rankgpt 4 and rankgpt 3.5 , using the gpt-4 and gpt-3.5-turbo models, respectively, from azure's openai api. to examine the effectiveness of open-source llms, we rerank with rankzephyr @xcite , an open-source reranking llm obtained by instruction-fine-tuning zephyr β @xcite to achieve competitive effectiveness with rankgpt models. baselines. we compare the reranking effectiveness of the llms using already established models as baselines. our baselines include two crossencoder models, the multilingual t5 (mt5) @xcite and afrimt5 @xcite , which is mt5 with continued pre-training on african corpora. the mt5 foot_0 and afrimt5 foot_1 rerankers were obtained from fine-tuning the base versions of both models on the ms marco passage collection @xcite for 100k iterations, with a batch size of 128. test collection. models are evaluated on ciral @xcite , a clir test collection consisting of four african languages: hausa, somali, swahili, and yoruba. queries in ciral are natural language factoid questions in english while passages are in the respective african languages. each language comprises between 80 and 100 queries, and evaluations are done using the pooled judgments obtained from ciral's passage retrieval task. 3 we also make use of ciral's translated passage collection foot_3 in our document translation scenario. the test collection's documents were translated from the african languages to english using the nllb machine translation model @xcite . we report ndcg@20 scores following the test collection's standard, and mrr@100. configurations. first-stage retrieval uses bm25 @xcite in the open-source pyserini toolkit @xcite . we use whitespace tokenization for passages in native languages and the default english tokenizer for the translated passages. our bm25 retrieval is implemented using document (bm25-dt) and query (bm25-qt) translations. for bm25-qt, queries are translated with google machine translation (gmt). we rerank the top 100 passages retrieved by bm25 using the sliding window technique by @xcite with a window of 20 and a stride of 10. experiments were conducted using the rankllm toolkit. 5 we use a context size of 4,096 tokens for rankgpt 3.5 and rankzephyr, and 8,192 tokens for rankgpt 4 . these context sizes are also maintained for the zero-shot llm translation experiments. for each model, translation is performed over three iterations and we vary the model's temperatures from 0 to 0.6 to allow variation in the translations. translations are only obtained for the gpt models since rankzephyr is suited only for reranking. reranking results are reported over a single run, except with the llm translations where we take the reciprocal rank fusion (rrf) @xcite of results from the three iterations. ## conclusion in this work, we evaluate zero-shot cross-lingual reranking with large language models (llms) on african languages. our suite covered three forms of llm-based reranking: rankgpt 4 , rankgpt 3.5 and rankzephyr. using the listwise reranking method, our results demonstrate that reranking in english via translation is the most optimal. we examine the effectiveness of llms in reranking for low-resource languages in the cross-lingual and african language monolingual scenarios and find that llms have comparable effectiveness in both scenarios but with better results in cross-lingual. in the process, we also establish that good translations obtained from the llms do improve their rerank-ing effectiveness in the african language reranking scenario as discovered with rankgpt 4 . additionally, while open-source models showcase slightly lower effectiveness than rankgpt 4 , they still largely improve over other proprietary models like rankgpt 3.5 , an important step towards the development of effective listwise rerankers for low-resource languages.