DIMSIM : Distilled Multilingual Critics for I ndic Text Simplification (2024)

Abstract: Self-correction techniques have recently emerged as a promising framework to improve the quality of responses generated by large language models (LLMs). Few-shot prompted LLMs act as critics to produce feedback for an input, which is further fed to a refiner (also an LLM) to produce an output. However, these critique-refine steps require multiple expensive LLM calls. To circumvent this large inference cost, we borrow inspiration from prior work on knowledge distillation and propose the use of critique distillation to train critic models. These are smaller sequence-to-sequence models that are trained on input-critique pairs generated by an LLM. We focus on the problem of text simplification for three Indian languages: Hindi, Bengali and Marathi. This task is a good fit for self-correction style techniques. It also hasn’t been systematically explored for Indian languages before. We train two separate critics that focus on lexical and structure complexity, and show that it is surprisingly more effective than using an LLM directly as a critic in both 0-shot and few-shot settings. We also show the benefits of training multilingual critics, as opposed to monolingual critics. Extensive human evaluations show that on average, raters find 80% of DIMSIM’s output to be simple and easy to read.

## introduction large language models (llms) are well-equipped to generate high-quality responses using only a few task-specific examples in its input prompts @xcite @xcite . however, these generations are prone to vulnerabilities such as "hallucinations" @xcite access to a small set of expert-written examples of text simplification in hindi. we adopt the more challenging zero-shot setting for bengali and marathi where we assume no access to text simplification instances in these two languages and only make use of the hindi examples. we decouple text simplification along two natural axes --structural and lexical complexity --and design two separate critic modules that provide critiques specific to each of these two dimensions ( §3.1). each distilled critic is a smaller sequenceto-sequence model (e.g., mt5 @xcite ), trained using synthetic data created via few-shot prompting of the llm followed by careful data filtering ( §3.2). apart from critique generation, we also show the merits of both eager and lazy refinement where the llm (used as a refiner) can either individually refine the outputs based on each trained critic's feedback or refine the output based on combined feedback from both critic modules, respectively ( §3.3). we evaluate our results using the standard sari metric @xcite ) for text simplification and also undertake detailed human evaluations. we find that trained critics, especially multilingual critics that are trained on data pooled together from hindi, bengali and marathi, offer improved cross-lingual transfer to bengali and marathi. the trained critics result in significant performance improvements of up to 2.4 absolute sari points for hindi and up to 2 absolute points for bengali and marathi in the zero-shot setting, compared to using llms at test time. we call our model dimsim: distilled multilingual critics for text simplification. ## dimsim: methodology in this work, we present a critique-refine framework for text simplification (ts) using multilingual llms. instead of a single critic that offers overall feedback for simplification, we decompose the simplification problem along two dimensions --structural complexity and lexical complexity -and we designate one critic to each dimension. the structure critic offers suggestions on how to alter the structure of a sentence, either by paraphrasing or splitting, to make it simpler. the lexical critic is more targeted in its critique by identifying archaic or difficult words in the text and offering simpler alternatives. henceforth, we will collectively refer to these two modules as sub-critics. finally, a refiner module processes the combined critique from the sub-critics to generate a simplified text output @xcite . to prime the sub-critics to offer informative critiques specific to each dimension, we need carefully designed few-shot examples. we create a corpus of 20 manually-curated input-output ts pairs for hindi. these span a range of simplification operations, and are created by trained language experts. this method is expensive, tedious, and requires access to trained annotators. it does not readily scale to new languages. therefore, we adapt the critique-refine framework to work in a zero-shot cross-lingual setting for bengali and marathi, using hindi few-shot examples. figure 1 offers a schematic illustration of dim-sim. section 3.1 describes the key design choices we make to build the sub-critics. section 3.2 further elaborates on generating synthetic data to train sub-critics. and, section 3.3 describes how the refiner module processes these critiques to produce the final simplified text. ## results tables 1 and 2 summarize our main results for hindi, bengali, and marathi. for hindi, dimsim outperforms all other methods, with a near 3 point improvement in sari scores (44.31) compared to ccsv, the closest competitor (41.96). it does so while making only 1 4 the number of llm calls. lazy refinement performs slightly better than eager refinement (in both directions). critique refinement with a single critic is not very effective, and falls behind chain-of-thought prompting by roughly 2 sari points. cross-lingual evaluations for bengali and marathi are reported in we observe that with cross-lingual + cot prompting, the llm tends to copy inputs more than 50% of the time. this is primarily because it fails to generate a comprehensive chain-of-thought string, and typically regresses to either producing an empty string or irrelevant text in bengali or marathi. using translated examples without cot slightly boosts sari scores for both bengali and marathi. however, with cot added, there is a 4 point drop in sari. recall that in the translation + cot approach, hindi phrases and suggested wordreplacements in the analysis string are also translated to bengali or marathi. however, there is no guarantee that words/phrases present in the translated analysis string are actually valid sub-strings of the translated input string. as a consequence, the llm has to learn from noisy in-context examples, resulting in a performance drop. ## conclusions in this work, we present a new framework for multilingual text simplification using llms for selfcorrection. we decouple the simplification problem along two salient dimensions and train individual critics using synthetic data for each critic generated via few-shot prompting of llms. we evaluate our framework on text simplification for three indian languages in a few-shot (for hindi) and zero-shot (for bengali and marathi) settings. while we focus entirely on text simplification in this work, we think our framework is broad enough to be applicable to other generation tasks such as formality transfer, abstractive summarization, etc. and leave this exploration to other tasks as future work. the fourth author would like to gratefully acknowledge a faculty grant from google research india supporting her research on multilingual models and is also very thankful for google cloud credits that helped support all the compute requirements. we are deeply indebted to the vani and verse solutions team (especially keshav and deepika agarwal) for helping us scale and expedite human evaluations on their platform.