Improving Cross-Domain Low-Resource Text Generation through LLM Post-Editing: A Programmer-Interpreter Approach (2024)

Abstract: Post-editing has proven effective in improving the quality of text generated by large language models (LLMs) such as GPT-3.5 or GPT-4, particularly when direct updating of their parameters to enhance text quality is infeasible or expensive. However, relying solely on smaller language models for post-editing can limit the LLMs’ ability to generalize across domains. Moreover, the editing strategies in these methods are not optimally designed for text generation tasks. To address these limitations, we propose a neural programmer-interpreter approach that preserves the domain generalization ability of LLMs while editing their output. The editing actions in this framework are specifically devised for text generation. Extensive experiments demonstrate that the programmer-interpreter significantly enhances GPT-3.5’s performance in logical form-to-text conversion and low-resource machine translation, surpassing other state-of-the-art (SOTA) LLM post-editing methods in cross-domain settings.

## introduction large pre-trained language models like gpt-3.5 1 or gpt-4 2 have gained significant attention in natural language research. however, fine-tuning these models for specific tasks is challenging due to limited computational resources or inaccessible parameters. consequently, many researchers resort to using web apis for instructing llms, leveraging zero-shot or few-shot in-context learning, enabling the llms to tackle tasks they weren't explicitly trained for. unfortunately, this approach falls short when tackling some low-resource sequence generation tasks in machine translation (mt), and logical form (lf)-to-text translation, as shown in @xcite ; @xcite . in such cases, minimal task-specific data was available during the 1 https://platform.openai.com/docs/models/gpt-3-5-turbo 2 https://platform.openai.com/docs/models/gpt-4-and-gpt4-turbo ## programmer-interpreter approach the objective in lf-to-text and mt tasks using llms is to generate a high-quality output text y, denoted as y ′ = arg max @xmath0, c), given an input x (e.g., lf, source-language utterance) and an exemplar pool @xmath1, y j , y * j , a * j )} |c| @xmath2. here, x i and y j are the ground truth input-output pairs, y * j is the imperfect translation of x i , and a * j represents the oracle edit actions that can modify y * j into y j . our approach focuses on achieving highquality generation through iterative refinement of the initial output text produced by an llm. specifically, the iterative refinement framework includes three-parameterized modules: a generator, a programmer, and an interpreter, foot_0 the generator corresponds to the llm (e.g. gpt-3.5, gpt-4). it produces the initial output text, y 0 , given the input x, a set of examples retrieved by the function m (x, c) when performing in-context learning. the programmer, a module that creates editing actions a i given x and the current imperfect output y i , is a pre-trained sequence-to-sequence @xcite language model, such as mt5 @xcite or flan-t5 (chung et al., 2022), fine-tuned on a synthetic dataset. the interpreter, essentially also an llm, refines the imperfect intermediate output y i by processing instructions that incorporate predicted editing actions and few-shot editing examples, retrieved via the function a(x, c). please note that the programmer has much fewer parameters than the llm used by the generator and interpreter. after several iterative refinements, we arrive at the final output y t generated by the llm. during generation, we assume no access to the parameters of the llms but only obtain the output text by providing prompting instructions. the implementation details of each module are as follows: generator. to generate the initial output, we supply a prompt composed of a few-shot set of exemplar pairs, denoted as m (x, c) = {(x j , y j )} m @xmath3, selected from a pool of reference pairs c. this is accompanied by an instruction prompting the llm to produce output y 0 based on the input x. the retrieval function identifies the closest pairs by calculating the cosine similarity of tf-idf features between x and other instances of x in c. programmer. after obtaining the initial or intermediate output y i from either the generator or the interpreter, we combine the input x and y i into a single sequence and feed it to the programmer to generate a sequence of edit actions a i . we create a synthetic training set t , extracted from the example pool c, for fine-tuning the programmer. each pair in t is defined as (x concat , a * ), where x concat is the concatenated sequence of x and y * , serving as the input for the programmer. the output a * is the sequence of oracle edit actions, synthetically generated based on the reference pairs in c. for each reference @xmath4, we calculate the word-level edit distance to the imperfect translation y * , generating intermediate edit actions. only insert-word and delete-word actions are retained in the sequence, forming the final training sequence a * for the programmer. if y * is identical to the reference y, the action is labeled as "noaction", indicating that no refinement is needed for that instance. unlike pive, which generates the imperfect translation y * by scrambling the original y, we directly use the initial output y 0 from the generator as y * in both c and t . this approach enables the programmer to learn an action distribution that more effectively corrects translation errors from llms. interpreter. to edit the intermediate output y i , we engage the llm in the interpreter role by providing it with prompting instructions. given the edit instructions a i and a pair (y i , x), the llm can insert or delete words in order to generate the modified text y i+1 . we also incorporate a few-shot examples that demonstrate editing procedures, extracted from c and denoted as a(x, c) = {(x j , y j , y * j , a * j )} n @xmath5. these examples are selected based on the cosine similarity between the tf-idf features of x and those in c. furthermore, to mimic action prediction errors from the programmer, we adopt an adversarial in-context learning strategy, similar to the approach in @xcite . this involves corrupting the action sequence by deleting oracle actions with a certain probability d%. if an action is not deleted, we swap it with other actions from c at the same probability d%. through this manipulation, we have discovered that the llm's exceptional text generalization ability enables it to effectively comprehend the editing instructions. as a result, it can generate high-quality text after performing the necessary edits, even if the predicted actions from the programmer are not completely accurate. see 3 experiments setup. in our experiments, we default to using gpt-3.5-turbo-0301 as the llm for the generator in both the zero-shot and few-shot settings. for the interpreter, we use gpt-3.5-turbo-0301 in the zero-shot setting and gpt-3.5-turbo-16k foot_1 in the few-shot setting. for the generator used across all settings and baselines, we consistently use 0 and 5 shots for mt and lf-to-text, respectively. for the interpreter in the few-shot setting, we apply 10 and 5 action examples for mt and lf-to-text, respectively, with a 50% action corruption probability. for the mt and lf-to-text tasks, we employ mt5base and flan-t5-base as the backbones of the programmers, respectively. these backbone choices are driven by our emphasis on a computationally efficient setup, ensuring the models fit within an nvidia v100 with 16gb memory. we train our programmers with a development set to select the optimal model. our search for the best learning rate includes [5e-5, 1e-4, 2e-4], while the range of epochs considered is @xcite @xcite , with batch sizes 4. gpts require no fine-tuning. each generation of 1096 tokens costs approximately $0.0015. for self-correct and self-refine, we perform five editing iterations. prog-refine and algo-refine stop when more than 95% of action is 'noaction'. ## conclusions we present a programmer-interpreter method that iteratively refines llm outputs using edit actions from a fine-tuned programmer and an llm interpreter. our approach combines the task-specific encoding capacity of a fine-tuned model with the domain generalization strength of the llm, incorporating specifically designed actions for text generation. the experiments confirm its efficacy, showing significant improvements in llm-generated text quality for low-resource mt and lf-to-text tasks. moreover, our approach outperforms established baselines in cross-domain scenarios.