Sample Design Engineering: An Empirical Study on Designing Better Fine-Tuning Samples for Information Extraction with LLM s (2024)

Abstract: Large language models (LLMs) have achieved significant leadership in many NLP tasks, but aligning structured output with generative models in information extraction (IE) tasks remains a challenge. Prompt Engineering (PE) is renowned for improving IE performance through prompt modifications. However, the realm of the sample design for downstream fine-tuning, crucial for task-specific LLM adaptation, is largely unexplored. This paper introduces **Sample Design Engineering** (SDE), a methodical approach to enhancing LLMs’ post-tuning performance on IE tasks by refining input, output, and reasoning designs. Through extensive ID and OOD experiments across six LLMs, we first assess the impact of various design options on IE performance, revealing several intriguing patterns. Based on these insights, we then propose an integrated SDE strategy and validate its consistent superiority over heuristic sample designs on three complex IE tasks with four additional LLMs, demonstrating the generality of our method. Additionally, analyses of LLMs’ inherent prompt/output perplexity, zero-shot, and ICL abilities illustrate that good PE strategies may not always translate to good SDE strategies.

## introduction information extraction (ie) aims to extract structured information from unstructured text, which is highly valuable in a wide range of industrial scenarios. the emergence of large language models (llms) such as gpt-3 @xcite , llama @xcite has broadened the capabilities of language models to tackle various complex ie tasks with a single model. nonetheless, a fundamental challenge arises from the discrepancy between the unstructured nature of the † equal contribution * corresponding authors, emails: han.songqiao@shufe.edu.cn, hlhuang@shufe.edu.cn llms llms prompt engineering input output input output frozen, very large llms trainable, smaller open-source llms llms for zero-shot/icl for downstream-tuning llms' generative paradigm and the requirement for structured output. in this background, prompt engineering (pe) has become a key area in leveraging cutting-edge llms to address this challenge @xcite @xcite . ## experiments ii: a robust integrated sde strategy based on the experimental evidence from the previous section, we propose an empirically strong sde strategy (termed as es-sde) using the wellperforming options: a combination of inst-first, no-mi input designs and lines, pu(placeholders for unmentioned), txtlabel output designs. we don't use the cot design because of its high annotation cost and relatively unstable output. in this section, we conduct comprehensive experiments to validate its effectiveness across different downstream tasks, as well as the robustness against perturbations in instructions or generation. ## can pe guide sde? effective pe can reveal a llm's strengths and preferences. we explore if pe can guide sde by crafting zero-shot and icl prompts according to different sde options. figure 6 reports the average rankings of sde options and their corresponding prompts in the masa id tasks, with detailed results in appendix a.8. for both pe and sde evaluations, inst-first and cot works well. however, there are also many inconsistent patterns between pe and sde, such as the performance of ou, and the comparison between natural and lines. @xcite showed that the lower perplexity (ppl) generally leads to better prompt designs. inspired by this, we conduct ppl analysis on the icl prompts/predictions. there are also some discrepancies between the ppl scores and the performance in pe and sde. for instance, ou has poor ppl scores, but performs well in zero-shot scenarios, and json shows weaker performance in sde compared to lines, despite its better ppl score. these findings highlight a complex landscape where prompt design patterns do not always align with sde effectiveness, underscoring the nuanced relationship between pe and sde. ## conclusion in this study, we introduce sde as an effective method to enhance the downstream-tuning performances of llms on ie tasks. through comprehensive id and ood experiments involving six llms, we demonstrate the effects of various sample design strategies, uncovering some interesting patterns that are consistent across different llms. building on these findings, we develop the es-sde approach, which integrates the most effective options. our experiments on three new tasks with four additional llms consistently show es-sde's superiority over baseline methods. further analysis of the relationship between pe and sde suggests that effective prompt designs do not necessarily translate to successful sample designs. this observation opens up avenues for more detailed investigations into the mechanisms of sde in future research.