Generate-then-Revise: An Effective Synthetic Training Data Generation Framework For Event Detection Retrieval (2024)

Abstract: “Large language models (LLMs) struggle with event detection (ED) due to the structured and vari-able number of events in the output. Existing supervised approaches rely on a large amount ofmanually annotated corpora, facing challenges in practice when event types are diverse and theannotated data is scarce. We propose Generate-then-Revise (GtR), a framework that leveragesLLMs in the opposite direction to address these challenges in ED. GtR utilizes an LLM to gen-erate high-quality training data in three stages, including a novel data revision step to minimizenoise in the synthetic data. The generated data is then used to train a smaller model for evalua-tion. Our approach demonstrates significant improvements on the low-resource ED. We furtheranalyze the generated data, highlighting the potential of synthetic data generation for enhancingED performance.Introduction”

## introduction event detection (ed) aims to extract structured information from natural language text. previous researches typically involve supervised fine-tuning on a large amount of manually annotated corpora, facing challenges in practice due to the vast number of event types in real-world scenarios and the high cost of manually annotating data. recently, large language models (llms) like chatgpt and llama2 have made remarkable progress in many tasks in low-resource scenarios, enabling some tasks to achieve performance comparable to fully supervised learning in zero-shot or few-shot conditions. however, the success of large models in natural language understanding and reasoning tasks is difficult to directly replicate in ed @xcite @xcite . current research indicates that directly using prompt instructions to instruct models to perform ed generally results in poor performance @xcite . as shown in figure 1 , researchers evaluate the average performance of several llms on ed in a test set containing 250 documents. among them, mixtral-8x7b-instruct-v0.1 achieves the best trigger identification (ti) f1 score of 37.5 in a 64-shot setting, while for trigger classification (tc), the performance of various models is even worse, and there is no linear increase in performance with the increase in samples.the failure of llms in ed is related to the characteristics of the task itself. ed requires generating a structured list of events of varying lengths. although these models are not adept at structured output tasks, recent research has found that models can generate meaningful inputs based on outputs in the opposite direction @xcite , effectively transforming the original computation from p(y k |x) to p(x|y k ). this suggests that we can leverage the asymmetry to synthesize high-quality data for ed. data generation avoids the difficulties of outputting structured content while reducing reliance on manually annotated data. however applying llms to data generation on ed is non-trivial. unlike classification tasks like sentiment analysis (which use sentence-level labels like positive or negative) @xcite , ed requires generating data with token-level labels @xcite , i.e, each sentence should contain both the correct triggers and event types. simply prompting llms can lead to noisy data, as shown in figure 1 , including accompanying events, competing events and incorrect events. unlabled accompanying events emerge frequently in sythesized data and can disrupt the training process. existing works leveraging llms to generate labeled data in ed often overlook these challenges, primarily relying on sophisticated prompts to produce training data. ma's approach @xcite leverages llms for data augmentation but fails to address the noise issues discussed earlier. while existing research offers noise mitigation techniques for classification tasks (e.g., noise-robust loss functions) @xcite , these methods are ineffective against the specific problem of accompanying events in ed. to address this gap, we identify three key challenges to overcome when mitigating noise in labeled data generation: (1) aligned generation: the llm must generate data that strictly adheres to the specified triggers and event types. (2) internal consistency: the triggers and event types within each generated sample need to be consistent and mutually supportive. (3) accompanying event handling: we need a mechanism to identify and label accompanying events during generation. to address these issues, we propose a sentence-level event data generation framework called gtr. gtr synthesizes high-quality automatically annotated data in three stages, which can be used in zeroshot scenarios without manual annotation and can also enhance existing data in few-shot settings. the synthesized data also includes adversarial examples, which can significantly improve the robustness of the model. in addition, considering that synthesized data inevitably contains some noise, we introduce a dynamic weighting module to score the synthesized data, reducing the interference of noisy data and improving the performance of models trained on synthesized data. ## related works zero-shot event detection zero-shot event detection (ed) aims to identify triggers and event types without accessing training data @xcite @xcite . previous approaches for zero-shot ed rely on external tools such as amr and framenet to learn event features from seen events, and apply them to unseen events @xcite . these methods assume that seen and unseen events have similarities, which may not always hold true in realworld scenarios. recently, applying llms to zero-shot tasks has gained popularity . compared to fully supervised learning, llms alleviate the trouble of limited data. however, when it comes to ed, the prompt-based method still yields relatively unsatisfactory performance @xcite @xcite . ## data generation framework in this section, we introduce how gtr generates high-quality labeled training data for ed, i.e. s @xmath0, y event }. ## dynamic weighting data revision can mitigate the aforementioned noise issues. as the scale of generated data increases, residual noise will inevitably continue to grow, constraining further improvements in model performance. therefore, we also propose a dynamic weighting module g(x, θ) that assigns weight coefficients to c c l 2 0 2 4 samples in s syn . ## experiments datasets we perform experiments on three different ed datasets including ace05 @xcite , maven @xcite and commodities news @xcite , which contain 33, 168 and 18 event types respectively. we refer to previous works to partition the test set from ace05 @xcite and commodities news @xcite for evaluating. for maven, we use the officially provided validation set for testing. baselines we train a model without weighting on synthetic data and compare it with these baselines in zero-shot setting: uie @xcite , zeop @xcite ) zed @xcite , gpt-3.5turbo and gtr prompt. compared with gtr, gtr prompt omits the data generation stage and treats the targeted test set as unlabeled generated data (generated triggers are used in this stage). then gtr prompt regards the labeled data obtained through the data revision stage as final results. we train a model with weighting and compare the model with these baselines in few-shot setting: uie @xcite , star @xcite , gpt-3.5-turbo. ## conclusion ed has long faced challenges of data scarcity and the time-consuming nature of manual annotation. the development of llms brings a ray of hope to address these issues. in this paper, we proposes gtr, the data synthesis framework, which includes three stages: trigger expansion, data generation, and data revision, along with a dynamic weighting module for synthetic data. in the experimental section, we test the generative performance of gtr in zero-shot and low-shot scenarios. compared to direct prompting and other generation methods, our framework exhibits significant advantages, with the dynamic weighting mechanism improving the model's performance ceiling. we hope that the research in this paper can inspire future work in the field of data synthesis.