GPT - F in RE : In-context Learning for Financial Relation Extraction using Large Language Models (2023)

Abstract: Relation extraction (RE) is a crucial task in natural language processing (NLP) that aims to identify and classify relationships between entities mentioned in text. In the financial domain, relation extraction plays a vital role in extracting valuable information from financial documents, such as news articles, earnings reports, and company filings. This paper describes our solution to relation extraction on one such dataset REFinD. The dataset was released along with shared task as a part of the Fourth Workshop on Knowledge Discovery from Unstructured Data in Financial Services, co-located with SIGIR 2023. In this paper, we employed OpenAI models under the framework of in-context learning (ICL). We utilized two retrieval strategies to find top K relevant in-context learning demonstrations / examples from training data for a given test example. The first retrieval mechanism, we employed, is a learning-free dense retriever and the other system is a learning-based retriever. We were able to achieve 3rd rank overall. Our best F1-score is 0.718.

## introduction the emergence of large language models (llms) such as gpt-3 @xcite [12] represents a significant advancement in natural language processing (nlp). these models have expertise in variety of domains and hence they can be used as it is in multiple nlp tasks. traditionaly language models use separate pre-training-and fine-tuning pipelines @xcite [3] [5] [4] @xcite where fine-tune stage follows pre-training. models are fine-tuned on a task-specific dataset in a fullysupervised manner. more recently a new paradigm known as in-context learning (icl) @xcite is being used which formulates an nlp task such that llms make predictions by learning from demonstrations. these demonstrations are presented to the llms in the context prompt itself. under the framework of icl, llms achieve remarkable performance rivaling previous fully-supervised methods even with only a limited number of demonstrations provided in the prompt in various tasks such as solving math problems, commonsense reasoning, text classification, fact retrieval, natural language inference, and semantic parsing @xcite [14] @xcite . recently, icl based approach @xcite is utilized for relation extraction (re) task. re seeks to identify a semantic relationship between a given entity pair mentioned in a sentence, which is the central task for knowledge retrieval requiring a deep understanding of natural language. the approach achieves improvements over not only existing gpt-3 baselines, but also on fully-supervised baselines. specifically, it achieves sota performances on the semeval and scierc datasets, and competitive performances on the tacred and ace05 datasets. retrieval of examples to demonstrate is a key factor in the overall performance on these pipelines. llms can relate to the presented "to be predicted" data point more if the contextual examples predicted are similar to it. more relevant examples help us to leverage more out from llms both in terms of improvement in performance and less hallucination as examples can demonstrate model not to hallucinate in some cases. in this paper, we employed gpt-3.5 turbo and gpt-4 under the framework of icl for the relation extraction task on refind dataset. we utilized two retrieval strategies to find top k relevant in-context learning demonstrations / examples from training data for a given test example. the first mechanism we have employed is a learning-free dense retriever. the other system we have utilized is a learning-based retriever @xcite . let c denote the input context and e1 in c, e2 in c denote the pair of entity pairs. given a set of predefined relation classes r, relation extraction aims to predict the relation y in r between the pair of entities (e1, e2) within the context c, or if there is no predefined relation between them, predict @xmath0. ## gpt-finre gpt-re is formalized under the icl framework, using gpt models as shown in figure -3 . ## experiments due to limited access and cost associated with openai, we performed 4 primary experiments on the test dataset. we tried various rule based heuristics to improve the f1score, but it didn't work as expected. we used retriever implementations from 1 . ## results the results are shown in table-1. our best f1-score is 0.718. we got 4th position in the shared-task. we find that gpt 4 performs better than gpt 3.5 turbo. we also find that learning based retriever (epr) outperforms learning-free retriever (knn with openai embeddigs). ## future work in future we want to utilize gpt 4 for epr. we also want to use different retrieval approaches such as compositional exemplars for in-context learning (ceil) @xcite . ## conclusion this work explores the potential of gpt + icl on financial relation extraction (refind dataset). we used two retrieval mechanisms to find similar examples: (1) knn with openai embeddings (2) epr. we tried two different gpt models: (1) gpt 3.5 turbo and gpt 4. the experimental results show that gpt 4 with learning based retriever epr is giving the best f1-score of 0.718.