LLM -supertagger: Categorial Grammar Supertagging via Large Language Models (2024)

Abstract: Supertagging is an essential task in Categorical grammar parsing and is crucial for dissecting sentence structures. Our research explores the capacity of Large Language Models (LLMs) in supertagging for both Combinatory Categorial Grammar (CCG) and Lambek Categorial Grammar (LCG). We also present a simple method that significantly boosts LLMs, enabling them to outperform LSTM and encoder-based models and achieve state-of-the-art performance. This advancement highlights LLMs’ potential in classification tasks, showcasing their adaptability beyond generative capabilities. Our findings demonstrate the evolving utility of LLMs in natural language processing, particularly in complex tasks like supertagging.

## introduction in natural language processing (nlp), many downstream tasks utilize the syntax representation such as categorial grammar (cg) of sentences @xcite . those syntax trees are typically obtained through a two-step process @xcite @xcite : tag assignment, and tree construction. the step for assigning tags to words is called supertagging. one possible way to improve the performance of supertagging is using llms. llms @xcite @xcite have experienced many breakthroughs in recent years and have been applied in various nlp tasks @xcite . however, their application to cg supertagging has received little attention. indeed, a conventional point of view is that, because most llms are decoderbased, they are more suitable for generation tasks rather than classification tasks. previous studies have shown that llms tend to underperform in classification tasks (e.g. @xcite . smaller encoder-based models, like roberta @xcite , operating in supervised settings have demonstrated the ability to match or even surpass the performance of much larger and more resourceintensive decoder-based generative llms @xcite . because cg supertagging is a classification task, modern techniques have only adopted encoder-based models @xcite @xcite . the capability of llm-based supertagging methods has not been fully explored. in this work, we study the ability of llms for cg supertagging. we compare different llms and present a novel method for improving their performance at supertagging tasks. without applying our new method, we find that llms are less accurate than standard bilstm-based supertaggers. this is because the architecture of encoder/bilstm-based models allows them to extract features of context provided by subsequent words, making them more advantageous than llms. we find that this disadvantage of llms can be addressed by repeating the input sentences. this new method enhances the capability of decoder-based models in supertagging tasks. the main contributions of our works are: large language models for nlp tasks recently, there has been a trend toward using llms for nlp tasks. for instance, @xcite employs in-context learning on gpt-3 for relation extraction(re). @xcite ; @xcite adapts llms to the named entity recognition (ner) task. llms have also been employed for other tasks such as text summarization @xcite and sentiment analysis @xcite . however, @xcite argue that llm enhances text annotation but requires human validation due to varying task effectiveness. ## method in this section, we will show the methodologies employed to utilize llms for supertagging. the focus will be on two primary aspects: in-context learning and fine-tuning a generative llm. ## conclusion we examine llm's ability to perform supertagging tasks. we find that providing repeating input to llms can boost their performance. our results show that generative models may also be applied to capture the syntax of natural languages. future works may re-evaluate the capability of decoderbased auto-regressive models on other classification problems.