Multi-purpose neural network for F rench categorial grammars (2023)

Abstract: Categorial grammar (CG) is a lexicalized grammar formalism that can be used to identify and extract the semantics of natural language sentences. However, despite being used actively to solve natural language understanding tasks such as natural language inference or recognizing textual entailment, most of the tools exploiting the capacities of CG are available in a limited set of languages. This paper proposes a first step toward developing a set of tools enabling the use of CG for the French language by proposing a neural network tailored for part-of-speech and type-logical-grammar supertagging, located at the frontier between computational linguistics and artificial intelligence. Experiments show that our model can compete with state-of-the art models while retaining a simple architecture.

## introduction categorial grammar (cg) is a formalism whose foundations come from @xcite and @xcite . from there, we can find two major lines of research that were created, namely, combinatory cg (ccg) @xcite and type-logical grammar (tlg) @xcite which itself can be divided into two subtheories, namely, displacement calculus @xcite and multi-modal cg @xcite . other theories that build upon those theories also exist, such as hybrid tlcg @xcite and abstract cg @xcite . using these syntactic theories offers knowledge about each word passed in an input sentence. using the appropriate resources, the great amount of information provided by a supertag @xcite attributed to a given word in a sentence can be parsed efficiently to solve natural language understanding tasks such as natural language inference or recognizing textual entailment. this syntax-semantic interface can then be used by machines in order to answer various kinds of challenges, such as question answering and text summarization. the continuous development of ccg and tlg led to the progressive appearance of several annotated corpora in various languages, such as german @xcite , italian @xcite , japanese @xcite , and of course english @xcite . however, the number of treebanks and tools is very limited for the french language. because cg has a close affinity to lambda calculus, logic, and natural deduction proofs, we are motivated to develop the current state-of-the-art in this field for the french language. in this work, we propose a simple supertagger for part of speech (pos) and tlg tagging by exploiting the capacities of deep bidirectional encoder representation from transformers (bert) @xcite for unlabeled input sentences. we demonstrate that integrating into our architecture a small long short-term memory (lstm)based variational autoencoder (vae) while adapting the training pipeline allows us to increase the word-wise supertagging accuracy of our model. we also show experimentally that joining the training of both pos and tlg supertagging offers slightly increased overall accuracy while reducing the accuracy of tags seen rarely during training. ## related works french tlg and pos supertagging the tl-gbank @xcite is a type-logical treebank for french, developed from the french treebank, a lexical and syntactic resource by @xcite . because both corpora have been manually verified and rectified by their respective authors, they can be considered as the gold standard for french cg. alongside his tlgbank, moot pre-sented the supertagger deepgrail, foot_0 which is an lstm layer that uses elmo (embeddings from language models) vector embeddings of the unlabeled input data. this model successfully assigns 93.2 percent of words their correct tlg formula and presents an accuracy of 99.1 percent of correct pos supertags. since then, state-of-the-art tlg supertagging of this treebank has been achieved by @xcite with an accuracy of 95.92 percent. their approach revisits traditional models by proposing a framework based on heterogeneous dynamic graph convolutions and by decomposing the structure of the supertags. by doing so, they presented novel accuracy results on supertags that were rarely seen during the training phase. this generalization effort motivated us to explore different ways to regularize our architecture without losing overall model accuracy. camembert our approach is built around the use of camembert @xcite , which is a fine-tuned roberta model @xcite for french, which itself is based on bert @xcite . this model is attractive for the french language because it uses a subword tokenization where each word is divided, so it can exploit the numerous inflections that appear in the french language. in the study reported herein, we found only a few differences between the experimental results of camembert base and camembert large models. therefore, for the sake of computing speed and efficiency, we used only camembert base in our model because its architecture is three times smaller than its other version. ## tlg and pos supertagger model in this section, we describe the training data and procedure and present the different modules of our model. ## results in further investigations using this architecture are needed in future work to prove the efficiency of this model. however, its simple nature offers the opportunity to manipulate and adapt it easily, whether by modifying its structure or by simply adding new heads tailored to specific tasks. ## contributions and limitations with the goal in mind to provide a tool allowing to properly represent the syntax of input sentences formulated in natural language, we hope that future works will be able to extend the capacities of this architecture in order to exploit this syntaxsemantic interface. while our model has not improved the state of the art of french tlg supertagging, it presents an accessible and simple finetuning of existing transformer-based models. its modular architecture eases the adaptation of other existing techniques such as beam search to obtain more than a single prediction per word. however, this model fails to modelize the internal structure of the syntactic types in the sense that it does not learn to create new composed types (n/n, s\np) by assembling atomic types (n, np, s). the current state of the art presented by kogkalidis and @xcite solves this problem by using a graph-theoretic perspective. ## conclusion in this work, we investigated the different ways to regularize and fine-tune a supertagger for the french language, exploiting pre-trained unlabeled word embedding and a customized procedure utilizing a vae architecture. we used a goldstandard annotated corpus, tlgbank, to train a simple and adaptable model able to compete with the current state of the art of supertaggers. we have shown experimentally that a vae can be used to improve model regularization and that overall accuracy can be improved by using a multi-headed architecture.