Recurrent Neural Network CCG Parser (2023)

Abstract: The two contrasting approaches are end-to-end neural NLI systems and linguistically-oriented NLI pipelines consisting of modules such as neural CCG parsers and theorem provers. The latter, however, faces the challenge of integrating the neural models used in the syntactic and semantic components. RNNGs are frameworks that can potentially fill this gap, but conventional RNNGs adopt CFG as the syntactic theory. To address this issue, we implemented RNN-CCG, a syntactic parser that replaces CFG with CCG. We then conducted experiments comparing RNN-CCG to RNNGs with/without POS tags and evaluated their behavior as a first step towards building an NLI system based on RNN-CCG.

## introduction over the years, two contrasting approaches to natural language inference (nli) have emerged: endto-end neural nli systems based on large language models (llms) @xcite @xcite , which we call monomodular approaches, and linguistically-oriented nli pipelines consisting of syntactic parsers, semantic representations and theorem provers @xcite @xcite @xcite , which we call multi-modular approaches. while the former has become more popular in recent years and has shown remarkable progress with the increasing scale of llms, the latter offers high precision, explanatory properties and strength in higher-order reasoning such as arithmetic. both approaches have strengths and weaknesses and are expected to complement each other. a drawback of using neural networks in multimodular approaches is that their neural models are split between syntax and semantics. for example, the neural part-of-speech (pos) taggers cannot receive feedback from the results of the semantic component. the distributional representations in semantic components considered in works such as @xcite ; larsson (2020); @xcite are not connected to syntax. this gap between syntactic and semantic neural models is a potential weakness of multi-modular approaches compared to mono-modular approaches that seek to optimize the whole process of nli. the use of recurrent neural network grammars (rnngs) @xcite is a potential solution to bridge the gap between syntactic and semantic neural models in multi-modular approaches. rn-ngs provide syntactic parsers that can function as feeding input (syntactic structures) to semantic components, which is still a non-trivial task for large language models. furthermore, unlike standard syntactic parsers and large language models, rnngs provide embedded representations for phrasal constituents obtained by training on predicting syntactic structures, which we expect to be useful in a semantic component as well. one remaining challenge is that the underlying grammar of the current rnngs is contextfree grammar (cfg), while modern syntactic processing in the multi-modular approaches adopts mildly context-sensitive grammars such as combinatory categorial grammar (ccg) @xcite . therefore, in this study, we attempt to implement rnn-ccg, a syntactic parser that replaces the underlying grammar of rnngs from cfg to ccg, and compare the performance of rnn-ccg with rnngs, as a first step towards developing a complete nli system using rnn-ccg. technically, rnn-ccg can be built using almost the same techniques as rnngs, but we will show that its performance is slightly better than rnngs. shift pop the word vector from buffer and push it to stack. nt x push a vector corresponding to the nonterminal symbol x to stack. this non-terminal symbol x is marked as open. reduce pop from stack all the elements up to the first open non-terminal symbol x encountered. generate a new vector that encodes them and push it back to stack as a new element. at each time step, stack, buffer, and history of actions are encoded using lstms and rnns. parsing is performed by determining the next action at each parsing state based on this encoding. it is inefficient to recompute the encoding of stack every time; thus, rnngs adopts a mechanism called stack lstms @xcite for optimization. rnngs have been the subject of subsequent researches: stack-only rnngs @xcite , which eliminate buffer from the architecture and use only stack, a pytorch implementation model @xcite ) that enables parallel execution and learning of larger data, and a model that uses transformer instead of rnns @xcite . however, in this paper, we focus on comparing cfg and ccg as underlying syntactic theories of rnngs, adopting the simplest model presented in the original paper @xcite and conducting experiments focusing on the parsing aspect. ## rnn-ccg we implemented two models based on rnngs: rnn-cfg, which is a re-implementation of rn-ngs using cfg, and rnn-ccg, which uses ccg instead of a cfg for the grammar used in rnngs. by treating ccg syntactic categories as cfg terminal symbols, ccg is regarded as an instance of phrase-structure grammar, and the action selection is a multi-class classification task similar to the case of rnn-cfg. however, there was a problem with rnn-ccg in that its syntactic structures do not provide a layer for pos tags, which is insufficient to be used for semantic composition. therefore, in this research, we extend rnn-ccg so that the structures have syntactic categories corresponding to words. for the sake of comparison, we also implement rnn-cfg that outputs non-terminal symbols corresponding to words. ## conclusion in this study, we implemented rnn-ccg, a syntactic parser in which the grammar used inside the rnngs was replaced from cfg to ccg, and conducted comparative experiments with rnn-cfg, a reimplementation of classical rnngs. we also implemented their extensions with pos tags considering syntactic categories corresponding to words. the results showed that the implementation of rnn-ccg achieved a higher f1 score than rnn-cfg with respect to the prediction of actions. moreover, both models function effectively when considering pos tags, providing a better interface for semantic composition in the case of rnn-ccgs. overall, rnn-ccg is a prospective candidate of syntactic parsers in a modular nli approach that bridges the gap between neural networks within ccg parsers and semantic modules. future research could investigate the fusion of rnn-ccg with semantic composition and logical reasonings.