Logical Transformers: Infusing Logical Structures into Pre-Trained Language Models (2023)

Abstract: Natural language contains rich logical structures and logical information, and correctly detecting and accurately understanding these logical structures and information underlying natural language texts is very crucial for NLP models’ performance on many important NLU and NLG tasks. Existing pre-trained language models based on the transformer architecture mostly adopt a classical design for constructing their input embeddings that ignores the logical structures underlying natural language texts, thus limiting their ability to better capture and encode key logical information in the input sequences. To overcome such limitations, in this paper we first propose a novel approach to construct logic-aware input embeddings for transformer language models through a combination of logic detection, logic mapping and hierarchical logical projections, and then develop a corresponding new modeling paradigm that can upgrade existing transformer language models into logical transformers to boost their performance on different NLU and NLG tasks. Our empirical experiments on four important and challenging NLU and NLG tasks demonstrate that our proposed logical transformer language models can achieve superior performance over their baseline transformer models through a deeper understanding of the logical structures of texts.

## introduction natural language contains rich logical structures and logical information @xcite ) that are crucial to a deep and accurate understanding of its meaning. therefore, the ability to correctly detect and accurately understand the logical structures and information within natural language texts is very crucial for nlp models' performance on many important natural language understanding (nlu) and natural language generation (nlg) tasks. the types of logics contained in natural language are very diverse, including not only mathematically well-defined propositional logic and firstorder logic @xcite , but also more general types of natural and structural logical relationships that people frequently use in natural language texts to convey and communicate their ideas and meanings more effectively and clearly. in recent years we have witnessed huge progress and success in many fields of natural language processing brought about by the introduction of all different kinds of pre-trained language models @xcite @xcite @xcite @xcite based on the transformer architecture @xcite . most existing pre-trained language models adopt the classical approach for constructing the input embeddings that are fed into the encoder parts of the language models, which can be summarized as the summation of the following three key components @xcite : this classical design of the input embeddings has been proven to be very effective at capturing important semantic and positional features from natural language texts and helping pre-trained language models to learn good contextualized representations of the input textual sequences @xcite . however, it also has a very important limitation -it doesn't consider or try to explicitly encode the logical structures underlying the text inputs, which are also very crucial for the deep and accurate understanding of the meaning of the text inputs. therefore, in order to overcome this limitation and to enable pre-trained language models to better capture and understand the important logical structures underlying natural language texts, in this paper we propose a novel approach to construct logicaware input embeddings for transformer-based pre-trained language models and a corresponding new modeling framework that can upgrade existing transformer language models into logical transformers to boost their performance on different nlu and nlg tasks. our new approach consists of two major modules: (1) logic detection and mapping, and (2) multi-layer hierarchical logical projections. it has the following key advantages: ## logical relationships and keywords in this work, we consider logical relationships in natural language texts as the underlying relationships among different language constituents that carry meaningful information regarding logical understanding and reasoning of the texts. in natural language, such logical relationships are usually indicated by logically-connective keywords and phrases. in this paper, we define a taxonomy of 12 most commonly seen types of logical relationships and their corresponding sets foot_1 of logical keywords (including phrases foot_2 ) for natural language foot_3 : 1. conjunction: a conjunction logical relationship indicates that the two language constituents involved are presented jointly in addition to each other. its logical keywords are: ## logic detection and mapping in this section, we describe our logic detection and mapping module based on keyword detection and constituency parsing. for each sentence s in the source text, we first perform constituency parsing @xcite over s to obtain its constituency parsing tree t (s). in this paper, we use the berkeley neural parser @xcite to perform constituency parsing. a l w p r j h e p f 6 3 j e j c w a z w w b i p m 5 f z m j s c d a b f q 0 i b j r l / 8 l n y r t x t v e n v d u n v i 4 c u g c x a a r 5 k i a a q j b 1 e j t r n a d e k i v 6 n w k r w f r z x p f t w 5 y + c w z + g x r 4 x v s w o 9 @xmath0 figure 3 : an example execution of algorithm 1 on the example sentence 'but i try very hard not to go that way because it would be too easy for them.' over the three detected logical keywords 'but', 'because', 'not'. then we search through all the constituent nodes in t (s) to detect the ones that exactly matches the keyword strings of the logical keywords as defined in section 2. let n key (s) denote the set of constituent node in t (s) that matches logical keywords. then for each logical keyword node n @xmath1, we fetch its parent constituent node pa(n k ). now we have three different cases: ## experiments in order to evaluate our proposed logical transformer architecture's performance boost on different nlu and nlg tasks with different transformer language models, in our experiments, we test it on three nlu datasets and one nlg dataset. ## related work recently there has been increasing interest in improving pre-trained language models' logical reasoning ability @xcite . for example, @xcite proposed a new method for parsing natural language into the forms of propositional logic and first-order logic using dual reinforcement learning. @xcite proposed a new unsupervised adversarial pre-training method, called logigan, in order to enhance language models' abilities of logical reasoning. @xcite proposed a new logiformer architecture based on a two-branch graph transformer network to improve language models' performance on interpretable logical reasoning. in contrast to these previous work that mostly focus on introducing new training methods or constructing complex model architectures, our proposed method in this paper only modifies the input embeddings and is thus more straightforward london, england (reuters) --harry potter star daniel radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on monday, but he insists the money won't cast a spell on him. daniel radcliffe as harry potter in "harry potter and the order of the phoenix" to the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. "i don't plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar," he told an australian interviewer earlier this month. "i don't think i'll be particularly extravagant. "the things i like buying are things that cost about 10 pounds --books and cds and dvds." at 18, radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film "hostel: part ii," currently six places below his number one movie on the uk box office chart. details of how he'll mark his landmark birthday are under wraps. his agent and publicist had no comment on his plans. "i'll definitely have some sort of party," he said in an interview. "hopefully none of you will be reading about it." radcliffe's earnings from the first five potter films have been held in a trust fund which he has not been able to touch. despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. "people are always looking to say 'kid star goes off the rails,'" he told reporters last month. "but i try very hard not to go that way because it would be too easy for them." his latest outing as the boy wizard in "harry potter and the order of the phoenix" is breaking records on both sides of the atlantic and he will reprise the role in the last two films. watch i-reporter give her review of potter's latest » . there is life beyond potter, however. the londoner has filmed a tv movie called "my boy jack," about author rudyard kipling and his son, due for release later this year. he will also appear in "december boys," an australian film about four boys who escape an orphanage. earlier this year, he made his stage debut playing a tortured teenager in peter shaffer's "equus." meanwhile, he is braced for even closer media scrutiny now that he's legally an adult: "i just think i'm going to be more sort of fair game," he told reuters. e-mail to a friend . copyright 2007 reuters. all rights reserved.this material may not be published, broadcast, rewritten, or redistributed. figure 6 : detected logical keywords in an example article from the cnn/dailymail dataset @xcite . it contains 7 different types of logical relationships: conjunction, disjunction, negation, comparative, adversative, temporal, and causal. and easily generalizable to different types of transformer language models. ## conclusion in this paper we introduced a new modeling paradigm for transformer language models that detects and extracts important logical structures and information from input texts and then integrates them into the input embeddings through carefully designed multi-layer hierarchical logical projections to infuse logical structures into pretrained language models. our empirical experiments on four important and challenging nlu and nlg tasks showed that our proposed logical transformer language models consistently perform better than their corresponding baseline transformer language models through a deeper understanding of the key logical structures underlying natural language texts.