L ogi T orch: A P y T orch-based library for logical reasoning on natural language (2022)

Abstract: Logical reasoning on natural language is one of the most challenging tasks for deep learning models. There has been an increasing interest in developing new benchmarks to evaluate the reasoning capabilities of language models such as BERT. In parallel, new models based on transformers have emerged to achieve ever better performance on these datasets. However, there is currently no library for logical reasoning that includes such benchmarks and models. This paper introduces LogiTorch, a PyTorch-based library that includes different logical reasoning benchmarks, different models, as well as utility functions such as co-reference resolution. This makes it easy to directly use the preprocessed datasets, to run the models, or to finetune them with different hyperparameters. LogiTorch is open source and can be found on GitHub.

## introduction machine reasoning over natural language has been an object of research since the 1950s @xcite . one prototypical task in the domain is textual entailment: given a premise (such as "i ate a cake"), the goal is to determine whether a hypothesis ("i ate something sweet") is entailed or not. other logical reasoning tasks are question answering, multiple choice question answering, and proof generation. lately, deep learning models have shown impressive performance on tasks such as these, in particular transformer-based models such as bert @xcite and gpt-3 @xcite . however, the models can be distracted easily by trap words, syntactic variations (kassner and schütze, 2020), or negation @xcite @xcite @xcite . hence, the question of whether these models can logically reason on text is still open @xcite . new models are being created incessantly (e.g., logigan @xcite and logiformer @xcite in 2022), and new datasets are being created to evaluate these models, including, e.g., logiqa @xcite and proofwriter @xcite . the initiative of open-sourcing toolkits has accelerated the progress in the field of natural language processing, driven by projects such as transformers @xcite from huggingface and stanza @xcite from stanford. however, this progress has not yet arrived in the field of logical reasoning: researchers still have to find and download different models, parameterize them, find the corresponding datasets, bring them into suitable formats, and fine-tune the models. the datasets are maintained on different web pages, exhibit different formats (json vs. full text, numerical vs. textual labels, etc.), and follow different conventions, which makes it cumbersome to apply one model across several sources. the models themselves are implemented in different frameworks, have different input and output formats, require different dependencies, and differ in the way of running them, which makes it burdensome to exchange one model for another. some models are not even available online, but have to be re-implemented from scratch based on the diagrams in the scientific publications. all of this hinders reproducibility, re-usability, comparability, and ultimately scientific progress in the area. in this paper, we propose to bring the benefits of open source libraries to the domain of logical reasoning: we build a python library, logitorch, that includes 14 datasets and 4 implemented models for 3 different logical reasoning tasks. all models can be called in a unified way, all datasets of one task are available in the same standardized format, and all models can be run with all datasets of the same task. all models have been re-implemented from the research papers that proposed them, and they have been validated by subjecting them to the same experiments as the original papers, with comparable results. more models and benchmarks are in preparation. logitorch works on top of pytorch @xcite , and uses the transformers library. it also includes utility functions used for preprocessing, such as coreference resolution and discourse delimitation. the rest of the paper is organized as follows. section 2 discusses the design and components of logitorch, and describes the datasets, utility functions, and models. section 3 shows the experimental results of our implemented models on different logical reasoning tasks. we conclude in section 4. ## logitorch logitorch is our python library for logical reasoning on natural language text. figure 1 shows the tree structure of our library. it is built on top of pytorch and consists of 5 parts: datasets. we gathered different logical reasoning datasets that allow users to evaluate the reasoning capabilities of deep learning models on natural language. once a dataset is called from logitorch, it is downloaded, and wrapped into an object that inherits the dataset class of pytorch. this means that all datasets are accessible via the same interface. we describe the datasets in detail in section 2.1. data collators. different models require different preprocessing steps for the same data and same task: one model may work on numerical vectors, the other on textual input. hence, we designed, for each pair of a dataset and a model, a data collator that brings the dataset into the format required by the model. utilities. some models require supplementary features in addition to the input text. for example, the dagn model @xcite requires the discourse structure of the input in order to create a logical graph representation of it. for such cases, logitorch provides different utility functions, most notably for discourse structure analysis, coreference resolution, and logical expression extraction, which we discuss in section 2.2. models. logitorch provides several deep learning models that have been designed to perform logical reasoning tasks such as proof generation and textual entailment. for each model, we either provide an implementation from scratch, or a wrapper over its original implementation. for the logitorch datasets qa mcqa proof_qa te data_collators utilities models pl_models transformer-based models, we use the transformers library from huggingface for the implementation of the models. we describe the models in detail in section 2.3. pytorch lightning models. for each implemented model, we also provide a pytorch lightning version. it includes the model, the optimizer, the training loop, and the validation evaluation. for example, the prover model @xcite has a pytorch lightning version called plprover. this allows users to play with features such as multi-gpu and fast-low precision training without modifying the training loop. ## evaluation we compared the performance of each model in logitorch to the performance of the model in the original paper on the same datasets: we trained the ruletaker model on the training set of rule-taker with language reasoning paths up to depth 5 and tested it on its testing set; we trained the prover and proofwriter models on the training set of proofwriter with language reasoning paths up to depth 5 and tested them on the corresponding testing set; and we trained the bertnot model (a pretrained bert base cased model) on the mlm task, with the negated wikipedia corpus provided by @xcite (included in logitorch), finetuned the model on each te dataset (mnli, snli, and rte) and tested it on its negated counterparts @xcite . all models use the same settings as in the original papers. ## conclusion we have introduced logitorch, a python library for logical reasoning on natural language. it is built on top of pytorch in combination with the transformers and pytorch lightning libraries. logitorch includes an extensive list of textual logical reasoning datasets and utility functions, and different implemented models. the library allows researchers and developers to easily use logical reasoning datasets and train logical reasoning models with just a few lines of code. the library is available on github and is under active development. for future work, we will add new datasets, and implement models such as dagn, focal reasoner, and logigan with their utility functions for feature engineering. finally, we want to invite researchers and developers to contribute to logi-torch. we believe that such a library will lower the hurdles to research in the area, foster re-usability, encourage comparative evaluation, strengthen reproducibility, and advance the culture of open software and data.