ETHAN at S em E val-2020 Task 5: Modelling Causal Reasoning in Language Using Neuro-symbolic Cloud Computing (2020)

Abstract: I present ETHAN: Experimental Testing of Hybrid AI Node implemented entirely on free cloud computing infrastructure. The ultimate goal of this research is to create modular reusable hybrid neuro-symbolic architecture for Artificial Intelligence. As a test case I model natural language comprehension of causal relations from open domain text corpus that combines semi-supervised language model (Huggingface Transformers) with constituency and dependency parsers (Allen Institute for Artificial Intelligence.)

## introduction several hard open problems of machine learning and ai are intrinsically related to causality (?). one of them is the algorithmization of counterfactuals @xcite : what algorithms would allow machines to recognize crucial semantic difference between sentences like if oswald didn't kill kennedy, someone else did, if oswald hadn't killed kennedy, someone else would have. @xcite what additional knowledge would a machine require to do such a binary classification? but going even further, what linguistic features of the two sentences would need to be isolated and analyzed before any form of causal inference can be performed? if the first question conveniently re-frames the linguistic problem into statistical one, then the second question requires machine to identify some basic causal relation in the text, namely the antecedent and consequent. semeval-2020 task 5 @xcite asks these two question in a form of subtasks 1 and 2 1 . even for a single language (english) the task so fundamental to natural language processing (nlp) that it bridges the major divide in the artificial intelligence (ai) research between the good old fashioned ai (gofai) and the methodology that led to the latest advances, namely the deep learning (dl). it is worth mentioning the intense debates 2 around the future of ai which inevitably circles back to the role of causality in human and machine understanding of language or any data for that matter. my results add to the position that any successful approach must be a hybrid between symbolic and connectionist sides of the divide (marcus, 2020) . i was able to achieve a sota f1 score 0.878 on the subtask 1 in a short time using open source software. however, i found subtask 2 significantly more challenging, another approach is to combine multiple modules each specialized in precise execution of one or few nlp tasks, as opposed to the steps in a pipeline. training of such modules can be done separately or jointly using the same or different datasets @xcite . moreover, the modules can be implemented using different methods of machine learning as appropriate for a particular task. pretrained dl models can be first fine-tuned on an intermediate task, before fine-tuning again on the target task of interest. these models, also referred to as stilts can target a narrowly defined model behavior or linguistic phenomenon @xcite . linguists distinguish between two different complex constructions involving if, tense, aspect, and modality: simple past conditionals vs. past perfect would-conditionals. the subtasks 1 and 2 target two substantially different linguistic phenomena, namely counterfactuals: a counter-factual statement can be converted to a contrapositive with a true antecedent and consequent. consider the "post-traumatic stress" example discussed above; it can be transposed into "because her post-traumatic stress was not avoided, (we know) a combination of paroxetine and exposure therapy was not prescribed". such knowledge can be not only used for analyzing the specific statement but also be accumulated across corpora to develop domain causal knowledge @xcite this objective requires an epistemic modality -we know. that is -counterfactual past perfect would conditionals (aka "subjunctives") carry additional information as compared to regular past conditionals (aka "indicatives") -the contrapositive conversion merely makes explicit the information already present in its surface form by converting it into its non-modal logical form. therefore, analyzing the surface form should allow that signal to be detected automatically. i will show how syntactic dependency parsing can be used to that end. ## system overview ethan is implemented as "cloud native node" that is essentially a pure algorithm executed in the computing cloud and using tabular data as input and output. its logic closely follows the aforementioned linguistic interactions between if and would. according to @xcite , there are nine importantly distinct tense combinations in counterfactuals and its logical forms can be reduced to canonical conterfactuals form if p, (then) q, where p antecedent and q is its consequent. i further reduced it to combinatorial interactions between the part-of-speech(pos) tags obtained from the sentence constituency parsing @xcite . ## conclusions dl is very effective at isolated tasks but is not sufficient for causal inference over language. causal modeling of language requires a modular architecture combining dl models trained on intermediate tasks with rule-based specialist modules. expansion can be achieved by layering symbolic modules with neural networks on a basis of the universal dependency scheme. cloud computing provides a flexible and open environment for developing hybrid ai modules at a very low cost.