Cold-Start and Interpretability: Turning Regular Expressions into Trainable Recurrent Neural Networks (2020)

Abstract: Neural networks can achieve impressive performance on many natural language processing applications, but they typically need large labeled data for training and are not easily interpretable. On the other hand, symbolic rules such as regular expressions are interpretable, require no training, and often achieve decent accuracy; but rules cannot benefit from labeled data when available and hence underperform neural networks in rich-resource scenarios. In this paper, we propose a type of recurrent neural networks called FA-RNNs that combine the advantages of neural networks and regular expression rules. An FA-RNN can be converted from regular expressions and deployed in zero-shot and cold-start scenarios. It can also utilize labeled data for training to achieve improved prediction accuracy. After training, an FA-RNN often remains interpretable and can be converted back into regular expressions. We apply FA-RNNs to text classification and observe that FA-RNNs significantly outperform previous neural approaches in both zero-shot and low-resource settings and remain very competitive in rich-resource settings.

## introduction over the past several years, neural network approaches have rapidly gained popularity in natural language processing (nlp) because of their impressive performance and flexible modeling capacity. nevertheless, symbolic rules are still an indispensable tool in various industrial nlp applications. regular expressions (re) are one of the most representative and useful forms of symbolic rules and are widely used for solving tasks such as pattern matching @xcite and intent classification @xcite . re-based systems are highly interpretable and therefore support fine-grained human inspection and manipulation. for example, individual re rules in a system can be easily added, revised, or removed to quickly adapt the system to changes in the task specification. moreover, re-based systems do not require a training stage with labeled data and hence can be quickly deployed with decent performance in zero-shot scenarios. however, res rely on human experts to write and often have high precision but moderate to low recall; re-based systems cannot evolve by training on labeled data when available and thus usually underperform neural networks in rich-resource scenarios. how to combine the advantages of symbolic rules and neural networks is an open question and is drawing increasing attention recently. one possible way is to use rules to constrain neural networks, usually in the manner of regularization via knowledge distillation @xcite and multi-task learning @xcite , or by tuning the output logits of neural networks @xcite . in this way, information from rules can be injected into neural networks, though the neural networks still require training and remain black boxes that are hard to interpret and manipulate. another way of utilizing rules is to design novel neural network architectures inspired by rule systems @xcite @xcite . models designed based on this idea usually achieve better interpretability, but they must be trained on labeled data and cannot be directly converted from rules or manually specified by human experts because of their structural differences from rule systems. in this paper, we propose finite-automaton recurrent neural networks (fa-rnn), a novel type of recurrent neural networks that is designed based on the computation process of weighted finite-state automata. because of the equivalence between 1 : re for matching sentences asking about distance, and a matched sentence. '$' is the wildcard. '|' is the or operator. '*' is the kleene star operator. we also show the finite automaton converted from the re. s 2 is the final state. res and finite-state automata, we can convert any res into an fa-rnn, which can be deployed in zero-shot and cold-start scenarios. when there are labeled data, the fa-rnn can also be trained in the same way as any neural network, which improves its prediction accuracy over the original res. the fa-rnn has good interpretability. when converted from res, it is (approximately) equivalent to the res and is fully interpretable. even after training, it often remains highly interpretable and can be converted back into res. the interpretability of fa-rnns opens the possibility of fine-grained manipulation such as integrating new res into a trained fa-rnn and disabling old res that are used to initialize an fa-rnn. we apply fa-rnns to the text classification task and compare them with neural network baselines as well as existing approaches of integrating res and neural networks. our experiments find that fa-rnns show clear advantages in both zero-shot and low-resource settings and remain very competitive in rich-resource settings. ## method we show step-by-step how we can convert res to a novel type of recurrent neural networks called fa-rnns. ## interpretability we regard the approximate equivalence between re/wfa and our re-initialized model as indication of good interpretability based on the following two reasons: 1) for people who are familiar with res and automata, our model is interpretable once converted back into a wfa; 2) for non-experts who are unfamiliar with automata and res, we may run a re/wfa of a specific classification label on an input sentence and show which part of the sentence contributes to the (best) matching of the re/wfa with the sentence, which can be easily understood by non-experts. note that not only can an fa-rnn be easily converted back into a re/wfa at initialization, but the conversion can also be done after training. we can use the trained parameters of the fa-rnn θ @xmath0, d1 , d2 , ĝ and word embedding matrix e w to reconstruct the wfa tensor t . in our experiments, we find that although the reconstructed re systems underperform the corresponding trained fa-rnns because of thresholding and rounding during reconstruction, they often outperform the original res. the reconstructed re systems achieve 73.6% accuracy for qc (+9.2% compared with the original res) and 87.45% for atis (+0.45% compared with the original res). for sms, the reconstructed res underperform the original ones (-1.2%) probably because the original res are already good enough. we show an example in fig. 4 , in which our model can be seen to learn interesting new patterns such as 'jet' and '737'. we show another example in appendix.f. good interpretability of our models opens the possibility of fine-grained manipulation of the model, e.g., adding new res without retraining the model. to inject a new set of res, we convert them to a new fa-rnn with parameters θ @xmath1, d 1 , d 2 , g and merge it into the original trained fa-rnn with parameters θ re by concatenating the parameter matrices: to add new logical aggregation rules, we can update the aggregation layer parameters similarly by concatenation. to disable an re in an fa-rnn, we reconstruct the wfa, remove all the states of the re from the wfa except those that can be reached from states of other res, and finally convert the wfa back to an fa-rnn. neural networks enhanced by rules hu et al. (2016); li and rush (2020) use rules to constrain neural networks by knowledge distillation and posterior regularization. awasthi et al. (2020) inject rule knowledge into neural networks using multitask learning. lin et al. (2020) train a trigger matching network using additional annotation and use the output of trigger matching results as the attention of a sequence labeler. rocktäschel et al. (2015); xu et al. (2018); hsu et al. (2018) use parsed rule results to regularize neural network predictions by additional loss terms. li and srikumar (2019); luo et al. (2018) inject declarative knowledge in the form of parsed re results or first-order expressions into neural networks by hacking the prediction logits or the attention scores. hu et al. (2016); hsu et al. (2018) use rules as additional input features. all these previous methods use matching results or truth values of rules to enhance existing neural models. in contrast, we directly turn res into a novel type of trainable networks. relating neural networks and wfa schwartz et al. (2018) propose a type of neural networks for learning soft surface patterns (a subset of res), which is inspired by wfas but cannot be converted from wfas or surface patterns. in contrast, our fa-rnn can be initialized from res and converted back to res. peng et al. (2018); dodge et al. (2019) formulate the update of each hidden dimension of various rnn architectures as a small wfa (2-4 states). weiss et al. (2018); merrill (2019) provide theoretical analysis of various neural networks and their accepting languages. our work differs from these more theoretical studies in that we aim for a practical text classification approach. omlin et al. (1998); giles et al. (1999) show the equivalence between wfa and second-order rnn. the main differences between our model and theirs include the following. first, compared with the undecomposed version of our fa-rnn, their rnn model involves nonlinear activation functions which complicate the model. second, our fa-rnn further decomposes the tensor parameter, integrate word embeddings, and propose the gated and bidirectional extensions. third, while their work is mostly theoretical, we empirically show the usefulness of our model in text classification. 8 conclusion and future work we propose a type of recurrent neural networks called fa-rnn. it can be initialized from res and can also learn from data, hence applicable to various scenarios including zero-shot, cold-start, low-resource and rich-resource scenarios. it is also interpretable and can be converted back into res. our experiments on text classification show that it outperforms previous neural approaches in both zero-shot and low-resource scenarios and is very competitive in rich-resource scenarios. in the future, we plan to apply fa-rnn to other tasks and explore other variants of fa-rnn. we release our data, re rules and code at https://github.com/ jeffchy/re2rnn . gail weiss, yoav goldberg, and eran yahav. 2018. on the practical computational power of finite precision rnns for language recognition. arxiv preprint arxiv:1805.04908. jingyi xu, zilu zhang, tal friedman, yitao liang, and guy van den broeck. 2018. a semantic loss function for deep learning with symbolic knowledge. 35th international conference on machine learning, icml 2018. shanshan zhang, lihong he, slobodan vucetic, and eduard dragut. 2018. regular expression guided entity mention mining from noisy web data. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 1991-2000, brussels, belgium. association for computational linguistics.