Neuralizing Regular Expressions for Slot Filling (2021)

Abstract: Neural models and symbolic rules such as regular expressions have their respective merits and weaknesses. In this paper, we study the integration of the two approaches for the slot filling task by converting regular expressions into neural networks. Specifically, we first convert regular expressions into a special form of finite-state transducers, then unfold its approximate inference algorithm as a bidirectional recurrent neural model that performs slot filling via sequence labeling. Experimental results show that our model has superior zero-shot and few-shot performance and stays competitive when there are sufficient training data.

## introduction neural approaches almost dominate recent natural language processing (nlp) research. the tremendous success of neural networks benefits from their strong capability of learning from a large amount of annotated data. on the other hand, systems based on symbolic rules, while being no longer the mainstream approach to nlp research, are still very widely used in practice because of their interpretability, trustworthiness and decent zero-shot performance in data-scarce scenarios. therefore, how to integrate neural and symbolic approaches while retaining their respective advantages is becoming an active research direction. in this paper, we try to integrate neural networks with regular expressions (re), one of the most widely used forms of symbolic rules. we focus on the slot filling task (sf), a typical application scenario of res that aims to identify words in a sentence that carry specific information. for example, given the sentence "show me the flights from new york to dallas.", sf aims to tag the span "new york" as the content of slot fr.city (the departing city). an re specifies text patterns of both con-tents and contexts and uses capturing groups to mark target contents. existing methods that integrate neural networks and symbolic rules either use rule outputs as pseudo-labels to distill knowledge into neural networks @xcite , or use rule outputs to influence attention weights or output logits of neural networks @xcite @xcite . these integration methods can outperform both rules and pure neural networks when there are sufficient training data. however, since they still require training, their performance is well below that of rules in zero-shot and low-resource settings. more recently, @xcite proposed a novel method that converts res into neural networks, which can match the performance of the original res without training and can compete with previous integration methods when trained. unfortunately, their method is designed for text classification and cannot be easily extended to the sf task because it does not model re capturing groups and only computes sentence-level scores. in this work, we propose to neuralize res with capturing groups for the sf task. our method is inspired by @xcite but differs from their work in many aspects. specifically, we treat sf as a sequence labeling task with the bio tag scheme and convert res with capturing groups into finite-state transducers (fst) with restricted output dependencies. we propose an approximate fst inference algorithm and unfold it as a neural model that resembles bidirectional rnn models for sequence labeling. our model is approximately equivalent to the original res and can be further improved by training on labeled data. we also propose several techniques to enhance the model without harming its initial approximation to the res. we conduct our experiments on three popular sf datasets involving two domains and two languages. results show that our model has superior performance on zero-shot and low-resource settings compared with all the previous methods and remains competitive on rich-resource settings. we provide our source code as well as a handy toolkit for writing and converting res in https: //github.com/jeffchy/re2nn-seq . ## method in this section, we introduce the several steps of neuralizing res (as shown in fig. 1 ) as well as the decoding and training methods. @xcite rigorously show that any re with capturing groups can be converted into an fst. the equivalence of the re and fst is reflected in: (1) re matches a sentence if and only if the corresponding fst has at least one accepting path; ## analysis ablation study we conduct an ablation study on the 100% training samples on the three datasets in utilizing unlabeled data we assume we do not have unlabeled data in previous experiments. however, with clean unlabeled data, we can use res to produce pseudo-labels and use them to train baselines. we compare the results of training bi-gru+(none/i/o)+crf using different amounts of unlabeled data and fstrnn without any training on snips (fig 4 ). results show that we need ## conclusion and future work in this work, we neuralize a symbolic re system for slot filling into a trainable neural network model. the model approximates res well initially and can be trained to improve itself with labeled data. experiments in various settings show the advantages of our method. to the best of our knowledge, we are the first to neuralize res into neural networks for the slot filling task. for future work, we want to explore the possibility of converting an fstrnn back into res for better interpretability, and investigate more on utilizing bert. we believe our methodology can be extended to other tasks such as ner and qa. we also hope our methods can provide insights into theoretical analysis on the relations between neural models and regular languages.