Enhancing Interpretable Clauses Semantically using Pretrained Word Representation (2021)

Abstract: Tsetlin Machine (TM) is an interpretable pattern recognition algorithm based on propositional logic, which has demonstrated competitive performance in many Natural Language Processing (NLP) tasks, including sentiment analysis, text classification, and Word Sense Disambiguation. To obtain human-level interpretability, legacy TM employs Boolean input features such as bag-of-words (BOW). However, the BOW representation makes it difficult to use any pre-trained information, for instance, word2vec and GloVe word representations. This restriction has constrained the performance of TM compared to deep neural networks (DNNs) in NLP. To reduce the performance gap, in this paper, we propose a novel way of using pre-trained word representations for TM. The approach significantly enhances the performance and interpretability of TM. We achieve this by extracting semantically related words from pre-trained word representations as input features to the TM. Our experiments show that the accuracy of the proposed approach is significantly higher than the previous BOW-based TM, reaching the level of DNN-based models.

## introduction tsetlin machine (tm) is an explainable pattern recognition approach that solves complex classification problems using propositional formulas @xcite . @xcite , numerical data- @xcite , and image classification @xcite are recent areas of application. in natural language processing (nlp), tm has provided encouraging trade-offs between accuracy and interpretability for various tasks. these include sentiment analysis (sa) @xcite , word sense disambiguation (wsd) @xcite , and novelty detection @xcite . because tm nlp models employ bag-ofwords (bow) that treat each word as independent features, it is easy for humans to interpret them. the models can be interpreted simply by inspecting the words that take part in the conjunctive clauses. however, using a simple bow makes it challenging to attain the same accuracy level as deep neural network (dnn) based models. a key advantage of dnn models is distributed representation of words in a vector space. by using a single-layer neural network, @xcite introduced such a representation, allowing for relating words based on the inner product between word vectors @xcite . one of the popular methods is skip-gram, an approach that learns word representations by predicting the context surrounding a word within a given window length. however, skip-gram has the disadvantage of not considering the co-occurrence statistics of the corpus. later, @xcite glove -a model that combines the advantages of local window-based methods and global matrix factorization @xcite . the foundation for the above vector representation of words is the distributional hypothesis that states that "the word that occurs in the same contexts tend to have similar meanings" @xcite . this means that in addition to forming a rich high-dimensional representation of words, words that are closer to each other in vector space tend to represent similar meaning. as such, vector representations have been used to enhance for instance information retrieval @xcite , name entity recognition @xcite , and parsing @xcite . the state of the art in dnn-based nlp has been advanced by incorporating various pre-trained word representations such as glove @xcite , word2vec @xcite , and fasttext @xcite . indeed, building semantic representations of the words has been demonstrated to be a vital factor for improved performance. most dnn-based models utilize the pre-trained word representations to initialize their word embeddings. this provides them with additional semantic information that goes beyond a traditional bow. however, in the case of tm, such word representations cannot be directly employed because they consist of floating-point numbers. first, these numbers must be converted into boolean form for tm to use, which may result in information loss. secondly, replacing the straightforward bow of a tm with a large number of floating-point numbers in fine-grained boolean form would impede interpretability. in this paper, we propose a novel preprocessing technique that evades the above challenges entirely by extracting additional features for the bow. the additional features are found using the pre-trained distributed word representations to identify words that enrich the bow, based on cosine similarity. in this way, tm can use the information from word representations for increasing performance, and at the same time retaining the interpretability of the model. the rest of the paper is organised as follows. we summarize related work in section 2. the proposed semantic feature extraction for tm is then explained in section 3. in section 4, we present the tm architecture employing the proposed feature extension. we provide extensive experiment results in section 5, demonstrating the benefits of our approach, before concluding the paper in section 6. ## related work conventional text classification usually focuses on feature engineering and classification algorithms. one of the most popular feature engineering approaches is the derivation of bow features. several complex variants of bow have been designed such as n-grams @xcite and entities in ontologies @xcite . apart from bow approaches, tang et al. demonstrated a new mechanism for feature engineering using a time series model for short text samples @xcite . there are also several techniques to convert text into a graph and sub-graph @xcite . in general, none of the above methods adopt any pre-trained information, hence have inferior performance. deep learning-based text classification either depends on initializing models from pre-trained word representations, or on jointly learning both the word-and document level representations. various studies report that incorporating such word representations, embedding the words, significantly enhances the accuracy of text classification @xcite . another approach related to pre-trained word embedding is to aggregate unsupervised word embeddings into a document embedding, which is then fed to a classifier @xcite . despite being empowered with world knowledge through pre-trained information, dnns such as bert @xcite and xlnet @xcite can be very hard to interpret. one interpretation approach is to use attention-based models, relying on the weights they assign to the inputs. however, more careful studies reveal that attention weights in general do not provide a useful explanation @xcite . researchers are thus increasingly shifting focus to other kinds of machine learning, with the tm being a recent approach considered to provide human-level interpretability @xcite @xcite . it offers a very simple model consisting of multiple tsetlin automata (tas) that select which features take part in the classification. however, despite promising performance, there is still a performance gap to the dnn models that utilize pre-trained word embedding. yet, several tm studies demonstrate high degree of interpretability through simple rules, with a marginal loss in accuracy @xcite @xcite . a significant reason for the performance gap between tm-based and state-of-the-art dnn-based nlp models is that tm operates on boolean inputs, lacking a method for incorporating pre-trained word embeddings. without pre-trained information, tms must rely on labelled data available for supervised learning. on the other hand, incorporating high-dimensional booleanized word embedding vectors directly into the tm would significantly reduce interpretability. in this paper, we address this intertwined challenge. we propose a novel technique that boosts the tm bow approach, enhancing the bow with additional word features. the enhancement consists of using cosine similarity between glove word representations to obtain semantically related words. we thus distill information from the pre-trained word representations for utilization by the tm. to this end, we propose two methods of feature extension: (1) using the k nearest words in embedding space and (2) using words within a given similarity threshold, measured as cosine angle (θ). by adopting the two methods, we aim to reduce the current performance gap between interpretable tm and black-box dnn, by achieving either higher or similar accuracy. ## boosting tm bow with semantically related words here, we introduce our novel method for boosting the bow of tm with semantically related words. the method is based on comparing pretrained word representations using cosine similarity, leveraging distributed word representation. there are various distributional representations of words available. these are obtained from different corpora, using various techniques, such as word2vec, glove, and fasttext. we here use glove because of its general applicability. ## experiments and results in this section, we evaluate our tm-based solution with the input features enhanced by distributed word representation. here we use glove pretrained word vector that is trained using commoncrawl with the configuration of 42b tokens, 1.9m vocab, uncased, and 300d vectors. ## conclusion in this paper, we aimed to enhance the performance of tsetlin machines (tms) by introducing a novel way to exploit distributed feature representation for tms. given that a tm relies on bag-of-words (bow), it is not possible to introduce pre-trained word representation into a tm directly, without sacrificing the interpretability of the model. to address this intertwined challenge, we extended each word feature by using cosine similarity on the distributed word representation. we proposed two techniques for feature extension: (1) using the k nearest words in embedding space and (2) including words within a given cosine angle (θ). through this enhancement, the tm bow can be boosted with pre-trained world knowledge in a simple yet effective way. our experiment results showed that the enhanced tm not only achieve competitive accuracy compared to state of the art, but also outperform some of the sophisticated deep neural network (dnn) models. in addition, our bow boosting also improved the interpretability of the model by increasing the scope of each clause, semantically relating more samples. we thus believe that our proposed approach significantly enhance the tm in the accuracy/interpretability continuum, establishing a new standard in the field of explainable nlp.