An Analysis of Capsule Networks for Part of Speech Tagging in High- and Low-resource Scenarios (2020)

Abstract: Neural networks are a common tool in NLP, but it is not always clear which architecture to use for a given task. Different tasks, different languages, and different training conditions can all affect how a neural network will perform. Capsule Networks (CapsNets) are a relatively new architecture in NLP. Due to their novelty, CapsNets are being used more and more in NLP tasks. However, their usefulness is still mostly untested. In this paper, we compare three neural network architectures—LSTM, CNN, and CapsNet—on a part of speech tagging task. We compare these architectures in both high- and low-resource training conditions and find that no architecture consistently performs the best. Our analysis shows that our CapsNet performs nearly as well as a more complex LSTM under certain training conditions, but not others, and that our CapsNet almost always outperforms our CNN. We also find that our CapsNet implementation shows faster prediction times than the LSTM for Scottish Gaelic but not for Spanish, highlighting the effect that the choice of languages can have on the models.

## introduction neural networks have become a common tool in natural language processing (nlp) for many tasks, but are different architectures better suited for different tasks, languages, and/or resources? to try to answer this question, we examine the performance of two common neural network architectures, long short-term memory networks (lstm) @xcite and convolutional neural networks (cnn) @xcite , against the newer capsule networks (capsnets), another neural network architecture based on cnns @xcite . while lstms and cnns are common in nlp, capsule networks are relatively new to the field. due to their recency, it's not always clear if or when they are better than other widely used sequence models. this paper investigates the capsnet architecture in comparison with lstms and cnns. for our analysis, we apply these three architectures to a part of speech (pos) tagging task, on two languages, and using both low-and high-resource scenarios. much of the focus of nlp research is on resource-rich languages like english. however, the performance of different models can depend on the linguistic properties of the language under study @xcite and the amount of training data available. to compare the performance of these architectures under different training conditions, we look at spanish-another resource-rich language-and scottish gaelic-a low-resource language using different amounts of training data. this comparison is a step in the right direction, but it does have the limitations of comparing neural network architectures implemented in different frameworks and only comparing two languages. the main contribution of this paper is comparing the lstm, cnn, and capsnet architectures across different training conditions. our analysis finds that none of the architectures consistently performs best across training conditions. this illustrates how different languages and training conditions can inform which architecture is best suited for a given nlp task, and that there is no obviously correct answer. ## related work capsnets are a relatively new type of neural network. @xcite introduces the architecture, with modifications by @xcite (dynamic routing) and @xcite (em routing). a capsnet is essentially a modified ver-sion of a cnn that trades max pooling for a more data-retentive process called routing by agreement. instead of the prediction with the highest score getting chosen, the weighted sum of all predictions are considered for classification. essentially, a cap-snet uses convolution to create first round predictions for objects-primary capsules-and then utilizes routing by agreement to predict the presence of higher level objects-secondary capsules. many implementations of capsnets are designed for image recognition @xcite @xcite . however, the capsnet architecture is being applied more and more to nlp tasks, including chinese word segmentation @xcite , and multi-label text classification and question answering @xcite . this paper continues this path by investigating how capsnets compare to other neural network architectures for the task of part of speech tagging. ## data our comparison considers two languages: spanish 1 and scottish gaelic 2 . spanish is a resource-rich language, being the second most spoken language by number native speakers, fourth most spoken language by total number of speakers, and the third or fourth most widely used language on the internet 3 . scottish gaelic is a low-resource language, with 57,375 fluent speakers in scotland per the 2011 census 4 . the spanish data come from the ud spanish ancora treebank 5 . the scottish gaelic data come from the ud arcosg treebank 6 . both corpora use 17 part of speech tag classes. to study how different low-resource conditions affect training, we artificially create training partitions of different sizes. from the original training data (train100), we create partitions consisting of 50% (train50), 10% (train10), and 1% (train1) of the training sentences. the amount of data for each partition is shown in ## approach in this section, we describe the implementation details of our capsnet, cnn, and lstm methods. our capsnet and cnn implementations build on top of yeung et al.'s implementation foot_0 , which was kept as close as possible to the architectures described by @xcite . importantly, we tried to keep all three models as close to each other as possible in order to make our comparison as faithful as possible. however, certain differences persist for this project-for example, the capsnet and cnn are implemented in python using tensorflow foot_1 and keras foot_2 , whereas the lstm is implemented in scala using dynet. 10 the hyperparameters for our capsnet and cnn implementations were chosen to be as close as possible to the original implementation. the hyperparameters of the lstm were chosen to be a reasonable approximation to the capsnet and cnn models. it is important to note that our comparison does not attempt to compare the best of the best of each architecture. ## results in addition to our four training data conditions per language, we evaluate the use of learning the word embeddings during training for all models ("learn" vs. "no learn"), yielding 24 training conditions per language. we trained all models five times with a different random seed and averaged the results. each condition trained for 10 epochs, with early stopping after 2 epochs if the loss did not improve. the results are given in ## conclusion in this paper, we compare the performance of three neural network architectures-lstm, cnn, and capsnet-on part of speech tagging and find that lstms are not always better under the common hyperparameters investigated. we examine how the best performing model changes under different high-and low-resource training conditions using spanish and scottish gaelic. we show that the relatively new capsnet architecture performs nearly as well as the more complex lstm under certain conditions and outperforms the cnn under most conditions we examined. these results suggest that there is no one obviously clear choice for a model architecture, and that the properties of a language and the amount of training data can affect which architecture performs best. future work should address the limitations of this paper. specifically, future effort should consider more training conditions, including other languages; the consistency of these results within groups of similar languages; and making the implementation of these architectures closer, to guarantee the performance differences are due to the architecture and not an artifact of how they were implemented.