A Deep Learning System for Automatic Extraction of Typological Linguistic Information from Descriptive Grammars (2021)

Abstract: Linguistic typology is an area of linguistics concerned with analysis of and comparison between natural languages of the world based on their certain linguistic features. For that purpose, historically, the area has relied on manual extraction of linguistic feature values from textural descriptions of languages. This makes it a laborious and time expensive task and is also bound by human brain capacity. In this study, we present a deep learning system for the task of automatic extraction of linguistic features from textual descriptions of natural languages. First, textual descriptions are manually annotated with special structures called semantic frames. Those annotations are learned by a recurrent neural network, which is then used to annotate un-annotated text. Finally, the annotations are converted to linguistic feature values using a separate rule based module. Word embeddings, learned from general purpose text, are used as a major source of knowledge by the recurrent neural network. We compare the proposed deep learning system to a previously reported machine learning based system for the same task, and the deep learning system wins in terms of F1 scores with a fair margin. Such a system is expected to be a useful contribution for the automatic curation of typological databases, which otherwise are manually developed.

## introduction and background linguistic typology is an area of linguistics concerned with analysis of and comparison between natural languages of the world in terms of their structural and functional attributes. among others, major objectives of the area are exploring the range of possibilities for expressing different linguistic categories, trying to understand the extent to which the presence of different features depend on one another in larger patterns, and to study how the global distribution of language traits has come about through an interplay of tendencies inherent to languages and historical contingencies @xcite . for achieving such aims, historically, the area has relied heavily on scholars having to read textual documents (commonly known as descriptive grammars) describing languages, manually extracting values of a pre-defined set of features, and then comparing languages based on the extracted feature values. to make the whole exercise systematic and structured, traditionally, the features are expressed in the form of questions e.g. 'what is the order of adjectives and noun in language x?', and scholar's job is then to find answers to such questions by reading descriptive documents about language x. the answers are often formulated to be simple strings e.g. 'na', 'an', or 'both' representing the fact that nouns precede adjectives, nouns follow adjectives, or 'both nouns may follow or precede adjectives' respectively in the case of above give question. for easy storage and retrieval, the questions and their answers are recorded in special kind of databases known as typological databases. there exist many such databases and a fuller list is available at languagegoldmine.com/. these databases are later used to compare and analyze languages to achieve the above mentioned objectives of the area. in addition, information in such databases has proven to be useful for a number of natural language processing (nlp) related tasks. a survey on the usefulness of typological information in nlp can be found in @xcite . as can be imagined, the manual development of typological databases is an expensive enterprise both in terms of time and efforts, and also their qual-ity and coverage is bound by human brain capacities. extensive digitization efforts (e.g. @xcite ) and the advancement of computational methodologies, including nlp, offer many possibilities for easing the task of developing linguistic typological databases. a maximally automated approach would allow for the generation of such databases at a hitherto unparalleled scale, increasing both the number of features and languages that can be analyzed and compared taking the area to new heights. however, this requires developing methodologies and systems for automatic extraction of typological linguistic information from descriptive documents. this exactly is a major objective of the study reported in this paper. previously, a few approaches and systems have been reported for the automatic extraction of typological information, but a practical system still remains to be be developed. pattern matching and machine learning based classification are the two main computational paradigms that have been exploited so far (see section 3). pattern matching has its own limitations as it is impossible to design patterns which can cover every possible scenario. similarly, machine learning classification algorithms are heavily dependent on feature engineering, which have its own limitations and features are often expensive to compute. to address some of these issues, we report a deep learning based system in this study (section 4). first, textual descriptions are manually annotated (section 4.1) with special structures called semantic frames, which are based on the theory of frame-semantics (section 2). those annotations are learned by a recurrent neural network using word embeddings learned from general purpose text as the major source of knowledge (section 4.2). the trained model is used to annotate the un-annotated data, and the annotations are then converted to linguistic feature values using a separate rule based module (section 8). our frame annotation system beats previously reported systems on a test set with a fair margin in terms of f1-scores. ## frame-semantics, framenet, and frame-semantic parsing frame semantics is a theory of meanings in natural languages @xcite . it stipulates that meanings of words can be best understood with reference to the situations they invoke in the minds of the speakers. the concrete manifestation of frame semantics is a computational lexical resource called a framenet. the first such resource was the english berkeley framenet (bnf) @xcite which has inspired work on framenets for many other languages. the "lexical entry" in a framenet is called a (semantic) frame, which is a script-like description of a prototypical event, object, relation, or scenario. a frame consists of triggers -the lexical units (words) which evoke a specific situationand additional components called frame-elements that fills in various semantic slots of the frame. below is an example sentence manually labeled with the commerce_selling semantic frame and its frame-elements i.e. seller, buyer, and price. the frame is triggered by the word 'sold'. jimmy seller [sold] commerce_selling [a car] goods [to lester] buyer in [2000 usd] price . the process of automatically performing the above type of annotation/analysis is called semantic parsing. the first such frame semantic parser was proposed by @xcite , which has been followed by a number of other systems/approaches (e.g. @xcite ). ## related work previously, a few experimental techniques and associated systems have been reported for automatic extraction of typological information. in @xcite , the authors have reported on simple pattern matching and syntactic parsing based systems. the systems have modest accuracy and recall and are very restricted with respect to the number of features they can target. to overcome some of those limitations, in @xcite the authors exploited frame-semantics and machine learning approaches to build a system that can extract information about a few experimental features. the systems is based on the work reported in @xcite where the authors proposed to use frame-semantics to represent the typological linguistic information. they also developed a domain specific framenet (lingfn) containing semantic frames for the linguistic domain representing various linguistic terms and phenomenas (e.g. verb, noun, agreement, inflection, etc.). the developed lingfn was used by @xcite to annotate textual descriptions of languages with linguistic domain frames (more details in section 4.1) and develop an automatic typlogical information extraction system using machine learning. in @xcite , the authors reported a two step strategy to detect the parts of the text that possibly contains value of a given feature, and then extracting the feature value from it. for the first step their approach relies on keyword spotting and pattern matching, while on machine learning classification for the second step. in @xcite , the author reported a simple keyword based approach for extracting values of one typological feature about tones for many languages from thousands of documents with an overall accuracy of 89.1%. ## proposed system figure 1 shows the complete architecture of the system that we propose. as shown, it has a clear division between three components: data annotation, deep learning, and typological information formulation. in the following subsection, the first two components will be explained, while the explanation of the third component is deliberately deferred until section 8 for a better flow. ## experiments the data was divided into two major sets labeled as 'full' and 'filtered'. the former is the data set where the annotation of all frames and frameelements were preserved, while in the later the annotation of two problematic frame elements (i.e. 'data' and 'data_translation' 2 ) were removed from the training data. the reason for removing 2 in lingfn, many frames have data and data_translation frame-elements, which are used to represent examples of various morphology or grammatical linguistic categories of the language described in the document. as an example consider the following annotation in which both data and data_translation frame-elements have been used to annotate an example numeral (i.e. ghrī) and its translation (i.e. 'one'): 'the numeral [ghrī]_data , [one]_data_translation , is used as an indefinite article. these two frame-elements is that they contain non-english words for which the embeddings are missing in the learned embedding space consequently impacting the model's performance adversely. to show their impact, we conducted experiments with and without those frame-element annotations. with those two data sets, the experiments were conducted with the following three settings: in the first setting, word-embeddings were used to compute the token level representations (i.e. eq 1), and consequently all other computations involving token representations. in the second setting, wordembeddings were replaced with character embedding, while in the third setting they were replaced with mimic embeddings. the motivation behind these three settings follows. since the word embedding space was learned from a general-purpose text, it understandably does not contain many of the the domain specific words in the data used in this study (i.e. the descriptive grammars). in addition, there are the non-english words, i.e the transliterated forms of the words of a specific language given as examples in the descriptive documents. in the annotated data, such words often appear as a part of the 'data' frameelement. for all those cases, a default word embedding was used by the system while learning and predicting the frame annotations. this means that the knowledge that words can bring to the system while learning was not available to the system resulting in a decreased system performance (as obvious from the results discussed in section 6). as a solution to this out-of-vocabulary words issue, we experimented with the other two settings. in the second setting, word embeddings were replaced with character embeddings while computing eq 1, resulting in different token and span level representations (eq 2 and 3 ). this was based on the intuition that even if a complete word has not been seen in the training set, the characters it contains have, hence, utilizing the character-level knowledge. this is particularly pertinent in the case of 'data' frame element. as is apparent from the results in given in section 6, this technique improved the result considerably. a primary computational concern with this solution from the outset was the need to train character embeddings, which is computationally a highly expensive task. for that we relied on automatic conversion of a set of pre-trained word embeddings into character embeddings foot_2 . in this technique, for each word in a set of word embeddings, each character is given a vector inferred from the parent word. then, for each word in the embeddings, when the character is seen again this vector is adjusted to reflect its average in the entire set. this technique is a useful workaround to get a meaningful set of character embeddings. in the third setting, word embeddings of the unknown words were inferred from the existing wordembedding space and used while computing input (eq 1). the embedding inferring technique has been proposed by @xcite in which a character-based bidirectional lstm was used to infer vectors for a list of unknown words and to add them to the existing word-embeddings. they call their method the 'mimick', and it predicts an embedding for a word that should fit into the same space based on the sequence of character embeddings for the characters in the given word. originally, the authors used their technique in the context of ocr error correction and spelling variations issue, but in our case we have used it for the out-of-vocabulary issue. it appeared to be equally useful in our case as is obvious from the results given in the next section. ## results in summary, we achieved the best results with character-level embedding when applied to the full data, and with the mimic technique when data and data_translation were excluded. ## comparison to a previous system ## typological information formulation module in the proposed system, once a sentence has been automatically annotated with semantic frames and their frame-elements (the output from the second part of the system architecture), the annotations can be converted to a typolological feature value as an answer to a typological question. currently, we rely on a rule-based module for the conversion from annotations to feature values as explained in @xcite . this involves writing small modules as shown in algorithm 1 given in appendix a for converting annotations to feature values. as can be seen the algorithm simply loops through the set of frames (line 2) and frame elements, and depending on their contents (line 3, 7, 8, 10, and 12), it assigns appropriate features value for the adjective-noun-order feature (line 9, 11, and 13) discussed in the introduction section. similar modules can be used for other typological features. appendix b shows a set of order related feature and their values automatically extracted from descriptive grammar of 'ulwa' language @xcite using the developed semantic parser, and the rule-based feature formulation module given in appendix a. note all features were extracted using only sequence foot_3 semantic frame from lingfn.