Exploring the Language of Data (2020)

Abstract: We set out to uncover the unique grammatical properties of an important yet so far under-researched type of natural language text: that of short labels typically found within structured datasets. We show that such labels obey a specific type of abbreviated grammar that we call the Language of Data, with properties significantly different from the kinds of text typically addressed in computational linguistics and NLP, such as ‘standard’ written language or social media messages. We analyse orthography, parts of speech, and syntax over a large, bilingual, hand-annotated corpus of data labels collected from a variety of domains. We perform experiments on tokenisation, part-of-speech tagging, and named entity recognition over real-world structured data, demonstrating that models adapted to the Language of Data outperform those trained on standard text. These observations point in a new direction to be explored as future research, in order to develop new NLP tools and models dedicated to the Language of Data.

## introduction structured data, such as database tables or xml trees, frequently contain short natural language labels that either describe the data structure itself (attribute names) or provide content (attribute values). in highprecision data integration and analysis applications, where mistakes in interpretation pose a significant risk, such as in health data analytics or emergency response, an unambiguous parsing of such pieces of text is crucial. conventional nlp tools, such as supervised sequence labellers or embeddings trained on full sentences, do not, however, perform well on structured data. firstly, data labels may be very short and thus devoid of context necessary for adequate nlp performance. secondly, they are not selfcontained: their interpretation relies on the data structure surrounding them, such as other values of the same attribute or record. and thirdly, their grammar shows significant difference with respect to the language on which nlp tools are typically trained. this paper explores the grammar of structured data labels-that we succinctly designate as the language of data or lod for short-according to three main hypotheses: (1) uniqueness: it is markedly different from other forms of language typically studied in computational linguistics, such as short social media messages, encyclopaedic articles, literature, or newswire; (2) uniformity: it displays coherent grammatical characteristics across domains, data sources, and even languages; and (3) usefulness: its distinctive features can be exploited to improve the performance of computational language analysis. following an empirical approach, our paper verifies the first two hypotheses through an extensive corpus analysis, and the third one through actually building and evaluating an initial set of nlp tools tailored to the grammar of short text labels. accordingly, the contributions of this paper are (1) a freely available, expert-annotated, 120k-token, bilingual (english-italian) dataset on which most of our analysis is based; ## state of the art the analysis of the grammar of short pieces of text dates back to @xcite and @xcite who first described the block language of english newspaper headlines and of the disjunctive language of advertising in general. these early works have inspired @xcite to analyse the 'language of the street' (i.e. signs and inscriptions), as well as ourselves to regard the short sentences of structured data as belonging to a specific form of these grammars. nevertheless, as we show later, the language of data also presents specificities with respect to 'headlinese' and advertising slogans. more recent research regarding short text concentrates on social media messages (e.g. tweets) often with a focus on opinion mining, web search queries with a focus on information retrieval, and on schema or classification labels with a focus on disambiguation and/or downstream semantic alignment tasks. among these, works on understanding short social media messages (see @xcite for a survey) are mostly irrelevant to us due to fundamental linguistic differences in terms of syntactic structure, noisiness, discursivity, use of an extremely informal spoken register, etc., as demonstrated in our paper. works on the matching or disambiguation of data schemas or classification labels are, on the other hand, highly relevant to us in terms of the underlying objectives. @xcite analyse the grammar of short labels used in classification hierarchies, the results of which can be exploited for the optimisation of nlp tasks, such as in cross-lingual alignment @xcite . classification labels, however, are usually expert-curated and thus follow strict and regular grammars exclusively constituted of noun phrases with coordinating conjunctions. we therefore consider the language of such labels as a narrow, specific subset of the lod. works on the disambiguation of data schemas and underlying data values @xcite typically approach the problem from a point of view of lexical semantics-i.e. the label is considered as a bag of words or word meanings-and concentrate on the exploitation of the context provided by the tabular or tree-based data structure for the purposes of disambiguation. they do not consider the grammar of the labels, which is the focus of our research. finally, in preliminary work @xcite , the authors explore the problem of word sense disambiguation over text in structured data, and mention some of the key characteristics of the lod in order to motivate the development of adapted nlp pipelines. they do not, however, provide any corpus-based empirical proof of the linguistic characteristics assumed, which is the main objective of our work. results on the analysis of search queries are also relevant given that, as we demonstrate, they are similar in certain aspects to structured data contents, such as in average length or their limited use of parts of speech. @xcite considers search queries as noun phrases and defines their semantic constituents as intent heads and modifiers. still, they use the results of regular pos taggers and chunkers as input features for learning-based methods, all the while being aware of their sub-optimal performance on short text. @xcite instead consider that such queries 'do not follow grammar rules' and attempt to identify the former constituents without relying on any syntactic feature. we situate our work in addition, rather than in opposition, to those described above. we believe that the grammar-agnostic approaches that rely on keywords, lexical semantics, and structured context can be successfully complemented by tools and resources built in awareness of the underlying specific grammar. ## the language of data: characteristics and methodology through an empirical, bottom-up corpus analysis of data labels, the results of which we present in sections 4-6, we have synthesised the main characteristics of the lod as follows: (1) abbreviated grammar used for brevity, which manifests itself in the shortness of labels, the frequent use of abbreviations, nonstandard word separation or lack thereof ('firstname'), as well as a block language syntax that is based on the omission of words ('age [of] pension'); (2) disjunctive grammar, which consists of independent non-finite and minor sentences ('road closed'); (3) named entities appear in a large proportion of labels; and (4) formal writing, which manifests itself in orthography inspired from formal languages ('zipcode' or 'zip code'), change of word order ('smith john'), as well as the presence of abstract expressions ('@xmath0. the term block language was first used by @xcite who examined the sparing syntax of newspaper headlines that frequently omit function and other words. later, @xcite used the term abbreviated grammar to define more generally the grammar of public signs and the linguistic register of advertising. in the language of data, the frequent use of abbreviated forms on both word and phrase level may be motivated partly by space saving concerns (e.g. in data entry forms), partly by an inspired adoption of the register of informative signs, e.g. in street language. the term disjunctive grammar, also coined by leech, refers to a syntax consisting of non-finite or minor (verb-less) sentences as independent clauses. 2 we explain the dominance of such constructs in the lod by the structured nature of data itself: a data value is always interpreted in the context of a data record (i.e. the 'row') and an attribute (the 'column'). in terms of semantics, the data value, the record, and the attribute correspond to the object, subject, and predicate of a full sentence, respectively. thus, a data value 'dark brown' within a record describing 'mr jones' and under the attribute 'hair' can be transformed into the full sentence 'the hair of mr jones is dark brown.' none of the three labels needs to be fully grammatical, as the data structure implies the syntactic 'glue' that joins them. as to the presence of formal writing in the lod, we attribute it to the computational context. the use of word separators such as underscores or camelcasing is likely inspired by computer programming practices. the change of word order often happens due to sorting or classification requirements. the frequency of interpretative clues (such as providing the unit of measure in attribute names) is explained by disambiguation needs for data entry or subsequent interpretation and analytics. finally, the fact that the primary purpose of structured data is to describe or refer to real-world entities (such as mr jones) explains the high proportion of named entities within data. our analysis of the lod was based on a new, human-annotated 120k-token corpus of natural language labels extracted from structured data, that we used to compute statistics and insights on the characteristics of the lod, as well as to train a first batch of dedicated nlp tools. annotation and subsequent validation were carried out by two language experts with adequate knowledge of both english and italian. ## surface statistics and orthography orthographic and other surface aspects of texts play a crucial role in computational tasks: as they are often dealt with as part of preprocessing, their incorrect handling affects the entire downstream pipeline. label length. figure 1 shows token count distributions for all header (dotted), value (continuous), and unstructured (dashed) corpora, while phrase and word separation. in the lod, non-standard phrase and word separation practices are frequent: the use of punctuation to/separate/words, camelcasing, titlecasing, or the overall omissionoftheseparator. the characters '/', '|', and '-' are also frequently used to separate distinct phrases and sentences, a practice very rare outside of the lod. abbreviations and acronyms are pervasively used in the lod. ## parts of speech we analysed and compared the distribution of pos and named entity categories in each annotated corpus shown in lod vs 'standard' text and tweets. we take as reference representing 'standard' text the four genrespecific subsets of the brown corpus: newswire, adventure, government documents, and belles lettres. as observed in figure 2 (chart to the right), the four genres are characterised by very similar patterns on the diagram, showing only minor differences in pos distribution. the wiki corpora are also similar, with fewer pronouns while more numbers and named entities, as it is expected from encyclopaedic text. the patterns of the lod (right and middle), in contrast, are fundamentally different, marked by an abundance of nouns and the absence of pronouns (0.3-0.6% as opposed to 3-12% of standard text) and adverbs (0.3-1.5% as opposed to 3-6%). the lod also has much fewer verbs (1-4% as opposed to 12-17%), articles and determiners (0.2-3% as opposed to 7-10%), and prepositions (in english: 4-7% as opposed to 10-13%, in italian: 8-16% as opposed to 17%). 6 as these differences are much more pronounced than cross-genre variations in the brown corpus, the language of data cannot be considered as merely another genre of written language. tweets, while also short labels, are even more dissimilar, having the largest proportion of verbs, adverbs, and pronouns among our corpora. these findings point in the direction that parsing short data labels is likely to be a very different problem with respect to both short social media messages and 'standard' text. lod headers vs data values. both the five header and the five data value corpora show a high degree of internal uniformity. headers are characterised by a major dominance of common nouns (48-66% as opposed to 15-22% in standard text) and a relatively high proportion of adjectives (7-11% as opposed to 5-8%). common nouns also dominate data values, although to a lesser extent (28-35%). values, however, are characterised by a very large number of named entities (20-42%), reflecting the fact that the typical function of structured data is to describe objects of the real world. names do appear in headers as well, although much more rarely (2-10%). ## syntax our analysis of the syntax of the lod is motivated by both descriptive and practical goals: even if full syntactic parsing is becoming less common in state-of-the-art nlp, word context windows of varying length play a major role in the training of most machine learning models (from word embeddings to semantic taggers), the contents of which depend on the syntactic structure of the text. instead of an onerous full syntactic annotation, we based our analysis on a three-step classification of labels. first, we applied a coarse-grained classification into: (1) fully grammatical sentences that follow standard syntax; (2) sub-sentence phrases that still follow standard syntax; and (3) non-standard syntax. what to consider as 'standard syntax' was left to the judgment of the annotator as to whether the phrase in question would feel grammatically out of place within standard written media, such as a book or a magazine. 7 as a second step, a manual analysis of the non-standard labels has identified a set of recurrent syntactic constructs: concatenation, compression, non-standard word order, interpretational clues, and abstract expressions. we classified all such labels with respect to these inductive categories. the frequency of these constructs in each corpus is reported in the right-hand side of compression, that we classify among the devices of block language, consists of the omission of certain grammatical elements, such as function words or verbs. this style is well known from news headlines, e.g. 'officer [who was] fired for refusing to kill [a] bear wins [a] legal battle.' in the lod, the same principle is pervasively used, more in italian where function words are more frequent (5-29% of all labels, such as: 'acquisto [al] bordo'), 'imprese [che] prevedono assunzioni') but to a lesser extent also in english (1-5%: 'operate [an] uninsured vehicle', '[the] year [the mine was] opened'). in the last case, the context of the dataset, that describes mines, provides the necessary interpretation. another form of compression, very frequent in english, consists of converting one or more phrases into noun adjuncts, such as 'death country' instead of 'country of death', in order to spare the preposition. concatenation of nouns and noun phrases without any grammatical marker (e.g. coordinating conjunction or comma) is the second major device of block language. for example, the label 'cooroy mary river road timber bridge rehabilitation' is a drastically shortened and rearranged form of the sentence 'rehabilitation of the timber bridge on mary river road in cooroy.' concatenation is also used for enumeration ('ristorante bar pizzeria'). it is an equally frequent form of non-standard syntax in both languages (11-30% of headers, 2-6% of values). non-standard word order is characteristic of formal or administrative style, and is often motivated by sorting needs. it manifests itself in inverted names ('fantozzi, ugo', 'agriculture, department of') and postpositive adjectives ('investigation non-specific', 'population growth, annual'). typically 1-2% of labels contain such constructs. interpretational clues are frequent in headers (1-10%) but appear also in values (1-2%). they consist of a short instruction at the end of the label, often delimited by punctuation, that serves as a clue for filling in or interpreting the data value. it most often provides a unit of measure ('duration (s)', 'expenditure $') or scope ('total (2016)'), but it may provide also provide other kinds of clarifications ('dog -non std'). abstract expressions express their content in a formulaic manner, such as '@xmath1.' the correct interpretation of such labels usually requires domain knowledge. 0-7% of all labels were found to contain such expressions. in synthesis, our findings confirm the lod almost exclusively to follow disjunctive grammar (97-100% of labels). the overwhelming presence of minor or non-finite sentences is further attested by a low frequency of verbs (1-4% in headers, 2-5% in values) and the high proportion of non-finite verbs among these (65-93% of all verbs inside headers, 53-72% inside values). most disjunctive sentences consist of independent noun phrases, as proven by the large proportion of nouns (see section 5). block language (consisting of compression and concatenation) appears in 17-40% of headers and 3-30% of values. the syntax of formal writing is found in 3-13% of headers and 2-3% of values. finally, as shown by the name column in once again, among our control corpora, only the query corpus shows similar behaviour, in particular to the header corpora. noun phrases dominate search queries as well. the kind of non-standard syntax employed within queries, however, is mostly limited to concatenation, a phenomenon that may contribute to the common perception of queries to be 'non-grammatical' @xcite . tweets, in contrast, show a very different profile: standard grammatical sentences are common (50%) as well as syntactic structures specific to spoken language (28%). disjunctive grammar is much more rare (17%), and block language devices are mostly limited to compression (6%), typically eliminating sentence-initial pronouns ('just came home'). in summary, while tweets and the lod do share some common (and computationally challenging) syntactic structures, they belong to two completely different language registers. ## experiments on nlp tasks our experiments have evaluated the performance of three classic low-level nlp tasks over the lod: tokenisation, pos tagging, and ner. these three tasks are often used in knowledge and information extraction, fundamental for semantic data integration and analytics over structured data @xcite . they are typically solved through supervised machine learning trained on annotated 'standard' text. our goals were (1) to test the performance of state-of-the-art tools, trained on standard text, over lod header and value datasets; (2) to evaluate the usefulness hypothesis: whether the same tools adapted to the lod (e.g. with models trained on lod datasets) would perform better, and to what extent; and our approach was to re-train state-of-the-art tools using our lod corpora. for cross-comparability, we mapped our annotations (the pos and ner tags) to those used by these tools. our test corpora were the australian headers and values datasets, as well as the annotated trec query subcorpus. our training corpus, in turn, was the fusion of all english-language lod datasets, obviously excluding the dataset used for testing. tokenisation. our analysis in section 4 showed the high frequency of non-standard word separation in data headers, and to a lesser extent in values. we compared three solutions: (1) a classic supervised learning-based approach using the opennlp tokeniser tool; (2) a simple rule-based (regular expression) tokeniser that we developed specifically for the lod; and (3) the opennlp tokeniser re-trained on our lod corpora. the results in pos tagging. over standard english text, pos tagging is a largely solved problem with state-of-the-art results around 95%. over the lod, pos tagging is expected to be harder: labels are very short, their orthography is different (e.g. inconsistent capitalisation makes the recognition of proper nouns harder), and the distribution of part of speech categories is also different. we compared a classic opennlp maximum entropy tagger, the state-of-the-art stanford pos tagger, and the opennlp tagger retrained on the lod. ## conclusions and future work despite the diversity of structured datasets used in our study, we observed uniform tendencies in the grammatical phenomena observed. at the same time, significant differences were revealed between the grammar of attribute names and values: among others, the latter tend to be longer, contain more named entities, and are closer in grammar to standard text. in terms of the uniqueness of the lod, major differences were revealed in comparison to standard language on all grammatical levels examined. only the query corpus showed similarity to the lod (especially to attribute names), hinting at the transferability of nlp models across the two kinds of corpora: a hypothesis eventually disproved by nlp experiments. we examined the usefulness of our results on a first set of nlp tasks. the results are encouraging as they show that training on lod corpora in itself leads to major improvements in precision and recall. at the same time, we consider our models as merely the first step in the direction of developing nlp tools dedicated to the lod. a promising research direction involves the creation and use of unsupervised word embeddings and language models from large lod corpora, as a basis for subsequent fine-tuning to specific tasks. another future work concerns the use of constraints imposed by the data structure itself on the interpretation of the labels. the combination of data-structure-based and corpus-based methods for label disambiguation, informed by the grammar of the lod, is a direction yet to be investigated.