Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020)

Abstract: The success of pretrained contextual encoders, such as ELMo and BERT, has brought a great deal of interest in what these models learn: do they, without explicit supervision, learn to encode meaningful notions of linguistic structure? If so, how is this structure encoded? To investigate this, we introduce latent subclass learning (LSL): a modification to classifier-based probing that induces a latent categorization (or ontology) of the probe’s inputs. Without access to fine-grained gold labels, LSL extracts emergent structure from input representations in an interpretable and quantifiable form. In experiments, we find strong evidence of familiar categories, such as a notion of personhood in ELMo, as well as novel ontological distinctions, such as a preference for fine-grained semantic roles on core arguments. Our results provide unique new evidence of emergent structure in pretrained encoders, including departures from existing annotations which are inaccessible to earlier methods.

## introduction the success of self-supervised pretrained models in nlp @xcite @xcite ) on many tasks @xcite has stimulated interest in how these models work, and what they learn about language. recent work on model analysis @xcite indicates that they may learn a lot about linguistic structure, including part of speech @xcite , syntax @xcite , word sense @xcite , and more @xcite . many of these results are based on predictive methods, such as probing, which measure how well a linguistic variable can be predicted from intermediate representations. however, the ability of * work performed while at google. figure 1 : lsl overview. a probing classifier over contextual embeddings produces multi-class latent logits, which are marginalized into a single logit trained on binary classification. in this example, "pierre vinken" is identified as a named entity and assigned to latent class 2, which aligns well with the person label. we treat the classes as clusters representing a latent ontology that describes the underlying representation space. figure 2 visualizes latent logits in more detail. to tackle these questions, we propose a systematic way to extract latent ontologies, or discrete categorizations of a representation space, which we call latent subclass learning (lsl); see figure 1 for an overview. in lsl, we use a binary classification task (such as detecting entity mentions or syntactic dependency arcs) as weak supervision to induce a set of latent clusters relevant to that task (i.e., entity or dependency types). as with predictive methods, the choice of task allows us to explore varied phenomena, and induced clusters can be quantified and compared to gold annotations. but also, as with descriptive methods, our clusters can be inspected and qualified directly, and observations have high specificity: agreement with external (e.g., gold) categories provides strong evidence that those categories are salient in the representation space. we describe the lsl classifier in section 3, and apply it to the edge probing paradigm @xcite in section 4. in section 5 we evaluate lsl on multiple encoders, including elmo and bert. we find that lsl induces stable and consistent ontologies, which include both striking rediscoveries of gold categories-for example, elmo discovers personhood of named entities and bert has a notion of dates-and novel ontological distinctionssuch as fine-grained core argument semantic roleswhich are not easily observed by fully supervised probes. overall, we find unique new evidence of emergent latent structure in our encoders, while also revealing new properties of their representations which are inaccessible to earlier methods. probing involves training lightweight classifiers over features produced by a pretrained model, and assessing the model's knowledge by the probe's performance. probing has been used for low-level properties such as word order and sentence length @xcite , as well as phenomena at the level of syntax @xcite , semantics @xcite @xcite , and discourse structure @xcite . error analysis on probes has been used to argue that bert may sim-ulate sequential decision making across layers @xcite , or that it encodes its own, soft notion of syntactic distance @xcite . predictive methods such as probing are flexible: any task with data can be assessed. however, they only track predictability of pre-defined categories, limiting their descriptive power. in addition, a powerful enough probe, given enough data, may be insensitive to differences between encoders, making it difficult to interpret results based on accuracy @xcite . so, many probing experiments appeal to the ease of extraction of a linguistic variable @xcite . existing work has measured this by controlling for probing model capacity, either using relative claims between layers and encoders @xcite @xcite or using explicit measures to estimate and trade off capacity with accuracy @xcite ). an alternative is to control amount of supervision, by restricting training set size (zhang and bowman, 2018), comparing learning curves @xcite , or using description length with online coding @xcite . we extend this further by removing the distinction between gold categories in the training data and reducing the supervision to binary classification, as explained in section 3. this extreme measure makes our test high specificity, in the sense that positive results-i.e., when comprehensible categories are recovered by our probe-are much stronger, since a category must be essentially invented without direct supervision. ## approach we propose a way to extract latent linguistic ontologies from pretrained encoders and systematically compare them to existing gold ontologies. we use a classifier based on latent subclass learning (section 3.1), which is applicable in any binary classification setting. 1 we propose several quantitative metrics to evaluate the induced ontologies (section 3.2), providing a starting point for qualitative analysis (section 5) and future research. ## experimental setup we adopt a similar setup to @xcite and @xcite , training probing models over several contextualizing encoders on a variety of linguistic tasks. ## results and analysis named entities as shown in date percent org person . . . event law lang. bert .70 .60 .54 .48 • .03 .02 .01 elmo .38 .28 .35 .81 • .02 .01 .01 nonterminals patterns in npmi (figure 3a ) suggest basic syntactic notions: complete clauses (s, top, sinv) form a group, as do phrase types which take subjects (sbar, vp, pp), and wh-phrases (whadvp, whpp, whnp). dependencies patterns in npmi (figure 3b ) indicate several salient groups: verb arguments (nsubj, obj, obl, xcomp), left-heads (det, nmod:poss, compound, amod, case), right-heads (acl, acl:relcl, nmod foot_5 ), and punct. semantic roles patterns in npmi (figure 3c) roughly match intuition: primary core arguments (arg0, arg1) are distinguished, as well as modals (argm-mod) and negation (argm-neg), while trailing arguments (arg2-5) and modifiers (argm-tmp, loc, etc.) form a large group. on one hand, this reflects surface patterns: primary core arguments are usually close to the verb, with arg0 on the left and arg1 on the right; trailing arguments and modifiers tend to be prepositional phrases or subordinate clauses; and modals gold label p / r / f1 argm-mod .62 / .41 / .49 arg0 .52 / .17 / .26 arg1 .50 / .09 / .15 argm-neg .36 / .60 / .45 arg2 .28 / .13 / .18 and negation are identified by lexical and positional cues. on the other hand, this also reflects error patterns in state-of-the-art systems, where label errors can sometimes be traced to ontological choices in propbank, which distinguish between arguments and adjuncts that have very similar meaning @xcite . while the number of induced classes roughly matches gold for most tasks, induced ontologies for semantic roles are considerably more diverse, with a diversity measure close to 20 for elmo and bert ( this follows intuition for propbank core argument labels, which have predicate-specific meanings. other approaches based on frame semantics @xcite , proto-roles @xcite , or levin classes @xcite have more explicit fine-grained roles. concurrent work @xcite shows that the choice of semantic role formalism meaningfully affects the behavior of supervised probes; further comparisons using lsl probing may help shed light on the origins of such differences. ## discussion our exploration of latent ontologies has yielded some surprising results: elmo knows people, bert knows dates, and both sense scalar and measurable values, while distinguishing between small and large numbers. both models preferentially split core semantic roles into many fine-grained figure 3 : pairwise gold label npmis on selected categories for ontologies induced from bert-large on selected tasks. blue is positive npmi, representing that gold labels are preferentially grouped together (i.e., conflated by the model) relative to chance. red is negative npmi, representing that gold labels are wellseparated. perfectly matching ontologies would be 1 (blue) along the diagonal and -1 (red) in all off-diagonal cells. counts are summed over all 5 runs to better reflect the underlying representations, though variance was low and our observed trends hold across all runs. this issue is particularly important when looking for deep, inherent understanding of linguistic structure, which by nature must generalize. for supervised systems, generalization is often measured by out-of-distribution objectives like out-of-domain performance @xcite , transferability @xcite , targeted forms of compositionality @xcite , or robustness to adversarial inputs @xcite . recent work also advocates for counterfactual learning and evaluation @xcite to mitigate confounds, or contrastive evaluation sets @xcite to rigorously test local decision boundaries. overall, these techniques target discrepancies between salient features in a model and causal relationships in a task. in this work, we extract such features directly and investigate them by comparing induced and gold ontologies. this identifies some very strong cases of transferability from the binary detection task to detection tasks over gold subcategories, such as elmo's people and bert's dates ( our results point to a paradigm of probing with latent variables, for which lsl is one potential technique. we have only scratched the surface of what may emerge with such methods: while our probing test is high specificity, it is low power; extant latent structure may still be missed. lsl probing may produce different ontologies due to many factors, such as tokenization @xcite , encoder architecture @xcite , probe architecture @xcite , data distribution @xcite , pretraining task @xcite , or pretraining checkpoint. any such factors may be at work in the differences we observe between elmo and bert: for example, bert's tokenization method may not as readily induce personhood features due to splitting of rare words (like names) in byte-pair encoding. furthermore, concurrent work @xcite has already found qualitative evidence of syntactic dependency types emergent in the special case of multilingual structural probes @xcite . with lsl, we provide a method that can be adapted to a variety of probing settings to both quantify and qualify this kind of structure. ## conclusion we introduced a new model analysis method based on latent subclass learning: by factoring a binary classifier through a forced choice of latent subclasses, latent ontologies can be coaxed out of input features. using this approach, we showed that encoders such as bert and elmo can be found to hold stable, consistent latent ontologies on a variety of linguistic tasks. in these ontologies, we found clear connections to existing categories, such as personhood of named entities. we also found evidence of ontological distinctions beyond traditional gold categories, such as distinguishing large and small numbers, or preferring fine-grained semantic roles for core arguments. in latent subclass learning, we have shown a general technique to uncover some of these features discretely, providing a starting point for descriptive analysis of our models' latent ontologies. the high specificity of our method opens doors to more insights from future work, which may include investigating how lsl results vary with probe architecture, developing intrinsic quality measures on latent ontologies, or applying the technique to discover new patterns in settings where gold annotations are not present. alex wang, yada pruksachatkun, nikita nangia, amanpreet singh, julian michael, felix hill, omer levy, and samuel r. bowman. 2019b. super-glue: a multi-task benchmark and analysis platform for natural language understanding. in h. wallach, h. larochelle, a. beygelzimer, f. d'alché-buc, e. fox, and r. garnett, editors, advances in neural information processing systems 32, pages 3261-3275. curran associates, inc. alex wang, amanpreet singh, julian michael, felix hill, omer levy, and samuel bowman. 2018. glue: a multi-task benchmark and analysis platform for natural language understanding. in proceedings of the 2018 emnlp workshop black-boxnlp: analyzing and interpreting neural networks for nlp, pages 353-355. association for computational linguistics. ralph weischedel, martha palmer, mitchell marcus, eduard hovy, sameer pradhan, lance ramshaw, nianwen xue, ann taylor, jeff kaufman, michelle franchini, et al. 2013. ontonotes release 5.0 ldc2013t19. linguistic data consortium, philadelphia, pa. zhiyong wu, yun chen, ben kao, and qun liu. 2020. perturbed masking: parameter-free probing for analyzing and interpreting bert. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 4166-4176. association for computational linguistics. kelly zhang and samuel bowman. 2018. language modeling teaches you more than translation does: lessons learned through auxiliary syntactic task analysis. in proceedings of the 2018 emnlp workshop blackboxnlp: analyzing and interpreting neural networks for nlp, pages 359-361. association for computational linguistics. yuhao zhang, victor zhong, danqi chen, gabor angeli, and christopher d. manning. 2017. positionaware attention and supervised data improve slot filling. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 35-45. association for computational linguistics. yukun zhu, ryan kiros, rich zemel, ruslan salakhutdinov, raquel urtasun, antonio torralba, and sanja fidler. 2015. aligning books and movies: towards story-like visual explanations by watching movies and reading books. in proceedings of the ieee international conference on computer vision, pages 19-27.