Made of Steel? Learning Plausible Materials for Components in the Vehicle Repair Domain (2023)

Abstract: We propose a novel approach to learn domain-specific plausible materials for components in the vehicle repair domain by probing Pretrained Language Models (PLMs) in a cloze task style setting to overcome the lack of annotated datasets. We devise a new method to aggregate salient predictions from a set of cloze query templates and show that domain-adaptation using either a small, high-quality or a customized Wikipedia corpus boosts performance. When exploring resource-lean alternatives, we find a distilled PLM clearly outperforming a classic pattern-based algorithm. Further, given that 98% of our domain-specific components are multiword expressions, we successfully exploit the compositionality assumption as a way to address data sparsity.

## introduction connecting a symptom to an underlying cause is a crucial building block for natural language understanding across domains. for example, as illustrated in fig. 1 , a standard approach that human mechanics in vehicle repair shops apply when tracing the cause of a symptom, is to exploit the link between a vehicle component and the component's materials. this study tackles the task of automatically learning plausible materials for vehicle components. the information is crucial in vehicle repair shops, as mechanics are faced with constantly growing vehicle complexity, making it hard to manually identify the cause of a malfunction. while domain-specific information on vehicle components is often available, plausible domainspecific material information typically needs to be gathered from external data sources. for example, from "brake disk are usually manufactured from gray cast iron", one may retrieve the information that brake disks consist of iron (and possibly further materials not mentioned here). we propose a novel approach to leverage stateof-the-art pretrained language models (plms) for the task of learning plausible materials in highly domain-specific contexts, such as vehicle components in the vehicle repair domain. our approach can also be applied to similarly specific domains, e.g., to learn which materials or substances are plausibly used in products in the health domain, textile industry, etc. as we tackle a task from material science, we make a contribution to a field that is not only very challenging, but heavily underinvestigated from an nlp perspective. we focus on the plm roberta @xcite and develop an approach that successfully learns plausible materials in english for components in the vehicle repair domain. we further exploit domain-adaptation variants and show (i) that harnessing a small highquality domain-specific corpus boosts the performance over an out-of-the-box vanilla roberta, and (ii) how similar performance can be reached by using a widely accessible data source such as wikipedia if no domain-specific data is available. to overcome the typical lack of annotated datasets for training and fine-tuning models in highly domain-specific tasks, we probe roberta in a cloze task setting @xcite , compare variants for aggregating the results, and analyze the influence of a varied set of query templates on model predictions. while plms come with great advantages, their hunger for storage, data, and computing power is even greater. we thus compare the performance of the vanilla and domain-adapted versions of roberta with (i) a distilled version, i.e., the much smaller distilroberta @xcite , and (ii) the seminal pattern-based bootstrapping algorithm basilisk @xcite . finally, we tackle the challenge of handling a dataset with a vast majority of multiword expressions, i.e., 98% of our targets are noun compounds (such as seat heating switch led). we address the corresponding severe data sparsity by assuming that many domain-specific compounds are compositional @xcite such that our domain-specific model may fall back to information regarding the compound's head (in the example above: the right-most simplex noun in the compound led), and thus improve on the data sparsity. prompting and cloze query engineering recent work has leveraged cloze-task style settings to probe the knowledge that plms acquire during pretraining, thus targeting linguistic capabilities @xcite @xcite , the understanding of rare words @xcite ) and conceptual abstractions @xcite , as well as factual and commonsense knowledge @xcite ). an emerging strand of research highlights plm sensitivity to the input in a cloze-task style setting. for example, @xcite demonstrate that plms show deficiencies in prediction consistency when presented cloze-style query paraphrases. others exploit this flaw by explicitly considering paraphrases of a cloze query in addressing a model's sensitivity to a specific input query to elicit a desired output @xcite . @xcite show that models lack robustness in their ability to harness relevant context information in the face of cloze tasks containing distracting contextual cues. in reverse, priming the model using trigger tokens @xcite and lexical cues @xcite might aid in guiding the model to predict the desired output. specifically, @xcite propose autoprompt to develop automatically constructed prompts or patterns to elicit knowledge from pretrained plms for a variety of tasks. in contrast to previous work, we semi-automatically construct a variety of prompts to cover a salient set of paraphrases for eliciting plausible materials in a plm; this allows for meaningful comparison and interpretation of the predictions' quality and the influence of the cloze queries. semantic plausibility while classical distributional models tend to model selectional preferences and thematic fit instead of capturing semantic plausibility @xcite , we have recently seen advances to model plausibility across various dimensions, including physical and abstract semantic plausibility @xcite . sota models for event plausibility however still rely on straightforward conditional probabilities of co-occurrences as estimated by distributional models @xcite . considering our task of learning plausible materials for vehicle components, we go beyond selectionally preferred component materials as predicted with high probabilities, aiming also for less frequently observed cases that are nevertheless plausible. ## plm experiments our plm experiments investigate to what extent and how plms encode domain-specific component materials when prompted with cloze query variants, as illustrated in fig. 2 : we use the vehicle component dataset to construct a set of 504 cloze query templates for each individual component. we first probe a roberta model without any modifications and gather the top-5 predictions for each component, resulting in a total @xmath0. to aggregate these predictions, we experiment with three aggregation methods and determine the top-5 most plausible predictions per component and aggregation method for expert evaluation. in a followup experiment, we perform domain-adaptation of roberta and distilroberta for the vehicle repair domain using our two domain-specific corpora, and compare the probing results of the vanilla and the domain-adapted models. the following subsections describe our plm experiments in detail. ## basilisk experiments: a classic in comparison to the plm experiments, we explore the seminal pattern-based bootstrapped learner basilisk @xcite for learning domain-specific plausible materials for vehicle components. basilisk is designed to learn highquality semantic lexicons for one or more categories, provided an unstructured natural language text corpus plus seed words for each semantic category to be learnt. the resource-lean learner leverages extraction patterns representing linguistic contexts, thus exploiting lexico-syntactic structures to capture word meaning. as the starting step for basilisk, we select ten seed nouns from the most frequent words in do-main foot_4 that belong to the semantic class material. since basilisk depends on extraction patterns to supply contextual support for additional words belonging to the same semantic class, we rely on dependency parsing to create domain-specific syntactic contextual patterns by using the seed words obtained from domain and only the domain corpus. we then apply basilisk as bootstrapping algorithm to select the best-performing patterns for a pattern pool and subsequently fill a candidate word pool with the extractions of the highest-scoring patterns. we retrieve, for example, s e e d * <gdep>:<compound>:<dependent>:alloy on the basis of which the word alloy gets added to the candidate word pool. we stop basilisk after n bootstrapping rounds and store the retrieved semantic lexicon of n plausible materials for a given vehicle component. to then connect these possible materials with the components, we first process the underlying unstructured data sources and only keep sentences containing at least one component name mention. second, we filter the obtained sentences with the semantic lexicon and store all component-material candidate matches. in this way, we generate 396,887 extraction patterns, and use the basilisk implementation @xcite with all corresponding patterns and seed words on the domain dataset. we limit the candidate word pool size to n " 200, and generate possible material candidates for both the do-main and the wiki datasets. ## evaluation to our knowledge, no gold standard is available for the task of plausible material extraction in the vehicle repair domain. we thus perform an expert evaluation of the quality of predictions for each slot obtained from the vanilla and the domain-adapted roberta models. to compare plm-based results to resource-leaner methods, we also evaluate results from the pattern-based basilisk algorithm and distilroberta. evaluation task for each model, we select and evaluate 100 randomly sampled components, foot_5 (i) balancing the number of constituents per target component, (ii) providing the full component vs. providing only its head, and (iii) varying the underlying corpus (for basilisk only). for both 1-and 2constituent words, foot_6 we sample 30 components with 20 full/10 head samples with equal amounts from domain/wiki. for 3-constituent words, we sample 20 instances with equal amounts of full/head and domain/wiki samples. for both 4-and 5constituent words, we sample 10 head instances with equal amounts of domain/wiki samples (cf. app. c for further details). we obtain the aggregated top-5 predictions for vehicle components from each model. for basilisk, we sample up to five answer options whenever more than five entries are extracted for a given component. to minimize annotator bias, we compile a set of up to 18 materials for each vehicle component while keeping track of prediction origins. this way, annotators see each evaluated component only once and rate all predictions from all models for evaluation setup we present each component instance with corresponding materials to three experts foot_7 from the vehicle repair domain. the annotators are asked to rate the plausibility of material predictions for each vehicle component, by ticking the correctly identified materials. they are also provided the options "none of these" if no material is plausible, and "i do not know the answer" if this is the case. inter-annotator agreement (iaa) is shown in ## results and discussion on the left, we show accuracy on component level across n annotators. accuracy is defined as the proportion of components for which at least one of a model's top-5 material predictions is rated plausible by the annotators. on the right, we display accuracy on material level where accuracy is defined as the proportion of plausible material predictions among a model's top-5 predictions, as rated by n expert annotators. foot_8 ## conclusion we tackled the task of learning domain-specific plausible materials for components in the vehicle repair domain from a novel perspective by probing sota language models in a cloze task style setting to overcome the lack of annotated datasets. based on a diverse set of semi-automatically constructed cloze queries combined with a suitable aggregation method, we presented a new method to efficiently extract knowledge regarding vehicle components and their materials as acquired by a plm. while showing that domain-adaptation using either a small, high-quality or a customized wikipedia corpus boosts performance, we also demonstrated the power of resource-lean alternatives such as the plm distilroberta, and found that the bottleneck for domain-adaptation with respect to our task might not be model size but rather corpus quality and suitability. finally, we successfully exploited the compositionality assumption for our domainspecific multiword expressions as a way to address data sparsity.