Accelerating Human Authorship of Information Extraction Rules (2022)

Abstract: We consider whether machine models can facilitate the human development of rule sets for information extraction. Arguing that rule-based methods possess a speed advantage in the early development of new extraction capabilities, we ask whether this advantage can be increased further through the machine facilitation of common recurring manual operations in the creation of an extraction rule set from scratch. Using a historical rule set, we reconstruct and describe the putative manual operations required to create it. In experiments targeting one key operation—the enumeration of words occurring in particular contexts—we simulate the process or corpus review and word list creation, showing that several simple interventions greatly improve recall as a function of simulated labor.

## introduction to maximize accuracy and robustness under the state of the art in information extraction (ie), one trains machine learning (ml) models, typically underpinned by neural language models, on large numbers of sentence-level annotations @xcite @xcite . if annotations are sufficiently numerous, this approach yields robust extraction capabilities that are difficult to implement through other means. and it has methodological advantages, inasmuch as the annotations serve as a precise extensional definition of a given extraction problem, one that can be trivially exploited in the development of improved extractors through new learning algorithms and architectures. however, this approach imposes certain costs that are not immediately apparent and that manifest as diminished agility, including: most concerningly, the community's shared focus on a small number of canonical problems, while it fosters replicability and fundamental progress, inhibits progress on methods that would enable the practical deployment of ie on a truly broad range of problems. many real-world problems involve data or use cases too distinctive to be solved with community models, and many candidate customers of ie lack the resources for adequate data annotation. rule-based approaches to ie offer an alternative for the deployment of competent novel extractors @xcite . while they suffer from certain limitations-limited retargetability, reduced recall, etc.-they possess one significant advantage over ml-based approaches, as illustrated in figure 1 . specifically, in the early stages of an effort pursuing novel extractors, they support very rapid deployment. hours of effort often suffice to implement usable extractors, where an equivalent ml-based extractor would require days or weeks. we argue that this "early deployment advantage" makes rule-based ie an important tool in real-world settings. importantly, rule-based methods and ml are not mutually exclusive. we have previously presented evidence that rule-based extractors can be used to annotate training data for ml, and that the resulting models generalize the rules in useful ways @xcite . in this paper, we consider a tighter integration between rule-based ie and ml, one in which ml facilitates the authorship of rules by offering options and suggestions to the human technician. in a new extraction problem area lacking annotations, the rule author is confronted with a difficult search problem-a difficulty that increases with the expressiveness of the rule language. we hypothesize that ml can be used to simplify the search in ways that dramatically reduce effort. this paper is an attempt to illuminate the dimensions along which such assistance is possible. we approach this through analysis of a historical rule set for extracting quantitative claims from the scientific literature on solar materials. by inspecting how various language features were used in pursuit of a performant extraction model, we attempt to infer some of the operations employed by the author in the initial construction and subsequent refinement of an improving rule set. and we provide preliminary quantitative evidence that some simple interventions could have substantially accelerated a key operation: the creation of problem-specific word lists. to summarize, we make the following contributions in this paper: ## related work the use of declarative, efficiently executable rules for information extraction was a common feature of early work in the area, which led to the creation of several rule frameworks @xcite @xcite . motivated by the difficulty of purely manual rule creation, early applications of machine learning to the problem sought to facilitate aspects of the authoring process, particularly the creation of what we call word sets and what the literature often calls dictionaries or semantic lexicons @xcite . this line of research led to some general methods for exploiting syntagmatic search (contextual patterns) for the assembly of paradigmatic resources (lexicons) @xcite , but by treating the lexicon as an end in its own right, it begged the question of ultimate utility for the downstream task of information extraction. early successes in lexicon induction gave rise to research pursuing end-to-end extraction through supervised rule or pattern induction @xcite @xcite . this work offered the advantage of improved replicability and retargetability, replacing the highly technical activity of rule creation with the more transparent activity of data annotation. however, once annotated data was available in sufficient volumes, rule-based representations were eventually outperformed by less constrained representations better able to integrate diverse signals in the data @xcite @xcite . the center of gravity in subsequent research has focused on models able to exploit large volumes of annotated data and the acquisition of data in sufficient volumes to realize their advantages. because annotation overhead hampers application of these methods to new problems, the field continues to investigate approaches to reducing annotation overheads, including few-shot learning @xcite @xcite and transfer learning @xcite @xcite . in some cases, these methods make it possible to achieve impressive competence in a new task with very few training examples. but note that such approaches, inasmuch as they often transfer extraction knowledge from known target types or from highly resourced domains to adjacent ones, do not eliminate the need for annotation. and it is often questionable whether the resulting models are sufficiently performant for downstream use without supplementation. if rule-based approaches to extraction have ceased to be a major research focus, they remain an important tool in the toolkit of practitioners @xcite and available as features of several general-purpose nlp toolkits @xcite @xcite . although new rule frameworks occasionally feature in the more recent literature @xcite @xcite , these works are almost exclusively descriptive, failing to provide empirical benchmarks that would facilitate continued research in the area. in particular, the process of authoring rules has received no prior empirical scrutiny, making it difficult to corroborate perceived advantages of rule-based methods. ## a historical rule set as part of a project attempting to document progress in solar materials research, we developed an extractor for quantatitive "claims," statements that communicated some important scientific measurement. our experimental data consisted of approximately 160k abstracts from the web of science foot_0 on solar energy research from 1968 to 2014. as is typical in projects like this, ie was not the focus of the effort, but only a means to assemble structured data for downstream analysis, which in this case sought to summarize diachronic progress on several key research dimensions. as shown in figure 2 , a claim is a binary relation between two domain-specific entities or concepts: a quantitative expression or measurement, and the corresponding metric or quantity being measured. for greatest flexibility in downstream analysis, our definition of claim was inclusive, encompassing any expression reflecting the result of a scientific measurement. as the example in the figure illustrates, the two phrasal extraction targets pose different challenges. measurements, consisting typically of a number and a unit of measurement, exhibit strong orthographic regularities, parts of which could be exploited with regular expressions. metrics, on the other hand, are noun phrases. to address this extraction challenge, we employed valet, a recently described ie rule syntax and framework implemented in python @xcite . the earlier version of valet used in this work lacked several of the features of the current framework. in particular, the rule author had no access to syntactic information. thus, the problem of extraction amounted to scanning tokens in the input stream sequentially, relying on orthographic and lexical clues to decide when the left and right boundaries of the two phrasal targets were observed. we briefly describe valet's provisions for such scanning to simplify later exposition. readers interested in more detail or a review of valet's more recent features are referred to the paper or the more extensive documentation in the public release. a statement or rule in valet is a sequence of three things: a name, a piece of syntax indicating the type of rule, and an expression defining the rule's behavior. the evaluation of such a statement yields an extractor, which can be applied directly to text (e.g., via scripts from the command line) or incorporated into subsequent statements through reference to the rule's name. the rules in this study relied on two types of expressions, token class expressions and phrase expressions. the example token class expression determiner: { a an the }i defines a case-insensitive extractor matching the individual words listed between the braces. phrase expressions employ a regular expression syntax to match multi-token sequences, enabling the rule author to mix previously defined token classes with literal tokens. in addition, phrase expressions can co-refer, enabling context-free composition. consider the rules: the person phrasal extractor in this example recognizes person mentions prefixed by an honorific, incorporating a separate phrase extractor for sequences of capitalized tokens (caps) by reference. (unlike in standard character-level regular expressions, the optional '.' has no special significance and matches period tokens in the input literally.) the rule set used to extract claims consists of 34 rules (15 token class expressions and 19 phrasal the devices exhibited a short-circuit current density of -6.14 ma/cm(2), an … metric measurement claim claim -> @metric @between @measurement notmetricword : { . a an the and of to is are … } metricword : not &notmetricword metrichead : { efficiency voltage density … } metric -> metricword* metrichead quant -> ( -| + ) ? &num ( . &num ) unit : { hz khz mhz angstrom m nm cal kcal … } unitphrase -> &unit ( . ? &unit ) * ( / &unit ) ? measurement -> @quant @unitphrase extractors). we next review the structure of this rule set and consider what it implies about the process of human rule development. ## rule search a human technician confronting a new extraction problem faces a daunting challenge. even if they possess the means to observe the effects of any changes to rules, a protracted exploratory process is required to arrive at an effective solution to any extraction problem other than the most trivial. to understand where ml and automation might facilitate that process, we first seek to form an intuition regarding the solution structure for extracting claims, as representative of a broader class of similar problems, then enumerate potential points of intervention in the putative search that produced this solution. ## discussion this work is an initial step in a line of inquiry that could lead to better tooling in support of more agile extraction. the key insight is that once we have a performant rule set, one that we are willing to treat as authoritative, we can simulate the process that led to its creation and experiment with new modes of facilitation in pursuit of greater labor savings and model robustness. critical to such research, and a focus of future work, is a credible cost model that quantifies levels of authoring effort. not only would such a model provide a more precise characterization of the "early deployment advantage" of rules over ml, but it could help widen this advantage as an objective function for simulations of the authoring process. of course, this approach has certain shortcomings. for one, any model, including our historical rule set, that is not developed and vetted against a thoroughly annotated data sample is typically an approximation, usually one that is recall-limited. in our previous work, we sought to overcome this limitation by using the rule set to generate a large annotated sample to train a high-recall sequence labeler @xcite . here, we treat the rules as definitional, but it seems clear that some of the "false positive" elements nominated by our corpusanalytic rankers belong in the definition. for example, only one of the top ten terms nominated for metrichead by it-set after two iterations of review was in the historical word set, but many of the excluded nominations appear plausible (e.g., reflectance, oxidation, or transmittance). many of these words presumably occur rarely (if at all) as part of claim expressions, and our performance metric's emphasis on maximizing recall punishes rankers that promote terms in the tail of the distribution, but a complete account of claim language in this domain might want to include them. a salient feature of all of these results is our ability to reach full recall quickly using a high-quality anchoring rule and a relatively simple ranking policy. but this outcome may in partly reflect a circularity in the experimental methodology. our anchoring rules are elements of the historical model, and they therefore necessarily enable us to review all sentences that the rule set considers relevant. a key unanswered question is: what do these anchors miss? our previous work, which used this rule set to train an ml extractor, yielded apparently valid claim expressions that the rule set does not sanction @xcite . perhaps methods such as it-set and ft-centroid, which seem wasteful of human effort, can be used to identify alternative or develop more general anchors. more generally, the structure of a historical rule set is a reflection of the rule language and tooling available to the author, and conclusions drawn from a study of such a rule set may overlook promising points of integration between rule-based methods and machine learning or corpus analytics. for example, the current valet framework supports on-demand application of it-set via an interactive dialog presenting a large list of words deemed to be close to a chosen word in the text. the user can select any of the words in the list and ask the development ui to generate a new word set expression. similarly, valet offers a "radius" statement that matches words within some distance of a seed set in lexical embedding space. and we have begun investigating a trainable word set feature that engages the user in an active learning loop to derive a customized word matcher, one that can in principle exploit contextual embeddings. while such features are potentially powerful, they sacrifice transparency and fine-grained controltwo attractive aspects of rule-based methods. in this respect, they are in the tradition of alternative approaches to rapid ie deployment, such as snorkel @xcite , which seeks to learn performant extractors from collections of noisy "labeling functions." such approaches, for problems on which they work, can lead to impressive labor savings, but they are difficult to control and optimize. but note that while snorkel-like approaches and traditional rule-based methods approach the ie objective from different angles-snorkel through redundant, high-recall labelers, rule-based methods through high-precision set covering-they are fundamentally compatible and offer interesting opportunities for hybridization. trivially, a framework like valet can be used to conveniently implement labeling functions. by the same token, snorkel points the way to a mode of rule set application distinct from the typical disjunctive mode. ## conclusion rule-based methods remain an important component of any toolset addressing the broader problem of information extraction, especially in cases where existing extraction models or sources of annotated data are misaligned to new use cases. a trained technician, outfitted with a suitable rule authoring framework, can create a performant extractor for a new problem in a fraction of the time required to produce a ml model of comparable accuracy. moreover, we have shown that some simple facilitations, based on an analysis of the rule authoring process, can serve to increase this "early deployment advantage." and by treating rule development as the focus of empirical investigation, we have pointed the way toward future systems in which rules and ml are combined creatively to lower the barrier to entry in the creation of custom extraction solutions.