A Human-machine Interface for Few-shot Rule Synthesis for Information Extraction (2022)

Abstract: We propose a system that assists a user in constructing transparent information extraction models, consisting of patterns (or rules) written in a declarative language, through program synthesis. Users of our system can specify their requirements through the use of examples,which are collected with a search interface. The rule-synthesis system proposes rule candidates and the results of applying them on a textual corpus; the user has the option to accept the candidate, request another option, or adjust the examples provided to the system. Through an interactive evaluation, we show that our approach generates high-precision rules even in a 1-shot setting. On a second evaluation on a widely-used relation extraction dataset (TACRED), our method generates rules that outperform considerably manually written patterns. Our code, demo, and documentation is available at https://clulab.github.io/odinsynth .

## introduction rule-based methods for information extraction address the opacity of neural architectures by producing models that are completely transparent, i.e., they are usually a collection of rules written in a declarative language. such models are better suited for incremental improvements, as each individual rule can be explicitly interpreted. however, these benefits do not come for free: users of such systems must be familiar with the underlying declarative rule language, and, potentially, with representations of syntax such as syntactic dependencies. none of these are trivial to users outside of natural language processing (nlp), which, we argue, should be the target users of these systems. to mitigate the above limitation, we propose a human-machine interface (hmi) that: (a) lets users synthesize rules from natural language examples, and correct them without necessarily understanding the rule syntax (although expert users who do have access to the actual rule produced), (b) generates rules in a few-shot setting, i.e., from a very small number of examples. the latter contribution is possible because our rule synthesis engine has been pretrained on a large collection of rules that were automatically generated from a large text corpus @xcite . in other words, our rule synthesis approach is akin to prompting for language models @xcite . that is, we first train an open-domain rule synthesis model, and then we guide its predictions to the task of interest using a small number of examples of the desired extractions (the "prompt"). we include two types of evaluations that prove the value of the proposed approach. the first evaluation focuses on interactive sessions where users generate rules that extract mentions of named entity classes, e.g., city or academic institution from a single example extraction. using six different users, we show that, despite the minimal supervision, the tool produces named entities in the corresponding classes with high precision, e.g., precision at 20 (p@20) at over 75% for the city class. this suggests that the proposed hmi is useful to domain scientists that need to perform information extraction quickly, without understanding the underlying nlp technology. for completeness, we also include a traditional, non-interactive evaluation, on the tacred dataset @xcite , through which we show that the rules synthesized by our approach outperform the manually-written rules by over 4 f1 points, even though our synthesis component is not re-trained on the tacred data. ## architecture at a high level, the proposed system consists of two main modules: a rule synthesis component, and an ui module to facilitate rapid rule prototyping with minimal programming or linguistic knowledge. the rule synthesis component consists of: (a) a searcher that explores the possible rule space, and (b) a neural scorer that prioritizes next steps during rule generation. figure 1 summarizes the overall architecture. the user selects a handful of examples after an initial query. then, our system proposes a rule, together with its potential extractions. the user then decides if this rule is satisfactory. if not, the user can either ask for a new rule or add new examples (positive or negative). this process repeats until the generated rule is accepted. we describe these components in detail in the next two sections. ## rule synthesis our proposed approach for rule generation follows closely the method proposed in @xcite . for completeness, we summarize it here as well. our rule synthesis method uses enumerative search that is guided by a transformer-based scoring mechanism, and is optimized using search-space pruning heuristics. our transformer model scores each potential next state (where a state contains the incompletely generated rule up to this point), given the current state, such that the number of states to be explored is minimized. specifically, our system consists of two main components: a searcher, with branch and bound @xcite as the underlying algorithm. the searcher uses the scores assigned by the scorer (below) to determine the order of exploration, choosing the state with the highest score, regardless of its position in the search tree. as such, it is important for the scorer to assign high scores to states that are in the sub-tree that leads to the desired final rule, and lower scores to all other states; a scorer, with a transformer backbone that is initialized with a pretrained model, but finetuned through self-supervision, i.e., over automatically generated rules (see section 3.2). the scorer uses the current state and the specification, i.e., the natural language examples to be matched by the generated rule, to score each potential next state. the searcher is responsible for exploring the states in priority order (as determined by the scorer), and deciding if a given state is successful (i.e., it is a valid query and correctly extracts the requested highlighted words and nothing more). the search space can be interpreted as a tree, where the root is the initial candidate solution and the children of a node n are the candidate solutions that the node n could expand into. given this, the searcher can be seen as iteratively applying a sequence of three operations: (a) expand the current state according to the domain-specific language grammar, (b) score each expanded candidate next state and insert them into the priority queue, and (c) select from the queue the state with the highest score to be the next state. we repeat this process until we find a solution or we reach our step limit. the scorer assigns a numerical value to states to establish the order of exploration. we explore two variants: a static variant, which assigns static weights to states based on their components, and a contextual neural variant based on a self-supervised method that assigns contextual state scores based on the current context. as a simple example, consider a user that wants to learn named entities belonging to the class city. she may start with a specification based on the sentence "regina romero is the mayor of tucson, arizona, having been elected after. . . ", in which she highlights "mayor of" as the relevant context representative of this class, and "tucson" as the desired entity to be extracted. from this specification, our method would generate the rule: [@xmath0, foot_0 which picks up "mayor" followed by a preposition (part-of-speech tag in) as the context, and a sequence of 1 or more proper nouns as the entity to be extracted. ## user interface we accompany the above rule synthesis component with a user interface (ui) to facilitate rapid prototyping with minimal programming or linguistic knowledge. we showcase the ui in figure 2 , split into 5 blocks (a-e). initially, the user has to do an initial search for sentences of interest (a). then, she selects any relevant sentence, highlighting the parts for which she wishes to obtain a rule, e.g., highlighting the capital city of as the context and amman as the entity of interest (b). the system then returns a potential rule which satisfies the current constraints, together with what the rule extracts (c). note that we display the rule for the benefit of expert users, but most users are not required to understand the format of the rule. that is, a user can understand the rule's impact by analyzing what such a rule extracts. she may add negative examples (a negative example is a sentence on which the output rule should not match anything), or ask for a new rule (d). this process is repeated until the user is satisfied with the given rule (e). ## conclusion and future work we introduced a human-machine interface that lets users synthesize rules from natural language examples, and correct them without necessarily understanding the rule syntax. in an interactive evaluation, we showed that our method is capable to rapidly generate highprecision rules for the extraction of named entities from a single example in natural language. we also demonstrated that in a traditional, non-interactive evaluation on the tacred dataset, our method produces rules that outperform manually-written rules, despite the fact that our rule synthesis engine is not retrained on the tacred data. while these initial results are exciting, there is plenty of work left to do. first, the rules generated by the system are "surface" rules, i.e., they act over sequences of tokens. adding support for the generation of rules over syntax would allow for capturing more complicated relations and over greater distances. second, the system is configured to generate a single rule that captures the whole specification, which may include multiple (positive or negative) examples. this may force the system to produce complicated rules. a better alternative would be to produce several simpler rules that together capture the whole specification. lastly, the system generates a sequence of rule candidates that are presented to the user one-by-one, which may bias the user's perspective and impact usability. for example, the current system tends to initially propose rules that are overly general, which yield low-precision results. to better understand users' preferences (i.e., do users prefer high-precision or high-recall rules initially?) user studies must be carried out.