Leveraging Approximate Pattern Matching with BERT for Event Detection (2024)

Abstract: We describe a new weakly supervised method for sentence-level event detection, based exclusively on linear prototype patterns like “people got sick” or “a roadside bomb killed people”. We propose a new BERT based algorithm for approximate pattern matching to identify event phrases, semantically similar to these prototypes. To the best of our knowledge, a similar approach has not been used in the context of event detection. We experimented with two event corpora in the area of disease outbreaks and terrorism and we achieved promising results in sentence level event identification: 0.78 F1 score for new disease cases detection and 0.68 F1 in detecting terrorist attacks. Results were in line with some state-of-the-art systems.

## introduction early event extraction systems predominantly rely on pattern matching and linguistic rules @xcite . this approach remains particularly effective in well-defined domains, such as disease outbreaks, biomedical papers, disasters, security, and socio-political developments, where language is clearly structured @xcite ; @xcite ; @xcite . in specific contexts, linguistic rules can offer competitive precision and enhanced transparency compared to machine learning (ml) models @xcite . linguistic rules can also be used for automatic corpus annotation, when new domains of event extraction are being considered and no training data is available @xcite . the transparency inherent in linguistic rules is particularly vital in real-world event extraction applications. end users can provide feedback on the performance of specific keywords and phrases, thereby improving the accuracy and breadth of the rule set. in this work we argue that the combination of manually crafted linear patterns and large language models (llm) is a promising avenue for combining the strengths of the knowledge based approaches and the llm in the domain of event detection. we experimented in the domains of security and health, but we think that the approach is applicable across a wide range of domains and event classes. llm like bert @xcite offer the capability to create utterance abstractions, using the contextualized word embeddings, which can be received from the embedding layer of the bert neural network, see figure 1 . in the context of event detection, this allows for creating simple linear patterns as prototypes, e.g. "people have got a disease", and using the bert contextualized embeddings of both prototypes and analysed text to find the semantic relation between the patterns and their lexical variations in the text, e.g. "children have got influenza". more concretely, we propose the following approach, starting with prototype patterns (here we consider the event types new disease cases and terrorist attacks) like "disease outbreak", "number people were infected" or "bomb exploded", to discovers in the test set sentences containing text fragments, containing words with similar bert embedding vectors -"influenza outbreak", "covid was discovered in 2 foreign nationals", or "blast killed", etc. these phrases are supposed to be semantically similar to the prototypes, because of their similarity in the bert encoding. our experiments demonstrated that bert-based pattern matching is able to infer event mentions which have significant lexical and syntactic differences with respect to the prototype patterns. we tested our approach on the task of detecting sentences containing events of a predefined event type. two event classes were considered in our experiments: new disease cases and terrorist attacks. for both of them we achieved performance much higher than the baseline. moreover, for the event type new disease cases, the achieved performance was in line with other systems, based on supervised ml. ## related work event detection at various levels: token, sentence and document level has been largely addressed in previous work. the case shared task on protest detection and the participating systems @xcite tackle event detection at all the three levels. various methods for sentencelevel event detection are studied in @xcite . a survey of the existing event detection and extraction approaches were presented in @xcite and @xcite pattern matching is a well established method for extracting event triggers and arguments. earlier event extraction systems massively exploited lexico-syntactic patterns ( xiang and wang, 2019 related to the prototype pattern matching we propose here, is another bert based entity matching approach @xcite . bert pattern matching is also used in question answering to find similar questions @xcite . supervised learning, considering event triggers, is described in several works: @xcite , @xcite , @xcite , and @xcite . ## the approach the approximate pattern matching approach is designed to identify in a test corpus the sentences containing event descriptions. at the same time, the algorithm identifies an n-gram in each of these sentences, matching best one of the prototypes. in our experiments we considered two event types -new disease cases and terrorist attacks, however we think that the method is applicable across various domains and event classes. ## experiments to assess the efficacy and adaptability of our event detection methodology, we collected a test set of two distinct event corpora, each derived from a disparate domain: disease outbreaks @xcite and politically motivated violence and disasters @xcite . a unique targeted event type was specified for each corpus. our approach involves the systematic crafting of a set of carefully tailored linear patterns for the specific event types. the formulation of patterns drew upon insights derived from a development set, encompassing 300 sentences extracted from each respective corpus. this was done in the following steps: generally, the creation of linguistic patterns is a intricate process, usually encompassing a combination of machine learning techniques and expert assessment @xcite . however, in this study, our approach involved crafting patterns primarily based on linguistic expertise, with the development set used to assess their coverage and precision. the pattern development process was not the central focus of our work. instead, we followed a pragmatic approach akin to what an average pattern developer might undertake, aiming for optimal results without substantial time investment. in order to test the accuracy of the prototypes, we randomly selected a test subset from each corpus, non overlapping with the development set. before we run the algorithm, we had to set the α eventness threshold. our observation on the development set was that the threshold delivers meaningful results in the interval 0.6 to 0.7. therefore, we have run the evaluation with three different values for α: 0.6, 0.65, and 0.7. we applied our approximate pattern matching detection of sentences on each corpus for each of the three α threshold values. we have also defined a baseline -exact pattern matching: if even one pattern is contained as a substring in a sentence, then the sentence is considered to contain an event. in experiments showed that our method outpaced by a considerable margin the baseline. at the same matching n-gram with surroundings pattern ...the number of confirmed covid-19 cases... number of infected the number of zika virus cases has crossed 100 ... number of infected ...raising the death toll due to the disease to 11 ... death toll from the outbreak ...the situation where the observed number of cases exceeds... number of infected ...the number of people testing positive for the infection rose... testing positive for virus ...321 new domestically transmitted coronavirus cases... confirmed disease cases ...proportion of those testing positive to the total tests... number of infected ...new clusters of coronavirus infections are igniting concerns... new infection cases ...new confirmed coronavirus infections have hit a record... confirmed disease cases analysing the errors for the terrorist attack event type, we saw that there are text fragments matched against the patterns, where terrorists were victims, rather than attackers. some sentences describing assassinations and kidnappings, especially in the middle east were also erroneously labeled as terrorist attacks. for example, the phrase "victim of an assassination attempt" erroneously matched the pattern "victim of a terrorist attack". also "air raid killed civilians" erroneously matched the pattern "market bomb targeted civilians". these and other pattern matching errors clearly show that in some cases the bert pattern matching may be misled by particular phrases in certain contexts. ## conclusions results from moreover, some of the detected n-grams are relevant as event detecting phrases and they themselves can constitute prototypes. following this line of thinking, the approximate pattern matching algorithm can also be used for learning of new patterns. as a conclusion, our experiments show that bert-based pattern matching is an efficient weakly supervised event classifier. this method combines the simplicity and transparency of the pattern-based approaches and the implicit semantic knowledge, encoded in large language models like bert.