Bayesian Information Extraction Network

Dynamic Bayesian networks (DBNs) offer an elegant way to integrate various aspects of language in one model. Many existing algorithms developed for learning and inference in DBNs are applicable to probabilistic language modeling. To demonstrate the potential of DBNs for natural language processing, we employ a DBN in an information extraction task. We show how to assemble wealth of emerging linguistic instruments for shallow parsing, syntactic and semantic tagging, morphological decomposition, named entity recognition etc. in order to incrementally build a robust information extraction system. Our method outperforms previously published results on an established benchmark domain. 1 Information Extraction Information extraction ( IE) is the task of filling in template information from previously unseen text which belongs to a pre-defined domain. The resulting database is suited for formal queries and filtering.IE systems generally work by detecting patterns in the text that help identify significant information. Researchers have shown [Freitag and McCallum, 1999; Ray and Craven, 2001 ] that a probabilistic approach allows the construction of robust and well-performing systems. However, the existing probabilistic systems are generally based on Hidden Markov Models ( HMMs). Due to this relatively impoverished representation, they are unable to take advantage of the wide array of linguistic information used by many non-probabilistic IE systems. In addition, existing HMM -based systems model each target category separately, failing to capture relational information, such as typical target order, or the fact that each element only belongs to a single category. This paper shows how to incorporate a wide array of knowledge into a probabilistic IE system, based on dynamic Bayesian networks ( DBN)—a rich probabilistic representation that generalizes HMMs. Let us illustrateIE by describing seminar announcements which got established as one of the most popular benchmark domains in the field[Califf and Mooney, 1999; Freitag and McCallum, 1999; Soderland, 1999; Roth and Yih, 2001; Ciravegna, 2001 ]. People receive dozens of seminar announcements weekly and need to manually extract information and paste it into personal organizers. The goal of an IE system is to automatically identify target fields such as location and topic of a seminar, date and starting time, ending time and speaker. Announcements come in many formats, but usually follow some pattern. We often find a header with a gist in the form “ PostedBy: john@host.domain; Who: Dr. Steals; When: 1 am; ” and so forth. Also in the body of the message, the speaker usually precedes both location and starting time, which in turn precedes ending time as in: ‘‘Dr. Steals presents in Dean Hall at one am.’’ The task is complicated since some fields may be missing or may contain multiple values. This kind of data falls into the so-called semi-structured text category. Instances obey certain structure and usually contain information for most of the expected fields in some order. There are two other categories: free text and structured text. Instructured text , the positions of the information fields are fixed and values are limited to pre-defined set. Consequently, theIE systems focus on specifying the delimiters and order associated with each field. At the opposite end lies the task of extracting information fromfree textwhich, although unstructured, is assumed to be grammatical. Here IE systems rely more on syntactic, semantic and discourse knowledge in order to assemble relevant information potentially scattered all over a large document. IE algorithms face different challenges depending on the extraction targets and the kind of the text they are embedded in. In some cases, the target is uniquely identifiable (singleslot), while in others, the targets are linked together in multislot association frames. For example, a conference schedule has several slots for related speaker, topic and time of the presentation, while a seminar announcement usually refers to a unique event. Sometimes it is necessary to identify each word in a target slot, while some benefit may be reaped from partial identification of the target, such as labeling the beginning or end of the slot separately. Many applications involve processing of domain-specific jargon like Internetese —a style of writing prevalent in news groups, e-mail messages, bulletin boards and online chat rooms. Such documents do not follow a good grammar, spelling or literary style. Often these are more like a stream-of-consciousness ranting in which asciiart and pseudo-graphic sketches are used and emphasis is provided by all-capitals, or using multiple exclamation signs. As we exemplify below, syntactic analysers easily fail on such Position 1 2 3 4 5 6 7 8 9 10 Tag <speaker > <location> <s-time> Phrase Doctor Steals Presents in Dean Hall at 1 am . Lemma Dr. present in hall at am PoS NNP NNP(VB) VB(NNS) IN NNP NN(NNP) IN CD NN(RB) . Syn.segm. NP NP (VP) VP PP NP NP PP NP NP(VP) Semantic Title LstName LstName Location Time Length 3 6 8 2 4 4 2 1 2 1 Case Upper Upper Upper lower Upper Upper lower lower Table 1: Sample phrase and its representation in multiple feature values for ten tokens.