waashk/dblp huggingface dataset:
automatic refinement of a pos tagger using a reliable parser and plain text corpora. this paper proposes a new unsupervised learning method for obtaining english part-of-speech (pos) disambiguation rules which would improve the accuracy of a pos tagger. this method has been implemented in the experimental system apras (automatic pos rule acquisition system), which extracts pos disambiguation rules from plain text corpora by utilizing different types of coded linguistic knowledge, i.e., pos tagging rules and syntactic parsing rules, which are already stored in a fully implemented mt system.in our experiment, the obtained rules were applied to 1.7% of the sentences in a non-training corpus. for this group of sentences, 78.4% of the changes made in tagging results were an improvement. we also saw a 15.5% improvement in tagging and parsing speed and an 8.0% increase of parsable sentences.
inherited feature-based similarity measure based on large semantic hierarchy and large text corpus. we describe a similarity calculation model called ifsm (inherited feature similarity measure) between objects (words/concepts) based on their common and distinctive features. we propose an implementation method for obtaining features based on abstracted triples extracted from a large text corpus utilizing taxonomical knowledge. this model represents an integration of traditional methods, i.e., relation based similarity measure and distribution based similarity measure. an experiment, using our new concept abstraction method which we call the flat probability grouping method, over 80,000 surface triples, shows that the abstraction level of 3000 is a good basis for feature description.
extracting important sentences with support vector machines. extracting sentences that contain important information from a document is a form of text summarization. the technique is the key to the automatic generation of summaries similar to those written by humans. to achieve such extraction, it is important to be able to integrate heterogeneous pieces of information. one approach, parameter tuning by machine learning, has been attracting a lot of attention. this paper proposes a method of sentence extraction based on support vector machines (svms). to confirm the method's performance, we conduct experiments that compare our method to three existing methods. results on the text summarization challenge (tsc) corpus show that our method offers the highest accuracy. moreover, we clarify the different features effective for extracting different document genres.
toward a redefinition of yes/no questions. while both theoretical and empirical studies of question-answering have revealed the inadequacy of traditional definitions of yes-no questions (ynqs), little progress has been made toward a more satisfactory redefinition. this paper reviews the limitations of several proposed revisions. it proposes a new definition of ynqs based upon research on a type of conversational implicature, termed here scalar implicature, that helps define appropriate responses to ynqs. by representing ynqs as scalar queries it is possible to support a wider variety of system and user responses in a principled way.
a measure of term representativeness based on the number of co-occurring salient words. we propose a novel measure of the representativeness (i.e., indicativeness or topic specificity) of a term in a given corpus. the measure embodies the idea that the distribution of words co-occurring with a representative term should be biased according to the word distribution in the whole corpus. the bias of the word distribution in the co-occurring words is defined as the number of distinct words whose occurrences are saliently biased in the co-occurring words. the saliency of a word is defined by a threshold probability that can be automatically defined using the whole corpus. comparative evaluation clarified that the measure is clearly superior to conventional measures in finding topic-specific words in the newspaper archives of different sizes.
an efficient treatment of japanese verb inflection for morphological analysis. because of its simple appearance, japanese verb inflection has never been treated seriously. in this paper we reconsider traditional lexical treatments of japanese verb inflection, and propose a new treatment of verb inflection which uses newly devised segmenting units. we show that our proposed treatment minimizes the number of lexical entries and avoids useless segmentation. it requires 20 to 40% less chart parsing computation and it is also suitable for error correction in optical character readers.
analysis of japanese compound nouns by direct text scanning. this paper aims to analyze word dependency structure in compound nouns appearing in japanese newspaper articles. the analysis is a difficult problem because such compound nouns can be quite long, have no word boundaries between contained nouns, and often contain unregistered words such as abbreviations. the non-segmentation property and unregistered words cause initial segmentation errors which result in erroneous analysis. this paper presents a corpus-based approach which scans a corpus with a set of pattern matchers and gathers cooccurrence examples to analyze compound nouns. it employs boot-strapping search to cope with unregistered words: if an unregistered word is found in the process of searching the examples, it is recorded and invokes additional searches to gather the examples containing it. this makes it possible to correct initial oversegmentation errors, and leads to higher accuracy. the accuracy of the method is evaluated using the compound nouns of length 5, 6, 7, and 8. a baseline is also introduced and compared.
a method of measuring term representativeness - baseline method using co-occurrence distribution. this paper introduces a scheme, which we call the baseline method, to define a measure of term representativeness and measures defined by using the scheme. the representativeness of a term is measured by a normalized characteristic value defined for a set of all documents that contain the term. normalization is done by comparing the original characteristic value with the characteristic value defined for a randomly chosen document set of the same size. the latter value is estimated by a baseline function obtained by random sampling and logarithmic linear approximation. we found that the distance between the word distribution in a document set and the word distribution in a whole corpus is an effective characteristic value to use for the baseline method. measures defined by the baseline method have several advantages including that they can be used to compare the representativeness of two terms with very different frequencies, and that they have well-defined threshold values of being representative. in addition, the baseline function for a corpus is robust against differences in corpora; that is, it can be used for normalization in a different corpus that has a different size or is in a different domain.
drawing pictures with natural language and direct manipulation. a multimodal user interface allows users to communicate with computers using multiple modalities, such as a mouse, a keyboard or voice, in various combined ways. this paper discusses a multimodal drawing tool, whereby the user can use a mouse, a keyboard and voice effectively. also, it describes an interpretation method, by which the system integrates voice inputs and pointing inputs using context.
building a large knowledge base for a natural language system. a sophisticated natural language system requires a large knowledge base. a methodology is described for constructing one in a principled way. facts are selected for the knowledge base by determining what facts are linguistically presupposed by a text in the domain of interest. the facts are sorted into clusters, and within each cluster they are organized according to their logical dependencies. finally, the facts are encoded as predicate calculus axioms.
natural language access to structured text. this paper discusses the problem of providing natural language access to textual material. we are developing a system that relates a request in english to specific passages in a document on the basis of correspondences between the logical representations of the information in the request and in the passages. in addition, we are developing procedures for automatically generating logical representations of text passages, directly from the text, by means of an analysis of the coherence structure of the passages.
a multilayered approach to the handling of word formation. the treatment of word formations has until recently been a neglected topic in natural language ai research. this paper proposes a multilayered approach to word formation which treats derivatives and compounds on several different levels of processing within a natural language dialogue system. analysis and generation strategies being developed for the dialogue system ham-ans are described. identification of word formations, semantic interpretation, and evaluation in the context of a dialogue are the main levels of analysis on which the system successively attempts to infer the implicit relations between word formation components. generation of word formations is viewed as a process comparable to the generation of elliptical utterances.
translating into free word order languages. in this paper, i discuss machine translation of english text into a relatively "free" word order language, specifically turkish. i present algorithms that use contextual information to determine what the topic and the focus of each sentence should be, in order to generate the contextually appropriate word orders in the target language.
webdiplomat: a web-based interactive machine translation system. we have implemented an interactive, web-based, chat-style machine translation system, supporting speech recognition and synthesis, local- or third-party correction of speech recognition and machine translation output, and online learning. the underlying client-server architecture, implemented in javatm, provides remote, distributed computation for the translation and speech subsystems. we further describe our web-based user interfaces, which can easily produce different useful configurations.
towards a more careful evaluation of broad coverage parsing systems. since treebanks have become available to researchers a wide variety of techniques has been used to make broad coverage parsing systems. this makes quantitative evaluation very important, but the current evaluation methods have a number of drawbacks such as arbitrary choices in the treebank and the difficulty in measuring statistical significance. we suggest a more detailed method for testing a parsing system using constituent boundaries, with a number of measures that give more information than current measures, and evaluate the quality of the test. we also show that statistical significance cannot be calculated in a straightforward way, and suggest a calculation method for the case of bracket recall.
varying cardinality in metonymic extensions to nouns. meaning shifting phenomena such as metonymy have recently attracted increasing interest of researchers. though these phenomena have been addressed by plenty of computational methods, the impacts of cardinalities of metonymically related items have been widely ignored in all of them. motivated by this lack of analysis, we have developed a method for representing expectations and knowledge about the cardinalities of metonymically related entities and for exploiting this information to build logical forms expressing metonymic relations, the entities related, and their cardinalities. the representation of lexically motivated knowledge is realized as an enhancement to pustejovsky's generative lexicon, and the process of building logical forms takes into account overwriting of default information and mismatch of cardinality requirements. our method enables a precise attachment of sentence complements, and it supports reference resolution in the context of metonymic expressions.
best analysis selection in inflectional languages. ambiguity is the fundamental property of natural language. perhaps, the most burdensome case of ambiguity manifests itself on the syntactic level of analysis. in order to face up to the high number of obtained derivation trees, this paper describes several techniques for evaluation of the figures of merit, which define a sort order on parsing trees. the presented methods are based on language specific features of synthetical languages and they improve the results of simple stochastic approaches.
learning the space of word meanings for information retrieval systems. several methods to represent meanings of words have been proposed. however. they are not useful for information retrieval systems because they cannot deal with the entities which cannot be universally represented by symbols.in this paper, we propose a notion of semantic space. semantic space is an euclidean space where words and entities are put. a word is one point in the space. the meanings of the word are represented as the space configuration around the word. the entities that cannot be represented by symbols can be identified in the space by the location the entity should be settled in. we also give a learning mechanism for the space. we prove the effectiveness of the proposed method by an experiment on information retrieval for the study of japanese literature.
presuppositions as beliefs. most theories of presupposition implicitly assume that presuppositions are facts, and that all agents involved in a discourse share belief in the presuppositions that it generates. these unrealistic assumptions can be eliminated if each presupposition is treated as the belief of an agent. however, it is not enough to consider only the beliefs of the speaker; we show that the beliefs of other agents are often involved. we describe a new model, including an improved definition of presupposition, that treats presuppositions as beliefs and considers the beliefs of all agents involved in the discourse. we show that treating presuppositions as beliefs makes it possible to explain phenomena that cannot be explained otherwise.
pause as a phrase demarcator for speech and language processing. in spontaneous speech understanding a sophisticated integration of speech recognition and language processing is especially crucial. however, the two modules are traditionally designed independently, with independent linguistic rules. in japanese speech recognition the bunsetsu phrase is the basic processing unit and in language processing the sentence is the basic unit. this difference has made it impractical to use a unique set of linguistic rules for both types of processing. further, spontaneous speech contains unexpected utterances other than well formed sentences, while linguistic rules for both speech and language processing expect well-formed sentences. they therefore fail to process everyday spoken language. to bridge the gap between speech and language processing, we propose that pauses be treated as phrase demarcators and that the interpausal phrase be the basic common processing unit. and to treat the linguistic phenomena of spoken language properly, we survey relevant features in spontaneous speech data. we then examine the effect of integrating pausal and spontaneous speech phenomena into syntactic rules for speech recognition, using 118 sentences. our experiments show that incorporating pausal phenomena as purely syntactic constraints degrades recognition accuracy considerably, while the additional degradation if some further spontaneous speech features are also incorporated.
construction of corpus-based syntactic rules for accurate speech recognition. this paper describes the syntactic rules which are applied in the japanese speech recognition module of a speech-to-speech translation system. japanese is considered to be a free word/phrase order language. since syntactic rules are applied as constraints to reduce the search space in speech recognition, applying rules which take into account all possible phrase orders can have almost the same effect as using no constraints. instead, we take into consideration the recognition weaknesses of certain syntactic categories and treat them precisely, so that a minimal number of rules can work most effectively. in this paper we first examine which syntactic categories are easily misrecognized. second, we consult our dialogue corpus, in order to provide the rules with great generality. based on both studies, we refine the rules. finally, we verify the validity of the refinement through speech recognition experiments.
a rule induction approach to modeling regional pronunciation variation. this paper describes the use of rule induction techniques for the automatic extraction of phonemic knowledge and rules from pairs of pronunciation lexica. this extracted knowledge allows the adaptation of speech processing systems to regional variants of a language. as a case study, we apply the approach to northern dutch and flemish (the variant of dutch spoken in flanders, a part of belgium), based on celex and fonilex, pronunciation lexica for northern dutch and flemish, respectively. in our study, we compare two rule induction techniques, transformation-based error-driven learning (tbedl) (brill, 1995) and c5.0 (quinlan, 1993), and evaluate the extracted knowledge quantitatively (accuracy) and qualitatively (linguistic relevance of the rules). we conclude that, whereas classification-based rule induction with c5.0 is more accurate, the transformation rules learned with tbedl can be more easily interpreted.
using knowledge to facilitate factoid answer pinpointing. in order to answer factoid questions, the webclopedia qa system employs a range of knowledge resources. these include a qa typology with answer patterns, wordnet, information about typical numerical answer ranges, and semantic relations identified by a robust parser, to filter out likely-looking but wrong candidate answers. this paper describes the knowledge resources and their impact on system performance.
dealing with conjunctions in a machine translation environment. the paper presents an algorithm, written in prolog, for processing english sentences which contain either gapping, right node raising (rnr) or reduced conjunction (rc). the dcg (definite clause grammar) formalism (pereira & warren 80) is adopted. the algorithm is highly efficient and capable of processing a full range of coordinate constructions containing any number of coordinate conjunctions ('and', 'or', and 'but'). the algorithm is part of an english-chinese machine translation system which is in the course of construction.
planning argumentative texts. this paper presents proverb a text planner for argumentative texts. proverb's main feature is that it combines global hierarchical planning and unplanned organization of text with respect to local derivation relations in a complementary way. the former splits the task of presenting a particular proof into subtasks of presenting subproofs. the latter simulates how the next intermediate conclusion to be presented is chosen under the guidance of the local focus.
a chinese corpus for linguistic research. this is a project note on the first stage of the construction of a comprehensive corpus of both modern and classical chinese. the corpus is built with the dual aim of serving as the central database for chinese language processing and for supporting in-depth linguistic research in mandarin chinese.
segmentation standard for chinese natural language processing. this paper proposes a segmentation standard for chinese natural language processing. the standard is proposed to achieve linguistic felicity, computational feasibility, and data uniformity. linguistic felicity is maintained by defining a segmentation unit to be equivalent to the theoretical definition of word, and by providing a set of segmentation principles that are equivalent to a functional definition of a word. computational feasibility is ensured by the fact that the above functional definitions are procedural in nature and can be converted to segmentation algorithms, as well as by the implementable heuristic guidelines which deal with specific linguistic categories. data uniformity is achieved by stratification of the standard itself and by defining a standard lexicon as part of the segmentation standard.
parsing in parallel. the paper is a description of a parallel model for natural language parsing, and a design for its implementation on the hypercube multiprocessor. the parallel model is based on the semantic definite clause grammar formation and integrates syntax and semantics through the communication of processes. the main processess, of which there are six, contain either purely syntactic or purely semantic information, giving the advantage of simple, transparent algorithms dedicated to only one aspect of parsing. communication between processes is used to impose semantic constraints on the syntactic processes.
disambiguation of morphological analysis in bantu languages. the paper describes problems in disambiguating the morphological analysis of bantu languages by using swahili as a test language. the main factors of ambiguity in this language group can be traced to the noun class structure on one hand and to the bi-directional word-formation on the other. in analyzing word-forms, the system applied utilizes swatwol, a morphological parsing program based on two-level formalism. disambiguation is carried out with the latest version (april 1996) of the constraint grammar parser (cgp). statistics on ambiguity are provided. solutions for resolving different types of ambiguity are presented and they are demonstrated by examples from corpus text. finally, statistics on the performance of the disambiguator are presented.
reversible resolution with an application to paraphrasing. this paper describes a reversible resolution method based on proof procedures. a modotonic semantics is given for a subset of the core language engine (cle)'s quasi-logical form (qlf) [ac92] which defines a relation (cat) realised by a series of declarative statements describing possible resolutions for terms. the application of paraphrasing is used to demonstrate the method in a useful function in a natural language processing environment.
layout and language: integrating spatial and linguistic knowledge for layout understanding tasks. complex documents stored in a flat or partially marked up file format require layout sensitive preprocessing before any natural language processing can be carried out on their textual content. contemporary technology for the discovery of basic textual units is based on either spatial or other content insensitive methods. however, there are many cases where knowledge of both the language and layout is required in order to establish the boundaries of the basic textual blocks. this paper describes a number of these cases and proposes the application of a general method combining knowledge about language with knowledge about the spatial arrangement of text. we claim that the comprehensive understanding of layout can only be achieved through the exploitation of layout knowledge and language knowledge in an inter-dependent manner.
on trying to do things with words: another plan-based approach to speech act interpretation. usual plan-based approaches to speech act interpretation require that performing a speech act implies its success. these approaches are thus useless for describing failing illocutionary or perlocutionary acts. we propose an alternative plan-based view of speech acts centered around the notion of trying to do - as opposed to actually doing - an action. this approach is contrasted with that of perrault which aims to overcome similar problems.