waashk/dblp huggingface dataset:
estimating upper and lower bounds on the performance of word-sense disambiguation programs. we have recently reported on two new word-sense disambiguation systems, one trained on bilingual material (the canadian hansards) and the other trained on monolingual material (roget's thesaurus and grolier's encyclopedia). after using both the monolingual and bilingual classifiers for a few months, we have convinced ourselves that the performance is remarkably good. nevertheless, we would really like to be able to make a stronger statement, and therefore, we decided to try to develop some more objective evaluation measures. although there has been a fair amount of literature on sense-disambiguation, the literature does not offer much guidance in how we might establish the success or failure of a proposed solution such as the two systems mentioned in the previous paragraph. many papers avoid quantitative evaluations altogether, because it is so difficult to come up with credible estimates of performance.this paper will attempt to establish upper and lower bounds on the level of performance that can be expected in an evaluation. an estimate of the lower bound of 75% (averaged over ambiguous types) is obtained by measuring the performance produced by a baseline system that ignores context and simply assigns the most likely sense in all cases. an estimate of the upper bound is obtained by assuming that our ability to measure performance is largely limited by our ability obtain reliable judgments from human informants. not surprisingly, the upper bound is very dependent on the instructions given to the judges. jorgensen, for example, suspected that lexicographers tend to depend too much on judgments by a single informant and found considerable variation over judgments (only 68% agreement), as she had suspected. in our own experiments, we have set out to find word-sense disambiguation tasks where the judges can agree often enough so that we could show that they were outperforming the baseline system. under quite different conditions, we have found 96.8% agreement over judges.
scalable inference and training of context-rich syntactic translation models. statistical mt has made great progress in the last few years, but current translation models are weak on re-ordering and target language fluency. syntactic approaches seek to remedy these problems. in this paper, we take the framework for acquiring multi-level syntactic translation rules of (galley et al., 2004) from aligned tree-string pairs, and present two main extensions of their approach: first, instead of merely computing a single derivation that minimally explains a sentence pair, we construct a large number of derivations that include contextually richer rules, and account for multiple interpretations of unaligned words. second, we propose probability estimates and a training procedure for weighting these rules. we contrast different approaches on real examples, show that our estimates based on multiple derivations favor phrasal re-orderings that are linguistically better motivated, and establish that our larger rules provide a 3.63 bleu point increase over minimal rules.
discourse segmentation of multi-party conversation. we present a domain-independent topic segmentation algorithm for multi-party speech. our feature-based algorithm combines knowledge about content using a text-based algorithm as a feature and about form using linguistic and acoustic cues about topic shifts extracted from speech. this segmentation algorithm uses automatically induced decision rules to combine the different features. the embedded text-based algorithm builds on lexical cohesion and has performance comparable to state-of-the-art algorithms based on lexical information. a significant error reduction is obtained by combining the two knowledge sources.
identifying agreement and disagreement in conversational speech: use of bayesian networks to model pragmatic dependencies. we describe a statistical approach for modeling agreements and disagreements in conversational interaction. our approach first identifies adjacency pairs using maximum entropy ranking based on a set of lexical, durational, and structural features that look both forward and backward in the discourse. we then classify utterances as agreement or disagreement using these adjacency pairs and features that represent various pragmatic influences of previous agreement or disagreement on the current utterance. our approach achieves 86.9% accuracy, a 4.9% increase over previous work.
a synopsis of learning to recognize names across languages. the development of natural language processing (nlp) systems that perform machine translation (mt) and information retrieval (ir) has highlighted the need for the automatic recognition of proper names. while various name recognizers have been developed, they suffer from being too limited; some only recognize one name class, and all are language specific. this work develops an approach to multilingual name recognition that uses machine learning and a portable framework to simplify the porting task by maximizing reuse and automation.
semantic-head based resolution of scopal ambiguities. we introduce an algorithm for scope resolution in underspecified semantic representations. scope preferences are suggested on the basis of semantic argument structure. the major novelty of this approach is that, while maintaining an (scopally) underspecified semantic representation, we at the same time suggest a resolution possibility. the algorithm has been implemented and tested in a large-scale system and fared quite well: 28% of the utterances were ambiguous, 80% of these were correctly interpreted, leaving errors in only 5.7% of the utterance set.
machine-learned contexts for linguistic operations in german sentence realization. we show that it is possible to learn the contexts for linguistic operations which map a semantic representation to a surface syntactic tree in sentence realization with high accuracy. we cast the problem of learning the contexts for the linguistic operations as classification tasks, and apply straightforward machine learning techniques, such as decision tree learning. the training data consist of linguistic features extracted from syntactic and semantic representations produced by a linguistic analysis system. the target features are extracted from links to surface syntax trees. our evidence consists of four examples from the german sentence realization system code-named amalgam: case assignment, assignment of verb position features, extraposition, and syntactic aggregation.
exploring asymmetric clustering for statistical language modeling. the n-gram model is a stochastic model, which predicts the next word (predicted word) given the previous words (conditional words) in a word sequence. the cluster n-gram model is a variant of the n-gram model in which similar words are classified in the same cluster. it has been demonstrated that using different clusters for predicted and conditional words leads to cluster models that are superior to classical cluster models which use the same clusters for both words. this is the basis of the asymmetric cluster model (acm) discussed in our study. in this paper, we first present a formal definition of the acm. we then describe in detail the methodology of constructing the acm. the effectiveness of the acm is evaluated on a realistic application, namely japanese kana-kanji conversion. experimental results show substantial improvements of the acm in comparison with classical cluster models and word n-gram models at the same model size. our analysis shows that the high-performance of the acm lies in the asymmetry of the model.
distribution-based pruning of backoff language models. we propose a distribution-based pruning of n-gram backoff language models. instead of the conventional approach of pruning n-grams that are infrequent in training data, we prune n-grams that are likely to be infrequent in a new document. our method is based on the n-gram distribution i.e. the probability that an n-gram occurs in a new document. experimental results show that our method performed 7--9% (word perplexity reduction) better than conventional cutoff methods.
improved source-channel models for chinese word segmentation. this paper presents a chinese word segmentation system that uses improved source-channel models of chinese sentence generation. chinese words are defined as one of the following four types: lexicon words, morphologically derived words, factoids, and named entities. our system provides a unified approach to the four fundamental features of word-level chinese language processing: (1) word segmentation, (2) morphological analysis, (3) factoid detection, and (4) named entity recognition. the performance of the system is evaluated on a manually annotated test set, and is also compared with several state-of-the-art systems, taking into account the fact that the definition of chinese words often varies from system to system.
unsupervised learning of dependency structure for language modeling. this paper presents a dependency language model (dlm) that captures linguistic constraints via a dependency structure, i.e., a set of probabilistic dependencies that express the relations between headwords of each phrase in a sentence by an acyclic, planar, undirected graph. our contributions are three-fold. first, we incorporate the dependency structure into an n-gram language model to capture long distance word dependency. second, we present an unsupervised learning method that discovers the dependency structure of a sentence using a bootstrapping procedure. finally, we evaluate the proposed models on a realistic application (japanese kana-kanji conversion). experiments show that the best dlm achieves an 11.3% error rate reduction over the word trigram model.
approximation lasso methods for language modeling. lasso is a regularization method for parameter estimation in linear models. it optimizes the model parameters with respect to a loss function subject to model complexities. this paper explores the use of lasso for statistical language modeling for text input. owing to the very large number of parameters, directly optimizing the penalized lasso loss function is impossible. therefore, we investigate two approximation methods, the boosted lasso (blasso) and the forward stagewise linear regression (fslr). both methods, when used with the exponential loss function, bear strong resemblance to the boosting algorithm which has been used as a discriminative training method for language modeling. evaluations on the task of japanese text input show that blasso is able to produce the best approximation to the lasso solution, and leads to a significant improvement, in terms of character error rate, over boosting and the traditional maximum likelihood estimation.
improving language model size reduction using better pruning criteria. reducing language model (lm) size is a critical issue when applying a lm to realistic applications which have memory constraints. in this paper, three measures are studied for the purpose of lm pruning. they are probability, rank, and entropy. we evaluated the performance of the three pruning criteria in a real application of chinese text input in terms of character error rate (cer). we first present an empirical comparison, showing that rank performs the best in most cases. we also show that the high-performance of rank lies in its strong correlation with error rate. we then present a novel method of combining two criteria in model pruning. experimental results show that the combined criterion consistently leads to smaller models than the models pruned using either of the criteria separately, at the same cer.
towards a self-extending parser. this paper discusses an approach to incremental learning in natural language processing. the technique of projecting and integrating semantic constraints to learn word definitions is analyzed as implemented in the politics system. extensions and improvements of this technique are developed. the problem of generalizing existing word meanings and understanding metaphorical uses of words is addressed in terms of semantic constraint integration.
refined lexikon models for statistical machine translation using a maximum entropy approach. typically, the lexicon models used in statistical machine translation systems do not include any kind of linguistic or contextual information, which often leads to problems in performing a correct word sense disambiguation. one way to deal with this problem within the statistical framework is to use maximum entropy methods. in this paper, we present how to use this type of information within a statistical machine translation system. we show that it is possible to significantly decrease training and test corpus perplexity of the translation models. in addition, we perform a rescoring of n-best lists using our maximum entropy model and thereby yield an improvement in translation quality. experimental results are presented on the so-called "verbmobil task".
unifying parallels. i show that the equational treatment of ellipsis proposed in (dalrymple et al., 1991) can further be viewed as modeling the effect of parallelism on semantic interpretation. i illustrate this claim by showing that the account straightforwardly extends to a general treatment of sloppy identity on the one hand, and to deaccented foci on the other. i also briefly discuss the results obtained in a prototype implementation.
generating minimal definite descriptions. the incremental algorithm introduced in (dale and reiter, 1995) for producing distinguishing descriptions does not always generate a minimal description. in this paper, i show that when generalised to sets of individuals and disjunctive properties, this approach might generate unnecessarily long and ambiguous and/or epistemically redundant descriptions. i then present an alternative, constraint-based algorithm and show that it builds on existing related algorithms in that (i) it produces minimal descriptions for sets of individuals using positive, negative and disjunctive properties, (ii) it straightforwardly generalises to n-ary relations and (iii) it is integrated with surface realisation.
efficient parsing for french. parsing with categorial grammars often leads to problems such as proliferating lexical ambiguity, spurious parses and overgeneration. this paper presents a parser for french developed on an unification based categorial grammar (fg) which avoids these problems. this parser is a bottom-up chart parser augmented with a heuristic eliminating spurious parses. the unicity and completeness of parsing are proved.
higher-order coloured unification and natural language semantics. in this paper, we show that higher-order coloured unification - a form of unification developed for automated theorem proving - provides a general theory for modeling the interface between the interpretation process and other sources of linguistic, non semantic information. in particular, it provides the general theory for the primary occurrence restriction which (dalrymple et al., 1991)'s analysis called for.
coreference handling in xmg. we claim that existing specification languages for tree based grammars fail to adequately support identifier managment. we then show that xmg (extensible meta-grammar) provides a sophisticated treatment of identifiers which is effective in supporting a linguist-friendly grammar design.
generating with a grammar based on tree descriptions: a constraint-based approach. while the generative view of language processing builds bigger units out of smaller ones by means of rewriting steps, the axiomatic view eliminates invalid linguistic structures out of a set of possible structures by means of well formedness principles. we present a generator based on the axiomatic view and argue that when combined with a tag-like grammar and a flat semantics, this axiomatic view permits avoiding drawbacks known to hold either of top-down or of bottom-up generators.
acquiring receptive morphology: a connectionist model. this paper describes a modular connectionist model of the acquisition of receptive inflectional morphology. the model takes inputs in the form of phones one at a time and outputs the associated roots and inflections. simulations using artificial language stimuli demonstrate the capacity of the model to learn suffixation, prefixation, infixation, circumfixation, mutation, template, and deletion rules. separate network modules responsible for syllables enable to the network to learn simple reduplication rules as well. the model also embodies constraints against association-line crossing.
conceptual coherence in the generation of referring expressions. one of the challenges in the automatic generation of referring expressions is to identify a set of domain entities coherently, that is, from the same conceptual perspective. we describe and evaluate an algorithm that generates a conceptually coherent description of a target set. the design of the algorithm is motivated by the results of psycholinguistic experiments.
a geometric view on bilingual lexicon extraction from comparable corpora. we present a geometric view on bilingual lexicon extraction from comparable corpora, which allows to re-interpret the methods proposed so far and identify unresolved problems. this motivates three new methods that aim at solving these problems. empirical evaluation shows the strengths and weaknesses of these methods, as well as a significant gain in the accuracy of extracted lexicons.
processing broadcast audio for information access. this paper addresses recent progress in speaker-independent, large vocabulary, continuous speech recognition, which has opened up a wide range of near and mid-term applications. one rapidly expanding application area is the processing of broadcast audio for information access. at limsi, broadcast news transcription systems have been developed for english, french, german, mandarin and portuguese, and systems for other languages are under development. audio indexation must take into account the specificities of audio data, such as needing to deal with the continuous data stream and an imperfect word transcription. some near-term applications areas are audio data mining, selective dissemination of information and media monitoring.
growing semantic grammars. a critical path in the development of natural language understanding (nlu) modules lies in the difficulty of defining a mapping from words to semantics: usually it takes in the order of years of highly-skilled labor to develop a semantic mapping, e.g., in the form of a semantic grammar, that is comprehensive enough for a given domain. yet, due to the very nature of human language, such mapping invariably fail to achieve full coverage on unseen data. acknowledging the impossibility of stating a priori all the surface forms by which a concept can be expressed, we present gsg: an empathic computer system for the rapid deployment of nlu front-ends and their dynamic customization by non-expert end-users. given a new domain for which an nlu front-end is to be developed, two stages are involved. in the authoring stage, gsg aids the developer in the construction of a simple domain model and a kernel analysis grammar. then, in the run-time stage, gsg provides the end-user with an interactive environment in which the kernel grammar is dynamically extended. three learning methods are employed in the acquisition of semantic mappings from unseen data: (i) parser predictions, (ii) hidden understanding model, and (iii) end-user paraphrases. a baseline version of gsg has been implemented and preliminary experiments show promising results.
priority union and generalization in discourse grammars. we describe an implementation in carpenter's typed feature formalism, ale, of a discourse grammar of the kind proposed by scha, polanyi, et al. we examine their method for resolving parallelism-dependent anaphora and show that there is a coherent feature-structural rendition of this type of grammar which uses the operations of priority union and generalization. we describe an augmentation of the ale system to encompass these operations and we show that an appropriate choice of definition for priority union gives the desired multiple output for examples of vp-ellipsis which exhibit a strict/sloppy ambiguity.
dynamic programming for parsing and estimation of stochastic unification-based grammars. stochastic unification-based grammars (subgs) define exponential distributions over the parses generated by a unification-based grammar (ubg). existing algorithms for parsing and estimation require the enumeration of all of the parses of a string in order to determine the most likely one, or in order to calculate the statistics needed to estimate a grammar from a training corpus. this paper describes a graph-based dynamic programming algorithm for calculating these statistics from the packed ubg parse representations of maxwell and kaplan (1995) which does not require enumerating all parses. like many graphical algorithms, the dynamic programming algorithm's complexity is worst-case exponential, but is often polynomial. the key observation is that by using maxwell and kaplan packed representations, the required statistics can be rewritten as either the max or the sum of a product of functions. this is exactly the kind of problem which can be solved by dynamic programming over graphical models.
xml-based data preparation for robust deep parsing. we describe the use of xml tokenisation, tagging and mark-up tools to prepare a corpus for parsing. our techniques are generally applicable but here we focus on parsing medline abstracts with the anlt wide-coverage grammar. hand-crafted grammars inevitably lack coverage but many coverage failures are due to inadequacies of their lexicons. we describe a method of gaining a degree of robustness by interfacing pos tag information with the existing lexicon. we also show that xml tools provide a sophisticated approach to pre-processing, helping to ameliorate the 'messiness' in real language data and improve parse performance.
on interpreting f-structures as udrss. we describe a method for interpreting abstract flat syntactic representations, lfg f-structures, as underspecified semantic representations, here underspecified discourse representation structures (udrss). the method establishes a one-to-one correspondence between subsets of the lfg and udrs formalisms. it provides a model theoretic interpretation and an inferential component which operates directly on underspecified representations for f-structures through the translation images of f-structures as udrss.
segment-based hidden markov models for information extraction. hidden markov models (hmms) are powerful statistical models that have found successful applications in information extraction (ie). in current approaches to applying hmms to ie, an hmm is used to model text at the document level. this modelling might cause undesired redundancy in extraction in the sense that more than one filler is identified and extracted. we propose to use hmms to model text at the segment level, in which the extraction process consists of two steps: a segment retrieval step followed by an extraction step. in order to retrieve extraction-relevant segments from documents, we introduce a method to use hmms to model and retrieve segments. our experimental results show that the resulting segment hmm ie system not only achieves near zero extraction redundancy, but also has better overall extraction performance than traditional document hmm ie systems.