waashk/dblp huggingface dataset:
a flexible distributed architecture for nlp system development and use. we describe a distributed, modular architecture for platform independent natural language systems. it features automatic interface generation and self-organization. adaptive (and non-adaptive) voting mechanisms are used for integrating discrete modules. the architecture is suitable for rapid prototyping and product delivery.
analysis system of speech acts and discourse structures using maximum entropy model. we propose a statistical dialogue analysis model to determine discourse structures as well as speech acts using maximum entropy model. the model can automatically acquire probabilistic discourse knowledge from a discourse tagged corpus to resolve ambiguities. we propose the idea of tagging discourse segment boundaries to represent the structural information of discourse. using this representation we can effectively combine speech act analysis and discourse structure analysis in one framework.
hybrid approaches to improvement of translation quality in web-based english-korean machine translation. the previous english-korean mt system that was the transfer-based mt system and applied to only written text enumerated a following brief list of the problems that had not seemed to be easy to solve in the near future: 1) processing of non-continuous idiomatic expressions 2) reduction of too many ambiguities in english syntactic analysis 3) robust processing for failed or illformed sentences 4) selecting correct word correspondence between several alternatives 5) generation of korean sentence style. the problems can be considered as factors that have influence on the translation quality of machine translation system. this paper describes the symbolic and statistical hybrid approaches to solutions of problems of the previous english-to-korean machine translation system in terms of the improvement of translation quality. the solutions are now successfully applied to the web-based english-korean machine translation system "fromto/ek" which has been developed from 1997.
techniques to incorporate the benefits of a hierarchy in a modified hidden markov model. this paper explores techniques to take advantage of the fundamental difference in structure between hidden markov models (hmm) and hierarchical hidden markov models (hhmm). the hhmm structure allows repeated parts of the model to be merged together. a merged model takes advantage of the recurring patterns within the hierarchy, and the clusters that exist in some sequences of observations, in order to increase the extraction accuracy. this paper also presents a new technique for reconstructing grammar rules automatically. this work builds on the idea of combining a phrase extraction method with hhmm to expose patterns within english text. the reconstruction is then used to simplify the complex structure of an hhmm the models discussed here are evaluated by applying them to natural language tasks based on conll-2004 and a sub-corpus of the lancaster treebank
analysis and synthesis of the distribution of consonants over languages: a complex network approach. cross-linguistic similarities are reflected by the speech sound systems of languages all over the world. in this work we try to model such similarities observed in the consonant inventories, through a complex bipartite network. we present a systematic study of some of the appealing features of these inventories with the help of the bipartite network. an important observation is that the occurrence of consonants follows a two regime power law distribution. we find that the consonant inventory size distribution together with the principle of preferential attachment are the main reasons behind the emergence of such a two regime behavior. in order to further support our explanation we present a synthesis model for this network based on the general theory of preferential attachment.
using machine-learning to assign function labels to parser output for spanish. data-driven grammatical function tag assignment has been studied for english using the penn-ii treebank data. in this paper we address the question of whether such methods can be applied successfully to other languages and treebank resources. in addition to tag assignment accuracy and f-scores we also present results of a task-based evaluation. we use three machine-learning methods to assign cast3lb function tags to sentences parsed with bikel's parser trained on the cast3lb treebank. the best performing method, svm, achieves an f-score of 86.87% on gold-standard trees and 66.67% on parser output - a statistically significant improvement of 6.74% over the baseline. in a task-based evaluation we generate lfg functional-structures from the function-tag-enriched trees. on this task we achive an f-score of 75.67%, a statistically significant 3.4% improvement over the baseline.
tracking initiative in collaborative dialogue interactions. in this paper, we argue for the need to distinguish between task and dialogue initiatives, and present a model for tracking shifts in both types of initiatives in dialogue interactions. our model predicts the initiative holders in the next dialogue turn based on the current initiative holders and the effect that observed cues have on changing them. our evaluation across various corpora shows that the use of cues consistently improves the accuracy in the system's prediction of task and dialogue initiative holders by 2-4 and 8-13 percentage points, respectively, thus illustrating the generality of our model.
response generation in collaborative negotiation. in collaborative planning activities, since the agents are autonomous and heterogenous, it is inevitable that conflicts arise in their beliefs during the planning process. in cases where such conflicts are relevant to the task at hand, the agents should engage in collaborative negotiation as an attempt to square away the discrepancies in their beliefs. this paper presents a computational strategy for detecting conflicts regarding proposed beliefs and for engaging in collaborative negotiation to resolve the conflicts that warrant resolution. our model is capable of selecting the most effective aspect to address in its pursuit of conflict resolution in cases where multiple conflicts arise, and of selecting appropriate evidence to justify the need for such modification. furthermore, by capturing the negotiation process in a recursive propose-evaluate-modify cycle of actions, our model can successfully handle embedded negotiation subdialogues.
dialogue management in vector-based call routing. this paper describes a domain independent, automatically trained call router which directs customer calls based on their response to an open-ended "how may i direct your call?" query. routing behavior is trained from a corpus of transcribed and hand-routed calls and then carried out using vector-based information retrieval techniques. based on the statistical discriminating power of the n-gram terms extracted from the caller's request, the caller is 1) routed to the appropriate destination, 2) transferred to a human operator, or 3) asked a disambiguation question. in the last case, the system dynamically generates queries tailored to the caller's request and the destinations with which it is consistent. our approach is domain independent and the training process is fully automatic. evaluations over a financial services call center handling hundreds of activities with dozens of destinations demonstrate a substantial improvement on existing systems by correctly routing 93.8% of the calls after punting 10.2% of the calls to a human operator.
constraint-based sentence compression: an integer programming approach. the ability to compress sentences while preserving their grammaticality and most of their meaning has recently received much attention. our work views sentence compression as an optimisation problem. we develop an integer programming formulation and infer globally optimal compressions in the face of linguistically motivated constraints. we show that such a formulation allows for relatively simple and knowledge-lean compression models that do not require parallel corpora or large-scale resources. the proposed approach yields results comparable and in some cases superior to state-of-the-art.
developing a flexible spoken dialog system using simulation. in this paper, we describe a new methodology to develop mixed-initiative spoken dialog systems, which is based on the extensive use of simulations to accelerate the development process. with the help of simulations, a system providing information about a database of nearly 1000 restaurants in the boston area has been developed. the simulator can produce thousands of unique dialogs which benefit not only dialog development but also provide data to train the speech recognizer and understanding components, in preparation for real user interactions. also described is a strategy for creating cooperative responses to user queries, incorporating an intelligent language generation capability that produces content-dependent verbal descriptions of listed items.
generating parallel multilingual lfg-tag grammars from a metagrammar. we introduce a metagrammar, which allows us to automatically generate, from a single and compact metagrammar hierarchy, parallel lexical functional grammars (lfg) and tree-adjoining grammars (tag) for french and for english: the grammar writer specifies in compact manner syntactic properties that are potentially framework-, and to some extent language-independent (such as subcategorization, valency alternations and realization of syntactic functions), from which grammars for several frameworks and languages are automatically generated offline.
on parsing strategies and closure. this paper proposes a welcom hypothesis: a computationally simple deviceis sufficient for processing natural language. traditionally it has been argued that processing natural language syntax requires very powerful machinery. many engineers have come to this rather grim conclusion; almost all working parsers are actually turing machines (tm). for example, woods believed that a parser should have tm complexity and specifically designed his augmented transition networks (atns) to be turing equivalent.(1) "it is well known (cf. [chomsky64]) that the strict context-free grammar model is not an adequate mechanism for characterizing the subtleties of natural languages." [woods70]if the problem is really as hard as it appears, then the only solution is to grin and bear it. our own position is that parsing acceptable sentences is simpler because there are constraints on human performance that drastically reduce the computational complexity. although woods correctly observes that <u>competence</u> models are very complex, this observation may not apply directly to a <u>performance</u> problem such as parsing.the claim is that performance limitations actually <u>reduce</u> parsing complexity. this suggests two interesting questions: (a) how is the performance model constrained so as to reduce its complexity, and (b) how can the constrained performance model naturally approximate competence idealizations?
stress assignment in letter to sound rules for speech synthesis. this paper will discuss how to determine word stress from spelling. stress assignment is a well-established weak point for many speech synthesizers because stress dependencies cannot be determined locally. it is impossible to determine the stress of a word by looking through a five or six character window, as many speech synthesizers do. well-known examples such as degr&aacute;de / d&egrave;grad&aacute;tion and t&eacute;legraph / tel&eacute;graphy demonstrate that stress dependencies can span over two and three syllables. this paper will present a principled framework for dealing with these long distance dependencies. stress assignment will be formulated in terms of waltz' style constraint propagation with four sources of constraints: (1) syllable weight, (2) part of speech, (3) morphology and (4) etymology. syllable weight is perhaps the most interesting, and will be the main focus of this paper. most of what follows has been implemented.
char_align: a program for aligning parallel texts at the character level. there have been a number of recent papers on aligning parallel texts at the sentence level, e.g., brown et al (1991), gale and church (to appear), isabelle (1992), kay and r&ouml;senschein (to appear), simard et al (1992), warwick-armstrong and russell (1990). on clean inputs, such as the canadian hansards, these methods have been very successful (at least 96% correct by sentence). unfortunately, if the input is noisy (due to ocr and/or unknown markup conventions), then these methods tend to break down because the noise can make it difficult to find paragraph boundaries, let alone sentences. this paper describes a new program, char_align, that aligns texts at the character level rather than at the sentence/paragraph level, based on the cognate approach proposed by simard et al.
word association norms, mutual information and lexicography. the term word association is used in a very particular sense in the psycholinguistic literature. (generally speaking, subjects respond quicker than normal to the word "nurse" if it follows a highly associated word such as "doctor.") we will extend the term to provide the basis for a statistical description of a variety of interesting linguistic phenomena, ranging from semantic relations of the doctor/nurse type (content word/content word) to lexico-syntactic co-occurrence constraints between verbs and prepositions (content word/function word). this paper will propose a new objective measure based on the information theoretic notion of mutual information, for estimating word association norms from computer readable corpora. (the standard method of obtaining word association norms, testing a few thousand subjects on a few hundred words, is both costly and unreliable.) the proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, making it possible to estimate norms for tens of thousands of words.
partially specified signatures: a vehicle for grammar modularity. this work provides the essential foundations for modular construction of (typed) unification grammars for natural languages. much of the information in such grammars is encoded in the signature, and hence the key is facilitating a modularized development of type signatures. we introduce a definition of signature modules and show how two modules combine. our definitions are motivated by the actual needs of grammar developers obtained through a careful examination of large scale grammars. we show that our definitions meet these needs by conforming to a detailed set of desiderata.
the wild thing. suppose you are on a mobile device with no keyboard (e.g., a cell or pda). how can you enter text quickly? t9? graffiti? this demo will show how language modeling can be used to speed up data entry, both in the mobile context, as well as the desk-top. the wild thing encourages users to use wildcards (*). a language model finds the k-best expansions. users quickly figure out when they can get away with wildcards. general purpose trigram language models are effective for the general case (unrestricted text), but there are important special cases like searching over popular web queries, where more restricted language models are even more effective.
natural language input to a computer-based glaucoma consultation system. a "front end" for a computer-based glaucoma consultation system is described. the system views a case as a description of a particular instance of a class of concepts called "structured objects" and builds up a representation of the instance from the sentences in the case. the information required by the consultation system is then extracted and passed on to the consultation system in the appropriately coded form. a core of syntactic, semantic and contextual rules which are applicable to all structured objects is being developed together with a representation of the structured object glaucoma-patient. there is also a facility for adding domain dependent syntax, abbreviations and defaults.
speech acts and rationality. this paper derives the basis of a theory of communication from a formal theory of rational interaction. the major result is a demonstration that illocutionary acts need not be primitive, and need not be recognized. as a test case. we derive searle's conditions on requesting from principles of rationality coupled with a gricean theory of imperatives. the theory is shown to distinguish insincere or nonserious imperatives from true requests. extensions to indirect speech acts, and ramifications for natural language systems are also briefly discussed.
memory-based learning of morphology with stochastic transducers. this paper discusses the supervised learning of morphology using stochastic transducers, trained using the expectation-maximization (em) algorithm. two approaches are presented: first, using the transducers directly to model the process, and secondly using them to define a similarity measure, related to the fisher kernel method (jaakkola and haussler, 1998), and then using a memory-based learning (mbl) technique. these are evaluated and compared on data sets from english, german, slovene and arabic.
performatives in a rationally based speech act theory. a crucially important adequacy test of any theory of speech acts is its ability to handle performatives. this paper provides a theory of performatives as a test case for our rationally based theory of illocutionary acts. we show why "i request you..." is a request, and "i lie to you that p" is self-defeating. the analysis supports and extends earlier work of theorists such as bach and harnish [1] and takes issue with recent claims by searle [10] that such performative-as-declarative analyses are doomed to failure.
parsing the wsj using ccg and log-linear models. this paper describes and evaluates log-linear parsing models for combinatory categorial grammar (ccg). a parallel implementation of the l-bfgs optimisation algorithm is described, which runs on a beowulf cluster allowing the complete penn treebank to be used for estimation. we also develop a new efficient parsing algorithm for ccg which maximises expected recall of dependencies. we compare models which use all ccg derivations, including non-standard derivations, with normal-form models. the performances of the two models are comparable and the results are competitive with existing wide-coverage ccg parsers.
scaling conditional random fields using error-correcting codes. conditional random fields (crfs) have been applied with considerable success to a number of natural language processing tasks. however, these tasks have mostly involved very small label sets. when deployed on tasks with larger label sets, the requirements for computational resources mean that training becomes intractable.this paper describes a method for training crfs on such tasks, using error correcting output codes (ecoc). a number of crfs are independently trained on the separate binary labelling tasks of distinguishing between a subset of the labels and its complement. during decoding, these models are combined to produce a predicted label sequence which is resilient to errors by individual models.error-correcting crf training is much less resource intensive and has a much faster training time than a standardly formulated crf, while decoding performance remains quite comparable. this allows us to scale crfs to previously impossible tasks, as demonstrated by our experiments with large label sets.
machine translation versus dictionary term translation - a comparison for english-japanese news article alignment. bilingual news article alignment methods based on multi-lingual information retrieval have been shown to be successful for the automatic production of so-called noisy-parallel corpora. in this paper we compare the use of machine translation (mt) to the commonly used dictionary term lookup (dtl) method for reuter news article alignment in english and japanese. the results show the trade-off between improved lexical disambiguation provided by machine translation and extended synonym choice provided by dictionary term lookup and indicate that mt is superior to dtl only at medium and low recall levels. at high recall levels dtl has superior precision.
models for sentence compression: a comparison across domains, training requirements and evaluation measures. sentence compression is the task of producing a summary at the sentence level. this paper focuses on three aspects of this task which have not received detailed treatment in the literature: training requirements, scalability, and automatic evaluation. we provide a novel comparison between a supervised constituent-based and an weakly supervised word-based compression algorithm and examine how these models port to different domains (written vs. spoken text). to achieve this, a human-authored compression corpus has been created and our study highlights potential problems with the automatically gathered compression corpora currently used. finally, we assess whether automatic evaluation measures can be used to determine compression quality.
the distributional inclusion hypotheses and lexical entailment. this paper suggests refinements for the distributional similarity hypothesis. our proposed hypotheses relate the distributional behavior of pairs of words to lexical entailment -- a tighter notion of semantic similarity that is required by many nlp applications. to automatically explore the validity of the defined hypotheses we developed an inclusion testing algorithm for characteristic features of two words, which incorporates corpus and web-based feature sampling to overcome data sparseness. the degree of hypotheses validity was then empirically tested and manually analyzed with respect to the word sense level. in addition, the above testing algorithm was exploited to improve lexical entailment acquisition.
an experiment in hybrid dictionary and statistical sentence alignment. the task of aligning sentences in parallel corpora of two languages has been well studied using pure statistical or linguistic models. we developed a linguistic method based on lexical matching with a bilingual dictionary and two statistical methods based on sentence length ratios and sentence offset probabilities. this paper seeks to further our knowledge of the alignment task by comparing the performance of the alignment models when used separately and together, i.e. as a hybrid system. our results show that for our english-japanese corpus of newspaper articles, the hybrid system using lexical matching and sentence length ratios outperforms the pure methods.
ranking algorithms for named entity extraction: boosting and the voted perceptron. this paper describes algorithms which rerank the top n hypotheses from a maximum-entropy tagger, the application being the recovery of named-entity boundaries in a corpus of web data. the first approach uses a boosting algorithm for ranking problems. the second approach uses the voted perceptron algorithm. both algorithms give comparable, significant improvements over the maximum-entropy baseline. the voted perceptron algorithm can be considerably more efficient to train, at some cost in computation on test examples.
a new statistical parser based on bigram lexical dependencies. this paper describes a new statistical parser which is based on probabilities of dependencies between head-words in the parse tree. standard bigram probability estimation techniques are extended to calculate probabilities of dependencies between pairs of words. tests using wall street journal data show that the method performs at least as well as spatter (magerman 95; jelinek et al. 94), which has the best published results for a statistical parser on this task. the simplicity of the approach means the model trains on 40,000 sentences in under 15 minutes. with a beam search strategy parsing speed can be improved to over 200 sentences a minute with negligible loss in accuracy.