waashk/dblp huggingface dataset:
hybrid methods for pos guessing of chinese unknown words. this paper describes a hybrid model that combines a rule-based model with two statistical models for the task of pos guessing of chinese unknown words. the rule-based model is sensitive to the type, length, and internal structure of unknown words, and the two statistical models utilize contextual information and the likelihood for a character to appear in a particular position of words of a particular length and pos category. by combining models that use different sources of information, the hybrid model achieves a precision of 89%, a significant improvement over the best result reported in previous studies, which was 69%.
paraphrasing using given and new information in a question-answer system. the design and implementation of a paraphrase component for a natural language question-answer system (co-op) is presented. a major point made is the role of given and new information in formulating a paraphrase that differs in a meaningful way from the user's question. a description is also given of the transformational grammar used by the paraphraser to generate questions.
an equivalent pseudoword solution to chinese word sense disambiguation. this paper presents a new approach based on equivalent pseudowords (eps) to tackle word sense disambiguation (wsd) in chinese language. eps are particular artificial ambiguous words, which can be used to realize unsupervised wsd. a bayesian classifier is implemented to test the efficacy of the ep solution on senseval-3 chinese test set. the performance is better than state-of-the-art results with an average f-measure of 0.80. the experiment verifies the value of ep for unsupervised wsd.
collocation translation acquisition using monolingual corpora. collocation translation is important for machine translation and many other nlp tasks. unlike previous methods using bilingual parallel corpora, this paper presents a new method for acquiring collocation translations by making use of monolingual corpora and linguistic knowledge. first, dependency triples are extracted from chinese and english corpora with dependency parsers. then, a dependency triple translation model is estimated using the em algorithm based on a dependency correspondence assumption. the generated triple translation model is used to extract collocation translations from two monolingual corpora. experiments show that our approach outperforms the existing monolingual corpus based methods in dependency triple translation and achieves promising results in collocation translation extraction.
using language resources in an intelligent tutoring system for french. this paper presents a project that investigates to what extent computational linguistic methods and tools used at geta for machine translation can be used to implement novel functionalities in intelligent computer assisted language learning. our intelligent tutoring system project is still in its early phases. the learner module is based on an empirical study of french as used by acadian elementary students living in new-brunswick, canada. additionally, we are studying the state of the art of systems using artificial intelligence techniques as well as nlp resources and/or methodologies for teaching language, especially for bilingual and minority groups.
statistical sense disambiguation with relatively small corpora using dictionary definitions. corpus-based sense disambiguation methods, like most other statistical nlp approaches, suffer from the problem of data sparseness. in this paper, we describe an approach which overcomes this problem using dictionary definitions. using the definition-based conceptual co-occurrence data collected from the relatively small brown corpus, our sense disambiguation system achieves an average accuracy comparable to human performance given the same contextual information.
tailoring lexical choice to the user's vocabulary in multimedia explanation generation. in this paper, we discuss the different strategies used in comet (coordinated multimedia explanation testbed) for selecting words with which the user is familiar. when pictures cannot be used to disambiguate a word or phrase, comet has four strategies for avoiding unknown words. we give examples for each of these strategies and show how they are implemented in comet.
a mention-synchronous coreference resolution algorithm based on the bell tree. this paper proposes a new approach for coreference resolution which uses the bell tree to represent the search space and casts the coreference resolution problem as finding the best path from the root of the bell tree to the leaf nodes. a maximum entropy model is used to rank these paths. the coreference performance on the 2002 and 2003 automatic content extraction (ace) data will be reported. we also train a coreference system using the muc6 data and competitive results are obtained.
combining trigram and winnow in thai ocr error correction. for languages that have no explicit word boundary such as thai, chinese and japanese, correcting words in text is harder than in english because of additional ambiguities in locating error words. the traditional method handles this by hypothesizing that every substrings in the input sentence could be error words and trying to correct all of them. in this paper, we propose the idea of reducing the scope of spelling correction by focusing only on dubious areas in the input sentence. boundaries of these dubious areas could be obtained approximately by applying word segmentation algorithm and finding word sequences with low probability. to generate the candidate correction words, we used a modified edit distance which reflects the characteristic of thai ocr errors. finally, a part-of-speech trigram model and winnow algorithm are combined to determine the most probable correction.
an iterative algorithm to build chinese language models. we present an iterative procedure to build a chinese language model (lm). we segment chinese text into words based on a word-based chinese language model. however, the construction of a chinese lm itself requires word boundaries. to get out of the chicken-and-egg problem, we propose an iterative procedure that alternates two operations: segmenting text into words and building an lm. starting with an initial segmented corpus and an lm based upon it, we use a viterbi-liek algorithm to segment another set of data. then, we build an lm based on the second set and use the resulting lm to segment again the first corpus. the alternating procedure provides a self-organized way for the segmenter to detect automatically unseen words and correct segmentation errors. our preliminary experiment shows that the alternating procedure not only improves the accuracy of our segmentation, but discovers unseen words suprisingly well. the resulting word-based lm has a perplexity of 188 for a general chinese corpus.
coordination as a direct process. we propose a treatment of coordination based on the concepts of functor, argument and subcategorization. its formalization comprises two parts which are conceptually independent. on one hand, we have extended the feature structure unification to disjunctive and set values in order to check the compatibility and the satisfiability of subcategorization requirements by structured complements. on the other hand, we have considered the conjunction et (and) as the head of the coordinate structure, so that coordinate structures stem simply from the subcategorization specifications of et and the general schemata of a head saturation. both parts have been encoded within hpsg using the same resource that is the subcategorization and its principle which we have just extended.
the representation of multimodal user interface dialogues using discourse pegs. the three-tiered discourse representation defined in (luperfoy, 1991) is applied to multimodal human-computer interface (hci) dialogues. in the applied system the three tiers are (1) a linguistic analysis (morphological, syntactic, sentential semantic) of input and output communicative events including keyboard-entered command language atoms, nl strings, mouse clicks, output text strings, and output graphical events; (2) a discourse model representation containing one discourse object, called a peg, for each construct (each guise of an individual) under discussion; and (3) the knowledge base (kb) representation of the computer agent's 'belief' system which is used to support its interpretation procedures. i present evidence to justify the added complexity of this three-tiered system over standard two-tiered representations, based on (a) cognitive processes that must be supported for any non-idealized dialogue environment (e.g., the agents can discuss constructs not present in their current belief systems), including information decay, and the need for a distinction between understanding a discourse and believing the information content of a discourse; (b) linguistic phenomena, in particular, context-dependent nps, which can be partially or totally anaphoric; and (c) observed requirements of three implemented hci dialogue systems that have employed this three-tiered discourse representation.
segmented and unsegmented dialogue-act annotation with statistical dialogue models. dialogue systems are one of the most challenging applications of natural language processing. in recent years, some statistical dialogue models have been proposed to cope with the dialogue problem. the evaluation of these models is usually performed by using them as annotation models. many of the works on annotation use information such as the complete sequence of dialogue turns or the correct segmentation of the dialogue. this information is not usually available for dialogue systems. in this work, we propose a statistical model that uses only the information that is usually available and performs the segmentation and annotation at the same time. the results of this model reveal the great influence that the availability of a correct segmentation has in obtaining an accurate annotation of the dialogues.
an architecture for dialogue management, context tracking, and pragmatic adaptation in spoken dialogue systems. this paper details a software architecture for discourse processing in spoken dialogue systems, where the three component tasks of discourse processing are (1) dialogue management, (2) context tracking, and (3) pragmatic adaptation. we define these three component tasks and describe their roles in a complex, near-future scenario in which multiple humans interact with each other and with computers in multiple, simultaneous dialogue exchanges. this paper reports on the software modules that accomplish the three component tasks of discourse processing, and an architecture for the interaction among these modules and with other modules of the spoken dialogue system. a motivation of this work is reusable discourse processing software for integration with non-discourse modules in spoken dialogue systems. we document the use of this architecture and its components in several prototypes, and also discuss its potential application to spoken dialogue systems defined in the near-future scenario.
statistical machine translation by parsing. in an ordinary syntactic parser, the input is a string, and the grammar ranges over strings. this paper explores generalizations of ordinary parsing algorithms that allow the input to consist of string tuples and/or the grammar to range over string tuples. such algorithms can infer the synchronous structures hidden in parallel texts. it turns out that these generalized parsers can do most of the work required to train and apply a syntax-aware statistical machine translation system.
a portable algorithm for mapping bitext correspondence. the first step in most empirical work in multilingual nlp is to construct maps of the correspondence between texts and their translations (bitext maps). the smooth injective map recognizer (simr) algorithm presented here is a generic pattern recognition algorithm that is particularly well-suited to mapping bitext correspondence. simr is faster and significantly more accurate than other algorithms in the literature. the algorithm is robust enough to use on noisy texts, such as those resulting from ocr input, and on translations that are not very literal. simr encapsulates its language-specific heuristics, so that it can be ported to any language pair with a minimal effort.
an iterative implicit feedback approach to personalized search. general information retrieval systems are designed to serve all users without considering individual needs. in this paper, we propose a novel approach to personalized search. it can, in a unified way, exploit and utilize implicit feedback information, such as query logs and immediately viewed documents. moreover, our approach can implement result re-ranking and query expansion simultaneously and collaboratively. based on this approach, we develop a client-side personalized web search agent pair (personalized assistant for information retrieval), which supports both english and chinese. our experiments on trec and htrdp collections clearly show that the new approach is both effective and efficient.
a word-to-word model of translational equivalence. many multilingual nlp applications need to translate words between different languages, but cannot afford the computational expenses of inducing or applying a full translation model. for theses applications, we have designed a fast algorithm for estimating a partial translation model, which accounts for translational equivalence only at the word level. the model's precision/recall trade-off can be directly controlled via one threshold parameter. this feature makes the model more suitable for applications that are not fully statistical. the model's hidden parameters can be easily conditioned on information extrinsic to the model, providing an easy way to integrate pre-existing knowledge such as part-of-speech, dictionaries, word order, etc., our model can link word tokens in parallel texts as well as other translation models in the literature. unlike other translation models, it can automatically produce dictionary-sized translation lexicons, and it can do so with over 99% accuracy.
a compositional semantics for focusing subjuncts. a compositional semantics for focusing subjuncts---words such as only, even, and also---is developed from rooth's theory of association with focus. by adapting the theory so that it can be expressed in terms of a frame-based semantic formalism, a semantics that is more computationally practical is arrived at. this semantics captures pragmatic subtleties by incorporating a two-part representation, and recognizes the contribution of intonation to meaning.
syntax-based semi-supervised named entity tagging. we report an empirical study on the role of syntactic features in building a semi-supervised named entity (ne) tagger. our study addresses two questions: what types of syntactic features are suitable for extracting potential nes to train a classifier in a semi-supervised setting? how good is the resulting ne classifier on testing instances dissimilar from its training data? our study shows that constituency and dependency parsing constraints are both suitable features to extract nes and train the classifier. moreover, the classifier showed significant accuracy improvement when constituency features are combined with new dependency feature. furthermore, the degradation in accuracy on unfamiliar test cases is low, suggesting that the trained classifier generalizes well.
a multi-neuro tagger using variable lenghts of contexts. this paper presents a multi-neuro tagger that uses variable lengths of contexts and weighted inputs (with information gains) for part of speech tagging. computer experiments show that it has a correct rate of over 94% for tagging ambiguous words when a small thai corpus with 22,311 ambiguous words is used for training. this result is better than any of the results obtained using the single-neuro taggers with fixed but different lengths of contexts, which indicates that the multi-neuro tagger can dynamically find a suitable length of contexts in tagging.
generalized multitext grammars. generalized multitext grammar (gmtg) is a synchronous grammar formalism that is weakly equivalent to linear context-free rewriting systems (lcfrs), but retains much of the notational and intuitive simplicity of context-free grammar (cfg). gmtg allows both synchronous and independent rewriting. such flexibility facilitates more perspicuous modeling of parallel text than what is possible with other synchronous formalisms. this paper investigates the generative capacity of gmtg, proves that each component grammar of a gmtg retains its generative power, and proposes a generalization of chomsky normal form, which is necessary for synchronous cky-style parsing.
parsing japanese honorifics in unification-based grammar. this paper presents a unification-based approach to japanese honorifics based on a version of hpsg (head-driven phrase structure grammar). utterance parsing is based on lexical specifications of each lexical item, including honorifics, and a few general psg rules using a parser capable of unifying cyclic feature structures. it is shown that the possible word orders of japanese honorific predicate constituents can be automatically deduced in the proposed framework without independently specifying them. discourse information change rules (dicrs) that allow resolving a class of anaphors in honorific contexts are also formulated.
some chart-based techniques for parsing ill-formed input. we argue for the usefulness of an active chart as the basis of a system that searches for the globally most plausible explanation of failure to syntactically parse a given input. we suggest semantics-free, grammar-independent techniques for parsing inputs displaying simple kinds of ill-formedness and discuss the search issues involved.
statistical decision-tree models for parsing. syntactic natural language parsers have shown themselves to be inadequate for processing highly-ambiguous large-vocabulary text, as is evidenced by their poor performance on domains like the wall street journal, and by the movement away from parsing-based approaches to text-processing in general. in this paper, i describe spatter, a statistical parser based on decision-tree learning techniques which constructs a complete parse for every sentence and achieves accuracy rates far better than any published result. this work is based on the following premises: (1) grammars are too complex and detailed to develop manually for most interesting domains; (2) parsing models must rely heavily on lexical and contextual information to analyze sentences accurately; and (3) existing n-gram modeling techniques are inadequate for parsing models. in experiments comparing spatter with ibm's computer manuals parser, spatter significantly outperforms the grammar-based parser. evaluating spatter against the penn treebank wall street journal corpus using the parseval measures, spatter achieves 86% precision, 86% recall, and 1.3 crossing brackets per sentence for sentences of 40 words or less, and 91% precision, 90% recall, and 0.5 crossing brackets for sentences between 10 and 20 words in length.
efficiency, robustness and accuracy in picky chart parsing. this paper describes picky, a probabilistic agenda-based chart parsing algorithm which uses a technique called probabilistic prediction to predict which grammar rules are likely to lead to an acceptable parse of the input. using a suboptimal search method, picky significantly reduces the number of edges produced by cky-like chart parsing algorithms, while maintaining the robustness of pure bottom-up parsers and the accuracy of existing probabilistic parsers. experiments using picky demonstrate how probabilistic modelling can impact upon the efficiency, robustness and accuracy of a parser.
an lr category-neutral parser with left corner prediction. in this paper we present a new parsing model of linguistic and computational interest. linguistically, the relation between the parser and the theory of grammar adopted (government and binding (gb) theory as presented in chomsky (1981, 1986a, b) is clearly specified. computationally, this model adopts a mixed parsing procedure, by using left corner prediction in a modified lr parser.
is it the right answer? exploiting web redundancy for answer validation. answer validation is an emerging topic in question answering, where open domain systems are often required to rank huge amounts of candidate answers. we present a novel approach to answer validation based on the intuition that the amount of implicit knowledge which connects an answer to a question can be quantitatively estimated by exploiting the redundancy of web information. experiments carried out on the trec-2001 judged-answer collection show that the approach achieves a high level of performance (i.e. 81% success rate). the simplicity and the efficiency of this approach make it suitable to be used as a module in question answering systems.
a multilingual paradigm for automatic verb classification. we demonstrate the benefits of a multilingual approach to automatic lexical semantic verb classification based on statistical analysis of corpora in multiple languages. our research incorporates two interrelated threads. in one, we exploit the similarities in the crosslinguistic classification of verbs, to extend work on english verb classification to a new language (italian), and to new classes within that language, achieving an accuracy of 86.4% (baseline 33.9%). our second strand of research exploits the differences across languages in the syntactic expression of semantic properties, to show that complementary information about english verbs can be extracted from their translations in a second language (chinese). the use of multilingual features improves classification performance of the english verbs, achieving an accuracy of 83.5% (baseline 33.3%).
reaping the benefits of interactive syntax and semantics. semantic feedback is an important source of information that a parser could use to deal with local ambiguities in syntax. however, it is difficult to devise a systematic communication mechanism for interactive syntax and semantics. in this article, i propose a variant of left-corner parsing to define the points at which syntax and semantics should interact, an account of grammatical relations and thematic roles to define the content of the communication, and a conflict resolution strategy based on independent preferences from syntax and semantics. the resulting interactive model has been implemented in a program called compere and shown to account for a wide variety of psycholinguistic data on structural and lexical ambiguities.
annotation schemes and their influence on parsing results. most of the work on treebank-based statistical parsing exclusively uses the wall-street-journal part of the penn treebank for evaluation purposes. due to the presence of this quasi-standard, the question of to which degree parsing results depend on the properties of treebanks was often ignored. in this paper, we use two similar german treebanks, t&uuml;ba-d/z and negra, and investigate the role that different annotation decisions play for parsing. for these purposes, we approximate the two treebanks by gradually taking out or inserting the corresponding annotation components and test the performance of a standard pcfg parser on all treebank versions. our results give an indication of which structures are favorable for parsing and which ones are not.