waashk/dblp huggingface dataset:
flsa: extending latent semantic analysis with features for dialogue act classification. we discuss feature latent semantic analysis (flsa), an extension to latent semantic analysis (lsa). lsa is a statistical method that is ordinarily trained on words only; flsa adds to lsa the richness of the many other linguistic features that a corpus may be labeled with. we applied flsa to dialogue act classification with excellent results. we report results on three corpora: callhome spanish, maptask, and our own corpus of tutoring dialogues.
compilation of weighted finite-state transducers from decision trees. we report on a method for compiling decision trees into weighted finite-state transducers. the key assumptions are that the tree predictions specify how to rewrite symbols from an input string, and the decision at each tree node is stateable in terms of regular expressions on the input string. each leaf node can then be treated as a separate rule where the left and right contexts are constructable from the decisions made traversing the tree from the root to the leaf. these rules are compiled into transducers using the weighted rewite-rule rule-compilation algorithm described in (mohri and sproat, 1996).
multilingual legal terminology on the jibiki platform: the lexalp project. this paper presents the particular use of "jibiki" (papillon's web server development platform) for the lexalp1 project. lexalp's goal is to harmonise the terminology on spatial planning and sustainable development used within the alpine convention2, so that the member states are able to cooperate and communicate efficiently in the four official languages (french, german, italian and slovene). to this purpose, lexalp uses the jibiki platform to build a term bank for the contrastive analysis of the specialised terminology used in six different national legal systems and four different languages. in this paper we present how a generic platform like jibiki can cope with a new kind of dictionary.
a stochastic finite-state word-segmentation algorithm for chinese. we present a stochastic finite-state model for segmenting chinese text into dictionary entries and productively derived words, and providing pronunciations for these words; the method incorporates a class-based model in its treatment of personal names. we also evaluate the system's performance, taking into account the fact that people often do not agree on a single segmentation.
accurate collocation extraction using a multilingual parser. this paper focuses on the use of advanced techniques of text analysis as support for collocation extraction. a hybrid system is presented that combines statistical methods and multilingual parsing for detecting accurate collocational information from english, french, spanish and italian corpora. the advantage of relying on full parsing over using a traditional window method (which ignores the syntactic information) is first theoretically motivated, then empirically validated by a comparative evaluation experiment.
named entity transliteration with comparable corpora. in this paper we investigate chinese-english name transliteration using comparable corpora, corpora where texts in the two languages deal in some of the same topics --- and therefore share references to named entities --- but are not translations of each other. we present two distinct methods for transliteration, one approach using phonetic transliteration, and the second using the temporal distribution of candidate pairs. each of these approaches works quite well, but by combining the approaches one can achieve even better results. we then propose a novel score propagation method that utilizes the co-occurrence of transliteration pairs within document pairs. this propagation method achieves further improvement over the best results from the previous step.
two kinds of metonymy. we propose a distinction between two kinds of metonymy: "referential" metonymy, in which the referent of an np is shifted, and "predicative" metonymy, in which the referent of the np is unchanged and the argument place of the predicate is shifted instead. examples are, respectively, "the hamburger is waiting for his check" and "which airlines fly from boston to denver". we also show that complications arise for both types of metonymy when multiple coercing predicates are considered. finally, we present implemented algorithms handling these complexities that generate both types of metonymic reading, as well as criteria for choosing one type of metonymic reading over another.
dimlex: a lexicon of discorse markers for text generation and understanding. discourse markers ('cue words') are lexical items that signal the kind of coherence relation holding between adjacent text spans; for example, because, since, and for this reason are different markers for causal relations. discourse markers are a syntactically quite heterogeneous group of words, many of which are traditionally treated as function words belonging to the realm of grammar rather than to the lexicon. but for a single discourse relation there is often a set of similar markers, allowing for a range of paraphrases for expressing the relation. to capture the similarities and differences between these, and to represent them adequately, we are developing dimlex, a lexicon of discourse markers. after describing our methodology and the kind of information to be represented in dimlex, we briefly discuss its potential applications in both text generation and understanding.
structure and intonation in spoken language undestanding. the structure imposed upon spoken sentences by intonation seems frequently to be orthogonal to their traditional surface-syntactic structure. however, the notion of "intonational structure" as formulated by pierrehumbert, selkirk, and others, can be subsumed under a rather different notion of syntactic surface structure that emerges from a theory of grammar based on a "combinatory" extension to categorial grammar. interpretations of constituents at this level are in turn directly related to "information structure", or discourse-related notions of "theme", "rheme", "focus" and "presupposition". some simplifications appear to follow for the problem of integrating syntax and other high-level modules in spoken language systems.
type-raising and directionality in combinatory grammar. the form of rules in combinatory categorial grammars (ccg) is constrained by three principles, called "adjacency", "consistency" and "inheritance". these principles have been claimed elsewhere to constrain the combinatory rules of composition and type raising in such a way as to make certain linguistic universals concerning word order under coordination follow immediately. the present paper shows that the three principles have a natural expression in a unification-based interpretation of ccg in which directional information is an attribute of the arguments of functions grounded in string position. the universals can thereby be derived as consequences of elementary assumptions. some desirable results for grammars and parsers follow, concerning type-raising rules.
alternating quantifier scope in ccg. the paper shows that movement or equivalent computational structure-changing operations of any kind at the level of logical form can be dispensed with entirely in capturing quantifier scope ambiguity. it offers a new semantics whereby the effects of quantifier scope alternation can be obtained by an entirely monotonic derivation, without type-changing rules. the paper follows fodor (1982), fodor and sag (1982), and park (1995, 1996) in viewing many apparent scope ambiguities as arising from referential categories rather than true generalized quantifiers.
constructivist development of grounded construction grammar. the paper reports on progress in building computational models of a constructivist approach to language development. it introduces a formalism for construction grammars and learning strategies based on invention, abduction, and induction. examples are drawn from experiments exercising the model in situated language games played by embodied artificial agents.
spontaneous lexicon change. the paper argues that language change can be explained through the stochasticity observed in real-world natural language use. this thesis is demonstrated by modeling language use through language games played in an evolving population of agents. we show that the artificial languages which the agents spontaneously develop based on self-organisation, do not evolve even if the population is changing. then we introduce stochasticity in language use and show that this leads to a constant innovation (new forms and new form-meaning associations) and a maintenance of variation in the population, if the agents are tolerant to variation. some of these variations overtake existing linguistic conventions, particularly in changing populations, thus explaining lexicon change.
trainable sentence planning for complex information presentations in spoken dialog systems. a challenging problem for spoken dialog systems is the design of utterance generation modules that are fast, flexible and general, yet produce high quality output in particular domains. a promising approach is trainable generation, which uses general-purpose linguistic knowledge automatically adapted to the application domain. this paper presents a trainable sentence planner for the match dialog system. we show that trainable sentence planning can produce output comparable to that of match's template-based generator even for quite complex information presentations.
a competition-based explanation of syntactic attachment preferences and garden path phenomena. this paper presents a massively parallel parser that predicts critical attachment behaviors of the human sentence processor, without the use of explicit preference heuristics or revision strategies. the processing of a syntactic ambiguity is modeled as an active, distributed competition among the potential attachments for a phrase. computationally motivated constraints on the competitive mechanism provide a principled and uniform account of a range of human attachment preferences and garden path phenomena.
a semantic approach to ie pattern induction. this paper presents a novel algorithm for the acquisition of information extraction patterns. the approach makes the assumption that useful patterns will have similar meanings to those already identified as relevant. patterns are compared using a variation of the standard vector space model in which information from an ontology is used to capture semantic similarity. evaluation shows this algorithm performs well when compared with a previously reported document-centric approach.
hahacronym: a computational humor system. computational humor will be needed in interfaces, no less than other cognitive capabilities. there are many practical settings where computational humor will add value. among them there are: business world applications (such as advertisement, e-commerce, etc.), general computer-mediated communication and human-computer interaction, increase in the friendliness of natural language interfaces, educational and edutainment systems. in particular in the educational field it is an important resource for getting selective attention, help in memorizing names and situations etc. and we all know how well it works with children.automated humor production in general is a very difficult task but we wanted to prove that some results can be achieved even in short time. we have worked at a concrete limited problem, as the core of the european project hahacronym. the main goal of hahacronym has been the realization of an acronym ironic re-analyzer and generator as a proof of concept in a focalized but non restricted context. to implement this system some general tools have been adapted, or developed for the humorous context. systems output has been submitted to evaluation by human subjects, with a very positive result.
precise n-gram probabilities from stochastic context-free grammars. we present an algorithm for computing n-gram probabilities from stochastic context-free grammars, a procedure that can alleviate some of the standard problems associated with n-grams (estimation from sparse data, lack of linguistic structure, among others). the method operates via the computation of substring expectations, which in turn is accomplished by solving systems of linear equations derived from the grammar. the procedure is fully implemented and has proved viable and useful in practice.
sentence planning as description using tree adjoining grammar. we present an algorithm for simultaneously constructing both the syntax and semantics of a sentence using a lexicalized tree adjoining grammar (ltag). this approach captures naturally and elegantly the interaction between pragmatic and syntactic constraints on descriptions in a sentence, and the inferential interactions between multiple descriptions in a sentence. at the same time, it exploits linguistically motivated, declarative specifications of the discourse functions of syntactic constructions to make contextually appropriate syntactic choices.
user-defined nonmonotonicity in unification-based formalisms. a common feature of recent unification-based grammar formalisms is that they give the user the ability to define his own structures. however, this possibility is mostly limited and does not include nonmonotonic operations. in this paper we show how nonmonotonic operations can also be user-defined by applying default logic (reiter, 1980) and generalizing previous results on nonmonotonic sorts (young and rounds, 1993).
processing complex sentences in the centering framework. we extend the centering model for the resolution of intra-sentential anaphora and specify how to handle complex sentences. an empirical evaluation indicates that the functional information structure guides the search for an antecedent within the sentence.
never look back: an alternative to centering. i propose a model for determining the hearer's attentional state which depends solely on a list of salient discourse entities (s-list). the ordering among the elements of the s-list covers also the function of the backward-looking center in the centering model. the ranking criteria for the s-list are based on the distinction between hearer-old and hearer-new discourse entities and incorporate preferences for inter- and intra-sentential anaphora. the model is the basis for an algorithm which operates incrementally, word by word.
functional centering. based on empirical evidence from a free word order language (german) we propose a fundamental revision of the principles guiding the ordering of discourse entities in the forward-looking centers within the centering model. we claim that grammatical role criteria should be replaced by indicators of the functional information structure of the utterances, i.e., the distinction between context-bound and unbound discourse elements. this claim is backed up by an empirical evaluation of functional centering.
a machine learning approach to pronoun resolution in spoken dialogue. we apply a decision tree based approach to pronoun resolution in spoken dialogue. our system deals with pronouns with np-and non-np-antecedents. we present a set of features designed for pronoun resolution in spoken dialogue and determine the most promising features. we evaluate the system on twenty switchboard dialogues and show that it compares well to byron's (2002) manually tuned system.
automated inversion of logic grammars for generation. we describe a system of reversible grammar in which, given a logic-grammar specification of a natural language, two efficient prolog programs are derived by an off-line compilation process: a parser and a generator for this language. the centerpiece of the system is the inversion algorithm designed to compute the generator code from the parser's prolog code, using the collection of minimal sets of essential arguments (msea) for predicates. the system has been implemented to work with definite clause grammars (dcg) and is a part of an english-japanese machine translation project currently under development at nyu's courant institute.
information retrieval using robust natural language processing. we developed a prototype information retrieval system which uses advanced natural language processing techniques to enhance the effectiveness of traditional key-word based document retrieval. the backbone of our system is a statistical retrieval engine which performs automated indexing of documents, then search and ranking in response to user queries. this core architecture is augmented with advanced natural language processing tools which are both robust and efficient. in early experiments, the augmented system has displayed capabilities that appear to make it superior to the purely statistical base.
summarization-based query expansion in information retrieval. we discuss a semi-interactive approach to information retrieval which consists of two tasks performed in a sequence. first, the system assists the searcher in building a comprehensive statement of information need, using automatically generated topical summaries of sample documents. second, the detailed statement of information need is automatically processed by a series of natural language processing routines in order to derive an optimal search query for a statistical information retrieval system. in this paper, we investigate the role of automated document summarization in building effective search statements. we also discuss the results of latest evaluation of our system at the annual text retrieval conference (trec).
a corpus-based approach to automatic compound extraction. an automatic compound retrieval method is proposed to extract compounds within a text message. it uses n-gram mutual information, relative frequency count and parts of speech as the features for compound extraction. the problem is modeled as a two-class classification problem based on the distributional characteristics of n-gram tokens in the compound and the non-compound clusters. the recall and precision using the proposed approach are 96.2% and 48.2% for bigram compounds and 96.6% and 39.6% for trigram compounds for a testing corpus of 49,314 words. a significant cutdown in processing time has been observed.
an improved extraction pattern representation model for automatic ie pattern acquisition. several approaches have been described for the automatic unsupervised acquisition of patterns for information extraction. each approach is based on a particular model for the patterns to be acquired, such as a predicate-argument structure or a dependency chain. the effect of these alternative models has not been previously studied. in this paper, we compare the prior models and introduce a new model, the subtree model, based on arbitrary subtrees of dependency trees. we describe a discovery procedure for this model and demonstrate experimentally an improvement in recall using subtree patterns.
incorporating speech recognition confidence into discriminative named entity recognition of speech data. this paper proposes a named entity recognition (ner) method for speech recognition results that uses confidence on automatic speech recognition (asr) as a feature. the asr confidence feature indicates whether each word has been correctly recognized. the ner model is trained using asr results with named entity (ne) labels as well as the corresponding transcriptions with ne labels. in experiments using support vector machines (svms) and speech data from japanese newspaper articles, the proposed method outperformed a simple application of text-based ner to asr results in ner f-measure by improving precision. these results show that the proposed method is effective in ner for noisy inputs.
lexical transfer using a vector-space model. building a bilingual dictionary for transfer in a machine translation system is conventionally done by hand and is very time-consuming. in order to overcome this bottleneck, we propose a new mechanism for lexical transfer, which is simple and suitable for learning from bilingual corpora. it exploits a vector-space model developed in information retrieval research. we present a preliminary result from our computational experiment.
experiments and prospects of example-based machine translation. ebmt (example-based machine translation) is proposed. ebmt retrieves similar examples (pairs of source phrases, sentences, or texts and their translations) from a database of examples, adapting the examples to translate a new input. ebmt has the following features: (1) it is easily upgraded simply by inputting appropriate examples to the database; (2) it assigns a reliability factor to the translation result; (3) it is accelerated effectively by both indexing and parallel computing; (4) it is robust because of best-match reasoning; and (5) it well utilizes translator expertise. a prototype system has been implemented to deal with a difficult translation problem for conventional rule-based machine translation (rbmt), i.e., translating japanese noun phrases of the form "n1 no n2" into english. the system has achieved about a 78% success rate on average. this paper explains the basic idea of ebmt, illustrates the experiment in detail, explains the broad applicability of ebmt to several difficult translation problems for rbmt and discusses the advantages of integrating ebmt with rbmt.
using predicate-argument structures for information extraction. in this paper we present a novel, customizable ie paradigm that takes advantage of predicate-argument structures. we also introduce a new way of automatically identifying predicate argument structures, which is central to our ie paradigm. it is based on: (1) an extended set of features; and (2) inductive decision tree learning. the experimental results prove our claim that accurate predicate-argument structures enable high quality ie results.
keyword extraction using term-domain interdependence for dictation of radio news. in this paper, we propose keyword extraction method for dictation of radio news which consists of several domains. in our method, newspaper articles which are automatically classified into suitable domains are used in order to calculate feature vectors. the feature vectors shows term-domain interdependence and are used for selecting a suitable domain of each part of radio news. keywords are extracted by using the selected domain. the results of keyword extraction experiments showed that our methods are robust and effective for dictation of radio news.
hierarchical directed acyclic graph kernel: methods for structured natural language data. this paper proposes the "hierarchical directed acyclic graph (hdag) kernel" for structured natural language data. the hdag kernel directly accepts several levels of both chunks and their relations, and then efficiently computes the weighed sum of the number of common attribute sequences of the hdags. we applied the proposed method to question classification and sentence alignment tasks to evaluate its performance as a similarity measure and a kernel function. the results of the experiments demonstrate that the hdag kernel is superior to other kernel functions and baseline methods.
convolution kernels with feature selection for natural language processing tasks. convolution kernels, such as sequence and tree kernels, are advantageous for both the concept and accuracy of many natural language processing (nlp) tasks. experiments have, however, shown that the over-fitting problem often arises when these kernels are used in nlp tasks. this paper discusses this issue of convolution kernels, and then proposes a new approach based on statistical feature selection that avoids this issue. to enable the proposed method to be executed efficiently, it is embedded into an original kernel calculation process by using sub-structure mining algorithms. experiments are undertaken on real nlp tasks to confirm the problem with a conventional method and to compare its performance with that of the proposed method.