waashk/dblp huggingface dataset:
an english generator for a case-labelled dependency representation. the paper describes a program which has been constructed to produce english strings from a case-labelled dependency representation. the program uses an especially simple and uniform control structure with a well defined separation of the different knowledge sources used during generation. furthermore, the majority of the system's knowledge is expressed in a declarative form, so in principle the generator's knowledge bases could be used for purposes other than generation. the generator uses a two-pass control structure, the first translating from the semantically orientated case-labelled dependency structures into surface syntactic trees and the second translating from these trees into english strings.the generator is very flexible: it can be run in such a way as to produce all the possible syntactically legitimate variations on a given utterance, and has built in facilities to do some synonym substitution. it has been used in a number of application domains: notably as a part of a free text retrieval system and as part of a natural language front end to a relational database system.
rule filtering by pattern for efficient hierarchical translation. we describe refinements to hierarchical translation search procedures intended to reduce both search errors and memory usage through modifications to hypothesis expansion in cube pruning and reductions in the size of the rule sets used in translation. rules are put into syntactic classes based on the number of non-terminals and the pattern, and various filtering strategies are then applied to assess the impact on translation speed and quality. results are reported on the 2008 nist arabic-to-english evaluation task.
learning to interpret utterances using dialogue history. we describe a methodology for learning a disambiguation model for deep pragmatic interpretations in the context of situated task-oriented dialogue. the system accumulates training examples for ambiguity resolution by tracking the fates of alternative interpretations across dialogue, including subsequent clarificatory episodes initiated by the system itself. we illustrate with a case study building maximum entropy models over abductive interpretations in a referential communication task. the resulting model correctly resolves 81% of ambiguities left unresolved by an initial handcrafted baseline. a key innovation is that our method draws exclusively on a system's own skills and experience and requires no human annotation.
weakly supervised approaches for ontology population. we present a weakly supervised approach to automatic ontology population from text and compare it with two other unsupervised approaches. in our experiments we populate a part of our ontology of named entities. we considered two high level categories-geographical locations and person names and ten sub-classes for each category. for each sub-class we automatically learn a syntactic model from a list of training examples and a parsed corpus. a novel syntactic indexing method allowed us to use large quantities of syntactically annotated data. the syntactic model for each named entity sub-class is a set of weighted syntactic features, i.e. words which typically co-occur with the members of the class in the corpus. the method is weakly supervised, since no manually annotated corpus is used in the learning process. the syntactic models are used to classify the unknown named entities in the test set. the method achieved promising results, i.e. 65% accuracy, and outperforms significantly the other two approaches.
translation and extension of concepts across languages. we present a method which, given a few words defining a concept in some language, retrieves, disambiguates and extends corresponding terms that define a similar concept in another specified language. this can be very useful for cross-lingual information retrieval and the preparation of multi-lingual lexical resources. we automatically obtain term translations from multilingual dictionaries and disambiguate them using web counts. we then retrieve web snippets with co-occurring translations, and discover additional concept terms from these snippets. our term discovery is based on co-appearance of similar words in symmetric patterns. we evaluate our method on a set of language pairs involving 45 languages, including combinations of very dissimilar ones such as russian, chinese, and hebrew for various concepts. we assess the quality of the retrieved sets using both human judgments and automatically comparing the obtained categories to corresponding english wordnet synsets.
natural language generation as planning under uncertainty for spoken dialogue systems. we present and evaluate a new model for natural language generation (nlg) in spoken dialogue systems, based on statistical planning, given noisy feedback from the current generation context (e.g. a user and a surface realiser). we study its use in a standard nlg problem: how to present information (in this case a set of search results) to users, given the complex tradeoffs between utterance length, amount of information conveyed, and cognitive load. we set these trade-offs by analysing existing match data. we then train a nlg policy using reinforcement learning (rl), which adapts its behaviour to noisy feedback from the current generation context. this policy is compared to several baselines derived from previous work in this area. the learned policy significantly outperforms all the prior approaches.
the syntactic regularity of english noun phrases. approximately, 10,000 naturally occurring noun phrases taken from the lob corpus were used firstly, to evaluate the np component of the alvey anlt grammar (grover et al., 1987, 1989) and secondly, to retest sampson's (1987a) claim that this data provide evidence for the lack of a clear-cut distinction between grammatical and 'deviant' examples. the examples were sorted and classified on the basis of the lexical and syntactic analysis undertaken as part of the lob corpus project (sampson, 1987b). tokens of each resulting type were parsed using the anlt grammar and the results analysed to determine the success rate of the parses and the generality of the rules employed.
lexical morphology in machine translation: a feasibility study. this paper presents a feasibility study for implementing lexical morphology principles in a machine translation system in order to solve unknown words. multilingual symbolic treatment of word-formation is seducing but requires an in-depth analysis of every step that has to be performed. the construction of a prototype is firstly presented, highlighting the methodological issues of such approach. secondly, an evaluation is performed on a large set of data, showing the benefits and the limits of such approach.
feature-based method for document alignment in comparable news corpora. in this paper, we present a feature-based method to align documents with similar content across two sets of bilingual comparable corpora from daily news texts. we evaluate the contribution of each individual feature and investigate the incorporation of these diverse statistical and heuristic features for the task of bilingual document alignment. experimental results on the english-chinese and english-malay comparable news corpora show that our proposed discrete fourier transform-based term frequency distribution feature is very effective. it contributes 4.1% and 8% to performance improvement over pearson's correlation method on the two comparable corpora. in addition, when more heuristic and statistical features as well as a bilingual dictionary are utilized, our method shows an absolute performance improvement of 23.2% and 15.3% on the two sets of bilingual corpora when comparing with a prior information retrieval-based method.
an annotation scheme for discourse-level argumentation in research articles. in order to build robust automatic abstracting systems, there is a need for better training resources than are currently available. in this paper, we introduce an annotation scheme for scientific articles which can be used to build such a resource in a consistent way. the seven categories of the scheme are based on rhetorical moves of argumentation. our experimental results show that the scheme is stable, reproducible and intuitive to use.
syntactic phrase reordering for english-to-arabic statistical machine translation. syntactic reordering of the source language to better match the phrase structure of the target language has been shown to improve the performance of phrase-based statistical machine translation. this paper applies syntactic reordering to english-to-arabic translation. it introduces reordering rules, and motivates them linguistically. it also studies the effect of combining reordering with arabic morphological segmentation, a preprocessing technique that has been shown to improve arabic-english and english-arabic translation. we report on results in the news text domain, the un text domain and in the spoken travel domain.
co-dispersion: a windowless approach to lexical association. we introduce an alternative approach to extracting word pair associations from corpora, based purely on surface distances in the text. we contrast it with the prevailing window-based co-occurrence model and show it to be more statistically robust and to disclose a broader selection of significant associative relationships - owing largely to the property of scale-independence. in the process we provide insights into the limiting characteristics of window-based methods which complement the sometimes conflicting application-oriented literature in this area.
combining clues for word alignment. in this paper, a word alignment approach is presented which is based on a combination of clues. word alignment clues indicate associations between words and phrases. they can be based on features such as frequency, part-of-speech, phrase type, and the actual wordform strings. clues can be found by calculating similarity measures or learned from word aligned data. the clue alignment approach, which is proposed in this paper, makes it possible to combine association clues taking different kinds of linguistic information into account. it allows a dynamic tokenization into token units of varying size. the approach has been applied to an english/swedish parallel text with promising results.
bayesian word sense induction. sense induction seeks to automatically identify word senses directly from a corpus. a key assumption underlying previous work is that the context surrounding an ambiguous word is indicative of its meaning. sense induction is thus typically viewed as an unsupervised clustering problem where the aim is to partition a word's contexts into different classes, each representing a word sense. our work places sense induction in a bayesian context by modeling the contexts of the ambiguous word as samples from a multinomial distribution over senses which are in turn characterized as distributions over words. the bayesian framework provides a principled way to incorporate a wide range of features beyond lexical co-occurrences and to systematically assess their utility on the sense induction task. the proposed approach yields improvements over state-of-the-art systems on a benchmark dataset.
japanese dependency structure analysis based on maximum entropy models. this paper describes a dependency structure analysis of japanese sentences based on the maximum entropy models. our model is created by learning the weights of some features from a training corpus to predict the dependency between bunsetsus or phrasal units. the dependency accuracy of our system is 87.2% using the kyoto university corpus. we discuss the contribution of each feature set and the relationship between the number of training data and the accuracy.
generating a non-english subjectivity lexicon: relations that matter. we describe a method for creating a non-english subjectivity lexicon based on an english lexicon, an online translation service and a general purpose thesaurus: wordnet. we use a pagerank-like algorithm to bootstrap from the translation of the english lexicon and rank the words in the thesaurus by polarity using the network of lexical relations in wordnet. we apply our method to the dutch language. the best results are achieved when using synonymy and antonymy relations only, and ranking positive and negative words simultaneously. our method achieves an accuracy of 0.82 at the top 3,000 negative words, and 0.62 at the top 3,000 positive words.
a general, abstract model of incremental dialogue processing. we present a general model and conceptual framework for specifying architectures for incremental processing in dialogue systems, in particular with respect to the topology of the network of modules that make up the system, the way information flows through this network, how information increments are 'packaged', and how these increments are processed by the modules. this model enables the precise specification of incremental systems and hence facilitates detailed comparisons between systems, as well as giving guidance on designing new systems.
lattice parsing to integrate speech recognition and rule-based machine translation. in this paper, we present a novel approach to integrate speech recognition and rule-based machine translation by lattice parsing. the presented approach is hybrid in two senses. first, it combines structural and statistical methods for language modeling task. second, it employs a chart parser which utilizes manually created syntax rules in addition to scores obtained after statistical processing during speech recognition. the employed chart parser is a unification-based active chart parser. it can parse word graphs by using a mixed strategy instead of being bottom-up or top-down only. the results are reported based on word error rate on the nist hub-1 word-lattices. the presented approach is implemented and compared with other syntactic language modeling techniques.
contextual phrase-level polarity analysis using lexical affect scoring and syntactic n-grams. we present a classifier to predict contextual polarity of subjective phrases in a sentence. our approach features lexical scoring derived from the dictionary of affect in language (dal) and extended through wordnet, allowing us to automatically score the vast majority of words in our input avoiding the need for manual labeling. we augment lexical scoring with n-gram analysis to capture the effect of context. we combine dal scores with syntactic constituents and then extract n-grams of constituents from all sentences. we also use the polarity of all syntactic constituents within the sentence as features. our results show significant improvement over a majority class baseline as well as a more difficult baseline consisting of lexical n-grams.
enhancing unlexicalized parsing performance using a wide coverage lexicon, fuzzy tag-set mapping, and em-hmm-based lexical probabilities. we present a framework for interfacing a pcfg parser with lexical information from an external resource following a different tagging scheme than the treebank. this is achieved by defining a stochastic mapping layer between the two resources. lexical probabilities for rare events are estimated in a semi-supervised manner from a lexicon and large unannotated corpora. we show that this solution greatly enhances the performance of an unlexicalized hebrew pcfg parser, resulting in state-of-the-art hebrew parsing results both when a segmentation oracle is assumed, and in a real-word parsing scenario of parsing unsegmented tokens.
flexible answer typing with discriminative preference ranking. an important part of question answering is ensuring a candidate answer is plausible as a response. we present a flexible approach based on discriminative preference ranking to determine which of a set of candidate answers are appropriate. discriminative methods provide superior performance while at the same time allow the flexibility of adding new and diverse features. experimental results on a set of focused what ...? and which ...? questions show that our learned preference ranking methods perform better than alternative solutions to the task of answer typing. a gain of almost 0.2 in mrr for both the first appropriate and first correct answers is observed along with an increase in precision over the entire range of recall.
language id in the context of harvesting language data off the web. as the arm of nlp technologies extends beyond a small core of languages, techniques for working with instances of language data across hundreds to thousands of languages may require revisiting and recalibrating the tried and true methods that are used. of the nlp techniques that has been treated as "solved" is language identification (language id) of written text. however, we argue that language id is far from solved when one considers input spanning not dozens of languages, but rather hundreds to thousands, a number that one approaches when harvesting language data found on the web. we formulate language id as a coreference resolution problem and apply it to a web harvesting task for a specific linguistic data type and achieve a much higher accuracy than long accepted language id approaches.
large-coverage root lexicon extraction for hindi. this paper describes a method using morphological rules and heuristics, for the automatic extraction of large-coverage lexicons of stems and root word-forms from a raw text corpus. we cast the problem of high-coverage lexicon extraction as one of stemming followed by root word-form selection. we examine the use of pos tagging to improve precision and recall of stemming and thereby the coverage of the lexicon. we present accuracy, precision and recall scores for the system on a hindi corpus.
tbl-improved non-deterministic segmentation and pos tagging for a chinese parser. although a lot of progress has been made recently in word segmentation and pos tagging for chinese, the output of current state-of-the-art systems is too inaccurate to allow for syntactic analysis based on it. we present an experiment in improving the output of an off-the-shelf module that performs segmentation and tagging, the tokenizer-tagger from beijing university (pku). our approach is based on transformation-based learning (tbl). unlike in other tbl-based approaches to the problem, however, both obligatory and optional transformation rules are learned, so that the final system can output multiple segmentation and pos tagging analyses for a given input. by allowing for a small amount of ambiguity in the output of the tokenizer-tagger, we achieve a very considerable improvement in accuracy. compared to the pku tokenizer-tagger, we improve segmentation f-score from 94.18% to 96.74%, tagged word f-score from 84.63% to 92.44%, segmented sentence accuracy from 47.15% to 65.06% and tagged sentence accuracy from 14.07% to 31.47%.
evaluating the inferential utility of lexical-semantic resources. lexical-semantic resources are used extensively for applied semantic inference, yet a clear quantitative picture of their current utility and limitations is largely missing. we propose system- and application-independent evaluation and analysis methodologies for resources' performance, and systematically apply them to seven prominent resources. our findings identify the currently limited recall of available resources, and indicate the potential to improve performance by examining non-standard relation types and by distilling the output of distributional methods. further, our results stress the need to include auxiliary information regarding the lexical and logical contexts in which a lexical inference is valid, as well as its prior validity likelihood.
an alignment algorithm using belief propagation and a structure-based distortion model. in this paper, we first demonstrate the interest of the loopy belief propagation algorithm to train and use a simple alignment model where the expected marginal values needed for an efficient em-training are not easily computable. we then improve this model with a distortion model based on structure conservation.
semitic morphological analysis and generation using finite state transducers with feature structures. this paper presents an application of finite state transducers weighted with feature structure descriptions, following amtrup (2003), to the morphology of the semitic language tigrinya. it is shown that feature-structure weights provide an efficient way of handling the templatic morphology that characterizes semitic verb stems as well as the long-distance dependencies characterizing the complex tigrinya verb morphotactics. a relatively complete computational implementation of tigrinya verb morphology is described.
character-level dependencies in chinese: usefulness and learning. we investigate the possibility of exploiting character-based dependency for chinese information processing. as chinese text is made up of character sequences rather than word sequences, word in chinese is not so natural a concept as in english, nor is word easy to be defined without argument for such a language. therefore we propose a character-level dependency scheme to represent primary linguistic relationships within a chinese sentence. the usefulness of character dependencies are verified through two specialized dependency parsing tasks. the first is to handle trivial character dependencies that are equally transformed from traditional word boundaries. the second furthermore considers the case that annotated internal character dependencies inside a word are involved. both of these results from character-level dependency parsing are positive. this study provides an alternative way to formularize basic character-and word-level representation for chinese.
sentiment summarization: evaluating and learning user preferences. we present the results of a large-scale, end-to-end human evaluation of various sentiment summarization models. the evaluation shows that users have a strong preference for summarizers that model sentiment over non-sentiment baselines, but have no broad overall preference between any of the sentiment-based models. however, an analysis of the human judgments suggests that there are identifiable situations where one summarizer is generally preferred over the others. we exploit this fact to build a new summarizer by training a ranking svm model over the set of human preference judgments that were collected during the evaluation, which results in a 30% relative reduction in error over the previous best summarizer.
word lattices for multi-source translation. multi-source statistical machine translation is the process of generating a single translation from multiple inputs. previous work has focused primarily on selecting from potential outputs of separate translation systems, and solely on multi-parallel corpora and test sets. we demonstrate how multi-source translation can be adapted for multiple monolingual inputs. we also examine different approaches to dealing with multiple sources, including consensus decoding, and we present a novel method of input combination to generate lattices for multi-source translation within a single translation model.
company-oriented extractive summarization of financial news. the paper presents a multi-document summarization system which builds company-specific summaries from a collection of financial news such that the extracted sentences contain novel and relevant information about the corresponding organization. the user's familiarity with the company's profile is assumed. the goal of such summaries is to provide information useful for the short-term trading of the corresponding company, i.e., to facilitate the inference from news to stock price movement in the next day. we introduce a novel query (i.e., company name) expansion method and a simple unsupervized algorithm for sentence ranking. the system shows promising results in comparison with a competitive baseline.