waashk/dblp huggingface dataset:
trainable, scalable summarization using robust nlp and machine learning. we describe a trainable and scalable summarization system which utilizes features derived from information retrieval, information extraction, and nlp techniques and on-line resources. the system combines these features using a trainable feature combiner learned from summary examples through a machine learning algorithm. we demonstrate system scalability by reporting results on the best combination of summarization features for different document sources. we also present preliminary results from a task-based evaluation on summarization output usability.
parsing free word order languages in the paninian framework. there is a need to develop a suitable computational grammar formalism for free word order languages for two reasons: first, a suitably designed formalism is likely to be more efficient. second, such a formalism is also likely to be linguistically more elegant and satisfying. in this paper, we describe such a formalism, called the paninian framework, that has been successfully applied to indian languages.this paper shows that the paninian framework applied to modern indian languages gives an elegant account of the relation between surface form (vibhakti) and semantic (karaka) roles. the mapping is elegant and compact. the same basic account also explains active-passives and complex sentences. this suggests that the solution is not just adhoc but has a deeper underlying unity.a constraint based parser is described for the framework. the constraints problem reduces to bipartite graph matching problem because of the nature of constraints. efficient solutions are known for these problems.it is interesting to observe that such a parser (designed for free word order languages) compares well in asymptotic time complexity with the parser for context free grammars (cfgs) which are basically designed for positional languages.
zero morphemes in unification-based combinatory categorial grammar. in this paper, we report on our use of zero morphemes in unification-based combinatory categorial grammar. after illustrating the benefits of this approach with several examples, we describe the algorithm for compiling zero morphemes into unary rules, which allows us to use zero morphemes more efficiently in natural language processing. then, we discuss the question of equivalence of a grammar with these unary rules to the original grammar. lastly, we compare our approach to zero morphemes with possible alternatives.
unsupervised sense disambiguation using bilingual probabilistic models. we describe two probabilistic models for unsupervised word-sense disambiguation using parallel corpora. the first model, which we call the sense model, builds on the work of diab and resnik (2002) that uses both parallel text and a sense inventory for the target language, and recasts their approach in a probabilistic framework. the second model, which we call the concept model, is a hierarchical model that uses a concept latent variable to relate different language specific sense labels. we show that both models improve performance on the word sense disambiguation task over previous unsupervised approaches, with the concept model showing the largest improvement. furthermore, in learning the concept model, as a by-product, we learn a sense inventory for the parallel language.
problem solving applied to language generation. this research was supported at sri international by the defense advanced research projects agency under contract n00039--79--c--0118 with the naval electronic systems command. the views and conclusions contained in this document are those of the author and should not be interpreted as representative of the official policies either expressed or implied of the defense advanced research projects agency, or the u. s. government. the author is grateful to barbara grosz, gary hendrix and terry winograd for comments on an earlier draft of this paper.
unsupervised part-of-speech tagging employing efficient graph clustering. an unsupervised part-of-speech (pos) tagging system that relies on graph clustering methods is described. unlike in current state-of-the-art approaches, the kind and number of different tags is generated by the method itself. we compute and merge two partitionings of word graphs: one based on context similarity of high frequency words, another on log-likelihood statistics for words of lower frequencies. using the resulting word clusters as a lexicon, a viterbi pos tagger is trained, which is refined by a morphological component. the approach is evaluated on three different languages by measuring agreement with existing taggers.
alternative phrases and natural languages information retrieval. this paper presents a formal analysis for a large class of words called alternative markers, which includes other(than), such(as), and besides. these words appear frequently enough in dialog to warrant serious attention, yet present natural language search engines perform poorly on queries containing them. i show that the performance of a search engine can be improved dramatically by incorporating an approximation of the formal analysis that is compatible with the search engine's operational semantics. the value of this approach is that as the operational semantics of natural language applications improve, even larger improvements are possible.
a practical nonmonotonic theory for reasoning about speech acts. a prerequisite to a theory of the way agents understand speech acts is a theory of how their beliefs and intentions are revised as a consequence of events. this process of attitude revision is an interesting domain for the application of nonmonotonic reasoning because speech acts have a conventional aspect that is readily represented by defaults, but that interacts with an agent's beliefs and intentions in many complex ways that may override the defaults. perrault has developed a theory of speech acts, based on rieter's default logic, that captures the conventional aspect; it does not, however, adequately account for certain easily observed facts about attitude revision resulting from speech acts. a natural theory of attitude revision seems to require a method of stating preferences among competing defaults. we present here a speech act theory, formalized in hierarchic autoepistemic logic (a refinement of moore's autoepistemic logic), in which revision of both the speaker's and hearer's attitudes can be adequately described. as a collateral benefit, efficient automatic reasoning methods for the formalism exist. the theory has been implemented and is now being employed by an utterance-planning system.
the structure of shared forests in ambiguous parsing. the context-free backbone of some natural language analyzers produces all possible cf parses as some kind of shared forest, from which a single tree is to be chosen by a disambiguation process that may be based on the finer features of the language. we study the structure of these forests with respect to optimality of sharing, and in relation with the parsing schema used to produce them. in addition to a theoretical and experimental framework for studying these issues, the main results presented are:- sophistication in chart parsing schemata (e.g. use of look-ahed) may reduce time and space efficiency instead of improving it,- there is a shared forest structure with at most cubic size for any cf grammar,- when o(n3) complexity is required, the shape of a shared forest is dependent on the parsing schema used.though analyzed on cf grammars for simplicity, these results extend to more complex formalisms such as unification based grammars.
a flexible approach to cooperative response generation in information-seeking dialogues. this paper presents a cooperative consultation system on a restricted domain. the system builds hypotheses on the user's plan and avoids misunderstandings (with consequent repair dialogues) through clarification dialogues in case of ambiguity. the role played by constraints in the generation of the answer is characterized in order to limit the cases of ambiguities requiring a clarification dialogue. the answers of the system are generated at different levels of detail, according to the user's competence in the domain.
nltk: the natural language toolkit. the natural language toolkit is a suite of program modules, data sets and tutorials supporting research and teaching in computational linguistics and natural language processing. nltk is written in python and distributed under the gpl open source license. over the past year the toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the python language. this paper reports on the simplified toolkit and explains how it is used in teaching nlp.
a memory-based approach to learning shallow natural language patterns. recognizing shallow linguistic patterns, such as basic syntactic relationships between words, is a common task in applied natural language and text processing. the common practice for approaching this task is by tedious manual definition of possible pattern structures, often in the form of regular expressions or finite automata. this paper presents a novel memory-based learning method that recognizes shallow patterns in new text based on a bracketed training corpus. the training data are stored as-is, in efficient suffix-tree data structures. generalization is performed on-line at recognition time by comparing subsequences of the new text to positive and negative evidence in the corpus. this way, no information in the training is lost, as can happen in other learning systems that construct a single generalized model at the time of training. the paper presents experimental results for recognizing noun phrase, subject-verb and verb-object patterns in english. since the learning approach enables easy porting to new domains, we plan to apply it to syntactic patterns in other languages and to sub-language patterns for information extraction.
a flexible approach to natural language generation for disabled children. natural language generation (nlg) is a way to automatically realize a correct expression in response to a communicative goal. this technology is mainly explored in the fields of machine translation, report generation, dialog system etc. in this paper we have explored the nlg technique for another novel application-assisting disabled children to take part in conversation. the limited physical ability and mental maturity of our intended users made the nlg approach different from others. we have taken a flexible approach where main emphasis is given on flexibility and usability of the system. the evaluation results show this technique can increase the communication rate of users during a conversation.
lexicalization in crosslinguistic probabilistic parsing: the case of french. this paper presents the first probabilistic parsing results for french, using the recently released french treebank. we start with an unlexicalized pcfg as a baseline model, which is enriched to the level of collins' model 2 by adding lexicalization and subcategorization. the lexicalized sister-head model and a bigram model are also tested, to deal with the flatness of the french treebank. the bigram model achieves the best performance: 81% constituency f-score and 84% dependency accuracy. all lexicalized models outperform the unlexicalized baseline, consistent with probabilistic parsing results for english, but contrary to results for german, where lexicalization has only a limited effect on parsing performance.
parsing ambiguous structures using controlled disjunctions and unary quasi-trees. the problem of parsing ambiguous structures concerns (i) their representation and (ii) the specification of mechanisms allowing to delay and control their evaluation. we first propose to use a particular kind of disjunctions called controlled disjunctions: these formulae allows the representation and the implementation of specific constraints that can occur between ambiguous values. but an efficient control of ambiguous structures also has to take into account lexical as well as syntactic information concerning this object. we then propose the use of unary quasi-trees specifying constraints at these different levels. the two devices allow an efficient implementation of the control of the ambiguity. moreover, they are independent from a particular formalism and can be used whatever the linguistic theory.
intentions and indormation in discourse. this paper is about the flow of inference between communicative intentions, discourse structure and the domain during discourse processing. we augment a theory of discourse interpretation with a theory of distinct mental attitudes and reasoning about them, in order to provide an account of how the attitudes interact with reasoning about discourse structure.
acceptability prediction by means of grammaticality quantification. we propose in this paper a method for quantifying sentence grammaticality. the approach based on property grammars, a constraint-based syntactic formalism, makes it possible to evaluate a grammaticality index for any kind of sentence, including ill-formed ones. we compare on a sample of sentences the grammaticality indices obtained from pg formalism and the acceptability judgements measured by means of a psycholinguistic analysis. the results show that the derived grammaticality index is a fairly good tracer of acceptability scores.
knowledge acquisition from texts: using an automatic clustering method based on noun-modifier relationship. we describe the early stage of our methodology of knowledge acquisition from technical texts. first, a partial morpho-syntactic analysis is performed to extract "candidate terms". then, the knowledge engineer, assisted by an automatic clustering tool, builds the "conceptual fields" of the domain. we focus on this conceptual analysis stage, describe the data prepared from the results of the morpho-syntactic analysis and show the results of the clustering module and their interpretation. we found that syntactic links represent good descriptors for candidate terms clustering since the clusters are often easily interpreted as "conceptual fields".
trigger-pair predictors in parsing and tagging. in this article, we apply to natural language parsing and tagging the device of trigger-pair predictors, previously employed exclusively within the field of language modelling for speech recognition. given the task of predicting the correct rule to associate with a parse-tree node, or the correct tag to associate with a word of text, and assuming a particular class of parsing or tagging model, we quantify the information gain realized by taking account of rule or tag trigger-pair predictors, i.e. pairs consisting of a "triggering" rule or tag which has already occurred in the document being processed, together with a specific "triggered" rule or tag whose probability of occurrence within the current sentence we wish to estimate. this information gain is shown to be substantial. further, by utilizing trigger pairs taken from the same general sort of document as is being processed (e.g. same subject matter or same discourse type)---as opposed to predictors derived from a comprehensive general set of english texts---we can significantly increase this information gain.
the effect of corpus size in combining supervised and unsupervised training for disambiguation. we investigate the effect of corpus size in combining supervised and unsupervised learning for two types of attachment decisions: relative clause attachment and prepositional phrase attachment. the supervised component is collins' parser, trained on the wall street journal. the unsupervised component gathers lexical statistics from an unannotated corpus of newswire text. we find that the combined system only improves the performance of the parser for small training sets. surprisingly, the size of the unannotated corpus has little effect due to the noisiness of the lexical statistics acquired by unsupervised learning.
towards history-based grammars: using richer models for probabilistic parsing. we describe a generative probabilistic model of natural language, which we call hbg, that takes advantage of detailed linguistic information to resolve ambiguity. hbg incorporates lexical, syntactic, semantic, and structural information from the parse tree into the disambiguation process in a novel way. we use a corpus of bracketed sentences, called a treebank, in combination with decision tree building to tease out the relevant aspects of a parse tree that will determine the correct parse of a sentence. this stands in contrast to the usual approach of further grammar tailoring via the usual linguistic introspection in the hope of generating the correct parse. in head-to-head tests against one of the best existing robust probabilistic parsing models, which we call p-cfg, the hbg model significantly outperforms p-cfg, increasing the parsing accuracy rate from 60% to 75%, a 37% reduction in error.
a phrase-based statistical model for sms text normalization. short messaging service (sms) texts behave quite differently from normal written texts and have some very special phenomena. to translate sms texts, traditional approaches model such irregularities directly in machine translation (mt). however, such approaches suffer from customization problem as tremendous effort is required to adapt the language model of the existing translation system to handle sms text style. we offer an alternative approach to resolve such irregularities by normalizing sms texts before mt. in this paper, we view the task of sms normalization as a translation problem from the sms language to the english language and we propose to adapt a phrase-based statistical mt model for the task. evaluation by 5-fold cross validation on a parallel sms normalized corpus of 5000 sentences shows that our method can achieve 0.80702 in bleu score against the baseline bleu score 0.6958. another experiment of translating sms texts from english to chinese on a separate sms text corpus shows that, using sms normalization as mt preprocessing can largely boost sms translation performance from 0.1926 to 0.3770 in bleu score.
development and evaluation of a broad-coverage probabilistic grammar of english-language computer manuals. we present an approach to grammar development where the task is decomposed into two separate subtasks. the first tasks linguistic, with the goal of producing a set of rules that have a large coverage (in the sense that the correct parse is among the proposed parses) on a blind test set of sentences. the second task is statistical, with the goal of developing a model of the grammar which assigns maximum probability for the correct parse. we give parsing results on text from computer manuals.
going beyond aer: an extensive analysis of word alignments and their impact on mt. this paper presents an extensive evaluation of five different alignments and investigates their impact on the corresponding mt system output. we introduce new measures for intrinsic evaluations and examine the distribution of phrases and untranslated words during decoding to identify which characteristics of different alignments affect translation. we show that precision-oriented alignments yield better mt output (translating more words and using longer phrases) than recall-oriented alignments.
automatic compensation for parser figure-of-merit flaws. best-first chart parsing utilises a figure of merit (fom) to efficiently guide a parse by first attending to those edges judged better. in the past it has usually been static; this paper will show that with some extra information, a parser can compensate for fom flaws which otherwise slow it down. our results are faster than the prior best by a factor of 2.5; and the speedup is won with no significant decrease in parser accuracy.
discourse entities in janus. this paper addresses issues that arose in applying the model for discourse entity (de) generation in b. webber's work (1978, 1983) to an interactive multimodal interface. her treatment was extended in 4 areas: (1) the notion of context dependence of des was formalized in an intensional logic, (2) the treatment of des for indefinite nps was modified to use skolem functions, (3) the treatment of dependent quantifiers was generalized, and (4) des originating from non-linguistic sources, such as pointing actions, were taken into account. the discourse entities are used in intra- and extra-sentential pronoun resolution in bbn janus.
on the decidability of functional uncertainty. we show that feature logic extended by functional uncertainty is decidable, even if one admits cyclic descriptions. we present an algorithm, which solves feature descriptions containing functional uncertainty in two phases, both phases using a set of deterministic and non-deterministic rewrite rules. we then compare our algorithm with the one of kaplan and maxwell, that does not cover cyclic feature descriptions.
a comparison of document, sentence, and term event spaces. the trend in information retrieval systems is from document to sub-document retrieval, such as sentences in a summarization system and words or phrases in question-answering system. despite this trend, systems continue to model language at a document level using the inverse document frequency (idf). in this paper, we compare and contrast idf with inverse sentence frequency (isf) and inverse term frequency (itf). a direct comparison reveals that all language models are highly correlated; however, the average isf and itf values are 5.5 and 10.4 higher than idf. all language models appeared to follow a power law distribution with a slope coefficient of 1.6 for documents and 1.7 for sentences and terms. we conclude with an analysis of idf stability with respect to random, journal, and section partitions of the 100,830 full-text scientific articles in our experimental corpus.
a complete and recursive feature theory. various feature descriptions are being employed in constrained-based grammar formalisms. the common notational primitive of these descriptions are functional attributes called features. the descriptions considered in this paper are the possibly quantified first-order formulae obtained from a signature of features and sorts. we establish a complete first-order theory ft by means of three axiom schemes and construct three elementarily equivalent models.one of the models consists of so-called feature graphs, a data structure common in computational linguistics. the other two models consist of so-called feature trees, a record-like data structure generalizing the trees corresponding to first-order terms.our completeness proof exhibits a terminating simplification system deciding validity and satisfiability of possibly quantified feature descriptions.
outilex, a linguistic platform for text processing. we present outilex, a generalist linguistic platform for text processing. the platform includes several modules implementing the main operations for text processing and is designed to use large-coverage language resources. these resources (dictionaries, grammars, annotated texts) are formatted into xml, in accordance with current standards. evaluations on efficiency are given.
entity-based cross-document coreferencing using the vector space model. cross-document coreference occurs when the same person, place, event, or concept is discussed in more than one text source. computer recognition of this phenomenon is important because it helps break "the document boundary" by allowing a user to examine information about a particular entity from multiple text sources at the same time. in this paper we describe a cross-document coreference resolution algorithm which uses the vector space model to resolve ambiguities between people having the same name. in addition, we also describe a scoring algorithm for evaluating the cross-document coreference chains produced by our system and we compare our algorithm to the scoring algorithm used in the muc-6 (within document) coreference task.