waashk/dblp huggingface dataset:
semantics-based representation for multimodal interpretation in conversational systems. to support context-based multimodal interpretation in conversational systems, we have developed a semantics-based representation to capture salient information from user inputs and the overall conversation. in particular, we present three unique characteristics: fine-grained semantic models, flexible composition of feature structures, and consistent representation at multiple levels. this representation allows our system to use rich contexts to resolve ambiguities, infer unspecified information, and improve multimodal alignment. as a result, our system is able to enhance understanding of multimodal inputs including those abbreviated, imprecise, or complex ones.
representing information need with semantic relations. information retrieval systems can be made more effective by providing more expressive query languages for users to specify their information need. this paper argues that this can be achieved through the use of semantic relations as query primitives, and describes a new technique for extracting semantic relations from an online dictionary. in contrast to existing research, this technique involves the composition of basic semantic relations, a process akin to constrained spreading activation in semantic networks. the proposed technique is evaluated in the context of extracting semantic relations that are relevant for retrieval from a corpus of pictures.
a chinese characters coding scheme for computer input and internal stopage. a coding scheme for inputting chinese characters by means of a conventional keyboard has been developed. the code for each chinese character is composed of two strings of keys, one corresponds to the spelling and the other the ideographic property of the character. each code requires no more than seven keys (average five and a half keys) and 99.5% of the ten thousand characters in a dictionary 'xiandai hanyu cidian' have unique codes. each input code can be packed into 32 bits for internal representation.
motivations and methods for text simplification. long and complicated sentences prove to be a stumbling block for current systems relying on nl input. these systems stand to gain from methods that syntactically simplify such sentences. to simplify a sentence, we need an idea of the structure of the sentence, to identify the components to be separated out. obviously a parser could be used to obtain the complete structure of the sentence. however, full parsing is slow and prone to failure, especially on complex sentences. in this paper, we consider two alternatives to full parsing which could be used for simplification. the first approach uses a finite state grammar (fsg) to produce noun and verb groups while the second uses a supertagging model to produce dependency linkages. we discuss the impact of these two input representations on the simplification process.
putting frames in perspective. this paper attempts to bridge the gap between framenet frames and inference. we describe a computational formalism that captures structural relationships among participants in a dynamic scenario. this representation is used to describe the internal structure of framenet frames in terms of parameters for event simulations. we apply our formalism to the commerce domain and show how it provides a flexible means of accounting for linguistic perspective and other inferential effects.
coupling an automatic dictation system with a grammar checker. automatic dictation systems (ads) are nowadays powerful and rellable. however, some inadequacies of the underlying models still cause errors. in this paper, we are essentially interested in the language model implemented in the linguistic component, and we leave aside the acoustic module. more precisely, we aim at improving this linguistic model by coupling the ads with a syntactic parser, able to diagnose and correct grammatical errors. we describe the characteristics of such a coupling, and show how the performance of the ads improves with the actual coupling realized for french between the tangora ads and the grammar checker developed at the ibm france scientific center.
word sense disambiguation of adjectives using probabilistic networks. in this paper, word sense disambiguation (wsd) accuracy achievable by a probabilistic classifier, using very minimal training sets, is investigated. we made the assumption that there are no tagged corpora available and identified what information, needed by an accurate wsd system, can and cannot be automatically obtained. the lesson learned can then be used to focus on what knowledge needs manual annotation. our system, named bayesian hierarchical disambiguator (bhd), uses the internet, arguably the largest corpus in existence, to address the sparse data problem, and uses wordnet's hierarchy for semantic contextual features. in addition, bayesian networks are automatically constructed to represent knowledge learned from training sets by modeling the selectional preference of adjectives. these networks are then applied to disambiguation by performing inferences on unseen adjective-noun pairs. we demonstrate that this system is able to disambiguate adjectives in unrestricted text at good initial accuracy rates without the need for tagged corpora. the learning and extensibility aspects of the model are also discussed, showing how tagged corpora and additional context can be incorporated easily to improve accuracy, and how this technique can be used to disambiguate other types of word pairs, such as verb-noun and adverb-verb pairs.
maximum entropy models for word sense disambiguation. a maximum entropy-based word sense disambiguation system is presented, consisting of individual word experts that are trained on both labeled and partially labeled corpora. the classification probabilities from the individual word experts are integrated using a new search algorithm, which balances time complexity and accuracy. the model is evaluated using established procedures on the english-all-words task from the senseval-2 workshop, a large test set consisting of words from all word groups to be disambiguated. lastly, an ongoing project that integrates pos tagging, parsing, and sense disambiguation in one system is presented. once in place, it will be boot-strapped with existing partially labeled corpora, to process and then train from them. the goal is to show that with each successive iteration, the accuracy of all three processes, pos tagging, parsing, and wsd, will improve as the system learns from more accurate, self-generated training data.
a logic-based government-binding parser for mandarin chinese. mandarin chinese is a highly flexible and context-sensitive language. it is difficult to do the case marking and index assignment during the parsing of chinese sentences. this paper proposes a logic-based government-binding approach to treat this problem. the grammar formalism is specified in a formal way. uniform treatments of movements, arbitrary number of movement non-terminals, automatic detection of grammar errors beforehand, and clear declarative semantics are its specific features. many common linguistic phenomena of chinese sentences are represented with this formalism. for example, topic-comment structures, the ba-constructions, the bei-constructions, relative clause constructions, appositive clause constructions, and serial verb constructions. a simple pronoun resolution is touched upon. the expressive capabilities and the design methodologies show this mechanism is also suitable for other flexible and context-sensitive languages.
towards automatic generation of natural language generation systems. systems that interact with the user via natural language are in their infancy. as these systems mature and become more complex, it would be desirable for a system developer if there were an automatic method for creating natural language generation components that can produce quality output efficiently. we conduct experiments that show that this goal appears to be realizable. in particular we discuss a natural language generation system that is composed of spot, a trainable sentence planner, and fer-gus, a stochastic surface, realizer. we show how these stochastic nlg components can be made to work together, that they can be ported to new domains with apparent ease, and that such nlg components can be integrated in a real-time dialog system.
automatic semantic classification for chinese unknown compound nouns. the paper describes a similarity-based model to present the morphological rules for chinese compound nouns. this representation model serves functions of 1) as the morphological rules of the compounds, 2) as a mean to evaluate the properness of a compound construction, and 3) as a mean to disambiguate the semantic ambiguity of the morphological head of a compound noun. an automatic semantic classification system for chinese unknown compounds is thus implemented based on the model. experiments and error analyses are also presented.
nlp and ir approaches to monolingual and multilingual link detection. this paper considers several important issues for monolingual and multilingual link detection. the experimental results show that nouns, verbs, adjectives and compound nouns are useful to represent news stories; story expansion is helpful; topic segmentation has a little effect; and a translation model is needed to capture the differences between languages.
a part-of-speech-based alignment algorithm. to align bilingual texts becomes a crucial issue recently. rather than using length-based or translation-based criterion, a part-of-speech-based criterion is proposed. we postulate that source texts and target texts should share the same concepts, ideas, entities, and events. simulated annealing approach is used to implement this alignment algorithm. the preliminary experiments show good performance. most importantly, the experimental objects are chinese-english texts, which are selected from different language families.
a rule-based and mt-oriented approach to prepositional phrase attachment. prepositional phrase is the key issue in structural ambiguity. recently, researches in corpora provide the lexical cue of prepositions with other words and the information could be used to partly resolve ambiguity resulted from prepositional phrases. two possible attachments are considered in the literature; either noun attachment or verb attachment. in this paper, we consider the problem from viewpoint of machine translation. four different attachments are told out according to their functionality: noun attachment. both lexical knowledge and semantic knowledge are involved resolving attachment in the proposed mechanism. experimental results show that considering more types of prepositional phrases is useful in machine translation.
information-based case grammar. in this paper we propose a framework of information-based case grammar (icg). this grammatical formalism entails that the lexical entry for each word contain both semantic and syntactic feature structures. in the feature structure of a phrasal head, we encode syntactic and semantic constraints on grammatical phrasal patterns in terms of thematic structures, and encode the precedence relations in terms of adjunct structures. such feature structures denote partial information which defines the set of legal phrases. they also provide sufficient information to identify thematic roles. with this formalism, parsing and thematic analysis can be achieved simultaneously. due to the simplicity and flexibility of information-based case grammar, context dependent and discontinuous relations such as agreements, coordinations, long-distance dependencies, and control and binding, can be easily expressed. icg is a kind of unification-based formalism. therefore it inherits the advantages of unification-based formalisms and more.
a muitilingual news summarizer. huge multilingual news articles are reported and disseminated on the internet. how to extract the key information and save the reading time is a crucial issue. this paper proposes architecture of multilingual news summarizer, including monolingual and multilingual clustering, similarity measure among meaningful units, and presentation of summarization results. translation among news stories, idiosyncrasy among languages, implicit information, and user preference are addressed.
word identification for mandarin chinese sentences. chinese sentences are composed with string of characters without blanks to mark words. however the basic unit for sentence parsing and understanding is word. therefore the first step of processing chinese sentences is to identify the words. the difficulties of identifying words include (1) the identification of complex words, such as determinative-measure, reduplications, derived words etc., (2) the identification of proper names, (3) resolving the ambiguous segmentations. in this paper, we propose the possible solutions for the above difficulties. we adopt a matching algorithm with 6 different heuristic rules to resolve the ambiguities and achieve an 99.77% of the success rate. the statistical data supports that the maximal matching algorithm is the most effective heuristics.
identification and classification of proper nouns in chinese texts. various strategies are proposed to identify and classify three types of proper nouns in chinese texts. clues from character, sentence and paragraph levels are employed to resolve chinese personal names. character, syllable and frequency conditions are presented to treat transliterated personal names. to deal with organization names, keywords, prefix, word association and parts-of-speech are applied. for fair evaluation, large scale test data are selected from six sections of a newspaper. the precision and the recall for these three types are (88.04%, 92.56%), (50.62%, 71.93%) and (61.79%, 54.50%), respectively. when the former two types are regarded as a category, the performance becomes (81.46%, 91.22%). compared with other approaches, our approach has better performance and our classification is automatic.
a new design of prolog-based bottom-up parsing system with government-binding theory. this paper addresses the problems of movement transformation in prolog-based bottom-up parsing system. three principles of government-binding theory are employed to deal with these problems. they are empty category principle, c-command principle, and subjacency principle. a formalism based upon them is proposed. translation algorithms are given to add these linguistic principles to the general grammar rules, the leftward movement grammar rules, and the rightward movement grammar rules respectively. this approach has the following specific features: the uniform treatments of leftward and rightward movements, the arbitrary number of movement non-terminals in the rule body, and automatic detection of grammar errors before parsing. an example in chinese demonstrates all the concepts.
unknown word extraction for chinese documents. there is no blank to mark word boundaries in chinese text. as a result, identifying words is difficult, because of segmentation ambiguities and occurrences of unknown words. conventionally unknown words were extracted by statistical methods because statistical methods are simple and efficient. however the statistical methods without using linguistic knowledge suffer the drawbacks of low precision and low recall, since character strings with statistical significance might be phrases or partial phrases instead of words and low frequency new words are hardly identifiable by statistical methods. in addition to statistical information, we try to use as much information as possible, such as morphology, syntax, semantics, and world knowledge. the identification system fully utilizes the context and content information of unknown words in the steps of detection process, extraction process, and verification process. a practical unknown word extraction system was implemented which online identifies new words, including low frequency new words, with high precision and high recall rates.
an acquisition model for both choosing and resolving anaphora in conjoined mandarin chinese sentences. anaphoric reference is an important linguistic phenomenon to understand the discourse structure and content. in chinese natural language processing, there are both the problems of choosing and resolving anaphora. in mandarin chinese, several linguists have attempted to propose criteria to explain the phenomenon of anaphora but with controversial results. on the other hand, search-based computational techniques for resolving anaphora are neither the best way to resolve chinese anaphora nor to facilitate choosing anaphora. thus, to facilitate both choosing and resolving anaphora with accuracy and efficiency, we propose a case-based learning model g-unimem to automatically acquire anaphoric regularity from a sample set of training sentences, which are annotated with a list of features. the regularity acquired from training was then tested and compared with other approaches in both choosing and resolving anaphora.
mining tables from large scale html texts. table is a very common presentation scheme, but few papers touch on table extraction in text data mining. this paper focuses on mining tables from large-scale html texts. table filtering, recognition, interpretation, and presentation are discussed. heuristic rules and cell similarities are employed to identify tables. the f-measure of table recognition is 86.50%. we also propose an algorithm to capture attribute-value relationships among table cells. finally, more structured data is extracted and presented.
recovering latent information in treebanks. many recent statistical parsers rely on a preprocessing step which uses hand-written, corpus-specific rules to augment the training data with extra information. for example, head-finding rules are used to augment node labels with lexical heads. in this paper, we provide machinery to reduce the amount of human effort needed to adapt existing models to new corpora: first, we propose a flexible notation for specifying these rules that would allow them to be shared by different models; second, we report on an experiment to see whether we can use expectation-maximization to automatically fine-tune a set of hand-written rules to a particular corpus.
syntactic ambiguity resolution using a discrimination and robustness oriented adaptive learning algorithm. in this paper, a discrimination and robustness oriented adaptive learning procedure is proposed to deal with the task of syntactic ambiguity resolution. owing to the problem of insufficient training data and approximation error introduced by the language model, traditional statistical approaches, which resolve ambiguities by indirectly and implicitly using maximum likelihood method, fail to achieve high performance in real applications. the proposed method remedies these problems by adjusting the parameters to maximize the accuracy rate directly. to make the proposed algorithm robust, the possible variations between the training corpus and the real tasks are also taken into consideration by enlarging the separation margin between the correct candidate and its competing members. significant improvement has been observed in the test. the accuracy rate of syntactic disambiguation is raised from 46.0% to 60.62% by using this novel approach.
looking for candidate translational equivalents in specialized, comparable corpora. previous attempts at identifying translational equivalents in comparable corpora have dealt with very large 'general language' corpora and words. we address this task in a specialized domain, medicine, starting from smaller non-parallel, comparable corpora and an initial bilingual medical lexicon. we compare the distributional contexts of source and target words, testing several weighting factors and similarity measures. on a test set of frequently occurring words, for the best combination (the jaccard similarity measure with or without tf.idf weighting), the correct translation is ranked first for 20% of our test words, and is found in the top 10 candidates for 50% of them. an additional reverse-translation filtering step improves the precision of the top candidate translation up to 74%, with a 33% recall.
an augmented chart data structure with efficient word lattice parsing scheme in speech recognition applications. in this paper, an augmented chart data structure with efficient word lattice parsing scheme in speech recognition applications is proposed. the augmented chart and the associated parsing algorithm can represent and parse very efficiently a lattice of word hypotheses produced in speech recognition with high degree of lexical ambiguity without changing the fundamental principles of chart parsing. every word lattice can be mapped to the augmented chart with the ordering and connection relation among word hypotheses being well preserved in the augmented chart. a jump edge is defined to link edges representing word hypotheses physically separated but practically possible to be connected. preliminary experimental results show that with the augmented chart parsing all possible constituents of the input word lattice can be constructed and no constituent needs to be built more than once. this will reduce the computation complexity significantly especially when serious lexical ambiguity exists in the input word lattice as in many speech recognition problems. this augmented chart parsing is thus a very useful and efficient approach to language processing problems in speech recognition applications.
named entity recognition: a maximum entropy approach using global information. this paper presents a maximum entropy-based named entity recognizer (ner). it differs from previous machine learning-based ners in that it uses information from the whole document to classify each word, with just one classifier. previous work that involves the gathering of information from the whole document often uses a secondary classifier, which corrects the mistakes of a primary sentence-based classifier. in this paper, we show that the maximum entropy framework is able to make use of global information directly, and achieves performance that is comparable to the best previous machine learning-based ners on muc-6 and muc-7 test data.
an english-to-korean machine translator: mates/ek. this note introduces an english-to-korean machine translation system mates/ek, which has been developed as a research prototype and is still under upgrading in kaist (korea advanced institute of science and technology). mates/ek is a transfer-based system and it has several subsystems that can be used to support other mt-developments. they are grammar developing environment systems, dictionary developing tools, a set of augmented context free grammars for english syntactic analysis, and so on.
recherches sur la representation des connaissances le systeme arches. ce papier pr&eacute;sente la description formelle d'un syst&egrave;me symbolique de repr&eacute;sentation de connaissances, le syst&egrave;me arches. comme tout syst&egrave;me formel, il est form&eacute; de deux composantes interd&eacute;pendantes. la premi&egrave;re est relative aux modalit&eacute;s de repr&eacute;sentation des connaissances qui sont d&eacute;termin&eacute;es par le langage objet de arches et l'organisation alg&eacute;brique de ses &eacute;l&eacute;ments. la deuxi&egrave;me est relative &agrave; son activit&eacute; inf&eacute;rentielle qui permet de mettre en oeuvre deux types de raisonnement: le raisonnement d&eacute;ductif pour lequel un principe de r&eacute;solution a &eacute;t&eacute; d&eacute;fini; et le raisonnement analogique fond&eacute; sur un mod&egrave;le analogique particulier qui rend consistant le syst&egrave;me arches.