waashk/dblp huggingface dataset:
synchronous morphological analysis of grapheme and phoneme for japanese ocr. we developed a novel language model for japanese based on grapheme-phoneme tuples, which is one order of magnitude smaller than word-based models. we also developed an alignment algorithm of graphemes and phonemes for both ordinary text and ocr output. we show, by experiment, that the combination of the grapheme-phoneme tuple ngram model and the grapheme-phoneme alignment algorithm significantly improve character recognition accuracy if both grapheme and phoneme representations are given.
time period identification of events in text. this study aims at identifying when an event written in text occurs. in particular, we classify a sentence for an event into four time-slots; morning, daytime, evening, and night. to realize our goal, we focus on expressions associated with time-slot (time-associated words). however, listing up all the time-associated words is impractical, because there are numerous time-associated expressions. we therefore use a semi-supervised learning method, the na&iuml;ve bayes classifier backed up with the expectation maximization algorithm, in order to iteratively extract time-associated words while improving the classifier. we also propose to use support vector machines to filter out noisy instances that indicates no specific time period. as a result of experiments, the proposed method achieved 0.864 of accuracy and outperformed other methods.
japanese ocr error correction using character shape similarity and statistical language model. we present a novel ocr error correction method for languages without word delimiters that have a large character set, such as japanese and chinese. it consists of a statistical ocr model, an approximate word matching method using character shape similarity, and a word segmentation algorithm using a statistical language model. by using a statistical ocr model and character shape similarity, the proposed error corrector outperforms the previously published method. when the baseline character recognition accuracy is 90%, it achieves 97.4% character recognition accuracy.
integration of large-scale linguistic resources in a natural language understanding system. knowledge acquisition is a serious bottleneck for natural language understanding systems. for this reason, large-scale linguistic resources have been compiled and made available by organizations such as the linguistic data consortium (comlex) and princeton university (wordnet). systems making use of these resources can greatly accelerate the development process by avoiding the need for the developer to re-create this information.in this paper we describe how we integrated these large scale linguistic resources into our natural language understanding system. client-server architecture was used to make a large volume of lexical information and a large knowledge base available to the system at development and/or run time. we discuss issues of achieving compatibility between these disparate resources.
a part of speech estimation method for japanese unknown words using a statistical model of morphology and context. we present a statistical model of japanese unknown words consisting of a set of length and spelling models classified by the character types that constitute a word. the point is quite simple: different character sets should be treated differently and the changes between character types are very important because japanese script has both ideograms like chinese <i>(kanji)</i> and phonograms like english <i>(katakana)</i>. both word segmentation accuracy and part of speech tagging accuracy are improved by the proposed model. the model can achieve 96.6% tagging accuracy if unknown words are correctly segmented.
an empirical model of acknowledgement for spoken-language systems. we refine and extend prior views of the description, purposes, and contexts-of-use of acknowledgment acts through empirical examination of the use of acknowledgments in task-based conversation. we distinguish three broad classes of acknowledgments (other&larr;ackn, self&larr;other&larr;ackn, and self+ackn) and present a catalogue of 13 patterns within these classes that account for the specific uses of acknowledgment in the corpus.
question answering with lexical chains propagating verb arguments. this paper describes an algorithm for propagating verb arguments along lexical chains consisting of wordnet relations. the algorithm creates verb argument structures using verbnet syntactic patterns. in order to increase the coverage, a larger set of verb senses were automatically associated with the existing patterns from verbnet. the algorithm is used in an in-house question answering system for re-ranking the set of candidate answers. tests on factoid questions from trec 2004 indicate that the algorithm improved the system performance by 2.4%.
a clustered global phrase reordering model for statistical machine translation. in this paper, we present a novel global reordering model that can be incorporated into standard phrase-based statistical machine translation. unlike previous local reordering models that emphasize the reordering of adjacent phrase pairs (till-mann and zhang, 2005), our model explicitly models the reordering of long distances by directly estimating the parameters from the phrase alignments of bilingual training sentences. in principle, the global phrase reordering model is conditioned on the source and target phrases that are currently being translated, and the previously translated source and target phrases. to cope with sparseness, we use n-best phrase alignments and bilingual phrase clustering, and investigate a variety of combinations of conditioning factors. through experiments, we show, that the global reordering model significantly improves the translation accuracy of a standard japanese-english translation task.
classification of modality function and its application to japanese language analysis. this paper proposes an analysis method for japanese modality. in this purpose, meaning of japanese modality is classified into four semantic categories and the role of it is formalized into five modality functions. based on these formalizations, information and constraints to be applied to the modality analysis procedure are specified. then by combining these investigations with case analysis, the analysis method is proposed. this analysis method has been applied to japanese analysis for machine translation.
temporal inferences in medical texts. the objectives of this paper are twofold, whereby the computer program is meant to be a particular implementation of a general natural language [nl] processing system [nlps] which could be used for different domains. the first objective is to provide a theory for processing temporal information contained in a well-structured, technical text. the second objective is to argue for a knowledge-based approach to nlp in which the parsing procedure is driven by extra linguistic knowledge.the resulting computer program incorporates enough domain-specific and general knowledge so that the parsing procedure can be driven by the knowledge base of the program, while at the same time employing a descriptively adequate theory of syntactic processing, i.e., x-bar syntax. my parsing algorithm not only supports the prevalent theories of knowledge-based parsing put forth in ai, but also uses a sound linguistic theory for the necessary syntactic information processing.
revision learning and its application to part-of-speech tagging. this paper presents a revision learning method that achieves high performance with small computational cost by combining a model with high generalization capacity and a model with small computational cost. this method uses a high capacity model to revise the output of a small cost model. we apply this method to english part-of-speech tagging and japanese morphological analysis, and show that the method performs well.
large-scale induction and evaluation of lexical resources from the penn-ii treebank. in this paper we present a methodology for extracting subcategorisation frames based on an automatic lfg f-structure annotation algorithm for the penn-ii treebank. we extract abstract syntactic function-based subcategorisation frames (lfg semantic forms), traditional cfg category-based subcategorisation frames as well as mixed function/category-based frames, with or without preposition information for obliques and particle information for particle verbs. our approach does not predefine frames, associates probabilities with frames conditional on the lemma, distinguishes between active and passive frames, and fully reflects the effects of long-distance dependencies in the source data structures. we extract 3586 verb lemmas, 14348 semantic form types (an average of 4 per lemma) with 577 frame types. we present a large-scale evaluation of the complete set of forms extracted against the full comlex resource.
guessing parts-of-speech of unknown words using global information. in this paper, we present a method for guessing pos tags of unknown words using local and global information. although many existing methods use only local information (i.e. limited window size or intra-sentential features), global information (extra-sentential features) provides valuable clues for predicting pos tags of unknown words. we propose a probabilistic model for pos guessing of unknown words using global information as well as local information, and estimate its parameters using gibbs sampling. we also attempt to apply the model to semi-supervised learning, and conduct experiments on multiple corpora.
whose thumb is it anyway? classifying author personality from weblog text. we report initial results on the relatively novel task of automatic classification of author personality. using a corpus of personal weblogs, or 'blogs', we investigate the accuracy that can be achieved when classifying authors on four important personality traits. we explore both binary and multiple classification, using differing sets of n-gram features. results are promising for all four traits examined.
hypertext authoring for linking relevant segments of related instruction manuals. recently manuals of industrial products become large and often consist of separated volumes. in reading such individual but related manuals, we must consider the relation among segments, which contain explanations of sequences of operation. in this paper, we propose methods for linking relevant segments in hypertext authoring of a set of related manuals. our method is based on the similarity calculation between two segments. our experimental results show that the proposed method improves both recall and precision comparing with the conventional tf.idf based method.
minimum error rate training in statistical machine translation. often, the training procedure for statistical machine translation models is based on maximum likelihood or related criteria. a general problem of this approach is that there is only a loose relation to the final translation quality on unseen text. in this paper, we analyze various training criteria which directly optimize translation quality. these training criteria make use of recently proposed automatic evaluation metrics. we describe a new algorithm for efficient training an unsmoothed error count. we show that significantly better results can often be obtained if the final evaluation criterion is taken directly into account as part of the training procedure.
a polynomial-time algorithm for statistical machine translation. we introduce a polynomial-time algorithm for statistical machine translation. this algorithm can be used in place of the expensive, slow best-first search strategies in current statistical translation architectures. the approach employs the stochastic bracketing transduction grammar (sbtg) model we recently introduced to replace earlier word alignment channel models, while retaining a bigram language model. the new algorithm in our experience yields major speed improvement with no significant loss of accuracy.
improved statistical alignment models. in this paper, we present and compare various single-word based alignment models for statistical machine translation. we discuss the five ibm alignment models, the hidden-markov alignment model, smoothing techniques and various modifications. we present different methods to combine alignments. as evaluation criterion we use the quality of the resulting viterbi alignment compared to a manually produced reference alignment. we show that models with a first-order dependence and a fertility model lead to significantly better results than the simple models ibm-1 or ibm-2, which are not able to go beyond zero-order dependencies.
constraint projection: an efficient treatment of disjunctive feature descriptions. unification of disjunctive feature descriptions is important for efficient unification-based parsing. this paper presents constraint projection, a new method for unification of disjunctive feature structures represented by logical constraints. constraint projection is a generalization of constraint unification, and is more efficient because constraint projection has a mechanism for abandoning information irrelevant to a goal specified by a list of variables.
discriminative training and maximum entropy models for statistical machine translation. we present a framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source-channel approach as a special case. all knowledge sources are treated as feature functions, which depend on the source language sentence, the target language sentence and possible hidden variables. this approach allows a baseline machine translation system to be extended easily by adding new feature functions. we show that a baseline statistical machine translation system is significantly improved using this approach.
understanding unsegmented user utterances in real-time spoken dialogue systems. this paper proposes a method for incrementally understanding user utterances whose semantic boundaries are not known and responding in real time even before boundaries are determined. it is an integrated parsing and discourse processing method that updates the partial result of understanding word by word, enabling responses based on the partial result. this method incrementally finds plausible sequences of utterances that play crucial roles in the task execution of dialogues, and utilizes beam search to deal with the ambiguity of boundaries as well as syntactic and semantic ambiguities. the results of a preliminary experiment demonstrate that this method understands user utterances better than an understanding method that assumes pauses to be semantic boundaries.
improving statistical natural language translation with categories and rules. this paper describes an all level approach on statistical natural language translation (snlt). without any predefined knowledge the system learns a statistical translation lexicon (stl), word classes (wcs) and translation rules (trs) from a parallel corpus thereby producing a generalized form of a word alignment (wa). the translation process itself is realized as a beam search. in our method example-based techniques enter an overall statistical approach leading to about 50 percent correctly translated sentences applied to the very difficult english-german verbmobil spontaneous speech corpus.
towards a model of face-to-face grounding. we investigate the verbal and nonverbal means for grounding, and propose a design for embodied conversational agents that relies on both kinds of signals to establish common ground in human-computer interaction. we analyzed eye gaze, head nods and attentional focus in the context of a direction-giving task. the distribution of nonverbal behaviors differed depending on the type of dialogue move being grounded, and the overall pattern reflected a monitoring of lack of negative feedback. based on these results, we present an eca that uses verbal and nonverbal grounding acts to update dialogue state.
dependency parsing with an extended finite state approach. this article presents a dependency parsing scheme using an extended finite-state approach. the parser augments input representation with "channels" so that links representing syntactic dependency relations among words can be accommodated and iterates on the input a number of times to arrive at a fixed point. intermediate configurations violating various constraints of projective dependency representations such as no crossing links and no independent items except sentential head are filtered via finite-state filters. we have applied the parser to dependency parsing of turkish.
translating a unification grammar with disjunctions into logical constraints. this paper proposes a method for generating a logical constraint-based internal representation from a unification grammar formalism with disjunctive information. unification grammar formalisms based on path equations and lists of pairs of labels and values are better than those based on first-order terms in that the former is easier to describe and to understand. parsing with term-based internal representations is more efficient than parsing with graph-based representations. therefore, it is effective to translate unification grammar formalism based on path equations and lists of pairs of labels and values into a term-based internal representation. previous translation methods cannot deal with disjunctive feature descriptions, which reduce redundancies in the grammar and make parsing efficient. since the proposed method translates a formalism without expanding disjunctions, parsing with the resulting representation is efficient.
morphological disambiguation by voting constraints. we present a constraint-based morphological disambiguation system in which individual constraints vote on matching morphological parses, and disambiguation of all the tokens in a sentence is performed at the end by selecting parses that receive the highest votes. this constraint application paradigm makes the outcome of the disambiguation independent of the rule sequence, and hence relieves the rule developer from worrying about potentially conflicting rule sequencing. our results for disambiguating turkish indicate that using about 500 constraint rules and some additional simple statistics, we can attain a recall of 95--96% and a precision of 94--95% with about 1.01 parses per token. our system is implemented in prolog and we are currently investigating an efficient implementation based on finite state transducers.
an algorithm for one-page summarization of a long text based on thematic hierarchy detection. this paper presents an algorithm for text summarization using the thematic hierarchy of a text. the algorithm is intended to generate a one-page summary for the user, thereby enabling the user to skim large volumes of an electronic book on a computer display. the algorithm first detects the thematic hierarchy of a source text with lexical cohesion measured by term repetitions. then, it identifies boundary sentences at which a topic of appropriate grading probably starts. finally, it generates a structured summary indicating the outline of the thematic hierarchy. this paper mainly describes and evaluates the part for boundary sentence identification in the algorithm, and then briefly discusses the readability of one-page summaries.
term recognition using technical dictionary hierarchy. in recent years, statistical approaches on atr (automatic term recognition) have achieved good results. however, there are scopes to improve the performance in extracting terms still further. for example, domain dictionaries can improve the performance in atr. this paper focuses on a method for extracting terms using a dictionary hierarchy. our method produces relatively good results for this task.
dependency parsing of japanese spoken monologue based on clause boundaries. spoken monologues feature greater sentence length and structural complexity than do spoken dialogues. to achieve high parsing performance for spoken monologues, it could prove effective to simplify the structure by dividing a sentence into suitable language units. this paper proposes a method for dependency parsing of japanese monologues based on sentence segmentation. in this method, the dependency parsing is executed in two stages: at the clause level and the sentence level. first, the dependencies within a clause are identified by dividing a sentence into clauses and executing stochastic dependency parsing for each clause. next, the dependencies over clause boundaries are identified stochastically, and the dependency structure of the entire sentence is thus completed. an experiment using a spoken monologue corpus shows this method to be effective for efficient dependency parsing of japanese monologue sentences.
constituent-based accent prediction. near-perfect automatic accent assignment is attainable for citation-style speech, but better computational models are needed to predict accent in extended, spontaneous discourses. this paper presents an empirically motivated theory of the discourse focusing nature of accent in spontaneous speech. hypotheses based on this theory lead to a new approach to accent prediction, in which patterns of deviation from citation form accentuation, defined at the constituent or noun phrase level, are automatically learned from an annotated corpus. machine learning experiments on 1031 noun phrases from eighteen spontaneous direction-giving monologues show that accent assignment can be significantly improved by up to 4%-6% relative to a hypothetical baseline system that would produce only citation-form accentuation, giving error rate reductions of 11%-25%.
automatic extraction of aspectual information from a monolingual corpus. this paper describes an approach to extract the aspectual information of japanese verb phrases from a monoligual corpus. we classify verbs into six categories by means of the aspectual features which are defined on the basis of the possibility of co-occurrence with aspectual forms and adverbs. a unique category could be identified for 96% of the target verbs. to evaluate the result of the experiment, we examined the meaning of -teiru which is one of the most fundamental aspectual markers in japanese, and obtained the correct recognition score of 71% for the 200 sentences.
a speech-first model for repair detection and correction. interpreting fully natural speech is an important goal for spoken language understanding systems. however, while corpus studies have shown that about 10% of spontaneous utterances contain self-corrections, or repairs, little is known about the extent to which cues in the speech signal may facilitate repair processing. we identify several cues based on acoustic and prosodic analysis of repairs in a corpus of spontaneous speech, and propose methods for exploiting these cues to detect and correct repairs. we test our acoustic-prosodic cues with other lexical cues to repair identification and find that precision rates of 89--93% and recall of 78--83% can be achieved, depending upon the cues employed, from a prosodically labeled corpus.
learning to say it well: reranking realizations by predicted synthesis quality. this paper presents a method for adapting a language generator to the strengths and weaknesses of a synthetic voice, thereby improving the naturalness of synthetic speech in a spoken language dialogue system. the method trains a discriminative reranker to select paraphrases that are predicted to sound natural when synthesized. the ranker is trained on realizer and synthesizer features in supervised fashion, using human judgements of synthetic voice quality on a sample of the paraphrases representative of the generator's capability. results from a cross-validation study indicate that discriminative paraphrase reranking can achieve substantial improvements in naturalness on average, ameliorating the problem of highly variable synthesis quality typically encountered with today's unit selection synthesizers.