waashk/dblp huggingface dataset:
english-to-korean transliteration using multiple unbounded overlapping phoneme chunks. we present in this paper the method of english-to-korean (e-k) transliteration and back-transliteration. in korean technical documents, many english words are transliterated into korean words in various forms in diverse ways. as english words and korean transliterations are usually technical terms and proper nouns, it is hard to find a transliteration and its variations in a dictionary. therefore an automatic transliteration system is needed to find the transliterations of english words without manual intervention.to explain e-k transliteration phenomena, we use phoneme chunks that do not have a length limit. by applying phoneme chunks, we combine different length information with easy. the e-k transliteration method has three steps. in the first, we make a phoneme network that shows all possible transliterations of the given word. in the second step, we apply phoneme chunks, extracted from training data, to calculate the reliability of each possible transliteration. then we obtain probable transliterations of the given english word.
syllable-based model for tiie korean morphology. this paper describes a syllable-based computational model for the korean morphology. in this model, morphological analysis is considered as a process of candidate generation and candidate selection. in order to increase the performance of the system, the number of candidates is highly reduced and the system requires small number of dictionary accesses. idiosynchratic features of a syllable, formalized as a characteristic function, make it possible to reject implausible candidates before dictionary confirmation. instead of a letter, syllable is a basic processing unit for the practical implementation of the morphological analyzer.
the rumors system of russian synthesis. the rumors synthesizer of russian is an integral part of the jarap experimental system of japanese-russian automatic translation, although it can also have other applications. morphologically, it is based, primarily, on a. a. zaliznyak's model of russian inflexion. syntactical functions of rumors rely on word-order and dependency data as input information. the synthesizer is implemented on ibm pc, ms dos, in turbo pascal.
lfg generation produces context-free languages. this paper examines the generation problem for a certain linguistically relevant subclass of lfg grammars. our main result is that the set of strings that such a grammar relates to a particular f-structure is a context-free language. this result obviously extends to other context-free based grammatical formalisms, such as patr, and also to formalisms that permit a context-free skeleton to be extracted (perhaps some variants of hpsg). the proof is constructive: from the given f-structure a particular context-free grammar is created whose yield is the desired set of strings. many existing generation strategies (top-down, bottom-up, head-driven) can be understood as alternative ways of avoiding the creation of useless context-free productions. our result can be established for the more general class of lfg grammars, but that is beyond the scope of the present paper.
recognizing text genres with simple metrics using discriminant analysis. a simple method for categorizing texts into pre-determined text genre categories using the statistical standard technique of discriminant analysis is demonstrated with application to the brown corpus. discriminant analysis makes it possible use a large number of parameters that may be specific for a certain corpus or information stream, and combine them into a small number of functions, with the parameters weighted on basis of how useful they are for discriminating text genres. an application to information retrieval is discussed.
vocnets - a tool for handling finite vocabularies. a method is proposed for storing a finite vocabulary in a manner which makes it convenient to recognize words and substrings of words. the representation, which can be generated automatically from a list of words or from given representations of other sets by means of which the vocabulary has been defined through set or string operations, has the form of a modified finite-state grammar, a form eliminating the multiplicative effects of conjunction, complementation, etc., on the node sets of conventional finite-state representations.
a freely available wide coverage morphological analyzer for english. this paper presents a morphological lexicon for english that handle more than 317000 inflected forms derived from over 90000 stems. the lexicon is available in two formats. the first can be used by an implementation of a two-level processor for morphological analysis (karttunen and wittenburg, 1983; antworth, 1990). the second, derived from the first one for efficiency reasons, consists of a disk-based database using a unix hash table facility (seltzer and yigit, 1991). we also built an x window tool to facilitate the maintenance and browsing of the lexicon. the package is ready to be integrated into an natural language application such as a parser through hooks written in lisp and c.to our knowledge, this package is the only available free english morphological analyzer with very wide coverage.
features and values. the paper discusses the linguistic aspects of a new general purpose facility for computing with features. the program was developed in connection with the course i taught at the university of texas in the fall of 1983. it is a generalized and expanded version of a system that stuart shieber originally designed for the patr-ii project at sri in the spring of 1983 with later modifications by fernando pereira and me. like its predecessors, the new texas version of the "dg (directed graph)" package is primarily intended for representing morphological and syntactic information but it may turn out to be very useful for semantic representations too.
an experimental parser for syntemic grammars. we describe a general parsing method for systemic grammars. systemic grammars contain a paradigmatic analysis of language in addition to structural information, so a parser must assign a set of grammatical features and functions to each constituent in addition to producing a constituent structure. our method constructs a parser by compiling systemic grammars into the notation of functional unification grammar. the existing methods for parsing with unification grammars have been extended to handle a fuller range of paradigmatic descriptions. in particular, the patr-ii system has been extended by using disjunctive and conditional information in functional descriptions that are attached to phrase structure rules. the method has been tested with a large grammar of english which was originally developed for text generation. this testing is the basis for some observations about the bidirectional use of a grammar.
modularizing codescriptive grammars for efficient parsing. unification-based theories of grammar allow to integrate different levels of linguistic descriptions in the common framework of typed feature structures. dependencies among the levels are expressed by coreferences. though highly attractive theoretically, using such codescriptions for analysis creates problems of efficiency. we present an approach to a modular use of codescriptions on the syntactic and semantic level. grammatical analysis is performed by tightly coupled parsers running in tandem, each using only designated parts of the grammatical description. in the paper we describe the partitioning of grammatical information for the parsers and present results about the performance.
finite-state description of semitic morphology: a case study of ancient accadian. this paper discusses the problems of description and computational implementation of phonology and morphology in semitic languages, using ancient akkadian as an example. phonological and morphophonological variations are described using standard finite-state two-level morphological rules. interdigitation, prefixation and suffixation are described by using an intersection of two lexicons which effectively defines lexical representations of words.
answering it with charts: dialogue in natural language and charts. a methodology is proposed for taking queries and requests expressed in natural language as input and answering them in charts through organizing that interaction into felicitous dialogue. charts and graphics, as well as languages, are important modes of communication. this is especially true of those which are used frequently when people analyze huge amount of data interactively, in order to find out its characteristics or to resolve questions about it. this paper raises the problem that in such situations the correctness of the charts depends on the context, and proposes a framework to resolve it. the core of the framework is a logical form that includes the specifications of the user's perspective and the proper treatment of the logical form for handling utterance fragments. the framework has been implemented and confirmed to be appropriate.
exploiting lexical regularities in designing natural language systems. this paper presents the lexical component of the start question answering system developed at the mit artificial intelligence laboratory. start is able to interpret correctly a wide range of semantic relationships associated with a alternate expressions of the arguments of verbs. the design of the system takes advantage of the results of recent linguistic research into the structure of the lexicon, allowing start to attain a broader range of coverage than many existing systems while maintaining modular organization.
linguistic error correction of japanese sentences. this paper describes a newly developed linguistic error correction system, which can correct errors and rejections of japanese sentences by using linguistic knowledge.conventional optical character readers (ocr) need human assistance to correct their recognition errors and rejections. an operator must teach the ocr correct answers whenever an illegible character pattern occurs. if this error correction operation is mechanized, the throughput of the ocr will increase.this linguistic error correction system offers means of automated error correction by analysing sentences of the ocr outputs linguistically. this system grammatically selects legal letters from the candidates which can not be decided uniquely by pattern recognition only, and recommends grammatically and semantically meaningful letters for the illegible letter.
fertilization of case frame dictionary for robust japanese case analysis. this paper proposes a method of fertilizing a japanese case frame dictionary to handle complicated expressions: double nominative sentences, non-gapping relation of relative clauses, and case change. our method is divided into two stages. in the first stage, we parse a large corpus and construct a japanese case frame dictionary automatically from the parse results. in the second stage, we apply case analysis to the large corpus utilizing the constructed case frame dictionary, and upgrade the case frame dictionary by incorporating newly acquired information.
anaphora for everyone: pronominal anaphora resolution without a parser. we present an algorithm for anaphora resolution which is a modified and extended version of that developed by (lappin and leass, 1994). in contrast to that work, our algorithm does not require in-depth, full, syntactic parsing of text. instead, with minimal compromise in output quality, the modifications enable the resolution process to work from the output of a part of speech tagger, enriched only with annotations of grammatical function of lexical items in the input text stream. evaluation of the results of our implementation demonstrates that accurate anaphora resolution can be realized within natural language processing frameworks which do not---or cannot--- employ robust and reliable parsing components.
probabilistic tagging with feature structures. the described tagger is based on a hidden markov model and uses tags composed of features such as part-of-speech, gender, etc. the contextual probability of a tag (state transition probaility) is deduced from the contextual probabilities of its feature-value-pairs.this approach is advantageous when the available training corpus is small and the tag set large, which can be the case with morphologically rich languages.
aligning more words with high precision for small bilingual corpora. in this paper, we propose an algorithm for aligning words with their translation in a bilingual corpus. conventional algorithms are based on word-by-word models which require bilingual data with hundreds of thousand sentences for training. by using a word-based approach, less frequent words or words with diverse translations generally do not have statistically significant evidence for confident alignment. consequently, incomplete or incorrect alignments occur. our algorithm attempts to handle the problem using class-based rules which are automatic acquired from bilingual materials such as a bilingual corpus or machine readable dictionary. the procedures for acquiring these rules is also described. we found that the algorithm can align over 80% of word pairs while maintaining a comparably high precision rate, even when a small corpus was used in training. the algorithm also poses the advantage of producing a tagged corpus for word sense disambiguation.
parallel replacement in finite state calculus. this paper extends the calculus of regular expressions with new types of replacement expressions that enhance the expressiveness of the simple replace operator defined in karttunen (1995). parallel replacement allows multiple replacements to apply simultaneously to the same input without interfering with each other. we also allow a replacement to be constrained by any number of alternative contexts. with these enhancements, the general replacement expressions are more versatile than two-level rules for the description of complex morphological alternations.
a spelling correction program based on a noisy channel model. this paper describes a new program, correct, which takes words rejected by the unix&reg; spell program, proposes a list of candidate corrections, and sorts them by probability. the probability scores are the novel contribution of this work. probabilities are based on a noisy channel model. it is assumed that the typist knows what words he or she wants to type but some noise is added on the way to the keyboard (in the form of typos and spelling errors). using a classic bayesian argument of the kind that is popular in the speech recognition literature (jelinek, 1985), one can often recover the intended correction, c, from a typo, t, by finding the correction c that maximizes pr(c) pr(t/c). the first factor, pr(c), is a prior model of word probabilities; the second factor, pr(t/c), is a model of the noisy channel that accounts for spelling transformations on letter sequences (e.g., insertions, delections, substitutions and reversals). both sets of probabilities were trained on data collected from the associated press (ap) newswire. this text is ideally suited for this purpose since it contains a large number of typos (about two thousand per month).
generation of informative texts with style. an approach to the computational treatment of style is presented in the case of generation of informative texts. we regard the style mostly as a means of controlled selection of alternatives faced at each level of text generation. the generation technique, as well as the style specification, are considered at four levels --- content production, discourse generation, surface structure development, and lexical choice. a style is specified by the frequency of occurrence of certain features examined through observation of particular texts. the algorithm for text generation ensures efficient treatment of the style requirements.
incremental sentence generation: implications for the structure of a syntactic processor. human speakers often produce sentences incrementally. they can start speaking having in mind only a fragmentary idea of what they want to say, and while saying this they refine the contents underlying subsequent parts of the utterance. this capability imposes a number of constraints on the design of a syntactic processor. this paper explores these constraints and evaluates some recent computational sentence generators from the perspective of incremental production.
tsnlp - test suites for natural language processing. the growing language technology industry needs measurement tools to allow researchers, engineers, managers, and customers to track development, evaluate and assure quality, and assess suitability for a variety of applications.the tsnlp (test suites for natural language processing) project has investigated various aspects of the construction, maintenance and application of systematic test suites as diagnostic and evaluation tools for nlp applications. the paper summarizes the motivation and main results of tsnlp: besides the solid methodological foundation of the project, tsnlp has produced substantial (i.e. larger than any existing general test suites) multi-purpose and multi-user test suites for three european languages together with a set of specialized tools that facilitate the construction, extension, maintenance, retrieval, and customization of the test data.the publicly available results of tsnlp represent a valuable linguistic resource that has the potential of providing a wide-spread pre-standard diagnostic and evaluation tool for both developers and users of nlp applications.
intelligent handling of weather forecasts. some typical cases of intelligent handling of weather forecasts such as translation, visualization, etc. are decomposed into two subprocesses--analysis and synthesis. specific techniques are presented for analysis and synthesis of weather forecast texts as well as for generation of weather maps. these techniques deal with the weather forecasts at different levels--syntactic, discourse and semantic. they are based on a conceptual model underlying weather forecasts as well as on formal descriptions of the means of expression used in particular natural and cartographic sublanguages.
automatic proofreading of frozen phrases in german. frozen phrases are introduced as a new level of automatic proofreading in between the standard level of spelling verification of isolated words and the level of grammar checking. the design and the implementation of a corresponding proofreading system are described in detail.
topic-comment structure of texts (and its contribution to the automatic processing of texts). in general, topic is informally defined as that part of the sentence which the sentence is about and comment as what is said in the sentence. this is, of course, not a formal definition but so far noone has succeeded to provide a viable formal definition for these notions. in the analysis of sentences and texts these notions can nevertheless be used with considerable success.
a novel disambiguation method for unification-based grammars using probabilistic context-free approximations. we present a novel disambiguation method for unification-based grammars (ubgs). in contrast to other methods, our approach obviates the need for probability models on the ubg side in that it shifts the responsibility to simpler context-free models, indirectly obtained from the ubg. our approach has three advantages: (i) training can be effectively done in practice, (ii) parsing and disambiguation of context-free readings requires only cubic time, and (iii) involved probability distributions are mathematically clean. in an experiment for a mid-size ubg, we show that our novel approach is feasible. using unsupervised training, we achieve 88% accuracy on an exact-match task.
an hpsg-to-cfg approximation of japanese. we present a simple approximation method for turning a head-driven phrase structure grammar into a context-free grammar. the approximation method can be seen as the construction of the least fixpoint of a certain monotonic function. we discuss an experiment with a large hpsg for japanese.
feature structure based semantic head driven generation. this paper proposes a generation method for feature-structure-based unification grammars. as compared with fixed arity term notation. feature structure notation is more flexible for representing knowledge needed to generate idiomatic structures as well as general constructions. the method enables feature structure retrieval via multiple indices. the indexing mechanism, when used with a semantic head driven generation algorithm, attains efficient generation even when a large amount of generation knowledge must be considered. our method can produce all possible structures in parallel, using structure sharing among ambiguous substructures.
identifying the coding system and language of on-line documents on the internet. this paper proposes a new algorithm that simultaneously identifies the coding system and language of a code string fetched from the internet, especially world-wide web. the algorithm uses statistic language models to select the correctly decoded string as well as to determine the language. the proposed algorithm covers 9 languages and 11 coding systems used in eastern asia and western europe. experimental results show that the level of accuracy of our algorithm is over 95% for 640 on-line documents.
parsing with category cooccurance restrictions. this paper summarizes the formalism of category cooccurrence restrictions (ccrs) and describes two parsing algorithms that interpret it. ccrs are boolean conditions on the cooccurrence of categories in local trees which allow the statement of generalizations which cannot be captured in other current syntax formalisms. the use of ccrs leads to syntactic descriptions formulated entirely with restrictive statements. the paper shows how conventional algorithms for the analysis of context free languages can be adapted to the ccr formalism. special attention is given to the part of the parser that checks the fulfillment of logical well-formedness conditions on trees.
qpatr and constraint threading. qpatr is an ms-dos arity/prolog implementation of the patr-ii formalism for unification grammar. the formalism has been extended to include the constraints of lfg as well as negation and disjunction, which are implemented with the disjunction and negation-as-failure of prolog itself. a technique of constraint threading is employed to collect negative and constraining conditions in prolog difference lists. the parser of qpatr uses a left-corner algorithm for context-free grammars and includes a facility for identifying new lexical items in input on the basis of contextual information.
top-down predictive linking and complex-feature-based formalisms. automatic compilation of the linking relation employed in certain parsing algorithms for context-free languages is examined. special problems arise in the extension of these algorithms to the possibly infinite domain of feature structures. a technique is proposed which is designed specifically for left-recursive categories and is based on the generalization of their occurrences in a derivation. particular attention is drawn to the top-down predictive character of the linking relation and to its significance not only as a filter for increasing the efficiency of syntactic analysis but as a device for the top-down instantiation of information, which then serves as a key to the directed analysis of inflected forms as well as "unknown" or "new" words.