waashk/dblp huggingface dataset:
korean language engineering: current status of the information platform. language engineering implements functions of a language and information via computers. the need for language engineering platforms has been generally recognized and several researches are being undertaken around the world. our goal is to establish korean information platform of linguistic resources and tools for korean language and information communities. the platform will support, researchers and engineers with well-developed and standardized resources and application tools thereby avoiding duplicate activities from scratch and amplifying overall effort on the domain. this paper reports the components and the current status of the project, and the importance of the effort.
a comparative evaluation of data-driven models in translation selection of machine translation. we present a comparative evaluation of two data-driven models used in translation selection of english-korean machine translation. latent semantic analysis(lsa) and probabilistic latent semantic analysis (plsa) are applied for the purpose of implementation of data-driven models in particular. these models are able to represent complex semantic structures of given contexts, like text passages. grammatical relationships, stored in dictionaries, are utilized in translation selection essentially. we have used k-nearest neighbor (k-nn) learning to select an appropriate translation of the unseen instances in the dictionary. the distance of instances in k-nn is computed by estimating the similarity measured by lsa and plsa. for experiments, we used trec data(ap news in 1988) for constructing latent semantic spaces of two models and wall street journal corpus for evaluating the translation accuracy in each model. plsa selected relatively more accurate translations than lsa in the experiment, irrespective of the value of k and the types of grammatical relationship.
unsupervised named entity classification models and their ensembles. this paper proposes an unsupervised learning model for classifying named entities. this model uses a training set, built automatically by means of a small-scale named entity dictionary and an unlabeled corpus. this enables us to classify named entities without the cost for building a large hand-tagged training corpus or a lot of rules.our model uses the ensemble of three different learning methods and repeats the learning with new training examples generated through the ensemble learning. the ensemble of various learning methods brings a better result than each individual learning method. the experimental result shows 73.16% in precision and 72.98% in recall for korean news articles.
decision-tree based error correction for statistical phrase break prediction in korean. in this paper, we present a new phrase break prediction architecture that integrates probabilistic approach with decision-tree based error correction. the probabilistic method alone usually suffers from performance degradation due to inherent data sparseness problems and it only covers a limited range of contextual information. moreover, the module can not utilize the selective morpheme tag and relative distance to the other phrase breaks. the decision-tree based error correction was tightly integrated to overcome these limitations.the initially phrase break tagged morpheme sequence is corrected with the error correcting decision tree which was induced by c4.5 from the correctly tagged corpus with the output of the probabilistic predictor. the decision tree-based post error correction provided improved results even with the phrase break predictor that has poor initial performance. moreover, the system can be flexibly tuned to new corpus without massive retraining.
a two-level morphological analysis of korean. the two-level morphology model has received a great deal of attention and has been implemented for languages like finnish, english, japanese, russian, french, and so on. however, this model has been claimed to be inappropriate for korean morphological analysis, because the complex conjugation (inflection) and agglutination in word formation, and the syllable-based representation of words may lead to a huge number of two-level morphological rules. in this paper, we show that the two-level model can be successfully applied to korean and its rule size is limited to only 52. an extension of two-level morphology is described for korean language.
structural feature selection for english-korean statistical machine translation. when aligning texts in very different languages such as korean and english, structural features beyond word or phrase give useful information. in this paper, we present a method for selecting structural features of two languages, from which we construct a model that assigns the conditional probabilities to corresponding tag sequences in bilingual english-korean corpora. for tag sequence mapping between two languages, we first define a structural feature function which represents statistical properties of empirical distribution of a set of training samples. the system, based on maximum entropy concept, selects only features that produce high increases in loglikelihood of training samples. these structurally mapped features are more informative knowledge for statistical machine translation between english and korean. also, the information can help to reduce the parameter space of statistical alignment by eliminating syntactically unlikely alignments.
human factors and linguistic considerations: keys to high-speed chinese character input. with a keyboard and supporting system developed at cornell university, input methods used to identify ideographs are adaptations of well-known schemes; innovation is in the addition of automatic machine selection of ambiguously identified characters.the unique feature of the cornell design is that a certain amount of intelligence has been built into the machine. this allows an operator to take advantage of the fact that about 60% of chinese characters in text are paired with other characters to form two-syllable compounds or phrase words. in speech and writing these pairings eliminate about 95% of the ambiguities created by ambiguously identified syllables.
typed feature structures as descriptions. a description is an entity that can be interpreted as true or false of an object, and using feature structures as descriptions accrues several computational benefits. in this paper, i create an explicit interpretation of a typed feature structure used as a description, define the notion of a satisfiable feature structure, and create a simple and effective algorithm to decide if a feature structure is satisfiable.
improvement in customizability using translation templates. this paper outlines customization of a machine translation system using translation templates, which enable users to represent the bilingual knowledge needed for complex translation. to evaluate their effectiveness, we analyzed a bilingual text to estimate the improvement in customizability. the result shows that about 60% of mistranslated sentences can be translated as model translations by combining the proposed framework with the conventional customizing functions.
interaction between structural changes in machine translation. this paper discusses complex structural changes during transfer within a non-destructive transfer framework. though the description of each individual structural change is not difficult, special provision must be made when they are combined, because interaction between them sometimes causes unexpected problems. transfer of coordinate structures is also discussed as this sometimes necessitates a structural change and interacts with other structural changes in a problematic way.
japanese sentence analysis for automatic indexing. a new method for automatic keyword extracting and "role" setting is proposed based on the japanese sentence structure analysis. the analysis takes into account the following features of japanese sentences, i.e., the structure of a sentence is determined by the noun-predicate verb dependency, and the case indicating words (kaku-joshi) play an important role in deep case structure. by utilizing the meaning of a noun as it depends on each prodicate verb, restricted semantic processing becomes possible. an automatic indexing system, equipped with a man-machine interactive error-correcting function, has been developed. the evaluation of the system is performed by applying it in news information retrieval. the results of this evaluation show that the system can be put to practical use.
hypertags. srinivas (97) enriches traditional morpho-syntactic pos tagging with syntactic information by introducing supertags. unfortunately, words are assigned on average a much higher number of supertags than traditional pos. in this paper, we develop the notion of hypertag, first introduced in kinyon (00a) and in kinyon (00b), which allows to factor the information contained in several supertags into a single structure and to encode functional information in a systematic manner. we show why other possible solutions based on mathematical properties of trees are unsatisfactory and also discuss the practical usefulness of this approach.
portuguese analysis with tree adjoining grammars. this article approaches syntactical analysis of portuguese language based upon a formalism called tree adjoining grammars (tags) [joshi 85]. it briefly describes the formalism and its main operations, outlines a portuguese subset for analysis, and presents a parser developed according tags concepts in order to validate an application of the formalism for this language.
multi-tape two-level morphology: a case study in semitic non-linear morphology. this paper presents an implemented multi-tape two-level model capable of describing semitic non-linear morphology. the computational framework behind the current work is motivated by [kay 1987]; the formalism presented here is an extension to the formalism reported by [pulman and hepple 1993]. the objectives of the current work are: to stay as close as possible, in spirit, to standard two-level morphology, to stay close to the linguistic description of semitic stems, and to present a model which can be used with case by the semitist. the paper illustrates that if finite-state transducers (fsts) in a standard two-level morphology model are replaced with multi-tape auxiliary versions (afsts), one can account for semitic root-and-pattern morphology using high level notation.
computing prosodic morphology. this paper establishes a framework under which various aspects of prosodic morphology, such as templatic morphology and inflxation, can be handled under two-level theory using an implemented multi-tape two-level model. the paper provides a new computational analysis of root-and-pattern morphology based on prosody.
on a device in dictionary operations in machine translation. a special programme converting classes of words of international usage directly from english to czech is described in its application in an experiment of machine translation as well as in general environments. the words undergo special morphemic analysis, they are adapted morphemically and orthographically to the target language form and, in the experimental version, they are assigned pertinent grammatical and semantic information.
tranditional means in machine translation. the chronic problems of machine translation cannot be solved in a fully automatic way. human intervention is inevitable. the development of "traditional" means in connexion with advances of computer technology represent most substantial contribution to further progress in the field of machine translation. some of the problems are illustrated using the example of the apac32 project.
scaled log likelihood ratios for the detection of abbreviations in text corpora. we describe a language-independent, flexible, and accurate method for the detection of abbreviations in text corpora. it is based on the idea that an abbreviation can be viewed as a collocation, and can be identified by using methods for collocation detection such as the log likelihood ratio. although the log likelihood ratio is known to show a good recall, its precision is poor. we employ scaling factors which lead to a strong improvement of precision. experiments with english and german corpora show that abbreviations can be detected with high accuracy.
pattern matching in the textract information extraction system. in information extraction systems, pattern matchers are widely used to identify information of interest in a sentence. in this paper, pattern matching in the textract information extraction system is described. it comprises a concept search which identifies key words representing a concept, and a template pattern search which identifies patterns of words and phrases. textract using the matcher performed well in the tipster/muc-5 evaluation. the pattern matching architecture is also suitable for rapid system development across different domains of the same language.
incremental sentence production with a parallel marker-passing algorithm. this paper describes a method of incremental natural language generation using a parallel marker-passing algorithm for modeling simultaneous interpretation. semantic and syntactic knowledge are represented in a memory network in which several types of markers are passed around in order to make inference, and explore implicit parallelism of sentence production. the model is consistent with several psycholinguistic studies. the model is actually implemented as a part of the &phi;dmdialog real-time speech-to-speech dialog translation system developed at the center for machine translation at carnegie mellon university, and publicly demonstrated since march 1989.
semantic network array processor as a massively parallel computing platform for high performance and large-scale natural language processing. this paper demonstrates the utility of the semantic network array processor (snap) as a massively parallel platform for high performance and large-scale natural language processing systems. snap is an experimental massively parallel machine which is dedicated to, but not limited to, the natural language processing using semantic networks. in designing the snap, we have investigated various natural language processing systems and theories to determine the scope of the hardware support and a set of micro-coded instructions to be provided. as a result, snap employs an extended marker-passing model and a dynamically modifiable network model. a set of primitive instructions is micro-coded to directly support a parallel marker-passing, bitoperations, numeric operations, network modifications, and other essential functions for natural language processing. this paper demonstrates the utility of snap for various paradigms of natural language processing. we have discovered that the snap provides milliseconds or microsectonds performance on several important applications such as the memory-based parsing and translation, classification-based parsing, and vlkb search. also, we argue that there are numerous opportunities in the nlp community to take advantages of the computational power of the snap.
embedded sublanguages and natural language processing. most recent systems for the large-scale intelligent processing of natural language texts are designed to accept only a restricted variety of language. in certain cases this restricted subseto of the language constitutes a <u>sublanguage</u>, for which it may be possible to write a relatively precise and compact <u>sublanguage grammar</u>. several research groups are currently exploiting the restrictions in scientific and technical sublanguage grammars for tasks such as information retrieval and automatic translation.
synthesizing weather forecasts from formatted data. this paper describes a system (rareas) which synthesizes marine weather forecasts directly from formatted weather data. such synthesis appears feasible in certain natural sublanguages with stereotyped text structure. rareas draws on several kinds of linguistic and non-linguistic knowledge and mirrors a forecaster's apparent tendency to ascribe less precise temporal adverbs to more remote meterological events. the approach can easily be adapted to synthesize bilingual or multi-lingual texts.
hypothesis selection in grammar acquisition. this paper presents some techniques for selecting linguistically adequate hypotheses of new grammatical knowledge to be used as resources of grammatical knowledge acquisition. in our framework of linguistic knowledge acquisition, a rule-based hypothesis generator is invoked in case of parsing failures and all the possible hypotheses of new grammar rules or lexical entries are generated from partial parsing results. although each hypothesis could recover the defects of the existing grammar, the greater part of hypotheses are linguistically unnatural. the techniques we propose here prevent such unnatural hypotheses from being generated without discarding plausible ones and make the following corpus-based acquisition process more efficient and more reliable.
"dialog navigator": a question answering system based on large text knowledge base. this paper describes a dialog based qa system, dialog navigator, which can answer questions based on large text knowledge base. in real world qa systems, vagueness of questions is a big problem. our system can navigates users to the desired answers using the following methods: asking users back with dialog cards, and description extraction of each retrieved text. another feature of the system is that it retrieves relevant texts precisely, using question types, synonymous expression dictionary, and modifier-head relations in japanese sentences.
complex: a computational lexicon for natural language systems. although every natural language system needs a computational lexicon, each system puts different amounts and types of information into its lexicon according to its individual needs. however, some of the information needed across systems is shared or "identical" information. this paper presents our experience in planning and building complex, a computational lexicon designed to be a repository of shared lexical information for use by natural language processing (nlp) systems. we have drawn primarily on explicit and implicit information from machine-readable dictionaries (mrd's) to create a broad coverage lexicon.
degrees of stativity: the lexical representation of verb aspect. l'acquisition automatique de connaissance lexicale &agrave; partir de larges corpus s'est essentiellement occup&eacute;e des ph&eacute;nom&egrave;nes de co-occurrence, aux d&eacute;pens des traits lexicaux inh&eacute;rents. nous pr&eacute;sentons ici une m&eacute;thodologie qui permet d'obtenir l'information s&eacute;mantique sur l'aspect du verbe en analysant automatiquement un corpus et en appliquant des tests linguistiques &agrave; l'aide d'une s&eacute;rie d'outils d'analyse structurale. lorsque ces deux t&acirc;ches sont accomplies, nous proposons une r&eacute;presentation de l'aspect du verbe qui associe une valeur de mesure pour les diff&eacute;rents types d'&eacute;v&egrave;nements. les mesures refl&egrave;tent l'usage typique du verbe, et par cons&eacute;quent une mesure de r&eacute;sistance ou de non-r&eacute;sistance &agrave; la coercion dans le contexte de la phrase. les r&eacute;sultats que nous rapportons ici ont &eacute;t&eacute; obtenus de deux mani&egrave;res: en extrayant l'information n&eacute;cessaire &agrave; partir du corpus &eacute;tiquet&eacute; de francis and ku&ccaron;era (1982), et en faisant tourner un analyseur syntaxique (mccord 1980, 1990) sur le corpus du reader's digest afin d'extraire une information plus pr&eacute;cise sur l'usage du verbe dans le texte.
the bicord system: combining lexical information from bilingual corpora and machine readable dictionaries. our goal is to explore methods for combining structured but incomplete information from dictionaries with the unstructured but more complete information available in corpora for the creation of a bilingual lexical data base. this paper concentrates on the class of action verbs of movement, and builds on earlier work on lexical correspondences between languages and specific to this verb class. the languages we explore here are english and french. we first examine the way prototypical verbs of movement are translated in the collins-robert (collins 1978, henceforth cr) bilingual dictionary. we then analyze the behavior of some of these verbs in a large bilingual corpus. we take advantage of the results of linguistic research on verb types (e.g. levin, to appear) coupled with data from machine readable dictionaries to motivate corpus-based text analysis for the purpose of establishing lexical correspondences with the full range of associated translations and then attach frequencies to translations.
machine-readable dictionaries in text-to-speech systems. this paper presents the results of an experiment using machine-readable dictionaries (mrds) and corpora for building concatenative units for text to speech (tts) systems. theoretical questions concerning the nature of phonemic data in dictionaries are raised; phonemic dictionary data is viewed as a representative corpus over which to extract n-gram phonemic frequencies in the language. dictionary data are compared to corpus data, and phoneme inventories are evaluated for coverage. a methodology is defined to compute phonemic n-grams for incorporation into a tts system.
data types in computational phonology. this paper examines certain aspects of phonological structure from the viewpoint of abstract data types. our immediate goal is to find a format for phonological representation which will be reasonably faithful to the concerns of theoretical phonology while being rigorous enough to admit a computational interpretation. the longer term goal is to incorporate such representations into an appropriate general framework for natural language processing.
robust interpretation of user requests for text retrieval in a multimodal environment. we describe a parser for robust and flexible interpretation of user utterances in a multi-modal system for web search in newspaper databases. users can speak or type, and they can navigate and follow links using mouse clicks. spoken or written queries may combine search expressions with browser commands and search space restrictions. in interpreting input queries, the system has to be fault-tolerant to account for spontanous speech phenomena as well as typing or speech recognition errors which often distort the meaning of the utterance and are difficult to detect and correct. our parser integrates shallow parsing techniques with knowledge-based text retrieval to allow for robust processing and coordination of input modes. parsing relies on a two-layered approach: typical meta-expressions like those concerning search, newspaper types and dates are identified and excluded from the search string to be sent to the search engine. the search terms which are left after preprocessing are then grouped according to co-occurrence statistics which have been derived from a newspaper corpus. these co-occurrence statistics concern typical noun phrases as they appear in newspaper texts.
recognition of abstract objects - a decision theory approach within natural language processing. the daisy/alibaba-system developed within the wai-project represents both a specific solution to the automatic indexing problem and a general framework for problems in the field of natural language processing, characterized by fuzziness and uncertainty. the wai approach to the indexing problem has already been published [3], [5]. this paper however presents the underlying paradigm of recognizing abstract objects. the basic concepts are described, including the decision theory approach used for recognition.