Kunji : A Resource Management System for Higher Productivity in Computer Aided Translation Tools (2019)

Abstract: Complex NLP applications, such as machine translation systems, utilize various kinds of resources namely lexical, multiword, domain dictionaries, maps and rules etc. Similarly, translators working on Computer Aided Translation workbenches, also require help from various kinds of resources - glossaries, terminologies, concordances and translation memory in the workbenches in order to increase their productivity. Additionally, translators have to look away from the workbenches for linguistic resources like Named Entities, Multiwords, lexical and lexeme dictionaries in order to get help, as the available resources like concordances, terminologies and glossaries are often not enough. In this paper we present Kunji, a resource management system for translation workbenches and MT modules. This system can be easily integrated in translation workbenches and can also be used as a management tool for resources for MT systems. The described resource management system has been integrated in a translation workbench Transzaar. We also study the impact of providing this resource management system along with linguistic resources on the productivity of translators for English-Hindi language pair. When the linguistic resources like lexeme, NER and MWE dictionaries were made available to translators in addition to their regular translation memories, concordances and terminologies, their productivity increased by 15.61%.

## introduction nlp applications -machine translation systems and translation workbenches (which can have multiple mt systems integrated), are complex in nature as they are built with various complex heterogeneous nlp modules. these complex nlp applications are compute and knowledge intensive in nature and require various types of resources at different levels of processing. the translators using the translation workbenches have to look for various resources -glossaries, terminologies, concordances and translation memory in order to supplement their translation tasks. it is still hard to maintain consistency while maintaining high productivity. sometimes, the aid the translators get from the resources is not enough and they have to look for linguistic resources -named entities, lexical, bilingual, multiwords and lexeme dictionaries -offline or online for the correct and appropriate meaning of a given phrase or word leading to additional cognitive load on translators. in mt system, when mt modules execute on the given input, various types of resources namely lexical, bilingual, domain, multiword dictionaries, paradigms, maps files and rules are required at various stages of the processing. with the exploration of sampark and anusaaraka mt systems @xcite , it has been found that the resources can be used in a more efficient way in the both systems. these resources are developed by by nlp researchers who face resource management issues like exporting, importing, storage etc. for a large scale nlp application development, inefficient storage and management of these resources can become a bottleneck for productivity. for the translation tasks and mt applications' development for indian languages, the above mentioned issues become more critical as the digital content available in indian languages are way less compared to other languages. as hindi is one of the top 5 spoken languages of the world but still its digital content available on the web is less than 0.05% while the content of the other indian languages are still lower. hence the need of a resource management system arises which can ad-dress the issues of both translators as well as mt modules' development. in this paper, we present a resource management system named kunji, which addresses the issues related to resources of both translation workbenches and mt system development for indian languages. it can be used in translation workbenches to facilitate the translators to use additional linguistic resources -domain, lexical, named entities, multiword and wsd dictionaries -along with the terminologies, concordances and glossaries. it facilitates the translators in two ways -i) reduction of their repetitive load by providing searching and filter mechanism and ii) reuse of their previously translated words, phrases and sentences by providing the provision of personal and domain dictionaries. it shifts the problem of recall to recognition which causes less cognitive load on translators by providing them provision to search the corresponding terms and recognize. it also facilitates the nlp researchers to manage the resources for mt modules in a robust manner while addressing their issues of management, export, import, formats and efficient search etc. it also lets them evaluate and verify their work from other senior expert. such system facilitates the large scale mt applications development by making the resources available to multiple modules and their facilitation to process and reuse them efficiently in an mt system. additionally we performed an experiment to study the impact of providing this resource management system along with linguistic resources on the productivity of translators. the described resource management system, kunji, has been integrated in the translation workbench transzaar @xcite . the resources with linguistic features like lexeme, ner and mwe dictionaries have been provided in kunji to facilitate translators in addition to their regular translation memories, concordances and terminologies for english-hindi language pair. we observed that the productivity of the translators increased by 15.61% when the resources with linguistic features were provided to the translators in addition to their regular resources (translation memories, glossaries, concordances and terminologies). in this paper, background and motivation for kunji is described in section-2 followed by related work in section-3. the detailed architecture and functionality of kunji is described in section-4. section-5 describes the experiment performed on kunji followed by its results and discussion in section-6. in section-7, we conclude our work. ## background and motivation in current scenario, various translation workbenches(cat tools) are available. some of them offer the resources like translation memories(tm), concordances, terminologies and glossaries to translators but there is a lack of the resource management systems which provide the provision by which a translator can use additional linguistic resources like ne, mwe and lexeme dictionaries. additionally there is a lack of the provisions of reuse, management and verification of these resources which can lead a process to improvement and development of the mt systems. there are several tools for resource development but they are built with separate purposes and languages as they are for task specific. like for annotators they have separate tools in indian languages like sanchay @xcite but it is desktop based application. some tools like gate @xcite addresses the some of the problems faced by nlp researchers while developing nlp resources but it is desktop based application. it motivated us to develop kunji, a resource management tool, which facilitates the translators in translator workbenches as well as mt system development process. it tries to address the issues faced by both translators as well as mt module developers. it allows the translators to use and manage the linguistic resources along with their terminologies and glossaries. ## related work there are some translation workbenches with separate resource management systems such as: smartcat foot_0 , memsource foot_1 and matecat @xcite are web based cat tools, provide central resources management for a project but they do not provide the provision for the linguistic resources. casmacat @xcite is a translation workbench which is web based and offers advanced functionality for computer-aided translation. it offers tms but it also lacks the provision for the linguistic resources. pet @xcite is a tool to postedit to for evaluating the quality of translations. it evaluates the efforts required in translations in order to be fixed. sdl trados, memoq 3 and anubis @xcite are cat tools which are not web based and both lack the provision to use linguistic resources. brat @xcite is basically a web based annotation tool which focuses mainly on text annotations to enhance annotators productivity by closely integrating nlp technology into the annotation process. it doesnt address the issues in resources management with respect to translators and large scale nlp application development. creating language resources for nlp in indian languages @xcite defines a novel idea in which the development of lexical resources is linked with an example nlp application like mt then it can act as a test bed for the developing resources and provide constant feedback. sanchay @xcite provides an extra layer of easily customizable language encoding support for less computerized languages along with an editor named sanchay with different types of fonts and language encoding supports but it is desktop based application. gate @xcite enables users to develop and deploy the language engineering tools and resources in a robust manner. it is a desktop based tool which supports many nlp and information extraction tasks in multiple languages. nancy et al @xcite presents an abstract data model and its implementation for linguistic annotations. annomarket @xcite described a cloud-based open platform for text mining, which aims to assist the development and deployment of robust, largescale text processing applications. we see that in some translation workbenches, the resource management systems for tm, terminologies and glossaries are available but they don't provide the provision for linguistic resources as well. ## kunji : resource management system we propose kunji-a resources management system, which is a web based system based on microservices architecture. it can be easily integrated to a translation workbench and provides a mechanism to facilitate translators to use and manage the various types of resources (i.e. like terminologies, glossaries and domain dictionaries along with linguistic resources(lexeme dictionar-3 https://www.memoq.com/en ies, ners, mwes etc.)). it does not only facilitate the translators but also facilitates the mt modules development by facilitating language researchers to manage the language engineering resources in a robust manner for various language processing tasks and evaluate them which boosts the process of large scale mt development. ## experiment the described resource management system -kunji has been integrated in translation workbenches transzaar @xcite . we study the impact of providing this resource management system in cat tool-transzaar along with linguistic resources on the productivity of translators for english-hindi language pair. the various types of resources namely terminologies, glossaries, tm along with linguistic resources like lexeme, ne and mwe dictionaries were made available in the resources management tool in the workbench. ## results and discussion the results of experiments are given in from the scenario-wise time consumed per sentence translation by each of the translators is presented in there are some observations of the translators about the data set. the set-1 is the toughest of all four sets. it is evident from table-7 that, translators took most time in translation of this set in each of the scenario. the set-4 is the easiest of the sets as less time is consumed in its translation. translators found the linguistic dictionaries (lex- ## conclusion we have proposed a resource management system which addresses the issues of the translators as well as mt system development. using kunji, a translator can utilize, reuse and manage his resources which can be shared across projects and can be used further. also the nlp researchers get benefited using it in their resources management in mt system development so that the resources can be utilized and managed in an efficient way. it also provides the mechanism which allows a translator to get help from the linguistic resourceslexeme, ner and mwe dictionaries in addition to their regular translation memories, concordances and terminologies. we performed experiments which show the effect of the linguistic resources in the productivity of the translators. we see that their productivity increased by 15.61% when they use linguistic resources along with their regular tm, terminologies and glossaries as mentioned in