Building a Corpus for Corporate Websites Machine Translation Evaluation. A Step by Step Methodological Approach (2021)

Abstract: The aim of this paper is to describe the process carried out to develop a paral-lel corpus comprised of texts extracted from the corporate websites of south-ern Spanish SMEs from the sanitary sector which will serve as the basis for MT quality assessment. The stages for compiling the parallel corpora were: (i) selection of websites with content translated in English and Spanish, (ii) downloading of the HTML files of the selected websites, (iii) files filtering and pairing of English files with their Spanish equivalents, (iv) compilation of individual corpora (EN and ES) for each of the selected websites, (v) merging of the individual corpora into a two general corpus one in English and the other in Spanish, (vi) selection a representative sample of segments to be used as original (ES) and reference translations (EN), (vii) building of the parallel corpus intended for MT evaluation. The parallel corpus generated will serve to future Machine Translation quality assessment. In addition, the monolingual corpora generated during the process could as a base to carry out research focused on linguistic – bilingual or monolingual − analysis.

## background nowadays, thanks to the development of information and communication technologies, companies are able to spread messages globally, allowing them to open new markets. web 2.0 tools, such as websites, provides enterprises, especially small and medium-sized ones (smes) with great opportunities for internationalization @xcite . in fact, in the european union (eu), more than 99% of all enterprises -save for the financial business sector -are smes @xcite and 77% of them have a website @xcite . however, language barriers often pose a challenge for companies when it comes to the multilingual dissemination of corporate information and that is why machine transla-tion (mt) can be a resource with great potential for solving this problem. nevertheless, mt quality is generally inferior to that reached by professional human translations. consequently, mt evaluation, by means of automatic and human metrics, plays a key role for determining mt quality as well as for mt systems to be improved. however, the assessment of mt systems implies cognitive linguistic, social, cultural and technical processes @xcite . as a result, assessing mt can present difficulties as in the majority of cases there is not just one correct translation @xcite . in addition, it is important to note that there is a great lack of consensus in relation to translation quality assessment and approaches may differ according to the individuals, groups or contexts in which quality is assessed. therefore, there are a number of metrics and criteria for undertaking the evaluation of mt systems. however, generally speaking, there are two main types of mt evaluation: human and automatic. on the one hand, most automated metrics establish comparisons between the output of an mt system and one or more reference translation @xcite . some of these measures such as wer, per or ter are based on the levenshtein or edit distance @xcite , the difference among this metrics is that some of this metrics consider word or phrases reordering as and edit operation @xcite . other measures, such as bleu @xcite -the most popular metric-are based on precision and carried out at the level of n-grams, indivisible language units. bleu employs a modified precision that considers the maximum number of each n-gram appearance in the reference translation and applies a brevity penalty that is added to the measurement calculation. other precision-centred metrics, for example, are nist @xcite , rouge @xcite , f-measure @xcite and meteor @xcite . on the other hand, human evaluation revolves around adequacy-based on semantic quality-and fluency-based on syntactic quality. for adequacy, evaluation reference translations or the original text, if the evaluators have language knowledge, are required. in the case of fluency evaluation, as the evaluation is monolingual, no reference translation nor the original text are necessary. human evaluation can be carried out by means of ranking, likert-type ordinal scales, gap filling tasks or by identifying, annotating, classifying and correcting translation errors, amongst others @xcite . human evaluation, despite demanding more time, effort and costs, is considered to be more reliable than automatic metrics, as their capacity to evaluate syntactic and semantic equivalence is limited @xcite . however, human evaluation cannot be reproduced, is less objective than automatic metrics and requires evaluators to fulfill certain criteria and to be trained prior the evaluation task. therefore, it is advisable to combine various metrics that evaluate different aspects in order to assure the reliability of the results. in the light of this scenario, the aim of this paper is to describe the process carried out to develop a parallel corpus (spanish -english) comprised of texts extracted from corporate websites of southern spanish smes from the sanitary sector which will serve as the basis for future mt quality assessment tasks. ## methodology prior research set the basis for the corpus presented in this paper @xcite . the latter study analyzed 1425 andalusian smes, belonging to what is referred to as group q: healthcare and social services activities according to the cnae-2009 classification (spanish national classification of economic activities). this economic sector was chosen as the healthcare sector is the second biggest group as regards business creation in net terms according to official reports @xcite . the sample was selected using information from the sectoral ranking of companies by turnover offered by the spanish source eleconomista.es, a daily newspaper with special focus on economics, finance, and business. the data from this company ranking comes from the in-forma d&b s.a.u. (s.m.e.) database -which boasts the spanish association for standardization and certification (aenor) quality certificate -and is fed from several public and private sources. this study concluded that around a half of the analyzed smes had a website, but only 10% of them offered their content translated to one or more languages. the final goal of the research project is to evaluate mt applied to corporate information available on smes websites, hence, reference translations were needed to build the parallel corpus. therefore, those companies offering translated content to english and spanish served to build the corpus described in this paper, which responds to a sequential sampling strategy meaning that first phase results @xcite determined the methodology of the next phase @xcite . the stages for compiling the parallel corpora were: (i) selection of websites with content translated in english and spanish, (ii) downloading of the html files of the selected websites, (iii) files filtering and pairing of english files with their spanish equivalents, (iv) compilation of individual corpora (en and es) for each of the selected websites, (v) merging of the individual corpora into a two general corpus one in english and the other in spanish, (vi) selection a representative sample of segments to be used as original (es) and reference translations (en) and, (vii) building of the parallel corpus intended for mt evaluation. ## conclusion and future research the aim of this paper was to describe step by step the methodological approach followed to build a parallel corpus which will be used for mt quality evaluation -including both human and automatic assessment -of corporate websites belonging to smes from the healthcare sector. to this end, to the final xls file, containing the original spanish segments and their equivalents in english, one more column was added containing mt output, therefore, the xls file contains automatic translations (en) together with their original translation (es) and reference translations (en), which are both essentials to mt quality assessment. however, more columns could be added in the future to include automatically generated translations from more mt systems to compare their performance. further analysis concerning mt error identification, annotation and classification could also be carried out using the parallel corpus as a base, along with the evaluation of the post-editing process. in addition, the parallel corpora can be easily enlarged by adding segments from the monolingual corpora, which are already formatted and numbered in order to make the process as efficient as possible. the parallel corpus can also be uploaded to corpus analysis tools such as sketch engine for further linguistic analysis. on another note, the monolingual corpora generated in english and spanish can also serve to carry out linguistic research, including comparison between languages or monolingual analysis. these two corpora, given its considerable volume, could be used to train purpose-built mt systems and they can also serve to enlarge the knowledge concerning the features of corporate texts from the healthcare sector.