S mart B i C : Smart Harvesting of Bilingual Corpora from the I nternet (2024)

Abstract: SmartBiC, an 18-month innovation project funded by the Spanish Government, aims at improving the full process of collecting, filtering and selecting in-domain parallel content to be used for machine translation and language model tuning purposes in industrial settings. Based on state-of-the-art technology in the free/open-source parallel web corpora harvester Bitextor, SmartBic develops a web-based application around it including novel components such as a language- and domain-focused crawler and a domain-specific corpora selector. SmartBic also addresses specific industrial use cases for individual components of the Bitextor pipeline, such as parallel data cleaning. Relevant improvements to the current Bitextor pipeline will be publicly released.

## introduction obtaining a suitable amount of parallel corpora to train neural machine translation (nmt) or large language models (llms) is a challenging task, which becomes particularly difficult for domainspecific areas. this is seen as a severe limitation for the full development of nmt and llms in industrial settings. smartbic, an innovation project ending in september 2023 with support from the spanish government through the nextgenera-tioneu funds, addresses this limitation from an industrial perspective. the main goal of smartbic is to ease the collection, filtering and selection of indomain parallel corpora by exploring multilingual websites and external corpora. to that end, smart-bic builds upon bitextor, foot_0 a free/open-source parallel corpora harvester, adding the following improved and novel components described in detail in section 3: a smart crawler, a smart cleaner, and a smart corpora selector. in-domain corpora produced with smartbic will be used in industrial use cases, particularly to train domain-specific nmt systems, but also to explore their usefulness as llm training or tuning datasets. this will extrinsically evaluate the quality and usefulness of the results of the project and point to improvements for future work. ## bitextor as the core technology bitextor @xcite has been used to produce petabytes of parallel corpora from multilingual web-crawled content in previous eu-funded projects such as paracrawl.eu or macocu.eu. among other alternatives, such as the ilsp-fc focused crawler @xcite , bitextor is chosen for its language support, active developer community and modular design. ## new components in smartbic smartbic addresses the following limitations in bitextor in order to achieve the goals of the project: (1) crawlers in bitextor download websites ignoring the language pair indicated in the input, only a tiny portion of the crawled content makes it to the output, (2) no topic or domain constraints are currently handled by bitextor, in-domain and generic content is mixed in the output parallel corpus and (3) seed urls need to be manually provided to the crawler and no mechanism to automatically discover further interesting urls is available. smartbic adds as new components to bitextor to overcome these limitations (see figure 1): the smart crawler module which improves the current generic crawler by integrating both language pair and domain focus during crawling and includes: (1) two new crawlers with complementary crawling strategies which filter content by language pair, reducing the amount of information downloaded; (2) a mechanism to automatically find urls from another url, a text or a keyword list exploiting these keywords, potentially similar urls are discovered using browser-based queries. the smart cleaner, a stand-alone component that enhances and provides the functionality of the bitextor cleaning pipeline; it includes: (1) improved and new filters with customisable thresholds; (2) support for new language pairs, that is, new models for bicleaner; (3) custom filtering, cleaning statistics and new output formats. the smart selector independent module which picks the most relevant set of in-domain data from already crawled or generic corpora and includes: (1) a separate, standalone cleaning step to filter out noisy sentence pairs and to score the remaining ones with bicleaner (zaragoza-bernabeu et al., 2022); (2) a multilingual zero-shot classifier which adds domain scores to each sentence in a sentence pair; (3) a customisable sentence pair selector based on size, domain, cleaning scores and content quality. relevant components resulting from this project will be released through bitextor.