Supervised Feature-based Classification Approach to Bilingual Lexicon Induction from Specialised Comparable Corpora (2023)

Abstract: This study, submitted to the BUCC2023 shared task on bilingual term alignment in comparable specialised corpora, introduces a supervised, feature-based classification approach. The approach employs both static cross-lingual embeddings and contextual multilingual embeddings, combined with surface-level indicators such as Levenshtein distance and term length, as well as linguistic information. Results exhibit improved performance over previous methodologies, illustrating the merit of integrating diverse features. However, the error analysis also reveals remaining challenges.

## introduction the current contribution represents a submission to the bucc2023 shared task on bilingual term alignment in comparable specialised corpora 1 , specifically for the english-french language pair. the task can alternatively be phrased as bilingual lexicon induction (bli) for terminology. it holds significant potential: it can benefit end-users with ad hoc bilingual terminology construction from relatively easily available comparable corpora, and offers researchers a probing task to assess the crosslingual lexico-semantic knowledge of language models. this complex task encompasses many current challenges in natural language processing. first, there are the challenges related to the data. with parallel corpora, identifying an equivalent term in the aligned sentence is, if not simple, at least a task with limited possible answers. with comparable corpora, the task becomes exponentially harder. there is no straightforward place in the corpus to start looking for equivalents, and no guarantee that there will be a valid cross-lingual equivalent for 1 2023 building and using comparable corpora shared task website: https://comparable.limsi.fr/bucc2023/bucc2023task.html each term. this makes it difficult both to construct a gold standard dataset and to automate the task. for the shared task, the former issue was addressed by creating comparable corpora based on parallel corpora @xcite . moreover, the shared task starts from a predefined list of candidate terms, so the focus is only on the cross-lingual alignment, and not term identification. besides the data-related challenges, there are conceptual challenges. terminological equivalence must be defined (should terms and meanings be considered in context? how close does the meaning have to be, for a term to be considered valid? do equivalents need to have the same syntactic function or can, e.g., an adjective be a valid equivalent for a noun?). this issue is circumvented in the shared task because the dataset was created based on parallel data, where the equivalence can be defined in context. as will be seen in the error analysis, this also means there are remaining questions as to the equivalence of, for instance, false positives. a final challenge concerns the choice of lexical items, in this case: single-and multi-word terms. popular embedding-based approaches still struggle with accurate representations for multi-words. including multi-words alongside single-words, with pairs of different lengths, forces participants to develop a methodology that handles both. for instance, the french equivalent for train station is gare, and for database it is base de données. additionally, terminology is typically not as common as general vocabulary, so methodologies need to be more robust for smaller datasets and lower frequencies. this paper starts with information on the shared task dataset and setup, and a section on related research. next, the methodology is described, followed by the results and a brief error analysis, before summarising the findings and looking ahead in the conclusion. number of: training test tokens in src corp. 19,358,505 4,464,919 tokens in tgt corp. 21,378,916 14,158,415 gs term pairs 2,519 1,970 src terms 3,132 1,270 tgt terms 2,984 9,712 src terms not in txt 17 0 tgt terms not in txt 30 9 looking at the sizes of the datasets (see all occurrences of all terms were identified in the lowercased and tokenised corpora. most, but not all terms were found. in the training data, terms that were not found did not appear in the gold standard list of term pairs. however, in the test data some terms among the gold standard term pairs were not found in the corpus. therefore, with this methodology, these terms could not be found by the system either. this was only the case for four pairs in the test data. there is a relatively even distribution between single-and multi-word terms in all parts of the corpora: 41% and 61% single-word terms in source and target training data; and 48% and 44% in source and target test data respectively). the corpora contain texts from many different domains, and they are not very specialised. there are many very general terms (e.g., water bottle, slow, remarks, young adults), and much fewer specialised terms (e.g., sovereignty, probiotic, legal person). in both train and test data, there are many instances that would not conventionally be called terms e.g., whosoever, very long time, necessarily, mere, friendly atmosphere, etc. this is due to the automatic creation of the dataset, based on automatic term extraction with termsuite @xcite . this is not to say that termsuite's performance is bad, but there will inevitably be errors. moreover, termsuite is meant to work well on domain-specific specialised corpora and, while the bucc corpora are somewhat specialised, they cover many different domains. a ranked list of term pairs for the test data had to be submitted, with the most confidently predicted pair at the top. up to five submissions were allowed per team. this list was evaluated through uninterpolated average precision (ap), with an evaluation script provided on github foot_0 . ## related research last year, two teams submitted three runs each to this shared task @xcite . team jozef stefan institute (jsi) @xcite trained an svm binary classifier @xcite , using features based on both the shared task resources, and external (freely available) resources. they originally experimented with cross-lingual embeddings and sentence transformers, but chose a featurebased approach instead, due to unsatisfying results. this approach was based on previous work @xcite , where they incorporated "the cosine similarity values of the crosslingual and sentence transformer models into features of the machine learning model" @xcite . they use four types of features. the "cognate-based features" take into account the specific differences between language. for instance, words ending in -ology in english are likely to end in -ologie in french. their "dictionary-based features" rely on giza++ word alignment @xcite . the "embedding-based features" use cosine similarity scores from cross-lingually aligned embeddings and five language models. the final group of "combined features" combines parts of all three other groups. team cuni @xcite submitted three different systems: one with cross-lingually aligned static embeddings, one with contextual multilingual embeddings, and one with unsupervised phrase-based machine translation. for the former, they trained fasttext embeddings @xcite for both languages and aligned them cross-lingually using the muse tool @xcite . for the contextual embeddings they worked with multilingual bert @xcite and xlm @xcite . finally, they used the monoses tool @xcite to train an unsupervised phrase-based machine translation model on the provided comparable corpora. they also submitted a combined approach using both the cross-lingually aligned embeddings and phrase-based machine translation. on the test set, the cuni team obtained the highest uninterpolated average precision score (0.2816) with their combined system, closely followed by two of the submissions of jsi (0.2685 and 0.2674). team jsi concluded that "careful feature engineering could still produce better results than more novel deep learning approaches", though they admit their system is "quite resource intensive" @xcite . team cuni concluded that they were able to get the highest mean average precision (map) on the train set with the xlmmodel, fine-tuned on the task dataset. the task organisers noticed that 10.7% of the gold standard term pairs were not found by any of the six submit-ted systems. a recurring issue was when multiple equivalents were present in the gold standard data, and the systems did not find all options. multiword terms were also found to be more difficult @xcite . of course, there is other related research outside of the shared task, though rarely including multi-words. generally, it is interesting to see experiments where the information from language models is supplemented with additional (linguistic) information. researchers argue that "there is still room in the nlp toolbox for methods that utilise discrete, symbolic linguistic knowledge; in fact, the two paradigms can be successfully combined for an amplified effect" @xcite . specifically for bli, there is also a call for more rigour on the definition of the task and the used datasets. @xcite address the challenges related to evaluating bli. focusing on the popular and valuable muse dataset @xcite , they identify several issues: there is an overrepresentation of proper nouns, of graphically similar (or identical) word pairs, and of high frequency words. a similar argument is made in the work of @xcite , who, additionally, talk about the gaps in the gold standard datasets. some of these issues are notably less present in the bucc shared task dataset because it focuses on terminology, making it more interesting and challenging. nevertheless, the gold standard data is still automatically generated, so any research requires thorough evaluation that goes beyond simple scores to identify system strengths and weaknesses. ## results the official results for the shared task can be found in as described, further experiments were performed to analyse the system and results in more detail. the experiments focused on the impact of: the scoring used for optimisation, the features, and the threshold value (i.e., the minimum predicted probability score for equivalence). originally, this threshold was always set at 50% (only pairs the system actually predicted as equivalent), but since it was observed that uninterpolated average precision could be further improved by lowering this threshold, scores were also calculated at a cut-off point of 25%. for each experiment, uninterpolated average precision (ap) is reported as defined by shared task, as well as precision (p), recall (r), and f1-score (f1). additionally, f1-score of the true label in the classification task (f1 true) is included, and the number of predicted equivalent pairs above the threshold (#), concerning the features, experiments were performed with: all described features (all: 45 features), only the cosine similarity features and levenshtein distance (cos&lev; 5 features), only the cosine similarity features (cos: 4 features), or a limited set of features, including cos & lev, the difference in frequency, the four combined length features, the three part-of-speech features that are not about specific tags, and the difference in the average number of tokens recognised as named entities (limited: 14 features). the latter was meant to reduce some of the redundant information in the features, as there were many with both separate values for source and target terms, as well as a feature combining that information. the results of these additional experiments can be seen in the first observation about the results in the next observation is that a lower threshold results in (much) higher scores for ap. for the experiments on the training data, this improvement in ap is due to an increase in recall (up to .94), but the drop in precision results in a lower f1-score. for the experiments on the test data, ap is also highest with the lower threshold thanks to an improved recall, but in this case, the f1-scores are not much affected. lowering the threshold has a higher impact on the number of predictions for the test data experiments. for the training data experiments, only 245 to 338 more pairs are extracted (+39% to 55%), whereas for the experiments on the test data, lowering the threshold results in up to 1433 more pairs, i.e., an increase of up to 122%. being able to easily adjust this threshold depending on the requirements of the experiment is a considerable advantage. conversely to the results of the officially submitted runs, scoring used for optimisation has only a very minor impact, so, for the experiments on the test data, all training was optimised with roc. predictably, the features do influence results. clearly and unsurprisingly, the cosine features are most important, and results are not bad with just those features. the addition of levenshtein is, predictably, an advantage for the english-french language pair as well. interestingly, the other features also add relevant information, though the individual features are much less important. the system with more limited features appears to efficiently capture the relevant information. feature importance scores of a system trained with limited features and optimised with roc are shown in ## error analysis the output of the system trained on the shared task training data and tested on the test data (including all features) was analysed in more detail. among the most confidently predicted pairs, there is a good mix of single-and multi-word terms, so not all multi-word term pairs were difficult to predict correctly. at the top of this list, there are a lot of pairs with a low levenshtein distance, though not exclusively. for instance, at rank 7 there is the pair typical recipe and recette typique, and at rank 31 disabled children and enfants handicapés. the first false positive is found at rank 74, where economic difficulties is aligned with problèmes économiques (literally economic problems). while a more literal equivalent is available, this pair could certainly be considered equivalent in many contexts. this is seen for many of the highly ranked false positives: they either could be equivalent in certain contexts, or they should have been considered equivalents in the first place, e.g., strategic game and jeu stratégique, and direct taxes and taxes directes. out of 1127 ranked equivalents, there were 325 false positives and 58 of those could be considered equivalent in many contexts, with an additional 33 deemed strongly related or potentially equivalent in some contexts. while these results require a more thorough analysis (with inter-annotator agreement), these numbers are an indication of the importance of a nuanced definition of equivalence and a thorough error analysis. naturally, some terms are also clearly misaligned. one of the, probably less serious, common misalignments is between terms with a different number. for instance, the singular tumor is aligned with the plural tumeurs, and the reverse is done for wine bottles and bouteille de vin. there are also a few false positives due to different partsof-speech, for instance, infected was matched to infection. however, this only occurred eight times, so the part-of-speech features may have already prevented some of these mismatches. multi-word terms with relatively general words were also found to be difficult. the term access control system was linked to 16 different french terms with a probability of at least 25%. a couple of other categories of terms that cause multiple false positives are: numbers, family relations, and colours. for instance, eighth is most confidently correctly aligned with huitième, but then also (with much lower probability) to dix-septième (seventeenth). similarly, aunt is correctly matched to tante with a 94% probability, but then also to oncle (uncle) (91%), mère (mother) (86%), père (father) (71%), neveu (nephew) (69%), and so on. similar issues are found for colours. sometimes cultural differences play a role, for instance when pound is wrongly, but understandably, matched to kilo. while performance on multi-word terms was not especially bad, the rather simplistic approach of averaging embeddings has clear downsides. this can be seen in misalignments where word order plays a role, e.g., wine bottles is misaligned to vin en bouteille (botteled wine), and product safety to produits de sécurité (safety products). ## conclusion this contribution to the bucc2023 shared task on bilingual term alignment in comparable specialised corpora presents a supervised approach with a feature-based classifier that combines features from embeddings with other information, including edit distance and linguistic characteristics. results are promising and the system outperforms those from last year's submissions. though the efficient random forest classifier is used, preparing the experiments is, admittedly, computationally expensive, since all source language terms are matched with all target language terms, and contextual features are calculated for each pair. however, it also provides interesting insights, for instance showing the relative importance of the various features. the error analysis illustrates various challenges, both in terms of the dataset and in terms of system weaknesses. future research is planned to look into rich datasets for bli from specialised corpora, to facilitate more thorough work on this task. further experiments will include more features and compare different embeddings, as well as experiments with different types of classifiers. a more elaborate error analysis and the inclusion of more language pairs could further improve our understanding of the cross-lingual knowledge captured (or not) by both static and contextual embeddings.