Intrinsic vs. Extrinsic Evaluation of C zech Sentence Embeddings: Semantic Relevance Doesn’t Help with MT Evaluation (2025)

Abstract: In this paper, we compare Czech-specific and multilingual sentence embedding models through intrinsic and extrinsic evaluation paradigms. For intrinsic evaluation, we employ Costra, a complex sentence transformation dataset, and several Semantic Textual Similarity (STS) benchmarks to assess the ability of the embeddings to capture linguistic phenomena such as semantic similarity, temporal aspects, and stylistic variations. In the extrinsic evaluation, we fine-tune each embedding model using COMET-based metrics for machine translation evaluation. Our experiments reveal an interesting disconnect: models that excel in intrinsic semantic similarity tests do not consistently yield superior performance on downstream translation evaluation tasks. Conversely, models with seemingly over-smoothed embedding spaces can, through fine-tuning, achieve excellent results. These findings highlight the complex relationship between semantic property probes and downstream task, emphasizing the need for more research into “operationalizable semantics” in sentence embeddings, or more in-depth downstream tasks datasets (here translation evaluation).

## introduction machine translation (mt) evaluation has advanced significantly in recent years, finally moving beyond traditional surface-level metrics like bleu @xcite towards more sophisticated approaches based on neural networks and contextualized embeddings. state-of-the-art mt evaluation metrics such as comet @xcite and bleurt @xcite use sentence embeddings to better capture semantic similarity between translations and references, achieving much higher correlation with human judgments than traditional metrics. however, the rapid development of new embedding models presents mt researchers with a challenging choice. although multilingual models such as labse @xcite and xlm-roberta @xcite have shown strong crosslingual capabilities, there are also language-specific models that claim superior performance for a selected language. for morphologically rich languages like czech, it remains unclear whether these specialized sentence embeddings offer advantages over multilingual alternatives when used in mt evaluation. in this paper, we examine the evaluation of english-to-czech machine translation and compare several state-of-the-art czech-specific models against multilingual models using both intrinsic evaluation and extrinsic evaluation. to this end, we see the task of machine translation evaluation (mte) and quality estimation (qe), i.e. mte without professionally translated reference sentences, as methods for extrinsic evaluation of sentence embeddings. for intrinsic evaluation, we assess how well the examined sentence embeddings reflect semantic properties exemplified in two datasets: costra @xcite and semantic textual similarity (sts, @xcite . in sum, our goal is to understand whether the performance of a model in intrinsic semantic tasks correlates with its usability for mt evaluation, potentially simplifying the selection of embeddings. ## related work several studies have raised concerns about the use of sts as an evaluation metric. for instance, @xcite , @xcite argue that, while sts can capture certain semantic similarities, it does not reliably predict how effective sentence representa-tions will be for downstream tasks. these works highlight how sts tasks often encourage surfacelevel heuristics or oversimplified semantic similarity patterns that may not generalize to more complex applications like entailment or paraphrasing detection. to address these limitations, new intrinsic evaluation methods such as evalrank @xcite and sentbench @xcite have been proposed, both of which exhibit a stronger correlation with extrinsic evaluation measures. these benchmarks evaluate sentence representations through information retrieval, sentence ordering, and probing tasks, offering a more holistic view of embedding quality that aligns better with actual downstream task performance. it is important to note that these previous experiments did not specifically focus on machine translation evaluation, which seems to be very close to sts-it also involves comparing pairs of sentences to assess their semantic closeness. @xcite report a negative correlation between the translation quality of transformer models measured by bleu and the semantic properties (assessed using sts) of the sentence embeddings derived from the transformer model. in contrast, @xcite demonstrate a strong positive correlation between sts performance and translation quality for both transformer and rnn-based models. more recently, @xcite have advocated for the use of semantic-aware metrics such as bertscore @xcite and comet @xcite in mt evaluation, showing that these outperform bleu in correlating with human judgments. these models incorporate contextual embeddings and often exhibit closer alignment with human-perceived meaning, bringing mt evaluation closer to the goals of intrinsic semantic understanding. ## models for our experiments, we used several stateof-the-art sentence embedding models, employing both czech-specific and multilingual variants. the czech encoders include three basesize transformer architectures, each using masked language modeling as their primary pretraining objective-czert-b-cased (czert, @xcite , fernet-c5 (fernet, @xcite and robeczech @xcite . to provide a broader comparative analysis, we also experimented with multilingual sentence embedding models trained on datasets that contain czech texts. these include labse @xcite , a model generating language-agnostic representations for more than one hundred languages with remarkable cross-lingual alignment, since its training objective was machine translation. furthermore, we evaluated two large models: xlm-roberta-large (xlm-r, @xcite and multilingual-e5-large (me5, @xcite , a model pretrained using a contrastive learning approach on a diverse range of tasks, including natural language inference and question answering across multiple languages. as a baseline model, we employed a bert architecture @xcite with randomly initialized weights. the only component inherited from the pretrained 'bert-base-multilingual-cased' model is the tokenizer. this means that while the model processes input according to the tokenization patterns learned from multilingual data, it does not benefit from any pretrained language representations. we refer to this configuration as random bert. this setup isolates and assess the contribution of the tokenizer alone, establishing a lower performance bound and offering a meaningful point of comparison to evaluate the benefits of pretraining. ## intrinsic evaluation we first evaluate the embeddings using a series of semantic benchmarks to determine their ability to accurately capture various semantic properties of a sentence. ## extrinsic evaluation-mte and qe extrinsic evaluation utilizes sentence embeddings as feature vectors for machine learning algorithms in downstream nlp tasks-mte and qe in our tasks. it serves well to choose the best method for a particular task but not as an absolute metric of embedding quality, as the performance of the embeddings does not correlate across different tasks @xcite . ## results and discussion when comparing the results of mte and qe with those of the intrinsic evaluation tasks, we can observe an interesting inversion. although both evaluation approaches aim to capture semantic similarity, the performance of the embeddings changed significantly after fine-tunings. specifically, xlm-r and fernet embeddings, which performed poorly in intrinsic evaluation, became the best performing mte and qe metrics. in contrast, simcse, which dominated intrinsic evaluations, ranks among the worst performing metrics in mte and qe. these results are in line with related research (section 2), which shows that sts performance may not accurately predict effectiveness in downstream tasks. there are several plausible hypotheses that might explain these discrepancies. let us at least mention them here-unfortunately, their thorough testing is beyond the scope of this article. first, xlm-r and fernet might perform poorly in intrinsic tasks because their representation spaces are not tuned for fine-grained semantic differences. however, when fine-tuned on a translation quality task, the model might learn to emphasize those aspects of the embedding space that are important for distinguishing translation quality. the fine-tuning process for comet-based evaluation might be effectively reconfiguring the xlm-r embedding space, transforming its initially oversmoothed representations into task-specific features that are highly discriminative for translation quality. although xlm-r raw embeddings appear to be all clustered together (see we should also not forget about the different embedding sizes, which played an important role in the observed behavior. the small embeddings-simcse and retromae-were among the worst performing comet estimators. large embeddings, such as those produced by xlm-r, offer a higherdimensional space that can capture more nuanced semantic and syntactic features. when fine-tuning with the comet estimator-which adds two hidden layers with sizes 3072 (resp. 2048) and 1024the richer representation provided by larger em-beddings could allow the model to extract and emphasize the translation-specific signals more effectively. interestingly, we can see not too much difference between the monolingual vs. multilingual embedding performances-they seem to be equally represented among the best performing embeddings in both intrinsic and extrinsic tasks. the size of the embeddings seem not to matter in the intrinsic tasks-the top 3 best performing embeddings (simcse, labse and me5) are small, base and large, respectively. the correlation analysis between different evaluation methodologies, visualized by heatmaps in figure 1 , reveals interesting patterns of how different evaluation methodologies relate to each other. these patterns provide valuable insight into the reliability and consistency of various embedding evaluation approaches. the heatmaps highlight a strong alignment among all intrinsic tasks (costra-, sts, and mte baselines). moreover, there is a strong correlation between the segment-level and the system-level metrics, indicating that aggregated segment scores provide reliable system-level insights. in particular, we observe strong correlations between segmentlevel metrics (segment mte and segment qe showing correlations of 0.97 and 0.91 for 2021 and 2022 respectively), suggesting that these evaluation approaches capture similar aspects of translation quality despite their methodological differences. however, one of the most striking findings is the weak and sometimes even negative correlation between intrinsic evaluation metrics (costra-, sts) and the system-level quality estimation scores system qe. this discrepancy is particularly evident in the 2022 data, where costra-shows a negative correlation (-0.52) with system qe, challenging the assumption that better semantic representation capabilities necessarily translate to improved mt evaluation performance. these findings indicate that intrinsic measures, while useful for general semantic similarity, may not sufficiently reflect translation-specific nuances required for mte or qe. consequently, intrinsic criteria alone appear inadequate for selecting optimal sentence embeddings for these specific tasks. further research is needed to identify intrinsic evaluation methods that better capture the subtleties relevant to machine translation. additionally, it would be valuable to explore in more detail the types of errors penalized in manual mt quality assessments to determine whether these errors predominantly concern sentence meaning or other aspects that should be preserved in translation. ## conclusion and future work we experimented with several evaluation methods for both czech and multilingual sentence embeddings, considering intrinsic semantic tasks and downstream application in machine translation evaluation and quality estimation. our key findings include the following: in future work, we intend to replicate these experiments across multiple languages to investigate whether the observed behavior is specific to the czech language or if it generalizes to other languages. in addition, we plan to conduct a more thorough analysis to better understand the underlying reasons for the differences in performance between the evaluation methods.