E mbed_ L lama: Using LLM Embeddings for the Metrics Shared Task (2023)

Abstract: Embed_llama is an assessment metric for language translation that hinges upon the utilization of the recently introduced Llama 2 Large Language Model (LLM), specifically, focusing on its embedding layer, with the aim of transforming sentences into a vector space that establishes connections between geometric and semantic proximities

## introduction the assessment of algorithm-generated translations entails the utilization of evaluation metrics that furnish quantitative scores to objectively gauge the precision of the model's output. various methodologies are employed to juxtapose machinegenerated translations with their human-generated counterparts, contingent upon the specific evaluation metric employed. in recent years, the realm of machine translation (mt) has witnessed notable advancements in terms of both translation accuracy and linguistic fluency. over time, there has been a significant enhancement in the correlation coefficient between human assessments and the automated evaluation of sentences generated by machines. earlier metrics, such as bleu @xcite or chrf++ @xcite , predominantly relied on the textual overlap between reference translations and the machine-generated counterparts. in contrast, contemporary approaches, exemplified by comet @xcite , harness recent breakthroughs in natural language processing(nlp) and transformer models, enabling them to consider not only individual words, but also to leverage contextual semantics for a more comprehensive evaluation. in the domain of machine learning (ml) applied to nlp, the embedding layer assumes a pivotal role within neural network architectures, particularly in tasks centered on textual data. its central objective lies in the transformation of discrete tokens, encompassing entities like words or characters into continuous vector representations. these vector representations, which maintain continuity, are amenable to acquisition and manipulation by neural networks and are commonly referred to as word embeddings. ## embed_llama the initial component in a natural language processing (nlp) model is typically an embedding layer, which serves the purpose of converting the distinct identifiers of tokens within the input sentence into a vectorized representation. in this context, it is essential to emphasize that sentences conveying similar semantic content should exhibit proximity in the vector space, irrespective of the presence of word-level overlap, in contrast to sentences chosen randomly. embed_llama draws inspiration from word2vec @xcite using llama 2 @xcite , a contemporary open-source pretrained model. rather than needing to train an extra nlp model for the purpose of assessing translation quality, a viable alternative approach involves uti-lizing a pre-trained, extensive neural network like llama 2, which has been originally trained for nexttoken prediction. this approach allows for the investigation of how closely related sentences evolve across the model's various layers, all without incurring the supplementary expenses associated with fine-tuning or training anew. ## hyperparameter as the objective of this current study revolves around the utilization of a pre-trained network, our sphere of influence is limited to a sole hyperparameter, namely, the number of blocks to retain before computing the cosine distance. ## conclusion although the authors initially anticipated that em-bed_llama would exhibit suboptimal performance for a majority of language pairs, except for those involving english, due to the apparent constraints posed by a limited vocabulary size and the nature of the dataset llama 2 was trained on, the actual performance did not exhibit a significant underperformance. the results from previous iterations of the wmt metrics shared task, presented in appendix a, indicate that this approach may not meet the contemporary state-of-the-art standards exemplified by metricx_xxl (unpublished) and comet-22 @xcite . the methodology involving the utilization of a non-finetuned, pre-trained large language model (llm) to assess translation quality through vector space similarity comparisons remains a prospective avenue of inquiry. this prospect gains relevance in light of forthcoming open-source models characterized by expansive vocabularies and training data encompassing diverse languages.