Training and Evaluating N orwegian Sentence Embedding Models (2023)

Abstract: We train and evaluate Norwegian sentence embedding models using the contrastive learning methodology SimCSE. We start from pre-trained Norwegian encoder models and train both unsupervised and supervised models. The models are evaluated on a machine-translated version of semantic textual similarity datasets, as well as binary classification tasks. We show that we can train good Norwegian sentence embedding models, that clearly outperform the pre-trained encoder models, as well as the multilingual mBERT, on the task of sentence similarity.

## introduction recently there have been a huge increase in the capabilities of natural language processing systems. the new dominant paradigm is using large language models such as bert @xcite or gpt @xcite as a starting model which one adapts to any given task one wishes to solve. there exists several different versions of bert-type encoder models in norwegian @xcite , @xcite , @xcite . it is well-known that berttype models that give contextual words embeddings do not give particularly good sentence embeddings @xcite . for this reason we train and evaluate norwegian sentence embedding models, using the pre-trained encoder models as starting points. we train models using the state of the art sim-cse methodology, similarly to the original paper @xcite . like them, we train both unsupervised and supervised models. we start with a pretrained bidirectional language encoder model such as bert or roberta @xcite . for the unsupervised version we sample texts from the norwegian colossal corpus (ncc) dataset @xcite . we then pass them through the model using two different dropout masks and predict contrastively which pairs within a batch represent the same text. for the supervised version, we train on a machine-translated version of natural language inference (nli) data, where we use sentences related by "entailment" as positive sentences, and sentences labeled as contradiction as hard negative sentences. we train on both the norwegian dataset, and a combined dataset of both norwegian and english nli data, and show that the latter gives better results for sentence representations in norwegian. we evaluate our models on a machine translated version of semantic textual similarities (sts) datasets, as well as on the sequence classification problems in norwegian "talk of norway" and the binary classification version of the norec review dataset @xcite . our main contributions are: ## related work the fundamental technique we build on is that of training large transformer models @xcite . in particular, we utilize the large encoder models bidirectional encoder representations from transformers (bert) and robustly optimized bert (roberta) by using them as pre-trained starting points. our work builds upon existing language models trained in norwegian. the national library of norway has trained bert models in norwegian @xcite , which we call nb-bert, which exists in both base and large size. also, the language technology group at the university of oslo has trained their version of a bert for norwegian called norbert @xcite . there is also a wikibert model trained on norwegian wikipedia @xcite . we also test the multilingual version of bert @xcite , which is trained in norwegian and many other languages. our work uses existing methodology for making sentence embedding models. the first paper to improve bert to make better sentence representations by training it for that purpose, was the sentence-bert paper @xcite , which trained sentence embedding models by using siamese networks. we build upon the newer simple contrastive learning of sentence embeddings (simcse) methodology @xcite , which uses a contrastive training objective to create sentence embeddings from a pre-trained encoder. the idea behind both of these works is that of finding a training procedure that better extracts the knowledge about sentences that already exists in the pre-trained encoder model. most existing work in the literature on making sentence embeddings are either in english or uses multilingual models. examples of the latter are mbert and several other approaches such as @xcite , @xcite and @xcite . ## data for the unsupervised models, we sample data from the norwegian colossal corpus (ncc) @xcite . this is a dataset of different smaller norwegian text corpuses that has been col-sentence: deltakerne mente at hvis interessenter var seriøse om å forbedre finansrapporteringsmodellen, ville en gruppe bli opprettet og finansiert spesielt for dette formålet. for supervised training we train with data collected for the task of natural language inference (nli). this task is that of taking a pair of sentences and predicting the relationship between them as either "entailment", "neutral" or "contradiction". the authors of the simcse paper use nli data to create triples of a sentence with one positive and one hard negative and show that this data work well for training sentence models using contrastive learning, thus we follow this practice. we use a dataset that has been curated for training in norwegian by the national library of norway. 1 the original data is based on the english datasets the stanford natural language inference (snli) corpus @xcite and multi-genre natural language inference (mnli) dataset @xcite . the norwegian data is machine translated from the mnli dataset and has about 128 thousand triples. there is also a combined norwegian and english version of the dataset made by taking a combination of the translated norwegian mnli data and english mnli and snli data. 2 also included are extra combined norwegian/english sentence triples: for each of the translated triples there is a joint norwegian/english triple consisting of one or two sentences in each of english and norwegian, see figure 1 for an example. the english/norwegian dataset contains about 531 thousand triples of sentences. for evaluation we also machine translate the standard english datasets for semantic textual similarity sts12-16 @xcite , @xcite , @xcite , @xcite , @xcite , stsbenchmark @xcite , and sick relatedness @xcite . the task is predicting how similar a pair of sentences are to each other on a scale of 0-5. we use these datasets only for validation and testing and never for training. in fig. 2 we see two examples from the translated sts benchmark dataset. the usage of translated datasets is a weakness compared to having original data in norwegian. this project can also be viewed as an exploration of what performance it is possible to get from auto-translated english datasets: to the degree they are shown to be useful, one will have much more data one could potentially work with in norwegian language processing. we note that for sentence similiarity, a similar exploration of translated data has been done for swedish in (isbister and sahlgren, 2020). they conclude that they do not recommend the usage of automatically translated sts datasets for fine-tuning, but that it should probably have limited negative consequences for comparing models. we partly follow their recommendation: we only use translated sts data for valdiation and evaluation, but we do perform supervised training on translated nli data. ## experiments our experiments follow the implementations in the simcse paper closely. we start with a pretrained encoder model that is either bert or roberta. for unsupervised training we sample one million texts from the ncc dataset. we then pass each text through the model using two different dropout masks to obtain two different text representations s i and s + i for each text. here dropout functions as a form of continuous augmentation of embeddings. then we contrastively predict which pairs of texts within a batch are the same using cross-entropy loss on the cosine similarity scores. in other words, the loss for text i is given by the english models are included as a sanity check: since we are using automatically translated datasets to choose the best models, we want to compare their performance with some models that are expected to perform worse than norwegian models. for the same reason we also test on the english sts datasets. we train the supervised models using nli data where each sentence has one paired sentenced labeled as entailment, which is regarded as a positive sample, and one sentence labeled with contradiction, which is considered a negative sample. we thus obtain three different sentence representations s i , s + i , s - i . as in the simcse paper, we train contrastively trying to predict the positive pairs, and add the negative sentence representation s - i to the loss function as follows: for training supervised models we start with the following models: we train with the same settings as in the sim-cse paper: we set a max sequence length of 32, and use the learning rates and batch sizes given in the appendix of the simcse paper (which vary by model type and size). each model is trained on a single nvidia 3090 gpu. for some models we have to use gradient accumulation to achieve the correct batch size due to lack of ram, which changes training dynamics a bit, since contrastive loss depends on the entire batch. we do not see any noticable effects on results from this. we train with the adam optimizer with linear weight decay and put a multi-layer perceptron (mlp) on top of the model for training. unsupervised we train for one epoch, and supervised for three. the best model is selected by evaluating on the dev part of the sts benchmark dataset. for evaluation we test both with and without this mlp, and find that generally, testing without the mlp gives slightly better results. we train three versions of each model and report average scores. the models are also fine-tuned on two norwegian sequence classification tasks. talk of norway (ton) is a subset of the norwegian parliament speeches dataset @xcite , where the task is to classify whether the speech was given by sv or frp (politically left or right, respectively) selected in @xcite . 3 norec is a dataset of reviews in norwegian from different domains such as movies, video games and music @xcite . from this dataset one can extract a binary classification task by taking the subset of reviews that are clearly positive or negative and letting the task be to classify them as positive or negative @xcite . we take the text representations made by the model before the mlp, and add a linear classification layer on top and fine-tune the entire model on the training dataset. for both the fine-tuning datasets we do a grid search for hyperparameters under the following conditions (these are the same hyperparameters as in the finetuning examples in the appendix of the original bert paper @xcite ): we use the macro f1 score on the validation set to select the best model for each training run. we do three training runs and report the average of test scores. ## results sentence similarity we evaluate the trained models on the semantic textual similarity datasets. we evaluate our models both on the norwegian version of the datasets, and the original english. we report spearman's correlation for the sts datasets. ## results classification we report macro f1 score for the binary classification tasks. ## discussion we believe that our models perform well on the semantic sentence similarity task, even if we do not have any strict comparison since this is the first evalutation of norwegian sentence embedding models on the sts data. the norwegian dataset corresponds to the english one, so the scores of english models on english sts and norwegian models on norwegian sts should in principle correspond to each other, but because of the extra noise added by the automatic translation we are not surprised that the norwegian numbers are a bit worse. we see that the models improve a lot compared to before training, and because they perform quite well even for the english sts datasets, we are confident that they have indeed learned something useful in norwegian. the supervised models perform better than our unsupervised models even though the supervised models are trained on machine translated data. this shows that machine translated data could be useful for doing nlp in smaller languages, at least for some tasks such as ours. the difference in the numbers we get for unsupervised and supervised training are similar to the ones in the original sim-cse paper. it is a bit unclear to what extent the specific content and language of the training data is important for performing well on sts tasks. for example, one can improve the performance of english simcse by training on unrelated image data @xcite . this might be because the task is a form of clustering, and images and text in other languages are structurally similar enough that the models learn something useful. from doing our experiments we get comparisons of the different norwegian language models. this is because this method of making sentence embeddings is mostly a way of extracting the knowledge already learned by the models, since the amount of training we do is much smaller than the amount the models already have been pre-trained. an unsuprising conclusion is that the scale of the model is the most important factor in making good language models. nb-bert-large is the best model by clear margins for all of our evaluations. this conforms to the general tendency in recent nlp that scaling up models is more effective than tailoring data or architecture on a given scale. next, we find that for binary classification the models nb-bert-base and nor-bert perform quite similary, while wikibert is generally a bit weaker, while all of them clearly outperform mbert. for sentence similarity we find different rankings among models: here unsupervised wikibert is the second best model, while the supervised version is the weakest of the norwegian supervised models. supervised nb-bert-base is clearly the second best model, while norbert performs worse on the sts task. we see that training sentence embedding models slightly improves performance on the binary classification tasks, but not by much compared with the base models. there is no clear tendency on whether training supervised or unsupervised improves performance on classification more, since the numbers we get are similar in both cases.