Applying the transformer architecture on the task of headline selection for F innish news texts (2024)

Abstract: The paper evaluates the possibilities of using transformer architecture in creating headlines for news texts in Finnish. The authors statistically analyse the original and generated headlines according to three criteria: informativeness, relevance and impact. The study also substantiates for the first time the effectiveness of a fine-tuned text-to-text transfer transformer model within the task of generating headlines for news articles in Finnish. The results show that there is no statistically significant difference between the scores obtained by the original and generated headlines on the mentioned criteria of informativeness, relevance and impact.

## introduction the headline of any text plays one of the most important roles. today, online media are ahead of their paper counterparts in terms of popularity, timeliness and mass appeal. the diversity of internet media emphasises the importance of the task of creating unique and attractive headlines. more and more attention is paid to search engine optimisation, the main goal of which is to attract new users and increase website traffic. since electronic media is a sphere of accumulation of huge arrays of text data, the question of its optimisation and automation is among the problems of modern computational linguistics. creating headlines specially adapted to search engines can help increase the visibility of news articles and, consequently, increase traffic. moreover, creating headlines manually is a labour-intensive and time-consuming process, which does not meet the requirements of the responsiveness of modern electronic media. the aim of this paper is to critically evaluate the effectiveness of transformer architecture in creating headlines for news texts in finnish. the paper statistically analyses original and generated headlines according to three criteria: informativeness, relevance and impact. it is also the first attempt to substantiate the effectiveness of a fine-tuned text-to-text transfer transformer model in the task of generating headlines for news articles in finnish. ## related work the popularity of transformers has led to the development of a wide range of pre-trained language models based on this architecture. transformers' strengths, such as its deep understanding of natural language and its ability to focus on particularly meaningful information in input data, have made models based on it effective tools, including for headers genetation. applying fine-tuning to pre-trained models on specific tasks allows the knowledge gained from pre-training to be extended to a new task. the results of fine-tuning pre-trained models to generate news headlines in russian were demonstrated in @xcite using the mbart and bertsumabs models as examples. the multilingual bart model (mbart) is a language model based on the transformer architecture, consisting of an encoder and an autoregressive decoder, and pre-trained on large-scale monolingual corpora covering 25 languages @xcite . the training process is built on reconstructing a document from its noisy version, which leads to significant improvements in machine translation quality at both sentence level and whole document level (ibid). the bertsumabs model uses bidirectional encoder representations from transformers (bert) as the encoder and a randomly initialised 6-layer transformer as the decoder @xcite . the decoder is pre-trained and the decoder is trained from scratch, so the tuning may be unstable. to overcome the mismatch, the optimisers of the encoder and decoder are separated (ibid). the fine-tuning of the selected models led to a significant improvement in the results: the rouge metric scores increased on average by 2.9 points compared to the previous leading edge performance of the phrase-based attentional transformer model @xcite and by 2.0 points compared to copynet @xcite . the performance of bertsumabs was also shown to be higher within the task of generating headlines in russian; moreover, the bertsumabs model produced headlines of a more abstract nature, while mbart was more prone to copying @xcite . human evaluation confirmed the effectiveness of the developed models: headlines produced by bertsumabs were selected by five or more experts in 32% of cases, while original headlines were selected by five or more experts in 28% of cases (ibid). in @xcite , the authors discuss finnish news headline generation using gpt-2. the key issue in developing the model was the possibility of using the proposed system as an auxiliary tool in real journalistic practice. the generated headlines were expertly evaluated by journalists from a finnish publishing house. as a result, the generated headlines were very close to being practically usable, and although the concrete implementation is not yet ready to become a fully automated headline generation system, as it still needs human control, the algorithm may well be applicable to potential needs. ## models as part of our work, it was decided to use finetuning of the pre-trained model. during the training, experiments were conducted on different variations of the gpt-2 and t5 models for finnish, provided by the finnish-nlp community and publicly available on hugging face (finnish-nlp, hugging face). in the first stages, both gpt-2 and t5 demonstrated the ability to generate coherent results for the task at hand, but in order to increase the novelty of the study it was decided to continue working with the t5 models. t5 @xcite is a model introduced in 2020 by the google ai team, which has fundamentally the same standard structure as the original transformer, consisting of 12 pairs of encoder-decoder blocks, using the self-awareness mechanism, direct communication network and ecoder-decoder attention described in @xcite . one of the characteristics of the structure of t5 models is the use of relative scalar embeddings, which are a type of positional coding. unlike absolute positional encodings, which assign a unique encoding to each position in a sequence, relative positional encodings encode the relative position between two tokens. to do this, the difference between the positions of the two tokens is computed, which is later used to compute a scalar value that is added to the embedding of the token. the language model for finnish, finnish t5, was pre-trained on a combination of 6 datasets: the finnish language subset of the mc4 dataset and wikipedia, yle finnish news archive 2011-2018 and 2019-2020, finnish news agency archive (stt) and the suomi24 sentences corpus. the raw datasets were automatically cleaned to improve quality and weed out non-finnish examples. the result was a raw dataset containing about 76 gb of text. the texts were not lower-cased, so the model retained case sensitivity. a masked language modelling (mlm) task was used in the pre-training process. all variations of the t5 model presented by the finnish-nlp community were studied and the following ones were selected as a result of preparatory experiments: ## experiments for the experiments, it was decided to select 10 non-training articles from each of the six sources that were included in the fine-tuning corpus. 10 news articles from the news resource yle were also added, as this company is one of the leading news providers in finland: according to yle's annual report 2023, the weekly reach of yle's online and mobile services (including yle areena) was 81% of the population (yle's annual reports, 2023). the fine-tuning did not emphasise learning any particular topic category, so articles for the experiments were also selected from different topics. to fully analyse the model, headlines were generated using the most sophisticated implementation of finnish t5 (t5-large-nl36finnish) to see the results of the untuned model. the following hyperparameters were used for this network (see a case of correct formation of a compound word deserves a positive assessment. in the input text there were the words "ravintolamoguli", consisting of two "ravintola" ('restaurant') and "moguli" ('tycoon'), and "yökerhojen", where "yökerho" ('nightclub'), '-jen' is the plural ending of genitive). taking these words as a basis, the model generated a new compound word in the title "yökerhomoguli". it is also worth noting the case where the model attempted to capture the headline formation style characteristic of finnish news outlets. for the resources participating in the study, the following headline structure is typical: a word or phrase summarising the essence of the message, containing a key named entity or indicating the source of information, followed by ':' and the main part of the headline, e.g. "ranskalaistutkijat: kännykkäkielto ei riitärajat tarvitaan kaikkeen ruutuaikaan, myös television" ('french researchers: banning mobile phones is not enough -we need restrictions on all screen time, including tv') or "ennuste sen kuin paranee: luvassa jopa 24 astetta" ('the forecast is improving: up to 24 degrees is expected'. the model generated the following example headline: "tiktok: nuoret etsivät tietoa ja seuraavat uutisia useimmiten" ('young people are more likely to seek information and follow the news'). for the generated and original headlines we calculated the average rouge (see ## results the simpler network t5-mini-nl8-finnish as a result of fine-tuning stably generated the token 'äijä', recognised as problematic because it does not carry any semantic load, but nevertheless fulfills the role of a full member in the generated headers. the most productive was the byt5-base-finnish network fine-tuning, which is based on processing text directly at the byte level. this is most likely the reason why the network does not allow the generation of a problematic token. for the previously discussed finnish t5 implementations, it is assumed that there were errors in the training of the model or tokenisers. the results obtained with the pre-trained byt5base-finnish network are diverse: by content, the generated headings were categorised into classes such as nominative, predicative and punctuated; by the techniques used, they were categorised as transitive, nominative, opinion and clickbait headings. among the 70 headings generated, only three factual errors were found. the questionnaire included 5 randomly selected news articles that participated in the study. a total of 20 experts participated in the survey. for each criterion, the maximum amount of points a headline could receive if each of the 20 experts gave a score of 5 was 100 points. the average score on a five-point scale for original headlines on the informativeness was 3.90, for generated headlines reached 3.57. the student's t-test resulted in a p-value above 0.05, so the null hypothesis of no statistically significant differences between the mean scores on the informativeness is accepted. for the relevance, the original headlines received a mean score of 3.86 on a five-point scale, while the generated headlines received a mean score of 3.75. as in the case of testing the previous criterion, the t-test confirmed the null hypothesis that there are no statistically significant differences between the sample scores on the relevance criterion. for original headlines, the mean score on a five-point scale on the impact is 3.69, while for generated headlines it is 3.67. for the t-test, the null hypothesis is also confirmed, namely that there is no statistically significant difference between the sample scores on the impact criterion. thus, the hypothesis put forward in this study is confirmed: there is no statistically significant difference between the scores obtained by the original and generated headlines according to the criteria of informativeness, relevance and impact. this means that the results of neural network work are close in quality to the results of human work. ## conclusion in this paper, the results of a practical application of the transformer-based finnish t5 model were studied in a task of generating finnish headlines from input news text. a critical evaluation was carried out, noting both the strengths and promising aspects of the different implementations of the model, as well as points still in need of improvement. the byt5-base-finnish network performed the best. the experts' evaluations indicate that the network shows sufficiently high potential in all the criteria considered (informativeness, relevance and impact) to be useful, for example, as an auxiliary tool for creating headlines in finnish. news content authors can use the headline variant proposed by the network as a basis for further work. to be fully self-sufficient, the network still needs to improve its reliability, namely to get rid of factual errors. such a problem can be solved by learning from a larger example dataset, but this also requires more powerful computational resources.