Building a S wedish Open-Domain Conversational Language Model (2021)

Abstract: We present on-going work of evaluating the, to our knowledge, first large generative language model trained to converse in Swedish, using data from the online discussion forum Flashback. We conduct a human evaluation pilot study that indicates the model is often able to respond to conversations in both a human-like and informative manner, on a diverse set of topics. While data from online forums can be useful to build conversational systems, we reflect on the negative consequences that incautious application might have, and the need for taking active measures to safeguard against them.

## introduction dialog is an important means through which machines can exhibit intelligence toward humans, which is interesting from a general ai perspective. but dialog also constitutes a natural interface for humans to interact with technology, which opens up for a breadth of applications involving complex information acquisition, automation of tasks and smart support systems. a promising direction towards this goal is the development of open domain conversational systems using large neural networks. early approaches to neural conversational systems rely on various forms of recurrent neural networks (rnn) trained autoregressively to model the textual sequences @xcite @xcite . more recently, as large pretrained transformer networks have come to dominate progress in nlp in general @xcite @xcite @xcite , approaches such as dialogpt @xcite , meena @xcite and blender @xcite have proven the architecture's applicability in open domain dialog systems as well. however, as the research effort is predominantly put into making progress on english, the importance of making progress in other languages as well has been noted @xcite . each language is its own unique challenge for many reasons, but the difference in availability of resources is a major one, in particular for data-driven methods. we argue this is also important to keep the public debate on the risks and ethical aspects of large scale language models open to non-english speaking communities. toward those ends, we present the first (to our knowledge) attempt to build a large scale open domain dialog system in swedish based on data from flashback, one of the largest social discussion forums in sweden. we also present early indicative results on a human evaluation to assess its response generation capabilities across a wide range of topics. ## data and preprocessing flashback foot_0 is a swedish online forum that launched in 1996 and has since grown to become one of the country's most popular social medias @xcite . in the various sub forums, a breadth of topics are openly discussed including computers and programming, economics, politics, sports and science. to the general public however, the forum is also widely known for housing an anonymous safe haven for controversial subjects such as prostitution, drugs and conspiracy theories @xcite . due to its consistent popularity over the last two decades, it arguably today makes up sweden's biggest single source of general conversational text. on flashback, posts are chronologically organized into threads. in a single thread, the discussion is centered around a specific topic typically number of layers 48 dimensionality 1600 feed-forward dim 5400 number of heads 16 number of parameters 1.4b max context length 400 batch size 512 optimizer adam vocabulary size 52,000 the data was tokenized into strings of bpe tokens @xcite using a customly trained vocabulary. due to flashback's organization of posts into a single linear feed (unlike the tree structure on e.g. reddit), it is common that users quote the previous post they respond to, to avoid confusion. as a quote holds important contextual information to a post, we chose to explicitly include this in the way we formatted the threads. more details of how the data was formatted into strings can be found in appendix a. ## model following previous works on open-domain dialogue systems @xcite , we trained an auto-regressive language model using a slightly modified transformer @xcite decoder as proposed by @xcite . that is, for an input sequence of tokens x 1 , ..., x n , the language model is trained to maximize the likelihood of the joint probability: we denote our model flashback-gpt, where gpt is an acronym for generative pre-trained transformer as first coined by @xcite . the hyper-parameters chosen are similar to those of the largest variant of gpt-2 @xcite , and are detailed in the model was trained on 16 nvidia tesla v100 sxm2 gpus for 7 days, equivalent to 86,250 gradient updates. the learning rate was increased linearly for the first 5,000 steps up until 5e-5, after which it was kept constant. we used the deepspeed @xcite library to optimize memory efficiency across the devices during training. ## evaluation evaluating natural language generation systems is known to be hard. even though it is common to conduct automatic evaluations due to their low cost, a human evaluation often serves as an additional validation of the results. however, designing a human evaluation to measure a specific quantity is also not trivial since there is always room for interpretation among the human annotators. therefore, we present a pilot study where the main aim is merely to get early indications rather than definite results, and to guide the design of bigger future studies. we design our pilot to measure our quantity of interest: to which extent is the model capable of participating in social discussion forums across a diverse set of topics? to that end, we seek to measure two quantities: humanlikeness and informativeness. as language models can often be inconsistent and show lack of commonsense knowledge, humanlikeness is supposed to answer if there is anything in a response that seems off, suggesting it has not been written by a human. however, a response can be humanlike but still uninformative. the notion of "informativeness" is particularly interesting in our setting as forums can be relatively knowledge centric, and uninformative responses such as i don't know add little to the discussion. ## broader implications conversational models such as that presented in this paper can be understood as part of a broader transformation of communication. as argued by guzman and lewis (2020), we are now moving away from the traditional view of communication as anchored in human such. how we apply and evaluate conversational models going forward may come to alter the way we relate to each other as communicators, and ultimately, humans. there is need for informed discussion around what constitutes desirable use. while highlighting the risks of these emerging technologies could be considered detrimental, we believe it to be an important means towards enabling the inclusion of diverse perspectives in this discussion. a prominent issue related to nlp is found in the notion of bias. explicit and implicit biases concerning gender, race or disability can be embedded in e.g. text corpora @xcite , word embeddings @xcite and generative models @xcite . employing biased conversational models risks scaling systematic discrimination of various groups in society. when developing conversational technologies, we must acknowledge that they can be used for malicious purposes. as generative language technology improves and grows in swedish, so will its ability to manipulate and deceive at scale. as noted by the swedish defence research agency (foi), recent developments within generative language technology present risks of increased computer-generated false news and comments -predominately on social media -possibly posing a national security threat @xcite . potential harm must also be considered on the individual level. in 2020, a gpt-3-powered (brown et al., 2020) bot engaged in reddit-forums with 30 million users about sensitive topics such as suicide and conspiracy theories @xcite . with the indicative model performance demonstrated in this article, such human-machine communication could soon transpire in swedish. ## conclusions and future work we demonstrate that flashback can provide a base on which to build general conversational systems in swedish. while our early results suggest the model is often capable to converse across a diverse set of topics, more work remains to examine its utility on various conversational tasks. we also believe developing methods for grounding the responses in additional data is an interesting direction to further the performance on in-formativeness in particular. however, we also believe particular care should be taken as the underlying data is known to contain toxic content. this points to the importance of putting our model through further scrutiny in following work, to better understand its biases, how they are manifested in downstream tasks, and how they can be mitigated. towards those ends, we intend to make the model available for such purposes, and more information is available at https://github.com/tobiasnorlund/ flashback-gpt listing 1: example of a formatted training record. the usernames are anonymized.