C laire: Large Language Models for Spontaneous F rench Dialogue (2024)

Abstract: Nous présentons la famille de modèles Claire, une collection de modèles de langage conçus pour améliorer les tâches nécessitant la compréhension des conversations parlées, tel que le résumé de réunions. Nos modèles résultent de la poursuite du pré-entraînement de deux modèles de base exclusivement sur des transcriptions de conversations et des pièces de théâtre. Aussi nous nous concentrons sur les données en français afin de contrebalancer l’accent mis sur l’anglais dans la plupart des corpus d’apprentissage. Cet article décrit le corpus utilisé, l’entraînement des modèles ainsi que leur évaluation. Les modèles, les données et le code qui en résultent sont publiés sous licences ouvertes, et partagés sur Hugging Face et GitHub.

## introduction a lot of information is shared through spoken conversation -in meetings, medical appointments, assistance calls, lectures, to give just a few examples. due to the spontaneous nature of such interactions, the kind of language we employ in these contexts can differ considerably from that found in text documents used to train large language models (llms). we hesitate, repeat ourselves, revise our wording in mid-sentence, leading to utterances that are ungrammatical by written standards. we might employ slightly different vocabulary and even syntactic constructions. (1) a : ok, c'est quoi le plus, le souvenir, le premier souvenir, le plus clair que tu as en tête, de nous deux ? b : le souvenir ? a : ton premier souvenir de nous en fait. b : le premier souvenir de nous le plus clair que j'ai, euh, je pense que c'est en algérie. je crois que c'est en algérie. et je sais pas, si tu te souviens. on était sur la place. et, euh, et on courait. in (1), it takes multiple lines and two turn changes for the speakers to establish that the question under discussion is simply, "quel est ton premier souvenir de nous ?". a reformulates their question multiple times in the first line, leading to constructions such as "le souvenir le premier souvenir" that we would not expect to find in a grammar book. there are filled pauses with expressions like "euh", which do not appear in written text. and we have focus constructions such as "le premier souvenir de nous le plus clair que j'ai, je pense que c'est..." that are characteristic of spoken language. a further contrast with most written documents is that conversation involves multiple speakers. one might hypothesize that a foundation model that has been trained to be sensitive to the kinds of interactions that we see in spontaneous conversations might serve as a better base for downstream fine-tuning on language generation and understanding tasks that focus on natural conversation. a model for meeting summarization or transcript querying, for example, would need to be able to navigate these types of interactions. it might be interesting to create chatbots that could produce more natural and informal language to improve the feel of interaction. in this paper, we describe the first step that we took to addressing our hypothesis : the creation of foundation models trained on transcripts of spontaneous conversation and theater plays. given what would inevitably be a small dataset, we decided to start with foundation models and continue pretraining-that is, to continue training with an autoregressive language modeling objective-rather than training dialogue models from scratch. we were curious to see how the additional french training impacted the quality of french responses and also how conversational we could get our models to be without specific chat-style training. for the latter reason, we chose pure foundation models that had not received specific instruction or chat training. the result was two models : claire-7b, based on falcon-7b, 1 and claire-mistral-7b, based on mistral-7b. we also decided to focus on french. a drawback of many current llms is that most of their training data is in english, which is a double cause for concern. not only might these models struggle to produce high-quality responses in other languages, but they will also have a poor representation of non-anglophone cultures. consider this example of mistral-7b foot_0 completion : the resulting claire models are quite conversational in french and even have a touch of humor. this is an example output from claire foot_1 : we emphasize that our models are foundation models trained to produce extended sequences of dialogue in one go, not to engage in back and forth chat with a human, and that it is expected for them to imitate spoken dialogue features, including disfluencies. in what follows, we present our dataset and data preparation approach as well as an overview of our model training and evaluation approach. ## data while the majority of data used to train high-profile llms comes from english (llama2, for example, uses 89.7% english data @xcite ), the large-scale datasets tend to include some data from other languages, especially web-crawled data. roots (responsible open-science opencollaboration text sources @xcite , which bigscience assembled for training bloom @xcite , contains 1.6tb of data from 59 languages, including french. 4 refinedweb @xcite , oscar @xcite , and redpajama @xcite are filtered and refined versions of commoncrawl data dumps 5 prepared specifically for llm training. these datasets do not concentrate on dialogue data as we did for our models, however. on the dialogue side, there have been recent efforts to assemble collections of dialogue datasets that can be used to train conversational ai agents, though these focus on english. dialogstudio @xcite regroups some well-known english spoken dialogue corpora, such as ami @xcite , icsi @xcite and mediasum @xcite . it also contains a variety of short, written dialogues collected through crowdsourcing. other similar but smaller-scale collections include instructdial @xcite and parlai @xcite . our dataset (see hunter et al. (2023) and https://huggingface.co/datasets/openllm-france/clairedialogue-french-0.1 ) is broken down in as reflected in for parliamentary proceedings, we expect the conversations to be slightly more formal and to include multiple participants. theater plays imitate multiparty spoken conversation to some extent, but have their own polished, theatrical style. in interviews, we expect more question/answer sequences, whereas free conversations are generally unguided discussions. the meeting category includes both real and simulated meetings and are expected to contain more structured interactions, though arguably less structured than those found in debates, where turns are controlled to a large extent by a moderator and questions are posed in a formal manner. finally, assistance interactions involve one person asking for information from another in a professional context while presentations involve longer monological sequences, frequently accompanied by question/answer sequences. the second column of ## data preparation and augmentation as our datasets came from a variety of sources, we had to contend with diverse data formats, and preparing a high-quality dataset for training often required dataset-specific solutions. some corpora came with punctuation, while others did not. different corpora adopted different annotation conventions for background noises, laughter, and other sounds. we attempted to standardize these annotations using a few tags like @xcite , as in (2), so that they could be easily found later if desired. the [pii] tag, which stands for "personally identifying information," marks anonymized content. because part of our aim was to train a model to represent the interactive turn taking characteristic of conversation, it was necessary that all transcripts contain speaker labels. recovering these labels was not always straightforward and sometimes required going into separate files to recombine this information with the transcript. we also standardized the style of the speaker label so that it could be easily identified for data augmentation, as explained below. in the end, we opted for a format in which speaker names were encased in square brackets and followed by a colon before the closing bracket as shown in example (2). with our conventions in place for marking non-verbal sounds, anonymized content and speaker labels, we were able to exploit the square brackets to augment our data. when dealing with machine learning models, we always have a choice : we can normalize the data so that the model is insensitive to things it should not be sensitive to, or we can augment the data to increase model robustness, without adding a preprocessing, normalization brick. we chose the latter option. first, we generated multiple variants for speaker labels by either enclosing the label in brackets, as described above and shown in (4-a) and (4-b), or by marking speaker changes with dashes as in (4-c). the bracketed variants contained either "intervenant" or a proper name. the former was chosen as a replacement for "speaker" after preliminary tests with falcon suggested that this label helped the model return responses in french. the option of using a dash (-) in conversations with only two speakers also came from experiments with falcon and mistral in which both models produced dialogues formatted in this way. for augmentation, we used the "intervenant" labels to add an anonymous alternative for corpora that used names and we used randomly generated proper names to add a named alternative for datasets that were anonymized. the choice between using a first and last name or only a first name throughout a transcript added another dimension for augmentation. next, we played with changing case and removing punctuation. ultimately, we wanted to design our models to be robust to different transcript formats, keeping in mind that these transcripts, many of which will be produced by asr systems, may need to handle transcripts without case or punctuation. combining these three options for data augmentation-speaker labels, case and punctuation-left us with up to nine different formats that we could pull from for training. (in some cases, as when a dataset did not contain punctuation or case, we did not have all nine options of course.) a final step in the data preparation pipeline involved cutting individual documents from our data set into smaller chunks that would fit into the context windows of falcon and mistral, which are limited to 2048 and 4096 tokens, respectively. to do this, we split tokenized documents so that each training sample begins with the start of a speech turn. ## model training because many of the original corpora in our training set were shared for research purposes only, we published claire-falcon and claire-mistral under a non-commercial license (cc by-sa-nc 4.0). simultaneously, however, we released two models under an apache 2.0 license, claire-apache and claire-mistral-apache, trained only on the datasets from our corpus that allow commercial use. to train our claire models, we used lora @xcite with bfloat16 precision. lora is a lightweight technique that greatly reduces the number of parameters to be trained, making training more efficient. because training with lora enforces the change in model parameters to be of low rank, an additional advantage of this method is that it greatly reduces the chances of catastrophic forgetting, in which a pretrained model forgets what it learned from its previous training. while lora is often associated with fine-tuning, we note that we applied it to all layers of the original models and used it to continue unsupervised training with an auto-regressive objective, and so our task was closer to pretraining than traditional fine-tuning. our hyperparameter configuration was : lora with rank @xmath0, adamw optimizer with a learning rate of 1e -4 , batch size of 128, dropout ratio of 0.05, weight decay regularization factor of 0.01, and gradient clipping of norm 1. training was carried out on the jean zay supercomputer run by genci (grand equipement national de calcul intensif) and installed at idris, the national computer center put in place for the cnrs (centre national de la recherche scientifique). to take advantage of jean zay's multi-gpu nodes, we used fully sharded data parallel (fsdp) @xcite , which shards a model's parameters, gradients and optimizer states across different workers instead of making a full copy of these states on each gpu, greatly facilitating multi-gpu training. we trained with 8 a100 gpus 10 , each with 80gb of memory and processed tokens at a rate between 7 and 8 million tokens per hour. figure 1 shows the convergence curves for the different variants of the model. the monitored loss is cross-entropy, which is in the case of multi-class classification (with discrete targets that are text tokens here) equivalent to the average negative log-likelihood. note that perplexity of the model is then the exponentiated loss. convergence curves show how fast that loss decreases on several hold-out validation subsets. the noisy background curve in light blue is the cost on the training samples estimated in real time during training. the first scale indicates the number of sequences that the model has seen up to a certain point, where a sequence can be a whole conversation or a proper part of a conversation when context size limitations forced to split the conversation. the second scale shows the number of training (subword) tokens, including padding tokens that are ignored in the loss function : this total number of tokens simply equals the number of sequences multiplied by the training context size (2048 for falcon, 4096 for 10. only claire-falcon was trained on a single gpu, with a slightly different batch size (132 instead of 128). mistral). the last scale measures training gpu hours. with around 50h gpu hours, training a claire model consumes a total energy around 25 kwh on the jean zay supercomputer, equivalent to around 1.5kg c02 eq following the estimation of @xcite . the thick grey line is the online validation curve that we used for early stopping ; that is, training is automatically stopped when the online validation loss does not decrease for some time. the colored lines show the loss curves for our data by subcategories. to get these results, we produced model checkpoints every hour and then tested the different types of data at the various checkpoints. the legend in the figure on the right sorts the data sets by level of learning difficulty. we see that the theater pieces were the most difficult to learn, probably because they were quite heterogeneous in nature, ranging from old french up through modern plays. after that come conversations, then meetings, speeches, debates, interviews, the national assembly, which is our biggest dataset, and then call-center type dialogues, which seem to be the easiest. figure 1 shows that overfitting appears earlier and more prominently for the apache models, which is explained by the fact that they did not see all of the data presented in section 2. it might also seem to suggest that that claire-mistral (bottom left) learns more efficiently than claire-falcon (top left) and converges to a lower loss. however, the evaluation results presented in section 5 reveal the opposite, showing that average token perplexity is not relevant to compare models with different tokenization and different context sizes. ## human evaluation evaluation of causal foundation models-which have only been trained with an autoregressive training objective and not fine-tuned to follow instructions or perform a specific task-is a nebulous affair. the only gold data against which model output can be compared is data held out from the training set for validation, which is just a diverse collection of (punctuated) sequences of words. for this reason, foundation models are often evaluated for perplexity, or how well they learn to match a probability distribution over word sequences with the actual probability distribution of the validation set. this does not mean that foundation models should not be evaluated using standard benchmarks, of course : such evaluation can facilitate the comparison between a model's behavior before and after fine-tuning and can be used to shed light on how different training corpora can impact the model. it's just that the output of these evaluations should be taken with a grain of salt. in our case, we saw that our claire models responded to their continued training and wanted to get a clearer idea of how. in particular, we wanted to see to what extent the dialogue data encouraged a style of language that appears interactive, conversational and spontaneous. standard benchmarks, however, tend to focus on knowledge and reasoning-based tasks, including general knowledge tasks (e.g., squadv2 ; @xcite , natural language inference (e.g., hellaswag ; zellers et al., 2019), coding (e.g., mbpp ; austin et al., 2021), maths (gsm8k ; @xcite , and occasionally a combination of multiple domains (e.g., mmlu from @xcite and big-bench from bench authors ( 2023 )). not even mt-bench @xcite , which assesses chat interactions, focuses on a model's ability to generate spoken-style conversational interactions. there are also limited options for benchmarks in french. one approach is to translate the existing resources into other languages, as is done by bactrian-x @xcite , taco @xcite and mt-bench-fr 11 , though it has been noted that the translation process can compromise the quality of the dataset, and thus of the evaluation. another approach is to use the high-quality but limited evaluation resources available for a specific target language as bawden et al. to evaluate our claire models, we developed our own approach to human evaluation that targets conversational abilities, evaluating model output along three dimensions : interaction, fluency and relevance. for interaction, our main interest was to determine not whether the model produced coherent language but rather if it succeeded in acting like it was engaging in conversation. we looked for evidence of turn taking, direct addressing of the "other speaker(s)", and expressions used to smooth conversation such as "well yeah". (5) shows an example question from the interaction dimension. en cas de dialogue avec plusieurs échanges, semble-t-il que les interlocuteurs cherchent à engager une discussion (en se tutoyant/vouvoyant directement, répondant aux questions, utilisant des expressions conversationnelles comme "oui, c'est vrai", etc.) ? fluency questions focused on such factors as whether the model consistently output french, whether the french was acceptable for oral conversation, and more generally, whether the model output seemed human. for relevance, evaluators were asked to judge whether the response actually addressed the prompt, whether the response stayed more or less on topic, whether it was logically coherent. we carried out two evaluation campaigns. in both, each evaluator was asked to review a set of five surveys, where each survey included one prompt together with the generated responses from the 11. https://huggingface.co/datasets/bofenghuang/mt-bench-french four models that we wished to compare : falcon, mistral, claire-falcon and claire-mistral. for each generated output, evaluators were asked 13 questions (covering the three dimensions) and additionally asked to rank the four generated outputs. we created 8 distinct groups of surveys, leading to 40 unique surveys for each campaign. each survey was reviewed by two people. for each campaign, we made a list of 10 prompts and varied the form of the speaker label along two dimensions. in the first campaign, we used either the [intervenant :] label or no speaker label at all, keeping a 50/50 distribution in our surveys. the idea here was two-fold : use monologue-style prompts to test whether the claire models would output dialogue even without dialogue-specific prompting and also minimize the bias towards our models, which were trained on [intervenant :] labels. in the second campaign, all prompts included speaker labels. here we opted for a 50/50 distribution between labels with brackets and those without. the former category included [intervenant :] labels but also proper names in brackets ; the latter included turns marked with dashes and proper names without brackets. the idea again was to make sure that half of the prompt variants were in a style that had been seen by the claire models while the other half remained neutral for the base models. we also randomly selected two generated responses per prompt variant (per model) so that each of the 10 prompts was seen in four different ways (2 speaker-label styles x 2 samples) throughout the surveys. in the second survey, we varied the nature of the prompt. in the first round, our ten prompts (a mixture of interrogative and declarative prompts) were what we call "culture prompts". these, like dis-moi, où est-ce que je peux trouver des cèpes ?, were designed to discuss concepts that were assumed to be better represented in french training data than in english in order to target the models' cultural training as well. in the second round, we kept five culture prompts that we felt were the most natural from the first round and added five "casual" prompts in which we asked the model generic questions such as "what will you cook for us today ?" or "what are your favorite french novels ?". this shows claire-falcon's response to the casual prompt : our two campaigns of evaluation revealed certain interesting tendencies (see appendix b for more details). first, while mistral had a clear tendency to output english or switch to code or other data formats when prompted in french, our claire-mistral model did not, suggesting that a small amount of additional french training was sufficient to correct the problem. claire-falcon was consistently rated above claire-mistral, while mistral was consistently rated below the other three models. note that this ranking of mistral runs contrary to what one might expect from the perplexity results in figure 1 in the appendix, underscoring our warning from section 4 that perplexity does not permit a direct comparison of the models. in both rounds of evaluation, the question that correlated the most with a model's overall preference rank was "does the output seem human ?", which underscores the difficulty of evaluating foundation models on a task like spontaneous dialogue generation. in second place, positive responses to whether the model generated excessive disfluencies were negatively correlated with model preference, even though such disfluencies are common in spoken language and ultimately quite "human". the comparison of claire-falcon and falcon was more complex. in the first round, claire-falcon was the clear winner, but this advantage disappeared in the second round. in the end, we traced the difference in results to the nature of the prompts : the results for claire-falcon and falcon on culture prompts were consistent in the two rounds of evaluation. however, falcon outperforms claire-falcon on casual prompts, which were introduced only in round two, explaining why claire-falcon was roughly tied with falcon in the second round. these results show how much llm performance can be influenced by subtle, and often seemingly unexplainable, differences in prompts. because of the incredible cost of human annotation, we decided to carry out our evaluation experiments using gpt-3.5 and gpt-4 as evaluators, following the llm-as-a-judge evaluation paradigm. unfortunately, testing on december 14 and 15, 2023, we found that both models, and especially gpt-3.5, were unable to differentiate the performance of the models we were evaluating, especially on subjective questions, adding to the already existing body of evidence that llms are unable to match or replace human evaluators when it comes to assessing human preference @xcite . ## conclusion we have presented the claire family of language models, a set of models designed to focus on dialogue dynamics with the aim of improving performance on downstream tasks involving spoken dialogue understanding. this work complements that of @xcite , though our datasets are largely manually transcribed and include more spontaneous spoken language (as opposed to language from tv or radio). our models were produced by continuing the training of the falcon and mistral 7b foundation models on conversation transcripts and theater plays. a key feature of our models, apart from the dialogue component, is their focus on french, contributing to a growing trend to emphasize multi-linguality in language models @xcite @xcite . our two principal models are released under a cc by-sa-nc 4.0 license 12 while the other two, trained on a subset of our data that allows commercial use, have an apache 2.0 license 13 . the github repository with code used to train our claire models is also available publicly 14 . we are currently working on extending the approach presented in this paper to train a bilingual english-french dialogue model with the objective of fine-tuning the resulting model for meeting summarization in order to test the impact of our dialogue pretraining on downstream tasks involving dialogues.