MEETING : A corpus of F rench meeting-style conversations (2024)

Abstract: We present the MEETING corpus, a dataset of roughly 95 hours of spontaneous meeting-style conversations in French. The corpus is designed to serve as a foundation for downstream tasks such as meeting summarization. In its current state, it offers 25 hours of manually corrected transcripts that are aligned with the audio signal, making it a valuable resource for evaluating ASR and speaker recognition systems. It also includes automatic transcripts and alignments of the whole corpus which can be used for downstream NLP tasks. The aim of this paper is to describe the conception, production and annotation of the corpus up to the transcription level as well as to provide statistics that shed light on the main linguistic features of the corpus.

## introduction spontaneous, multiparty conversation poses particular problems for both speech and natural language processing tasks. disfluent speech, non-standard or incorrect grammar, idiosyncratic speaker styles, and overlapping speech and interruptions -not to mention transcription errors introduced when transcripts are automatically generated -complicate the task of understanding conversation transcripts, especially for nlp models that have been trained largely on text @xcite . these linguistic features together with complicated acoustic conditions, shared microphones, different accents, and rapid speech aggravate the problem for speech processing tasks such as transcription and speaker identification @xcite . progress on these topics is hindered by a lack of data especially for specific tasks such as meeting summarization. few people are comfortable sharing recordings of their discussions and even if they are, preparing the data for training poses a significant hurdle. data scarcity is even more serious when we look to languages other than english. the principal goal of the summ-re project is to develop models for conversation and meeting summarization with a particular focus on french. a critical step in accomplishing this goal has been to collect and prepare a corpus of roughly 100 hours of meeting-style conversations in french. the aim of this paper is to present this corpus and to detail its conception and production. icsi consists of natural, weekly meetings that last about one hour each for a total of roughly 72 hours of recordings. on average, the meetings involve six participants that can include undergraduates, graduate students, and professors who meet to discuss technical topics related to natural language processing, computational linguistics and even the icsi corpus itself. as icsi contains real meetings between people who were actively collaborating on projects at the time of recording, the interactions are more natural than those in ami. at the same time, they draw on technical vocabulary and specialized subjects as well as a considerable amount of shared knowledge between participants, which can complicate the interpretation of the content for models that do not have this knowledge. while such a scenario is undeniably realistic, the summ-re corpus is designed to strike a balance between naturalness and feasibility for automatic summarization. as such, it includes light guidance of the meeting structure and limits the impact of shared background knowledge by bringing in participants who in many cases did not know each other before participating in our corpus. while both ami and icsi are entirely in english, there have been recent efforts to expand to other languages. vcsum (versatile chinese meeting summarization dataset @xcite is a collection of transcripts and videos of roundtable meetings in chinese found on the internet. a total of 239 meetings were selected for a total of over 230 hours of recording time. the corpus is called "versatile" because it contains a variety of annotations that can be relevant for different summarization tasks. the elitr corpus @xcite contains transcripts for 113 technical project meetings in english but also for 53 meetings in czech, for a total of over 160 hours of content. like icsi, the meetings in elitr are natural, leading to many of the same advantages and drawbacks. unlike icsi, ami, vcsum and summ-re however, the elitr audio files have not been released and parts of the meetings are censored for privacy. there are also a variety of smaller, conversational corpora in french (for a recent list, see @xcite . cid (the corpus of interactional data @xcite , which contains eight one-hour dialogues between friends, has notable similarities with summ-re in that a major effort was involved in adding different levels of annotation, including dialogue-central information that can be exploited by downstream nlp models. with only eight hours of recording, however, it remains very small and is not focused specifically on meetings. ## corpus design to bypass the concern of sharing personal information, the summ-re corpus does not contain real meetings. the conversations were nevertheless designed to imitate certain important features of meeting-style conversation, developing situations in which participants have to make decisions or plan out a list of action items or report on things they have done. they were also designed to have some continuity with past meetings, as real meetings often do : almost every one of the 96 individual experiments in the corpus is made up of a series of three 20 minute meetings that develop a given topic. 1 in general, foot_0 the participants were asked to plan an event during the third meeting, while the second meeting focused on deciding what kind of event to plan, and the first meeting involved participants going around the table to discuss their experience or opinions about certain topics. each one-hour experiment contains the same set of participants-usually four but sometimes fewer-throughout. in some cases, meeting participants knew each other before the experiment. this condition adds an arguably realistic element to the interactions, as many meetings are held by people who have worked together before. however, because our meeting scenarios were artificial, we also feared that it would encourage playful interactions full of jokes and laughter. while such interactions are certainly possible in professional meetings, many meetings involve a higher level of seriousness and personal distance. in an effort to vary the interactions and imitate different levels of professionalism, we aimed to balance the number of groups in which all participants knew each other, some participants knew each other or no participants knew each other. to encourage participants to speak naturally and spontaneously while also pushing them to stick to a meeting-like agenda, we had to strike a balance between role playing and freedom for the participants to talk about their own experiences. to encourage freedom, we defined a set of topics that we assumed most participants would be comfortable discussing, including a) internet and technology-related topics like social networks and the societal influence of companies like google and amazon, b) films and tv series, c) fundraising, 3 d) music, e) environment-related topics such as global warming and renewable energy. within these larger topics, participants were allowed to choose a subtopic that interested them. to add structure to the conversations, we proposed a series of subtopics or responsibilities that each participant could choose to lead as well as a series of points, questions, or tasks (depending on the format of the meeting) that the participants might want to pursue. for each subtopic, the participants received an individual card or a collective document listing the points so that they had a "cheat sheet" if they struggled to develop their contribution to the conversation. here, for example, is the set of questions (translated from french to english here) given to a participant who chose to discuss amazon during a reporting meeting : you have chosen to be responsible for leading the discussion about amazon. you will need to be able to talk about this subject for 3-4 minutes. to help guide you, below is a list of questions to which you might respond (but you are free to choose other questions if you find them more suitable) : finally, for each 20 minute conversation, the group was asked to choose a moderator among them. in addition to managing the discussion for a 20 minute conversation (also following suggested guidelines), each moderator was assigned the task of taking notes and of making an oral summary at the end of the meetings based on these notes. to preserve the continuity of the conversation, the summary was recorded immediately after the conversation on the same record. the summary file was later extracted in post-processing. ## data acquisition the original plan was to record the entire summ-re corpus in the h2c2 4 . unfortunately, corpus collection began in 2020 and ran through 2021 when the covid pandemic was still a threat, so in-person meetings were not always an option. we thus adopted two different strategies for corpus collection : in person recordings and zoom. in all cases, the recordings shared certain characteristics. for instance, each one hour experiment involved the same set of participants. in general, there were four participants but sometimes, we were only able to find three and on very rare occasions, two. each participant had their own microphone for recording, as explained below. finally, each experiment was led by one of the co-authors of this paper, who would begin by giving instructions to the participants and making sure they understood the task, but would leave the room during recording to avoid interaction with the participants. most in-person recordings took place in the h2c2, though there are a few exceptions in which it was 4. https://plateformeh2c2.fr/ easier to go into peoples' homes. participants for these recordings were generally recruited through our platform, calls for participants in social media or announcements made in university courses. in total, 248 out of 283 meetings were recorded in person (see the h2c2 studio is composed of two rooms : while recording conditions in the h2c2 were near ideal, we encountered two problems that impacted the quality of the final recordings. first, although the room was spacious enough to leave a fair amount of distance between speakers and each headset microphone was adjusted to the voice of the person wearing it, sometimes a participant's voice would vary between the test conditions and the final recording conditions. a speaker might end up using a higher pitch during recording due to elevated emotion, for example. in such cases, individual microphones would often end up capturing the voices of multiple speakers. our efforts to isolate the contributions of the main speaker in such cases are described in section 5. the second complication resulted from efforts to navigate health and safety regulations enforced during the covid pandemic. to reduce contact between speakers, we first tried installing plexiglass barriers so that participants could see each other's mouths, but this led to an echo in the recordings. ultimately, we asked the participants to wear masks. this worked in most cases, but idiosyncratic approaches to mask wearing still led to suboptimal recording in some cases. ## annotation and post-processing the summ-re corpus is partitioned into train, dev and test data sets with proportions at roughly 75%, 12.5% and 12.5%, respectively. files were assigned one of three classes randomly with the constraint that the ratio of zoom experiments to in-person experiments as well as relative proportions of scenarios were kept roughly constant. after this automatic process, minor adjustments were made to ensure that certain files that had been manually corrected ended up in the dev set. this procedure resulted in 210 files in the train set, 36 files in the dev set and 37 files in the test set. the entirety of the summ-re corpus has been automatically transcribed and the entirety of the dev set, roughly 12 hours, has been manually corrected and annotated. correction of the test set is underway and will soon be complete. manual correction is performed in two phases. first, the transcripts are manually corrected for transcription errors. corrections at this stage are made with the software praat @xcite . in places where whisper misses a substantial 5. https ://www.prolific.com/ pause (i.e. on the order of hundreds of milliseconds) a pause marker # is added. if the ipu boundaries differ significantly from the true boundary, this is also manually corrected. all files are then manually verified for any obvious errors made during the first correction as well as minor adjustments such as adding non-linguistic annotations for non-speech sounds like laughs and coughs to improve tier boundary accuracy. on the dev set, corrections took one month and were carried out by one of the co-authors who was paid for the task and who is a native speaker of french. manual verification was performed by another co-author (non-native speaker). while the details of our automatic pipeline are described in @xcite , we give a brief overview here. at a high-level, the pipeline can be divided into two parts : the detection of inter-pausal units (ipus)-segments of audio in which the principal speaker is speaking-and the transcription of the words that were uttered in the ipu along with their start and end times. ipu detection was necessary because, as explained in section 4, individual microphones often captured the voices of multiple speakers. for this task, we employed out-of-the-box ipu annotation provided by the sppas package @xcite combined with an approach which relies on speaker diarization using the pyannote package @xcite . this second step was necessary because the voice intensity processed by sppas and the voice quality processed by pyannote contain complementary information. once the ipus were identified, we created new audio files in which background voices were replaced with silences and then passed the resulting audio files to whisper @xcite , openai's speech to text model. 6 the quality of the annotation and alignment predicted by our pipeline was extensively evaluated in yamasaki et al. (2023) on a 3.3 hour subset of the dev set. a final point is that in order to prevent identification of individual participants we developed a script to remove participant names from both audio files and transcripts and replace the corresponding audio intervals with a beep. this was done by : -identifying possible candidates for mentions of individual names by calculating the levenshtein distance of each token to participants' first and second names -manually filtering false positives -replacing the resulting list of names by anon and the corresponding wav interval with a single beep this anonymization scheme assumes the correctness of the transcription, which is not an issue for the 6. there are several variants of whisper models such as @xcite ; @xcite . we chose louradour (2023) after a detailed comparison of their performance as detailed in @xcite . we opted for the large v2 model as a starting point but added two custom features. the first employed prompting to encourage whisper to transcribe disfluencies, yielding a more faithful transcript. the second involved introducing precise word alignment through techniques developed in @xcite and the julius forced aligner @xcite . further details of both the ipu detection algorithm and the whisper transcription process can be found in @xcite . ## basic corpus statistics the summ-re corpus includes a total of 207 unique participants. due to the recruitment procedure described in section 4, our data is biased with respect to age, gender and occupation. in particular, the majority of participants were students with a mean age of 28.7, there are nearly three times more female than male participants, and a large portion of participants were monolingual french speakers who grew up in france. see 2 in the appendix for further metadata on participants. our data set consists of 96 sessions with 3 conversations for almost every session, yielding a total of 283 conversations of roughly 20 minutes each. 7 of these, 8 sessions (22 files) are pilot experiments and the remaining 88 sessions (261 files) are non-pilot experiments. as stated above (section 4) some meetings (35/283 files) were recorded on zoom due to covid restrictions. ## linguistic statistics dialogue corpora vary greatly depending on a wide range of contextual factors. to better understand the nature of a dialogue corpus, it is crucial to look at its linguistic, and in particular its interactional, properties. in this section, we focus on some core metrics that are are interesting proxies to characterize the corpus. namely, we look at (i) distribution of inter-pausal unit (ipu) durations (as a proxy to sentence length in the written realm) ; (ii) automatically extracted backchannels ; (iii) amount of overlapping speech ; (iv) filled pause count ; and (v) speaker dominance. together these metrics provide an insight about the degree of spontaneity and interaction in the corpus. ## conclusion we have presented the summ-re corpus, a new dataset containing 96 hours of spontaneous, multiparty meeting-style conversations in french. the corpus is the only french corpus of its kind, and one of the only large-scale meeting corpora in a language other than english. while the meetings are based on loose role-playing, they remain natural and are designed to elicit basic discursive interactions that we can expect from real meetings. we have offered preliminary analyses of the data to illustrate the level of spontaneity and interactivity in the corpus and have shown how this can vary depending on the type of meeting involved, the subject and-although this was not a part of the initial aim of the corpus-whether the conversation was recorded in-person or on zoom. the dataset is available on hugging face at https://huggingface.co/datasets/linagora/summ-re .