D eep C on: An End-to-End Multilingual Toolkit for Automatic Minuting of Multi-Party Dialogues (2022)

Abstract: In this paper, we present our minuting tool DeepCon, an end-to-end toolkit for minuting the multiparty dialogues of meetings. It provides technological support for (multilingual) communication and collaboration, with a specific focus on Natural Language Processing (NLP) technologies: Automatic Speech Recognition (ASR), Machine Translation (MT), Automatic Minuting (AM), Topic Modelling (TM) and Named Entity Recognition (NER). To the best of our knowledge, there is no such tool available. Further, this tool follows a microservice architecture, and we release the tool as open-source, deployed on Amazon Web Services (AWS). We release our tool open-source here http://www.deepcon.in .

## introduction and related work due to the covid-19 pandemic, a substantial part of the working population has seen a significant increase in virtual meetings, especially people working in information technology (it) industry and academia. by all means, meetings are the most vital component to ensure collaborative work and efficient to-and-fro communications. natural language processing (nlp) technologies provide users with a holistic experience in these online interactions. (i) remote conferences or meetings discussions held over an online platform are extremely important in today's globalized world and need interpretation. (ii) coherent translations of larger documents and dialogues and efficient systems with many sources or target languages. (iii) summarizing meetings in the form of structured minutes from speech can potentially save up to 80% foot_0 of time. with all these in mind, we designed an easyto-use and clean interface that provides (i) automatic minuting with dynamic length controlled outputs. (ii) isometric translation for five different languages: french, german, russian, italian, and hindi (iii) topic extraction to the best of our knowledge, there is no such tool available. however, some applications, such as deeptalk foot_1 , provide the fastest way to transform text from chats, emails, surveys, reviews, and social networks into a real business. it is a tool for end users that provides an interactive user interface for topic detection, sentiment analysis, autotagging, analytical interpretation, and summarization. however, this tool lacks in providing users with audio and video support. wordcab foot_2 intelligence adds customizable call summaries to applications so that users can revisit conversations in a fraction of the time. it is developer tool that allows developers to integrate their api and easily generate different types of summaries. this tool focuses on generating complex and customized summaries from the given transcript. happyscribe foot_3 provides automatic and human transcription services convert audio to text with 85-99% accuracy in 120+ languages and 45+ formats. it provides strong api integration that enables users to handle multi-lingual input. this tool is primarily built for generating transcripts from audio files or automatic subtitles. happyscribe does not provide any summarization and topic modeling capabilities that limit this platform's scope. hendrix ai foot_4 is your intelligent, award-winning ai assistant for meetings that automatically transcribes meeting notes, captures action items and other data points, and uses machine learning to identify productivity insights. hendrix ai is a highly analytical tool for zoom meetings. it gives users various outputs such as concise summaries, actionable outcomes, most commonly discussed topics, and meeting effectiveness. hendrix ai is limited to the zoom platform's (person2) um (person2) hmm. mhm. mhm. so, the general outline of the, the project will be the first to go through a functional design phase. you all get task in this in this space and ah then we will meet again and discuss this functional design and the same holds for the two faces. after this, the conceptual design and detailed design in with the final project should get 76. alright. but first we will do from training. okay. in all, in front of you, (person4) you are designed we must keep in mind that the selling price of the product will be about c25 (b) meeting minutes by deepcon: ## deepcon in previous sections, we discuss the various tools that provide similar features as deepcon. most of the tools are made as an api service for developers. a tool like deeptalk, accessible directly to users, does not provide audio and video support. deep-con, however, provides an easy-to-use interface for end users to utilize advanced transformer models without coding. deepcon also provides users with multiple features and support of audio and video files on one platform, which are not provided by various other tools. as seen in figure 2 , deepcon have various different components. in the following subsections we elaborate on each of the major components in our proposed system. ## pilot study to assess deepcon, we experiment it with is1000a recording of the ami meeting corpus @xcite that contains 4 speakers. the first step to use deepcon is for users to login using their credentials. if users are not registered they can use the sign-up link 9 on the landing page. next as shown in the figure 3 users can upload the .mp3 or .mp4 file and select the desired attributes. users can also add name, email, and number of speakers. we have provided some advanced options that can control the length and translations of the audio file. users can select the length of meeting-minutes from the three options: short, medium, and long. users can also select amongst five languages for translation. once users choose the appropriate options, they can upload the audio file via the upload button at the bottom and click on submit. after clicking the submit button, our front-end micro-service uploads the recording on the aws s3 bucket and sends a post request to our back-end micro-service that contains the user's details along with the unique process code. once this process is done, users get a confirmation email about the submission of the job. once the back-end processing is completed, users again receive an email notification from our back-end microservice. this notification informs users that their processing is completed. users can also download the outputs from our results page as seen in 4. the link of this page is also mailed to the user. we gave the users a feedback form and question based on transcript quality, adequacy, grammaticality correctness, and fluency for english minutes. we received an average score of 4.57, 4.71, 4.57, and 4.85 out of 5, respectively. for mt quality estimation, we provided the users with the questions for quality in french, german, russian, hindi, and italian translation and received an average score of 5, 4.83, 4.8, 5, 4.6 out of 5, respectively. the average quality of generated topics is 4.85 out of 5, and the overall user interface received a score of 4.85 with an additional comment of "improvements can be made in its ability to differentiate between the voice of respective speakers. also, there was a slight deviation from the actual words in the transcript, that can be improved as well". 9 https://forms.gle/wsbe6asdggqnxjsha ## design choices as mentioned in previous sections, we make use of microservice architecture. we utilize this methodology for three main reasons: as we utilize large pre-train models, it is not an efficient choice to do real-time processing for users. because every meeting recording can vary in length and users can select many optimization options, real-time processing can take considerable time and slows down the website for other users. with recent advancements in devops technology like kubernetes, microservice architecture has proven to be the most efficient, fault-tolerant, and robust deployment architecture. we use aws eks foot_8 as a container orchestration tool to deploy a highly scalable application. microservice architecture also enables an easy development process of the application. as we have independent services, changing one will not affect others, and it can lead to a faster and more efficient software development process. we also plan to make this app open source, and we believe microservice architecture will enable developers to work in the module of their interest. ## conclusion and future work in this paper, we present a tool for better management of meeting recordings by providing users the ability to generate meeting minutes, topics, and entities in six different languages. our development architecture is designed in a way that it can be scaled and optimized if traffic increases on the application. we can also extend this application to accommodate more languages like spanish, romanian, telugu, etc. the microservice architecture can further be abstracted by introducing a microservice for each functionality. this abstraction can result in more robustness and efficiency during high workloads. this application can also be extended as an api service for developers to integrate in their systems. one major application can be of building native apps for online meeting platforms like zoom, microsoft teams, and google meet the first step, as seen, is attaching a 'normal' prompt to the input source sentence. this will help the translation model generate output with a length ratio close to 1. the next step is to attach a reverse prompt and send input to the paraphrasing module. based on these reverse prompts, the paraphrasing module tries to shorten the long sentences and lengthen the short ones.