A Linguistic Annotation Framework to Study Interactions in Multilingual Healthcare Conversational Forums (2021)

Abstract: In recent years, remote digital healthcare using online chats has gained momentum, especially in the Global South. Though prior work has studied interaction patterns in online (health) forums, such as TalkLife, Reddit and Facebook, there has been limited work in understanding interactions in small, close-knit community of instant messengers. In this paper, we propose a linguistic annotation framework to facilitate analysis of health-focused WhatsApp groups. The primary aim of the framework is to understand interpersonal relationships among peer supporters in order to help develop NLP solutions for remote patient care and reduce burden of overworked healthcare providers. Our framework consists of fine-grained peer support categorization and message-level sentiment tagging. Additionally, due to the prevalence of code-mixing in such groups, we incorporate word-level language annotations. We use the proposed framework to study two WhatsApp groups in Kenya for youth living with HIV, facilitated by a healthcare provider.

## introduction good communication between patients and healthcare providers as a part of patient-centered care, where the patient is an equal partner in their healthcare decisions, has been shown to improve medical adherence @xcite (i.e., the practice of consistently taking the medicines as prescribed) . however in the global south, i.e. the countries, located primarily in the southern hemisphere, that have historically been identified as third world due to perceptions of their socio-economic status, technological advancements, and global dominance, high patient volumes and a need for cost-effective solutions can make patient-centered care difficult. in recent years, various online chat applications such as whatsapp and wechat are being increasingly used to ensure smooth access to both patient-provider communication and peer support beyond formal healthcare system @xcite @xcite . even though these peer support groups are effective in providing patient-centered care, they can be taxing for already overworked healthcare providers @xcite . there is therefore a necessity for designing technology to support both the providers and patients using such groups. prior work on online health forums focuses on analyzing participation @xcite @xcite , social connection and engagement @xcite @xcite , and modelling user behavior @xcite @xcite @xcite @xcite . further, existing research focuses on the linguistic analysis of conversations in social media platforms like facebook @xcite @xcite , reddit and twitter @xcite @xcite @xcite @xcite @xcite , but has paid little attention to analyzing the conversations in small, close-knit whatsapp communities. in this paper, we propose a hierarchical annotation framework to facilitate the analysis of health-focused whatsapp groups. our proposed hierarchical framework consists of fine-grained peer support categorization and message-level sentiment tagging. additionally, we address the prevalence of code-mixing in such whatsapp groups by incorporating word-level language annotation. we evaluate our framework on chat logs of two whatsapp groups of kenyan youth living with hiv, moderated by a healthcare provider. we also develop a user-friendly annotation interface to facilitate annotation. in the rest of the paper, we describe the annotation process using the framework, the challenges faced and our learnings. our annotation study shows that the annotators achieve high agreement for all categories, thereby indicating the efficacy of our linguistic annotation framework. we believe that this framework can be used to analyze inter-personal relationships among group members and is the first step towards developing nlp solutions to support healthcare workers providing remote patient care. while developing the framework, we emphasized on code-mixing. multilingualism in kenya contributes to a considerable linguistic diversity in a population that speaks kiswahili, kikiyu and english fluently @xcite . this leads to a high prevalence of codemixing in their linguistic interactions. this is further enriched by the use of sheng, a swahili and english-based cant (a mixed language or creole), by the urban youth in kenya. it emerged as a result of urbanization and globalization @xcite . any annotation framework for such conversations, therefore, has to take into account this multilinguality. several annotation schemas have been proposed for multilingual code-mixed messages on social media platforms @xcite @xcite @xcite @xcite @xcite @xcite . however, to the best of our knowledge, none of these look into instant-messaging based interactions among peer supporters using code-mixed african languages like kiswahili, sheng. we developed our annotation framework to support an ethnomethodologically-informed ethnographic analysis of the chat: a qualitative approach used in human-computer interaction (hci) research to understand social interactions @xcite . such analysis has proven useful in the design of computer systems as it reveals the practices and methods of those who will use these systems and which need to be designed for @xcite ; be-ing used to examine text interactions such as instant messages (o @xcite , internet forums @xcite and chat messages @xcite . the annotation framework was designed to be used to support and inform the ethnographic analysis, with the aim of identifying ways in which we might support both the healthcare provider and participants in these chat groups. additionally, the framework would be useful for sociolinguistic studies of multilingual conversations and the schema could be extended to understand other peer support settings such as mental health forums. another area where the framework might prove useful is for research around improving human-machine conversational agents @xcite ) by leveraging data annotated using these categories. ## dataset we studied whatsapp chat logs from two peersupport groups for kenyan youth, living with hiv. there were a total of 1,655 messages in group-1 (28 members, 14 female, 14 male, @xmath0,901 messages in group-2 (27 members, 21 female, 6 male, @xmath1. both the groups were facilitated by a healthcare provider with a background in public health, hiv testing, and counselling. the facilitator sent weekly messages on a range of topics, such as future goals, strategies for medical adherence, etc., responded to queries (within 12 hours), clarified any misinformation posted on the group, and referred medical-questions to a clinic. the chat records were in kiswahili, english and sheng, sometimes a mix of these languages in a single message. all the messages were anonymized and translated into english by a native speaker. each chat message in the dataset had the following information: anonymized speaker id, timestamp, original message and english-translated message. the data contains sensitive content dealing with people's deeply personal lives and tragedies, and therefore, even with anonymization there are ethical reasons for not making the dataset public. however, anyone wanting access to the data for research purposes can get in touch with the authors. ## annotation framework the primary objective of our proposed linguistic annotation framework is to facilitate studies on inter-personal relationships among the peersupporters. in order to satisfy the above objective, the framework is governed by the following guiding principles. 1) we want to model the conversational intent of the peer supporters behind the act of sending messages. 2) we need to assess the psychology of their behavior by understanding their sentiments and emotions. this is even more critical in a healthcare setup where it is important to maintain an overall positive outlook. 3) we also want to capture the morphosyntactic level of linguistic information which can be used to help us tease apart deeper interaction patterns and stylistic features of conversations. the framework (figure 1 ) has a hierarchical schema with five top-level categories that deal with linguistic and pragmatic phenomena observed in the chat. the categories include peer support (principle 1, 3.1), sentiment (principle 2, 3.2), word-level language, topic change, and typo (principle 3, 3.3, 3.4, 3.5). note that this framework can be further augmented to include more linguistic features and can be extended to perform similar studies on other peer support groups. in this section, we define the broader and fine-grained categories. ## annotation interface overview we developed an annotation interface (figure 2 ) to facilitate user-friendly annotation. the annotators were provided with detailed guidelines on how to use the interface for annotating the chat messages based on our proposed annotation framework. django framework (python) was used on the server-side for implementing this interface. this interface can be also used to annotate the peer support conversations in the mental health chat forums such as reddit, talklife. the annotators need to login into the interface with their credentials to start the annotation process as outlined in the guidelines provided to them. on logging in, a target message is displayed along with ten (or less) previous messages to provide context (figure 2a ). the right side of the screen shows question related to top-level categories: peer support, sentiment, and change in topic (figure 2b ). the annotators then have to choose appropriate labels. for the peer support category, the annotators have to further choose subcategory labels as well (figure 2c ). as a message can be associated with multiple subcategories, the annotators are asked to highlight specific portions of the message for each subcategory. e.g., in the message "hi...he got me pregnant..."-"hi" is tagged as chit-@xmath2. for typo, the annotators are asked to highlight spelling mistakes and abbreviations, and correct them wherever possible. for instance, in the message "morning members from today, hence forth untill further notice am the s.t.o (sitting allowance officer) ask any querry you get the right answer" the annotators needs to highlight the typos ("untill", "querry") and normalize those ("until", "query") using the interface. moreover, they should also annotate the abbreviation ("s.t.o"). the annotation requires a language tag for each word in the message. instead of specifying a language label for each word, the annotators are asked to highlight the word spans for each language category, facilitating quicker and less errorprone language identification. for example, in the message "wow, mamboz mrembo" ([translated]: "wow, hello beautiful")-"wow" is highlighted as english and the word span "mamboz mrembo" is highlighted as kiswahili. the annotators are expected to completely annotate each message before moving on to the next message. the submitted annotations are saved in a cloud database. the annotators have an option to revise and modify their current or previously submitted annotations during the process. if the annotators are unsure of how to annotate a particular message, the interface allows them an option to note and skip that message. this annotation interface can be leveraged easily to annotate any complex hierarchical annotation schema comprising of multiple levels. ## annotation experiments and observations the annotation was conducted in three phases. in phase-1, a small sample of 395 english-translated messages (105 messages from group-1, 290 messages from group-2) were annotated, and the corresponding original multilingual messages of the same set were annotated in phase-2. in phase-3, the entire multilingual dataset was annotated. below we discuss the challenges faced and the learnings from all the annotation phases. ## conclusion and future work in this paper, we presented a comprehensive annotation schema, combining several linguistic and pragmatic phenomena, to enable studies of what-sapp based conversations of youth living with hiv. unlike twitter and facebook, this requires understanding the content of previous discourse to annotate the sentiment and conversational intent of supporting the peers in specific examples. during the process of annotation, we have learnt that a good percentage of examples require understanding the context. to the best of our knowledge, we are the first to combine several standard annotation schemes to annotate the multilingual conversations in a small, close-knit community. we believe our schema can be used to flag important messages (based on user concerns) to the healthcare providers. this will in turn help to alleviate the need to scroll through all the messages individually to identify the ones that require a response from the facilitators. linguists can also use this framework to study the nature of multilingualism in such conversations from both sociolinguistic and structural perspectives. in future, we plan to use the framework to further study the nature of engagement and multilingual interactions in such peer-support groups in healthcare domain with the aim of enhancing remote patient care.