Use of NLP in the Context of Belief states of Ethnic Minorities in L atin A merica (2023)

Abstract: The major goal of our study is to test methods in NLP in the domain of health care education related to Covid-19 of vulnerable groups such as indigenous people from Latin America. In order to achieve this goal, we asked participants in a survey questionnaire to provide answers about health related topics. We used these answers to measure the health education status ofour participants. In this paper, we summarize the results from our NLP-application on the participants’ answers. In the first experiment, we use embeddings-based tools to measure the semantic similarity between participants’ answers and “expert” or “reference” answers. In the second experiment, we use synonym-based methods to classify answers under topics. We compare the results from both experiments with human annotations. Our results show that the tested NLP-methods reach a significantly lower accuracy score than human annotations in both experiments. We explain this difference by the assumption that human annotators are much better in pragmatic inferencing necessary to classify the semantic similarity and topic classification of answers.

## introduction indigenous people belong to the particularly vulnerable groups in the covid-19 era and are disproportionally affected by epidemics and other crises, as acknowledged by the united nations (united nations and affairs, 2020). beyond the general problems related to the socio-economic marginalization and the concomitant inaccessibility of health-care services (in particular in rural regions and remote communities), a major threat for indigenous people arises through miscommunication, either due to the sparsity of information material in indigenous languages or due to cultural differences hindering the interpretation/application of the recommended health measures @xcite . dissemination of reliable covid-19-re-lated information, adapted to cultural and linguistic background of indigenous peoples, is a major priority in epidemic crisis; @xcite ) @xcite (un, 13 april 2020). several initiatives of the european union (eu) and world health organization (who) address the problems in communication of health related information @xcite . these initiatives target communication of key health-related terms and concepts underlying them such as understanding of medical instructions. in the recent covid pandemic, it was documented that misconceptions about preventive measures against the spread of covid had a strong impact on the severity of the pandemic (un, 13 april 2020). in order to reduce health-illiteracy and avoid unnecessary spread of infectious diseases, it is necessary to observe people's understandings of infectious diseases and their treatments. for instance, some individuals have the perception that antibiotics are a "cure-all" drug and might take antibiotics to cure diseases caused by viruses, which is an improper use of antibiotics and can lead to severe damaging effects @xcite . given the urgency of measuring the accuracy of health-related concepts and uses, it is necessary to develop nlp tools that can ease and speed up the process related to health education measurement. the key outcome of our research project is testing nlp methodology targeting measurement of health education related to the covid-19 pandemics. ## state-of-the-art accuracy measurement of medical terms uses like antibiotics is currently missing due to two main reasons: a) missing data sources and methodologies that enable researchers to identify, characterize and measure actual uses of health related topics and concepts and b) missing statistical (in)accuracy measures of actual information status related to infectious diseases. it is thus not surprising that the initiative the social media mining 4 health (#smm4h) is addressing these problems in its agenda(klein, 2021) @xcite . this initiative uses social media data as a data source for solving health-related tasks and problems such as finding disease mentions and symptoms(klein, 2021) @xcite . however, this rich data source does not have demographic information necessary for the statistics on social variation in the health literacy study. in addition, social media does not represent all social groups including indigenous population that often has low internet access or uses other tools for communication. as a consequence, data from indigenous communities related to covid pandemics is very rare @xcite . in order to address these problems, we used a traditional methodology in social sciences in order to access the information about the health education status, namely the survey methodology. we asked health-related questions such as questions about virus propagation and treatment to our participants. in order to be able to measure the accuracy of health-related concepts and uses of our participants', it is necessary to compare their information status with "expert" knowledge or uses. in recent years, big progress has been made in semantic comparison of linguistic units such as words and sentences due to recent developments in neural language models such as bert @xcite . bert is a language model trained on a large amount of natural language data to predict words that have been masked out as shown in bert has been used to find out which word vectors are responsible for lexical meaning variation such as coach used as 'trainer' and 'vehicle'. a word vector is essentially a mathematical representation of the meaning of a word based on learning or memorizing the frequency at which a word appears in a particular linguistic context. the differences or similarities of word vectors have been used to predict semantic (dis)similarity of words @xcite and sentences (reimers and gurevych, 2020). however, previous approaches mainly focus on meaning differences in big data sources such as social media and very few of them address meaning differences in survey questionnaires of ethnic minorities. it is thus not known yet how well these models work in the low resource scenario given the specific topic domain and the specific format of answers. this paper presents results from testing vector-based approaches in the measurement of answer similarity in the low resource domain. ## methodology we carried out a survey study with our cooperation partners from latin america (marleen haboud, claudia crespo, fernando ortega pérez), in which indigenous groups speaking quechua or kichwa from peru and ecuador (around 150 people from each country) answered questions about covid-19 (10 yes-no questions and 10 open-ended questions). our task was to measure the accuracy of key concepts related to health. we tested how well the information status of indigenous groups matches the information and suggestions from reliable sources such as the world health organization (who), henceforth our reference corpus. for instance, according to the who, the virus covid-19 is distributed through contact, hence the suggestion to keep social distancing. we asked our participants about how the virus covid-19 is distributed in order to see how well their answer matches the information from who. the answers were collected in rural areas via free interviews by a local person knowing indigenous communities. the method of free interviews was particularly important in order to include individuals who are less accustomed to performing highly controlled tasks such as older and/or illiterate participants. due to lack of time and resources we did not transcribe the interviews. instead, the local interviewer summarized the answers to the questions in a digital form in spanish. consequently, the answers in this survey study do not directly reflect the information state of indigenous minorities. ## experiments and results we ran two experiments. the data and the code for both experiments can be found on github 1 . in our first experiment, we tested the sbert model for measuring the semantic similarity between the participants' answers and the "expected" answers from the reference corpus via cosine similarity (see sentence transformers based on reimers and gurevych, 2020). the following examples demonstrate some results of cosine similarity from the chosen method: word before mask after mask coach 'vehicle' i have driven my coach into the garage. i have driven @xmath0. coach 'trainer' i have a female coach. i have a @xmath1. in order to evaluate the validity of the similarity measure by sbert, we asked human annotators to annotate participants' answers from 0-5 as not similar (0) or similar (5). the annotators were four students of linguistics and one expert in medical anthropology. we divided the human ratings into three categories: similar (4-5), dissimilar (0-2), ambiguous (3) and selected the answers with high inter-speaker agreement. we translated the human ratings into correspondent cosine similarity scores: similar (>0.6), dissimilar (<0.4), ambiguous (> 0.4 @xmath2.6). our results show that the semantic similarity measured by cosine similarity using sbert is significantly lower (mean 0.2) than the semantic similarity acquired by human annotation (mean 0.7). our second experiment had the goal to find a computational method to classify a topic of an answer to an open-ended question. here is an example. survey question: why do you not want to be vaccinated? topics: a) afraid of side effects, b) my own decision, c).... an automatic classification of answers under the correspondent topics can ease the process of survey data analysis and provide a uniform way of measuring answers to open-ended questions. we asked human annotators to create topics for the interview questions and then to annotate answers according to these topics, e.g. "i can get thrombosis" was classified by human annotators as a) afraid of side effects. we tested automatic methods to classify answers under suggested topics. the underlying idea was to look for key words in the answers that semantically correspond to suggested topics. for this aim, we performed a synonym-based similarity task without stemming (task 1) and with stemming (task 2). in the first task, if the topic was a synonym of one of the tokens in the given answer, the classification was true. in the second task, if the topic stem was a synonym of the token stem in the given answer, the classification was true. the latter case ignores morphological variation of words and focuses only on the lexical stem. we preprocessed the given answers by tokenization, removing stop words and case lowering. the synonyms were taken from the nltk wordnet. we used a stemmer from nltk, to stem the synonym words: our results in ## discussion the computational approaches we tested have shown much lower accuracy compared to human annotations. the biggest problem we have identified is the lack of pragmatic inferencing humans are good at, but automatic models we tested are not. for instance, people answered to the question about how the virus distributes by saying "through crowd". due to a pragmatic inference human annotators can evaluate this answer as similar to the answer given by the reference corpus. "a crowd" implies pragmatically that social distancing cannot be obtained adequately and this can promote virus infection. however, none of our automatic models was able to predict a high similarity between the reference answer "through contact" and the participant's answer "through crowd". another example illustrating problems with pragmatic inferences is the annotation of vaccination side effects. while human annotators had no difficulties to classify "thrombosis" as a possible vaccination side-effect, our automatic methods were not able to do it. to sum up, one of the biggest challenges in our tasks was the lack of natural language understanding and inferencing (nli and nlu) by the computational models we tested. using nli and nlu in the context of low resource is reserved for future research. in the near future, we will test models trained on health-related topics, fragmented answers that represent the majority of our answers and models trained on nli-and nludatasets @xcite .