Assisting Breastfeeding and Maternity Experts in Responding to User Queries with an AI -in-the-loop Approach (2024)

Abstract: Breastfeeding and Maternity experts are a scarce resource and engaging in a conversation with mothers on such a sensitive topic is a time-consuming effort. We present our journey and rationale in assisting experts to answer queries about Breastfeeding and Maternity topics from users, mainly mothers. We started by developing a RAG approach to response generation where the generated response is made available to the expert who has the option to draft an answer using the generated text or to answer from scratch. This was the start of an ongoing effort to develop a pipeline of AI/NLP-based functionalities to help experts understand user queries and craft their responses.

## context breastfeeding and maternity experts are a scarce resource and engaging in a conversation with mothers on such a sensitive topic is a time-consuming effort. we present our effort in assisting experts to answer queries about breastfeeding and maternity topics from users, mainly mothers, using nlp and generative ai technology. from the user perspective, the interaction happens in a chat which is a premium service in an m-health application . in addition, informal discussions and observation of some experts at work revealed that they spend a lot of time going through the lengthy user messages and previous conversations in order to gather key information. we also learnt that experts informally keep a list of frequently used sentences, answers to common questions and list of blog post url links at hand to help them craft their response. in order to focus our efforts, we analyzed the types of messages that the experts receive from the users and identified five main types of user messages the experts might need help with, shown in we initially implemented a rag response generation approach that applies to any incoming message above a minimum length. the generated response is made available to the expert who has the choice query type example incoming message simple message about compatibility of some medication with breastfeeding complex unhappy baby and mother, with issues of pain, poor weight gain and latching difficulties multimodal weight-related question with weight tracker screenshot or tabular data "here and now" query about current affairs such as it problem, visits, workshops or follow-up to previous physical interaction incomplete user follow-up or response to current conversation to draft an answer using the generated text or to answer from scratch. we quickly realized the limitations of our approach such as the poor outcome for complex queries or 'here and now' queries that require extremely fresh documents/information or the burden of having an answer generated for every incoming message regardless whether it contains a query. furthermore, the approach did not address the need of the experts to parse the current conversationskimming through a long single paragraph message that seems to have been written in one spurt -in order to sift the wheat from the chaff. to address these issues, we then developed a pipeline of nlp/ai tasks to help the experts process the user queries and craft their responses. in both cases, our philosophy is to provide support to our human experts with an 'ai-in-the-loop' (or ai co-piloting) approach. the conversation is still conducted by the experts who make or validate all the decisions so that trust and security -which are a fundamental requirement for user attention in this domain -are not compromised. in the rest of the paper, for lack of space, we present the design and implementation of the multitask pipeline (section 2), including the answer generation task initially used. we then briefly explain our initial deployment of a simple response generation in the expert environment and the deployment of the more advanced pipeline in a development environment (section 3). this is followed by the evaluation of the answer generation task and a preliminary informal evaluation of the other tasks of the pipeline (section 4) before discussing related work (section 5) and drawing some conclusions regarding efforts so far and future work (section 6). ## design and implementation for an agile development, we currently implemented each task as a rest api function using prompt-based generative ai in a zero-shot or incontext learning setting with some of the most performant commercial models on the market (i.e., claude x foot_1 and gpt x foot_2 ). the tasks are implemented using the haystack python api to define pipelines and components, to call models and query data stores. @xcite for document chunk retrieval used in answer generation and blog recommendation, we use the aws opensearch data store. @xcite the tasks are pipelined together using an orchestrator, which performs the necessary checks and calls the different rest functions in order, concatenating the results of the different steps -including meta information such as cost, model and number of input and output tokens -in a final json structure . @xcite figure 1 in the appendix illustrates the tasks flowchart and presents their basic implementation. more specifically, in order to address the first four user interactions in text summarization. this task applies to 'complex' messages and involve detecting parts of the query that are about historical context (e.g., mother's previous pregnancies and health issues), current situation (e.g., what is the situation now), questions and emotions. @xcite summarization is both abstractive and extractive and an example output is shown in intent detection. this task is about characterizing the query, that is what it is about, who is talking and who is it for. thus we determine for each message 1) whether it contains a query; 2) whether it is a follow up to a visit, a workshop or some other event; 3) whether it is about a topic that is not breastfeeding (such as maternity in general or child rearing) and 4) who is the author of the message (health professional, friend, family, mother, etc.). we also identify main and secondary intent(s) in the message. the idea is that secondary intents are sub-issues that are related or collateral to the main issue, whilst main issues are independent of one another and must be addressed separately. there are currently around 60 possible intents with their optional definition/explanation, such as: compatibility: compatibility of products that the mother takes whilst breastfeeding medicines, vitamins, infusions, food supplements, aesthetic treatments, foods, drinks). shape of the breasts: such as hypoplasia; tubular breasts; smaller or larger breasts; asymmetrical breasts, soft breasts, one breast producing more than the other; flat, inverted or pierced nipples; breast augmentation, etc". information extraction. this is divided in two tasks: a general ie task to extract general information such as baby's age, mother's pregnancy, baby's prematurity; and an intent-specific ie task. currently we have two intent-specific ie tasks: one for compatibility and one for 'pain in the breast'. compatibility ie task is about identifying the object of compatibility (medication, food, drink, product, body or health treatment and activity). pain ie task is about identifying diagnosis (e.g., mastitis), treatments (e.g., antibiotics, apply-ing cold, cabbage leaves), symptoms (e.g., fever, pink area, red line) and body zone (e.g., left breast, areola). machine translation. this actually consists of two separate tasks: detection of source language and translation to spanish, and translation of the generated text back to the source language. although the current volume of queries that are not in spanish is small, we envisage that this architecture will allow us to offer the chat functionality to other language communities. blog url recommendation. this task uses a data store of 40k blog post chunks and retrieves the list of 100 chunks that are most similar to the query. the unique urls of those chunks are identified and an average score is computed for each of them and up to 3 of the ones with the highest score are returned as recommendations. image-to-text extraction. we used the multimodal capabilities of claude 3 and gpt4-vision to extract baby growth data from a table image. this task is triggered if one of the query's main intent is about baby weight and the user adjoined an image. it is currently implemented in a two-step approach: 1) extraction and optional translation of table header, 2) extraction of baby growth data according to the table header. retrieval augmented generation. we use a rag approach by retrieving the top 3 q&a pairs whose questions are most similar to the incoming query. we currently have in store over 20k standalone q&a pairs that were obtained by applying a conversation classification model trained on an automatically constructed dataset to detect conversation boundaries and selecting 'simple' conversations as our standalone q&a dataset, that is, conversations that consist of only one sequence of consecutive user messages followed by only one sequence of consecutive expert messages. an example input message with rag, gener-ated and expert responses is shown in ## deployment we currently have a version that is deployed in the experts production environment which only consists in applying retrieval augmented generation. the expert has the option to modify the generated message (i.e., it is presented in an editable text box). she also has the option to ignore the generated text altogether and draft her answer from scratch. we had to address a number of impromptu issues including overload of model service or occasional noise in the generated json output string that required some preprocessing. we also implemented the pipeline version described in section 2 and illustrated in figure 1 in the appendix in a development environment. the orchestrator in this pipeline applies in order optional translation to spanish, new conversation detection, summarization, intent detection either on extractive summary (if message is complex) or original query, general and intent-based information and imageto-text extraction, answer generation and optional translation of answer to source language, in addition to retrieval-based blog url recommendation. the tasks are parameterized so as to enable the personalization of some results, such as the preferred languages of the expert (she may be able to attend queries in different languages), or the user source language (so as to translate the rag generated response back into the user source language). ## initial evaluation and monitoring in this section we first report on the evaluation and monitoring of our initial implementation, i.e., the answer generation task, before discussing preliminary evaluation of the other tasks. ## related work although task-oriented chatbots and virtual agents have been at the forefront of ai and nlp applications and research for many years, for many domains this implementation remain challenging and costly and its adoption met with dissatisfaction or mistrust (kraus et al., 2023). the relatively recent advent of large language models (llms) and socalled generative ai has brought new promises but also new challenges such as hallucinations and poor relevance. in the healthcare domain, several approaches have been used to mitigate those issues such as llm fine tuning to adapt to diagnosis style and prompt engineering to improve consistency (shi et al., 2024), or applying knowledge-and nlpintensive approaches such as xia et al (2022) who combine symptom recognition and disambiguation and knowledge graph reasoning (which they call 'triage') before performing an entity-aware promptbased generation. other approaches aim to assist healthcare professionals instead of replacing them. for example, madeira et al (2020) provide chat operators of a mental healthcare service with query classification and a list of suggestions to be discussed. xie et al (2024) investigate how llms can help doctors in daily tasks that are "repetitive [by] nature (e.g., case summarization, preoperative education), relatively low medical risk (e.g., triage), [or require]... extensive information requirements (e.g., medication inquiry)." ## conclusions and future work we have presented our ongoing journey into developing ai-driven functionality to assist experts in addressing user queries about maternity and breastfeeding. starting with a rag approach, we gave our experts the option to draft their response from a generated text. the monitoring and feedback received allowed us to quickly realize that not every user message could be treated equally, so we followed up with a more complex pipeline for the conditional generation of answers, where we could guarantee a higher relevance, coverage and faithfulness (es et al., 2024). we also realized the need to help the experts not only draft their response but also understand the current and past conversation and so we expanded the pipeline with understanding tasks such as information extraction . our approach is to incrementally put in place and test a set of functionalities that can work for our experts. in doing so we must take into account the following criteria: cost. proprietary llms are costly. whilst those out-of-the-box models allow us to quickly get a grasp of the workability of our pipeline, we consider implementing some of our own models in the future for certain tasks, such as intent detection (because it has such as large input prompt). trust. one of the most-valued features of the app and associated chat is the trust it generates and builds amongst our users and this is something that cannot be compromised. this is why we favor extractive understanding through summarization and information extraction, so that the expert can always see the information in context and hence trust its veracity. trust is something we always need to keep in mind when developing our system.