A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries (2023)

Abstract: We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.

## introduction music recommendation systems (rss), fundamental to streaming services nowadays, learn from user listening history or music content which artists or tracks to suggest next @xcite . most of these algorithms provide personalized music content to the users when logging in the streaming apps or websites, or when triggered with pre-defined utterances via voice assistants @xcite . more recent conversational rss aim to help users to express their recommendation needs by supporting interactions via queries in natural language @xcite . however, despite existing in the scientific literature, such conversational rss are not widely deployed because of multiple issues, one being ner. the processing of recommendation queries entails the extraction of named entity mentions @xcite . this sub-task faces multiple challenges, even when queries are framed as pre-defined utterances. the transcriptions of the voice queries results in lower-case noisy text, often with misspellings @xcite . the lack of capitalisation in entities and misspelled words are often present in text-based queries too @xcite . music entities, or those coming from the creative content domains, are highly irregular: they do not follow inherent patterns as it is the case with people's names, and there is little to no separation between the vocabularies of entity and context words, especially for creative works @xcite (e.g. common words like "i" or "love" in track titles). also, new music entities appear all the time. major music streaming services ingest one new track almost every second @xcite . previous works have already shown that ner systems struggle with the aforementioned challenges @xcite @xcite . thus, multiple approaches have been proposed to address them, either focused 1) on collecting more and relevant data for training / fine-tuning standard ner sequential models @xcite ; or 2) on model's design choices that favour generalisation @xcite . most solutions focused on the latter objective have been motivated by the human ner linguistic behaviour, e.g. make the model rely more on context cues than on named entity mentions or learn from a few examples only, as humans do. however, apart from some scarce, partially related works @xcite , there is no systematic investigation of how humans actually perform ner on noisy text with many new and irregular named entities. moreover, in the case of music recommendation, we are not aware of any existing dataset of queries in natural language, annotated with named entities. thus, our goal is to investigate the human ner linguistic behavior when confronted with these challenging conditions. for that, we create musi-creconer, a new corpus of noisy natural language queries for music recommendation in english that simulates human-music assistant interactions. we then conduct a human subject research study to establish a human baseline and learn from it. finally, we perform a detailed comparison of humans and the most popular ner systems nowadays, finetuned transformers, that covers multiple evaluation schemes (strict named entity segmentation and typing, exact segmentation only, or partial segmentation with strict named entity typing) and scenarios including entities previously seen or unseen by the model or humans. the results showed that the task was challenging for humans. given an aggregated metric such as f1 score, human and algorithmic performances were on par. however, the detailed evaluation revealed that humans struggled more with recall while the best model with precision. the high recall obtained by the model was partially a result of entity exposure during pre-training or fine-tuning. also, music entities had different error patterns and, in some queries, had ambiguous context that made their segmentation and typing quite hard. to sum up, our research contribution foot_0 are: ## related work analysing human and algorithmic performance was done for multiple nlp tasks in the past. nangia and bowman (2019) ran an annotation campaign on the glue benchmark with the goal to estimate the effort needed by existing models to catch up with the humans under limited-data regimes. @xcite conducted a large-scale human study on topic shift identification in order to discover patterns of disagreements and consolidate the evaluation metrics. @xcite analysed the human behaviour for understanding ambiguous text-based or spoken sentences to guide the development of a machine learning system. multiple machine translation works challenged the human parity claim @xcite and proposed a secondary evaluation method to reveal detailed differences between humans or algorithms @xcite . compared to these, we benchmark humans and models on a different task-named entity recognition, but we share similar goals-to estimate the human-algorithmic performance gap and to identify patterns that could support the design of better evaluation methods or automatic solutions. human annotation is frequent in ner especially when targeting a new domain such as archaeology (brandsen et al., 2020), or a new language such as indonesian @xcite . however, we are not aware of any annotated corpus of conversational queries for recommendation in the music domain. some other related works propose corpora of noisy social media text containing new entities including irregular ones @xcite , a noisy dataset of movie-related queries @xcite , a dataset of music artist biographies annotated for entity linking @xcite , or a corpus of tweets associated with a classical music radio channel @xcite . previous works have showed that transformers fine-tuned for ner are strong baselines, especially when training data is scarce @xcite . a more recent line of research employs these pre-trained models as few-short learners @xcite . however, the results are still below those obtained with a fine-tuning approach. in order to improve the bare-bone fine-tuned transformers, other works adopted distant supervision @xcite , and the inclusion of gazetteers @xcite or contextual triggers @xcite . though these solutions are interesting and relevant to our problem and context, in the current research, we want to rely on the results of this study before making any design choices for an advanced ner system in the music domain. when conducting human subject studies, the quality of annotations (inter-rater agreement or reliability) is often assessed with kappa statistic or its variations @xcite . yet, for ner, or more generally for labelling phrases, this statistic is less applicable as the number of negative cases on which it relies is ill-defined @xcite . to address this issue, multiple imperfect solutions have been proposed such as to compute the kappa statistic at the token level @xcite -however, this does not reflect the task well as each token is not tagged individually; or to estimate the negative cases by enumerating all n-grams or noun phrases from a text-however, this lacks accuracy @xcite . @xcite show that when the number of negative cases gets very large, the kappa statistic approaches the f1 score. thus, f1 is considered a better metric, which we also adopt to measure the performance of humans and compare the ner human and algorithmic baselines. ## results and discussion we report scores using 4-fold cross-validation on the datasets presented in when comparing bert and the other models in ## conclusion in this work, we investigated the human linguistic behavior when performing ner in the music domain. we created musicreconer, a new corpus of complex noisy queries for recommendations annotated with artist and woa entities. we then designed and conducted a human subject research study to establish a human baseline and learn from its comparison with the most popular systems nowadays, fine-tuned transformers. we performed a thorough evaluation covering multiple metrics, schemes and scenarios, including a careful analysis of the impact of entity exposure on results. the results obtained by the algorithmic baselines were comparable to the human ones. yet, the detailed evaluation showed that humans yielded a better precision while the model had a better recall, linked also to entity exposure during pre-training and fine-tuning. thus, when evaluating fine-tuned pre-trained models, checking their performance on new entities shows their real generalisation ability. regarding the ner evaluation protocol, human performances were much better under a more relaxed schema focused on segmentation or typing only. such a schema could prove a more realistic setup to aim to when training models too. also, we noticed that the relevant schema depended on the entity type as artist was better segmented, while woa better typed. contrary to previous claims, we show that, in our domain, ner in challenging conditions such as noisy text, and irregular or novel entities is rather hard for humans even when provided with complex instructions and multiple examples. thus, although we could learn from the human linguistic behaviour, we should not, by default, assume their results to be a target for any nlp problem. for some tasks, it is common when establishing a human baseline to consider it as an upper bound for the model. this is not necessarily a desirable outcome in our case as it would imply mislabelling 1/3 woa entities. more generally, as we also showed by studying the impact of entity exposure, algorithms can store a lot more knowledge than humans and one may want to leverage this as much as possible. as for proposing a better system to perform music ner, one next step would be to continue the model's pre-training on more related data, in our case music, to get even more exposure, or to integrate gazetteers. still, given the rate of new entities in our domain, forcing the model to rely more on context, when context is not confusing, is another desirable future direction. in case of context ambiguity, asking questions to clarify the request and supporting user interaction in natural language could be ultimately the answer towards a more suitable, but still very challenging solution. we plan to explore these ideas as future work.