Information Extraction of Music Entities in Conversational Music Queries (2024)

Abstract: The detection of music entities such as songs or performing artists in natural language queries is an important task when designing conversational music recommendation agents. Previous research has observed the applicability of named entity recognition approaches for this task based on pre-trained encoders like BERT. In recent years, large language models (LLMs) have surpassed these encoders in a variety of downstream tasks. In this paper, we validate the use of LLMs for information extraction of music entities in conversational queries by few-shot prompting. We test different numbers of examples and compare two sampling methods to obtain few-shot examples. Our results indicate that LLM performance can achieve state-of-the-art performance in the task.

## introduction detecting music entities such as songs or musical artists in natural language queries is a key component of conversational music agents @xcite . in such queries, users request music entities they want to listen to in a conversational way as an alternative to traditional text search. the task of detecting music entities is typically modeled as named entity recognition (ner) which was earlier addressed by probabilistic approaches @xcite . more recently, pre-trained encoders demonstrated strong performance in the ner task in the music domain @xcite . with the advent of large language models (llms) such as gpt-3.5 for text generation tasks, these are increasingly used for ner and the related task of information extraction (ie) @xcite @xcite . several studies found that encoder-only models still outperform llms @xcite @xcite . however, llms are usually trained on much larger datasets than encoder-only models which theoretically makes these more likely to capture music knowledge to some extent. in this paper, we investigate the success of llms for ie of music entities in conversational music queries (e.g., give me some artists like metallica). we prompt llms to label utterances in the queries with respective labels (title and artist) and compare these to two strong baseline encoder-only models. we investigate the difference of two fewshot sampling methods and different numbers of few-shot examples. lastly, we outline some contextual cues captured by the best performing llm which we request in our prompt to reflect internal reasoning. we release our code publicly. 1 in the next section, we outline related work in ie and ner using llms. in section 3 we describe our proposed method. in section 4 we describe our used dataset and baselines before presenting the results in section 5. lastly, we close this paper with section 6. ## related work in this section we outline related work in ie and ner with llms. ner is a subtask of ie in which a sequence of labels per token or character is obtained. the more general task of ie refers to the extraction of relevant information in some structured form, but not necessarily a sequence of labels and possibly with additional steps (e.g., normalization). research towards the use of llms for ie comprises the direct use of llms for the task (e.g., by instruction tuning) or auxiliary use in combination with encoder-only models (e.g., bert). a line of research has validated the usefulness of llms for ner. beside @xcite which fine-tune llama-2, the most prevalent strategy appears to be prompting the llm @xcite @xcite . jung et al. (2024) relies on a single prompt without fewshot examples. wang et al. (2023a) prompt gpt-3 and provide few-shot examples retrieved by a nearest neighbor search. their approach achieves a performance close to the state-of-the-art based on bert @xcite . @xcite demonstrate that gpt-3.5 and gpt-4 beat other llms such as t5xxl in the task. the authors include entity label descriptions and request reasoning for predicted entities. @xcite state that llms underperform in comparison to encoder-only models due to reasons like hallucination and context limits in few-shot settings. they provide various means to overcome this issue, such as demonstration retrieval and self-verification. the mixed performance of llms for ie and ner motivates another paradigms which favors the auxiliary use of llms together with encoderonly models trained in a supervised fashion. @xcite argue that the lack of speciality of llms is a major factor. hence, they propose an approach to combine those with encoder-only models, utilizing the llm only for relabeling of initially uncertain predictions. similarly, @xcite state that llms are better to use for hard samples than for general use in ie. they suggest to rather use llm to re-rank of outputs obtained from a pre-trained encoder-only model. other works include the auxiliary llm purely for data augmentation @xcite or model distillation @xcite . in the following, we propose our approach for ie from music queries using llms. ## music entity extraction with llms the goal of our approach is the extraction of music entities from natural language queries. when users request music recommendations, the query can refer to various aspects such as genres, moods, titles (e.g., album or song titles) or performing artists (performers). we only focus on musical entities such as songs and albums (represented by their title) and performing artists (represented by their name). for each query, we aim to extract all the entities of this type and permit the possibility of no relevant entities being contained. thus, the query recommend me some rock songs should yield no result, while the query something similar to metallica st anger should yield the utterance metallica with the label performer and the utterance st anger instruction from the following text, which contains a user request for music suggestions, extract all the relevant entities that you find. ## results in this section, we present the results of our previously presented experiments. all of the results are obtained with 4-fold cross validation using the split from @xcite . in figure 2 we show f1 scores as a function of k for both sampling methods of few-shot examples. the effect of tf-idf sampling as opposed to random sampling seems to have no positive effect on the performance of mistral-7b and just a minor effect on gpt-3.5-turbo. mixtral-8x7b is the best performing llm and achieves higher f1 scores than the baselines for @xmath0. using tf-idf sampling, it exceeds the baseline for smaller values of @xmath1.80. however, the performance for @xmath2. at random sampling, the performance appears to stagnate for @xmath3, but experiments with even higher values are necessary to fully exploit the potential of even more examples. to gather more detailed insights in llm perfor-mance against the baselines, we report the precision and recall of both entity labels in ## conclusion and limitations in this paper, we performed ie of music entities in conversational music queries with three llms as an alternative to the previously suggested ner with encoder-only models like bert. we showed that tf-idf sampling to obtain similar few-shot examples to the query text can enhance the llm performance, especially in case of the best performing model mixtral-8x7b. the observed increase in f1 is mostly achieved by improved precision which leads to an overall improvement against the baselines. in future work, the inclusion of annotations for contextual cues could be helpful to encourage the llms to return those more frequently and possibly encourage better reasoning. further, our study motivates experiments with even more capable llms such as llama-3-70b.