PIAST : A Multimodal Piano Dataset with Audio, Symbolic and Text (2024)

Abstract: While piano music has become a significant area of study in Music Information Retrieval (MIR), there is a notable lack of datasets for piano solo music with text labels. To address this gap, we present PIAST (PIano dataset with Audio, Symbolic, and Text), a piano music dataset. Utilizing a piano-specific taxonomy of semantic tags, we collected 9,673 tracks from YouTube and added human annotations for 2,023 tracks by music experts, resulting in two subsets: PIAST-YT and PIAST-AT. Both include audio, text, tag annotations, and transcribed MIDI utilizing state-of-the-art piano transcription and beat tracking models. Among many possible tasks with the multimodal dataset, we conduct music tagging and retrieval using both audio and MIDI data and report baseline performances to demonstrate its potential as a valuable resource for MIR research.

## introduction piano music presents unique opportunities for music research due to its ability to express diverse styles using a single instrument and its superior transcription performance. given these characteristics, it has become a significant area of study in music information retrieval (mir), encompassing tasks such as classification @xcite , and music generation with various conditions @xcite . while these tasks require datasets that combine piano audio with various modalities such as midi, sheet music, or text, there is a notable scarcity of such comprehensive multimodal piano datasets. however, existing multimodal music datasets, particularly music-text datasets, rarely focus exclusively on piano music, and piano solo pieces comprise only a small portion of general music-text datasets. for instance, in the ecals dataset @xcite , a subset of the million song dataset @xcite , the number of piano solo tracks is very limited. we observed that excluding tracks tagged with instruments other than the piano or genres that could not be solely represented by the piano, only approximately 0.46% of the entire dataset can be identified as piano solo music. several piano datasets, such as maestro @xcite , have been developed in recent years, which provide classical piano performances primarily used for piano transcription. another classical piano dataset, giantmidi @xcite , is also commonly used in transcription tasks. other datasets like pop1k7 @xcite focus on the performance generation of pop piano music, while pijama @xcite is employed for performer identification tasks with their jazz piano data. however, these datasets are confined to a single genre and lack text labels. this absence of genre diversity within a single dataset and the lack of textual information underscores the need for a piano dataset with text information. some piano datasets contain emotion labels, such as emopia @xcite and vgmidi (ferreira and whitehead, 2019). however, these datasets are annotated only with emotion information based on either russell's four quadrants @xcite or the valence-arousal model @xcite . this limited annotation approach lacks the rich textual descriptions needed for text-based mir tasks. to address the limitations, we present multimodal piano music data with rich text annotations and transcribed midi. to build the dataset, we first created a piano-specific taxonomy with 31 tags that include genre, emotion, mood, and style information to encompass the broad and diverse musical range that the piano can express. based on this taxonomy, we collected data from youtube, transcribed it to midi format, and conducted an annotation process. the piast dataset consists of two subsets: piast-yt, 9,673 tracks collected from youtube, providing audio and text information (titles, tags, and descriptions), and piast-at, 2,023 tracks with annotations by music experts. this dual approach ensures both breadth and accuracy in the dataset. additionally, piast includes transcribed performance midi data alongside audio and text, enhancing its capabilities beyond existing methods @xcite . this paper details the dataset collection process and analyzes the data. we present baseline results for piano music annotation and retrieval tasks, utilizing the piast-yt and the piast-at datasets across audio and midi domains. the pi-ast dataset is available in our online repository foot_0 , and the source code for the experiments can be found on github foot_1 . ## piano music classification in this section, we present the application of our proposed dataset for piano music annotation and retrieval tasks in both the audio and midi domains. we employed a two-stage framework: 1) pre-training and 2) transfer learning. for pretraining, we used the piast-yt dataset to train a general-purpose piano-specific model with largescale audio, midi, and diverse text data. we leveraged text supervision through music-text joint embedding pre-training @xcite @xcite . in the transfer learning stage, we utilized the piast-at dataset to train a piano classification model as a downstream task. ## conclusion in this paper, we introduced piast, a piano dataset with audio, symbolic and text. our experiments demonstrated the dataset's effectiveness for piano music annotation and retrieval tasks, showing improvements with pre-training. the piast dataset supports various applications, including improved music retrieval, text-based music generation, music analysis, and emotion/genre classification. to further enhance the dataset, our future work will address tag imbalances by adding more samples and incorporating additional processed data such as lead sheets and chord annotations.