To make music composition more approachable, we designed the first AI-powered Google Doodle, the Bach Doodle [1] , where users can create their own melody and have it harmonized by a machine learning model (Coconet [22]) in the style of Bach. For users to input melodies, we designed a simplified sheet-music based interface. To support an interactive experience at scale, we re-implemented Coconet in TensorFlow.js [32] to run in the browser and reduced its runtime from 40s to 2s by adopting dilated depthwise separable convolutions and fusing operations. We also reduced the model download size to approximately 400KB through post-training weight quantization. We calibrated a speed test based on partial model evaluation time to determine if the harmonization request should be performed locally or sent to remote TPU servers. In three days, people spent 350 years worth of time playing with the Bach Doodle, and Coconet received more than 55 million queries. Users could choose to rate their compositions and contribute them to a public dataset, which we are releasing with this paper. We hope that the community finds this dataset useful for applications ranging from ethnomusicological studies, to music education, to improving machine learning models. Machine learning can extend our creative abilities by offering generative models that can rapidly fill in missing parts of our composition, allowing us to see a prototype of how a piece could sound. To celebrate J.S. Bach's 334th birthday, we designed the Bach Doodle to create an interactive experience where users can rapidly explore different possibilities in harmonization by tweaking their melody and requesting new harmonizations. The harmonizations are powered by Coconet [22], a versatile generative model of counterpoint that can fill in arbitrarily incomplete scores. Creating this first AI-powered doodle involved overcoming challenges in user interaction and interface design, and also technical challenges in both machine learning and in infrastructure for serving the models at scale. For inputting melodies, we designed a simplified sheet music interface that facilitates easy trial and error and found that users adapted to it quickly even when they were not familiar with western music notation. As most users do not have dedicated hardware to run machine learning models, we re-implemented Coconet in TensorFlow.js [32] so that it could run in the browser. We reduced the model run-time from 40s to 2s by adopting dilated depth-wise separable convolutions and fusing operations, and we reduced the model download size to ∼400KB through post-training weight quantization. To prepare for large-scale deployment, we calibrated a speed test to determine if a user's device is fast enough for running the model in the browser. If not, the harmonization requests were sent to remote TPU servers. Users in 80% of sessions explored multiple harmonizations, and 53.8% of the harmonizations were rated as positive. One complaint from advance users was the presence of parallel fifths (P5s) and octaves (P8s). We analyzed 21.8 million harmonizations and found that P5s and P8s occur on average 0.365/measure and 0.391/measure respectively. Furthermore, P5s and P8s were more common when user input was out of distribution, and fewer P5s and P8s were correlated with positive user feedback. Machine learning has been used in algorithmic music composition to support a wide range of musical tasks [5, 13, 19, 28, 29] . Melody harmonization is one of the canonical tasks [7, 11, 20, 26] , encourages human-computer interaction [3, 14, 21, 25, 33] , and is particularly approachable for novices. Different interfaces and tools have been developed to make the interaction experience more accessible. For example, in MySong [31], users can sing a melody and have the system harmonize it. In Hyperscore [12] , users can draw multiple levels of "motifs" on a graphical sketchpad and have them harmonized according to the tension curve they specified. Startups such as JukeDeck and Amper Music offer APIs that allow users to describe a piece through timing and mood tags. Opensource libraries such as Magenta.js [30] allow machine learning models to be arXiv:1907.06637v1 [cs.SD]
An example of their work is the Bach doodle REF , which is able to harmonize in the style of Bach the melodies that users create manually.
The Bach Doodle: Approachable music composition with machine learning at scale
ArXiv
ArXiv
Computer Science Engineering Mathematics