Keyness in song lyrics: Challenges of highly clumpy data (2023)

Abstract: Computer-assisted stylistic analyses regularly employ the calculation of keywords. We show that the inclusion of a separate dispersion measure in addition to a frequency measure into keyword analysis (or more generally: keyness analysis), as proposed by Gries (2021), is a necessary extension of said analyses. Using texts from the German Songkorpus, we demonstrate that traditional keyword calculations using only frequency measures lead to spurious results. Determining keywords by both measuring a word’s frequency and its dispersion in comparison to a reference corpus gives a more realistic view. This is especially relevant for our corpus, since song lyrics turn out to be extraordinarily clumpy data: Words that are very frequent in one artist’s subcorpus typically only occur in a few or even just a single one of their songs due to widespread word repetition within songs, e.g., in choruses. Song lyrics in our dataset are shown to not feature words that can be considered key at all. Our contribution is twofold: (1) We demonstrate the utility of Gries’ (2021) approach and (2) interpret the (lack of) results in terms of a genre-specific property which is that song lyrics are lexically autonomous works of art.

## corpus for our analysis, we use data made available as part of the newly compiled songkorpus -linguistic corpus of german song lyrics @xcite ). 1 the subcorpus consists of song lyrics performed by seven german artists (singers and bands, see ## traditional approaches to keyness in order to analyze an artist's language on a lexical level, keyness analysis in which one artist's word frequencies are compared to those of all other artists in a corpus seems to be an appropriate approach. this approach is straightforward both in its calculation (only word frequencies and a short formula are needed) and in its interpretation (words are attracted to or repelled by a corpus to a quantifiable degree). as mentioned above, statistical measures relying on significance are particularly suitable for stylistic analysis, and a widely used measure is log likelihood ratio @xcite . using a contingency table, the observed frequencies of a word (or lemma, n-gram etc.) in both a target corpus and a reference corpus are compared to the expected frequencies given an even distribution of the words' frequencies across both corpora. observed frequencies that deviate from expected frequencies most yield a high llr value and are interpreted as being most key for a given corpus. positive keywords are more frequent in the target corpus than expected and can be interpreted as being characteristic or typical for the target corpus texts, while negative keywords occur less frequently than expected and are interpreted as atypical. this type of analysis is a standard method in corpus linguistics and is implemented in many popular tools such as cqpweb @xcite or sketchengine @xcite . calculating keywords for the german singer-songwriter hannes wader using this approach in its most basic implementationneither requiring keywords to have a minimum absolute frequency in the target corpus nor excluding stopwordsyields the results shown in ignoring the ampersand and the punctuation marks at the top of the list, foot_1 the keyword with the highest llr is alledem ('all that'). the table also shows the range of each word, i.e., in how many different texts it occurs at least once, for both the target and reference corpus. as can be seen, the 57 occurrences of the word alledem in the hannes wader subcorpus stem from only three different songs, and the word only occurs in eight different songs in the reference corpus. this is due to the fact that wader recorded three different versions of trotz alledem ('in spite of all that'), a song based on a 19 th century german poem (based on an even older scottish one): das war 'ne heiße märzenzeit trotz regen, schnee und alledem! nun aber, da es blüten schneit, nun ist es kalt, trotz alledem! trotz alledem und alledem -trotz wien, berlin und alledem this particular keyword derives its keyness from the fact that it is repeated very often in a small number of songs. same goes for the runner-up ciao which occurs 30 times in the hannes wader corpus and not once in the reference corpus, but the word only ever occurs in one single song, an interpretation of the popular italian partisan hymn bella ciao. the word na, then, seems to be an interesting candidate for style analysis because it serves as an interjection (e.g., na, willy or na gut) indicating a colloquial style, but also as a non-lexical vocable (na na na na). upon further inspection, the word turns out to only occur in a very small number of songs, but it is not even evenly dispersed across said songs with 96% of its occurrences clustered in one single text where the word is used as a most repetitive non-lexical vocable. to conclude, relying on these llr values leads to misinterpretation, because single words may seem as typical of an artist while they are in fact typical of certain songs only. assessing the range values in addition to llr does certainly add valuable information. however, as seen in the na example, it hides how the occurrences of a word are distributed within this range. since llr-based keyword analysis of concrete word forms or lemmas is distorted by the repetitiveness of the genre of song lyrics, a focus on more abstract patterns seems to be promising. particularly appropriate are part-of-speech ngrams (pos-ngrams) which allow for capturing typical syntactic patterns and contextual embeddings that are especially informative for style analysis @xcite . for example, a pos-trigram analysis for the singer-songwriter konstantin wecker yields the results shown in as demonstrated, such a shift in focus to more abstract patterns can indirectly remedy the above-mentioned deficiency of an llr-based keyness analysis which is that dispersion across texts is not considered at all. 3 while one could also introduce range thresholds (e.g., a word or pattern must appear in at least 30% of all texts), this would be an arbitrary measure which also leaves it unclear whether within this subset a word is dispersed evenly across texts or predominantly occurs in just one of them. as seen, this is especially problematic for song lyrics, as they are particularly repetitive by their nature. not only choruses are repeated, but also single words or phrases may appear again and again in a given song. thus, song lyrics are especially susceptible to containing clumpy data, i.e., words or patterns which have a low dispersion. this makes it difficult to use traditional approaches to keyness analysis. the problem of (lacking) dispersion in keyness analysis has been discussed before @xcite , and most recently, @xcite proposed a new approach to calculating keyness which incorporates a word's dispersion over the corpus as well as its frequency into keyness calculations. this design promises to solve the problem of clumpy data, so we will turn to this approach in the following section. ## adding dispersion to the mix gries' (2021) newly proposed method turns keyness into a two-dimensional concept with one dimension being a measure that is based on word frequencies and a second one which measures the dispersion of a word over a corpus. the frequency-based measure is calculated using the so-called kullback-leibler divergence which determines the divergence of two probability distributions as follows 4 : where a is a word's relative frequency (i.e., its probability of occurrence) in a target corpus, b is the relative frequency (i.e., its probability of occurrence) of said word in a reference corpus and e and f are the respective proportions of both corpora in the complete dataset (i.e., their respective probabilities of occurrence). put simply, one asks: 'what is the probability 3 for a differentiation between dispersion in a corpus linguistic and a statistical sense, see @xcite ). 4 zero is inserted instead of log values where where n is the number of texts in a given corpus, pi is the proportion of all occurrences of a given word that occur in text i within the corpus and qi is said text's proportion within the whole of the corpus. target and reference corpus are compared by subtracting a word's two klddisp values for each corpus from one another. 5 this value is again normalized to fall in the range [-1, 1] where a value of 1 would indicate that a word is very dispersed in a target corpus compared to a reference corpus and -1 would mean that a word is very dispersed in the reference corpus while at the same time very clumpy in the target corpus. for the hannes wader subcorpus keyword candidate alledem, these two calculations yield a kldfreq value of 0.81 and a difference in klddisp values of -0.001, respectively. the very high frequency of the word compared to the reference corpus is reflected in a very high kldfreq result. at the same time, the vanishingly small difference in klddisp values adequately conveys that this word is not well dispersed over the hannes wader subcorpus at all (it is minimally more dispersed over the reference texts, even though this difference is negligible). thus, using gries' (2021) method, this word would not be considered key regardless of its frequency in the target corpus compared to its frequency in the reference corpus. due to its multidimensional nature, results of this type of keyness analysis can best be assessed by plotting the frequency and dispersion values against each other. figure 1 shows the results for hannes wader. as one can clearly see, no words obtain high values on both scales and, accordingly, there are no words that can be considered key for the chosen artist. looking at texts by the german hip hop group fettes brot, which can be expected to have lyrics very different from hannes wader, we observe very similar results (figure 2 ). this pattern holds true for every single artist from our corpus when compared to the rest of the corpus (figure 3 ). words tend to be scattered along the kldfreq axis with many words obtaining very high values while most words obtain very low values on the klddisp axis (this can be seen most easily looking at the marginal plots in figure 3 ). words that are far more frequent in the target corpus as compared to the reference corpus do exist for every artist, but they tend to not be more dispersed over the respective artist's repertoire in comparison to the other artists. song lyrics' word distributions, at least those in our dataset, seem to be extremely clumpy. gries' (2021) results for the clinton-trump corpus @xcite ) look somewhat different with words being scattered more across the klddisp axis rather than concentrating just slightly above zero like in our data. however, a replication of his results including marginal plots (see figure 4 ) reveals a generally similar distribution for both axes. there is only one fundamental difference between our results and results from the clinton-trump corpus: in contrast to the clinton-trump results, there simply aren't any 'real' keywords in our data. revisiting the originally promising results for more abstract patterns using the traditional llr approach (ch. 2), pos-trigrams actually plot very similarly compared to single words following the kld method (see figure 5 for all artists plotted on top of each other). this means that also the pattern nn kon nn, which appeared to be typical for konstantin wecker when employing an llr calculation ( when measuring dispersion, it is not always straightforward across what an item should be dispersed. the songkorpus is organized into individual song lyrics, but these lyrics in turn are part of albums, which are also annotated. in our analysis thus far, we considered song lyrics as the unit of text, and hence the klddisp measure compared dispersion values for words across song lyrics. however, one might also conduct the very same analysis on a higher level with albums as the unit of text. this might be a legitimate approach: while individual song lyrics are indisputably the 'real' unit in the sense that they were written as discrete texts by their respective author, albums also constitute natural units of text in a wider sense. grouping items on the album level potentially impacts the results of a keyness analysis that incorporates the use of dispersion values. one could assume that a lower number of units in which a word can appear leads to a higher probability of it being dispersed more evenly across these units. keywords computed 'albumwise' differ most from 'songwise' keywords for fettes brot compared to all other artists (see figure 6 ). there is a small number of words that are well dispersed over the band's albums as well as being more frequent in comparison to the rest of the corpus: fettes which is part of the band's name and tends to get mentioned on many of their releases, as well as parts of the band members' names such as boris, renz, könig and colloquial contractions such as auf'm ('on the'). figure 7 shows all 'albumwise' keywords for all artists compared to the rest of the corpus plotted into the same graph (all words are plotted seven times, once for each artist as reference artist). as can be seen, these results differ from the results for a 'songwise' keyness analysis and now look more similar to gries' results (see figure 4 ). however, there are only very few keywords for each artist (compared to many for clinton speeches vs. trump speeches and vice versa) and most words do not obtain high values on both scales, despite the lower number of text units on which the klddisp measure is based. the question of what level constitutes the most natural textual unit within a corpus will also arise in settings where researchers deal with corpus data such as novels (whole volumes vs. chapters), newspapers (whole issues vs. sections vs. articles), etc. an adequate level on which dispersion is measured should be chosen wisely in any case. ## conclusion as we could demonstrate in our analysis, word dispersion matters when analyzing keywords. song lyrics appear to be a case where 'traditional' keyword-oriented style analysis based on mere frequency counts falls short. our evaluation of gries' (2021) multidimensional approach to keyness clearly showed its usefulness. including a measure of a word's or pattern's dispersion over both the target and reference corpus made disappear results that would have given a false impression of typicality. words that would initially yield high llr values and could thus be interpreted to be key for a given artist were shown to be artifacts of word repetition within songs. a strength of gries' ( 2021 ) method is that the klddisp measure does not require the use of an arbitrary range threshold and also captures a word's distribution within its range of occurrence. another aspect we have briefly touched on is that when introducing a measure of dispersion as described above, one has to carefully reason about the levels that are 'naturally' present in the data. while the change of level from single songs to whole albums in our corpus did not alter our results in a substantial way, this might be different for other data. the more one knows about the underlying structure of a given corpus, the better one can control for a possibly clumpy dispersion. the virtually complete absence of 'actual' keywords in the songkorpus data might be a surprise. it becomes very plausible, however, when one inspects the data's specific pattern of frequent word repetitions within single songs and few repetitions across songs. the repetition within a song is a typical stylistic device in song lyrics and the fact that word repetition across songs rarely occurs suggests that song lyrics are independent works of art. a limitation of our study is that our subcorpus includes 7 different artists and, depending on how one makes these distinctions, only 2 to 5 genres. the observed results might not hold true for a corpus that is structured differently and features a greater number of artists or artists representing a different set of genres. while our subcorpus contained complete discographies of a small number of artists, the songkorpus archive also contains, e.g., a charts archive featuring a more diverse set of artists and genres with a smaller number of songs per artist. this dataset could be used for follow-up analyses. an awareness for the importance of dispersion for keyness analysis seems to generally be on the rise and methods that alleviate the risk of making false assumptions based on frequency-only-methods are being refined. besides gries' (2021) and egbert/biber's (2019) method, another very promising approach incorporating both frequency and dispersion measures using negative binomial regression has very recently been proposed by @xcite . available approaches for improving keyness analysis should be evaluated on a greater number of different corpora and their performance should be compared. there exist numerous text genres where one can expect data to be potentially clumpy and an inclusion of dispersion measures might be warranted. for example, newspapers, which are a popular source for general corpora, might have a very particular distribution of certain words across their sections. in these cases, clumpiness might pose less of a problem compared to song lyrics, which we suspect to be an extreme case, but controlling for dispersion should ideally become a standard procedure which should also be included in corpus analysis software. 6 keyness analysis in general has proven to be a useful tool for style analysis, partly because it is not based on strong presumptions on the side of the researcher. as is becoming clearer and clearer, however, the available methods of keyword calculation have relied too strongly on a latent assumption of a general correlation between frequency and dispersion. the 'naïve' keyword list calculation using log-likelihood-ratios or similar measures is in many cases an insufficient representation of the occurrence of words or larger patterns in corpus data.