Measuring Similarity by Linguistic Features rather than Frequency (2022)

Abstract: In the use and creation of current Deep Learning Models the only number that is used for the overall computation is the frequency value associated with the current word form in the corpus, which is used to substitute it. Frequency values come in two forms: absolute and relative. Absolute frequency is used indirectly when selecting the vocabulary against which the word embeddings are created: the cutoff threshold is usually fixed at 30/50K entries of the most frequent words. Relative frequency comes in directly when computing word embeddings based on co-occurrence values of the tokens included in a window size 2/5 adjacent tokens. The latter values are then used to compute similarity, mostly based on cosine distance. In this paper we will evaluate the impact of these two frequency parameters on a small corpus of Italian sentences whose main features are two: presence of very rare words and of non-canonical structures. Rather than basing our evaluation on cosine measure alone, we propose a graded scale of scores which are linguistically motivated. The results computed on the basis of a perusal of BERT’s raw embeddings shows that the two parameters conspire to decide the level of predictability.

## introduction this paper presents work carried out to verify whether current transformer based models like bert @xcite are able to cope with linguistically highly complex datasets and to what degree. in particular, bert tries to predict the next word or sentence on the basis of word embeddings as they have been represented in the pre-trained model: in the experiment we used only the output of the first projection layer of a deep learning model, the raw word embeddings. we organized an experiment on a small number of italian sentences taken from two domains: newspapers and poetry domain. they represent two levels of increasing difficulty in the possibility to predict the masked word that we in-tended to test. the experiment is organized on the hypothesis of increasing difficulty in predictability at the three levels of linguistic complexity that we intend to monitor: lexical, syntactic and semantic level. whereas lexical predictability may be based on word frequency and not just context, syntax and semantics strictly constrain meaning understanding. to test this hypothesis we alternate canonical and non-canonical version of the same sentence before processing them with the same dl model. in particular, we expect the poetry domain to introduce additional restrictions on the local word context due to the need to create metaphors which require non-literal meaning compositional processes. the notion of "similarity" which cosine measure is supposed to represent is very poorly defined. it can represent "semantic" similarity of a candidate word proposed by the model to the target one, in the sense of being semantically "related" or just semantically "associated" to the target word. however, whenever the two items are not identical nor semantically close in any sense they might still exhibit "linguistic" similarity which is shown by lexical, morphological, and syntactic features. these features are very important in their ability to reveal how close the prediction has been on the basis of frequency of (co)-occurence -the context, as it is being measured by word embeddings and their vector space models. in fact, the only number that is being used by neural networks and deep learning models is frequency of occurrence, that substitutes words in the overall computation. now, frequency of (co)-occurence which characterizes the context, is strictly dependent on absolute frequency and even though the corpora being used nowadays to build and train the models are huge and number by the terabytes, they will always be subjected to the zipfian laws that establish that the frequency of any word is inversely proportional to its rank in the frequency table. as a consequence, the reference dictionary on which basis the embeddings are built -usually comprising only the first 30/50k most frequent word forms -will always be a very partial picture of the corpus it should represent, and the use of subword units does not solve the problem (see @xcite ). in order to evaluate fine-grained levels of similarity between the predicted outcome and the expected result we devised a graded scale of linguistically based scoring table which is then used to produce a "predictability parameter". this parameter is highly correlated with the cosine measure used to gauge the similarity between predicted and expected, but produces a better distinction between linguistically evaluated classes. a further important element contributing to define the kind of response italian language exhibits to deep learning is its inherent language structure which is very different from english as discussed in what follows. ## the experimental setup: (co)-frequency and cosine measures do not coincide we assume that word predictability can be characterized by two parameters: word (co-occurrence) frequency/ies and linguistic complexity measured by the context, or a syntactic/semantic related scoring function. we evaluate word co-occurrence frequencies by means of embeddings as the cosine value made available by bert 4 in its first projection layer, using pretrained models and no finetuning. as said above, we used bert -with the italian model taken from uwac corpus, umbertocommoncrawl -and examined the output of the first or projection layer 5 . in this way we intended to check the predicting ability of bert on the masked word, by selecting in turn one content word at a time allowing bert to use the rest of the sentence as a context to make appropriate predictions. to this aim we ran bert by masking each content word and some function word, one at a time in order to be able to make a detailed error object fronting (the object comes before the subject at the beginning of the sentence) in sentence 2a and 5a; adjective extraction (the adjective is extracted and fronted from the noun phrase) in sentence 13a and 14a; ppadjunct preposing from participial clause in sentence 1b and 13a; lexical verb left extraction (the main verb -untensed nonfinite -is positioned before the auxiliary/modal) in sentence 3a; subject right dislocation (the subject is positioned after the complements) in sentence 3a and 6b; subject and object fronting (the subject comes before the object and both are positioned before the main verb) in sentence 4a and 5a; ppspecification extraction from the noun phrase and fronted to the left in sentence 5a; clitic left dislocation in sentence 8b; object right dislocation (the object is positioned after the indirect object or the adjuncts) in sentence 10b; parenthetical insertion (a parenthetical is inserted after the subject before the main verb) in sentence 11b and 16b; adjective right extraction (the adjective is extracted from the noun phrase and positioned after the noun adjuncts) in sentence 11b and 14a; ppspecification right stranding -the ppof is stranded to the right out of the noun phrase in sentence 14b; lexical verb right extraction (the main verb -untensed nonfinite -is positioned after the complements) in sentence 12a; double parenthetical insertions (after the subject and after the verb complex and before the complements) in sentence 15b and 16b; clitic left dislocation with subject fronted as hanging topic in sentence 18b. sentence 1.b -oggi ringrazio della cortesia in più occasioni dimostrata a me e ai miei colleghi. 1.bc oggi ringrazio della cortesia dimostrata a me e ai miei colleghi in più occasioni. the sentence belongs to the newswire domain: it is computed best in the canonical form, with 5 words over 8 while the non-canonical version has only 3 words predicted correctly -only "più/more", "occasioni/chances" and "miei/my". cosine values are not particularly high except for "miei/my" the possessive which being in its attributive position has a favourable predictive condition. "oggi" is wrongly predicted as being a separator with very high value, " s 0.99998". it can be noted that "ringrazio" is partially predicted by "grazie" in first position but very low value 0.14397. now the canonical version: ringrazio (0.0238), più (0.287), occasioni (0.545), dimostrata (0.165), miei (0.882). interesting to note that the three words predicted in both structural versions have the same cosine values. when we add the remaining 7 sentences, another word is predicted, colleghi (0.076). no connection with frequency values of the missing words: they are all positioned in the high part of the frequency list -excluding "più" and "miei" which are grammatical words and are positioned close to the top. frequency list: °-pi ù; °-miei; °-oggi; °-colleghi; °-occasioni; °-ringrazio; °-dimostrata; °-cortesia sentence 2.a -lei sola forse il freddo sognatore educherebbe al tenero prodigio. 2.ac forse il freddo sognatore educherebbe lei sola al tenero prodigio. the second sentence belongs to the poetry domain. the original non-canonical version has no candidate found in the first 5 positions. this may be due to presence of a rather infrequent word like "educherebbe/would+educate" as main verb which only appears listed low only in the upper list. on the contrary, the canonical form has three words predicted: first "forse/maybe ", second word "lei/she", and third word "solo"/alone but with wrong masculine morphology. however, these words are correctly predicted with low cosine values -forse (0.149), lei (0.0355) solo (0.0145). no version provides useful approximations of the meaning of the missing words even though "freddo/cold" is included in the high portion of the 50000 vocabulary. as to the remaining words, they are still included in the vocabulary but in the lower portion. it is important to note that the lack of prediction can only be motivated just because by combining not so frequent words in unusual combination has produced metaphors like "cold dreamer", "tender prodigy", in association with a verb like "educate". frequency list: °-solo; °-lei; °-forse; °-freddo; *-tenero; *-prodigio; *-sognatore; **-educherebbe sentence 3.a -penso a un verde giardino ove con te riprendere può a conversare l'anima fanciulla. 3.ac penso a un verde giardino ove l'anima fanciulla può riprendere a conversare con te. the non-canonical version of this sentence has two words correctly predicted, giardino/garden, ove/where and a third word with different morphology, in slot 5, pensa/think(3rd+person+singular+present+indicative), rather than penso(1st+person). in the canonical version we find correctly penso/think in second slot, and another word is added può/can, the modal auxiliary that is now positioned correctly in front of its main verb "riprendere/restart", which is by itself a very frequent verb. as to cosine values, we have the following low values for the canonical version: penso (0.085), giardino (0.194), ove (0.146), può (0.0865). the non-canonical version has a lower value for penso but a higher value for giardino (0.291). in the longer context, the interesting fact is constituted by the substitution of "pensa" with fino/until in the non-canonical version; while in the canonical version penso/think is moved to a worse position from second slot to last slot, slot 5 and a lower cosine value (0.06112). as to the non-predicted noun modifier "fanciulla/maid", this is certainly an unusual combination even though the two words are highly frequent. the result of the combination is of course a beautiful metaphor which combines "primavera"/spring with "fanciulla"/maid and the garden. notice the different position of penso+1st+pers, with respect to pensa+3rd+pers which is by far less frequent. now consider the word conversare/conversing which receives the following list of non-word predicted candidates: erare/?? (0.4455), rare/rare?? (0.16737), lare/?? (0.0549), mare/sea?? (0.0479), scere/?? (0.03124). apart from rare and mare which i don't regard being selected for their current meaning but just for being part of the list of subwords, the remaining segments are all meaningless and bear no semantically useful relation with the masked word conversare. frequency list: °-può; °-ove; °-anima; °-verde; °-penso; °-riprendere; *-pensa; *-fanciulla; *-conversare sentence 4.a -se primavera il mio cuor generoso soffocasti di spasimi sordi. 4.ac primavera, se soffocasti il mio cuor generoso di spasimi sordi. in this sentence only the phrase "mio cuor"/my heart is predicted in both structural versions. mio (0.291), cuor (0.394). the word "primavera", which is the first word in the canonical version, has no close prediction: as happens in all sentences, the prediction is totally missed whenever a content word appears in first position. in the non-canonical version, the word comes second, after the conjunction "se"/if, which predicts the appearance of an auxiliary be/have in their correct morphological word form -fossi/were, avessi/had in both cases with first person morphology, but also fosse/were, and the last two: con/with and solo/alone. the version with the addition of the 7 sentences has the worsening effect of introducing a subword in place of con/with, mmai which i assume derives from the wrongly split semmai/if+ever. the word has been wrongly split because the segment se is wrongly -at least in the word semmai -regarded as a legitimate segment due to its very high frequency. again the problem seems the unusual combination of the remaining words which are fairly common, apart from soffocasti/choked which is not included in the frequent nor in the rare wordform list; and spasmi/spasms which is only included in the upper list. in other words, it's their metaphorical import that prevents the correct prediction. however, it is the position that produces the worst results: the adjective "sordi/deaf" in predicative position is predicted as a punctuation mark in both structural versions. frequency list: °-se; °-mio; °-cuore; °-primavera; *-generoso; *-primavera; *-sordi; **-spasmi sentence 5.a -né l'oblioso incanto dell'ora il ferreo battito concede. 5.ac né il ferreo battito dell'ora concede l'oblioso incanto. this sentence is the worst case of the poetry domain lot: it has no word predicted neither in the non-canonical nor in the canonical version. this may be due to the presence of a very infrequent word "obliosi/oblivious". however, we notice the presence of an unusual combination of the attributive metaphoric use of "ferreo/iron-like", a rather unusual word. but of course, it is just the combination of words used to build a powerful metaphor that prevents predictions to take place. it is worthwhile noting that "incanto"/enchantment is substituted by ten candidates semantically loosely related to the domains evoked by the masked word: temporal dimension (rhythm, stepping, passing, proceeding, beat), and a condition of the contemplating mind (silence, rest, meaning, thought, sound). also another important remark regards the inability to predict the ambiguous word "ora"/hour, homograph with "ora"/now, thus clearly showing that context is the determining factor. frequency list: °-ora; °-né; °-concede; °-incanto; *-battito; **-ferreo; **-oblioso sentence 6.b -diventa così più acuta la contraddizione. 6.bc la contraddizione diventa così più acuta. this sentence has different predicted words in the two structural representations, diventa/becomes is present in both. then "così/so" and "più/more" are predicted in the canonical sentence -diventa (0.215), così (0.0439), più (0.559); while in the non-canonical structure only acuta/sharp is predicted, acuta (0.0441), and the cosine value for "diventa" is lower being in sentence first position. the canonical form has predicted the discourse marker "così/so" positioned in sentence center: not so in the non-canonical structure where we can again assume that it is the position right after the verb at the beginning of the sentence that does not allow the prediction, notwithstanding its high frequency. now consider the high frequency of "contraddizione" which is not predicted presumably because of its position at the end of the sentence: the first candidate is the subword "mente" with cosine value (0.16536), followed by sensibilità/sensibility, coscienza/conscience, gioia/joy. frequency list: °-pi ù; °-così; °-contraddizione; °-acuta; *-diventa sentence 7.b -buono invece in complesso il resto. 7.bc invece in complesso il resto è buono. no word was predicted in either versions. in order to transform the original non-canonical version in the corresponding canonical one we added the copula "è" that is missing in the original sentence. this is predicted in the canonical version but since it has been added we do not count it for the actual predictive task. all the words are very frequent. as will be clarified further on, whenever the first word of the sentence coincides with a discourse marker or a conjunction the prediction is very close if not equal. this is the case for the canonical form of the sentence starting with "invece"/rather, which has the five following best predictions: "ma"/but, "e"/and, "però"/however, "più"/more, "ed"/and, all belonging to the same grammatical category and in two cases, also to the same semantic type ("ma", "però"). considering the status of the adjective "buono"/good which comes in first position in the non-canonical structure and in second position in the canonical one, one can clearly realize the importance of the respective position and the context on the ability of bert to predict. in the first case, the word coming first position has no left context and there is no similarity, not even at a grammatical level: only conjunctions and verbs are predicted. on the contrary, in the canonical form, "buono" appears as predicate in a copulative structure and the predictions are very close: diverso/different, risolto/resolved, compiuto/achieved, secondario/secondary, positivo/positive. frequency list: °-invece; °-resto; °-complesso; *-buono sentence 8.b -una decisione importante ghitti l'ha riservata a dopo le feste. 8.bc ghitti ha riservato una decisione importante a dopo le feste. only one word is predicted in both versions but it is not the same word. the canonical version predicts "importante/important", (0,0605), the non-canonical version predicts "dopo/after", (0.0152). as can be noticed, the cosine values are very low and again the frequency of occurrence of the words contained in the sentence is fairly high -excluding the proper name "ghitti" which does not exist in the overall frequency list. the unexpected fact is constituted by the inability to predict the auxiliary "ha"/has in the non-canonical structure -as opposed to what happens in the canonical one -, and the association in fourth slot of a non-word like "vamteen", presumably a subword of some kind. the only explanation could be the presence of a past participle with feminine+singular ending which is only allowed by presence of the resumptive clitic "la" needed to construct the clitic left dislocation of the object np "una decisione importante". as said above, the canonical version predicts the presence of the auxiliary have in the correct form and also in two additional morphologically possible forms: "aveva"/had+3rd+pers and "avrebbe"/would+have+3rd+pers; final word predicted in the other auxiliary legal form "è"/is. frequency list: °-dopo; °-importante; °-decisione; °-riservata; °-feste; ***ukn-ghitti sentence 9.b -l'importante ora è aprirlo di più. 9.bc ora è importante aprirlo di più. this sentence is perhaps too short and only function words are captured by bert embeddings: ora/now (0.3825) più/more (0.0911). the ambiguous word "ora"/now is better predicted in the non-canonical structure -in first position -for the availability of right context -the canonical version predicts "ora" in fourth position (0.0844). again this is not relatable to a frequency problem but just structural problems, with the exception perhaps of the final word "aprirlo" which is only present in the very-low frequency list. in fact, in the canonical version, "aprirlo"/open+it is substituted by cliticized verbs -though semantically unrelated, however, showing that the morphology has been captured correctly. as to "importante"/important, it does not appear in the first five candidates, but it is predicted in sixth position (0.04902). frequency list: °-ora; **-aprirlo sentence 10.b -le sue informazioni darebbero anche agli orientamenti di democrazia laica maggiori spinte. 10.bc le sue informazioni darebbero maggiori spinte anche agli orientamenti di democrazia laica. this sentence has the same predicted word "maggiori/major" in both structural representations. as before, the words are all very frequent with the exception of "darebbero/+would+give, which is below the threshold and is only part of the "very+low" list. now consider the word spinte/boosts: predicted masked words are as follows: certezze/certainties (0.0852), garanzie/guarantees (0.0824), informazioni/information (0.04183), taria/tary (0.04003), opportunità/opportunities (0.0383). the fourth slot contains a subword, in fact a non-word, which is assigned a score higher than the one assigned to "opportunities". the question is that the masked word is not frequent enough to be able to collect the co-occurrences required. as a result, even very low scored embeddings are considered. the non-word gets a slightly better score when the text is considered as a whole with the last 7 sentences added, up to (0.06002), but remains always in fourth position. frequency list: °-anche; °-informazioni; °-sue; °-maggiori; °-democrazia; °-orientamenti; °-laica; *-spinte; *-darebbero sentence 11.b -in questo libro maria teresa, spiegano alla mondadori, darà esempi di carità concreti. 11.bc in questo libro maria teresa darà esempi di carità concreti, spiegano alla mondadori. in this sentence there is a striking difference in prediction between the two structures. the non-canonical version has only two words predicted, "libro/book" and "esempi/examples", libro (0.0242), esempi (0.653). on the contrary, in the canonical version bert manages to predict four words, "questo/this", "maria/mary", "teresa/therese", "esempi/examples", questo (0.767), maria (0.283), teresa (0.141), esempi (0.734). strangely enough, the word "libro" does not figure in the first five candidates. useless to say, the remaining words are all very frequent. the third run with a longer text including the following 7 sentences gives interesting results: "teresa" now becomes first candidate substituting the previously chosen first candidate "ci"/us. the word "esempi"/examples, predicted as first candidate, in the text is followed by "carità"/charity which is not predicted in both version: in its place, the first candidate is again "esempi", thus certifying that predictions are made one word at a time disregarding the textual context. now consider the adjective "concreti" which has been dislocated and is disjoined from its head, "esempi". the list of five candidates for the canonical version is the following: "cristiana+fem+sing"/christian (0.1919), '.' (0.0909), ',' (0.0387), "civile+sing"/civil (0.0383), "esemplare+sing"/exemplar (0.0222). none of the candidates is plural in number as it should be, if the morphology of italian has to be respected. on the contrary, the first candidate agrees both in number and gender with the preceding word "carità+fem+sing"/charity, which is not to be considered the correct nominal head. the non-canonical version has one punctuation mark less and an additional adjective "pastorale+sing"/pastoral. frequency list: °-questo; °-libro; °-esempi; °-carità; °-concreti; °-darà; °-spiegano; °-mondadori sentence 12.a -disse che gli hanno il cor di mezzo il petto tolto. 12.ac disse che gli hanno tolto il cuore di mezzo il petto. this sentence from the poetry subset has only one word in common "cor/heart" and an additional word predicted in the canonical structure, "tolto/taken+off". the cosine values are all very low, cor-cuore (0.1019), for the non-canonical, and cor-cuore (0.0756), tolto (0.156) in the other structure. interesting enough, when using the configuration with the whole text, also "mezzo/means" is predicted in second slot. frequency list: °-mezzo; °-cuore; °-petto; °-tolto; *-disse sentence 13.a -i ritrosi pareri e le non pronte e in mezzo a l'eseguire opere impedite. 13.ac i ritrosi pareri e le opere non pronte e impedite in mezzo a l'eseguire. no prediction found by bert in the two structural representationswith the exception of "mezzo"/means which however is only appearing in 8th position and not considered in this evlauation. however it is important to note that the previous seven predicted words are in fact only subwords, mostly meaningless, and some having a corresponding identical wordform with a totally different meaning. here they are: "dotti"/learned+mas+plur, "dotte"/learned+fem+plur, "tente"/meaningless, "sistenti"/meaningless, "sistenza"/meaningless,"difficoltà"/difficulty, "fami"/meaningless. as to their frequency, words are mostly frequent but there are two missing words in the overall frequency lists: "ritrosi/reluctant" and "impedite/hampered". these two words may have been supplemented as subwords but with no useful context for the current analysis. the five candidates appearing are as follows: for "ritrosi" we have -suoi/his+hers, non/not, buoni/good+masc+plur, mal/bad(truncated), loro/their+them+they; and for "impedite" -'.', buone/good+fem+plur, inutili/useless+plur, nuove/new+fem+plur, pubbliche/public+fem+plur. in all of these cases, even if the correct word has not been predicted, the morphology has been matched correctly. frequency list: °-mezzo; °-opere; °-pareri; °-eseguire; °-pronte; ***ritrosi; ***impedite sentence 14.a -un'eco di mature angosce rinverdiva a toccar segni alla carne oscuri di gioia. 14.ac un'eco di mature angosce rinverdiva a toccar segni di gioia oscuri alla carne. this is another sentence from poetry domain very hard to tackle and to understand. both the canonical and the non-canonical analyses have just one word found, "eco/echo" (0.0984). of course the main verb "rinverdiva" is not amongst the frequent words in the list: in fact, it is missing. the remaining words are frequent but they are organized in a peculiar structural configuration with the declared aim to produce metaphors. no changes or improvements when the sentence is analysed with the canonical version of the text. as we did for example 11, we now consider the discontinuous adjective "oscuri+masc+plur"/obscure and the morphology of the five candidates predicted. in the non-canonical version we have: "pieni+mas+plur"/full (0.5461), "piena+fem+sing"/full (0.0486), "e"/and, ',', "pieno+mas+sing"/full (0.0216). now the canonical version: "fino"/until (0.1139), "intorno"/around (0.1139), "dentro"/inside (0.1001), "sino"/until (0.0476), "vicino"/close (0.0437). as can be noticed, all of the predicted words for the non-canonical structure are function words and none -with the possible exclusion of the ambiguou "vicino+mas+sing" -is an adjective. the reason for this lack of grammatical match may be due to the presence of the articulated preposition "alle"/to the+fem+plur in the canonical version. in the non-canonical version the word "oscuri" was followed by a preposition "di" which is the most frequent wordform with 65 million occurrences. frequency list: °-alla; °-carne; °-gioia; °-segni; °-toccare; °-eco; *-oscuri; *-mature; *-angosce; ***rinverdiva sentence 15.b -il governo, quindi, pur rinunciando alla maggioranza assoluta, ha voluto, come già nell'imi, puntare a una privatizzazione graduale. 15.bc quindi, il governo ha voluto puntare a una privatizzazione graduale pur rinunciando alla maggioranza assoluta come già nell'imi. this long sentence belongs to the domain of the news and even in its non-canonical structure, it is more linear and thus more predictable. there are seven words predicted (over ten we masked) in the two versions: governo/government (0.304), maggioranza/majority (0.0377), assoluta/absolute (0.349), ha/has (0.977), voluto/wanted (0.491), puntare/aim (0.0385). the proper name imi is in the very low list. strangely enough the function word come/like (0.1925/0.9186) is predicted as first candidate in its non-canonical position, as second position ,but with a much lower cosine measure in canonical position. frequency list: °-governo; °-maggioranza; °-voluto; °-assoluta; °-puntare, °-privatizzazione; °-graduale; *-rinunciando; **-imi sentence 16.b -in una conferenza al viminale il ministro, quando viene interrogato sul senatore a vita, sulle prime non capisce il nome. 16.bc in una conferenza al viminale, quando viene interrogato sul senatore a vita sulle prime il ministro non capisce il nome. there are four words predicted in this long sentence, again in the domain of the news, in the canonical and the non-canonical structures. they are: ministro/minister (0.497), viene (0.795), senatore/senator (0.808), vita/life (0.996). again, most words are very frequent. an apparent difficulty is constituted by presence of a multiword: "sulle prime/at first" which may be hard to distinguish and differentiate on the basis of the context. in fact, in both structures, "prime" is substituted by riforme/reforms, banche/banks, dimissioni/resignation , pensioni/pensions, cose/things. frequency list: °-vita; °-viene: °-nome; °-ministro; °-prime; °-senatore: °-conferenza; °-capisce; *-interrogato; *-viminale sentence 17.b -primo intervento da fare, ha detto in questi giorni, è di attuare la riforma. 17.bc primo intervento da fare è di attuare la riforma, ha detto in questi giorni. this is another fairly simple sentence which has the major number of predicted words in the whole set in relation to the total number in the sentence. there are six words predicted both in the canonical and the non-canonical version: "fare/do" (0.818), "ha/has" (0.283), questi/these (0.961), giorni/days (0.83), riforma/reform (0.194). the only difference being the slot assigned to riforma/reform, which has first slot in the canonical version and second slot in the non-canonical one, preceded by costituzione/constitution. useless to say, the missing words are all very frequent. frequency list: °-fare; °-giorni; °-detto; °-intervento; °-riforma; °-primo; °-attuare sentence 18.b -io il privato lo concepisco come un metodo di lavoro, come contratti di lavoro, come modo di gestire insomma. 18.bc io concepisco il privato come un metodo di lavoro, come contratti di lavoro, come modo di gestire insomma. in this final sentence again belonging to the newswire domain, there are four words predicted: metodo/method (0.0618), lavoro/work (0.214), lavoro/work (0.214), modo/way (0.794). again very frequent missing words, apart from "concepisco/surmise" which is the only word present in the rare-words list. when analyzed with the canonical version of the text, the word lavoro/work moves from third to first slot, with a slightly improved cosine score. frequency list: °-lavoro; °-modo; °-io; °-contratti; °-privato; °-metodo; °-insomma; °-gestire; ***-concepisco. ## experimental results and discussion the evaluation has been carried out in three different configurations: on a first configuration, part of the sentences, the last 7 -are withheld with the aim to reduce the overall context at sentence level. this is done both for non-canonical and canonical structures. then the last 7 sentences are added and the cosine values verified to see if predictions have been modified. in other words, the context is both dependent on word frequency and word structural position. one example is highly representative of the interplay between frequency and context and is the word "ora", an ambiguous word with two homographshomophones: one meaning "now", an adverbial contained in sentence n. 9 -the newswire domain; and another meaning "hour", a (temporal) noun, contained in sentence n. 5 -the poetry domain. only the adverbial is predicted in both structural versions. the noun is contained in a sentence belonging to the poetry domain where the overall context is not supportive for that word predictability. in figure 1 . below we show weighted -by number of masked words -cosine values -by choosing always the value associated with the first candidate -when compared with weighted linguistic parameter by listing sentences in descending order according to their score. correlation evaluation between our linguistic parameter and cosine values is estimated at 0.8705 when computed on absolute values, but it goes down to 0.6349 when using weighted values. news texts have overall higher parameters in both evaluations: the descending trend is however much more linear for linguistic parameters than for the cosine ones. ## conclusion in this paper we have proposed a word predictability parameter based on linguistically motivated information that we have tested in a highly constrained context determined by the combination of three fundamental factors for a sentence meaning understanding perspective on the prediction task represented by bert masked task: use of infrequent words -as measured against the itwac frequency list -and their phrase level combination -word poetic usage for metaphors w.r.t possible semantic association -, and their larger sentential context in uncommon syntactic structures -noncanonical structures. in order to be able to evaluate the different impact of the three adversarial factors on masked word prediction, we have included in the dataset a higher number of sentences from newswire domain showing the same structural syntactic properties but lacking both the usage of very infrequent words -with a few exceptions -and their uncommon combination to produce metaphors. word predictability has then been measured by bert raw word embeddings and their cosine measure, by masking one content word at a time -and a few function words. each content word has then been searched in the frequency list made available by the itwac frequency list. the results have clearly shown the ability of newswire sentences to receive an overall higher word predictability score thanks to the smaller effect of adversarial factors we investigated. the answer to the question: is frequency or context the determining factor for transformer language models to predict the masked word, is both are. the news domain has less infrequent words and less uncommon noncanonical structures than the poetry domain, which is what explains the remarkable difference in final results.