Spectral Probing (2022)

Abstract: Linguistic information is encoded at varying timescales (subwords, phrases, etc.) and communicative levels, such as syntax and semantics. Contextualized embeddings have analogously been found to capture these phenomena at distinctive layers and frequencies. Leveraging these findings, we develop a fully learnable frequency filter to identify spectral profiles for any given task. It enables vastly more granular analyses than prior handcrafted filters, and improves on efficiency. After demonstrating the informativeness of spectral probing over manual filters in a monolingual setting, we investigate its multilingual characteristics across seven diverse NLP tasks in six languages. Our analyses identify distinctive spectral profiles which quantify cross-task similarity in a linguistically intuitive manner, while remaining consistent across languages—highlighting their potential as robust, lightweight task descriptors.

## introduction analyzing the contextualized embedding representations of pre-trained language models (lms) using lightweight probes @xcite has identified latent features in the untuned encoders which are highly relevant to downstream nlp tasks at various layer depths @xcite . orthogonally, linguistic phenomena are also encoded at different timescales: i.e., rapidly changing (sub-)word-level information versus slower changing sentence or paragraph-level information. decomposing contextualized embeddings into frequencies with different rates of change has yielded initial insights into the timescales at which these task-specific latent phenomena occur @xcite . these findings currently rely on handcrafted spectral filters and are limited to english. to enable more efficient analyses of finer-grained, continuous frequency spectra in contextualized representations covering more tasks and languages, this work contributes: given a sequence of embedding values foot_0 , decompose into composite frequency waves using dct 2 , apply the learned filter γ , retaining a subset of waves 3 , for which idct returns the filtered sequence of values 4 . ## probing for spectral profiles spectral probing (figure 1 ) builds on established signal processing methods @xcite and recent findings on the manual frequency filtering of contextual embeddings @xcite . the method automatically learns spectral profiles which measure the relevance of specific frequencies to a given task by amplifying or reducing contextual information with different rates of change. discrete cosine transform @xcite is an invertible method for decomposing any sequence of real values {x 0 , . . . , x n -1 } (e.g., all values of an embedding dimension) into a weighted sum over cosine waves with different frequencies. the number of frequencies equals the sequence length n , as the lowest frequency wave is a constant (@xmath0. inverting the dct (idct) using all x (k) n will return the original sequence. however, weighting coefficients for some k by 0 will return a filtered version. zeroing out all k above a threshold will only retain lower frequencies and make values oscillate with a slow rate of change. vice-versa, zeroing out all k below a threshold will only retain higher frequencies-amplifying short-term changes. fixed-band filters applying frequency filters to a sequence of contextualized embeddings extracts linguistic information at different timescales. within this formulation, the values across each embedding dimension are gathered into a real-valued sequence to which transformations such as the dct can be applied. in seminal work, tamkin et al. (2020) apply manually defined low (@xmath1, 1]), mid-low (@xmath2, 8]), mid (@xmath3, 33]), midhigh (@xmath4. capturing the full picture using manual, fixed-band filters is however not computationally feasible: relevant frequencies might not lie in a contiguous band, and furthermore, frequencies can not only be turned on or off (i.e., weighted 0 or 1), but can actually be weighted continuously in [0, 1]. learnable filters to capture the complete picture, we propose spectral probing which learns a continuous weighting of frequencies relevant to a task. in effect, the spectral filter is a vector γ ∈ r m for which each entry corresponds to the weight assigned to a particular frequency. before inverting the dct, each x (k) n is multiplied by the sigmoidscaled weight γ (k) ∈ [0, 1] which will then retain or filter out frequencies at index k. as m depends on the sequence length n , which changes across inputs, the spectral probe dynamically scales γ to the length at hand using adaptive mean pooling. in practice, we set m to the maximum input length for our given encoder (e.g., 512 for bert) and shrink γ appropriately, as a wave cannot cycle more often than there are values. it would however be equally possible to set m smaller than n and interpolate the filter up to the length required. overall, γ is a lightweight parameter which can be easily incorporated between the frozen encoder and probing head, and uses the existing training objective to jointly learn which frequencies to amplify or filter out. ## spectral profiling analysis each task's distinct spectral profile (figure 3 ) allows us to analyze their relation to the timescale hierarchy of linguistic structures, and quantify crosstask similarities within and across languages. for this we use the percentage-normalized l1 distance (i.e., 0%-100% overlap) between filters (figure 4 ). cross-task overlap overall, we observe a dichotomy between broad-frequency, token-level tasks and low-frequency, sequence-level tasks (figures 3 and 4a). in addition, there appears to be a hierarchy which depends on the timescales of the linguistic structures involved. notably, compared to prior fixed-band filters, none of the learned filters fully excludes low frequencies. for instance, highfrequency information is most important to retrieve pos, but reaching the performance of the original embeddings also requires some lower-frequency information-most likely to disambiguate difficult cases based on sentence-level context. dep appears to benefit the least from both lower and higher-frequency information. instead, the strong weight on mid-high frequencies matches the fact that dependency relations span multiple words and benefit from information at the phrase-level. ner sees a further decrease in high-frequency information, coupled with an uptick in lower frequencies. we hypothesize that phrase and sentencelevel information become more important for disambiguating certain entity types (e.g., org and loc). across the token-level tasks this shift from higher to lower frequencies is also reflected in filter overlap which decreases from syntactic to semantic token-level tasks, while their overlap with sentencelevel tasks increases (figure 4a ). the sequence-level tasks share low-frequency spectral profiles which overlap more with each other than do the token-level tasks. in fact, senti and topic overlap almost perfectly (although the latter involves less mid-range frequencies). this similarity is unlikely to be the result of the shared underlying dataset as both tasks also overlap with the unrelated xnli and jsnli datasets. at the same time, the pos and dep tasks, which also share datasets, have a lower overlap despite being based on the exact same inputs. overall, senti, topic and nli all appear to rely on information which is consistent across a sequence-explaining why simple methods such as mean-pooled sentence embeddings can be effective in these scenarios. qa provides an intermediate case: while it is reliant on low frequencies it also includes more mid-low and a small amount of higher frequency information. this is reflected in figure 4a , where it shares more overlap with the token-level tasks than all other sequence-level tasks. since probing for the correctness of a question-answer pair is dependent on finer-grained information than the general sentiment, topic or semantic coherence of a sequence, this inclusion of higher frequency information matches linguistic intuitions. cross-lingual consistency finally, we investigate the similarity of learned spectral profiles across languages. while figure 3 shows that there is some variance between the filters of different languages within a task, figure 4b shows that actual quantitative overlap between languages is high, ranging from 94%-98%. this holds even across distinctive pairs such as ja-en which differ substantially in factors such as sub-word length and distance between syntactic dependents. this strong consistency highlights the potential for spectral profiles to provide language-agnostic features for task characterization and comparison. linguistic information at different timescales is an, as of yet, underexplored dimension in contextualized embeddings. we propose a fully differentiable spectral probe which automatically learns to weigh frequencies that are relevant to a specific task and improves over prior fixed-band filters by capturing continuous mixtures over frequencies (section 2). this enables us to not only outperform the manual filters while using one probe instead of five, but to also identify that high-frequency tasks still benefit from low-frequency information (section 3.1). extending spectral probing to seven tasks in six languages, we trained task-specific filters which outperformed the original, unfiltered embeddings. the resulting spectral profiles furthermore shed light onto how linguistic information at different timescales relates to different task types (section 3.2). they not only match the linguistic intuitions underlying each task, but also enable quantitative comparisons between them. the analysis of the filters' overlap surfaced a clear dichotomy between token and sequence-level tasks, but also highlighted intersecting frequency ranges which contain information relevant across task types. finally, the language-agnostic nature of these spectral profiles highlights future avenues towards more robust task descriptors (section 4).