Teacher:In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
Teacher: Now, understand the problem? Solve this instance: The aim of this section is to validate the applicability of our theoretical resultswhich state that self-attention can perform convolutionand to examine whether self-attention layers in practice do actually learn to operate like convolutional layers, when being trained on standard image classification tasks. In particular, we study the relationship between self-attention and convolution with quadratic and learned relative positional encodings. We find that for both cases, the attention probabilities learned tend to respect the conditions of Lemma UNKREF15, corroborating our hypothesis. Nevertheless, to validate that our model learns a meaningful classifier we compare it to the standard ResNet18 BIBREF14 on the CIFAR-10 dataset BIBREF15.
Question: What numerical experiments they perform?
Student:
attention probabilities learned tend to respect the conditions of Lemma UNKREF15, corroborating our hypothesis validate that our model learns a meaningful classifier we compare it to the standard ResNet18