Improving Span Representation by Efficient Span-Level Attention (2023)

Abstract: High-quality span representations are crucial to natural language processing tasks involving span prediction and classification. Most existing methods derive a span representation by aggregation of token representations within the span. In contrast, we aim to improve span representations by considering span-span interactions as well as more comprehensive span-token interactions. Specifically, we introduce layers of span-level attention on top of a normal token-level transformer encoder. Given that attention between all span pairs results in O(n^4) complexity ( n being the sentence length) and not all span interactions are intuitively meaningful, we restrict the range of spans that a given span could attend to, thereby reducing overall complexity to O(n^3) . We conduct experiments on various span-related tasks and show superior performance of our model surpassing baseline models. Our code is publicly available at https://github.com/jipy0222/Span-Level-Attention .

## introduction many natural language processing tasks involve spans, making it crucial to construct high-quality span representations. in named entity recognition, spans are detected and typed with different labels @xcite ; in coreference resolution, mention spans are located and grouped @xcite @xcite ; in constituency parsing, spans are assigned scores for constituent labels, based on which a parse tree structure is derived @xcite @xcite . most existing methods compute span representations by shallowly aggregating token representations. they either pool over tokens within the span @xcite @xcite , or concatenate the starting and ending tokens @xcite 2021). the limitation of these methods lies in: (i) span representations are dominated by a subset of tokens, resulting in a potential lack of crucial information. (ii) intuitively, span interactions should play an important role in span encoding. for example, meanings of spans, especially constituents, can be composed from their sub-spans and disambiguated by their neighbouring spans. however, such span interactions are completely ignored in these methods. inspired by the utilization of self-attention in transformer @xcite , we introduce span-level self-attention to capture span interactions and improve span representations. however, computing attention scores for all span pairs leads to o(n 4 ) complexity (n for sequence length). in addition, not all span interactions are intuitively meaningful. therefore, we design four different span-level patterns to restrict the range of spans that a given span could attend to: inside-token, containment, adjacency and all-token (fig. 1). each of them allows only o(n) spans for attention, reducing the overall complexity to o(n 3 ). many existing studies also aim at improving span representations. yuan et al. (2022) utilize entity labels to define specific span representations for nested ner. zhou et al. (2022) use syntactic parse trees to enhance span encoding. zhu et al. (2022) improve span representations by stacking multiple span-token attention layers. wang et al. (2022) introduce intra-span attention to enhance span representations by computing attention between each given span and all other spans. compared to existing works, our method offers unique advantages: (i) we design span representations for a wide range of tasks without relying on external information such as labels and parse trees. (ii) we lay more emphasis on span interactions by incorporating span-level attention. (iii) we design span-level attention patterns to capture meaningful span interactions and reduce the overall complexity to an acceptable level, thereby ensuring both effectiveness and efficiency. ## method fig. 2 illustrates the architecture of our model, which we describe from bottom up. ## conclusion we propose to use span-level attention to improve span representations. in order to reduce the o(n 4 ) complexity and encourage more meaningful span interactions, we incorporate different attention patterns to limit the scope of spans that a particular span can attend to. experiments on various tasks validate the efficiency and effectiveness of our method.