Q: In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
In order to represent individual sentences, we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it. We also use interval segment embeddings to distinguish multiple sentences within a document. This way, document representations are learned hierarchically where lower Transformer layers represent adjacent sentences, while higher layers, in combination with self-attention, represent multi-sentence discourse. Position embeddings in the original Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings that are initialized randomly and fine-tuned with other parameters in the encoder.
Question: What is novel about their document-level encoder?
A:
Bert model have a maximum length of 512; we overcome this limitation by adding more position embeddings we insert external [cls] tokens at the start of each sentence, and each [cls] symbol collects features for the sentence preceding it document representations are learned hierarchically