In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.

[EX Q]: We evaluated our models on 3 different datasets:

CSAT dataset for CSAT prediction, consisting of spoken transcripts (automatic via ASR).

20 newsgroups for topic identification task, consisting of written text;

Fisher Phase 1 corpus for topic identification task, consisting of spoken transcripts (manual);
Question: What datasets did they use for evaluation?
[EX A]: CSAT dataset 20 newsgroups Fisher Phase 1 corpus

[EX Q]: Contextualized word embeddings, sentence embeddings, such as deep contextualized word representations BIBREF20 , BERT BIBREF22 , encode the complex characteristics and meanings of words in various context by jointly training a bidirectional language model. The second method is to use sentence embeddings, BERT. It is used to a generate single 786-dimensional sentence embedding from 10k-dimensional one-hot vector or distribution over previous words and then merge into a single context vector with two different merging methods. Using this approach, we can obtain a more dense, informative, fixed-length vectors to encode conversational-context information, $e^k_{context}$ to be used in next $k$ -th utterance prediction. We use contextual gating mechanism in our decoder network to combine the conversational-context embeddings with speech and word embeddings effectively. Our gating is contextual in the sense that multiple embeddings compute a gate value that is dependent on the context of multiple utterances that occur in a conversation. Let $e_w = e_w(y_{u-1})$ be our previous word embedding for a word $y_{u-1}$ , and let $e_s = e_s(x^k_{1:T})$ be a speech embedding for the acoustic features of current $k$ -th utterance $x^k_{1:T}$ and $e_c = e_c(s_{k-1-n:k-1})$ be our conversational-context embedding for $n$ -number of preceding utterances ${s_{k-1-n:k-1}}$ . Then using a gating mechanism:

$$g = \sigma (e_c, e_w, e_s)$$ (Eq. 15)

where $\sigma $ is a 1 hidden layer DNN with $ exttt {sigmoid}$ activation, the gated embedding $e$ is calcuated as

$$e = g \odot (e_c, e_w, e_s) \ h = ext{LSTM}(e)$$ (Eq. 16)

and fed into the LSTM decoder hidden layer. The output of the decoder $h$ is then combined with conversational-context embedding $e_c$ again with a gating mechanism,

$$g = \sigma (e_C, h) \ \hat{h} = g \odot (e_c, h)$$ (Eq. 17)

Then the next hidden layer takes these gated activations, $\hat{h}$ , and so on.
Question: How are sentence embeddings incorporated into the speech recognition system?
[EX A]: BERT generates sentence embeddings that represent words in context. These sentence embeddings are merged into a single conversational-context vector that is used to calculate a gated embedding and is later combined with the output of the decoder h to provide the gated activations for the next hidden layer.

[EX Q]: For POS tagging, we observe error reductions of respectively 0.71% for GSD, 0.81% for Sequoia, 0.7% for Spoken and 0.28% for ParTUT. For parsing, we observe error reductions in LAS of 2.96% for GSD, 3.33% for Sequoia, 1.70% for Spoken and 1.65% for ParTUT. On the XNLI benchmark, CamemBERT obtains improved performance over multilingual language models on the TRANSLATE-TRAIN setting (81.2 vs. 80.2 for XLM) while using less than half the parameters (110M vs. 250M). However, its performance still lags behind models trained on the original English training set in the TRANSLATE-TEST setting, 81.2 vs. 82.91 for RoBERTa. Both improvements result in a 2.36 point increase in the F1 score with respect to the best SEM architecture (BiLSTM-CRF), giving CamemBERT the state of the art for NER on the FTB.
Question: How much better was results of CamemBERT than previous results on these tasks?
[EX A]:
2.36 point increase in the F1 score with respect to the best SEM architecture on the TRANSLATE-TRAIN setting (81.2 vs. 80.2 for XLM) lags behind models trained on the original English training set in the TRANSLATE-TEST setting, 81.2 vs. 82.91 for RoBERTa For POS tagging, we observe error reductions of respectively 0.71% for GSD, 0.81% for Sequoia, 0.7% for Spoken and 0.28% for ParTUT For parsing, we observe error reductions in LAS of 2.96% for GSD, 3.33% for Sequoia, 1.70% for Spoken and 1.65% for ParTUT