qinchuanhui/UDA-QA huggingface dataset:
Question: What benchmark datasets they use?

Evidence:
raw_evidence: Our first task is the recently-introduced Visual Question Answering challenge (VQA) BIBREF22 . The VQA dataset consists of more than 200,000 images paired with human-annotated questions and answers, as in fig:vqa:qualitative-results.
The next set of experiments we consider focuses on GeoQA, a geographical question-answering task first introduced by Krish2013Grounded. This task was originally paired with a visual question answering task much simpler than the one just discussed, and is appealing for a number of reasons. In contrast to the VQA dataset, GeoQA is quite small, containing only 263 examples. Two baselines are available: one using a classical semantic parser backed by a database, and another which induces logical predicates using linear classifiers over both spatial and distributional features. This allows us to evaluate the quality of our model relative to other perceptually grounded logical semantics, as well as strictly logical approaches.

Answers:
VQA and GeoQA
Question: what is the proposed Progressive Dynamic Hurdles method?

Evidence:
raw_evidence: To address this problem we formulated a method to dynamically allocate resources to more promising architectures according to their fitness. This method, which we refer to as progressive dynamic hurdles (PDH), allows models that are consistently performing well to train for more steps. It begins as ordinary tournament selection evolutionary architecture search with early stopping, with each child model training for a relatively small $s_0$ number of steps before being evaluated for fitness. However, after a predetermined number of child models, $m$ , have been evaluated, a hurdle, $h_0$ , is created by calculating the the mean fitness of the current population. For the next $m$ child models produced, models that achieve a fitness greater than $h_0$ after $s_0$ train steps are granted an additional $s_1$ steps of training and then are evaluated again to determine their final fitness. Once another $m$ models have been considered this way, another hurdle, $h_1$ , is constructed by calculating the mean fitness of all members of the current population that were trained for the maximum number of steps. For the next $m$ child models, training and evaluation continues in the same fashion, except models with fitness greater than $m$0 after $m$1 steps of training are granted an additional $m$2 number of train steps, before being evaluated for their final fitness. This process is repeated until a satisfactory number of maximum training steps is reached. Algorithm 1 (Appendix) formalizes how the fitness of an individual model is calculated with hurdles and Algorithm 2 (Appendix) describes tournament selection augmented with progressive dynamic hurdles.

Answers:
allows models that are consistently performing well to train for more steps
Question: What is in the model search space?

Evidence:
raw_evidence: Our search space consists of two stackable cells, one for the model encoder and one for the decoder (see Figure 1 ). Each cell contains NASNet-style blocks, which receive two hidden state inputs and produce new hidden states as outputs; the encoder contains six blocks and the decoder contains eight blocks, so that the Transformer can be represented exactly. The blocks perform separate transformations to each input and then combine the transformation outputs together to produce a single block output; we will refer to the transformations applied to each input as a branch. Our search space contains five branch-level search fields (input, normalization, layer, output dimension and activation), one block-level search field (combiner function) and one cell-level search field (number of cells).

Answers:
Our search space consists of two stackable cells, one for the model encoder and one for the decoder , Each cell contains NASNet-style blocks, which receive two hidden state inputs and produce new hidden states as outputs, Our search space contains five branch-level search fields (input, normalization, layer, output dimension and activation), one block-level search field (combiner function) and one cell-level search field (number of cells).
Question: How does Progressive Dynamic Hurdles work?

Evidence:
raw_evidence: To address this problem we formulated a method to dynamically allocate resources to more promising architectures according to their fitness. This method, which we refer to as progressive dynamic hurdles (PDH), allows models that are consistently performing well to train for more steps. It begins as ordinary tournament selection evolutionary architecture search with early stopping, with each child model training for a relatively small $s_0$ number of steps before being evaluated for fitness. However, after a predetermined number of child models, $m$ , have been evaluated, a hurdle, $h_0$ , is created by calculating the the mean fitness of the current population. For the next $m$ child models produced, models that achieve a fitness greater than $h_0$ after $s_0$ train steps are granted an additional $s_1$ steps of training and then are evaluated again to determine their final fitness. Once another $m$ models have been considered this way, another hurdle, $h_1$ , is constructed by calculating the mean fitness of all members of the current population that were trained for the maximum number of steps. For the next $m$ child models, training and evaluation continues in the same fashion, except models with fitness greater than $m$0 after $m$1 steps of training are granted an additional $m$2 number of train steps, before being evaluated for their final fitness. This process is repeated until a satisfactory number of maximum training steps is reached. Algorithm 1 (Appendix) formalizes how the fitness of an individual model is calculated with hurdles and Algorithm 2 (Appendix) describes tournament selection augmented with progressive dynamic hurdles.

Answers:
It begins as ordinary tournament selection evolutionary architecture search with early stopping, with each child model training for a relatively small $s_0$ number of steps before being evaluated for fitness. However, after a predetermined number of child models, $m$ , have been evaluated, a hurdle, $h_0$ , is created by calculating the the mean fitness of the current population. For the next $m$ child models produced, models that achieve a fitness greater than $h_0$ after $s_0$ train steps are granted an additional $s_1$ steps of training and then are evaluated again to determine their final fitness. Once another $m$ models have been considered this way, another hurdle, $h_1$ , is constructed by calculating the mean fitness of all members of the current population that were trained for the maximum number of steps. For the next $m$ child models, training and evaluation continues in the same fashion, except models with fitness greater than $m$0 after $m$1 steps of training are granted an additional $m$2 number of train steps, before being evaluated for their final fitness. This process is repeated until a satisfactory number of maximum training steps is reached.
Question: Do they beat current state-of-the-art on SICK?

Evidence:
raw_evidence: To show the effectiveness of our approach, we show results on the SICK dataset BIBREF1, a common benchmark for logic-based NLI, and find MonaLog to be competitive with more complicated logic-based approaches (many of which require full semantic parsing and more complex logical machinery). We also introduce a supplementary version of SICK that corrects several common annotation mistakes (e.g., asymmetrical inference annotations) based on previous work by kalouli2017entail,kalouli2018. Positive results on both these datasets show the ability of lightweight monotonicity models to handle many of the inferences found in current NLI datasets, hence putting a more reliable lower-bound on what results the simplest logical approach is capable of achieving on this benchmark.

Answers:
No
Question: How do they combine MonaLog with BERT?

Evidence:
raw_evidence: In this work, we introduce a new logical inference engine called MonaLog, which is based on natural logic and work on monotonicity stemming from vanBenthemEssays86. In contrast to the logical approaches cited above, our starting point is different in that we begin with the following two questions: 1) what is the simplest logical system that one can come up with to solve empirical NLI problems (i.e., the system with minimal amounts of primitives and background knowledge)?; and 2) what is the lower-bound performance of such a model? Like other approaches to natural logic BIBREF15, BIBREF16, our model works by reasoning over surface forms (as opposed to translating to symbolic representations) using a small inventory of monotonicity facts about quantifiers, lexical items and token-level polarity BIBREF17; proofs in the calculus are hence fully interpretable and expressible in ordinary language. Unlike existing work on natural logic, however, our model avoids the need for having expensive alignment and search sub-procedures BIBREF18, BIBREF19, and relies on a much smaller set of background knowledge and primitive relations than MacCartneyManning.
Since our logic operates over surface forms, it is straightforward to hybridize our models. We investigate using MonaLog in combination with the language model BERT BIBREF20, including for compositional data augmentation, i.e, re-generating entailed versions of examples in our training sets. To our knowledge, our approach is the first attempt to use monotonicity for data augmentation, and we show that such augmentation can generate high-quality training data with which models like BERT can improve performance.
We perform two experiments to test MonaLog. We first use MonaLog to solve the problems in a commonly used natural language inference dataset, SICK BIBREF1, comparing our results with previous systems. Second, we test the quality of the data generated by MonaLog. To do this, we generate more training data (sentence pairs) from the SICK training data using our system, and performe fine-tuning on BERT BIBREF20, a language model based on the transformer architecture BIBREF23, with the expanded dataset. In all experiments, we use the Base, Uncased model of BERT.

Answers:
They use Monalog for data-augmentation to fine-tune BERT on this task
Question: How do they select monotonicity facts?

Evidence:
raw_evidence: MonaLog utilizes two auxiliary sets. First, a knowledge base ${K}$ that stores the world knowledge needed for inference, e.g., semanticist $\le $ linguist and swim $\le $ move, which captures the facts that $[\![\mbox{\em semanticist}]\!]$ denotes a subset of $[\![\mbox{\em linguist}]\!]$, and that $[\![\mbox{\em swim}]\!]$ denotes a subset of $[\![\mbox{\em move}]\!]$, respectively. Such world knowledge can be created manually for the problem at hand, or derived easily from existing resources such as WordNet BIBREF22. Note that we do not blindly add all relations from WordNet to our knowledge base, since this would hinge heavily on word sense disambiguation (we need to know whether the “bank” is a financial institution or a river bank to extract its relations correctly). In the current implementation, we avoid this by adding x $\le $ y or x $\perp $ y relations only if both x and y are words in the premise-hypothesis pair. Additionally, some relations that involve quantifiers and prepositions need to be hard-coded, since WordNet does not include them: every $=$ all $=$ each $\le $ most $\le $ many $\le $ a few $=$ several $\le $ some $=$ a; the $\le $ some $=$ a; on $\perp $ off; up $\perp $ down; etc.

Answers:
They derive it from Wordnet
Question: How does the model recognize entities and their relation to answers at inference time when answers are not accessible?

Evidence:
raw_evidence: In this paper we presented the Question Dependent Recurrent Entity Network, used for reasoning and reading comprehension tasks. This model uses a particular RNN cell in order to store just relevant information about the given question. In this way, in combination with the original Recurrent Entity Network (keys and memory), we improved the State-of-The-Art in the bAbI 1k task and achieved promising results in the Reading comprehension task on the CNN & Daily news dataset. However, we believe that there are still margins for improving the behavior for the proposed cell. Indeed, the cell has not enough expressive power to make a selective activation among different memory blocks (notice in Figure 2 (a) the gates open for all the memories). This does not seem to be a serious problem since we actually outperform other models, but it could be the key to finally pass all the bAbI tasks.
Dynamic Memory stores information of entities present in $T$ . This module is very similar to a Gated Recurrent Unit (GRU) BIBREF20 with a hidden state divided into blocks. Moreover, each block ideally represents an entity (i.e. person, location etc.), and it stores relevant facts about it. Each block $i$ is made of a hidden state $h_i\in \mathbb {R}^d$ and a key $k_i\in \mathbb {R}^d$ , where $d$ is the embedding size. The Dynamic Memory module is made of a set of blocks, which can be represent with a set of hidden states $\lbrace h_1,\dots ,h_z \rbrace $ and their correspondent set of keys $\lbrace k_1,\dots ,k_z \rbrace $ . The equation used to update a generic block $i$ are the following: $ g_i^{(t)} =& \sigma (s_t^T h_i^{(t-1)} + s_t^T k_i^{(t-1)} + s_t^T q ) &\text{(Gating Function)}&\\ \hat{h}_i^{(t)} =& \phi (U h_i^{(t-1)} + V k_i^{(t-1)} + W s_t ) &\text{(Candidate Memory)}&\\ h_i^{(t)} =& h_i^{(t-1)} + g_i^{(t)} \odot \hat{h}_i^{(t)} &\text{(New Memory)}&\\ h_i^{(t)} =& h_i^{(t)}/\Vert h_i^{(t)} \Vert &\text{(Reset Memory)}&\\ $
The addition of the $s_t^T q$ term in the gating function is our main contribution. We add such term with the assumption that the question can be useful to focus the attention of the model while analyzing the input sentences.

Answers:
gating function, Dynamic Memory
Question: What other solutions do they compare to?

Evidence:
raw_evidence: We evaluate the performance of our proposed method based on two evaluation criteria EM and F1 for the MC tasks. We compare our model with other strong competitive methods on the SQuAD leaderboard and TriviaQA leaderboard.

Answers:
 strong competitive methods on the SQuAD leaderboard and TriviaQA leaderboard
Question: How does the gatint mechanism combine word and character information?

Evidence:
raw_evidence: Using the fine-grained gating mechanism conditioned on the lexical features, we can accurately control the information flows between word-level and char-level. Intuitively, the formulation is as follows: INLINEFORM0
where INLINEFORM0 is the element-wise multiplication operator. when the gate has high value, more information flows from the word-level representation; otherwise, char-level will take the dominating place. It is practical in real scenarios. For example, for unfamiliar noun entities, the gates tend to bias towards char-level representation in order to care richer morphological structure. Besides, we not only utilize the lexical properties as the gating feature, we also concatenate them as a supplement of lexical information. Therefore, the final representation of words are computed as follows: INLINEFORM1

Answers:
when the gate has high value, more information flows from the word-level representation; otherwise, char-level will take the dominating place,  for unfamiliar noun entities, the gates tend to bias towards char-level representation in order to care richer morphological structure
Question: Which dataset do they use?

Evidence:
raw_evidence: For the pre-translation, we used a PBMT system. In order to analyze the influence of the quality of the PBMT system, we use two different systems, a baseline system and a system with advanced models. The systems were trained on all parallel data available for the WMT 2016. The news commentary corpus, the European parliament proceedings and the common crawl corpus sum up to 3.7M sentences and around 90M words.

Answers:
parallel data available for the WMT 2016
Question: How is the PBMT system trained?

Evidence:
raw_evidence: Both systems were optimized on the tst2014 using Minimum error rate training BIBREF20 . A detailed description of the systems can be found in BIBREF21 .

Answers:
systems were optimized on the tst2014 using Minimum error rate training BIBREF20
Question: Which NMT architecture do they use?

Evidence:
raw_evidence: The neural machine translation was trained using Nematus. For the NMT system as well as for the PreMT system, we used the default configuration. In order to limit the vocabulary size, we use BPE as described in BIBREF0 with 40K operations. We run the NMT system for 420K iterations and stored a model every 30K iterations. We selected the model that performed best on the development data. For the ensemble system we took the last four models. We did not perform an additional fine-tuning.

Answers:
trained using Nematus, default configuration
Question: Do they train the NMT model on PBMT outputs?

Evidence:
raw_evidence: In a second step, we will train a neural monolingual translation system, that translates from the output of the PBMT system INLINEFORM0 to a better target sentence INLINEFORM1 .

Answers:
Yes
Question: Is the corpus annotated?

Evidence:

Answers:
No
Question: How is the corpus normalized?

Evidence:
raw_evidence: KTC is composed of 31 educational textbooks published from 2011 to 2018 in various topics by the MoE. We received the material from the MoE partly in different versions of Microsoft Word and partly in Adobe InDesign formats. In the first step, we categorized each textbook based on the topics and chapters. As the original texts were not in Unicode, we converted the content to Unicode. This step was followed by a pre-processing stage where the texts were normalized by replacing zero-width-non-joiner (ZWNJ) BIBREF2 and manually verifying the orthography based on the reference orthography of the Kurdistan Region of Iraq. In the normalization process, we did not remove punctuation and special characters so that the corpus can be easily adapted our current task and also to future tasks where the integrity of the text may be required.

Answers:
by replacing zero-width-non-joiner (ZWNJ) BIBREF2 and manually verifying the orthography
Question: Is the corpus annotated with a phonetic transcription?

Evidence:

Answers:
No
Question: Is the corpus annotated with Part-of-Speech tags?

Evidence:

Answers:
No
Question: what evaluation methods are discussed?

Evidence:
raw_evidence: The most common approach is to treat the task as a document-level classification problem. Given a set of evaluation documents, each having a known correct label from a closed set of labels (often referred to as the “gold-standard”), and a predicted label for each document from the same set, the document-level accuracy is the proportion of documents that are correctly labeled over the entire evaluation collection. This is the most frequently reported metric and conveys the same information as the error rate, which is simply the proportion of documents that are incorrectly labeled (i.e. INLINEFORM0 ).
Authors sometimes provide a per-language breakdown of results. There are two distinct ways in which results are generally summarized per-language: (1) precision, in which documents are grouped according to their predicted language; and (2) recall, in which documents are grouped according to what language they are actually written in. Earlier work has tended to only provide a breakdown based on the correct label (i.e. only reporting per-language recall). This gives us a sense of how likely a document in any given language is to be classified correctly, but does not give an indication of how likely a prediction for a given language is of being correct. Under the monolingual assumption (i.e. each document is written in exactly one language), this is not too much of a problem, as a false negative for one language must also be a false positive for another language, so precision and recall are closely linked. Nonetheless, authors have recently tended to explicitly provide both precision and recall for clarity. It is also common practice to report an F-score INLINEFORM0 , which is the harmonic mean of precision and recall. The F-score (also sometimes called F1-score or F-measure) was developed in IR to measure the effectiveness of retrieval with respect to a user who attaches different relative importance to precision and recall BIBREF376 . When used as an evaluation metric for classification tasks, it is common to place equal weight on precision and recall (hence “F1”-score, in reference to the INLINEFORM1 hyper-parameter, which equally weights precision and recall when INLINEFORM2 ).

Answers:
document-level accuracy, precision, recall, F-score
Question: How is "hirability" defined?

Evidence:
raw_evidence: We have decided to focus on only one specific type of job: sales positions. After filtering based on specific job titles from the ROME Database, a list of positions was selected and verified by the authors and an expert from the Human Resources (HR). Finally, in a collaboration with an HR industry actor, we have obtained a dataset of French video interviews comprising more than 475 positions and 7938 candidates. As they watch candidates' videos, recruiters can like, dislike, shortlist candidates, evaluate them on predefined criteria, or write comments. To simplify the task, we set up a binary classification: candidates who have been liked or shortlisted are considered part of the hirable class and others part of the not hirable class. If multiple annotators have annotated the same candidates, we proceed with a majority vote. In case of a draw, the candidate is considered hirable. It is important to note that the videos are quite different from what could be produced in a laboratory setup. Videos can be recorded from a webcam, a smartphone or a tablet., meaning noisy environments and low quality equipment are par for the course. Due to these real conditions, feature extraction may fail for a single modality during a candidate's entire answer. One example is the detection of action units when the image has lighting problems. We decided to use all samples available in each modality separately. Some statistics about the dataset are available in Table TABREF33 . Although the candidates agreed to the use of their interviews, the dataset will not be released to public outside of the scope of this study due to the videos being personal data subject to high privacy constraints.

Answers:
candidates who have been liked or shortlisted are considered part of the hirable class