qinchuanhui/UDA-QA huggingface dataset:
Question: What is English mixed with in the TRAC dataset?

Evidence:
raw_evidence: In future, we will explore other methods to increase the understanding of deep learning models on group targeted text, although the categories are well defined we will look after if we further fine-tune the categories with more data. In the future, we are planning to pay attention on a generalized language model for code-mixed texts which can also handle Hindi-code-mixed and other multi-lingual code-mixed datasets (i.e., trying to reduce the dependencies on language-specific code-mixed resources).
The block diagram of the proposed system is shown in Figure FIGREF22. The proposed system does not use any data augmentation techniques like BIBREF14, which is the top performer in TRAC (in English code-mixed Facebook data). This means the performance achieved by our system totally depends on the training dataset provided by TRAC. This also proves the effectiveness of our approach. Our system outperforms all the previous state of the art approaches used for aggression identification on English code-mixed TRAC data, while being trained only from Facebook comments the system outperforms other approaches on the additional Twitter test set. The remaining part of this paper is organized as follows: Section SECREF2 is an overview of related work. Section SECREF3 presents the methodology and algorithmic details. Section SECREF4 discusses the experimental evaluation of the system, and Section SECREF5 concludes this paper.
The fine-grained definition of the aggressiveness/aggression identification is provided by the organizers of TRAC-2018 BIBREF0, BIBREF2. They have classified the aggressiveness into three labels (Overtly aggressive(OAG), Covertly aggressive(CAG), Non-aggressive(NAG)). The detailed description for each of the three labels is described as follows:
Overtly Aggressive(OAG) - This type of aggression shows direct verbal attack pointing to the particular individual or group. For example, "Well said sonu..you have courage to stand against dadagiri of Muslims".
Covertly Aggressive(CAG) - This type of aggression the attack is not direct but hidden, subtle and more indirect while being stated politely most of the times. For example, "Dear India, stop playing with the emotions of your people for votes."
Non-Aggressive(NAG) - Generally these type of text lack any kind of aggression it is basically used to state facts, wishing on occasions and polite and supportive.

Answers:
Hindi
Question: How have the differences in communication styles between Twitter and Facebook increase the complexity of the problem?

Evidence:
raw_evidence: Most of the above-discussed systems either shows high performance on (a) Twitter dataset or (b) Facebook dataset (given in the TRAC-2018), but not on both English code-mixed datasets. This may be due to the text style or level of complexities of both datasets. So, we concentrated to develop a robust system for English code-mixed texts, and uni-lingual texts, which can also handle different writing styles. Our approach is based on three main ideas:

Answers:
Systems do not perform well both in Facebook and Twitter texts
Question: What data/studies do the authors provide to support the assertion that the majority of aggressive conversations contain code-mixed languages?

Evidence:
raw_evidence: The informal setting/environment of social media often encourage multilingual speakers to switch back and forth between languages when speaking or writing. These all resulted in code-mixing and code-switching. Code-mixing refers to the use of linguistic units from different languages in a single utterance or sentence, whereas code-switching refers to the co-occurrence of speech extracts belonging to two different grammatical systemsBIBREF3. This language interchange makes the grammar more complex and thus it becomes tough to handle it by traditional algorithms. Thus the presence of high percentage of code-mixed content in social media text has increased the complexity of the aggression detection task. For example, the dataset provided by the organizers of TRAC-2018 BIBREF0, BIBREF2 is actually a code-mixed dataset.

Answers:
None
Question: What is the baseline?

Evidence:
raw_evidence: Emotions have been used in many natural language processing tasks and they showed their efficiency BIBREF35. We aim at investigating their efficiency to detect false information. In addition to EIN, we created a model (Emotion-based Model) that uses emotional features only and compare it to two baselines. Our aim is to investigate if the emotional features independently can detect false news. The two baselines of this model are Majority Class baseline (MC) and the Random selection baseline (RAN).

Answers:
Majority Class baseline (MC) , Random selection baseline (RAN)
Question: What datasets did they use?

Evidence:
raw_evidence: Evaluation Framework ::: Datasets ::: News Articles
Our dataset source of news articles is described in BIBREF2. This dataset was built from two different sources, for the trusted news (real news) they sampled news articles from the English Gigaword corpus. For the false news, they collected articles from seven different unreliable news sites. These news articles include satires, hoaxes, and propagandas but not clickbaits. Since we are interested also in analyzing clickbaits, we slice a sample from an available clickbait dataset BIBREF33 that was originally collected from two sources: Wikinews articles' headlines and other online sites that are known to publish clickbaits. The satire, hoax, and propaganda news articles are considerably long (some of them reach the length of 5,000 words). This length could affect the quality of the analysis as we mentioned before. We focus on analyzing the initial part of the article. Our intuition is that it is where emotion-bearing words will be more frequent. Therefore, we shorten long news articles into a maximum length of N words (N=300). We choose the value of N based on the length of the shortest articles. Moreover, we process the dataset by removing very short articles, redundant articles or articles that do not have a textual content.
With the complicated political and economic situations in many countries, some agendas are publishing suspicious news to affect public opinions regarding specific issues BIBREF0. The spreading of this phenomenon is increasing recently with the large usage of social media and online news sources. Many anonymous accounts in social media platforms start to appear, as well as new online news agencies without presenting a clear identity of the owner. Twitter has recently detected a campaign organized by agencies from two different countries to affect the results of the last U.S. presidential elections of 2016. The initial disclosures by Twitter have included 3,841 accounts. A similar attempt was done by Facebook, as they detected coordinated efforts to influence U.S. politics ahead of the 2018 midterm elections.
For this dataset, we rely on a list of several Twitter accounts for each type of false information from BIBREF6. This list was created based on public resources that annotated suspicious Twitter accounts. The authors in BIBREF6 have built a dataset by collecting tweets from these accounts and they made it available. For the real news, we merge this list with another 32 Twitter accounts from BIBREF34. In this work we could not use the previous dataset and we decide to collect tweets again. For each of these accounts, we collected the last M tweets posted (M=1000). By investigating these accounts manually, we found that many tweets just contain links without textual news. Therefore, to ensure of the quality of the crawled data, we chose a high value for M (also to have enough data). After the collecting process, we processed these tweets by removing duplicated, very short tweets, and tweets without textual content. Table TABREF35 shows a summary for both datasets.

Answers:
News Articles, Twitter
Question: What scoring function does the model use to score triples?

Evidence:
raw_evidence: Let $\mathcal {E}$ denote the set of entities and $\mathcal {R}$ the set of relation types. For each triple $(h, r, t)$ , where $h, t \in \mathcal {E}$ and $r \in \mathcal {R}$ , the STransE model defines a score function $f_r(h, t)$ of its implausibility. Our goal is to choose $f$ such that the score $f_r(h,t)$ of a plausible triple $(h,r,t)$ is smaller than the score $f_{r^{\prime }}(h^{\prime },t^{\prime })$ of an implausible triple $\mathcal {R}$0 . We define the STransE score function $\mathcal {R}$1 as follows:
$ f_r(h, t) & = & \Vert \textbf {W}_{r,1}\textbf {h} + \textbf {r} - \textbf {W}_{r,2}\textbf {t}\Vert _{\ell _{1/2}} $
using either the $\ell _1$ or the $\ell _2$ -norm (the choice is made using validation data; in our experiments we found that the $\ell _1$ norm gave slightly better results). To learn the vectors and matrices we minimize the following margin-based objective function: $ \mathcal {L} & = & \sum _{\begin{array}{c}(h,r,t) \in \mathcal {G} \\ (h^{\prime },r,t^{\prime }) \in \mathcal {G}^{\prime }_{(h, r, t)}\end{array}} [\gamma + f_r(h, t) - f_r(h^{\prime }, t^{\prime })]_+ $

Answers:
$ f_r(h, t) & = & \Vert \textbf {W}_{r,1}\textbf {h} + \textbf {r} - \textbf {W}_{r,2}\textbf {t}\Vert _{\ell _{1/2}} $
Question: What datasets are used to evaluate the model?

Evidence:
raw_evidence: As we show below, STransE performs better than the SE and TransE models and other state-of-the-art link prediction models on two standard link prediction datasets WN18 and FB15k, so it can serve as a new baseline for KB completion. We expect that the STransE will also be able to serve as the basis for extended models that exploit a wider variety of information sources, just as TransE does.

Answers:
WN18, FB15k
Question: How better are results for pmra algorithm  than Doc2Vec in human evaluation? 

Evidence:
raw_evidence: Regarding the evaluation itself, based on the three-modality scale (bad, partial or full relevance), models are clearly not equivalents (Figure FIGREF26). The D2V model has been rated 80 times as "bad relevance" while the pmra returned only 24 times badly relevant documents. By looking at the results ranking, the mean position for D2V was 14.09 (ranging from 13.98 for JPL to 14.20 for EL). Regarding the pmra, this average position was equal to 6.89 (ranging from 6.47 for EL to 7.23 for SJD).

Answers:
The D2V model has been rated 80 times as "bad relevance" while the pmra returned only 24 times badly relevant documents.
Question: What Doc2Vec architectures other than PV-DBOW have been tried?

Evidence:
raw_evidence: Among all available parameters to tune the D2V algorithm released by Gensim, six of them were selected for optimisation BIBREF14. The window_size parameter affects the size of the sliding window used to parse texts. The alpha parameter represents the learning rate of the network. The sample setting allows the model to reduce the importance given to high-frequency words. The dm parameter defines the training used architecture (PV-DM or PV-DBOW). The hs option defines whether hierarchical softmax or negative sampling is used during the training. Finally, the vector_size parameter affects the number of dimensions composing the resulting vector.

Answers:
PV-DM
Question: What four evaluation tasks are defined to determine what influences proximity?

Evidence:
raw_evidence: The goal here being to assess if D2V could effectively replace the related-document function on PubMed, five different document similarity evaluations were designed as seen on figure FIGREF9. These tasks were designed to cover every similarities, from the most general (the context) to the character-level similarity.
Methods ::: Evaluation ::: String length
To assess whether a similar length could lead to convergence of two documents, the size of the query document $D_{x}$ has been compared with the top-close document $C_{x}$ for 10,000 document randomly selected from the TeS after some pre-processing steps (stopwords and spaces were removed from both documents).
Methods ::: Evaluation ::: Words co-occurrences
A matrix of words co-occurrence was constructed on the total corpus from PubMed. Briefly, each document was lowered and tokenized. A matrix was filled with the number of times that two words co-occur in a single document. Then, for 5,000 documents $D_{x}$ from the TeS, all models were queried for the top-close document $C_{x}$. All possible combinations between all words $WD_{x} \in D_{x}$ and all words $WC_{x} \in C_{x}$ (excluding stopwords) were extracted, 500 couples were randomly selected and the number of times each of them was co-occurring was extracted from the matrix. The average value of this list was calculated, reflecting the proximity between D and C regarding their words content. This score was also calculated between each $D_{x}$ and the top-close document $C_{x}$ returned by the pmra algorithm.
Methods ::: Evaluation ::: Stems co-occurrences
The evaluation task explained above was also applied on 10,000 stemmed texts (using the Gensim’s PorterStemmer to only keep word’s roots). The influence of the conjugation form or other suffixes can be assessed.
Methods ::: Evaluation ::: MeSH similarity
It is possible to compare the ability of both pmra and D2V to bring closer articles which were indexed with common labels. To do so, 5,000 documents $D_{x}$ randomly selected from the TeS were sent to both pmra and D2V architectures, and the top-five closer articles $C_{x}$ were extracted. The following rules were then applied to each MeSH found associated with $D_{x}$ for each document $C_{x_i}$ : add 1 to the score if this MeSH term is found in both $D_{x}$ and $C_{x_i}$, add 3 if this MeSH is defined as major topic and add 1 for each qualifier in common between $D_{x}$ and Cxi regarding this particular MeSH term. Then, the mean of these five scores was calculated for both pmra and D2V.

Answers:
String length, Words co-occurrences, Stems co-occurrences, MeSH similarity
Question: What six parameters were optimized with grid search?

Evidence:
raw_evidence: Among all available parameters to tune the D2V algorithm released by Gensim, six of them were selected for optimisation BIBREF14. The window_size parameter affects the size of the sliding window used to parse texts. The alpha parameter represents the learning rate of the network. The sample setting allows the model to reduce the importance given to high-frequency words. The dm parameter defines the training used architecture (PV-DM or PV-DBOW). The hs option defines whether hierarchical softmax or negative sampling is used during the training. Finally, the vector_size parameter affects the number of dimensions composing the resulting vector.

Answers:
window_size, alpha, sample, dm, hs, vector_size
Question: What are the differences with previous applications of neural networks for this task?

Evidence:
raw_evidence: One common point in all the approaches yet has been the use of only textual features available in the dataset. Our model not only incorporates textual features, modeled using BiLSTM and augmented with an attention mechanism, but also considers related images for the task.

Answers:
This approach considers related images
Question: Is infinite-length sequence generation a result of training with maximum likelihood?

Evidence:
raw_evidence: We extended the notion of consistency of a recurrent language model put forward by BIBREF16 to incorporate a decoding algorithm, and used it to analyze the discrepancy between a model and the distribution induced by a decoding algorithm. We proved that incomplete decoding is inconsistent, and proposed two methods to prevent this: consistent decoding and the self-terminating recurrent language model. Using a sequence completion task, we confirmed that empirical inconsistency occurs in practice, and that each method prevents inconsistency while maintaining the quality of generated sequences. We suspect the absence of decoding in maximum likelihood estimation as a cause behind this inconsistency, and suggest investigating sequence-level learning as an alternative in the future.
Inconsistency may arise from the lack of decoding in solving this optimization problem. Maximum likelihood learning fits the model $p_{\theta }$ using the data distribution, whereas a decoded sequence from the trained model follows the distribution $q_{\mathcal {F}}$ induced by a decoding algorithm. Based on this discrepancy, we make a strong conjecture: we cannot be guaranteed to obtain a good consistent sequence generator using maximum likelihood learning and greedy decoding. Sequence-level learning, however, uses a decoding algorithm during training BIBREF25, BIBREF26. We hypothesize that sequence-level learning can result in a good sequence generator that is consistent with respect to incomplete decoding.

Answers:
There are is a strong conjecture that it might be the reason but it is not proven.
Question: What metrics are used in challenge?

Evidence:
raw_evidence: For evaluation, the Visual Dialog task employs four metrics. NDCG is the primary metric of the Visual Dialog Challenge which considers multiple similar answers as correct ones. The other three are MRR, recall@k, and mean rank where they only consider the rank of a single answer. Our experiments show the scores of NDCG and non-NDCG metrics from our image-only and joint models have a trade-off relationship due to their different ability (as shown in Sec.SECREF41) in completing Visual Dialog tasks: the image-only model has a high NDCG and low non-NDCG values while the joint model has a low NDCG and high non-NDCG values.

Answers:
NDCG, MRR, recall@k, mean rank
Question: What model was winner of the Visual Dialog challenge 2018?

Evidence:
raw_evidence: For the evaluation on the test-standard dataset of VisDial v1.0, we try 6 image-only model ensemble and 6 consensus dropout fusion model ensemble. As shown in Table TABREF48, our two models show competitive results compared to the state-of-the-art on the Visual Dialog challenge 2018 (DL-61 was the winner of the Visual Dialog challenge 2018). Specifically, our image-only model shows much higher NDCG score (60.16). On the other hand, our consensus dropout fusion model shows more balanced results over all metrics while still outperforming on most evaluation metrics (NDCG, MRR, R@1, and R@5). Compared to results of the Visual Dialog challenge 2019, our models also show strong results. Although ReDAN+ BIBREF26 and MReaL–BDAI show higher NDCG scores, our consensus dropout fusion model shows more balanced results over metrics while still having a competitive NDCG score compared to DAN BIBREF25, with rank 3 based on NDCG metric and high balance rank based on metric average.

Answers:
DL-61
Question: Which method for integration peforms better ensemble or consensus dropout fusion with shared parameters?

Evidence:
raw_evidence: As shown in Table TABREF46, consensus dropout fusion improves the score of NDCG by around 1.0 from the score of the joint model while still yielding comparable scores for other metrics. Unlike ensemble way, consensus dropout fusion does not require much increase in the number of model parameters.
As also shown in Table TABREF46, the ensemble model seems to take the best results from each model. Specifically, the NDCG score of the ensemble model is comparable to that of the image-only model and the scores of other metrics are comparable to those of the image-history joint model. From this experiment, we can confirm that the two models are in complementary relation.

Answers:
ensemble model
Question: How big is dataset for this challenge?

Evidence:
raw_evidence: We use the VisDial v1.0 BIBREF0 dataset to train our models, where one example has an image with its caption, 9 question-answer pairs, and follow-up questions and candidate answers for each round. At round $r$, the caption and the previous question-answer pairs become conversational context. The whole dataset is split into 123,287/2,000/8,000 images for train/validation/test, respectively. Unlike the images in the train and validation sets, the images in the test set have only one follow-up question and candidate answers and their corresponding conversational context.

Answers:
133,287 images
Question: What open relation extraction tasks did they experiment on?

Evidence:
raw_evidence: We first measure the utility of various components in Logician to select the optimal model, and then compare this model to the state-of-the-art methods in four types of information extraction tasks: verb/preposition-based relation, nominal attribute, descriptive phrase and hyponymy relation. The SAOKE data set is split into training set, validating set and testing set with ratios of 80%, 10%, 10%, respectively. For all algorithms involved in the experiments, the training set can be used to train the model, the validating set can be used to select an optimal model, and the testing set is used to evaluate the performance.

Answers:
verb/preposition-based relation, nominal attribute, descriptive phrase and hyponymy relation.
Question: How is Logician different from traditional seq2seq models?

Evidence:
raw_evidence: Logician is a supervised end-to-end neural learning algorithm which transforms natural language sentences into facts in the SAOKE form. Logician is trained under the attention-based sequence-to-sequence paradigm, with three mechanisms: restricted copy mechanism to ensure literally honestness, coverage mechanism to alleviate the under extraction and over extraction problem, and gated dependency attention mechanism to incorporate dependency information. Experimental results on four types of open information extraction tasks reveal the superiority of the Logician algorithm.

Answers:
restricted copy mechanism to ensure literally honestness, coverage mechanism to alleviate the under extraction and over extraction problem, and gated dependency attention mechanism to incorporate dependency information
Question: What's the size of the previous largest OpenIE dataset?

Evidence:
raw_evidence: Prior to the SAOKE data set, an annotated data set for OIE tasks with 3,200 sentences in 2 domains was released in BIBREF20 to evaluate OIE algorithms, in which the data set was said BIBREF20 “13 times larger than the previous largest annotated Open IE corpus”. The SAOKE data set is 16 times larger than the data set in BIBREF20 . To the best of our knowledge, SAOKE data set is the largest publicly available human labeled data set for OIE tasks. Furthermore, the data set released in BIBREF20 was generated from a QA-SRL data set BIBREF21 , which indicates that the data set only contains facts that can be discovered by SRL (Semantic Role Labeling) algorithms, and thus is biased, whereas the SAOKE data set is not biased to an algorithm. Finally, the SAOKE data set contains sentences and facts from a large number of domains.

Answers:
3,200 sentences
Question: How does propose model model that capture three-way interactions?

Evidence:
raw_evidence: We model interactions between observations from the environment, goal, and document using layers. We first encode text inputs using bidirectional LSTMs, then compute summaries using self-attention and conditional summaries using attention. We concatenate text summaries into text features, which, along with visual features, are processed through consecutive layers. In this case of a textual environment, we consider the grid of word embeddings as the visual features for . The final output is further processed by MLPs to compute a policy distribution over actions and a baseline for advantage estimation. Figure FIGREF18 shows the model.

Answers:
 We first encode text inputs using bidirectional LSTMs, then compute summaries using self-attention and conditional summaries using attention. We concatenate text summaries into text features, which, along with visual features, are processed through consecutive layers. In this case of a textual environment, we consider the grid of word embeddings as the visual features for . The final output is further processed by MLPs to compute a policy distribution over actions and a baseline for advantage estimation.