qinchuanhui/UDA-QA huggingface dataset:
Question: What are the qualitative experiments performed on benchmark datasets?

Evidence:
raw_evidence: Table TABREF18 shows the Spearman correlation values of GM$\_$KL model evaluated on the benchmark word similarity datasets: SL BIBREF20, WS, WS-R, WS-S BIBREF21, MEN BIBREF22, MC BIBREF23, RG BIBREF24, YP BIBREF25, MTurk-287 and MTurk-771 BIBREF26, BIBREF27, and RW BIBREF28. The metric used for comparison is 'AvgCos'. It can be seen that for most of the datasets, GM$\_$KL achieves significantly better correlation score than w2g and w2gm approaches. Other datasets such as MC and RW consist of only a single sense, and hence w2g model performs better and GM$\_$KL achieves next better performance. The YP dataset have multiple senses but does not contain entailed data and hence could not make use of entailment benefits of GM$\_$KL.
Table TABREF19 shows the evaluation results of GM$\_$KL model on the entailment datasets such as entailment pairs dataset BIBREF29 created from WordNet with both positive and negative labels, a crowdsourced dataset BIBREF30 of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset BIBREF31. The 'MaxCos' similarity metric is used for evaluation and the best precision and best F1-score is shown, by picking the optimal threshold. Overall, GM$\_$KL performs better than both w2g and w2gm approaches.
raw_evidence: Table TABREF9 shows the qualitative results of GM$\_$KL. Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed. For eg., the word `plane' in its 0th component captures the `geometry' sense and so are its neighbours, and its 1st component captures `vehicle' sense and so are its corresponding neighbours. Other words such as `rock' captures both the `metal' and `music' senses, `star' captures `celebrity' and `astronomical' senses, and `phone' captures `telephony' and `internet' senses.

Answers:
Spearman correlation values of GM_KL model evaluated on the benchmark word similarity datasets.
Evaluation results of GM_KL model on the entailment datasets such as entailment pairs dataset created from WordNet, crowdsourced dataset of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset.
Given a query word and component id, the set of nearest neighbours along with their respective component ids are listed
Question: How does this approach compare to other WSD approaches employing word embeddings?

Evidence:
raw_evidence: Table TABREF17 compares the performance of the approaches on the SCWS dataset. It is evident from Table TABREF17 that GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.

Answers:
GM$\_$KL achieves better correlation than existing approaches for various metrics on SCWS dataset.
Question: What tasks did they use to evaluate performance for male and female speakers?

Evidence:
raw_evidence: In this paper, we first highlight the importance of TV and radio broadcast as a source of data for ASR, and the potential impact it can have. We then perform a statistical analysis of gender representation in a data set composed of four state-of-the-art corpora of French broadcast, widely used within the speech community. Finally we question the impact of such a representation on the systems developed on this data, through the perspective of an ASR system.

Answers:
ASR
Question: What is the goal of investigating NLP gender bias specifically in the news broadcast domain and Anchor role?

Evidence:
raw_evidence: We found that an ASR system trained on unbalanced data regarding gender produces gender bias performance. Therefore, in order to create fair systems it is necessary to take into account the representation problems in society that are going to be encapsulated in the data. Understanding how women under-representation in broadcast data can lead to bias in ASR performances is the key to prevent re-implementing and reinforcing discrimination already existing in our societies. This is in line with the concept of “Fairness by Design" proposed by BIBREF31.
raw_evidence: Besides the social impact of gender representation, broadcast recordings are also a valuable source of data for the speech processing community. Indeed, automatic speech recognition (ASR) systems require large amount of annotated speech data to be efficiently trained, which leaves us facing the emerging concern about the fact that "AI artifacts tend to reflect the goals, knowledge and experience of their creators" BIBREF3. Since we know that women are under-represented in media and that the AI discipline has retained a male-oriented focus BIBREF4, we can legitimately wonder about the impact of using such data as a training set for ASR technologies. This concern is strengthened by the recent works uncovering gender bias in several natural language processing (NLP) tools such as BIBREF5, BIBREF6, BIBREF7, BIBREF8.

Answers:
create fair systems
 broadcast recordings are also a valuable source of data for the speech processing community, recent works uncovering gender bias in several natural language processing (NLP) tools
Question: Which corpora does this paper analyse?

Evidence:
raw_evidence: Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.
raw_evidence: Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.

Answers:
ESTER1, ESTER2, ETAPE, REPERE
ESTER1, ESTER2, ETAPE, REPERE
Question: How many categories do authors define for speaker role?

Evidence:
raw_evidence: As pointed out by the CSA report BIBREF1, women presence tends to be marginal within the high-audience hours, showing that women are represented but less than men and within certain given conditions. It is clear that a small number of speakers is responsible for a large number of speech turns. Most of these speakers are journalists, politicians, presenters and such, who are representative of a show. Therefore, we introduce the notion of speaker's role to refine our exploration of gender disparity, following studies which quantified women's presence in terms of role. Within our work, we define the notion of speaker role by two criteria specifying the speaker's on-air presence, namely the number of speech turns and the cumulative duration of his or her speaking time in a show. Based on the available speech transcriptions and meta-data, we compute for each speaker the number of speech turns uttered as well as their total length. We then use the following criteria to define speaker's role: a speaker is considered as speaking often (respectively seldom) if he/she accumulates a total of turns higher (respectively lower) than 1% of the total number of speech turns in a given show. The same process is applied to identify speakers talking for a long period from those who do not. We end up with two salient roles called Anchors and Punctual speakers:
the Anchor speakers (A) are above the threshold of 1% for both criteria, meaning they are intervening often and for a long time thus holding an important place in interaction;
the Punctual speakers (PS) on the contrary are below the threshold of 1% for both the total number of turns and the total speech time.

Answers:
 two salient roles called Anchors and Punctual speakers
Question: How big is imbalance in analyzed corpora?

Evidence:
raw_evidence: As expected, we observe a disparity in terms of gender representation in our data (see Table ). Women represent 33.16% of the speakers, confirming the figures given by the GMMP report BIBREF0. However, it is worth noticing that women account for only 22.57% of the total speech time, which leads us to conclude that women also speak less than men.

Answers:
Women represent 33.16% of the speakers
Question: What are four major corpora of French broadcast?

Evidence:
raw_evidence: Surprisingly, as data is said to be “the new oil", few data sets are available for ASR systems. The best known are corpora like TIMIT BIBREF10, Switchboard BIBREF11 or Fisher BIBREF12 which date back to the early 1990s. The scarceness of available corpora is justified by the fact that gathering and annotating audio data is costly both in terms of money and time. Telephone conversations and broadcast recordings have been the primary source of spontaneous speech used. Out of all the 130 audio resources proposed by LDC to train automatic speech recognition systems in English, approximately 14% of them are based on broadcast news and conversation. For French speech technologies, four corpora containing radio and TV broadcast are the most widely used: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four corpora have been built alongside evaluation campaigns and are still, to our knowledge, the largest French ones of their type available to date.

Answers:
ESTER1, ESTER2, ETAPE, REPERE
Question: What were their results on the classification and regression tasks

Evidence:
raw_evidence: Table TABREF29 gives three results from our submissions in the competition. The first is the baseline NBSVM solution, with an F1 of 0.7548. Second is our first random seed selected for the classifier which produces a 0.8083 result. While better than the NBSVM solution, we pick the best validation F1 from the 20 seeds we tried. This produced our final submission of 0.8099. Our best model achieved an five-fold average F1 of 0.8254 on the validation set shown in Figure FIGREF27 but a test set F1 of 0.8099 - a drop of 0.0155 in F1 for the true out-of-sample data. Also note that our third place entry was 1.1% worse in F1 score than first place but 1.2% better in F1 than the 4th place entry.
raw_evidence: Table TABREF29 gives three results from our submissions in the competition. The first is the baseline NBSVM solution, with an F1 of 0.7548. Second is our first random seed selected for the classifier which produces a 0.8083 result. While better than the NBSVM solution, we pick the best validation F1 from the 20 seeds we tried. This produced our final submission of 0.8099. Our best model achieved an five-fold average F1 of 0.8254 on the validation set shown in Figure FIGREF27 but a test set F1 of 0.8099 - a drop of 0.0155 in F1 for the true out-of-sample data. Also note that our third place entry was 1.1% worse in F1 score than first place but 1.2% better in F1 than the 4th place entry.

Answers:
F1 of 0.8099
F1 score result of 0.8099
Question: Do the authors conduct experiments on the tasks mentioned?

Evidence:
raw_evidence: What may be more surprising is that currently (July 2016), machine translation programs are unable to choose the feminine plural pronoun even when the only possible antecedent is a group of women. For instance, the sentence “The girls sang a song and they danced” is translated into French as “Les filles ont chanté une chanson et ils ont dansé” by Google Translate (GT), Bing Translate, and Yandex. In fact, I have been unable to construct any sentence in English that is translated into any language using the feminine plural pronun. Note that, since the masculine plural pronoun is used for groups of mixed gender in all these languages, it is almost certainly more common in text than the feminine plural; hence this strategy is reasonable faute de mieux. (It also seems likely that the erroneous use of a feminine plural for a masculine antecedent sounds even more jarring than the reverse.)

Answers:
Yes
No
Question: Did they collect their own datasets?

Evidence:
raw_evidence: A number of the Winograd schemas already published can be “converted” quite easily to this form. Indeed, the above example was constructed in this way; the original form was “Jane knocked on Susan's door, but she did not [answer/get an answer].” Of the 144 schemas in the collection at http://www.cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.html there are 33 that can plausibly be translated this way. (#'s 3, 4, 12, 15, 17, 18, 23, 25, 26, 27, 42, 43, 44, 45, 46, 57, 58, 60, 69, 70, 71, 77, 78, 79, 86, 97, 98, 106, 111, 114, 127, 132, 144) — essentially any schema that involves two people as possible referents where the content of the sentences makes sense as applied to groups of people rather than individuals.

Answers:
No
Question: What data do they look at?

Evidence:
raw_evidence: Care must be taken to avoid relying on, or seeming to rely on, objectionable stereotypes about men and women. One mechanism that can sometimes be used is to include a potentially problematic sentence in both directions. For instance, schema 23 from the WSC collection can be translated into both “The girls were bullying the boys so we [punished/rescued] them” and “The boys were bullying the girls, so we [punished/rescued] them,” thus avoiding any presupposition of whether girls are more likely to bully boys or vice versa.

Answers:
WSC collection
Question: Do they report results only on English datasets?

Evidence:
raw_evidence: For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017. A pre-processing step has been applied to replace URLs with a placeholder and to pad punctuation. The final vocabulary size was around 1.6 millions words. Additionally to the in-domain corpus we collected, we use GloVe vectors trained on Wikipedia articles in order to investigate the impact of out-of-domain word-vectors.
raw_evidence: For all the tasks in our experimental study, we use 36 millions English tweets collected between August and September 2017. A pre-processing step has been applied to replace URLs with a placeholder and to pad punctuation. The final vocabulary size was around 1.6 millions words. Additionally to the in-domain corpus we collected, we use GloVe vectors trained on Wikipedia articles in order to investigate the impact of out-of-domain word-vectors.

Answers:
Yes
Yes
Question: Which other hyperparameters, other than number of clusters are typically evaluated in this type of research?

Evidence:
raw_evidence: In this paper, we explore a hybrid approach, that uses text embeddings as a proxy to create features. Motivated by the argument that text embeddings manage to encode the semantics of text, we explore how clustering text embeddings can impact the performance of different NLP tasks. Although such an approach has been used in different studies during feature engineering, the selection of word vectors and the number of clusters remain a trial-end-error procedure. In this work we present an empirical evaluation across diverse tasks to verify whether and when such features are useful.

Answers:
selection of word vectors
Question: How were the cluster extracted? 

Evidence:
raw_evidence: We cluster the embeddings with INLINEFORM0 -Means. The k-means clusters are initialized using “k-means++” as proposed in BIBREF9 , while the algorithm is run for 300 iterations. We try different values for INLINEFORM1 . For each INLINEFORM2 , we repeat the clustering experiment with different seed initialization for 10 times and we select the clustering result that minimizes the cluster inertia.

Answers:
Word clusters are extracted using k-means on word embeddings
Question: what english datasets were used?

Evidence:
raw_evidence: Experimental Setup

Answers:
Answer with content missing: (Data section) Penn Treebank (PTB)
Question: which chinese datasets were used?

Evidence:
raw_evidence: Experimental Setup

Answers:
Answer with content missing: (Data section) Chinese with version 5.1 of the Chinese Penn Treebank (CTB)
Question: How did they determine fake news tweets?

Evidence:
raw_evidence: For our study, we used the number of retweets to single-out those that went viral within our sample. Tweets within that subset (viral tweets hereafter) are varied and relate to different topics. We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al. BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious. The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties.
raw_evidence: Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news. Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Exposure.
Characterization.
Polarization.

Answers:
an expert annotator determined if the tweet fell under a specific category
Exposure, Characterization, Polarization
Question: What is their definition of tweets going viral?

Evidence:
raw_evidence: One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers. Among the reasons for retweeting, Body et al. BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others. As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news). For our study, we consider that a tweet went viral if it was retweeted more than 1000 times.
raw_evidence: For our study, we used the number of retweets to single-out those that went viral within our sample. Tweets within that subset (viral tweets hereafter) are varied and relate to different topics. We consider that a tweet contains fake news if its text falls within any of the following categories described by Rubin et al. BIBREF7 (see next section for the details of such categories): serious fabrication, large-scale hoaxes, jokes taken at face value, slanted reporting of real facts and stories where the truth is contentious. The dataset BIBREF8 , manually labelled by an expert, has been publicly released and is available to researchers and interested parties.

Answers:
Viral tweets are the ones that are retweeted more than 1000 times
those that contain a high number of retweets
Question: What are the characteristics of the accounts that spread fake news?

Evidence:
raw_evidence: We found that 82 users within our sample were spreading fake news (i.e. they produced at least one tweet which was labelled as fake news). Out of those, 34 had verified accounts, and the rest were unverified. From the 48 unverified accounts, 6 have been suspended by Twitter at the date of writing, 3 tried to imitate legitimate accounts of others, and 4 accounts have been already deleted. Figure FIGREF28 shows the proportion of verified accounts to unverified accounts for viral tweets (containing fake news vs. not containing fake news). From the chart, it is clear that there is a higher chance of fake news coming from unverified accounts.
A useful representation for friends and followers is the ratio between friends/followers. Figures FIGREF31 and FIGREF32 show this representation. Notice that accounts spreading viral tweets with fake news have, on average, a larger ratio of friends/followers. The distribution of those accounts not generating fake news is more evenly distributed.
Figure FIGREF24 shows that, in contrast to other kinds of viral tweets, those containing fake news were created more recently. As such, Twitter users were exposed to fake news related to the election for a shorter period of time.
raw_evidence: Accounts spreading fake news appear to have a larger proportion of friends/followers (i.e. they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only. Together with the fact that, on average, tweets containing fake news have more URLs than those spreading viral content, it is possible to hypothesize that, both, the ratio of friends/followers of the account producing a viral tweet and number of URLs contained in such a tweet could be useful to single-out fake news in Twitter. Not only that, but our finding related to the number of URLs is in line with intuitions behind the incentives to create fake news commonly found in the literature BIBREF9 (in particular that of obtaining revenue through click-through advertising).

Answers:
Accounts that spread fake news are mostly unverified, recently created and have on average high friends/followers ratio
have a larger proportion of friends/followers (i.e. they have, on average, the same number of friends but a smaller number of followers) than those spreading viral content only
Question: What is the threshold for determining that a tweet has gone viral?

Evidence:
raw_evidence: One straightforward way of sharing information on Twitter is by using the retweet functionality, which enables a user to share a exact copy of a tweet with his followers. Among the reasons for retweeting, Body et al. BIBREF15 reported the will to: 1) spread tweets to a new audience, 2) to show one’s role as a listener, and 3) to agree with someone or validate the thoughts of others. As indicated, our initial interest is to characterize tweets containing fake news that went viral (as they are the most harmful ones, as they reach a wider audience), and understand how it differs from other viral tweets (that do not contain fake news). For our study, we consider that a tweet went viral if it was retweeted more than 1000 times.

Answers:
1000
Question: How is the ground truth for fake news established?

Evidence:
raw_evidence: The 1327 `viral' tweets were manually annotated as containing fake news or not. The annotation was carried out by a single person in order to obtain a consistent annotation throughout the dataset. Out of those 1327 tweets, we identified 136 as potentially containing fake news (according to the categories previously described), and the rest were classified as `non containing fake news'. Note that the categorization is far from being perfect given the ambiguity of fake news themselves and human judgement involved in the process of categorization. Because of this, we do not claim that this dataset can be considered a ground truth.

Answers:
Ground truth is not established in the paper