TV - A f D : An Imperative-Annotated Corpus from The Big Bang Theory and W ikipedia’s Articles for Deletion Discussions (2020)

Abstract: In this study, we created an imperative corpus with speech conversations from dialogues in The Big Bang Theory and with the written comments in Wikipedia’s Articles for Deletion discussions. For the TV show data, 59 episodes containing 25,076 statements are used. We manually annotated imperatives based on the annotation guideline adapted from Condoravdi and Lauer’s study (2012) and used the retrieved data to assess the performance of syntax-based classification rules. For the Wikipedia AfD comments data, we first developed and leveraged a syntax-based classifier to extract 10,624 statements that may be imperative, and we manually examined the statements and then identified true positives. With this corpus, we also examined the performance of the rule-based imperative detection tool. Our result shows different outcomes for speech (dialogue) and written data. The rule-based classification performs better in the written data in precision (0.80) compared to the speech data (0.44). Also, the rule-based classification has a low-performance overall for speech data with the precision of 0.44, recall of 0.41, and f-1 measure of 0.42. This finding implies the syntax-based model may need to be adjusted for a speech dataset because imperatives in oral communication have greater syntactic varieties and are highly context-dependent.

## discussion since our annotation guideline is pragmatic-oriented, the annotated imperatives data from the big bang theory shows more varieties of imperatives. in comparison, our predefined rule for classification is relatively limited. the rule's overall performance for the speech data is low in precision, recall and f-1 score. also, although the ruleclassification performs better in text data from the wikipedia afd discussion forum in precision, there are still some errors worth discussing. we see that data from both sources have similar causes of false-positive errors with rule classification, 60% of the false-positive from the tv show data and 70% of the false-positive from the wikipedia discussion are both resulted by the dropping of first-person singular pronoun or other possible pronouns and subjects at the beginning of the sentence (e.g. "see you"). this indicates that the rule might be too general in some cases when it simply tries to catch sentences with a base form of a verb at the beginning of the sentence. however, we also notice that these phrases from the speech data including "see you", "guess what", and "thank you" are commonly used phrases. it is possible for these errors to be largely avoided if the rule excludes these phrases from getting classified as positive. besides, in both data sources, 5% of false-positive errors are caused by questions starting with "do you/we" or "have you/we", which shows that this type of error might need specific processing steps. intriguingly, while only 2% of false-positives in the text data is caused by the pragmatic reason, 30% of falsepositive errors in the speech data are caused by the same reason. this may indicate that there are much more statements in the speech data that fit in the general syntactic forms of imperatives but not serve the functions of imperatives. also, we found that pos tagging and parsing have their limitations, in that in some cases they fail to parse "n't" or incorrectly tag verbs as adverbs. it leads to 20% of false positives in the text data and 20% of false negatives in the speech data. other than that, we found the rules in the rule-classification we design are not flexible enough for speech data. there are 80% of false negatives resulted from the rules being too limited and not representing how people construct imperatives in face-toface communication beyond adhering to the general syntactic structures. that said, imperative detection in speech data require more flexible rules and to account for commonly used sentence patterns (e.g. "would you like to join me for chinese food" or "could you get that"). ## conclusion in this paper, we used data from the tv show, the big bang theory for speech data and wikipedia articles for deletion comments for written text data to build an imperative corpus. we used an annotation guideline based on @xcite and a predefined classification rule based on the general syntactic forms of imperatives to classify retrieved data. we then used the corpus to assess the performance of the predefined rule. the assessment shows different performances for speech and text data. the low performance of the rule on speech data may also imply the flaws, limitations and inflexibility associated the contextfree feature of the rule-based model. the differences shown in the error analysis of the rule-based classification for both datasets also reveal some linguistic differences between written and face-to-face communication. additionally, this use case showcases the value of the corpus in our study, which provides a labeled imperatives dataset to develop, test, and improve not just rule-based classification models, but also supervised machine learning models in future studies. another factor worth future analysis is that our classification methods may work differently with auto transcribed text, as there can be transcription errors of text and sentence segmentation and other factors that can complicate the classification process.