Comparing Supervised Machine Learning Techniques for Genre Analysis in Software Engineering Research Articles (2021)

Abstract: Written communication is of utmost importance to the progress of scientific research. The speed of such development, however, may be affected by the scarcity of reviewers to referee the quality of research articles. In this context, automatic approaches that are able to query linguistic segments in written contributions by detecting the presence or absence of common rhetorical patterns have become a necessity. This paper aims to compare supervised machine learning techniques tested to accomplish genre analysis in Introduction sections of software engineering articles. A semi-supervised approach was carried out to augment the number of annotated sentences in SciSents (Avaliable on: ANONYMOUS). Two supervised approaches using SVM and logistic regression were undertaken to assess the F-score for genre analysis in the corpus. A technique based on logistic regression and BERT has been found to perform genre analysis highly satisfactorily with an average of 88.25 on F-score when retrieving patterns at an overall level.

## introduction written communication plays a fundamental role in scholarly development. evidence for this is the high number of estimated publications and journals @xcite @xcite . in this scenario, the reviewing process is a crucial pathway for improving publication quality, as it acts as a filter through which suitable research papers are selected for publication @xcite . in principle, although academic gatekeeping does not entail rigid language rubrics, scientific publications on the whole present standardised conventions such as preference for passive constructions, high nominal style, paper division in sections, and use of lexical and phrasal structures to indicate the function and purpose of each text portion @xcite . disseminated linguistic work aiming to systematically describe writing organization with a focus on the introduction section is the cars (create a research space) model @xcite . cars approaches genre analysis by introducing two concepts, namely moves and steps. whereas a move represents the objectives and functions of a text segment at an overall level, a step further elaborates on explaining how the rhetorical means are specifically used to perform the function of a move @xcite (see examples in computational approaches may be implemented to query linguistic segments automatically in research articles by indicating the presence or absence of commonly used rhetorical patterns. automatic approaches such as support vector machines (svm) @xcite can be employed to perform genre analysis due to its productive results regarding textual issues @xcite . nonetheless, they require annotated data, which are scant in the literature and not easily obtained, with the existing ones having limited amount of input @xcite @xcite @xcite @xcite . manual annotation is an arduous, expensive and time-consuming task as it requires expert human annotators. to tackle this issue, svm may be used as a semi-supervised approach, in which considerable amounts of labeled and unlabeled data are utilized together to form more solid classifiers @xcite . in this context, this work aims to evaluate supervised and semi-supervised machine learning techniques for automatic retrieval of rhetorical patterns within swales' cars genre analysis schema in research articles. this investigation was carried out into scisents foot_1 corpus and, for this reason, is restricted to the introduction section of software engineering articles. this paper has two main objectives: the first is to augment the number of annotations in scisents corpus, and the second is to compare and assess f-scores generated by supervised approaches for genre analysis. for such, we performed a comparison between svms and logistic regression techniques for the classification task. for sentence encoding, we evaluated novel approaches such as universal sentence encoder @xcite and bert @xcite . this paper proceeds as follows: firstly, we review state-of-the-art works on genre analysis automation. next, we present and describe the corpus and the semi-supervised annotation procedures. also, we comparatively discuss the main features of the techniques employed in the experiments. we then address the included implementation details and, finally, report the results. ## related work one relevant reference for genre analysis automation is the work of @xcite , who proposed a computer software tool for outlining a research article's structure. based on swales' schema, the tool (named mover) aimed at presenting to learners a panorama of the move structure utilized in ra. the tool scanned 100 information technology articles abstracts comprising 692 sentences. the abstracts were manually annotated on the grounds of a modified create a research space (cars) model proposed by @xcite . the model includes swales @xcite 3 moves, as well as 12 steps. since this is a general and small-sized model designed for the introduction section, not all steps appeared in the dataset. a modified bag of words was utilized to represent the text so it could be machine manipulated. in a traditional bag of words, dataset sentences would be tokenized in single words. however, the authors added clusters of sequential words in order to allow the system to operate at the discourse level, therefore naming the model as bag of clusters. as well as allowing the system to identify steps only possible to classify if preceding or subsequent steps are known, an additional "location" feature was added to the bag of clusters model. the model's output fed a naive bayes classifier which performed consistently with an average step accuracy rate of 68% (ranging from 17% -indicate gap -to 92% -announce research). the authors justified the poor results by the scarce training items from these steps. through error analysis, they observed that when the software presented flaws, the incorrectly categorised step tended to fall within the same move. in order to improve accuracy the two most probable classifications were used in a second experiment. in this turn, the user had to select the most appropriate option. with this procedure, accuracy achieved 86%. however, despite the productive result, the reduced number of articles and sentences was a hindrance for further validation. @xcite attempted to devise a pedagogical tool for automating discourse evaluation. the purpose was to appraise academic writing drafts in agreement with an adapted model based on cars, to compare it with other papers from the same discipline and to provide feedback to the student. to develop such a tool, a text-categorization approach using suport vector machine (svm) for sentence classification in research article introductions drawn on swales' rhetorical moves was employed. an experiment was conducted with a corpus named intelligent academic discourse evaluator (iade) consisting of 11,149 sentences from 401 introduction sections in 20 academic disciplines. each sentence was manually annotated within the moves from cars schema. to execute the classification, sentences were stemmed and represented in an n-dimensional vector (up to word trigrams). experiment results were encouraging (with an accuracy above 70%), but the dataset was relatively small and did not take steps into account. cotos and pendar (2016) made progress in their own 2008 work by increasing iade's size to 1,020 research articles across 51 disciplines. sentences were also annotated according to the cars model, but this time including both moves and steps. an svm classifier with the previous settings achieved a move accuracy of 72.6% and a step accuracy of 72.9%. @xcite presented a neural network architecture composed of a bi-lstm with crf as an automated approach to examine rhetorical structure in student writing. the embedding layer was initialised with a pre-trained representation of glove @xcite and was fine-tuned to the dataset to produce more accurate word representation. two datasets were used to test the model: iade @xcite and research writing tutor (rwt) comprising 900 full research articles (not only introduction sections) across 30 academic disciplines. rwt was manually annotated and sentences with one communicative goal and more than one functional strategy could be labeled with several steps; a sentence could be assigned with a secondary move/step tag if it had more than one communicative goal. experiments results achieved a precision and recall of 77%, and an f1-score of 76% for the classification task in rwt dataset. due to the data paucity problem present in the aforementioned works, this study proposes a semisupervised approach as a contribution towards genre analysis automation as far as the cars framework is concerned. a detailed explanation of the procedures can be found in the following sections. ## results we report the f-score averaged over the folds of our techniques in tables 2 , 3 , 4 and 5. each table column shows the result of an experiment type comprising a technique (svm or logistic regression), a sentence embedding technique (bert or universal sentence encoder), and an annotated set (scisents, semi-supervised round 1 and semisupervised round 2). ## discussion the present study was designed to augment the number of annotations in scisents corpus and to compare results in supervised machine learning techniques for genre analysis in software engineering research articles. the number of annotated sentences was increased from the 595 ones in scisents to 1309 through two semi-supervised rounds using svm. svm versus logistic regression: logistic regression produced higher outcomes than svm in 64% of the experiments. when associated with bert, logistic regression beats svm in 85% of cases, but when in conjunction with use, svm outperformed logistic regression in 57% of experiments. universal sentence encoder versus bert: vector representation provided by bert delivered higher scores than universal sentence encoder did in 75,5% of the tested sets. when bert was employed with logistic regression, the results overcame other experiments in 81% of cases. thus, from the pairs of techniques tested, the indicated one for genre analysis is logistic regression with bert. vector representation -sentence alone versus co-occurring sentences: one finding in the experiments in supervised machine learning techniques is that, in most cases, the use of sentence embedding generated from the sentence alone provided more productive results than those with the use of the actual sentence together with its preceding one. semi-supervised approach: when analysing the evolution of results throughout the annotation process within each experiment type we can notice that they did not always improve accordingly. in when approaching annotation evolution throughout although most of the best results for steps were achieved with a second round annotation set, setbacks were also present, thus indicating the need for more annotations for probabilities potentially corresponding to the categories set for steps. these annotations may also contribute to genre analysis regarding moves, despite results presenting insignificant improvements with a second round of annotations. ## conclusion the present study compared supervised machine learning techniques which automatically retrieved linguistic segments from research articles. firstly, we used a semi-supervised approach to increase the number of annotated sentences in scisents corpus. next, we used two supervised and two sentence embedding techniques to carry out genre analysis on the dataset. the results suggest that an approach based on logistic regression and bert presents higher scores for genre analysis. in addition, although a semi-supervised annotation process has proven to contribute to the overall procedure, it lacks elements with random probabilities for substantial improvement. as future work, the semi-supervised annotation process and the techniques hereby described can be used for annotating other sections of software engineering research articles. also, the same analyses could be applied to articles from other domains so that cross-disciplinary rhetorical differences could be identified.