Assisting Undergraduate Students in Writing S panish Methodology Sections (2020)

Abstract: In undergraduate theses, a good methodology section should describe the series of steps that were followed in performing the research. To assist students in this task, we develop machine-learning models and an app that uses them to provide feedback while students write. We construct an annotated corpus that identifies sentences representing methodological steps and labels when a methodology contains a logical sequence of such steps. We train machine-learning models based on language modeling and lexical features that can identify sentences representing methodological steps with 0.939 f-measure, and identify methodology sections containing a logical sequence of steps with an accuracy of 87%. We incorporate these models into a Microsoft Office Add-in, and show that students who improved their methodologies according to the model feedback received better grades on their methodologies.

## introduction in the mexican higher education system, most undergraduate students write a thesis (tesis de licenciatura) before graduation. the academic advisor and the student are typically both involved. throughout the process, the advisor spends time reviewing the draft that the student is building and gradually offering suggestions. this process becomes a cycle until the document meets established standards and/or institutional guidelines. this cycle is often slow due to the required changes in the structure of the thesis. one of the key components of such a thesis is a methodology section, which contains the steps and procedures used to develop the research. a methodology is supposed to provide a step-by-step explanation of the aspects necessary to understand and replicate the research including the techniques and procedures employed, the type of research, the population studied, the data sample, the collection instruments, the data selection process, the validation instrument, and the statistical analysis process @xcite . natural language processing techniques have the potential to assist students in writing such methodologies, as several aspects of good methodologies are visible from lexical and orthographic features of the text. a good methodology should have phrases or sentences that represent a series of steps, which may be written in a numbered list or in prose with sequential connectives like next. steps in a methodology section should have a predicate that represents the action of that step, like analyze or design. and the list of steps should be in a logical order, e.g., an explore step should typically appear before (not after) an implement step. good methodology sections should of course have much more beyond these simple features, but any methodology section that is missing these basic components is clearly in need of revision. we thus focus on designing machine-learning models to detect and evaluate the quality of such steps in a spanish-language student-written methodology section, and on incorporating such models into an interactive application that gives students feedback on their writing. our contributions are the following: ## background there is a long history of natural language processing research on interactive systems that assist student writing. essay scoring has been a popular topic, with techniques ranging from syntactic and discourse analysis @xcite , to list-wise learning-to-rank @xcite , to recurrent neural networks @xcite . yet the goal of such work is very different from ours, as we aim not to assign an overall score, but rather to provide detailed feedback on aspects of a good methodology that are present or absent from the draft. intelligent tutoring systems have been developed for a wide range of topics, including mechanical systems @xcite , qualitative physics @xcite , learning a new language @xcite , and introductory computer science @xcite . as we focus on assisting students in writing thesis methodology sections, the most relevant prior work focuses on analysis of essays. ets criterion (attali, 2004) uses features like n-gram frequency and syntactic analysis to provide feedback for grammatical errors, discourse structure, and undesirable stylistic features. the sat system @xcite combines lexical and grammatical properties with a perceptron learner to provide detailed sentence-by-sentence feedback about possible lexical and grammatical errors. revision assistant @xcite uses logistic regression over lexical and grammatical features to provide feedback on how individual sentences influence rubricspecific formative scores. all of these systems aim at general types of feedback, not the specific feedback needed for methodology sections. other related work touches on issues of logical organization. @xcite propose training sentence ordering models to differentiate between the original order of a well-written text and a permuted sentence order. @xcite continue in this paradigm, training an encoderdecoder network to read a series of sentences and reorder them for better coherence. our goal is not to reorder a student's sentences, but to provide more detailed feedback on whether the right structures (e.g., steps) are present in the methodology. more relevant work is @xcite , which combines lexical heuristics with sequence alignment models to score the organization of an essay. however, they provide only an overall score, and do not integrate this into any intelligent tutoring system. a final major difference between our work and prior work is that all the work above focused on the english language, while we provide feedback for spanish-language theses. ## data a collection was created using the coltypi foot_0 site. this site includes spanish-language theses within the information technologies subject area. the graduate level is composed of doctoral and master theses. the undergraduate level is composed of bachelor and advanced college-level technician (tsu) theses. all theses and research proposals in the collection have been reviewed at some point by a review committee. ## model: step identification the model for identifying which sentences represent steps (stepid) is a logistic regression foot_1 that takes a sentence as input, and predicts whether that sentence is a methodology step or not. the model considers the five types of features described in the following sections. ## model: logical sequence detection the model for detecting logical sequence (log-icseq) is a multilayer perceptron, with a single hidden layer of size two plus the number of features (weka's a layer specifier), that takes an entire methodology as input, and predicts whether it contains a logical sequence of steps or not. the model considers the features described in the following section. ## stepid and logicseq results both classifiers were evaluated using 10-fold crossvalidation on their respective parts of our annotated corpus. we thus find that despite our modest-sized data sets, accurate models based on language-model features can be trained to detect methodological steps in a thesis and identify whether those steps appear in a logical order. in the next section, we move from the intrinsic evaluation of our models on the annotated dataset to an extrinsic evaluation in a user study. ## pilot test we designed and performed a pilot test to assess the impact of using an application focused on the two models created, stepid and logicseq. the goal is to evaluate these models in an environment where students interact with the models while writing. our main research question is: what elements incorporated in the developed methods will have a positive impact on the student's final document? ## discussion we have demonstrated that with a small amount of training data, several carefully engineered features, and standard supervised classification algorithms, we can construct models that can reliably (0.939 f) detect the presence of steps in student-written spanish methodology sections, and reliably (87% accuracy) determine whether those steps are presented in a logical order. we have also shown that incorporating these models into an office add-in for microsoft word resulted in a system that students found useful and easy to use, and that the detections of the models were predictive of teacherassigned essay grades. there are some limitations to our study. first, because of the success of our simple models, we did not investigate more complex recent models like bert @xcite . such models might yield improved predictive performance but at a significant additional computational cost. second, the amount of data that we annotated was small, as it required a high level of expertise in the reviewing of spanish-language methodology sections. (we relied on spanish-speaking professors in computer science.) it would be good to expand the size of the dataset, but we take the high levels of performance of the models, and the fact that they make useful predictions on the unseen student-generated methodologies of the pilot test, as an indication that the dataset is already useful in its current size. finally, the pilot study was a controlled experiment, where specific problem statements were given as prompts. it would be interesting to measure the utility of the application for students writing their own theses. in the future, we would like to explore integrating other types of writing feedback into the turet office add-in, since students found its feedback about methodology steps both intuitive and helpful. though we focused on methodology sections in this article, our vision is a set of models that can provide useful feedback for all sections of a spanish-language student thesis.