Supporting S panish Writers using Automated Feedback (2021)

Abstract: We present a tool that provides automated feedback to students studying Spanish writing. The feedback is given for four categories: topic development, coherence, writing conventions, and essay organization. The tool is made freely available via a Google Docs add-on. A small user study with third-level students in Mexico shows that students found the tool generally helpful and that most of them plan to continue using it as they work to improve their writing skills.

## motivation and background there are a multitude of writing support tools available for students who wish to improve their english writing (e.g. grammarly, 1 writing mentor, 2 ginger, 3 microsoft word, or revision assistant 4 ). these tools for english vary in complexity from basic feedback on spelling errors to advanced feedback about structure, register, topic development, and use of evidence to support claims. in the context of writing feedback for spanish, there are automatic grammar checkers available (e.g. language-tool 5 or spanishchecker 6 ). however, there are no tools for spanish that offer the kind of comprehensive writing feedback that tools such as writing mentor and grammarly offer. there is a huge native spanish-speaking population (almost 500 million globally according 1 www.grammarly.com 2 mentormywriting.org 3 www.gingersoftware.com 4 www.revisionassistant.com/ 5 https://languagetool.org 6 https://spanishchecker.com/ to wikipedia foot_0 ) that we could potentially support by providing advanced nlp tools to help improve writing skills. ## related work in automated feedback studies have shown that automated feedback on student writing can have a positive impact on their learning @xcite @xcite @xcite . the nlp technologies used to provide feedback on writing have often gone hand-in-hand with the development of automated scoring systems. the intuition is that if the system is "measuring" some aspect of writing in order to be able to grade it, it could also use that same measurement in order to give feedback. however, there are also studies with mixed, or less favourable outcomes when students use tools that provide automated feedback on their writing @xcite @xcite . this is an active area of research, and one that requires significant resources to conduct valid user studies and evaluations. ## spanish writing mentor building on previous work that developed comprehensive automated writing feedback for english, we have developed a tool to similarly support spanish writers by providing automated feedback. mentormywriting.org/es. html contains information about the tool, links to the download page, a video describing the main features of the tool, as well as an faq section. the tool is implemented as a google doc add-on (front-end), freely available to download from the app store, with a server-based back-end processing student texts and computing feedback. it is an extension of the original work done for english @xcite ) and the add-on allows users to select either english or spanish on a per-document basis. figure 1 shows the language selection screen when the add-on is first started. the design of the back-end feedback engine was based on the corresponding english one. however, in terms of the implementation, much of the functionality naturally differs in order to account for the language differences. furthermore, we introduce some new functionality -most notably the section on well-organized writing -that could potentially also be made available for the english version of the tool. we take advantage of a number of publicly available tools to build our feedback components. we use the spanish stanford core nlp foot_2 for tokenization, tagging and constituency parsing @xcite . we use the spacy foot_3 spanish dependency parser @xcite , aligning the dependency relations to the tokenization provided by the stanford tools. we use the standalone version of spanish lan-guagetool foot_4 to compute a subset of the feedback relating to writing conventions (spelling and grammatical errors). ## user study we conducted a user study to collect initial usage and perception data from the tool. our participants were students in the universidad autónoma metropolitana, cuajimalpa, in mexico city. participants were recruited from two groups: (1) a group of students taking optional courses in the university writing center, which provides support to students who want to improve their writing skills and (2) a group of 3rd and 4th year undergraduate students taking an elective course in latin american literature. participation in the study was optional, and each student who took part received a certificate of participation upon completion. participants were asked to use spanish writing mentor to support their regular coursework writing assignments. no changes were made to the assignments. users were given instructions on how to use the tool three weeks before the end of the trimester, and could choose how to use the tool (if at all) during those three weeks. the user study focused only on extended writing. an investigation into the usefulness of the paragraph writing component remains for future work. our user study consists of two components: (1) a measure of writing ability before and after using the tool and (2) a user survey completed after the three weeks. in order to measure writing ability before and after using the tool, each participant completed a standardized assessment of writing ability. the assessment is usually administered as a placement test in the writing center to assign students to one of four levels: low (0-49), moderate (50-69), acceptable (70-89) or optimum (90-100). our user survey consisted of 13 questions (see appendix a) and participants were asked to complete it after they had handed in their final assignments. shows the number of unique users each month between october 2020 and march 2021. we see that the number of users peaked during our user study, but did not drop off entirely once the study was over. figure 6b shows the average time (in minutes) for an active session (i.e. we exclude sessions where no text was entered). we see differences in average usage across months, but for the months with the most users, the average time spent using the app was between 15 and 30 minutes. finally, figure 6c shows the distribution of the time (in seconds) for each section of the app from the time period october 2020 to march 2021. we restrict the plot to the interquartile range, since there were many extreme outliers (probably due to users switching away from the app and coming back later). even still, we see quite a range of values for the medial time spent per section. sections such as contractions, grammar errors, word choice, transition terms, well organized and long sentences engaged the users for longer times, while sections such as flow of ideas, pronoun use, title and section headers engaged the users less. ## conclusions we presented a tool to support writers of spanish by providing them automated feedback within a free google docs add-on. the tool was built by adapting an existing tool for english, and implementing spanish-languagespecific components. we conducted a small study with 13 post-secondary level students in mexico city, and in general found that they considered the tool helpful and were planning to continue using it and also recommend it to others. we see that their writing ability, as measured by a standardized test, improved between the pre-and post-tests, though we cannot yet say whether the spanish writing mentor app contributed to this improvement. we find that a small number of users engage with the app, even outside of planned user studies, which is encouraging.