Mind the Gap between the Application Track and the Real World (2023)

Abstract: Recent advances in NLP have led to a rise in inter-disciplinary and application-oriented research. While this demonstrates the growing real-world impact of the field, research papers frequently feature experiments that do not account for the complexities of realistic data and environments. To explore the extent of this gap, we investigate the relationship between the real-world motivations described in NLP papers and the models and evaluation which comprise the proposed solution. We first survey papers from the NLP Applications track from ACL 2020 and EMNLP 2020, asking which papers have differences between their stated motivation and their experimental setting, and if so, mention them. We find that many papers fall short of considering real-world input and output conditions due to adopting simplified modeling or evaluation settings. As a case study, we then empirically show that the performance of an educational dialog understanding system deteriorates when used in a realistic classroom environment.

## introduction modern nlp systems, powered by large language models (llms), now have the ability to perform well at foundational natural language understanding and generation tasks @xcite . such systems have also increased access and made inter-disciplinary contributions possible across fields such as medicine, law, education, and science. in nlp venues like acl, the growth in applied and inter-disciplinary work can be witnessed in the nlp applications track, which received the second-highest number of submissions at emnlp 2022. recently published research from these tracks includes work on complex and important tasks such as synthesizing code for visualization @xcite , classifying operational risk in finance @xcite , and verifiying scientific claims @xcite . however, the inherent complexities associated with real-world data distributions and workflows can lead to the actual problem being simplified into an artificial setting that does not realistically reflect the original motivation. for instance, systems may make assumptions about the input available (e.g., require providing pseudocode/docstrings for code generation), or only evaluate on manually curated clean data as opposed to noisier data such as automatic speech recognition (asr) outputs. motivated by this observation and in line with the acl 2023 theme track, we set out to investigate the relationship between the motivation described in the introductions and the actual experiments in application-focused nlp papers. we survey papers from the nlp applications tracks of acl 2020 and emnlp 2020. specifically, we ask if there are gaps between motivation and experimentation, in the form of i) sub-tasks that are required for the application, but haven't been mentioned in the paper ii) data distributions that are expected in real-world conditions, but haven't been included in the paper's modeling or evaluation. we find that authors do not always explicitly mention assumptions they make, and often operate in constrained scenarios highly different from their intended motivation. to empirically demonstrate the severity of this problem, we then present a case study investigating the performance of an educational dialog system, when the inputs are changed from manually transcribed data to transcripts from a state-of-the-art asr system. the purpose of the system is to classify utterances made by a student in a classroom into talkmoves @xcite that reflect the communication strategies they use, such as making a claim, relating to another student. we find that performance drops by 14.6 points (21.2%) when evaluting on google asr instead of human transcripts. however, asr was not identified as a key component of the evaluation pipeline by the original work. we argue that as the field grows and nlp models get better and better at simulated and constrained settings, it is important for us to explicitly consider additional complexities of our systems in practice. we then present suggestions for authors and organizers of conferences, towards this end. ## case study in this section, we present a case study of an application from the domain of education. the task involves classifying student utterances into talk moves @xcite , which are strategies provided by the academically productive talk framework @xcite , that students and teachers use for maintaining productive and respective discourse in a classroom. we empirically analyze the impact of evaluating this task only on a constrained, artificial environment, as opposed to a more realistic setting. ## discussion through the above survey and case study, we qualitatively and quantitatively examine the gap between task-focused solutions in nlp research, and realistic use cases. we first acknowledge that there has existed a long-standing tradition in nlp to contextualize current research efforts through potential future applications. looking at task-oriented dialog systems for example, early work such as @xcite was motivated by the need to design computational assistants to support humans in mechanical tasks, and discussed the construction of essential components such as discourse processors, despite missing key upstream and downstream components such as asr or dialog generation. investigating sub-problems and their respective solutions in environments that are distinct from real-word settings has largely been unavoidable and sometimes even desirable. however, we argue that with the growth of the field and with the progress enabled by llms and related advances, we now have the opportunity to examine how closely our experimental setups can reflect our long term goals. additionally, for papers that are explicitly in the applications track, which present new applications intended to satisfy a real-world user need, we believe it is even more important to consider the bigger picture, and accurately describe necessary next steps for making the application a reality. to bridge this gap, we propose a few initial recommendations: i) we suggest including a question on the responsible nlp checklist foot_2 pertinent to application-oriented papers, asking if the experimental setup has taken into account the real-world conditions of the application, ii) we recommend that authors describe any potential gaps between their motivation and proposed solution, and if so, state what is lost in the gap (such as asr), and iii) we call for work to investigate ways to explicitly account for the gap, such as simulating noisy input data in cases where accessing the true distributions is not possible. we invite discussion from the research community on other ways forward. ## related work our paper adds to a body of work on meta-analysis of nlp papers and the state of nlp research, particularly from the recently introduced theme tracks at *acl conferences @xcite @xcite . similarly to us in that the authors examine evaluation practices, @xcite points out problems with benchmarking, while @xcite proposes ways to improve leaderboards in order to truly track progress. other papers that critically examine evaluation and leaderboards include ribeiro et al. (2020); @xcite and @xcite . in contrast, we focus on discrepancies between proposed experimental settings and the stated motivation of research endeavours. in addition, @xcite discusses that, similar to problematic hype, underclaiming when talking about nlp models comes with risks, and bianchi and hovy (2021) highlights multiple concerning trends in nlp research. more broadly, lipton and steinhardt (2019) discuss concerns with ml scholarship, and church (2020) draws attention to downward trends in reviewing quality and how these can potentially be mitigated. ## conclusions we investigate the "gap" between the motivations of application-focused nlp papers and their actual experimental setting. through a survey of nlp applications papers from two nlp conferences, we find that i) necessary components for the application get overlooked when papers focus on subtasks and ii) realistic input sources such as asr are not being considered in downstream evaluations. we further highlight the severity of the latter issue through a case study on a dialog understanding system intended for classrooms, showing the drop in performance when asr input, expected in the real-world, is used. while we outline potential strategies to address this issue, we hope our work will spur further discussion about future steps.