CARE : Collaborative AI -Assisted Reading Environment (2023)

Abstract: Recent years have seen impressive progress in AI-assisted writing, yet the developments in AI-assisted reading are lacking. We propose inline commentary as a natural vehicle for AI-based reading assistance, and present CARE: the first open integrated platform for the study of inline commentary and reading. CARE facilitates data collection for inline commentaries in a commonplace collaborative reading environment, and provides a framework for enhancing reading with NLP-based assistance, such as text classification, generation or question answering. The extensible behavioral logging allows unique insights into the reading and commenting behavior, and flexible configuration makes the platform easy to deploy in new scenarios. To evaluate CARE in action, we apply the platform in a user study dedicated to scholarly peer review. CARE facilitates the data collection and study of inline commentary in NLP, extrinsic evaluation of NLP assistance, and application prototyping. We invite the community to explore and build upon the open source implementation of CARE.Github Repository: https://github.com/UKPLab/CAREPublic Live Demo: https://care.ukp.informatik.tu-darmstadt.de

## introduction individual and collaborative text work is at the core of many human activities, including education, business, and research. yet, reading text is difficult and takes considerable effort, especially for long and domain specific texts that require expert knowledge. while past years have seen great progress in analyzing and generating text with the help of ai -culminating in strong generative models like gpt-3 @xcite and chatgpt @xcite 2 -the progress in applications of ai to reading and collaborative text work is yet to match these achievements. the ability of modern generative models to create natural-sounding but factually flawed outputs @xcite stresses the need for supporting humans in critical text assessment. humans use annotations to read and collaborate over text, from hand-written print-out notes to highlights in collaborative writing platforms. this makes in-text annotations -inline commentaries -a promising vehicle for ai-based reading assistance. for example, an ai assistant could automatically classify the user's commentaries, or verify and provide additional information on the highlighted passages. yet, the lack of data and key insights limits the nlp progress in this area: from the foundational perspective, we lack knowledge about the language of inline commentaries, as most of this data is not openly available for research. from the applied perspective, little is known about the hands-on interactions between humans and texts, how they translate into nlp tasks, and how the impact of nlp-based assistance on text comprehension can be measured. while ethical, controlled data collection has been receiving increasing attention in the past years @xcite , data collection tools for inline commentary are missing, and so are the tools for applying and evaluating nlp models within a natural reading environment. to address these limitations, we introduce care: a collaborative ai-assisted reading environment, where users can jointly produce in-1 3 2 4 5 figure 2 : an inline commentary in care consists of a highlight (1) optionally associated with a commentary text (2), a label (3), a number of free-form tags (4) and metadata (5), e.g. user name and creation time. our contribution has multiple audiences. for nlp researchers, care makes it possible to efficiently collect inline commentary data in a standardized manner, and provides a generic interface for the extrinsic evaluation of nlp models. for application designers, care offers an extensible platform and behavioral metrics to study how humans interact with texts. for users, care enables the development of new, innovative applications built around ai-assisted reading such as interactive e-learning, community-based fact-checking, and research paper assessment. to foster the progress in ai-assisted reading research, we make the implementation openly available and easy to deploy and to extend. ## platform description care addresses the gap in existing solutions that prevents the study of inline commentary and aiassisted reading. here we review the main components of care from the user perspective, while the next section outlines the key technical aspects of our open implementation. we discuss the components in order of importance and refer to the appendix a for the illustration of a typical user journey. at the core of care is the reading component which allows users to attach inline commentaries to documents. to ensure that the visual representation of the document remains true to its source and stable across platforms, care focuses on pdf as the main source format 13 . an inline commentary can amount to a simple highlight attached to a continuous text span, can be associated with a free-text note, and can carry a label from a pre-configured label set, as well as any number of free-form tags (figure 2 ). it is possible to add document-level commentaries that are not attached to a span. inline commentaries are displayed in the dedicated care sidebar and can be navigated and edited. the process is collaborative: multiple users can leave inline commentaries on the same document and reply to them in real time. the commentaries are saved and can be revisited at a later point; the resulting data can be exported in an easy-to-use data format, individually or in aggregate, and displayed within the user interface of care. in addition to the textual data, care collects and exports basic behavioral metrics; for instance, the time of highlight creation and the users' scrolling behavior within the document. the second key component of care is ai assistance: the inline commentary data can be routed to an arbitrary external nlp module, which returns the prediction that can be displayed in the annotation component in close-to-real-time as labels, inline commentary replies, or via a custom ui. the interaction between users and ai assistance is mediated by a flexible broker system that distributes the processing tasks among a set of nlp models. multiple ai assistance model instances can be acting simultaneously, and the pool of models can be extended easily through registering a new model node at the broker backend. at the moment of writing, care provides examples to supports integration of any pre-trained model compatible with the huggingface transformers api @xcite by simply changing the configuration parameters. the model then has access to the inline commentary text, highlighted span from the main document, labels, tags and metadata. it is possible to adapt care to use models based on other frameworks. finally, care features a flexible and configurable dashboard that provides quick access to user and system settings, document and label set management, and study management. in particular, the user management component is responsible for registration, authentication and authorization; to encourage responsible data collection and ensure that the collected inline commentary data can be used in research, care features sample informed consent forms that users are presented upon registration, along with the necessary licensing disclaimers, which can be refined by the study administrator. ## system design care is designed to be generic, modular and extensible (figure 3 ). the ability to build and deploy care via a docker container makes it easy to set it up in new environments. while care features detailed documentation, here we provide a high-level overview of the system design. care follows a client-server architecture, preferring client-side operation whenever possible to speed up execution and reduce network traffic and server load. this results in a clear separation of responsibilities between the client, the server and the nlp assistance components of care and affords high modularity. while the main client-server pair is purely javascript-based, the ai models and the broker are implemented in python to facilitate the nlp assistance development by the natural language processing community. the web-based care client is responsible for frontend rendering and annotation functionality. the client is fully implemented in vue.js foot_11 , allowing dynamic rendering, modular frontend structure and reuse of original and third-party components. bootstrap foot_12 is used throughout the frontend to ensure consistent styling and responsive design; document rendering is handled via pdf.js 16 . in addition, we adopt the localization code from hypothes.is 17 to locate inline commentaries in the document. the care server, in turn, is responsible for synchronizing the data among clients, authentication and authorization, and for connecting to external services, including the ai-assistance broker. in line with the javascript-based frontend, the backend is implemented in node.js 18 as a cross-platform run-time environment. as keeping message transition time low is crucial for collaboration and ai assistance, we base all communication on the websocket protocol 19 . persistent bidirectional connection between the client and server components enables real-time exchange of messages and reduces communication time to the possible minimum by reducing the number of connection setups (i.e., three-way handshakes). ai assistance in care is implemented by routing user requests to separately hosted nlp models abstracted into skills: high-level machinereadable specifications of assistance functionalities including inputs, outputs and model configurations @xcite . the current implementation of nlp assistance in care is built on top of the huggingface pipeline, making it easy to integrate a wide range of pre-trained models; we provide sample code to facilitate building selfregistering docker containers for nlp model deployment. the interactions between the server and the nlp models is mediated by a broker system which distributes user requests among nlp models depending on the necessary skill. ## user study to evaluate and refine the reading environment of care in the context of a collaborative applied task (requirements a and b), and to ensure the data export functionality (c) and the extensibility (d) of the system, we have extended the base configuration of care to accommodate a custom reading scenario and conducted a user study. we describe the core components of the study here and refer to the appendix b.1 for details. task scholarly peer review is a prototypical example of close reading accompanied by notetaking, where an expert assesses a manuscript in terms of its originality, readability, validity and impact @xcite . we adopted critical reading that takes place during peer review as a basis for our task. the participants of the study were provided with a manuscript-to-review and instructed to leave self-contained annotations on the manuscript while reading. to incentivize reviewers to perform the task rigorously, we simulated a subsequent acceptance-decision-making phase based on the provided annotations. to support the scenario, we extended care to allow reviewer-paper assignment and decision-making functionality. ## care and ai assistance data collection care enables the collection of inline commentary data that can be used to study inline commentaries and to create new datasets for nlp assistance model development. the collaboration functionality of care allows gathering the data about reader interactions within the tool, and the support for free-form tagging and controlled labeling offers great opportunities for collecting user-generated silver data for model pre-training and fine-tuning. assisted reading out of the box, care supports integration of any pre-existing huggingface transformer model into the reading workflow, which opens a wide range of possibilities for applying previously developed models "in the wild". to provide feedback to the reader, a pre-trained model can be used to enrich inline commentaries with labels, i.e. prompting the reader to provide additional detail, assessing the politeness @xcite , specificity @xcite or sentiment @xcite of a commentary. in addition, the power and flexibility of modern generative models like t5 @xcite allow performing a wide range of text-to-text tasks to assist reading, from question answering to summarization of highlighted passages, with the results rendered as automatically generated replies to the user's inline commentaries. the care repository provides sample code for nlp model integration. ## conclusion and future work this paper has presented care -a new open platform for the study of inline commentary and ai-assisted reading. care enables efficient inline commentary and behavioral data collection for nlp, and supports a wide range of collaborative reading scenarios, while requiring minimal effort to use. the extensible nlp assistance interface allows using care for rapid prototyping and extrinsic evaluation of nlp modules that support reading and text-based collaboration. planned extensions of care include support for non-pdf document processing and automatic text highlighting, improved human-in-the-loop functionality and scalability, as well as further development of the onboard behavioral metrics. we invite the community to use our tool and contribute to its further development 20 . the experiments performed in this study involved human participants who gave explicit consent to the study participation and to the storage, modification and distribution of the collected data. the arbitrary username selection by the users ensured that the behavioral data did not allow any association with the participants unless they decided to reveal this information. we report the demographic distribution of the participants in the appendix. the documents used in the study are distributed under an open license. although we have attempted to reflect the reading-for-peer-review workflow as closely as possible, we note that the study might still not be fully representative of the reading practice during peer review, as the participants were strictly limited in time to perform the task, and the selected papers were not necessarily from the participants' domains of specialist expertise. any application of ai to assisting humans in performing real-world tasks bears risk. we stress the need to control for bias, harmful content and factuality of the ai models used to assist reading and text work -especially in the case of large pretrained generative models. we deem it equally important to educate the users of ai-assisted reading tools about the limitations and risks associated with the integrated assistance models. from the data collection perspective, we note that all data collected with care is humangenerated personal data, in particular the behavioral data. we thus require the users of the tool to provide explicit informed consent on the data collection upon registration. in addition, the users must explicitly agree with the optional collection of behavioral statistics before any of this data is transferred to the server (opt-in). we stress that while sufficient for controlled studies, in a real application environment these measures would need to be extended by allowing the users to change their decision at a later point and specify the parts of their data that are included into data collection. from the privacy perspective, care allows registration with an arbitrary username, first and last name, e-mail and password. the choice and management of the usernames and user identities are left to the study administrator -we note that if the usernames are not assigned at random and associated with additional data, this needs to be incorporated into the informed consent form upon registration. care implements standard security measures to protect the data, and complete access to the data (documents, inline commentaries, behavioral data) is restricted to the application and server administrator. the security mechanism of the broker and thus of the ai-assistance is currently set via a token defined during the installation of the platform. it is up to the administrator to ensure that the token is kept private, otherwise the models can be used by unwanted users. we stress that for some application scenarios -e.g. dealing with sensitive or confidential documents or performing advanced behavioral measurements -additional security measures should be considered to protect the data. user journey figure 5 illustrates a typical user journey for the reader using care. it starts with log-in or registration during which consent and licensing forms are submitted. afterwards, the user is presented with a dashboard where they can manage documents, label sets and additional settings and export inline commentary and behavioral data. each document can be opened for reading via the annotation component, where multiple users can annotate the document by leaving inline commentaries organized in a sidebar which also serves as an interface for ai assistance.