A G it H ub-based Workflow for Annotated Resource Development (2025)

Abstract: Computational linguists have long recognized the value of version control systems such as Git (and related platforms, e.g., GitHub) when it comes to managing and distributing computer code. However, the benefits of version control remain under-explored for a central activity within computational linguistics: the development of annotated natural language resources. We argue that researchers can employ version control practices to make development workflows more transparent, efficient, consistent, and participatory. We report a proof-of-concept, GitHub-based solution which facilitated the creation of a legal English treebank.

## introduction linguistic annotation is an important pillar of the empirical enterprise that supports modern computational linguistics. a recent review notes that "corpus resources... remain highly relevant for testing and studying @xcite systems" @xcite , even as these resources take a less central role in system training. by augmenting corpus data with high-quality annotations, "people skilled at language analysis can ensure meaningful evaluation of nlp systems" (ibid). however, creating a valuable annotated dataset is time-consuming and labor-intensive, and some common practices can undermine the usefulness and quality of the end result. for example, behind each "gold" annotation may be several non-trivial analytical decisions reached through careful adjudication. unfortunately, researchers tend not to make, or publicly share, detailed records of these processes. as a result of low project transparency, dataset users may have no way of determining the original justification for a given annotation. moreover, linguistic annotation practices tend to vary widely in terms of the assistive tools made available to annotators. providing annotators with access to tools that automatically visualize and/or validate annotations can facilitate more efficient and more consistent (i.e., less error-prone) resource development @xcite . however, there are high overhead costs for creating such tools from scratch, meaning that less mature annotation projects are often pursued with more primitive annotation technologies. finally, not all workflows permit the kinds of robust community participation that help to sustain linguistic annotation projects over time. though most projects are sustained primarily by the efforts of a core development team, outside researchers can make valuable contributions by identifying annotation errors or adding new annotations. to make full use of these non-core contributors, it is desirable to develop resources on platforms that facilitate open communication between a project's core developers and the broader research community. we argue that researchers can address these issues with resource development workflows that employ version control systems (such as git) and online services for interacting with such systems (such as github). though computational linguists have long recognized the value of version control for managing and distributing computer code, we demonstrate that version control systems and services also serve to make linguistic annotation procedures more transparent, efficient, consistent, and participatory. in what follows, we recap the core principles behind version control generally and git/github in particular. we then present our github-based annotation workflow in general form. next, we report a proof-of-concept implementation, which facilitated the creation of a treebank of legal english. ## version control and git/github in this section, we briefly review the concept of a version control system (vcs) and the core principles underlying git/github, with a focus on prop-erties that facilitate our proposed workflow. a vcs records changes to a file repository over time, allowing teams to track modifications, compare versions, and revert to previous states when needed. vcs adoption enables developers to create and modify files while maintaining a complete project history within the repository. git is a widely employed vcs. a git branch is a parallel instance of the repository with a change history that may diverge from that of the central version of the project (as reflected by the 'main' branch). branches allow project contributors to develop new features or fixes without affecting the main codebase before the changes are ready to be integrated. a git commit records the changes made to repository files at a specific point in time. each commit contains a unique hash identifier and includes a message describing the changes made. commits create a traceable history of modifications, allowing viewers to understand when and why particular changes were implemented. github is a web-based hosting service for managing and sharing git projects. while git provides the foundational version control capabilities, github extends these with a social platform that enables web-based collaboration. on github, pull requests enable developers to propose changes from their working branch to the main branch. pull requests serve as a collaborative space where team members can review file changes, provide feedback, and discuss modifications before changes are merged from a working branch to the main branch. github actions specify automated procedures triggered by repository events (such as commits or pull requests). actions serve to automate repetitive tasks such as testing code or writing files. ## application to linguistic annotation notably, github has already proven to be valuable for large-scale linguistic annotation projects such as universal dependencies (de marneffe et al., 2021), which employs github as a forum for discussing annotation guidelines and as a tool for maintaining existing datasets. 1 our proposed workflow goes a step further by integrating github directly at the resource development stage. this level of integration results in a comprehensive record of annotation decisions (and annotator discussions) for each individual annotation in the dataset. this workflow (figure 1 ) starts with two conceptual roles performed by project participants: the annotator role and the manager role. 2 the manager organizes the annotation project by populating a subdirectory of the repository with "stub" entries. these entries include pre-annotated text, possibly with some pre-processing (e.g., tokenization). these entries, and/or their associated filenames, may also include project-relevant metadata. from the main github branch where stub entries reside, the annotator creates a working branch. 3 within this working branch, the annotator completes stub entries, adding annotations according to the project guidelines. each time an annotator commits changes to their working branch, two github actions are automatically triggered: a visualization action and a validation action. the visualization action creates a graphical representation of the annotated data and commits it to the annotator's branch. the validation action triggers a script that heuristically verifies that the annotation conforms to conventions of the annotation schema. when an annotator completes their annotations, they initiate a pull request to merge their changes back into the main branch. the manager reviews the pull request. this review is facilitated by the action-generated graphical representation, which enables the manager to inspect the proposed contribution without having to manually read through the raw text of the annotation file. the manager and annotator can also review the output of the validation action to ensure the annotation is well-formed. the manager and annotator can discuss the proposed contributions by leaving comments on the pull request. ultimately, the manager has two options: approve the changes and merge them into the main branch, or request additional edits from the annotator. in the latter case, the annotator makes edits on the annotator branch and then requests a subsequent review from the manager. upon successful merging of annotated entries into the main branch, a statistics action is automatically triggered. this process updates project statistics, which may include information about overall project progress or summary statistics of the annotations themselves. in what follows, we show that this workflow can be implemented in a way that promotes the four values presented in section 1: transparency, consistency, efficiency and community participation. ## demonstration: treebanking we applied this workflow while developing a treebank of legal us english in cgelbank @xcite , a treebanking formalism that extends the descriptive theory of english syntax presented in the cambridge grammar of the english language (cgel, @xcite . the core team consisted of five researchers. each team member performed the tasks of the annotator role, while the tasks of the manager role were performed primarily by the two senior members of the team. one member working in the manager role populated the main branch with stub files in the project-native .cgel data format (figure 2 ; see @xcite 5 for more discussion), with each file corresponding to one sentence of the treebank. in addition to the raw sentence text and other relevant metadata, each stub file contained an automated tokenization of the sentence. the annotated sentences were sourced from us federal statutes as compiled in the us code by the office of the law revision counsel (olrc) of the us house of representatives. 4 the olrc publishes the us code in xml format according to a standardized schema known as united states legislative markup (uslm). each sentence of the # @xmath0... # @xmath1:det (dp :head (d :t "the")) :head (nom :head (n :t "attorney") :mod (adjp :head (adj :t "general")))) figure 2 : example of the .cgel data format, illustrating analysis of the noun phrase the attorney general. for each sentence, the assigned annotator created a new working branch from the main branch of the project's github-hosted repository. the annotator then manually corrected the automated tokenization and added lemma and part-of-speech tags according to cgelbank conventions @xcite . tree editing was facilitated by active-dop (van cranenburgh, 2018), a browser-based graphical treebanking tool which utilizes an active learning parser @xcite . to enable editing of .cgel-format trees, we extended a cgelbank-customized version of ac-tivedop reported by @xcite . once the annotator was finished using the tool, they exported the .cgel-format tree from activedop and appended it to the corresponding stub file. the an-notator then saved and committed their file changes to their working branch. some annotators opted to interface with git from the command line (and subsequently 'push' their commits to the project's github repository), while others utilized github's built-in text editor user interface to edit and commit changes directly from their web browser. once the annotator's changes had been committed to their working branch on github, a visualization action automatically generated a l a t e x rendering of the .cgel-format tree as a .pdf file and committed that file to the working branch. a second validation action verified that the tree did not have any obvious errors. the annotator then opened a pull request on the main branch. another team member, assuming the manager role, reviewed the pull request by inspecting the changed files. the l a t e x rendering provided the reviewer with a convenient, easy-to-read graphical representation of the user's annotation. the reviewer and annotator could discuss the annotation through comments left on the pull request. in the event that the reviewer requested changes, the annotator could modify the relevant .cgel file, which automatically re-triggered the visualization action to update the l a t e x .pdf of the tree. this procedure is partly illustrated in figure 3 . once the reviewer approved the annotation and merged it to the main branch, an automaticallytriggered action generated summary statistics of the treebank, including counts of lexical nodes and category/function labels, average tree depth, and a list of high-frequency lemmas. ## discussion our project repository foot_3 is not simply a static collection of gold annotations; the repository's commit history and pull request comments also form a dynamic public record of the decision-making processes that led to that gold data. this feature of our development workflow enhances project transparency, providing future dataset users with a means of determining how we adjudicated hard cases of linguistic analysis. as a new treebanking formalism with a relatively small research community, cgelbank lacks the breadth of specialized annotation tools enjoyed by more established projects, e.g., universal dependencies (de marneffe et al., 2021). 6 we used github actions -relatively simple scripts which execute in a github repository -to deliver some of the functionality of standalone annotation tools (i.e., automated visualization and validation), in addition to using and extending a bespoke cgel-bank annotation tool. we used these actions in a way that allowed the annotator and reviewer to efficiently discuss and adjudicate a proposed annotation. these actions -especially the automated valiation -also promote consistency by enabling annotators and reviewers to quickly spot errors. lastly, the public nature of github strongly encourages community participation. anyone with a github account can comment on the project by posting a github issue (a discussion thread used to track project-related matters). the broader community can also create pull requests to suggest corrections to the dataset (or to add new data). ## related work to a limited extent, previous work has discussed the utility of version control for developing annotated linguistic resources. @xcite recommend that annotators employ a vcs protocol to promote data security and integrity as a resource is developed. @xcite implements a git-based procedure to develop a dataset of phonetic transcriptions for three indigenous australian languages. on this procedure, annotators' contributions are tracked through git commits, and git "hooks" (automated scripts) automatically re-compute corpus statistics upon merge. our proposed workflow builds on this approach by leveraging the social functionality of github to facilitate adjudication, foster community participation, and create a persistent open record of the design and analysis choices that shape the final corpus product. previous work has also explored the value of vcs technologies for maintaining previouslydeveloped resources. @xcite and @xcite discuss how version control could help research communities record (and disseminate) changes and corrections to speech corpus annotations. @xcite design and implement a vcs for managing dynamic speech corpora of the kind envisioned by rosenberg. previous work has focused largely on applying vcs protocols in the context of annotated speech corpora. to our knowledge, we report the first application of a vcs-based workflow to syntactic treebanking. however, as discussed in section 3, github already plays a significant role in the ongoing maintenance of the universal dependencies project, including as a forum for discussing errors and updates to annotation conventions. ## limitations though our workflow offers several advantages for linguistic annotation, we have not presented a quantitative comparison of annotation speed or accuracy against alternative workflows. additionally, while github actions provide useful automation, developing and maintaining custom validation and visualization scripts requires a non-trivial number of technical prerequisites, including familiarity with the yaml-based workflow syntax associated with github actions. finally, annotators unfamiliar with version control in general (or git in particular) may face a learning curve associated with the core concepts of git repository management. ## conclusion we presented a github-based workflow for linguistic annotation. we provided a proof-of-concept implementation of this workflow for syntactic treebanking, demonstrating that this workflow promotes four values that enhance the usefulness and quality of annotated linguistic resources. future work could extend this approach to other types of linguistic annotation tasks beyond treebanking, such as semantic role labeling or discourse analysis. moreover, the workflow could be adapted to support multiple independent annotations followed by adjudication, leveraging git's branching model to manage parallel annotation efforts. finally, there are opportunities to integrate github with external annotation tools through the github apps framework, foot_5 which enables thirdparty software to directly perform common github operations such as writing commits, opening/commenting on pull requests, and triggering automated workflows. in ongoing work, we are extending such functionality to activedop (van cranenburgh, 2018), the tree editor employed in our cgelbank treebanking demonstration, so that annotators can participate in a github-based workflow without leaving the annotation environment. computational linguistics continues to depend on high-quality linguistic annotation to support empirically-informed natural language analysis and data-driven system development. by embracing version control practices and technologies, we can foster more rigorous, collaborative, and sustainable approaches to this essential practice.