Is NLP Ready for Standardization? (2022)

Abstract: While standardization is a well-established activity in other scientific fields such as telecommunications, networks or multimedia, in the field of AI and more specifically NLP it is still at its dawn. In this paper, we explore how various aspects of NLP (evaluation, data, tasks...) lack standards and how that can impact science, but also the society, the industry, and regulations. We argue that the numerous initiatives to rationalize the field and establish good practices are only the first step, and developing formal standards remains needed to bring further clarity to NLP research and industry, at a time where this community faces various crises regarding ethics or reproducibility. We thus encourage NLP researchers to contribute to existing and upcoming standardization projects, so that they can express their needs and concerns, while sharing their expertise.

## introduction most of the natural language processing community remains estranged from standardization. as this is already common practice in many computer science fields, including telecommunications, networks and multimedia, what is making nlp so special in that regard? @xcite has already asked the question of the potential barriers and benefits of standardization work in the broader field of artificial intelligence, which is now becoming a reality. here we propose to deepen that discussion by investigating the more specific context of nlp and its standardization needs. standards are normative documents produced by standards developing organizations (sdos) such as iso. in practice, they can be of various nature and content. some of them are terminological references that establish shared terms and definitions for a technical domain. for instance, the iso online browsing platform 1 indexes all 1 https://www.iso.org/obp existing iso definitions. other standards rather describe a reference framework, which can prove useful for bootstrapping new activities or rationalizing existing ones. standards can also provide technical specifications for data, systems or procedures. this notably includes quality specifications, as well as interoperability ones @xcite . for instance, the c language (iso/iec 9899), the mp3 coding (iso/iec 11172-3), the latin-1 charset (iso/iec 8859-1) and the 2-letter language codes (iso 639-1) are all examples of standards. technical specificiations are often associated with certification, and test suites can be developed to assess compliance with standards offering sufficient formalism (e.g. syntax checkers, protocol testing, or dedicated measurements against standard thresholds). standards are written by volunteer experts from various backgrounds (scientific, legal, standardization experts, etc.). in most sdos, registration is open to anyone willing to contribute, usually through a mirror committee within their national standards organization. experts collaborate within working groups and decisions are taken by consensus foot_0 -across countries, but also across backgrounds, across sectors, across technical fields. this approach makes standardization a rather slow process (with up to 3 years to establish some standards), but it also ensures the strength of the agreements. most sdos plan a mandatory revision of published standards every few years, without which they are declared obsolete and withdrawn, in order to ensure that standards remain up to date with technological and societal evolutions. standards are especially important for the indus-try and for regulatory authorities -but they can apply on very technical fields, including scientific topics, and therefore affect the research community as well. this paper investigates how nlp standardization could impact our community, offering both challenges and opportunities. after a brief review of the current state of nlp standardization initiatives ( §2), we explore standardization gaps pertaining to nlp evaluation ( §3), data and formats ( §4), tasks ( §5) and higher-level concepts ( §6). having built a broader view of how nlp standardization could benefit research, but also the society, industry and regulations ( §7), we then conclude on possible contributions that nlp researchers could add to those efforts ( §8). ## existing initiatives towards nlp standardization within the nlp community, most of the standardized material is actually de facto standards: data, tools or methodology that are consensually used throughout the field, even though they don't have any official status and have not necessarily gone through the formalization process that official standards offer. such de facto standards often result from past shared tasks: for instance, the mteval-v13a.pl evaluation script from the wmt shared tasks series has been used for years as the reference evaluation script by a large part of machine translation research. similarly, the conll-x format for dependency treebanks has been widely adopted following the corresponding conll shared task @xcite . as for event detection, the definition of the task itself is fully driven by the ace 2005 campaign @xcite , to the point that it is sometimes referred to as "ace event detection" @xcite . recent years have seen however the growth of the universal dependencies initiative @xcite , for establishing common guidelines for treebank annotation across languages; considering the breadth of its contributors and its sustained efforts for guidelines formalization, this project has now become very close to standardization work and could be considered as an sdo. regarding official standardization initiatives relevant for nlp, the most established one is iso's technical committee 37 (language and terminology), and more prominently its subcommittees 4 (language resource management, created in 2001) foot_1 and 5 (translation, interpreting and related technology, created in 2012). with a strong focus on corpora and annotation, these groups have notably released a number of annotation framework standards (e.g. for tiger-xml or tei), which are extensively used by the corresponding industry; they also co-organize with acl the isa workshop series on interoperable semantic annotation @xcite . yet this focus on data leaves the algorithmic and evaluation parts of nlp largely unaddressed. iso-iec's joint technical committee 1 (information technology) has created in 2017 its subcommittee 42 on artificial intelligence. this one is more concerned with algorithms, development methodology, and system evaluation, but at the higher level of ai in general and not delving into nlp-specific aspects. to date, its nlp-related activity has focused on defining a few major concepts, such as nlg, question answering, or ocr. concurrently, other global sdos such as itu-t have also explored nlp standardization, but mostly in the context of specific use cases (e.g. itu-t h.862.5 "emotion enabled multimodal user interface based on artificial neural networks") rather than the nlp field in general. hence their remains a gap in terms of nlp standardization. the european counterparts of iso-iec (cen-cenelec) have thus created in 2021 their own joint technical committee 21 on artificial intelligence, with a group dedicated to kickstarting activities on speech and nlp (ad-hoc group 4, ai systems for human language processing, on track to become persistent in 2022). this is the first page of a new chapter, roadmaps are being written today. now is thus the right time to question what can, and should, be standardized within our field. ## is nlp evaluation ready for standardization? the reproducibility crisis that has spread through the field in recent years @xcite has renewed the community's interest for fair and reliable evaluation. this has led in the 2020s to a blooming of workshops and shared tasks dedicated to evaluation means (whether human evaluation or automated metrics), such as eval4nlp, humeval, gem, or benchmarking: past, present and future @xcite @xcite . but those concerns are not new, as illustrated by the 4real workshops organized in the 2010s @xcite . even the terminology of reproducibility has been the topic of debate and clarification attempts for many years in the machine learning community @xcite . the existence of the lrec conference series itself is a token of that interest for standardized evaluation, with its first edition dedicating a whole workshop to the lack of a shared strategy, definition and infrastructure for system evaluation @xcite . for domains like human evaluation of nlg, the need for more consistent practices is clear enough, and @xcite have already advocated for producing standards on both the methodology and the terminology, based on their review of 20 years of nlg with conflicting evaluation criteria. 4 yet here we argue that even cases with seemingly straightforward automated evaluation can suffer today from the lack of standards. ## on the benefits of nlp standardization looking at the long-term impact of the universal dependencies project hints at how standardizing nlp could more generally benefit the field and its dynamics. the immediate benefit was a more faithful evaluation across languages, that enabled deeper investigation of cross-lingual transfer. but the existence of common guidelines also created a commu-nity incentive to produce more data, by providing the guidance and means for easier extension to dozens of zero-resourced languages. the creation of the project itself opened new fora for communitylevel collaboration and sharing, thereby giving dependency research a new boost. this has been an opportunity to reflect upon research practices, fostering more systematic studies on annotation scheme impact, better highlighting the gaps in linguistic coverage, and uncovering biases in our view of syntax. overall, standardization contributes to better driving research, both at the individual and institutional levels. a clear taxonomy makes it easier to identify scientific gaps, more compatible resources and tools offer richer experimental means, and shared definitions guarantee that we are all pushing in the same direction. standards also support community-wide adoption of good practices. even if abundantly discussed and well documented as checklists, especially regarding evaluation @xcite @xcite @xcite and documentation @xcite @xcite @xcite , consensual good practices guidelines are not necessarily implemented in practice, in research and even more crucially in industry. escalating them to formal standards makes it easier to enforce them. comparability is another clear benefit for nlp researchers, but even more so for users and consumers. interoperability can facilitate putting a researcher's ideas into users' hands, with easier integration into products. but it is also a matter of survival for smes, for packaging and distributing their products in a competitive environment where big tech standalone solutions dominate the market and smes struggle to propose large-scale alternatives. last but not least, standardizing nlp concepts is a necessary step to refine a shared roadmap to address ethical considerations in nlp @xcite -but it is also a prerequisite to regulating abuses and enforcing safe use of nlp that preserves individual rights. at a time where the eu is establishing its ai act, it has started collaborating with cen-cenelec to bring clarity on the terminology and processes needed to legally ensure key trust characteristics, such as robustness, trans-parency or fairness. standards used to support compliance with the ai act will thus be written by cen-cenelec/jtc 21 over the next few years. in that context, it is crucial that nlp standards (not only ai generic standards that do not fully apply to nlp specificities) are developed, to ensure that companies distributing nlp products in europe are able to comply with the regulation. this review of various aspects of the nlp ecosystem has shown how the lack of standards can cause confusion, inefficiency, and sometimes even render research efforts detrimental to scientific progress, through misinsterpretations and fostering of bad practices. building and formalizing consensus on key practices and concepts would enable instead a more reproducible, more insightful, more industryready and more ethical science. admittedly, not everything can be readily standardized: sometimes the scientific material to do so does not even exist yet. and sometimes research freedom and creativity need to be preserved by maintaining concurrent options. but these are precisely cases where it is even more important that the standardization ecosystem benefits from scientific expertise, in order to avoid over-standardizing the field, or widening the discrepancies between research and industry practices. we believe it is a matter of scientific responsibility to offer such guidance to those who are shaping the industrial and legal future of society-wide use of nlp. contributing to standardization means sharing our expertise and insights, but also our needs and our concerns, both as scientists and as citizens. nlp is ready and in need -now we have to get ready. there are numerous ways to taking part. while community-internal initiatives should be pursued and fostered, we also encourage european researchers to join cen-cenelec/jtc 21 foot_6 for contributing to its budding roadmap, and worldwide researchers to both pursue resource standardization efforts within iso/tc 37 foot_7 and help iso-iec/jtc 1/sc 42 foot_8 to deepen debates that have still only scratched the surface of the upcoming work. organize events, discuss, share, debate, draft, brainstorm, publish. and nlp standards will be within reach.