The scientific community is facing raising concerns about the reproducibility of research in many fields. To address this issue in Natural Language Processing, the CLEF eHealth 2016 lab offered a replication track together with the Clinical Information Extraction task. Herein, we report detailed results of the replication experiments carried out with the three systems submitted to the track. While all results were ultimately replicated, we found that the systems were poorly rated by analysts on documentation aspects such as "ease of understanding system requirements" (33%) and "provision of information while system is running" (33%). As a result, simple steps could be taken by system authors to increase the ease of replicability of their work, thereby increasing the ease of re-using the systems. Our experiments aim to raise the awareness of the community towards the challenges of replication and community sharing of NLP systems.
As part of a NLP challenge, REF report results on replicating experiments from three systems submitted to the CLEF eHealth track.
Replicability of Research in Biomedical Natural Language Processing: a pilot evaluation for a coding task
Louhi@EMNLP
Computer Science