AxCell : Automatic Extraction of Results from Machine Learning Papers (2020)

Abstract: Tracking progress in machine learning has become increasingly difficult with the recent explosion in the number of papers. In this paper, we present AxCell, an automatic machine learning pipeline for extracting results from papers. AxCell uses several novel components, including a table segmentation subtask, to learn relevant structural knowledge that aids extraction. When compared with existing methods, our approach significantly improves the state of the art for results extraction. We also release a structured, annotated dataset for training models for results extraction, and a dataset for evaluating the performance of models on this task. Lastly, we show the viability of our approach enables it to be used for semi-automated results extraction in production, suggesting our improvements make this task practically viable for the first time. Code is available on GitHub.

## introduction machine learning studies how machines learn with respect to a task, a performance metric, and a dataset @xcite . the (task, dataset, metric name, metric value) tuple can therefore be seen as representing a single result of a machine learning paper. to make progress as a field we need to make comparisons between results achieved with different methodologies. in light of the explosion in the number of machine learning publications in recent years, such comparisons have become more difficult. 2 this poses serious challenges to peer review, among others. for instance, across ten language modelling papers submitted to iclr 2018, the perplexity score of the best baseline differed by more than 50 points @xcite . one way to deal with the deluge of papers is to develop automatic approaches for extracting results from papers and aggregating them into leaderboards. authors typically publish their results in a tabular format in the paper, including a selection of comparisons between their approach and past papers. automatic extraction of result tuples from tables-and optionally metadata such as model names-enables a full comparison between published methods. online leaderboards for comparison have become increasingly common in the research community. but these are only available for a few tasks and do not aid the comparison of models across tasks. to fill the gap, result aggregation tools such as papers with code foot_3 and nlp-progress foot_4 utilise crowdsourced community contributions to populate paper leaderboards. however, human annotation of results can be laborious and error-prone, leading to omission or misreporting of paper results. automating at least some parts of the process can speed-up the annotation, reduce number of errors and lower the expert knowledge required to correctly annotate a paper. this motivates the need for a machine learning approach to create a comprehensive results resource for the field. existing state-of-the-art approaches for results extraction are brittle and noisy, relying on text formatting hints and tables extraction from pdf files @xcite . in contrast, we propose axcell, a pipeline for automatic extraction of results from machine learning papers. axcell breaks down the results extraction task into several subtasks including table type classification, table semantic segmentation and linking results to leaderboards. we employ an ulmfit-based classifier architecture @xcite to make full use of paper and table context to interpret tabular content, and extract results accordingly. as a whole, this paper makes three main contributions to the literature. first, we significantly improve over the state-of-the-art for results extraction with our axcell system. on the subset of the nlp-tdms dataset of @xcite where l a t e x code is available, our approach achieves a micro f 1 score of 25.8 compared to the state of the art of 7.5. secondly, we release a structured, annotated dataset for training models for results extraction, and an evaluation dataset for evaluating the performance of models on this task. lastly, our approach is used in an in-production setting at paperswithcode.com to semi-automatically (by aiding the human review) extract results from papers and track progress in machine learning. closer to our formulation, @xcite extract absolute metric values alongside the metric name, task and dataset. they also use text excerpts as well as direct tabular information to make inferences for table contents. they frame extraction as a natural language inference problem and apply an nli model based on a bert architecture @xcite to extract results from pdf files. the disadvantage of this approach is that using pdfs leads to a lot of noise in structural information such as the partition of a table into cells. in our work, we explicitly utilise the structural information from the l a t e x source to extract entire tables in order to perform semantic segmentation. we demonstrate that this structural information and segmentation are crucial for boosting extraction performance. ## our approach the task of paper results extraction is to take a machine learning paper as an input and extract results contained within the paper, specifically tuples of the form (task, dataset, metric name, metric value). as an example, if we were to take the efficient-net paper of @xcite as an input, some example results tuples we would want to extract would be (image classification, imagenet, top 1 accuracy, 84.4%), (image classification, ima-genet, top 5 accuracy, 97.1%) and (image classification, stanford cars, accuracy, 94.7%). to tackle this problem effectively we define subtasks that take us from paper to results. in particular, we introduce the axcell pipeline that consists of the following subtasks: (i) table type classification, identifying whether a table in a paper has relevant results; (ii) table segmentation, segmenting and classifying table cells according to whether they hold metrics, datasets, models, etc.; and (iii) linking results to leaderboards, taking the result tuples and matching them to an existing leaderboard of results. the end-to-end system is shown in figure 1 with reference to an example. we now introduce the different components of ax-cell. ## dataset in this section we explain the datasets we used for training and evaluating axcell for results extraction. the primary input we use for a training dataset is l a t e x source code of machine learning papers from arxiv.org. over 90% of considered papers have source code available. this allows us to obtain a high quality dataset without common artifacts that arise from extracting data directly from pdf files. for training our models we use two main datasets: we manually tune the linking and filtering performance of our method using a validation dataset: lastly we evaluate the end-to-end performance of axcell on our test set: we now describe in detail these datasets. ## performance studies due to working with machine learning papers from multiple domains (from cv to nlp to biology) and a multistep approach (where errors compound) the errors are characterized by a long-tail distribution and it is difficult to pin-point the biggest source of errors. in this section, we analyze the various steps of axcell in order to better understand their relative importance. ## future work we cover three possible extensions to our work for future research. first, we might want to consider methods that retrieve all results rather than just the principal results introduced in the paper. this includes extracting ablation studies to enable search over fine-grained comparison results. secondly, we could look more into automatic taxonomy discovery. currently, we assume a closeddomain approach with a taxonomy of leaderboards known in advance. while manually extending the taxonomy requires only adding the task, dataset and metric names, it becomes problematic to cover a large fraction of papers due to publication rate and long tail of leaderboards. finally, to relax the necessity of axcell to have access to l a t e x source we consider using the arxivpapers dataset as a corpus to train extraction working directly with pdf files. ## conclusions we presented a pipeline for extracting results from machine learning papers. our method performs well across various tasks and leaderboards within machine learning, with a taxonomy that can be easily extended without retraining. additionally we released a new collection of datasets for training and evaluating on the results extraction task. these datasets enable the training of more fine-grained feature extractors and detailed error analysis. we demonstrated that our approach achieves significant performance gains over the state-of-the-art. future work may want to build on our approach for more comprehensive extraction tasks, focussing on more types of result, as well as other information contained in papers such as architectural details and hyperparameters. tion. additionally, arxiv.org provided metadata of submitted papers is available in public domain. as a consequence of legal requirements we are not able to fully publish the dataset of articles in a ready to use form, with extracted texts and tables. in order to make research in this area reproducible and results comparable, we publish our extraction pipeline and detailed information of extraction results. in particular, each paper contained in the arxivpapers dataset includes the following fields: extraction of texts and tables from papers was run on a single machine with 48 cores / 96 threads cpu with 2.5 ghz base clock. see the corresponding jupyter notebooks for the average extraction time.