  Precision medicine has the potential to revolutionize healthcare, but much of
the data for patients is locked away in unstructured free-text, limiting
research and delivery of effective personalized treatments. Generating large
annotated datasets for information extraction from clinical notes is often
challenging and expensive due to the high level of expertise needed for high
quality annotations. To enable natural language processing for small dataset
sizes, we develop a novel enriched hierarchical annotation scheme and
algorithm, Supervised Line Attention (SLA), and apply this algorithm to
predicting categorical tumor attributes from kidney and colon cancer pathology
reports from the University of California San Francisco (UCSF). Whereas
previous work only annotated document level labels, we in addition ask the
annotators to enrich the traditional label by asking them to also highlight the
relevant line or potentially lines for the final label, which leads to a 20%
increase of annotation time required per document. With the enriched
annotations, we develop a simple and interpretable machine learning algorithm
that first predicts the relevant lines in the document and then predicts the
tumor attribute. Our results show across the small dataset sizes of 32, 64,
128, and 186 labeled documents per cancer, SLA only requires half the number of
labeled documents as state-of-the-art methods to achieve similar or better
micro-f1 and macro-f1 scores for the vast majority of comparisons that we made.
Accounting for the increased annotation time, this leads to a 40% reduction in
total annotation time over the state of the art.
