Simplifying annotation of intersections in time normalization annotation: exploring syntactic and semantic validation (2021)

Abstract: While annotating normalized times in food security documents, we found that the semantically compositional annotation for time normalization (SCATE) scheme required several near-duplicate annotations to get the correct semantics for expressions like Nov. 7th to 11th 2021. To reduce this problem, we explored replacing SCATE’s Sub-Interval property with a Super-Interval property, that is, making the smallest units (e.g., 7th and 11th) rather than the largest units (e.g., 2021) the heads of the intersection chains. To ensure that the semantics of annotated time intervals remained unaltered despite our changes to the syntax of the annotation scheme, we applied several different techniques to validate our changes. These validation techniques detected and allowed us to resolve several important bugs in our automated translation from Sub-Interval to Super-Interval syntax.

## introduction time normalization is the task of translating natural language expressions of time, e.g., three days ago, to computer-readable forms, e.g., 2021-11-04. time normalization is an important component of applications like monitoring patient symptoms @xcite , matching news events across languages @xcite , and studying datebased literature trends @xcite . as part of a larger project studying the causes and effects of food insecurity, we were interested in annotating normalized times in this highly timesensitive domain. several schemes have been proposed for annotating normalized times. timeml @xcite primarily focuses on times that can be described as a prefix of yyyy-mm-ddthh:mm:ss, including relative time expressions like next monday, but excluding time expressions like the past three summers that refer to more than one time interval. the semantically compositional annotation for time normalization (scate; @xcite scheme breaks time expressions down into individual temporal operators like next or between, and can therefore cover a wider variety of time expressions than timeml. the time event ontology (teo; @xcite draws ideas from both timeml and scate, to provide a simplified annotation scheme specifically targeted at clinical notes. because the scate scheme covered the widest variety of time expressions, we selected this scheme for annotating normalized times in food security documents. we annotated 17 documents (22k words), producing 2305 time annotations, and achieved an acceptable inter-annotator agreement of 0.808 f1. however, during the process we noticed that to get the correct temporal semantics of expressions like nov. 7th to 11th 2021, we had to add many near-duplicate annotations. under the scate scheme, to get this expression interpreted correctly as [2021-11-07 00:00:00, 2021-11-12 00:00:00), two month-of-year annotations are needed on nov. and two year annotations are needed on 2021, as shown in fig. 1 @xcite . these duplicate annotations are necessary because the between needs two intervals, one for 2021-11-07 and one for 2021-11-11, and a year with a chain of sub-interval links can represent only a single interval. for example, the top-most year in the figure, based on its chain of sub-interval links, is interpreted as 2021-11-11. because these types of time expressions were common in our food security documents, we explored a possible change to the scate scheme which would replace sub-interval links with super-interval links. in essence, we would reverse the links in the chains, removing the need for the additional annotations, as shown in figure 1 is linked to a different day and therefore cannot be shared, in the super-interval version the month-of-year can be shared because it is linked only to the year, which is also shared. our proposed change to scate is purely syntactic; if implemented correctly, it should not change the semantics of any complete time expression. but sub-interval is extremely common in scateannotated documents, such as the semeval 2018 task 6 annotated corpus @xcite , so the conversion from sub-interval to super-interval would have to be automatic, not manual. we thus designed an automatic conversion tool, and a series of validations to ensure that our conversions behaved as expected. because scate, in addition to defining a set of annotations syntactically, also defines a formal semantics for each annotation, we were able to validate our tool in two different ways: syntactically, ensuring that the xml annotations were valid scate annotations and lost no information, and semantically, ensuring that the time intervals inferrable from the scate annotations were identical before and after the conversion. to convert between anafora xml and label-studio json, we created an xml-to-json script that translated @xmath0. the structure of these json objects follows the labelstudio requirements. the objects for type and text-based properties have an id field, give the name via from_name, and give the span and value in the value sub-object. the objects for link-based properties have from_id and to_id fields and give the name via labels. figure 2 shows a sample of the original anafora xml and the corresponding labelstudio json, as well as a screenshot of labelstudio visualizing the scate annotations. ## validating xml-to-json before implementing the proposed sub-interval to super-interval conversion in the json files, we tested our anafora xml to labelstudio json conversion. we first implemented a json-to-xml script designed to be an inverse of the xml-to-json script described in section 2, which collected all json objects for each id and merged them into a @xmath1. we then tested that the composition of the two scripts (i.e., xml-to-json-to-<@xmath2.1830.0611@@xmath3,31</@xmath4.1830.0611@@xmath5: { "start": 29, "end": 31, "labels": ["month-of-year"]}, "id": "51@e@abc19980114.1830.0611@gold", "from_name": "type", }, { "value": { "start": 29, "end": 31, "choices": ["january"]}, "id": "51@e@abc19980114.1830.0611@gold", "from_name": "month-of-year-type", }, { "from_id": "51@e@abc19980114.1830.0611@gold", "to_id": "50@e@abc19980114.1830.0611@gold", "type": "relation", "labels": ["sub-interval"] }, the phrase post spans the offsets 6491 to 6495 within the phrase postoperative. it is likely that an annotation over the operative was lost in the original annotations. xml) produced anafora xml that was equivalent to the original anafora xml. performing this type of syntactic validation revealed a number of errors. annotation errors we found 43 cases in the semeval 2018 task 6 annotated corpus where there were errors in the original anafora xml. for example, fig. 3 shows @xmath6, but the document contains no entity with id 326@e@id159_clinic_470@gold. these were revealed as exceptions in the xml-to-json script when it attempted to look up the non-existing entity. properties with multiple values almost all scate annotations allow only a single value for each property. however, our xml-to-json-to-xml validation revealed that a small number of annotations in the semeval 2018 task 6 corpus had properties with multiple values. for example, as shown in fig. 4 , and in the phrase mondays and tuesdays was annotated as a union with @xmath7, mondays and tuesdays. our original code assumed one value per property, resulting in missing links in the generated xml. we resolved the issue by ensuring that properties with multiple values resulted in multiple json objects instead of just one. annotations with multiple spans almost all scate annotations are over just a single, continuous span of text. however, our xml-to-jsonto-xml validation revealed that a small number of annotations in the semeval 2018 task 6 corpus had multiple spans. for example, in fig. 5 , the event status, stable (an event from a clinical note that describes the state of the patient) was annotated to exclude the comma and space characters. similar problems arise in our food security data, where expressions like kiremt 2016 rainy season needs a discontinuous span for the season kiremt . . . rainy season. we resolved this issue by creating an extra json object for each additional span, where the id is suffixed with continued to allow for easy reconstruction of the multi-span annotation. ## validating sub-to-super-interval having validated our xml-to-json conversion, we implemented our sub-interval to super-interval conversion. our initial implementation simply changed the labels from sub-interval to super-interval and swapped the from_id and to_id fields. to perform semantic validation of this conversion, we took advantage of the interpretation api provided by @xcite 2 , which takes annotations as input and produces time intervals as output. for example, the between annotation in fig. 1 would be interpreted as the interval [2021-11-07 00:00:00, 2021-11-12 00:00:00). ## discussion the change from sub-interval to super-interval achieved our primary goal, successfully removing 69 near-duplicate annotations in our food security corpus, without changing the semantics of any time expression. there was only one place where the switch caused a new nearduplicate annotation to be added: meher 2016/17. the meher here refers to two different seasons, may to september 2016 and may to september 2017. with sub-intervals, both 2016 and 17 could link to the same meher season-of-year, but with super-intervals, there must be two season-of-years annotated, each pointing their super-interval to exactly one of 2016 or 17. we nonetheless consider the experiment a success: 69 near-duplicates removed, at the cost of just 1 near-duplicate added. in terms of validation strategies, we found it helpful to be working with an annotation scheme that had both a syntactic specification of how annotations should be applied to words in the text, and a formal semantic interpretation that converted the annotations to explicit intervals on the timeline. this allowed us to change the syntax of the annotation scheme, while making sure that we did not unintentionally change the semantics of the annotations. having both a syntactic specification and a formal semantic interpretation is uncommon in annotation schemes. at one end of the spectrum are purely syntactic annotation schemes, like universal dependencies (ud; @xcite , with no formal semantic interpretation at all. at the other end of the spectrum are purely semantic annota-109 tion schemes, like abstract meaning representation (amr; @xcite where only the formal logic-like interpretation is annotated, with no explicit links to individual words of the original text. our experience was that having both a syntactic and semantic specification made it easier to apply and validate improvements to the annotation scheme. all code produced during this work can be found at https://github.com/clulab/timenorm/ .