Original: Enterprise Alexandria: Online High-Precision Enterprise Knowledge Base Construction with Typed Entities
Revised: Enterprise Alexandria: Online High-Precision Enterprise Knowledge Base Construction with Typed Entities

Original: Anonymous authors

Original: Abstract
Revised: Abstract

Revised: We present Enterprise Alexandria, one of the core AI technologies behind Microsoft

Revised: Viva Topics 1 .

Original: We present Enterprise Alexandria, a new system for automatically constructing a knowledge base with high-precision and typed entities from private enterprise data such as emails, documents and intranet pages.
Revised: Enterprise Alexandria is a new system for automatically constructing a knowledge base with high-precision and typed entities from private enterprise data such as emails, documents and intranet pages.

Original: Built as an extension of Alexandria [Winn et al., 2019], the key novelty of Enterprise Alexandria is the ability in processing both the textual information and the structured metadata available in each document in an online learning fashion, making use of any manual curations that have happened in the interim.
Revised: Built as an extension of Alexandria [Winn et al., 2019], the key novelty of Enterprise Alexandria is the ability in processing both the textual information and the structured metadata available in each document in an online learning fashion, making use of any manual curations that have happened in the interim.

Original: This task is performed entirely eyes-oﬀ to respect the privacy of the user and the restricted access their documents.
Revised: This task is performed entirely eyes-oﬀ to respect the privacy of the user and the restricted access their documents.

Original: The knowledge discovery process uses a probabilistic program deﬁning the process of generating the data item from a set of unknown typed entities.
Revised: The knowledge discovery process uses a probabilistic program deﬁning the process of generating the data item from a set of unknown typed entities.

Original: Using probabilistic inference, Enterprise Alexandria can jointly discover a large set of entities with custom types speciﬁc to the organization.
Revised: Using probabilistic inference, Enterprise Alexandria can jointly discover a large set of entities with custom types speciﬁc to the organization.

Original: Experiments on three real-world datasets show that the system outperforms alternative methods with the ability to work eﬀectively at large scale.
Revised: Experiments on three real-world datasets show that the system outperforms alternative methods with the ability to work eﬀectively at large scale.

Original: 1. Introduction
Revised: 1. Introduction

Original: As the volume of textual information produced within organizations continues to grow dramatically [Radicati and Levenstein, 2019], it is critical for Automated Knowledge Base
Revised: As the volume of textual information produced within organizations continues to grow dramatically [Radicati and Levenstein, 2019] it is critical for Automated Knowledge Base

Original: Construction (AKBC) systems applied to enterprise scenarios to process data automatically and at large scale .
Revised: Construction (AKBC) systems applied to enterprise scenarios to process data automatically and at large scale .

Original: This is becoming a key research area in enterprise knowledge discovery [Loshin, 2001] where a key task is to turn unstructured text present in emails, documents and intranet pages into a single, continuously available, consistent and up-to-date knowledge base.
Revised: This is becoming a key research area in enterprise knowledge discovery [Loshin, 2001] where a key task is to turn unstructured text present in emails, documents and intranet pages into a single, continuously available, consistent and up-to-date knowledge base.

Original: In doing so, due to the restricted access of enterprise documents that are continuously edited by the users, the entire knowledge base must be constructed incrementally and eyes-oﬀ (i e., documents are not accessible for labeling) for each organization [Voigt and
Revised: In doing so, due to the restricted access of enterprise documents that are continuously edited by the users, the entire knowledge base must be constructed incrementally and eyes-oﬀ (i.e., documents are not accessible for labeling) for each organization [Voigt and

Original: Von dem Bussche, 2017].
Revised: Von dem Bussche, 2017].

Original: Furthermore, since any automated system will inevitably have gaps in the extracted knowledge, the management of such systems also involves the role of knowledge curators acting as a trusted source that can editorially curate or expand the mined knowledge [Maedche et al., 2003].
Revised: Furthermore, since any automated system will inevitably have gaps in the extracted knowledge, the management of such systems also involves the role of knowledge curators acting as a trusted source that can editorially curate or expand the mined knowledge [Maedche et al., 2003].

Original: Therefore it is important for an AKBC system to be designed to accommodate and learn from curated content.
Revised: Therefore it is important for an AKBC system to be designed to accommodate and learn from curated content.

Original: The aim is to achieve a virtuous cycle where people and machines continually work together to keep the knowledge accurate and complete.
Revised: The aim is to achieve a virtuous cycle where people and machines continually work together to keep the knowledge accurate and complete.

Original: A key characteristic of enterprise documents is their structured nature, including both structured text and associated metadata .
Revised: A key characteristic of enterprise documents is their structured nature, including both structured text and associated metadata .

Original: For instance, emails have a subject, sender, list of recipients and attached documents while documents have a creator and a list of editors, viewers and commentators as well as structure within the document, such as sections and titles.
Revised: For instance, emails have a subject, sender, list of recipients and attached documents while documents have a creator and a list of editors, viewers and commentators as well as structure within the document, such as sections and titles.

Original: We argue that both kinds of information are critical to build an accurate knowledge base.
Revised: We argue that both kinds of information are critical to build an accurate knowledge base.

Original: In this respect, there is a wealth of knowledge discovery methods that process text without explicitly modeling metadata [Yamada et al., 2020, Devlin et al., 2019, Zhang et al., 2019].
Revised: In this respect, there is a wealth of knowledge discovery methods that process text without explicitly modeling metadata [Yamada et al., 2020, Devlin et al., 2019, Zhang et al., 2019].

Original: Other systems can leverage the full data structure if some amount of training labels are available to infer the relationship between each structured ﬁeld (e.g. email sender) and the entity properties (e.g. project lead)
Revised: Other systems can leverage the full data structure if some amount of training labels are available to infer the relationship between each structured ﬁeld (e.g. email sender) and the entity properties (e.g. project lead)

Original: [Zhang, 2015].
Revised: [Zhang, 2015].

Original: Unfortunately, the privacy restrictions of enterprise data do not allow for their labeling by human annotators, making these methods unfeasible.
Revised: Unfortunately, the privacy restrictions of enterprise data do not allow for their labeling by human annotators, making these methods unfeasible.

Original: We address this gap by developing an automated system, Enterprise Alexandria (EA), as an extension of Alexandria [Winn et al., 2019], an existing system designed for Web-based information extraction.
Revised: We address this gap by developing an automated system, Enterprise Alexandria (EA), as an extension of Alexandria [Winn et al., 2019], an existing system designed for Web-based information extraction.

Original: EA is able to jointly discover entities and their associated types with very minimal supervision in compliance with the privacy requirements of enterprise data, which makes it the ﬁrst entity discovery method applicable to enterprise scenarios to the best of our knowledge.
Revised: EA is able to jointly discover entities and their associated types with very minimal supervision in compliance with the privacy requirements of enterprise data, which makes it the ﬁrst entity discovery method applicable to enterprise scenarios to the best of our knowledge.

Original: In EA, the probabilistic program is extended to generate both the document and its metadata from a set of latent entities.
Revised: In EA, the probabilistic program is extended to generate both the document and its metadata from a set of latent entities.

Original: Inference is partly performed automatically using an eﬃcient message-passing algorithm
Revised: Inference is partly performed automatically using an eﬃcient message-passing algorithm

Original: [Minka et al., 2018], and partly with a handwritten scalable algorithm that can process a large set of documents in batches and build the knowledge base incrementally, making use of any manual curations that have happened in the interim.
Revised: [Minka et al., 2018], and partly with a handwritten scalable algorithm that can process a large set of documents in batches and build the knowledge base incrementally, making use of any manual curations that have happened in the interim.

Original: Furthermore, EA adds an important ability to learn the set of types associated with each entity (e.g. product, event, team, project) from a pre-deﬁned type hierarchy.
Revised: Furthermore, EA adds an important ability to learn the set of types associated with each entity (e.g. product, event, team, project) from a pre-deﬁned type hierarchy.

Original: In order to ensure that the type hierarchy is suﬃcient, EA is also able to semi-automatically discover type names from enterprise data.
Revised: In order to ensure that the type hierarchy is suﬃcient, EA is also able to semi-automatically discover type names from enterprise data.

Original: The importance of determining the types of extracted entities receives support from a study of workers in two organizations, which showed that a wide variety of knowledge types are important for work, and that workers seek to engage with these diﬀerent types in diﬀerent ways.
Revised: The importance of determining the types of extracted entities receives support from a study of workers in two organizations, which showed that a wide variety of knowledge types are important for work, and that workers seek to engage with these diﬀerent types in diﬀerent ways.

Original: We argue that types are needed not just generically, but to support work in particular industries and organizations.
Revised: We argue that types are needed not just generically, but to support work in particular industries and organizations.

Original: Using real-world datasets, we show that the quality of the entities and their associated types extracted by EA outperforms the baselines by signiﬁcantly increasing the precision of the extracted knowledge.
Revised: Using real-world datasets, we show that the quality of the entities and their associated types extracted by EA outperforms the baselines by signiﬁcantly increasing the precision of the extracted knowledge.

Original: In summary, this paper provides the following contributions to the state of the art: ( i ) Findings from a qualitative study of members of two multinational organizations , which highlights how entity types are important for work.
Revised: In summary, this paper provides the following contributions to the state of the art: ( i ) Findings from a qualitative study of members of two multinational organizations, which highlights how entity types are important for work.

Original: ( ii )
Revised: ( ii )

Original: An extended probabilistic program that improves the existing Alexandria model and can perform entity discovery jointly from structured text and the associated metadata.
Revised: An extended probabilistic program that improves the existing Alexandria model and can perform entity discovery jointly from structured text and the associated metadata.

Original: ( iii )
Revised: ( iii )

Original: A new mechanism for updating the discovered knowledge incrementally with the ability to support curated content contributed by human users.
Revised: A new mechanism for updating the discovered knowledge incrementally with the ability to support curated content contributed by human users.

Original: ( iv ) A real-world evaluation showing that EA performs eﬃciently at scale while signiﬁcantly improving the precision of the extracted knowledge compared to four baselines on several real-world datasets.
Revised: ( iv ) A real-world evaluation showing that EA performs eﬃciently at scale while signiﬁcantly improving the precision of the extracted knowledge compared to four baselines on several real-world datasets.

Original: 2. User Study on Knowledge Categories
Revised: 2. User Study on Knowledge Categories

Original: To best inform the design of our model, and to gather data on real entity types as a baseline for our evaluation, we undertook a qualitative study with prospective end users who were employed in two diﬀerent industries.
Revised: To best inform the design of our model, and to gather data on real entity types as a baseline for our evaluation, we undertook a qualitative study with prospective end users who were employed in two diﬀerent industries.

Original: In the study we asked: How do organization members think about knowledge types, and what types of entities need to be discovered to support everyday work?
Revised: In the study we asked: How do organization members think about knowledge types, and what types of entities need to be discovered to support everyday work?

Original: The study was conducted in two workshops with 12 (5 = men, 7 = women) knowledge workers who were employed by one of two multinational organizations (6 participants in each).
Revised: The study was conducted in two workshops with 12 (5 = men, 7 = women) knowledge workers who were employed by one of two multinational organizations (6 participants in each).

Original: The ﬁrst organization was a large professional services ﬁrm; the second was part of the travel and tourism industry.
Revised: The ﬁrst organization was a large professional services ﬁrm; the second was part of the travel and tourism industry.

Original: Participants came from a variety of backgrounds in terms of role (e.g. designer, program manager, analyst, information architect) and ranged in seniority from junior to national leadership.
Revised: Participants came from a variety of backgrounds in terms of role (e.g. designer, program manager, analyst, information architect) and ranged in seniority from junior to national leadership.

Original: In order to provide context, participants were ﬁrst shown a short video that described how an automatically-generated knowledge base might support work.
Revised: In order to provide context, participants were ﬁrst shown a short video that described how an automatically-generated knowledge base might support work.

Original: Participants were then asked to think about the diﬀerent components that make up their work, which they might want represented in the
Revised: Participants were then asked to think about the diﬀerent components that make up their work, which they might want represented in the

Original: AKBC system.
Revised: AKBC system.

Original: Participants were given four examples of such components (project, budget, product launch, designer) in order to stimulate ideas.
Revised: Participants were given four examples of such components (project, budget, product launch, designer) in order to stimulate ideas.

Original: The study received ethical approval from the host Institutional Review Board (IRB).
Revised: The study received ethical approval from the host Institutional Review Board (IRB).

Original: Each participant provided informed consent for data collection and use prior to participation.
Revised: Each participant provided informed consent for data collection and use prior to participation.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Table 1 shows the 48 unique knowledge types generated by participants in total, which we manually grouped bottom-up (via an aﬃnity diagram) into eight superordinate classes to help summarize results.
Revised: Table 1 shows the 48 unique knowledge types generated by participants in total, which we manually grouped bottom-up (via an aﬃnity diagram) into eight superordinate classes to help summarize results.

Original: The knowledge categories were grouped based on the explicit meanings of the category names, descriptions and discussions provided by participants in the workshop.
Revised: The knowledge categories were grouped based on the explicit meanings of the category names, descriptions and discussions provided by participants in the workshop.

Original: While three of the knowledge categories reported by participants were conﬁrmations of the examples we gave to participants (project, budget, designers), the remaining 45 categories were generated by participants independently.
Revised: While three of the knowledge categories reported by participants were conﬁrmations of the examples we gave to participants (project, budget, designers), the remaining 45 categories were generated by participants independently.

Original: Rather than suggesting that all of these categories are necessary for every knowledge base, we suggest these categories capture this set of prospective users’ aspirations for a knowledge base.
Revised: Rather than suggesting that all of these categories are necessary for every knowledge base, we suggest these categories capture this set of prospective users’ aspirations for a knowledge base.

Original: Overall, this qualitative study demonstrates why types matter: a wide variety of entity types are important for work, and workers seek to engage with these diﬀerent types in diﬀerent ways.
Revised: Overall, this qualitative study demonstrates why types matter: a wide variety of entity types are important for work, and workers seek to engage with these diﬀerent types in diﬀerent ways.

Original: In the following
Revised: In the following

Original: CalendarEvent
Revised: CalendarEvent

Original: Name : “{Name} {Types} sync”
Revised: Name : “{Name} {Types} sync”

Original: Organizers : Subset(Members)
Revised: Organizers : Subset(Members)

Original: Attendees : Subset(Members)
Revised: Attendees : Subset(Members)

Original: IsRecurring : True +
Revised: IsRecurring : True +

Original: Structured Template
Revised: Structured Template

Original: Generated Data Item =
Revised: Generated Data Item =

Original: CalendarEvent
Revised: CalendarEvent

Original: Name : “Tokyo project sync”
Revised: Name : “Tokyo project sync”

Original: Organizers : John
Revised: Organizers : John

Original: Attendees : Alice, Bob
Revised: Attendees : Alice, Bob

Original: IsRecurring : True
Revised: IsRecurring : True

Original: Latent KB Entity
Revised: Latent KB Entity

Original: Entity
Revised: Entity

Original: Name : “Tokyo”
Revised: Name : “Tokyo”

Original: Types : Project, System
Revised: Types : Project, System

Original: Members : Alice, Bob, John discussion, we move on to describe  model to  entity 
Revised: Members : Alice, Bob, John discussion, we move on to describe a model to learn the entity types automatically as part of the task.

Original: 3. The Enterprise Alexandria Model
Revised: 3. The Enterprise Alexandria Model

Original: The Enterprise Alexandria model is an extended form of the probabilistic program used by
Revised: The Enterprise Alexandria model is an extended form of the probabilistic program used by

Original: Alexandria [Winn et al., 2019].
Revised: Alexandria [Winn et al., 2019].

Original: The main extensions are described below.
Revised: The main extensions are described below.

Original: 3.1 Structured Templates for joint processing of Text and Metadata
Revised: 3.1 Structured Templates for joint processing of Text and Metadata

Original: The Alexandria probabilistic program uses a set of templates to generate text.
Revised: The Alexandria probabilistic program uses a set of templates to generate text.

Original: EA extends this program to generate entire data items, such as emails or calendar events, using manuallyspeciﬁed structured templates .
Revised: EA extends this program to generate entire data items, such as emails or calendar events, using manuallyspeciﬁed structured templates .

Original: Figure 1 illustrates how a structured template can be used to generate an observed data item from a latent knowledge base entity.
Revised: Figure 1 illustrates how a structured template can be used to generate an observed data item from a latent knowledge base entity.

Original: In this example, the structured template is applied to a Project entity named “Tokyo” to create a recurring calendar event ‘Tokyo project sync’ with suitable organizers and attendees.
Revised: In this example, the structured template is applied to a Project entity named “Tokyo” to create a recurring calendar event ‘Tokyo project sync’ with suitable organizers and attendees.

Original: During inference this process is inverted, so a compatible calendar event will be matched against this structured template to produce the knowledge base entity.
Revised: During inference this process is inverted, so a compatible calendar event will be matched against this structured template to produce the knowledge base entity.

Original: A structured template consists of the type or types of data items which can be generated (such as emails, calendar events or documents) along with a set of properties for the generated data items each with a suitable value generator.
Revised: A structured template consists of the type or types of data items which can be generated (such as emails, calendar events or documents) along with a set of properties for the generated data items each with a suitable value generator.

Original: Value generators can be one of three kinds: ( i ) A template-based generator which generates a string given a template.
Revised: Value generators can be one of three kinds: ( i ) A template-based generator which generates a string given a template.

Original: If more than one template is provided, one is selected at random.
Revised: If more than one template is provided, one is selected at random.

Original: The Name property in Figure 1 uses this kind of value generator.
Revised: The Name property in Figure 1 uses this kind of value generator.

Original: (When ‘Types’ is used in a template, one type is selected at random.)
Revised: (When ‘Types’ is used in a template, one type is selected at random.)

Original: ( ii ) A subset generator which generates a collection value as a random subset of a speciﬁed collection in the entity.
Revised: ( ii ) A subset generator which generates a collection value as a random subset of a speciﬁed collection in the entity.

Original: The Organizers and Attendees property in Figure 1 use this kind of generator to generate subsets of the Members property.
Revised: The Organizers and Attendees property in Figure 1 use this kind of generator to generate subsets of the Members property.

Original: ( iii )
Revised: ( iii )

Original: A constant value generator which always gives a particular value.
Revised: A constant value generator which always gives a particular value.

Original: The IsRecurring property in Figure 1 uses a constant value generator which always returns ‘True’.
Revised: The IsRecurring property in Figure 1 uses a constant value generator which always returns ‘True’.

Original: 3.2 Entity Names and Variants
Revised: 3.2 Entity Names and Variants

Original: In the original Alexandria model, the prior probability of an entity name was uniform over the set of valid names.
Revised: In the original Alexandria model, the prior probability of an entity name was uniform over the set of valid names.

Original: EA uses a more sophisticated model (tuned on Wikipedia titles) which takes into account the number of words, the length of each word, and more.
Revised: EA uses a more sophisticated model (tuned on Wikipedia titles) which takes into account the number of words, the length of each word, and more.

Original: This means that two mentions of the same long name are less likely to be a random collision and more likely to be referring to the same entity than two mentions of the same short name.
Revised: This means that two mentions of the same long name are less likely to be a random collision and more likely to be referring to the same entity than two mentions of the same short name.

Original: This improves the ability of the system to disambiguate mentions of diﬀerent entities with the same name.
Revised: This improves the ability of the system to disambiguate mentions of diﬀerent entities sharing the same name.

Original: In enterprise data, it is common for an entity to be referred to by diﬀerent variations of a name, such as abbreviated forms, capitalizations, etc.
Revised: In enterprise data, it is common for an entity to be referred to by diﬀerent variations of a name, such as abbreviated forms, capitalizations, etc.

Original: EA allows an entity to have alternative names, as long as the alternatives are compatible variants of each other under a set of variants models.
Revised: EA allows an entity to have alternative names, as long as the alternatives are compatible variants of each other under a set of variants models.

Original: The supported variants models are:
Revised: The supported variants models are:

Original: (cid:136) Case and diacritic variants where names diﬀer only in case or in the presence/absence of accents or diacritics; (cid:136) Separator variants where the names diﬀer only in separators (such as ‘&’ or ‘+’) or the separators are removed entirely; (cid:136) Name phrase variants where a name “Tokyo” and a name phrase “Project Tokyo” can be identiﬁed; (cid:136) Acronym variants where one name is an expansion and the other an acronym or partial acronym.
Revised: (cid:136) Case and diacritic variants where names diﬀer only in case or in the presence/absence of accents or diacritics; (cid:136) Separator variants where the names diﬀer only in separators (such as ‘&’ or ‘+’) or the separators are removed entirely; (cid:136) Name phrase variants where a name “Tokyo” and a name phrase “Project Tokyo” can be identiﬁed; (cid:136) Acronym variants where one name is an expansion and the other an acronym or partial acronym.

Original: When computing the probability that two mentions refer to the same entity, if their names do not match but are compatible alternatives, then the probability is the same as if the names matched (on the more probable name), times a constant penalty.
Revised: When computing the probability that two mentions refer to the same entity, if their names do not match but are compatible alternatives, then the probability is the same as if the names matched (on the more probable name), times a constant penalty.

Original: 3.3 Type Discovery
Revised: 3.3 Type Discovery

Original: As in Alexandria, facts about an entity are represented by named, typed properties.
Revised: As in Alexandria, facts about an entity are represented by named, typed properties.

Original: EA includes a property named ‘Types’ which holds the types for the entity, selected from a predeﬁned set.
Revised: EA includes a property named ‘Types’ which holds the types for the entity, selected from a predeﬁned set.

Original: EA semi-automatically learns this set of allowed types through a process of type discovery .
Revised: EA semi-automatically learns this set of allowed types through a process of type discovery .

Original: This process starts with a small manually-provided seed set of types, such as { Project, Team } .
Revised: This process starts with a small manually-provided seed set of types, such as { Project, Team } .

Original: New types are then added iteratively as follows: ( i ) Entity discovery is run to extract entities from a set of enterprise data items, given these seed types.
Revised: New types are then added iteratively as follows: ( i ) Entity discovery is run to extract entities from a set of enterprise data items, given these seed types.

Original: ( ii )
Revised: ( ii )

Original: Fact retrieval is run using the names of these discovered entities, as described in Winn et al.
Revised: Fact retrieval is run using the names of these discovered entities, as described in Winn et al.

Original: However, the schema is modiﬁed to allow values of the Types property to be any string of 1–3 words, rather than one of the ﬁxed set of known types.
Revised: However, the schema is modiﬁed to allow values of the Types property to be any string of 1–3 words, rather than one of the ﬁxed set of known types.

Original: The resulting type names are ‘type candidates’.
Revised: The resulting type names are ‘type candidates’.

Original: ( iii )
Revised: ( iii )

Original: The type candidates are aggregated across entities and automatically ﬁltered based on their frequency and other checks, such as whether there is uncertainty in the posterior distribution over the type name.
Revised: The type candidates are aggregated across entities and automatically ﬁltered based on their frequency and other checks, such as whether there is uncertainty in the posterior distribution over the type name.

Original: ( iv ) The remaining type names are manually ﬁltered to exclude people types and ﬁle types.
Revised: ( iv ) The remaining type names are manually ﬁltered to exclude people types and ﬁle types.

Original: ( v ) The manually approved types are added to the type set and the process is repeated using this updated set.
Revised: ( v ) The manually approved types are added to the type set and the process is repeated using this updated set.

Original: Iteration continues until the number of newly added types becomes suﬃciently small to ensure that good overall coverage has been achieved.
Revised: Iteration continues until the number of newly added types becomes suﬃciently small to ensure that good overall coverage has been achieved.

Original: This type discovery process works because entities usually have multiple types.
Revised: This type discovery process works because entities usually have multiple types.

Original: For example, ‘Tokyo’ might be referred to as a project but also as a framework, system and toolkit.
Revised: For example, ‘Tokyo’ might be referred to as a project but also as a framework, system and toolkit.

Original: Each iteration of type discovery ﬁrst ﬁnds entities whose types are any known type and then uses fact retrieval to add in the other types for these entities, which may be unknown.
Revised: Each iteration of type discovery ﬁrst ﬁnds entities whose types are any known type and then uses fact retrieval to add in the other types for these entities, which may be unknown.

Original: Iteration of this process will discover any type, so long as there exists a chain of entities which connect that type to a seed type.
Revised: Iteration of this process will discover any type, so long as there exists a chain of entities which connect that type to a seed type.

Original: In practice, given a large set of enterprise data, the process appears to give good coverage of types across a wide variety of domains.
Revised: In practice, given a large set of enterprise data, the process appears to give good coverage of types across a wide variety of domains.

Original: Speciﬁcally, on our largest dataset, it was able to discover a universe of 171 types across diﬀerent domain that include “university”, “case”, “campaign”, etc.
Revised: Speciﬁcally, on our largest dataset, it was able to discover a universe of 171 types across diﬀerent domain that include “university”, “case”, “campaign”, etc.

Original: Note that the EA’s entity discovery process will surface only the subset of types that appear in the input dataset.
Revised: Note that the EA’s entity discovery process will surface only the subset of types that appear in the input dataset.

Original: For instance, types like “university” or “case” will be prevalent in the data belonging to an academic institution or a law ﬁrm, respectively.
Revised: For instance, types like “university” or “case” will be prevalent in the data belonging to an academic institution or a law ﬁrm, respectively.

Original: 4. Incremental Clustering
Revised: 4. Incremental Clustering

Original: In our scenario, we assume that new documents will appear, or be updated, at diﬀerent times and their overall volume will exceed the capacity of a single machine.
Revised: In our scenario, we assume that new documents will appear, or be updated, at diﬀerent times and their overall volume will exceed the capacity of a single machine.

Original: Therefore, we designed EA to process documents incrementally and update the knowledge base in an online fashion.
Revised: Therefore, we designed EA to process documents incrementally and update the knowledge base in an online fashion.

Original: This architecture is illustrated in Figure 2.
Revised: This architecture is illustrated in Figure 2.

Original: In detail, let the set of documents available at time t be D t .
Revised: In detail, let the set of documents available at time t be D t .

Original: Given a set of templates
Revised: Given a set of templates

Original: J , we use the Alexandria template matching system to produce a set of template matches { S j ,t } for D t .
Revised: J , we use the Alexandria template matching system to produce a set of template matches { S j ,t } for D t .

Original: These template matches are split into I batches 1 , ..., I of arbitrary size so that S j ,i,t is the i-th batch available at time t from the template set J .
Revised: These template matches are split into I batches 1 , ..., I of arbitrary size so that S j ,i,t is the i-th batch available at time t from the template set J .

Original: For simplicity of notation, we focus the description on a single time interval, as it trivially generalizes to any time interval, and so we drop t from the variables.
Revised: For simplicity of notation, we focus the description on a single time interval, as it trivially generalizes to any time interval, and so we drop t from the variables.

Original: Batch clustering :
Revised: Batch clustering :

Original: Using the probabilistic program, the system takes { S j ,i } as observations and applies probabilistic inference to produce a set of discovered entities E i .
Revised: Using the probabilistic program, the system takes { S j ,i } as observations and applies probabilistic inference to produce a set of discovered entities E i .

Original: For example, a document D1 titled “Project Tokyo overview” authored by Alice and a calendar event
Revised: For example, a document D1 titled “Project Tokyo overview” authored by Alice and a calendar event

Original: D2: “Tokyo team weekly sync” organized by Bob can be matched by structured templates containing the text templates “ { Types } { Name } overview” and “ { Name } { Types } weekly sync”, respectively.
Revised: D2: “Tokyo team weekly sync” organized by Bob can be matched by structured templates containing the text templates “ { Types } { Name } overview” and “ { Name } { Types } weekly sync”, respectively.

Original: Given these matches, batch clustering will output an entity { Name:
Revised: Given these matches, batch clustering will output an entity { Name:

Original: Tokyo, Types: { Project, Team } , Members: { Alice, Bob } , Evidences: { D1, D2 }} .
Revised: Tokyo, Types: { Project, Team } , Members: { Alice, Bob } , Evidences: { D1, D2 }} .

Original: Linking :
Revised: Linking :

Original: The entities discovered from the batch are linked to the knowledge base by the following steps: i Query the knowledge base for candidate entities Q i , using a set of key properties, such as the entity name.
Revised: The entities discovered from the batch are linked to the knowledge base by the following steps: i Query the knowledge base for candidate entities Q i , using a set of key properties, such as the entity name.

Original: To allow for name variants, the queries are extended to include diﬀerent normalized forms of the entity names.
Revised: To allow for name variants, the queries are extended to include diﬀerent normalized forms of the entity names.

Original: For example, an entity with a name
Revised: For example, an entity with a name

Original: [Figure]
Revised: [Figure]

Original: “Cloud Storage Explorer” is queryable by“cloudstorageexplorer” and “CSE” keys which allows it to cluster incrementally with entities named “CloudStorage Explorer” and “CSExplorer” whose normalized names will overlap.
Revised: “Cloud Storage Explorer” is queryable by“cloudstorageexplorer” and “CSE” keys which allows it to cluster incrementally with entities named “CloudStorage Explorer” and “CSExplorer” whose normalized names will overlap.

Original: ii Produce a set of entities R i by clustering together E i , Q i , iii Update the knowledge base by replacing the entities Q i with the result set R i which contains both new and updated entities.
Revised: ii Produce a set of entities R i by clustering together E i , Q i , iii Update the knowledge base by replacing the entities Q i with the result set R i which contains both new and updated entities.

Original: For example, if the knowledge base contains an entity: “ { Names: Tokyo, Types: { Project } , Members: { John } , Evidences: { D3 }} ”, the linking step will conﬂate it with the entity produced from the batch clustering step and so create an updated entity “ { Name: Tokyo, Types: { Project, Team } , Members: { Alice, Bob, John } , Evidences { D1, D2, D3 }} ”.
Revised: For example, if the knowledge base contains an entity: “ { Names: Tokyo, Types: { Project } , Members: { John } , Evidences: { D3 }} ”, the linking step will conﬂate it with the entity produced from the batch clustering step and so create an updated entity “ { Name: Tokyo, Types: { Project, Team } , Members: { Alice, Bob, John } , Evidences { D1, D2, D3 }} ”.

Original: Curation : During this process, the human curator can asynchronously edit the knowledge base and/or add new entities.
Revised: Curation : During this process, the human curator can asynchronously edit the knowledge base and/or add new entities.

Original: Both the AI mined and curated entities are included in Q i allowing mined and curated entities to be linked together into coherent merged entities.
Revised: Both the AI mined and curated entities are included in Q i allowing mined and curated entities to be linked together into coherent merged entities.

Original: The batch size can be adjusted based on the memory constraints of the executor – a small batch size requires less memory but increases the runtime for processing all the batches.
Revised: The batch size can be adjusted based on the memory constraints of the executor – a small batch size requires less memory but increases the runtime for processing all the batches.

Original: Entity Correctness : In Alexandria, fact retrieval was performed for known entities which restricted the set of valid names for template matching.
Revised: Entity Correctness : In Alexandria, fact retrieval was performed for known entities which restricted the set of valid names for template matching.

Original: EA aims instead to discover entities and so template matching is unrestricted.
Revised: EA aims instead to discover entities and so template matching is unrestricted.

Original: This results in the system discovering entities which we do not want in the KB, such as people, months, locations, document names and tiny entities which are mentioned only in passing.
Revised: This results in the system discovering entities which we do not want in the KB, such as people, months, locations, document names and tiny entities which are mentioned only in passing.

Original: To address this problem, we trained a ‘correctness’ model on a small sample of manually labeled entities extracted from eyes-on data.
Revised: To address this problem, we trained a ‘correctness’ model on a small sample of manually labeled entities extracted from eyes-on data.

Original: This model is a linear classiﬁer, using simple features like the number of documents mined from, the kinds of documents, the template diversity and so on, that can be applied post-linking to estimate the correctness of each entity.
Revised: This model is a linear classiﬁer, using simple features like the number of documents mined from, the kinds of documents, the template diversity and so on, that can be applied post-linking to estimate the correctness of each entity.

Original: Speciﬁcally, the classiﬁer was trained on a sample of manually labelled 857 entities (274 positive and 583 negative labels) taken from the Anonymous Eyes-oﬀ dataset (see Section 6.1 for more details).
Revised: Speciﬁcally, the classiﬁer was trained on a sample of manually labelled 857 entities (274 positive and 583 negative labels) taken from the Microsoft Eyes-oﬀ dataset (see Section 6.1 for more details).

Original: 5. Related Work
Revised: 5. Related Work

Original: The use of machine learning methods for large-scale automatic knowledge base construction has been intensively studied in the last decades [Dong et al., 2014, Carlson et al., 2010,
Revised: The use of machine learning methods for large-scale automatic knowledge base construction has been intensively studied in the last decades [Dong et al., 2014, Carlson et al., 2010,

Original: Szekely et al., 2015, Zhang, 2015].
Revised: Szekely et al., 2015, Zhang, 2015].

Original: The majority of these existing methods are focused on
Revised: The majority of these existing methods are focused on

Original: Web-based information extraction, where the aim is to link new facts extracted from text to a set of entities coming from a public knowledge base such as Wikipedia [Dong et al., 2014] or Satori [Winn et al., 2019]
Revised: Web-based information extraction, where the aim is to link new facts extracted from text to a set of entities coming from a public knowledge base such as Wikipedia [Dong et al., 2014] or Satori [Winn et al., 2019]

Original: ; this task is often referred to as entity linking [Zhang, 2015].
Revised: ; this task is often referred to as entity linking [Zhang, 2015].

Original: Within an organization, entities are likely to be private rather than public, and so linking to public entities is not possible.
Revised: Within an organization, entities are likely to be private rather than public, and so linking to public entities is not possible.

Original: Another line of related research on entity discovery deals with discovering new entities from scratch by processing text documents talking about these entities.
Revised: Another line of related research on entity discovery deals with discovering new entities from scratch by processing text documents talking about these entities.

Original: Some of these methods perform feature extraction over the processed text to extract features characterizing mentions of a particular entity to guide the entity mining process.
Revised: Some of these methods perform feature extraction over the processed text to extract features characterizing mentions of a particular entity to guide the entity mining process.

Original: On such a basis, it is possible to conﬂate entity values found in mentions sharing similar feature patterns [Zhang, 2015].
Revised: On such a basis, it is possible to conﬂate entity values found in mentions sharing similar feature patterns [Zhang, 2015].

Original: In particular, the Named Entity Recognition (NER) features are widely used for this task as powerful signals for detecting categorised entity names in free text.
Revised: In particular, the Named Entity Recognition (NER) features are widely used for this task as powerful signals for detecting categorised entity names in free text.

Original: Speciﬁcally, the NER models are able to detect entity names from the text while also providing the type of each entity name chosen from a pre-deﬁned set of types.
Revised: Speciﬁcally, the NER models are able to detect entity names from the text while also providing the type of each entity name chosen from a pre-deﬁned set of types.

Original: To date, the most successful NER methods are based on neural language models, such as
Revised: To date, the most successful NER methods are based on neural language models, such as

Original: BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robust
Revised: BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa (Robust

Original: Optimized BERT Approach)
Revised: Optimized BERT Approach)

Original: [Devlin et al., 2019, Liu et al., 2019], that are trained on a large public corpus that can generalize to a private corpus with a small amount of ﬁne tuning.
Revised: [Devlin et al., 2019, Liu et al., 2019], that are trained on a large public corpus that can generalize to a private corpus with a small amount of ﬁne tuning.

Original: While the categorized entity names extracted from NER models alone are not suﬃcient for constructing a full knowledge base, it is possible to adapt them to our task if an appropriate conﬂation algorithm is provided.
Revised: While the categorized entity names extracted from NER models alone are not suﬃcient for constructing a full knowledge base, it is possible to adapt them to our task if an appropriate conﬂation algorithm is provided.

Original: Therefore, in the next section, we will consider the most competitive NER models as baselines for evaluating our approach.
Revised: Therefore, in the next section, we will consider the most competitive NER models as baselines for evaluating our approach.

Original: 6. Experimental Evaluation
Revised: 6. Experimental Evaluation

Original: This section evaluates the ability of EA to perform entity discovery, compared to four rival approaches:
Revised: This section evaluates the ability of EA to perform entity discovery, compared to four rival approaches:

Original: BERT Base Fine-tuned (BERT F)
Revised: BERT Base Fine-tuned (BERT F)

Original: BERT is a state–of–the–art language model developed by Google that is commonly applied to NER tasks [Devlin et al., 2019].
Revised: BERT is a state–of–the–art language model developed by Google that is commonly applied to NER tasks [Devlin et al., 2019].

Original: It is pre-trained on a large corpus of English data in a self-supervised fashion and it makes use of 12 stacked layers of encoders with an attention mechanism to learn contextual relations between words in the text.
Revised: It is pre-trained on a large corpus of English data in a self-supervised fashion and it makes use of 12 stacked layers of encoders with an attention mechanism to learn contextual relations between words in the text.

Original: This version of BERT is ﬁne-tuned on the English version of the standard CoNLL
Revised: This version of BERT is ﬁne-tuned on the English version of the standard CoNLL

Original: 2003 dataset
Revised: 2003 dataset

Original: [Tjong Kim Sang and De Meulder, 2003] to recognize four types of entity names: location (LOC), organizations (ORG), person (
Revised: [Tjong Kim Sang and De Meulder, 2003] to recognize four types of entity names: location (LOC), organizations (ORG), person (PER) and

Original: PER) and Miscellaneous MISC)
Revised: Miscellaneous (MISC) 2 .

Original: BERT Base Multi-lingual Fine-tuned (BERT MF
Revised: BERT Base Multi-lingual Fine-tuned (BERT MF)

Original: A variant of BERT Base Fine - tuned that is further ﬁne- tuned on the CoNLL 2003 dataset to recognize the same four categories of entity names as BERT F in diﬀerent languages beyond English 2 .
Revised: A variant of BERT Base Finetuned that is further ﬁne-tuned on the CoNLL 2003 dataset to recognize the same four categories of entity names as BERT F in diﬀerent languages beyond English 3 .

Original: BERT Large Fine-tuned (BERT LF)
Revised: BERT Large Fine-tuned (BERT LF)

Original: This model similar to the above methods
Revised: This model similar to the above methods

Original: but it uses the large version of BERT with 24 layers of encoders and 16 attention heads.
Revised: but it uses the large version of BERT with 24 layers of encoders and 16 attention heads.

Original: It is also ﬁne-tuned on the English version of the CoNLL 2003 dataset to recognizecategories of entity names 3 .
Revised: It is also ﬁne-tuned on the English version of the CoNLL 2003 dataset to recognize 16 categories of entity names 4 .

Original: RoBERTa Large Fine-tuned (RoBERTa LF) RoBERTa is a language model developed by Facebook that improves the performance of BERT using an optimized training strategy with a longer training epochs, bigger batches and a larger dataset [Liu et al., 2019].
Revised: RoBERTa Large Fine-tuned (RoBERTa LF) RoBERTa is a language model developed by Facebook that improves the performance of BERT using an optimized training strategy with a longer training epochs, bigger batches and a larger dataset [Liu et al., 2019].

Original: This model is also ﬁne-tuned on the English version of the CoNLL 2003 dataset 4 .
Revised: This model is also ﬁne-tuned on the English version of the CoNLL 2003 dataset 5 .

Original: The original Alexandria system [Winn et al., 2019] requires entity names as input and therefore not applicable to entity discovery.
Revised: The original Alexandria system [Winn et al., 2019] requires entity names as input and therefore not applicable to entity discovery.

Original: All the benchmarks are freely available on the open-source huggingface platform.
Revised: All the benchmarks are freely available on the open-source huggingface platform.

Original: To make these models suitable for the entity discovery task, we must add an entity conﬂation model to conﬂate all the NER extractions referring to the same entity into a single entity.
Revised: To make these models suitable for the entity discovery task, we must add an entity conﬂation model to conﬂate all the NER extractions referring to the same entity into a single entity.

Original: Speciﬁcally, we use a standard case-sensitive linking by name conﬂation method that works

Original: huggingface.co/dslim/bert-base-NER 2.
Revised: huggingface.co/dslim/bert-base-NER 3.

Original: huggingface.co/wietsedv/bert-base-multilingual-cased-finetuned-udlassy-ner3. huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english 4. 
Revised: huggingface.co/wietsedv/bert-base-multilingual-cased-finetuned-udlassy-ner 4. huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english 5.

Original: huggingface.co/xlm-roberta-large-finetuned-conll03-english
Revised: huggingface.co/xlm-roberta-large-finetuned-conll03-english

Original:  as follows:
Revised: Speciﬁcally, we use a standard case-sensitive linking by name conﬂation method that works as follows:

Original: Firstly, all extractions sharing the same name based on case-sensitive string matching are grouped together.
Revised: Firstly, all extractions sharing the same name based on case-sensitive string matching are grouped together.

Original: Then, an entity is constructed by taking the union of all the documents’ meta–data.
Revised: Then, an entity is constructed by taking the union of all the documents’ meta–data.

Original: For instance, n extractions about an entity named x classiﬁed with type t 1 , ..., t n extracted from j documents, d i , ..., d j , each with associated people’s meta–data m i , ..., m j (e.g. the authors of the documents) are conﬂated into the entity: “ { Name: x ,
Revised: For instance, n extractions about an entity named x classiﬁed with type t 1 , ..., t n extracted from j documents, d i , ..., d j , each with associated people’s meta–data m i , ..., m j (e.g. the authors of the documents) are conﬂated into the entity: “ { Name: x ,

Original: Types: { u ( t 1 , ..., t n ) } , Members: { u ( m i , ..., m j ) } , Evidences: { d i , ..., d j }} ” where u ( . ) is the union function.
Revised: Types: { u ( t 1 , ..., t n ) } , Members: { u ( m i , ..., m j ) } , Evidences: { d i , ..., d j }} ” where u ( . ) is the union function.

Original: To remove obvious noise, we ﬁlter out small entities from all the methods, i.e., with less than 100 evidences, and we also remove a few categories of entities that were not included in our ground truth, i.e., person, date and ordinal numbers.
Revised: To remove obvious noise, we ﬁlter out small entities from all the methods, i.e., with less than 100 evidences, and we also remove a few categories of entities that were not included in our ground truth, i.e., person, date and ordinal numbers.

Original: Finally, we use a list of disallowed entity names, containing 1998 stop-words and common English n-grams corresponding to false entity names, that is applied to ﬁlter the output of each method.
Revised: Finally, we use a list of disallowed entity names, containing 1998 stop-words and common English n-grams corresponding to false entity names, that is applied to ﬁlter the output of each method.

Original: 6.1 Datasets
Revised: 6.1 Datasets

Original: Our evaluation involves the following three real-world datasets: Enron : A well-known public corpus of 517 , 401 emails generated from 150 employees of the Enron Corporation [Klimt and Yang, 2004].
Revised: Our evaluation involves the following three real-world datasets: Enron : A well-known public corpus of 517 , 401 emails generated from 150 employees of the Enron Corporation [Klimt and Yang, 2004].

Original: The dataset has full eyes-on access for research purposes (i.e., researchers can look at the data).
Revised: The dataset has full eyes-on access for research purposes (i.e., researchers can look at the data).

Original: Anonymous 5 Eyes-on : A dataset of 368 , 366 documents donated by Anonymous employees with restricted eyes-on access (i.e., the entities mentioned in the documents can be labeled by a selected pool of human judges).
Revised: Microsoft Eyes-on : A dataset of 368 , 366 documents donated by Microsoft employees with restricted eyes-on access (i.e., the entities mentioned in the documents can be labeled by a selected pool of human judges).

Original: For this dataset, we also have access to a ground truth with 1,786 human-curated labels for real entity names (1,205 positive labels and 581 negative labels) that exist within the Anonymous organization.
Revised: For this dataset, we also have access to a ground truth with 1,786 human-curated labels for real entity names (1,205 positive labels and 581 negative labels) that exist within the Microsoft organization.

Original: Anonymous Eyes-oﬀ : A larger dataset with 1 , 023 , 435 documents made available by Anonymous employees with eyes-oﬀ access only that is useful for evaluating performance at larger scale.
Revised: Microsoft Eyes-oﬀ : A larger dataset with 1 , 023 , 435 documents made available by Microsoft employees with eyes-oﬀ access only that is useful for evaluating performance at larger scale.

Revised: Both Microsoft datasets went through exhaustive privacy reviews in order to protect individual privacy and corporate conﬁdentiality

Original: 6.2 Performance Comparisons
Revised: 6.2 Performance Comparisons

Original: We evaluate the performance of the alternative approaches against two versions of our method: ( i ) EA size > 100 , that is our model with the basic entity size ﬁlter (i.e., > 100 evidences) that is applied to all methods.
Revised: We evaluate the performance of the alternative approaches against two versions of our method: ( i ) EA size > 100 , that is our model with the basic entity size ﬁlter (i.e., > 100 evidences) that is applied to all methods.

Original: ( ii ) EA , that is our model with the entity correctness classiﬁer described in Section 4.
Revised: ( ii ) EA , that is our model with the entity correctness classiﬁer described in Section 4.

Original: To evaluate the performance of the diﬀerent approaches, we measure the precision and coverage against the ground truth deﬁned as follows:
Revised: To evaluate the performance of the diﬀerent approaches, we measure the precision and coverage against the ground truth deﬁned as follows:

Original: Precision = p/ ( p + n ) and Relative Coverage = p/p EA .
Revised: Precision = p/ ( p + n ) and Relative Coverage = p/p EA .

Original: Here p and n are the number of positive and negative entity name labels in the ground truth, respectively, that are calculated by matching in the entity names discovered by each method to the ground truth labels, and p EA is the precision of EA.
Revised: Here p and n are the number of positive and negative entity name labels in the ground truth, respectively, that are calculated by matching in the entity names discovered by each method to the ground truth labels, and p EA is the precision of EA.

Original: Intuitively, the precision measures the proportion of discovered entity names with a positive label out of all the entities that can be matched to the ground truth, while the relative coverage measures the recall ratio between each method and EA.
Revised: Intuitively, the precision measures the proportion of discovered entity names with a positive label out of all the entities that can be matched to the ground truth, while the relative coverage measures the recall ratio between each method and EA.

Original: Absolute recall is not feasible to compute in our setting since it requires labels for entities that were not discovered by any system.
Revised: Absolute recall is not feasible to compute in our setting since it requires labels for entities that were not discovered by any system.

Original: Since Enron does not have any existing ground truth for entity names, for each method we drew a random sample of 100 entity names out of the names it discovered and manually labeled them positive or negative.
Revised: Since Enron does not have any existing ground truth for entity names, for each method we drew a random sample of 100 entity names out of the names it discovered and manually labeled them positive or negative.

Original: This ensures unbiased precision estimates.
Revised: This ensures unbiased precision estimates.

Original: For Anonymous Eyes-on , we used the existing ground truth.
Revised: For Microsoft Eyes-on , we used the existing ground truth.

Original: [Table]
Revised: [Table]

Original: In detail, Table 2 reports the performance of the ﬁve methods on the Enron and 
Revised: In detail, Table 2 reports the performance of the ﬁve methods on the Enron and Microsoft

Original: Anonymous Eyes-on datasets.
Revised: Eyes-on datasets.

Original: In these experiments, the EA methods used a set oftemplates and hierarchy of 176 entity types, both learned from the Anonymous Eyes-oﬀ dataset.
Revised: In these experiments, the EA methods used a set of 88 templates and hierarchy of 176 entity types, both learned from the Microsoft Eyes-oﬀ dataset.

Original: On both datasets, EA size > 100 and EA have the highest precision.
Revised: On both datasets, EA size > 100 and EA have the highest precision.

Original: In particular, the precision of EA is 5.11 (0.92 vs. 0.16 on Enron ) and 1.43 (0.83 vs. 0.58 Anonymous Eyes-on ) times higher than the second best method (BERT LF).
Revised: In particular, the precision of EA is 5.11 (0.92 vs. 0.16 on Enron ) and 1.43 (0.83 vs. 0.58 Microsoft Eyes-on ) times higher than the second best method (BERT LF).

Original: This is due to the combination of the template-based language model and the probabilistic clustering of EA that allows it to jointly discover entities with high precision and disambiguate diﬀerent entities with the same name.
Revised: This is due to the combination of the template-based language model and the probabilistic clustering of EA that allows it to jointly discover entities with high precision and disambiguate diﬀerent entities with the same name.

Original: For example, EA was able to ﬁnd two distinct entities named ‘MT6’ on Anonymous Eyes-on , while the same entities are incorrectly over-conﬂated together due to linking entities by name in the other methods.
Revised: For example, EA was able to ﬁnd two distinct entities named ‘MT6’ on Microsoft Eyes-on , while the same entities are incorrectly over-conﬂated together due to linking entities by name in the other methods.

Original: In addition, through its name variant model (see Section 3.2), EA was able to ﬁnd alternate names for several entities, such as { OneDriveWeb, OneDrive Web, OD
Revised: In addition, through its name variant model (see Section 3.2), EA was able to ﬁnd alternate names for several entities, such as { OneDriveWeb, OneDrive Web, OD

Original: Web } in Anonymous Eyes-on and { Phillips, PHILLIPS } in Enron , referring to the diﬀerent ways that people expressed the same entity in the free text.
Revised: Web } in Microsoft Eyes-on and { Phillips, PHILLIPS } in Enron , referring to the diﬀerent ways that people expressed the same entity in the free text.

Original: Instead, the same entity names were incorrectly under-conﬂated by the other methods because they do not strictly match with one another.
Revised: Instead, the same entity names were incorrectly under-conﬂated by the other methods because they do not strictly match with one another.

Original: In terms of coverage, EA has the highest coverage on Anonymous Eyes-on while the BERT-based methods have higher coverage but lower precision on Enron .
Revised: In terms of coverage, EA has the highest coverage on Microsoft Eyes-on while the BERT-based methods have higher coverage but lower precision on Enron .

Original: It is worth noting that in our application, given the presence of the knowledge curator that must editorially check the validity of each entity, we ﬁnd that methods with high coverage and low precision are less practical because they tend to overload the system with a very large and noisy set of entities, more than a human curator can cope with.
Revised: It is worth noting that in our application, given the presence of the knowledge curator that must editorially check the validity of each entity, we ﬁnd that methods with high coverage and low precision are less practical because they tend to overload the system with a very large and noisy set of entities, more than a human curator can cope with.

Original: Instead, high-precision methods like EA tend to provide a more manageable, yet substantial, set of entities that are much easier to deal with for the knowledge manager.
Revised: Instead, high-precision methods like EA tend to provide a more manageable, yet substantial, set of entities that are much easier to deal with for the knowledge manager.

Original: In this light, precision errors tend to be penalized more than providing additional coverage.
Revised: In this light, precision errors tend to be penalized more than providing additional coverage.

Original: Finally, EA is able to associate up to 9.5 times more types (152 vs. 16 on Enron ) to the discovered entities compared to the other methods that can only associate types from their (typically small) set of types that they have been ﬁne-tuned on.
Revised: Finally, EA is able to associate up to 9.5 times more types (152 vs. 16 on Enron ) to the discovered entities compared to the other methods that can only associate types from their (typically small) set of types that they have been ﬁne-tuned on.

Original: Overall, EA achieves signiﬁcantly higher precision and it retrieves more entity types on both datasets.
Revised: Overall, EA achieves signiﬁcantly higher precision and it retrieves more entity types on both datasets.

Original: 6.3 Memory Usage and Runtime
Revised: 6.3 Memory Usage and Runtime

Original: To assess the performance of EA at larger scale, we ran it on the Anonymous Eyes-oﬀ dataset.
Revised: To assess the performance of EA at larger scale, we ran it on the Microsoft Eyes-oﬀ dataset.

Original: For compliance reasons, we could not run the other methods on this dataset.
Revised: For compliance reasons, we could not run the other methods on this dataset.

Original: In this experiment, EA was able to process 1 , 023 , 435 documents and 45 , 645 , 618 template matches in 9.3h on a conventional machine, using 8.64GB of memory measured as the median of the last 50 batches (Memory@50).
Revised: In this experiment, EA was able to process 1 , 023 , 435 documents and 45 , 645 , 618 template matches in 9.3h on a conventional machine, using 8.64GB of memory measured as the median of the last 50 batches (Memory@50).

Original: It discovered 675 , 439 entities and 172 entity types.
Revised: It discovered 675 , 439 entities and 172 entity types.

Original: The clustering time is constant across batches, taking approximately 23 seconds for a batch of 10,000 template matches.
Revised: The clustering time is constant across batches, taking approximately 23 seconds for a batch of 10,000 template matches.

Original: The linking time increases approximatively linearly with a small slope (0.005) over batches as the size of the knowledge base grows over time (that can potentially be mitigated by cleaning up old entities).
Revised: The linking time increases approximatively linearly with a small slope (0.005) over batches as the size of the knowledge base grows over time (that can potentially be mitigated by cleaning up old entities).

Original: Overall, this result shows that the incremental clustering architecture adopted in EA scales gracefully on our largest dataset.
Revised: Overall, this result shows that the incremental clustering architecture adopted in EA scales gracefully on our largest dataset.

Original: 6.4 Type Retrieval
Revised: 6.4 Type Retrieval

Original: Figure 3 shows the types retrieved by EA on the two eyes-on datasets.
Revised: Figure 3 shows the types retrieved by EA on the two eyes-on datasets.

Original: Due to the lack of ground truth for types, we cannot perform a quantitative evaluation of their correctness.
Revised: Due to the lack of ground truth for types, we cannot perform a quantitative evaluation of their correctness.

Original: However, since it is known that Enron and Anonymous are companies specialized in the energy and software industry, respectively, the types appear relevant to the organization owning the data (e.g. “Site” or “Fund” for Enron and “Team”, “App” or “Deal” for Anonymous Eyes-on ).
Revised: However, since it is known that Enron and Microsoft are companies specialized in the energy and software industry, respectively, the types appear relevant to the organization owning the data (e.g. “Site” or “Fund” for Enron and “Team”, “App” or “Deal” for Microsoft Eyes-on ).

Original: When we focus on the ‘Technology’ and ‘Seed types’ classes from the user study (Table 1), we found that 40% of the entity types declared by the participants were also retrieved by EA for at least one entity in the Anonymous Eyes-on dataset.
Revised: When we focus on the ‘Technology’ and ‘Seed types’ classes from the user study (Table 1), we found that 40% of the entity types declared by the participants were also retrieved by EA for at least one entity in the Microsoft Eyes-on dataset.

Original: The nonretrieved types are mostly speciﬁc categories such as “Best practice”, “O365” or “Service measurement” that did not appear often in the training set.
Revised: The non-retrieved types are mostly speciﬁc categories such as “Best practice”, “O365” or “Service measurement” that did not appear often in the training set.

Original: As a result, we expect a substantial but not necessarely complete overlap between the two sets of types, even if the both belong to the technology industry.
Revised: As a result, we expect a substantial but not necessarily complete overlap between the two sets, even if they both belong to the technology industry.

Original: Area
Revised: Area

Original: Site
Revised: Site

Original: Fund
Revised: Fund

Original: P ar t y App City
Revised: P a

Original: [Figure]
Revised: [Figure]

Original: Complex
Revised: t

Original: [Figure]
Revised: [Figure]

Original: Grant Regulation

Original: Academy

Original: Feature

Original: Methodology
Revised: Methodology

Original: Branch
Revised: Branch

Original: Module
Revised: Module

Original: Portal Engine
Revised: Portal Engine

Original: Component
Revised: Component

Original: Memorandum Protocol
Revised: Memorandum Protocol

Original: Jurisdiction
Revised: Jurisdiction

Original: Hospital Expo
Revised: Hospital Expo

Original: Register
Revised: Register

Original: Factory
Revised: Factory

Original: Operating system
Revised: Operating system

Original: SteeringCommittee Contractor Symposium
Revised: SteeringCommittee Contractor Symposium

Original: Territory Treatment Condition Framework Gateway Device
Revised: Territory Treatment Condition Framework Gateway Device

Original: Registry Warehouse Sponsor
Revised: Registry Warehouse Sponsor

Original: Institute of Technology
Revised: Institute of Technology

Original: Manufacturer Subdivision Vendor
Revised: Manufacturer Subdivision Vendor

Original: Data Warehouse
Revised: Data Warehouse

Original: Market Area Work Order Leadership Conference Hardware Pharmacy Development Project User Interface
Revised: Market Area Work Order Leadership Conference Hardware Pharmacy Development Project User Interface

Original: Repository
Revised: Repository

Original: Compound API Original Equipment Manufacturer Graphical User Interface Appliance Health Service PurchaserBoroughToolbox Web app
Revised: Compound API Original Equipment Manufacturer Graphical User Interface Appliance Health Service PurchaserBoroughToolbox Web app

Original: SDK (a) Enron
Revised: SDK (a) Enron

Original: Team
Revised: Team

Original: App
Revised: App

Original: Solution
Revised: Solution

Original: Area
Revised: Area

Original: S t o r e Form
Revised: S t o r e Form

Original: Package
Revised: Package

Original: Module
Revised: Module

Original: Strategy
Revised: Strategy

Original: Guide
Revised: Guide

Original: Engine
Revised: Engine

Original: Market
Revised: Market

Original: Partner
Revised: Partner

Original: Branch Device
Revised: Branch Device

Original: Order
Revised: Order

Original: Building
Revised: Building

Original: Association
Revised: Association

Original: Industry
Revised: Industry

Original: SDK
Revised: SDK

Original: Institute
Revised: Institute

Original: Location
Revised: Location

Original: Party
Revised: Party

Original: City
Revised: City

Original: Survey
Revised: Survey

Original: Society Nation Government
Revised: Society Nation Government

Original: Unit
Revised: Unit

Original: Authority
Revised: Authority

Original: Web app Contract Division
Revised: Web app Contract Division

Original: Deal
Revised: Deal

Original: Investment
Revised: Investment

Original: Agency Fund
Revised: Agency Fund

Original: Utility
Revised: Utility

Original: Zone
Revised: Zone

Original: Campaign Complex
Revised: Campaign Complex

Original: Desk DLL Portfolio
Revised: Desk DLL Portfolio

Original: Toolkit
Revised: Toolkit

Original: Gateway
Revised: Gateway

Original: Campus
Revised: Campus

Original: Mobile App Registry
Revised: Mobile App Registry

Original: Award
Revised: Award

Original: Rest API
Revised: Rest API

Original: Data model Hardware
Revised: Data model Hardware

Original: Museum Vendor
Revised: Museum Vendor

Original: League
Revised: League

Original: Province
Revised: Province

Original: Methodology
Revised: Methodology

Original: Webservice
Revised: Webservice

Original: Operating system
Revised: Operating system

Original: Sector
Revised: Sector

Original: Factory
Revised: Factory

Original: Journal
Revised: Journal

Original: Festival
Revised: Festival

Original: Asset Convention
Revised: Asset Convention

Original: Acquisition Micro service
Revised: Acquisition Micro service

Original: Hospital Federation Territory
Revised: Hospital Federation Territory

Original: Register
Revised: Register

Original: Estate
Revised: Estate

Original: Regulation
Revised: Regulation

Original: Appliance
Revised: Appliance

Original: Condition
Revised: Condition

Original: Treatment
Revised: Treatment

Original: Graphical User Interface Data Warehouse
Revised: Graphical User Interface Data Warehouse

Original: Subsidiary
Revised: Subsidiary

Original: Grant
Revised: Grant

Original: Original Equipment Manufacturer
Revised: Original Equipment Manufacturer

Original: Bureau
Revised: Bureau

Original: BusinessModel Affiliate
Revised: BusinessModel Affiliate

Original: Sponsor
Revised: Sponsor

Original: Contractor Steering Committee
Revised: Contractor Steering Committee

Original: Toolbox
Revised: Toolbox

Original: Manufacturer Clinic Lease Coalition Warehouse Expo Seminary Brief Business Unit Health System SymposiumHoldings Institute of Technology
Revised: Manufacturer Clinic Lease Coalition Warehouse Expo Seminary Brief Business Unit Health System SymposiumHoldings Institute of Technology

Original: Health Service
Revised: Health Service

Original: Task Force Work Order Middleware Compound Jurisdiction Borough (b) Anonymous Eyes-On
Revised: Task Force Work Order Middleware Compound Jurisdiction Borough (b) Microsoft Eyes-On

Original: 7. Conclusions
Revised: 7. Conclusions

Original: We have introduced Enterprise Alexandria, a new AKBC system specializing in automatic knowledge discovery with typed entities from enterprise documents.
Revised: Enterprise Alexandria provides automatic knowledge discovery of typed entities from enterprise documents.

Original: The system extends Alexandria by consuming structured information coming from documents’ metadata. EA also introduces a new mechanism for discovering entities incrementally in an online fashion that allows millions of documents to be processed in a few hours with constrained computing resources. Furthermore, EA allows users to contributeto entity discovery process by accepting manual curations that are then merged with knowledge updates automatically extracted from the documents.
Revised: The system extends Alexandria by consuming structured information coming from documents’ metadata, discovering entities incrementally in an online fashion, and allowing users to contribute manual curations to the entity discovery process.

Original: Our evaluation shows that EA signiﬁcantly outperforms the alternatives in extracting entities with higher quality on several real-world datasets.
Revised: Our evaluation shows that EA signiﬁcantly outperforms the alternatives in extracting entities with higher quality on several real-world datasets.

Revised: EA is a key technology for the AI engine of Microsoft Viva Topics and it is currently servicing thousands of organizations and users 6 .