ANTS : A Framework for Retrieval of Text Segments in Unstructured Documents (2022)

Abstract: Text segmentation and extraction from unstructured documents can provide business researchers with a wealth of new information on firms and their behaviors. However, the most valuable text is often difficult to extract consistently due to substantial variations in how content can appear from document to document. Thus, the most successful way to extract this content has been through costly crowdsourcing and training of manual workers. We propose the Assisted Neural Text Segmentation (ANTS) framework to identify pertinent text in unstructured documents from a small set of labeled examples. ANTS leverages deep learning and transfer learning architectures to empower researchers to identify relevant text with minimal manual coding. Using a real world sample of accounting documents, we identify targeted sections 96% of the time using only 5 training examples.

## introduction datasets of text documents hold enormous amounts of raw information, particularly for social scientists and business researchers. an individual document contains not only declarative statements and facts, but also style, theme and sentiment information that can be used to evaluate diverse research questions. researchers have spent decades developing frameworks and techniques to distill text into features that are easily integrated into existing research practices. one common practice is to use vetted word lists to compute a score for a particular topic or theme. for example, @xcite use a vetted word list to identify the degree of uncertain language used in financial documents, and count the number of occurrences as a proxy for the amount of prospective discussion. dictionary approaches such as liwc @xcite ) match words to predefined psycholinguistic categories, allowing researchers to identify broad themes including "anxiety" and "religion". more computationally sophisticated methods such as word embeddings @xcite and topic modeling algorithms @xcite provide the capability to measure prevalence of topics within documents, as well as the relationships between words and how they may shift over time. the development of transformer models such as bert (bi-directional encoder representations from transformers) @xcite have opened a new frontier of text processing, with models trained to categorize, summarize or answer specific questions from input text. despite all these advancements, valuable pieces of information remain difficult to extract or categorize in large unstructured documents. word lists and dictionaries can fail to capture the immense variety of language that can be used to talk about a single topic. topic modeling algorithms may not capture a specific concept in one overarching topic. thus, to ensure maximum quality, many researchers resort to manual methods to effectively characterize the text data from their documents. one approach is to begin by identifying only the segments of interest, so that only relevant text can be utilized by subject matter experts or computational methods. to manually select these relevant subsets, researchers frequently work with undergraduates or other research assistants, or they post tasks to pools of remote workers using platforms like amazon's mechanical turk (mturk). in either method, using humans to extract specific pieces of text from large documents is costly and time consuming. we propose a general deep learning framework to provide assisted neural text segmentation (ants) as a way to facilitate identification of text segments of interest for researchers. the primary goal of this general framework aims to reduce the amount of time subject matter experts must spend manually coding documents or identifying and training effective research assistants. the ants framework has four steps: in this paper, we present a specific problem of extracting human capital disclosure (hcd) sections from form 10-k filings created by corporations for regulatory securities and exchange commission (sec) filings. we also illustrate a few strategies to elevate the performance of our model without annotating additional training data. through similar means, we hope to provide a less costly and time consuming pathway for researchers to identify relevant segments of text from unstructured documents. ## related work with modern advancements in deep learning technology and the increased need for processing large text datasets, researchers have been optimizing the task of automated text segmentation. common applications of this natural language processing (nlp) task include information retrieval @xcite , topic segmentation @xcite , and document summarization @xcite . these tasks can take either linear or hierarchical approaches, with the latter taking into account structural representation of topics within documents @xcite . generally, the development of neural models from scratch for text segmentation tasks requires large training datasets @xcite and high computational costs. in response, researchers have turned to pre-trained deep transformer models such as bert, which offers high performance on nlp tasks and the possibility of fine-tuning its base model towards specific domains. various transformer-based model architectures and linear, hierarchical, and multilevel models have been explored and evaluated for their performance on text segmentation. for domain-independent models, @xcite introduced three new bert architectures to segment documents and discourses by predicting on break points instead of classifying every piece of text. these novel architectures showed that a simple cross-segment bert model using only local context (sequences of tokens before and after a potential break point) can perform as competitively as more complex hierarchical bert models. yoong et al. (2021) also developed three bert models-bert-nsp, bert-sep and bert-segment-to perform a text tiling task (dividing a document or dialogue into semantically coherent text segments) and demonstrated that bert-sep, which considers the relatedness of adjacent sentences as well as information from the whole document, outperformed graph-based or bi-directional lstm (long short-term memory) models. @xcite developed a two-level transformer framework incorporating languagespecific or domain-specific pre-trained bert transformers as sentence encoders, which outperformed state-of-the-art text segmentation models on a semantic coherence measure. to develop domain-specific models, often with limited labeled training data, researchers have tested how transformer-based language models pretrained on large amounts of general-domain data can be leveraged and adapted for a specific domain. to extract content elements from regulatory filings and property lease agreements, @xcite segmented documents into paragraphs and trained bert at the paragraph level, which achieved reasonable accuracy. they also found that training with fewer than 100 documents was sufficient to achieve an f1 score similar to that of the same model trained with the entire set of documents. @xcite introduced finbert, a fine-tuned bert model for the financial domain, by conducting additional pre-training and fine-tuning of bert using text from financial news articles. finbert outperformed other pre-trained models with as few as 250 training examples in a sentiment analysis task involving financial phrases. we go beyond the works mentioned that only provided information retrieval, topic segmentation, or document summarization to extract any targeted section that a social science researcher needs through a quick and manual-labor saving framework. building on the above related works, we focus on refining a generic transformer model training -8g: ## data the securities and exchange commission (sec) requires that publicly traded companies file financial disclosures at regular intervals. the form 10-k is an annual report that contains information about a company's business operations, financial results, and management. in november 2020, the sec began to require its registrants to include a disclosure of their human capital resources in their form 10-k. this resulting section is of interest to accounting researchers who want to characterize how firms discuss their human capital and whether specific diversity metrics are divulged @xcite . we use mturk to train workers to identify the described human capital disclosures (hcd) section in 393 form 10-k documents filed by s&p 500 firms from november 2020 through march 2021. the hcd section is a single, continuous segment of text located within each form 10-k. it appears under various titles (e.g., "human capital", "human resources", "talent", "employee engagement"), which span a range of sub-section topics (e.g., hiring, benefits and compensation, diversity, culture) of different lengths and combinations. we employ human labor for this extraction task due to this lack of uniformity in the section names, content, and location of the hcd section among form 10-k documents. we use this manually collected data as a test case for our ants framework. in later sections, we will describe how documents are randomly sampled to obtain training and test sets to evaluate our framework. ## methods in this section, we describe the ants framework (outlined in figure 1 ) in more detail and how it is used in the scope of our specific test case. our framework expands upon the general structure and methodologies of machine learning systems. we employ a few strategies within the framework to maximize the performance of our fine-tuned model on our task without adding more labeled documents. in training, we explore a windowing method to expand the size of our input data. in prediction, we use an approach combining the prediction scores of individual sentences and blocks of sentences to optimize our ability to locate the targeted single, continuous hcd section. our implementation of the methods described below can be found at darc.stanford.edu/ants. ## results our results are reported using the aforementioned sample of 393 form 10-k documents from s&p 500 companies. we use three evaluation metrics: precision, recall, and jaccard index. precision represents how well our section identification algorithm captures positive sentences, penalizing the situation where sentences outside of the true hcd section are determined to be part of that section. in practice, however, extraneous sentences at the outside edges of the hcd section may be acceptable if most of the section itself is correctly labeled. for this, we rely on the recall score, which represents how much of the targeted section is captured. finally, to capture both the precision and recall metrics together, we use the jaccard index, which penalizes both false positives and false negatives. we calculate the three metrics described above for each predicted hcd section from a document varying the number of training documents for each model i . to characterize the performance of model i , we take the mean score from all predicted documents. documents without a predicted hcd section receive a score of zero for each metric. the plots in figure 3 show these mean scores. ## conclusions in this paper, we propose a practical framework to extract continuous segments of text from unstructured documents, with a particular focus on text-intensive research in business and social science. the ants framework utilizes a pre-trained bert model to identify targeted sections 96% of the time using only 5 training examples. this general framework can enable subject matter experts to accelerate their research by reducing the time commitment needed to extract large amounts of relevant text given a very small number of training examples. our proof of concept using human capital disclosure sections of sec filings demonstrates that manually coding only a few documents provides enough training data for a model to effectively identify the relevant section of the remaining documents. furthermore, the success of this framework opens a number of other valuable research questions from the same documents. for instance, what distinguishes the official hcd text from thematically similar data (as flagged by ants) in the remainder of the document? or, how did companies report human capital information prior to the sec's disclosure requirement? the ants framework provides the opportunity for researchers to use additional domain knowledge to integrate the sentence-level scores from a trained model. in this report we use a section picking algorithm that is constrained to identify only a single contiguous section to mirror the sec filing structure. an ants framework that could, for instance, identify boilerplate language from corporate charters, could be tuned based on the known length, location and number of boilerplate sections. the wealth of trained models also provides the opportunity to extract or flag relevant text from semi-structured documents (e.g., html), spoken text transcriptions or social media posts. while our proof of concept only returns flagged sections as an output, it suggests an application as a successor to "word list" based research methods. work such as @xcite describes counting words and phrases as a proxy for an underlying theme, such as uncertainty. using ants, researchers can identify sections of interest and prepare document scores from aggregating model results. taking human capital disclosures and diversity as an example, ants could be trained with language on workforce diversity from sec filings, and then used to generate a diversity score by counting the number of sentences discussing diversity. this work could circumvent the technical and arduous task of building word lists, and provide context aware metrics that can flag a diversifying workforce without false positives from a diversifying supply chain. taken together, the ants framework demonstrates a rich set of avenues that can be used to accelerate, augment and amplify the work of academic researchers in the social sciences. as deep learning tools are released on free and reduced cost platforms (e.g., colab, openai, huggingface), researchers will build effective datasets from larger, more diverse and more subtle text sources. we hope that ants can be leveraged to facilitate this growth in text data and democratize deep learning advances in new and unexpected ways.