Structured Extraction of Terms and Conditions from G erman and E nglish Online Shops (2022)

Abstract: The automated analysis of Terms and Conditions has gained attention in recent years, mainly due to its relevance to consumer protection. Well-structured data sets are the base for every analysis. While content extraction, in general, is a well-researched field and many open source libraries are available, our evaluation shows, that existing solutions cannot extract Terms and Conditions in sufficient quality, mainly because of their special structure. In this paper, we present an approach to extract the content and hierarchy of Terms and Conditions from German and English online shops. Our evaluation shows, that the approach outperforms the current state of the art. A python implementation of the approach is made available under an open license.

## introduction terms and conditions (t&cs) of online shops are rarely read and even more rarely understood @xcite , although we all still accept them. in recent years, the automated analysis of t&cs has become an interesting field of research @xcite . a structured extraction of t&cs from online shops is a necessary prerequisite for such further processing in an nlp pipeline. content extraction is a well-researched task and numerous open source libraries are available. existing approaches are predominantly designed for or based on news articles and blog posts. in this paper, we will show that the existing approaches are not well-suited to deal with t&cs, partially because of the strict hierarchical structures that can be found in such legal documents, which are not common in other types of content such as news articles. in this paper, we present a domain-specific content extraction approach for t&cs, that combines both, content extraction and hierarchy extraction, and an open source python library that implements this approach. needs to combine both. our evaluation shows that the library outperforms general-purpose content extraction approaches on t&cs from german and english online shops. the library is available on github: https://github.com/sebischair/ lowestcommonancestorextractor . ## related work the existing work on content extraction has been mainly focused on news articles or generic content extraction, often based on the dataset of the cleaneval competition @xcite . to the best of our knowledge, no work exists that specifically targets or is evaluated on t&cs. @xcite described the process of content extraction as a sequence labeling problem on a document broken down into a sequence of blocks. each block needs to be classified as either content or notcontent. @xcite work with a more detailed four-class separation. another approach is a boundary detection method where a heuristic needs to determine a start-and end-block framing the whole content, i.e. everything between start-and end-block is considered content whereas everything else is discarded as notcontent. according to @xcite , the classification needs to take both html structure and the actual content into account. "purely text-based or purely html-based approaches do not have perfect results." several approaches for classifying blocks as content or notcontent can be found in the literature. @xcite propose a classification inspecting the text on a functional level using a set of so-called shallow text features. shallow text features are statistical calculations on block-level looking at domain and language independent features like link density, the average sentence length, the uppercase ratio, etc. @xcite introduce an improvement to this algorithm by also taking the html tree structure into account. pomikálek (2011) introduced a similar approach using a low amount of features to determine the likelihood of a block being content or notcontent. uncertainties are dealt with within the next step, which involves an analysis of the relative position of a block in the html tree including the classification of its neighbors. this step is based on the assumption that content blocks are to be found near other content blocks (and vice versa). @xcite tried to solve the task by finding a maximum subsequence in tokenized html documents, where each token is assigned a score determined by token-level classifiers. different classifiers like simply assigning predefined scores to words and tags and more advanced classifiers which combined naive bayes classification with features of the surrounding tokens were investigated. there is a number of synonyms to the process of "content extraction" @xcite ) like boilerplate detection/removal @xcite , template matching @xcite and cleaning @xcite . in addition to extracting the content, a (relatively small) number of approaches also try to extract the hierarchy of the content. according to @xcite , the nested hierarchy of an html document can contribute some information to hierarchy extraction but does not necessarily coincide with the actual hierarchical structure of an html document. the html tags originally meant to structure a document and indicate headings are often misused for seo or not used at all. @xcite introduced a segmenting method extracting the hierarchical structure of html documents based on the differences in the visual styles in hierarchical headings. they defined a set of rules based on the way humans read hierarchically structured content. according to them, headings a characterized by more prominent visual styles (the same on one level of hierarchy) preceding the blocks they describe. @xcite used a similar approach with only nine parameters. the parameters include the number of child nodes, text length of nodes and the styling of succeeding content. in order to derive requirements for the extraction of t&cs, we compared existing extractors and our expected results. in addition, we manually inspected t&c pages to detect patterns and domain-specific characteristics. the test data was sampled from the data set by @xcite which is available under the cc by-sa 3.0 license on github foot_0 . we investigated the extraction results of the t&cs pages of 30 german and 20 english online shops. 15% of the english sample had to be adjusted, as the url from the data-set did not point to the actual t&cs page. ## approach and implementation this section covers the approach we developed based on the identified requirements and the design and implementation of the structuredlegalex-traction library that implements the approach in python. ## evaluation the performance of the developed algorithms is evaluated in this section. besides the sample used to derive the requirements, there is also another sample to evaluate the library to prelude overfitting. ## conclusion we introduced a new content extraction algorithm that performs better than existing solutions in its specific domain of t&c web pages. since the algorithm is based on some domain-specific assumptions, it is unclear how successful it would operate on a generic web corpus. further research in the field could answer this question. initial small tests looked promising under the assumption that the content of the page is not interrupted. the style extraction, which is currently based on selenium can be considered the performance bottleneck, as retrieving certain css properties takes a rather long time. one should look for a more efficient solution to extract the rendered style. in addition, several content extraction threads can op-erate in parallel. the general functionality of the rule-based approach to hierarchy extraction could be demonstrated. the general idea of @xcite was extended by an enumeration detection due to the frequent usage of enumerations to structure t&c pages. it is much more difficult to achieve similar success rates in hierarchy extraction as with content extraction, due to the many irregularities in visual representation. this is probably due to the fact that the operators of different online shops often want to highlight very different elements from the contract text visually. in cases where such outliers do not occur in the visual representation, hierarchy extraction yields good results.