Machine Extraction of Tax Laws from Legislative Texts (2021)

Abstract: Using a corpus of compiled codes from U.S. states containing labeled tax law sections, we train text classifiers to automatically tag tax-law documents and, further, to identify the associated revenue source (e.g. income, property, or sales). After evaluating classifier performance in held-out test data, we apply them to an historical corpus of U.S. state legislation to extract the flow of relevant laws over the years 1910 through 2010. We document that the classifiers are effective in the historical corpus, for example by automatically detecting establishments of state personal income taxes. The trained models with replication code are published at https://github.com/luyang521/tax-classification .

## introduction a foundational task in legal document analysis is to determine a document's area of legal practice -for example, tax law. in a law firm or court administrative office, distinguishing tax-related matters from others is necessary for assigning work to lawyers or judges who have the right expertise. in the scope of income reporting and payments, having the relevant statutes and cases to hand is necessary for complying with the law without overpaying. in empirical legal studies, measurement of the volume, complexity, or other features of tax laws is predicated on extracting those laws in the first place. this essential task -of distinguishing different legal topics, such as which laws are about tax -has traditionally been done by hand. tax law materials are assembled in books and databases by trained experts. building and maintaining these materials is costly, especially in updating corpora to accommodate new material, such as newly enacted statutes. a cheaper and quicker approach would have broad benefits in private legal practice, in the courts, and in academic research. this paper provides a method and tools for automatically tagging plain-text documents with their relevance to tax law. we use machine learning applied to the law texts to distinguish tax legislation from non-tax legislation in the jurisdictional context of u.s. states. further, within the set of tax laws, we train a second model to classify the associated source -e.g. personal income, sales, property. the approach works as follows. we start with a corpus of tax code sections, with class labels generated from section headers. the plain texts of the code sections are transformed to frequency distributions over a vocabulary of tens of thousands of phrases. using those phrase frequencies as inputs, we train a binary text classifier to all statutes to predict the label of tax-related or not. we train a second multinomial classifier on the subset of tax law statutes to predict the tax source label -individual income tax, corporate income tax, property tax, etc. for the first binary classifier, we get the best performance with a regularized logistic regression (95% accuracy). for the multinomial source classifier, we get the best performance with a random forest classifier (73% accuracy). 1 we then apply these classifiers to assign labels in an historical corpus of state session laws, issued biennially over the century starting in 1910 and ending in 2010. we validate that the model attends to tax-related language in the target corpus and is 90% accurate in a sample of sentences classified as tax-related or not. as an empirical validation, we show that the volume of legal texts related to personal income tax jumps up sharply in the bienniums when a state income tax is introduced, but there is no such jump for tax laws from other sources besides personal income. our classifier and this paper add to the growing literature in applied legal nlp @xcite @xcite . more specifically, we add to the work on automated classification of relevant legal topics (e.g. @xcite . substantively, the paper is related to others focusing on tax law @xcite . besides their applications in industry, these text-based legal tools will be useful for empirical legal studies on taxation, such as @xcite and choi (2020). ## corpus data our training corpus for learning to classify texts is a collection of compiled codes from u.s. states, obtained from lexisnexis in 2017. a document is a statute section. each section is labelled for whether it is related to taxes or not. within those related to taxes, we have additional labels for the category of taxation. these labels were produced through a partly automated iterative process, overseen by a lawyer with u.s. legal training. first, we applied automated pattern matching on the legislative section headings to identify sections related to taxes. 2 the lawyer annotator inspected the associated statutes, and we refined the pattern matches until obtaining high precision and recall. second, within the tax-related statutes, we produced another set of patterns to match different tax sources. these were also iteratively checked in conjunction with the lawyer annotator. tax-related statutes that did not get assigned to a source are assigned to an other category. the application data set for analyzing flows of legislation is the state session laws for 1900-2010, obtained from heinonline. these are 2,506,513 statutes coming from the annual or biennial session laws enacted by u.s. states in any given year. the data come as raw ocr from document scans, with a sample shown in appendix table a .1. the data are segmented into statutes through an automated process. the resulting statute sections are not annotated as being related to taxes or not. in addition, they do not contain section headers, so using the header-based pattern matching approach is not feasible. before getting to the modeling, we build a joint vocabulary from the historical session laws and modern codes. in both corpora, we break down the plain text into countable features. each statute section is represented as a frequency distribution over a vocabulary of words, bigrams (two-word phrases), and trigrams (three-word phrases). the vocabulary is filtered to the 50,000 most frequent words and phrases used in both corpora, after dropping any phrases appearing in more than 75% of documents. ## machine extraction of tax laws this section describes how we train a model to predict whether a statute section is tax related. we start with the annotated modern codes sections, and the n-gram features described above. the task is to train a text classifier to reproduce the annotated labels, extracted from the statute section headers, based on the statute body text. for evaluation, we split the set of codes into 80% training set and 20% test set. for the machine classification, we experimented with two models: logistic regression and a random forest classifier (e.g. @xcite ). 3 for each of these models, we performed grid search to select hyperparameters with five-fold cross-validation in the training set. the selected settings are reported in appendix table a.2. one of the hyperparameters was the degree of additional feature selection within the training set using a chisquared test. for example, for logistic regression, the model uses only the 2000 most predictive features, uses balanced class weights, and adds weak l2 regularization (@xmath0.01). thus, we are confident with the performance of the tuned logistic regression model, for the purposes of classifying documents as tax-related or not. ## classifying tax laws by tax source the next task is to classify the source of taxed value. the corpus used is the set of annotated statute sections that were labeled as tax-related. each statute is labeled according to one of the nine classes listed in logistic regression performed poorly on this task, so we focus on the random forest classifier. 4 again, hyperparameters were selected through fivefold cross-validated grid search (appendix table a @xcite ). we have a feature set of 10k n-grams, balanced class weights, and 500 decision trees in the ensemble. looking at the metrics separately by class, panel b shows that precision is quite high across all classes, but recall varies significantly. for example, income tax has a precision of .79 and recall of .73, while sales tax has a precision of .78 and recall of .67. the highest-recall category "other", is also the lowest-precision. this means that the classifier is somewhat conservative in identifying particular tax sources and often errs by putting documents in the "other" bin. for a qualitative assessment, appendix table a.5 shows the set of words and n-grams that the model most associates with each tax-source class. they are quite intuitive -for example, "gasoline" and "fuel" are associated with energy, "cigarettes" with excise, "decedent" with inheritance, and "property" with property. overall, these phrases suggest that the classifier is capturing legally relevant features, rather than correlated features. ## application to state session laws we now have two trained text classifiers. one can distinguish legislative documents related to tax law from legislative documents on other legal areas. the second one can take documents related to tax law and tell use the value source from which the tax is derived. now we show the usefulness of these classifiers by applying them to an historical corpus of 2.5m state session laws. for each statute, we form a predicted probability that it is about taxes using the first tax detection model, trained on the modern codes. we take as tax laws the set of statutes that have a greater than 50% chance of being part of the tax code. the resulting labels are tabulated in to obtain some qualitative confidence in the classifier, we applied it at the sentence level for a large sample of statute sections. we then ranked the sentences by predicted probability of being tax related. appendix table a .6 shows the top-20 sentences in the sample by predicted class probability on tax-related; appendix table a.7 shows an equivalent list but filtering out sentences containing the string "tax". the retrieved sentences are clearly and definitively related to tax law, and the assignment does not depend on the presence of the word "tax". for comparison, appendix table a .8 shows the 20 sentences with the lowest class probability (so least likely to be tax-related). on an inspection, these don't mention anything related to taxes, and tend to have more of a focus on courts and judicial issues. for a more comprehensive evaluation about the cross-domain accuracy, we sampled 100 sentences from the historical corpus, 50 from each predicted class (tax-related or not). the lawyer annotator performed a blind tagging of the sentences as taxrelated or not-tax-related. 5 the manual annotations matched the model predictions in 90% of the observations, quite close to the accuracy from the original domain. next, the tax-source classifier is applied to the documents in the historical state session laws and class predictions are formed. to test these expectations, we estimate a panel event study model with distributed lags and leads by biennium @xcite , with state and biennium fixed effects and standard errors clustered by state. because we would expect that legislation could be passed in the biennium before the tax was first collected, the coefficients are estimated relative to two bienniums before the introduction of the tax. we look at two outcomes. first, we use the log number of words in statutes classified as tax-related and related to income tax. second, we use the log number of words in statutes classified as tax-related but related to any tax source besides income tax. figure 2 panel a shows the coefficients and 95% confidence intervals from the first panel event study regression. we can see that there is a clear jump in income-tax legislation text in the biennium before the tax introduction, which increases even more drastically in the biennium where income taxes are first collected (a six log points increase). the effect on flow of legislation gradually decreases but remains consistently positive and statistically significant, reflecting that additional legislation is needed to maintain and clarify the policy -for example, to update rates, to add exemptions, and to close statutory loopholes. meanwhile, figure 2 panel b shows the equivalent event study effect on tax-related legislation for other tax sources besides income tax. there is no effect at all on this outcome, as expected for a placebo test. together, the event study regressions indicate that the tax classifier is precisely capturing information in a single sentence to make a confident determination, and the resulting top-ranked classes are noisy and mostly incorrect. the source classifier requires longer documents, such as full statute sections. we did not have sufficient annotation resources for a trained lawyer to read lengthy statute documents and determine a single tax source. further evaluation in application domains could be considered in future work. -2 0 2 4 6 8 <-10 -10 -8 -6 -4 -2 0 2 4 6 8 10 >10 years before and after implementation of state individual income tax effect on log words in income tax statutes b. placebo effect on other-source tax legislation -2 -1 0 1 2 3 <-10 -10 -8 -6 -4 -2 0 2 4 6 8 10 >10 years before and after implementation of state individual income tax effect on log words in other tax statutes the income-tax components of historical statutes, and not other potentially correlated components. ## conclusion in this paper, we have introduced a classifier for identifying tax laws in legislative corpora, and a second classifier for identifying the source within the set of tax laws. the approach uses a groundtruth corpus of u.s. state codes labelled based on the statute section headers and a random forest classifier applied to n-gram representations of the documents. the trained classifiers perform well in the original domain according to the standard test-set performance metrics. we provide trained models and code as an opensource package for the research community. in an application, we illustrate how to use the trained model in a new domain -a century of historical statutes from u.s. states. a number of validation steps provide confidence in the usefulness of the classifier for this application and others, whether in the legal tech industry or in empirical legal scholar-ship. notwithstanding the promising results from these validations, our approach for classifying tax laws has a number of limitations which could be addressed in future work. first, the model is trained and validated only for u.s. state legislation. it is unclear whether it would work for federal or local legislation in the united states, and it is unlikely the model would work well in other countries. extending the approach to other jurisdictions would present a clear and straightforward benefit. it could also be fruitful to experiment with models that work across jurisdictions, perhaps including metadata on the jurisdiction. second, the approach has been limited to classical machine learning models and has not used deep learning. while our initial experimentation with convolutional neural nets have been unsuccessful, there is clearly additional room for performance gains using state-of-the-art models, especially transformers (e.g. @xcite . a difficulty with the standard transformer models is the maximum context length, which is shorter than many statute documents. that difficulty might be addressed by creative document segmenting, or by adopting the newer generation of long-document transformers that use sparse attention (e.g. @xcite .