Inferring about fraudulent collusion risk on B razilian public works contracts in official texts using a B i- LSTM approach (2020)

Abstract: Public works procurements move US$ 10 billion yearly in Brazil and are a preferred field for collusion and fraud. Federal Police and audit agencies investigate collusion (bid-rigging), over-pricing, and delivery fraud in this field and efforts have been employed to early detect fraud and collusion on public works procurements. The current automatic methods of fraud detection use structured data to classification and usually do not involve annotated data. The use of NLP for this kind of application is rare. Our work introduces a new dataset formed by public procurement calls available on Brazilian official journal (Diário Oficial da União), using by 15,132,968 textual entries of which 1,907 are annotated risky entries. Both bottleneck deep neural network and BiLSTM shown competitive compared with classical classifiers and achieved better precision (93.0% and 92.4%, respectively), which signs improvements in a criminal fraud investigation.

## introduction in the last five years, brazil's federal government invested @xcite in 23,352 public works contracts adding up to r@xmath0 49.3 billion in may 2020). those works consist of all sorts of projects from oil refineries to ports, from soccer stadiums to power plants, from tunnels to dams, and are developed on a continentalsized territory, generating an endless and growing quantity of information regarding those projects. thereupon, public works procurements are a preferred field for collusion and fraud @xcite . brazilian federal police have been working on fraud investigations on public works for the last four decades and develop its investigation based on a highly skilled group of experts formed by civil, electrical, mechanical, computer engineers, and accountants @xcite . the types of fraud investigated are mainly collusion (bid-rigging), overpricing, and delivery fraud (quality and quantity of services and materials). we will bring the knowledge accrued during those decades to enhance our data understanding @xcite . as described in foundation (2020), public works contracts are made via procurement, that is the process of public administration uses to make all its contracts. every procurement step is usually publicized by a call for application, and any interested people (or enterprises) around the world can obtain data from all available government journals of brazil (the prominent public information journal in brazil is the diário oficial da união -dou). despite being easy to access, tables, texts and documents do not bring any other annotated data for classification, even less for fraud detection. those types of datasets have been studied, for example, by named-entity recognition in @xcite and linking open data, as in @xcite . on the other hand, detecting and proving fraud on construction procurements is a laborious task, consuming around one month of forensic expert work per procurement/contract. furthermore, it is essential to detect and combat fraud since a procurement first step because, as observed @xcite , over-pricing is hardly obtained without collusion as most prices are set during procurement. so, it has been the object of many studies @xcite @xcite @xcite , but none of them used unstructured data to produce its goals. based on these presented statements, this work is focused on present a new dataset with textual information extracted from brazilian public government journals with an annotated ground-truth by forensics experts. it is, also, included in this work an initial classification methodology using a bi-lstm model and all early results are compared with main state of the art techniques. the manuscript is organized as follows: in section 2 are presented the related works about fraudulent collusion on public works contracts in official texts. section 3 explains our proposed methodology. section 4 contains information about the results and discussion. section 5 provides conclusion points and introduce further works. ## related works it will be presented two parts: the first one, about fraud detection efforts, and the second one, about nlp classification. brazilian federal police has aspiring to improve its fraud detection mechanisms. therefore, as presented in vallim (2020) made a cbr model of paving services in the paraná state approaching paving works contracts, which are one of the most budget consuming services in a state or city level and focus of criminal activities. this model used procurement type, enterprises, contract, and georeferenced data, and aimed to classify collusion cases, all of them based in a manual approach. another way to prove and identify public procurement collusion is by the use of statistics and probability. those methods were explored on several federal police's studies and were based on joint behavior analysis of competitors who act to achieve bid-rigging. it was successfully applied to oil-related contracts using operation car wash foot_0 information @xcite and for infrastructure projects @xcite with capped first-price auctions. brazilian comptroller general of the union (cgu), a national auditing public agency, also has several initiatives to reach a reliable classifier for public procurement fraud. in ralha and @xcite was elaborated a unsupervised evaluator that, using a priori rules, evaluate the possibility of a certain group winning a given tender. they used structured data to bring suspect groups to be evaluated by experts. the work developed in @xcite focused on the evaluation of fraud risk in government agencies using a naïve bayes classifier for audit planning by the use of structured data and patterns of fraudulent activity. @xcite used traditional neural networks and deep neural networks (dnn) to elaborate an early alarm system. the cgu studies usually have as features and fraud indicators: the number of bids, estimated cost and price relations, relations between public and private parts, political links of political parties, etc. carvalho and carvalho (2016) achieved good results using bayesian models with structured data from penalties database. they used data enrolled from the federal civil servants, servants' roles and income, number of accounts judged irregular and number of regularity certificates on an agency unit, and affiliated servants of each management unit. @xcite uses ann and structured data on poland's highways public procurements. they used the number of enterprises, price differences, contract order in the same place, and set of propositions to assess its fraud risk. works using tf-idf in procurement documentation, as presented by @xcite , tested logistic regression, svm and naïve bayes on potential corruption. their model had no annotated data, so it was focused in finding one bid tenders which "could be potentially suspicious." natural language processing is not often used to classify public procurement documents for risk or fraud. the technology is used for assessing fraud risk in health care claims @xcite and financial reports @xcite . public works publications data are not uniform enough to be structured, and, even if it is possible, it would be extremely laborious and it might be done at the cost of losing some unknown or undetected features. all these studies suggest that fraud or risk public procurement classification has been developed based mostly on structured data and the use of nlp for this specific kind of classification is rare or nonexistent. ## proposed methodology the proposed methodology is presented in figure 1 and detailed on along with the next subsections. the workflow presented in figure 1 is formed by two actors: dataset and classification, respectively. ## results from every hundred tests ran for each model described above, it was computed precision, recall, and f1 score with respective standard deviation. those results, shown in the best precision was achieved by the bottleneck network with 92.8%, the best recall was achieved by naïve complement with 97,2%, but the best value for f1 score was by passive-aggressive with 93,4%. although the smaller f1-score, both neural networks classifiers had higher precision, in other words, they classify less false positives entries. and knowing that the final model can raise a flag for further investigations and, on a finite resources police, that 1% difference equaling to an average monthly 569 entries, it is preferable to increase precision and lower recall than the opposite. the use of classical deep artificial neural networks classifiers proved that it is possible to use it on natural language processing applications to classify the procurement publications dataset and reach a reliable model to sort out risky procurements. among the shelf feature extraction models, tf-idf showed to be a better abstraction for the dataset and the better classical and neural networks classifiers obtained f1-score results over 93%. both bottleneck deep neural network and bi-lstm proved to be competitive with traditional classifiers and achieved better precision, which is more desirable (over recall) in a criminal fraud investigation. another contribution is towards in to use a basic set of lstm models on a dataset without temporal correlation, as traditionally used by many other approaches. in the dataset used in our experiments, there is not any temporal correlation (or other features as dependency discourse parsing with word base, for example) among those collect data. despite this feature, the used lstm models achieved high performance, when compared with all baseline techniques. this issue opens new opportunities to possible developments using these models for solve situations when an application for text analysis is achieved their limits (or bounded by boundary conditions), as a temporal correlation in the text fragment to be classified, for example. ## conclusions and future works this work presented a preliminary evaluation of the dnn based on the lstm model applied to fraudulent collusion detection from public works contracts in official texts. this proposed approach was compared with several state-of-the-art text classifiers (baseline classifiers), and the proposed method achieved competitive results for precision, recall, and f1 score. the proposed annotated dataset of the brazilian public procurement calls allows researchers to explore a new form of fraud risk classification based on natural language processing and expert knowledge integration on its labeled data. the dataset covers already more than 22 years of publications, including mode than 15 million entries, and it will be available for academics researches. on the other hand, despite the full range of customization of neural networks, it is possible to achieve even better results. it will be studied in the next works, new techniques to improve and customize feature extraction for the specific dataset and all lstm models. for example, it is expected that data augmentation techniques should improve outcomes because of the small amount of annotated data available in the proposed dataset. still, it is one of the many ways to achieve performance improvements in the classification process.