Audit Report Coverage Assessment using Sentence Classification (2023)

Abstract: Audit reports are a window to the financial health of a company and hence gauging coverage of various audit aspects in them is important. In this paper, we aim at determining an audit report’s coverage through classification of its sentences into multiple domain specific classes. In a weakly supervised setting, we employ a rule-based approach to automatically create training data for a BERT-based multi-label classifier. We then devise an ensemble to combine both the rule based and classifier approaches. Further, we employ two novel ways to improve the ensemble’s generalization: (i) through an active learning based approach and, (ii) through a LLM based review. We demonstrate that our proposed approaches outperform several baselines. We show utility of the proposed approaches to measure audit coverage on a large dataset of 2.8K audit reports.

## introduction financial audit is a complex process used by organizations to assure the stakeholders about the quality and trustworthiness of the governance @xcite , @xcite . auditors examine data, documents, systems and processes, physical assets to ensure that they comply with the required standards, guidelines, laws and regulations and also to ensure that the reported financial information is fair and accurate. outside the organization, stakeholders use audited financial statements (fs) -such as balance sheet, income statement, cash-flow statement etc.-for making important decisions such as investments, loans, taxation and so forth. one important outcome of an audit is the audit report prepared by the auditors, wherein the auditor declares the fs are free from material misstatement, are fair and accurate and are presented in accordance with the relevant accounting standards. if not, the auditor * work done while working at tcs research identifies several types of issues, makes suggestions for improvement, and identifies instances of non-conformance, misinformation, irregularities, inconsistencies, errors, inaccuracies, frauds, lapses, non-compliance, violations etc. given the crucial importance of audits, and the high demands on the knowledge, experience and efforts of the auditing team, it is important to measure the quality of an audit in order to ensure that it was carried out efficiently and effectively. poor quality audits, whether intentional or not, can have disastrous consequences, such as frauds, loss of earnings, loss of goodwill, litigations, inability of the company to function as a going concern and even bankruptcy; e.g., see @xcite . there are many reasons why an audit can be of poor quality: lack of expertise in the auditing team @xcite , compromised auditor independence @xcite , biases, conservatism (recognize bad news rather than good news) @xcite and risk-averse attitudes of auditors, non-cooperation from management, insufficient time/efforts spent in auditing etc. the sarbanes-oxley act 2002 in the us is explicitly aimed at improving auditing and public information disclosure, in the light of persisting scandals fueled by auditing failures such as enron @xcite and @xcite . a good comprehensive audit report is an important indicator of a good audit. audit monitoring bodies such as the chartered accountants (ca) society of india have issued guidelines on the contents of audit reports wherein they describe a set of audit aspects which the auditor should touch upon and describe. in this paper, we focus on measuring the coverage of the audit report based on such statutory requirements, as one of the initial steps to gauge audit quality. we pose the problem of gauging coverage of the audit aspects in an audit report through classification of sentences in the report into one or more of these aspects. given the ## problem definition an audit report consists of various sections mentioning details about a company being audited, responsibility of management and auditor followed by remarks or comments by the auditors pertaining to company's business operations. generally, auditors adhere to standard audit checklist that includes scope of the audit, evidence collection, audit tests, result analysis and conclusions to be drawn from audit. moreover, auditors also have to comply with any legislation by local regulatory bodies. since, the goal in this paper is to determine the audit coverage and data under consideration is of indian companies, the coverage is checked with respect to a standard auditing checklist @xcite and companies (auditor's report) order, 2020 (caro) (icai, 2020). accordingly, the union of classes from both these sources is considered as given in ## proposed approach we propose a sieved approach which combines the power of multiple techniques such as rules, a standard bert based classifier, active learning and large language models. we explain the contribution of each of the techniques individually and then how they are combined in an ensemble for the final prediction on test data. ## audit report coverage analysis audit report coverage refers to the extent and scope of an audit report, detailing what aspects of an organization's financial statements, instruments and operations have been examined and reported on, by an external auditor. as iterated earlier, we aim to measure the audit coverage through classification of sentences into audit aspects specified in regulatory checklists. we consider that if a sentence is mapped to a class, it is generally commenting about aspects of that class and in turn achieves the objective of checking that class. at the document level, sentences commenting on the various classes can then be an indicator of which classes were checked and reported upon. we consider the set of 2812 reports for measuring audit report coverage on the complex classes and report on it from two perspectives: 1. checklist coverage: in this perspective, if a class appears at least once in the audit report then we can consider that aspect is covered in the audit. this is mainly important to check the compliance requirements where the regulatory bodies expect mandatory coverage of specific areas. figure 2 shows the number of reports in which the complex classes were reported on at least once. we can conclude that classes like human resource and payroll processing, litigations, operational and administrative expenses, purchase and procurement, and working capital show considerably lower coverage than other classes. the lower coverage could be due to (a) absence of litigations or (b) lower importance given by the auditor for that aspect. e.g., most companies may not be involved in litigations or issues relating to human resources, hence those aspects may be skipped. 2. weighted coverage: through this perspective, the number of sentences specifying a certain class can be considered as weight/importance devoted to the corresponding aspect. this can be used by the stakeholders for analyzing the weightage given by the audit report for a specific aspect, for e.g. while lending money to a firm the bank can check the focus given on aspects like payables, internal control, revaluation of fixed assets, etc. figure 3 shows the distribution of weightage for the complex classes over the considered reports. we observe that some of the classes such as cost records and working capital have comparatively less weightage than classes such as payables and fixed assets. this helps in understanding the importance auditors place on certain aspects and their implications on the functioning of the company. ## conclusion and future work in this paper, our objective was to find whether all the necessary audit aspects are being covered in an audit report. we proposed a set of 21 classes corresponding to these audit aspects. we proposed a weakly supervised approach for automatic multiclass multi-label classification of sentences in an audit report. due to absence of training data, we use a rule-based technique to automatically create labelled dataset for training a bert-based sentence classifier. further, we employ two novel ways to improve the generalization -(i) through an active learning based approach which needs manual annotation efforts and, (ii) through a llm based review which needs efforts for prompt engineering. given the complex and domain specific semantics of the classes and unavailability of labelled data, we were still able to achieve the f1-score of more than 75% with our approaches outperforming several baselines. we also showed the utility of the proposed classification approaches to measure audit coverage on a large dataset of 2.8k audit reports. as part of future work, we would like to explore open source llms further for our sentence classification problem. from domain point of view, we plan to extend our techniques for different stakeholders such as regulatory bodies or banks to automatically evaluate the audit reports in an unbiased way. we also plan to apply these techniques for audits in other domains like software quality audits or environment, social & corporate governance (esg) audits, using domain knowledge such as audit guidelines from the respective domains.