Satyrn: A Platform for Analytics Augmented Generation (2024)

Abstract: Large language models (LLMs) are capable of producing documents, and retrieval augmented generation (RAG) has shown itself to be a powerful method for improving accuracy without sacrificing fluency. However, not all information can be retrieved from text. We propose an approach that uses the analysis of structured data to generate fact sets that are used to guide generation in much the same way that retrieved documents are used in RAG. This analytics augmented generation (AAG) approach supports the ability to utilize standard analytic techniques to generate facts that are then converted to text and passed to an LLM. We present a neurosymbolic platform, Satyrn, that leverages AAG to produce accurate, fluent, and coherent reports grounded in large scale databases. In our experiments, we find that Satyrn generates reports in which over 86% of claims are accurate while maintaining high levels of fluency and coherence, even when using smaller language models such as Mistral-7B, as compared to GPT-4 Code Interpreter in which just 57% of claims are accurate.

## introduction in recent times, large language models (llms) have seen a meteoric rise in popularity due to their ability to generate fluent and coherent language. however, these models often struggle to produce truthful outputs @xcite @xcite . methods such as retrieval augmented generation (rag) @xcite have been developed to address such concerns. yet, they too have an important limitation: they only work when the required information is present in textual form. for many sources of information, such as relational databases, the sought after information must be derived via computation and analysis of those data. performing this analysis and providing the results to an llm is another way to augment their generation with a far broader array of information. such analytics augmented generation (aag) supports the use of standard analytics techniques to generate facts that are converted to text and used to guide the generation of accurate responses. this has motivated the development of methods in which an llm invokes tools and external computation engines for deriving information from structured data @xcite . this is also the approach of openai's code interpreter foot_0 which can be used for generating reports that present information derived from structured data. however, these approaches let the llms decide when to call these tools and write the queries required to derive the information, an inherently non-deterministic procedure that can make them unreliable in producing the set of information to communicate in a report. in this paper, we present satyrn foot_1 , a neurosymbolic platform which leverages aag for producing factual, fluent, and coherent reports. satyrn separates the analysis from the language generation process in order to steer llms using facts derived from relational databases. it deterministically produces the analysis plans required to satisfy the informational needs of a report, generates templated language describing the results, and then uses a language model to generate the final report. this separation of the analysis and generation steps along with the deterministic nature of the analysis planning improves factual accuracy compared to existing solutions like code interpreter. in our experiments on the report generation task, we see a factual accuracy of 86% using satyrn compared to 57% using code interpreter. one of satyrn's highlights is its ability to effectively utilize small, off-the-shelf models to achieve highly accurate results. in our experiments, the two local models we used were mistral-7b and 4-bit there are 50 states in total. the top 3 are alaska (969 acres), nevada (427 acres), and idaho (316.7 acres). report: ranking report target: state, name, oregon metric: median wildfire size input description: determine the rank of the target entity according to the metric plan: |1|:(retrieve_entity wildfire) |2|:(retrieve_attribute |1| fire_size) |3|:(retrieve_entity state) ... description: count the number of entities being ranked plan:|1|:(retrieve_entity state) |2|:(retrieve_attribute |1| id) |3|:(count_unique |2|) ... (2) report blueprints (3) analysis engine (4) language model report oregon ranks fourth in the nation when it comes to the median fire size. however, when compared to the states with the largest median fire sizes, oregon's fires are smaller. the top three states in this category are alaska, nevada, and idaho, with median fire sizes of 969 acres, 427 acres, and 316.7 acres respectively ... output (1) ring and data oregon is ranked 4 according to median fire size. there are 50 states in total. our main contributions are as follows: ## methods in this section, we present the components of satyrn that allow it to perform analytics on the data, irrespective of its domain, in order to generate accurate, fluent, and coherent reports grounded by this data. the high level architecture of satyrn is shown in figure 1 . beginning with the data in (1), we create a lightweight knowledge representation, called a ring, that provides simple semantic labels that describe how the data can be analyzed by satyrn. we introduce a new plan representation language called structured question representation (sqr) (pronounced seeker) that allows complex analyses to be specified according to the semantic enhancements. the input to satyrn is a structured information request and the output is a report satisfying this request with an answer and contextualizing information. this request is a json object describing the type of report to generate, the subject of the report, a metric to use when evaluating the subject, and any filters to apply to the data before analysis. report blueprints (2) specify the information to be derived via analysis of the data and are used to produce a set of executable plans. with the analysis engine (3), each of these plans is executed in order to derive the information. the outputs are formatted as statements and used as input to the llm (4) which generates the final report. ## experiments we evaluate the performance of satyrn along with two baselines on the task of generating reports for a specific information request. reports are evaluated based on their accuracy, fluency, and coher-ence. we create three report blueprints: a ranking report that ranks an entity's performance amongst its cohort, a time over time report that compares an entity's performance at two time periods, and a comparative benchmark report that compares an entity's performance with a benchmark value. further details on the specific information requirements of each report type can be found in appendix c.1. these three report blueprints are each used across eight domains: healthcare, environmental sustainability, urban housing, criminal justice, education, legal and judicial, socioeconomic, and business. a summary of the datasets can be found in appendix c.4. we create a ring for each dataset. ## results in terms of factual accuracy, satyrn outperforms the closest baseline by 29 points, even with smaller models, while maintaining similarly high degrees of fluency and coherence as measured by unieval. these primary results are captured by ## satyrn platform satyrn is a platform for enabling users to produce fluent, coherent, and accurate reports that are grounded by their data. satyrn's capabilities can be expanded with new datasets, new analytics, and new report types. to utilize a new dataset, a ring can be created. given a ring, all existing analyses can be applied to the data and all report blueprints can be used to generate reports with them. the only exception is when the data itself does not support a particular kind of report. for example, the time over time report used in the experiments requires the data be temporal so that analyses like change over time can be computed. to enable new analytics within satyrn, additional sqr plan templates can be added to satyrn. to enable a new report type to be generated, a report blueprint can be created. new blueprints can utilize any of the sqr plan templates available within satyrn to deterministically guarantee the desired information is produced when generating a report. this crosscompatibility of all components of the platform ensures that satyrn is scalable to new datasets, new analyses, and new report types. ## conclusion we present satyrn, a system that leverages aag to produce highly accurate, fluent, and coherent reports that adhere to information derived from data. we find that satyrn generates reports that contain over 86% accurate claims as compared to gpt-4 code interpreter in which just 57% of claims are accurate. notably, this is accomplished using a far smaller model, while preserving high levels of fluency and coherence. in the future, we plan to develop automatic validation methods to further improve the accuracy of generated reports. additionally, we plan to allow language to be used as input rather than structured information requests.