Structured Object Language Modeling ( SO - LM ): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising (2024)

Abstract: In this paper, we study the problem of generating structured objects that conform to a complex schema, with intricate dependencies between the different components (facets) of the object. The facets of the object (attributes, fields, columns, properties) can be a mix of short, structured facts, or long natural-language descriptions. The object has to be self-consistent between the different facets in the redundant information it carries (relative consistency), while being grounded with respect to world knowledge (absolute consistency). We frame the problem as a Language Modeling problem (Structured Object Language Modeling) and train an LLM to perform the task natively, without requiring instructions or prompt-engineering. We propose a self-supervised denoising method to train the model from an existing dataset of such objects. The input query can be the existing object itself, in which case the system acts as a regenerator, completing, correcting, normalizing the input, or any unstructured blurb to be structured. We show that the self-supervised denoising training provides a strong baseline, and that additional supervised fine-tuning with small amount of human demonstrations leads to further improvement. Experimental results show that the proposed method matches or outperforms prompt-engineered general-purpose state-of-the-art LLMs (Claude 3, Mixtral-8x7B), while being order-of-magnitude more cost-efficient.

## introduction following natural-language text generation and code generation by the state-of-the-art large language models (llms) @xcite @xcite @xcite , structured objects generation, also known as json @xcite correspondence: {atavanae,kiatkoo,hayro,shaobaij,qlimz, hameng,bouykari}@amazon.com notation) generation or key-value pairs object generation, is a challenging problem for existing llms @xcite . it is one of the most desired behaviour of llms when used in production settings beyond traditional chatbot applications. it allows llms to be used as autonomous agents that integrate seamlessly with apis, since json is the de-facto communication standard between apis, and the universal string serialization format of structured objects. it also allows to use llm outputs directly without any post-processing required, for example writing the output directly to a data store or passing it as input to subsequent functions. finally, it allows to optimize llm inference cost and number of calls and ensures self-consistency of the output by generating the entire object in a single llm call, instead of generating each field of the object independently by an llm query. general-purpose instruction-following and human-intent-aligned llms (chatbots) can be steered towards generating json objects outputs by specifying the requirement in their instruction prompts. multiple prompting techniques have been tried to ensure that the output is a valid json that conforms to a schema, with variable success @xcite @xcite . state-of-the-art llm services such as openai's gpt-4, anthropic's claude 3, and mistral ai's models, to name a few, have also recently introduced a "json-mode" that allows the user to steer the model's output towards generating json, but without strict guarantee and still requiring the model to be explicitly instructed to output the json @xcite . various wrapper libraries like jsonformer @xcite allow to decompose the json generation problem into multiple independent value generation queries for each key, then using the generated values to fill the schema of the object in a post-processing re-composition step. the drawbacks of the approaches mentioned above are 1) long prompts/instructions and extensive prompt-engineering process, 2) unstable llm's behaviour in response to prompt changes, 3) prerequisites for prompt preparation such as structured objects schema or "keys" in json, and 4) computationally expensive llms. to address these issues, we propose a selfsupervised learning model to learn a native json language model, or structured object language model (solm), that natively generates objects that conform to a given structure (schema, class, database model, relational model, api specification, etc). the proposed solm acts as an object generator, but also as an object self-regeneration machine. no instructions or prompt-engineering is required for the model, which intelligently and autonomously understands what is the best possible schema and output given the input payload. our model can also inherently perform multiple enhancement tasks while (re)generating the object. tasks include 1) creation of the structured object from unstructured noisy input, 2) auto-completion of incomplete structured input, 3) error detection and auto-correction of noisy structured input, 4) auto-normalization of noisy structured input to desired normalization schemes, 5) auto-dependency resolution and auto-enforcement of inter-dependent parts/facets of the object. in this work, we focus specifically on complex multi-facet objects with intricate dependencies between the different components (facets) of the object. the facets of the object (also known as attributes, fields, columns, properties) can be a mix of short, structured facts, or long, complex, naturallanguage descriptions. this type of structure naturally occurs in complex production use-case. examples include product listings in online stores, house listings, job listings, entity records, etc. the object has to be self-consistent between the different facets and redundant information it carries (relative consistency), while being grounded and consistent with respect to a world knowledge about the entity (absolute consistency). we use an online store product catalog as an example application. for these types of e-commerce listings, some parts of the structure (e.g. title, product description, feature bullets, etc) are free-form natural language type of content, while other parts (structured metadata) are short form data-type and enumerationconstrained type of content. the proposed structured object language model handles the interleaving of these different types of content and ensures self-consistency between the natural-language portions and the structured content portions. in this paper, starting from a general purpose 7b parameter pre-trained language model, we first train our structured object language model using novel targeted denoising functions in a selfsupervised manner (solm self supervised). the current model is then further fine tuned based on few human generated high quality human demonstrations to align the llm to human preferences (solm sft: supervised fine tuning). we compare this approach against prompt-engineering of sota llms, namely claude 3.0 sonnet and mixtral-8x7b-instruct, using two different promptengineering paradigms (whole object generation versus individual attributes generation). results show that the proposed solm model is able to match the performance of prompt-engineered claude 3.0 sonnet while being order of magnitude more cost-effective. ## self-supervised training in the following, we use the pre-trained mpt-7b as the backbone transformer architecture. mpt-7b is a decoder-only transformer pre-trained on english text and code including 1 trillion tokens (mo-saicml, 2023). however, the proposed approach can be applied to any generative model (encoderdecoder or decoder-only). mpt-7b supports alibi position encoding for long text processing and generation regardless of the training text length and flash attention @xcite for less gpu memory usage and a faster attention algorithm. the self-supervised learning approach does not require human labeled data and uses denoising techniques on a corpus of existing noisy data and tasks @xcite . the proposed noising functions are designed to minimize hallucination while preserving most of the pre-trained model's world knowledge. we define two modes of operations: one where the output has to strictly be grounded in data explicitly mentioned in the input payload/context, and one where the model is encouraged to rely on its inductive biases to guess the most plausible values even if not explicitly mentioned in the input payload. those two modes of operation allow to cover a range of applications of structured objects generation or regeneration. sec 6 shows use-cases for the two modes. ## supervised fine tuning through the use of denoising functions, the selfsupervised denoising trains the llm to generate structured objects that conform to the target schema. although it is effective in adapting a pretrained llm to the desired domain, as shown in section 6, the resulting model performance is limited as it is not explicitly trained to generate human preferred objects, especially on the subjective parts of the object (e.g. long description and free-form bullets). supervised fine tuning (sft) is commonly used in the literature to align an llm to desirable user responses @xcite . given a structured object, there exist a notion of human desirable or preferred responses. the desired output is one that contains all relevant and factual information representing the data. the key to sft lies in a demonstration dataset generated by human experts. ## data corpus we validate our approach in the domain of ecommerce structured product catalog data. ## conclusion this paper proposes a new approach to generate structured objects in a single pass without needing any prompt nor objects' schema. the structured object language model (solm) is trained using a novel self-supervised training method incorporating a combination of targeted noising functions to help create or improve structured objects with complete, correct, and normalized components. the self-supervised model is further fine-tuned on hu-man labeled data to improve the quality of the freeform text components (solm sft). in future work, we will extend the training stages by incorporating reinforcement learning from human feedback (rlhf) and preference optimization (dpo) to better capture human preference. prepare the input, we applied diverse noises (that are also different from the noising functions used in training) on each component and expected the llms to improve and correct the noisy component. for free-form texts we used rouge scores to compare the generated texts with the reference texts in the original structured objects. since the self-supervised model is specifically trained on the structured objects, it's expected to perform better. also, the zero-shot mixtral does not have any information about the structured objects schema. thus, only the self supervised solm is able to generate the list of tabular attributes that are missing in the structured object. fig. 3 shows the rouge scores of the zero-shot mixtral versus ssft for unstructured text generation.