Symbol- LLM : Towards Foundational Symbol-centric Interface For Large Language Models (2024)

Abstract: Although Large Language Models (LLMs) demonstrate remarkable ability in processing and generating human-like text, they do have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language(e.g., chemical molecular formula). Injecting a collection of symbolic data directly into the training of LLMs can be problematic, as it disregards the synergies among different symbolic families and overlooks the need for a balanced mixture of natural and symbolic data. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. First, we curated a data collection consisting of 34 tasks and incorporating 20 distinct symbolic families, intending to capture the interrelations and foster synergies between symbols. Then, a two-stage tuning framework succeeds in injecting symbolic knowledge without loss of the generality ability. Extensive experiments on both symbol- and NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models.

## introduction large language models (llms), such as gptseries @xcite @xcite and llama-series @xcite , boosted the performance in various natural language processing (nlp) tasks @xcite @xcite . the success of these models heavily relies on natural language (nl) as the primary interface 1 for interaction and reasoning. however, the nl-centric interface confines the inputs and outputs to an nl form, which can only address certain aspects of world knowledge, such as fact @xcite , commonsense @xcite . nevertheless, a substantial amount of abstract knowledge, notably in areas like molecular formula (e.g., c 6 h 12 o 6 ) and first-order logic (e.g., istriangle(x) → sumofangles(x, 180 • )), is more effectively represented in symbolic forms rather than in nl. 2 compared to the nl form, the symbolic form covers a wide spectrum of scenarios and tends to be more concise and clear, enhancing its communication effectiveness @xcite . in particular, when interacting with robots, symbolic command sequences (such as pickup, walk) are more accurate and efficient than nl. similarly, when using programming languages (like sql and python) to call external tools @xcite , expressing this structured information in nl form can be difficult. despite the symbolic form offering a wealth of information, deploying llms directly via a symbolic-centric interface poses a significant challenge. this is largely attributed to the fact that llms are trained via large-scale unsupervised pre-training on extensive general text datasets, which inherently lack a symbolic foundation. the most straightforward approach to incorporating symbolic knowledge into llms is through finetuning @xcite . however, the format of symbolic data significantly diverges from that used during pre-training. consequently, merely fine-tuning with large heterogeneous data can lead to catastrophic forgetting @xcite . meanwhile, existing injection methods primarily stress on specific symbols, it is important to note that symbolic forms can be quite complex and vary across tasks. training llm for a specific symbolic form is both time-consuming and labor-intensive. furthermore, treating each symbol independently often overlooks the interconnections between different symbols, e.g., atom unit (e.g., bornin(obama, usa)) in first-order logic (fol) is similar in form to function (e.g., query(paris, nwr(hotel))) in api calls . upon this observation, we conduct a comprehensive collection of 34 text-to-symbol generation tasks with 20 standard symbolic forms introduced with instruction tuning format. the symbolic data comes from three sources: (1) 88.3% of the data was collected from existing benchmarks. (2) 5.8% of the data was prompted by llms. compensating for the natural absence of symbolic representations in some nl-centric tasks, prompting powerful llms can generate more novel text-to-symbol pairs. (3) 5.9% of data was generated by introducing the symbol-evol strategy, with replaced symbolic definitions to prevent the model from memorizing specific symbols. the above sources are uniformly leveraged to capture the underlying connections between symbols from the data view. for the framework aspect, we apply a two-stage continual tuning framework including the injection stage and the infusion stage. the injection stage prioritizes the exploitation of the inherent connections between different symbols, thereby enabling the model to thoroughly learn a wide range of symbolic knowledge. after tuning llama-2-chat models with all collected symbolic data, we obtain symbol-llm base variants. the infusion stage focuses on balancing the model's dual capabilities by utilizing both symbolic data and general instruction tuning. after combining the general instruction-tuning data with the sampled symbolic data and tuning based on symbol-llm base , we can obtain symbol-llm instruct . finally, symbol-llm series models are widely tested on both symbolcentric and nl-centric tasks, which are verified to exhibit substantial superiority. our contributions are as follows, with key values and research insights attached in appendix a, b: transform the natural language sentence into action sequence from the candidate actions. i_turn_right means turning right. i_jump means jump... ## approach in this section, we first introduce the overall symbolic data collection procedure in section 2.1 and then describe the two-stage tuning framework and the comprehensive test settings in section 2.2. ## experiments in this section, we fully evaluate symbol-llm foot_1 on three parts of experiments: the symbolic tasks in sec. gpt-3.5, claude-1, and llama-2-chat column presents the baseline performances of prompting these models under the few-shot setting. single-sft represents the models fine-tuned with single-domain samples based on llama-2-chat. symbol-llm column represents the final obtained model after two-stage tuning, with beam size 1 for decoding. the subscript of the result denotes the variances between three generations (with different beam sizes). llm is well optimized with the two-stage framework in keeping general abilities. for 7b models, symbol-llm instruct shows consistent superiority on mmlu and bbh benchmarks, with ∼4% gains compared with llama-2-chat. for 13b models, although symbol-llm instruct slightly falls behind its llama counterpart, it achieves 7.20% performance advantages in bbh. the superiority on average is obvious. while symbol-llm may not yet match the performance of closed-source llms, its well-rounded general capability is notable. these remarkable performances can make symbol-llm stand out as a foundational llm, which combines the basic nl ability and proficiency in handling symbolic language. to verify the consistent superiority of symbol-llm in general tasks, we include a broader scope of evaluation in appendix i. ## analysis in this section, we include the ablation studies (sec.4.1) and the analysis on alignment and uniformity (sec.4.2). notably, additional supplementary experiments are attached in appendix k, including tuning design choices k.1, comparison with single sft k.2, and new symbol extrapolation k.3. ## conclusion and broader impact this work pioneeringly proposes to enhance the llm capability in symbol-centric tasks while preserving the performances on general tasks, leading to balanced symbol and nl interfaces. to address the challenges of capturing symbol interrelations and maintaining a balance in general abilities, we tackle the problem from both data and framework perspectives. data-wise, we include a collection of 13098 34 text-to-symbol tasks to systematically explore underlying symbol relations. framework-wise, we implement sft in a novel two-stage manner to mitigate catastrophic forgetting. extensive experiments across three task settings (i.e., symbolic tasks, general tasks, and symbol+delegation tasks) demonstrate symbol-llm's superiority in harmonizing symbol-and nl-centric capabilities. moreover, all models and resources will be made public to facilitate a broader range of research. notably, the exploration of the role of symbolic language is not limited to a single modality or the mentioned scenario (e.g., reasoning). it is worthy of extending the neural-symbolic paradigms to other multi-modal applications, such as visual web navigation (cheng et al., 2024), 3d understanding @xcite , and embodied agents @xcite .