Concise and Precise Context Compression for Tool-Using Language Models (2024)

Abstract: Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process.Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths.To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio.Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.

## introduction the advent of tool-using language models represents a significant extension of the capability of language models, including but not limited to interacting with the internet through the web browser, retrieving knowledge from an extended database, or even driving other models or devices @xcite @xcite ; cheng * work done during internship at huawei † corresponding author ## related work tool-using language models in terms of flexibility, the tool-using capability of language models can be broadly categorized into two types: supporting limited pre-defined tools as built-in features, or enabling arbitrary tools by reading corresponding documentation in the context. the first involves integrating predefined tools like search engines, calculators, and python interpreters @xcite @xcite @xcite @xcite @xcite . the second more dynamically trains lms to utilize tools by reading their documentation, expanding their application scope. this approach is exemplified by chatgpt plugins (openai, 2023) and further developments in open-source lms @xcite ., compressor decoder t 0 t 1 t 2 t 3 t 4 s s s c 0 c 1 c 3 doc query tool call ground truth other docs… per-token cross-entropy loss compressed documentation c 0 c 1 c 3 emb(t 2 ) emb(t 4 ) compressed plain blocks key / plain blocks from raw documentation ## method our approach uses two strategies to improve the basic soft context compression for tool-using language models. in this paper, we name the language model which generates the output as the decoder, and refer to the model which generates the compressed sequence as the compressor. although the decoder and the compressor are the same model in our work, we differentiate them from each other in our description for clear logic. we start with basic context compression, then introduce our two strategies, and finally describe our approach which integrates all of them. ## experiments 4.1 basic settings to evaluate our approach, we conduct experiments to train models with context compression and test the performance on tool-using benchmarks. our goal is to investigate the variations in model performance under different compression ratios and compression strategies. base models in all of our experiments, we use the same base model as both the compressor and the decoder, and use llama-7b @xcite to initialize all the base models. according to the manner of compression, we categorize base models into three cases: note that we refer to existing compression approaches such as rmt @xcite and autocompressor @xcite as basic compression, and the overall context compression case is equivalent to basic compression plus block compression. therefore, we study the influence of block compression strategy through extra analysis experiments in section 4.5 which provide fair comparison between our approach and these existing soft compression approaches. ## conclusion in this work, we propose an approach to compress tool documentation into concise and precise summary sequences. there are two main challenges in achieving our gold. first, context compression approaches suffer from uncontrollable compression loss, leading to key information loss. second, existing approaches cannot generate summary sequences of variable length, and thus cannot support preset compression ratio. we propose two compression strategies to deal with these challenges. to avoid key information loss, we propose the selective compression strategy which allows the key information to be retained as raw text tokens. to support preset compression ratios, we propose block compression which chunks the full sequence to be compressed into small blocks to realize variable length compression upon fixed length compression model. we evaluate our approach in two tool-using benchmarks. results show that our method can reach at least comparable performance to the baseline without compression, while achieving up to 16x compression ratio. furthermore, we explore the effects of our two compression strategies, and provide a training recipe of context compression for tool-using language models as a result.