U nified SKG : Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022)

Abstract: Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communities, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UnifiedSKG framework, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset. We use UnifiedSKG to benchmark T5 with different sizes and show that T5, with simple modifications when necessary, achieves state-of-the-art performance on almost all of the 21 tasks. We further demonstrate that multi-task prefix-tuning improves the performance on most tasks, largely improving the overall performance. UnifiedSKG also facilitates the investigation of zero-shot and few-shot learning, and we show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. We also use UnifiedSKG to conduct a series of controlled experiments on structured knowledge encoding variants across SKG tasks. UnifiedSKG is easily extensible to more tasks, and it is open-sourced at https://github.com/hkunlp/unifiedskg.

## introduction structured knowledge (e.g., web tables, knowledge graphs, and databases) stores large amounts of data in organized structures, forming a basis for a wide range of applications, e.g., medical diagnosis, personal assistants, and customer relations manage-ment. accessing and searching data in structured knowledge typically requires mastering query languages through professional training. to promote the efficiency of data access, structured knowledge grounding (skg) systems ground user requests in structured knowledge and produce various outputs, including computer programs (e.g., sql and sparql), table cell values, and natural language responses (figure 1 ). for example, semantic parsing @xcite converts natural language questions into formal programs; knowledge-base question answering @xcite derives answers from tables or knowledge graphs. skg has attracted significant interest and has been studied through different tasks defined by different communities. recent developments in tasks, models, and datasets for skg have led to taskspecific modeling advances, making each task's progress seemingly unique and incompatible. a main reason is that skg tasks are heterogeneous. different types of structured knowledge, such as databases or knowledge graphs, lead to highly specialized encoders @xcite @xcite . some skg tasks, e.g., semantic parsing, use customized decoders to generate programs @xcite . therefore, instead of solving common challenges in skg research, improvements in skg have been prone to be exclusive to a single task, domain, or dataset. in this paper, we propose the unifiedskg framework to advocate for a unifying view of 21 skg tasks across six task families and multiple data domains ( ## conclusions in this paper, we propose the unifiedskg framework to promote systematic research on struc-tured knowledge grounding by unifying 21 skg tasks. using unifiedskg as a benchmark, we demonstrate that finetuning t5 on individual tasks achieves state-of-the-art results on almost all 21 tasks. we show that multi-task prefix-tuning benefits most skg tasks, largely improving the overall performance. for structured knowledge encoding, we find that the effectiveness of encoding variations varies across tasks. moreover, unifiedskg is a challenging testbed for zero-shot and few-shot learning, shown by the poor results of large plms.