Papers
arxiv:2408.02442

Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models

Published on Aug 5
Authors:
,
,
,
,
,

Abstract

Structured generation, the process of producing content in standardized formats like JSON and XML, is widely utilized in real-world applications to extract key output information from large language models (LLMs). This study investigates whether such constraints on generation space impact LLMs' abilities, including reasoning and domain knowledge comprehension. Specifically, we evaluate LLMs' performance when restricted to adhere to structured formats versus generating free-form responses across various common tasks. Surprisingly, we observe a significant decline in LLMs' reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks.

Community

We recently added gpt-4o-mini-2024-07-18 results with OpenAI latest structure outputs API on our v3 edition

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2408.02442 in a model README.md to link it from this page.

Datasets citing this paper 3

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2408.02442 in a Space README.md to link it from this page.

Collections including this paper 2