sajjadanwar0/token-budgets-catalog huggingface dataset:
Framework: langchain (2023-10)
Issue: prevent a agent from exceeding the token limit
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bu; AUTO-STALE-CLOSED Feb 14 2024 by @dosubot bot as not planned (4+ months filing-to-close). Filed Oct 5 2023 by @Ethanmbinns tagged area: models + auto:question. Reporter quotable: "I am trying got find a way to prevent an agent from going over the token limit, I have looked at the docs, asked around and had no luck." Concrete error: "InvalidRequestError: This model maximum context length is 16384 tokens. However, your messages resulted in 17822 tokens. Please reduce the length of the messages." MAINTAINER ACKNOWLEDGES MISSING-PRIMITIVE: Collaborator @eyurtsev Oct 5 2023 pointed at AgentTokenBufferMemory class as workaround -- not a first-class budget primitive. CONTRIBUTOR-CONFIRMED-FRAMEWORK-LIMITATION (paper-worthy quote): @KastanDay (Contributor) Nov 2 2023: "This issue generally is one of the biggest obstacles to langchain agents -- it should truncate by default or expose a hook where we can pass a runnable to truncate the text, much like the trim_intermediate_steps runnable hook in AgentExecutor." -- explicit external-contributor admission framework lacks first-class budget enforcement. Nov 3 2023 KastanDay shipped FORK-BASED IMPLEMENTATION: "I created my own truncate function, it works beautifully." with full code at github.com/KastanDay/langchain-improved-agents. Fork-based-implementation-not-merged-upstream pattern at n=2 (parallels SMAG-002 @joaopauloschuler beyond-python-smolagents fork). MULTI-USER CONFIRMATION: @junoriosity (Scraiber admin) Oct 12-13 2023 confirmed same issue with Wolfram Alpha plugin: "I am receiving the same issue, when making use of some ChatGPT plugins." DOSUBOT-DEFLECTION-PATTERN at n=7 (LANG-017/018/019/020/024/025/028): Dosubot suggested per-LLM-call max_tokens OUTPUT limit, not aggregate-loop INPUT cap. Suggestion technically misses reporter actual question. Sub-mechanism within M-budget-primitive-missing: primitive-doesnt-exist (no first-class max_total_input_tokens or max_total_cost field at AgentExecutor level) + dosubot-deflection-from-real-question. Adds LangChain to M-budget-primitive-missing cluster -- now spans 4 frameworks (smolagents + LangChain + Mastra + CrewAI).
Framework: langchain (2023-09)
Issue: Reduce azure openai token usage
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bu; AUTO-STALE-CLOSED Dec 20 2023 by @dosubot bot as not planned (~14 weeks filing-to-close). Filed Sep 13 2023 by @easontsai tagged area: agent + area: models + auto:improvement + auto:question. Reporter quotable: "I am using the Azure openai gpt4-32k as llm in langchain. I implemented openai plugin by agent, but the cost is increasing at an incredible rate. I think the agent would ask gpt4 modal to understand the plugin openapi json that make the token usage increasing. any recommendations to reduce the token usage in agent?" Direct cost-runaway-without-cap evidence: gpt4-32k + plugin-openapi-json-injection-per-call = "incredible rate" cost growth. ZERO HUMAN ENGAGEMENT throughout 14 weeks of issue lifetime -- only dosubot bot replied with generic max_tokens setting + tiktoken counting tips, which do not address aggregate-loop-cost. Reporter never followed up. Auto-tag confirmation: framework auto-tagged as auto:improvement (framework-acknowledges-improvement-territory). Sub-mechanism within M-budget-primitive-missing: primitive-doesnt-exist + zero-human-engagement-then-stale-close. Pattern parallels SMAG-005, CRAI-006, CRAI-007. NEW PATTERN at the catalog level: bot-only-engagement-then-stale-close (zero human responses + dosubot generic deflection + stale-bot closure). HONEST EPISTEMIC CAVEAT: weaker evidence than other rows in batch (no maintainer engagement, no community confirmation, no concrete repro code) -- retained for: (a) explicit cost-framing language ("incredible rate"), (b) auto:improvement tag (framework-side-acknowledgement), (c) zero-human-engagement signature is paper-worthy as cost-observability-deflection-pattern. Reviewer-defensibility flag: this row is qualitatively weaker than LANG-028 within the same cluster; v23 prose should distinguish evidence-quality across cluster cases.
Framework: crewai (2025-11)
Issue: Agent loop is not stopped after max iterations reached
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bf with STRONGEST source-line-traced regression evidence in batch. RENUMBERED FROM ORIGINAL PASTE (was "CRAI-006" but ID collision with existing CRAI-006 #933 from batch 25; renumbered to CRAI-011). Filed Nov 6 2025 by @ehansis. CLOSED Nov 7 2025 (1 DAY filing-to-merge -- FASTEST FIX IN CATALOG) by @greysonlalonde (Contributor) via PR #3848 commit 19c5b9a + AI-bot commit 8105e74 "Fix agent loop not stopping after max iterations reached." Reporter quotable: "It seems to me like the agent loop is not being stopped once max_iter has been reached. handle_max_iterations_exceeded is called and a formatted answer is prepared, but this answer is being overwritten in following LLM calls and the loop is not stopped." EXACT REGRESSION COMMIT IDENTIFIED BY REPORTER (gold-standard archaeological evidence): "A previous revision had a break right after preparing the formatted answer, but this was lost in rev efe27bd." Path: crewAI/lib/crewai/src/crewai/agents/crew_agent_executor.py Line 208 in e4cc9a6. MAINTAINER CONFIRMS INTERNAL-TRACKING-VS-EXTERNAL-REPORT PATTERN: @greysonlalonde Nov 6 2025: "thanks for flagging this @ehansis, we have been tracking internally. I will tag this issue in a pr." -- maintainers KNEW about regression but had not fixed; external report triggered fix. NEW PATTERN (16th sub-shape in closure/engagement typology): maintainers-know-but-do-not-fix-until-external-report. AI-BOT-PR-PATTERN AT n=2 WITH MIXED OUTCOMES: @devin-ai-integration (AI bot) commit 8105e74 + PR #3848 -- SECOND AI-bot contribution in catalog (after CRAI-010 #4784). AI-bot success vs human failure pattern: CRAI-010 devin-ai PR #4784 was closed unmerged (failure); CRAI-011 devin-ai PR #3848 was merged successfully (success). AI-bot-PR pattern at n=2 with mixed outcome. SUB-MECHANISM within M-budget-primitive-missing: primitive-exists-but-broken-via-silent-regression. max_iter is documented framework feature, was working pre-rev efe27bd, regressed silently in subsequent revision. Distinct from SMAG-005 (primitive-doesnt-exist), MAST-005 (primitive-via-callback-only), MAST-004 (primitive-only-on-memory-retrieval). Pattern parallels LANG-027 (~12-month silent regression of retry decorator, exact commit 53ac1eb in PR #19420) -- both are fix-shipped-immediately-regressed at framework-feature level. POST-FIX SUSPICIOUS PROMOTIONAL COMMENT: @Saurabh0377 Nov 25 2025 left lengthy promotional comment about agentic-qa-api.onrender.com loop-injection-test middleware service. Promotional-spam-noise-pattern at n=2 (parallels MAST-017 @chorghemaruti64-creator Agenium spam). Strongest single piece of evidence: 1-day-filing-to-merge with exact regression commit identified BY REPORTER -- reporter did the framework archaeology themselves before maintainer engagement. FASTEST FIX IN CATALOG.
Framework: pydantic-ai (2025-03)
Issue: Error in multi-agent example: pydantic_ai.exceptions.UsageLimitExceeded
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bu; CLOSE-THEN-REOPEN-THEN-CLOSED-VIA-AI-BOT-PR pattern. Filed Mar 31 2025 by @dangmanhtruong1995. INITIALLY CLOSED Aug 15 2025 by Member @Kludex with deflection: "Since this issue a lot has changed, please read this documentation: ai.pydantic.dev/multi-agent-applications. I will close this preemptively, but happy to reopen if this is still an issue for you." REOPENED Aug 16 2025 by Kludex after reporter empirically retested with NEW error: "UsageLimitExceeded: Exceeded the total_tokens_limit of 300 (total_tokens=357)" -- close-then-reopen-after-empirical-counterevidence pattern at n=1. Reporter diagnosis Aug 16: "So I think the example should be changed so that the total token limit is big enough." CLAUDE-AI-BOT-PR PATTERN (NEW): @Kludex Aug 25 2025: "@claude Can you fix the example in the docs?" -> @claude (Anthropic Claude bot, Contributor tag) Aug 25 same day: opened PR #2662 + commit f0f3983 changing total_tokens_limit=300 to total_tokens_limit=500. Claude-bot PR explanation quote: "The user reported that the example fails with actual token usage of 353-357, which exceeds the documented limit of 300. I set it to 500 to provide a reasonable buffer above the observed usage. All pre-commit checks passed." Closed in PR #2662 same day. SECOND DISTINCT AI-BOT integration in catalog (after devin-ai-integration in CRAI-010/011). AI-bot-PR-attempt cumulative now at n=3 across 2 different bots: devin-ai-integration x2 (CRAI-010 closed-unmerged + CRAI-011 merged-success) + claude-bot x1 (PYAI-001 merged-success). FRAMEWORK-SHIPPED-MULTI-AGENT-DOCS-EXAMPLE-BROKEN pattern at n=2 (parallels ATGN-004 official-notebook-context-overflow). POSITIVE EVIDENCE FOR PAPER: Pydantic AI is FIRST framework in catalog with first-class UsageLimits(request_limit=N, total_tokens_limit=N) config field at agent.run_sync() level -- explicit declarative budget primitive that EXISTS. Story shifts from "frameworks lack primitive" to "frameworks have primitive but ship broken defaults that contradict their own docs." Sub-mechanism: framework-has-primitive-but-ships-broken-defaults-in-docs. NEW 6th sub-mechanism for M-budget-primitive-missing category. CLASSIFICATION CAVEAT: borderline cluster fit -- the primitive mechanism works correctly, the bug is documentation defaults too low. Could alternatively classify as standalone case rather than M-budget-primitive-missing sub-mechanism. FIRST PYDANTIC-AI ENTRY establishes 8th framework in catalog.
Framework: pydantic-ai (2026-02)
Issue: Automatic tool budget reminders via ToolReturn.content
Mechanism cluster: M-budget-primitive-missing
Notes: paper:fr; STRONGEST SINGLE QUOTE IN ARCHAEOLOGY EFFORT. OPEN at fetch tagged feature + capability-solveable. Filed Feb 19 2026 by @sarth6 (Contributor). PAPER-QUOTABLE OPENING SENTENCE: "UsageLimits(tool_calls_limit=N) is a kill switch. The model has no idea it is running low on tool calls until the hard cap fires UsageLimitExceeded and the entire run dies, often after the model already did useful work." CONCRETE BATCH-REJECTION-CLIFF EVIDENCE: "The hard cap checks at batch level. check_before_tool_call computes current + batch_size and rejects the entire batch if it would exceed the limit. Model requests 10 parallel calls with a limit of 6 -> all 10 rejected, zero execute." Reporter provides 3 detailed workarounds with each failure-mode characterized: (1) Dynamic @agent.instructions reading ctx.usage.tool_calls -- "puts the budget count in the wrong place. instructions maps to system_instruction in the Google API and {role: system} in OpenAI. That is the top of the context, before the model has read through any of the conversation history containing those tool calls. It is like putting page 8 of 10 in the header of page 1." + busts prompt cache; (2) tool return value pollution -- "pollutes the tool return schema. Instead of getting [...] directly, the model has to dig into {results: [...], remaining_budget: 5}. Every consumer of the tool type has to unwrap the wrapper"; (3) Manual ToolReturn.content -- "right position in the conversation. But it requires every tool author to manually track call counts, compute remaining budget, and format the reminder. If you have 5 tools, you are writing the same bookkeeping code 5 times." MAINTAINER ACTIVE ENGAGEMENT: Member @adtyavrdhn Feb 19 proposed @agent.on_tool_results hook abstraction. Collaborator @DouweM Mar 5: "This sounds like a good candidate for a hooks-based Capability once #2885 lands as part of #4303." Member @adtyavrdhn Mar 5: "Yep once middleware capabilities lands we will be able to close off a lot of these in one go." Reporter @sarth6 Mar 5 acknowledged: "Sounds good, I will work on a temporary package for now with the goal of deprecating it once that launches." AUTHOR-SHIPPED-EXTERNAL-PACKAGE: @sarth6 Mar 5 commit ff82bf9 + Mar 11 published pypi.org/project/pydantic-ai-tool-budget. External-package-as-de-facto-fix pattern at n=3 (parallels SMAG-002 + LANG-028). Cross-references: PR #3691 ToolPolicy + issue #2593 (tool_calls_limit recently merged) + issue #843 (closed original limit-tool-calls request). PROMOTIONAL-COMMENT-PATTERN at n=3: @fjnunezp75 Mar 15 GPU-Bridge service spam (after MAST-017 Agenium + CRAI-011 agentic-qa-api). Sub-mechanism within M-budget-primitive-missing: primitive-exists-but-only-as-hard-cliff-without-graceful-degradation-or-budget-awareness. Cross-ref to OAAS-002 same batch confirming SAME architectural pattern in OpenAI Agents SDK -- DUAL-PATH PROOF for hard-cliff-without-graceful-degradation variant. Strongest paper-quotable sentence: kill switch... the entire run dies.
Framework: pydantic-ai (2026-02)
Issue: Anthropic cache tokens double-counted in Langfuse/OTel
Mechanism cluster: M-cost-observability
Notes: paper:bu; closed Feb 24 2026 by Contributor @alexmojaki as not planned with maintainer-deflection-to-downstream-spec rationale: "this is in line with the spec in opentelemetry.io/docs/specs/semconv/registry/attributes/gen-ai/. I definitely think this is the most intuitive semantics, i.e. that usage.input_tokens in Python or gen_ai.usage.input_tokens in OTel represents the actual total number of input tokens and is not affected by whether some tokens are cached or whether you are using OpenAI or Anthropic. Langfuse choosing to add up these values is a bug IMO." Filed Feb 19 2026 by @ZihaoZhou tagged OpenTelemetry. CONCRETE DOUBLE-COUNTING MATH: input_tokens=5 (uncached) + cache_read_input_tokens=128955 + cache_creation_input_tokens=1253 -> genai-prices Usage.input_tokens=130213 (correctly summed). pydantic-ai then emits to OTel: gen_ai.usage.input_tokens=130213 AND gen_ai.usage.details.cache_read_tokens=128955 AND gen_ai.usage.details.cache_write_tokens=1253. Downstream Langfuse adds: 130213 + 128955 + 1253 = 260421 when real prompt was 130213 -- 2x inflation. REPORTER QUANTIFIED IMPACT: "Cache hit rate shows ~50% instead of ~99%; Cost estimates are ~2x inflated (and may incorrectly trigger Anthropic >200K tiered pricing in the UI); Token counts exceed Anthropic 200K hard limit, which should be impossible." SOURCE-LINE TRACED to genai-prices data.py Anthropic provider with 3 mappings sharing dest=input_tokens. NEW CLOSURE PATTERN (17th sub-shape): maintainer-deflection-to-downstream-spec -- distinct from upstream-deflection (DSPY-002). The OTel spec ambiguity becomes framework defense for not fixing. POST-CLOSURE 6-WEEK CASCADE confirms cross-tool-counting-discrepancy: @ZihaoZhou Feb 27 2026 filed langfuse/langfuse#12306; @ajastro Feb 28 2026 mirrored in test-repo; @csansoon Mar 31 2026 filed latitude-dev/latitude-llm#2555 + #2558; @JasonPan ~3 weeks before fetch mlflow/mlflow#23002 -- 4th downstream consumer affected. CROSS-ECOSYSTEM-COST-COUNTING-DISCREPANCY at n=4 systems (Pydantic AI + Langfuse + Latitude + MLflow). Sub-mechanism within M-cost-observability: cost-counts-inflated-via-cache-double-counting-cross-tool-spec-disagreement. NEW (4th) sub-mechanism for M-cost-observability category. First catalog row of single-cost-bug-propagating-across-4-downstream-tools.
Framework: pydantic-ai (2025-08)
Issue: Enforce usage limits by exact tool call count (not just request/turn count)
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bf; bug_fixed_by_framework. Reporter framing: 'single model turn that triggers many tool calls is not bounded by usage limits' -- documents that pre-fix Pydantic AI had NO enforcement of tool-call counts. A single model turn with many parallel tool calls bypassed request_limit entirely. Fixed by PR #2633 in Sep 2025 which added tool_calls_limit field directly to UsageLimits. Cross-references PYAI-002 (#4359) which POST-FIX documents the resulting tool_calls_limit as 'a kill switch' -- evidence that even when the primitive lands, it ships as hard-cliff without graceful degradation. Sub-mechanism within M-budget-primitive-missing: primitive-missing-then-shipped-as-hard-cliff. CHRONOLOGICAL PAIR with PYAI-002: pre-fix gap (PYAI-004) -> post-fix kill-switch framing (PYAI-002). Reporter @TODO_USER provides parallel tool-call evidence: request_limit checks at request boundary not per-tool-call within a single LLM turn.
Framework: pydantic-ai (2026-01)
Issue: Feature Request: First-class Context Compaction API
Mechanism cluster: M-context-amplification
Notes: paper:fr; feature_request. Reporter framing: 'Tool-heavy agents in production environments can [cause cost explosion]' -- explicit context-amplification incident with detailed API proposal for first-class compaction. CLOSED Apr 2026 by maintainer @DouweM deferring to two-pronged userland solution: (1) new 'capabilities' feature with lifecycle hooks (prerequisite PRs #2885 + #4303); (2) third-party VStorm summarization package. PUNT-TO-USERLAND closure pattern. Maintainer @DouweM Apr 2026: framework will NOT ship first-class compaction; users must compose hooks + external packages. Cross-references PYAI-007 (#4538) as PREREQUISITE: context_window primitive on ModelProfile needed before compaction logic can be threaded. Sub-mechanism within M-context-amplification: runtime-loop-level compaction-deferred-to-userland-hooks-plus-external-package. PUNT-TO-USERLAND closure pattern is the 18th closure sub-shape in catalog -- distinct from punt-to-spec (PYAI-003) and punt-to-vendor (CRAI-014).
Framework: pydantic-ai (2025-10)
Issue: Better handling when max tokens limit results in incomplete tool call arguments
Mechanism cluster: M-context-amplification
Notes: paper:bf; bug_fixed_by_framework. Reporter PROVIDER-ASYMMETRY EVIDENCE: 'I encounter this only with Anthropic models. From what I gather, it's pretty easy for me to hit the output token limit for some tool calls that I make for file editing, and it seems that this results in completely dropping whichever arg was being written, and leaving an incomplete tool args (even missing the closing brace).' Maintainer @DouweM accepted as a bug: 'I think the right solution would be for the developer to get a clearer error message, and then either increase the max_tokens settings, or modify the prompt/tool definitions to need fewer tokens.' Fixed via IncompleteToolCall exception in PR #3137. Anthropic-specific output-token truncation silently corrupts in-flight tool-call JSON. Sub-mechanism within M-context-amplification: runtime-agent-step-loop hitting output token limit silently drops tool-call args mid-write. PROVIDER-ASYMMETRY pattern: Anthropic-only output-truncation behavior -- parallels PYAI-003 (Anthropic-only cache-double-counting). M-context-amplification adds runtime-tool-call-arg-truncation as a distinct sub-mechanism.
Framework: pydantic-ai (2026-03)
Issue: Expose context window size on ModelProfile and usage limits/tokens on RunContext
Mechanism cluster: M-budget-primitive-missing
Notes: paper:mf; maintainer_framing + feature_request. MAINTAINER-FILED by @adtyavrdhn Mar 4 2026 -- HIGHEST-CONFIDENCE M-budget-primitive-missing entry in catalog (filed by framework maintainer rather than user). Maintainer proposes adding context_window primitive: '@dataclass(kw_only=True) class ModelProfile: context_window: int | None = None. The model's maximum context window size in tokens, if known. genai-prices already has this data (e.g. 128000 for gpt-4o, 1000000 for claude-sonnet-4-5), so we can auto-populate it.' Worked example for user-side cost-aware decisions in tools: 'if context_window and ctx.usage.input_tokens > context_window * 0.8: # compact, summarize, etc.' PREREQUISITE for PYAI-005 (#4137 First-class Context Compaction API) -- maintainer explicitly notes 'Prerequisite for Feature Request: First-class Context Compaction API'. Complements ContextWindowExceeded exception (#4060). Three years separate earliest 2023 budget-primitive requests (AGPT-001) from this 2026 maintainer-filed re-request; gap between requested mechanism and shipped mechanism has not narrowed at the type level. Sub-mechanism within M-budget-primitive-missing: context-window-not-exposed-as-primitive-maintainer-acknowledges-gap. Maintainer-filed entries cumulative now at n=2 (PYAI-007 + MAST-002 which was filed by Mastra lead maintainer @abhiaiyer91).
Framework: openai-agents (2025-03)
Issue: Tool Calling Running in Loop Until Max-Turn
Mechanism cluster: M-delegation-fanout
Notes: paper:bu; AUTO-STALE-CLOSED Oct 26 2025 by github-actions stale-bot at 7 MONTHS but with AGGRESSIVE 7-DAY-INACTIVITY-TO-STALE config (vs typical 30+ days other frameworks): @github-actions Oct 22 2025: "This issue is stale because it has been open for 7 days with no activity" -> Oct 26 2025 (3 days later): "This issue was closed because it has been inactive for 3 days since being marked as stale." 7-day-stale + 3-day-close = OpenAI Agents has very aggressive auto-stale config (paper-worthy operational signal). Filed Mar 17 2025 by @WSQsGithub originally tagged bug, RE-TAGGED needs-more-info by Collaborator @rm-openai removing bug label. Reporter quotable: "The agent is calling the one tool I provided for multiple times until the max_turn runs out. And each function calls have different search input parameters." Concrete SearchAgent setup with tool_choice=required + parallel_tool_calls=False + explicit instruction "Call your search tools ONLY ONCE" -- still loops. REPORTER-SELF-DIAGNOSIS SAME-DAY: @WSQsGithub Mar 17 2025 self-bisected to tool_choice=required-causes-runaway-loop. MAINTAINER CONFIRMATION OF FRAMEWORK FOOTGUN: Collaborator @rm-openai Mar 18 2025: "Yeah, tool choice required can be an unexpected footgun. See PR #203 for something that will help." Maintainer-explicit-admits-architectural-footgun pattern. MULTI-USER POST-CLOSURE CONFIRMATION (5+ months later): @LaurenceM10 May 5 2025 PAPER-QUOTABLE DILEMMA STATEMENT: "If tool usage is optional, the agent hallucinates and does not call the right tool. If tool usage is required, the agent enters an infinite loop and exhausts the token budget. I need a reliable way to enforce at least one tool call without triggering the MaxTokens loop." -- bidirectional-failure-mode-with-no-good-middle-ground at framework architectural level. Sub-mechanism within M-delegation-fanout: tool_choice=required-induces-fanout-loop OR forced-tool-call-without-stop-condition-induces-loop. NEW 6th sub-mechanism for M-delegation-fanout category. Adds OpenAI Agents SDK to M-delegation-fanout cluster as 4th framework. FIRST OPENAI-AGENTS ENTRY establishes 9th framework in catalog. Cross-references PR #203 (architectural fix in flight per maintainer).
Framework: openai-agents (2025-06)
Issue: Max turns exceeded
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bu; AUTO-STALE-CLOSED Jun 27 2025 by github-actions stale-bot at 16 DAYS filing-to-close (aggressive auto-stale config consistent with OAAS-001). Filed Jun 11 2025 by @fitzjalen tagged question. Reporter quotable: "I am wondering how we could control the agent not to exceed the maximum number of turns. Or, if we encounter this situation, can we give it a strike instruction to provide a FINAL answer based on previous thoughts/tool results?" REPORTER-SHIPPED-WORKAROUND-THAT-DOES-NOT-WORK: Reporter shows attempted-Hook with dynamic prompt that suffixes "You have reached the search limit. Please answer now." + agent.tools.clear() + agent.model_settings = replace(agent.model_settings, tool_choice=none). Reporter quote: "However, the agent mostly ignores it and encounters an error." Workaround-attempted-but-framework-resists pattern. MAINTAINER EXPLICIT MISSING-CAPABILITY ADMISSION: Collaborator @rm-openai Jun 16 2025: "Hmm good question. We do not have anything amazing here right now. Best existing solution is to try-catch the exception, then use RunErrorDetails to issue one final request to an LLM to get a final answer. Would welcome suggestions on a good path forward." First-party-maintainer-explicit-architectural-gap-admission pattern at n=3 in catalog (parallels ATGN-018 @afourney "no way to mitigate short of rewriting GroupChat" + MAST-004 @TylerBarnes "by design right now unfortunately"). POST-CLOSURE 5-MONTH-LATER FIX: @thomas-lab-openai (OpenAI staff handle) Nov 6 2025 PR #2052 "fix #844: add resume helpers for MaxTurnsExceeded runs" -- fix shipped 5 months after stale-close. NEW PATTERN (18th sub-shape): fix-ships-but-issue-stays-closed-as-stale (distinct from fix-shipped-not-released LANG-021 because here the fix did release but original tracking issue was already closed by stale-bot). POST-CLOSURE COMMUNITY ENGAGEMENT: @kawashishu Aug 15 2025 cost-implication observation: "If you do that, you probably will not benefit from Prompt Caching, causes slow performance and increased costs." -- workaround-has-additional-cost-implications pattern. Workarounds for cost issues themselves cause cost issues -- recursion strengthens type-level discipline argument. Sub-mechanism within M-budget-primitive-missing: primitive-as-strict-cliff-throws-exception (max_turns hits -> MaxTurnsExceeded raised -> no graceful-final-output mechanism). DUAL-PATH PROOF FOR M-budget-primitive-missing strict-cliff-without-graceful-degradation variant: PYAI-002 (Pydantic AI tool_calls_limit kill switch) + OAAS-002 (OpenAI Agents SDK max_turns kill switch) = 2 frameworks confirm SAME architectural pattern in same batch. Adds OpenAI Agents SDK to M-budget-primitive-missing cluster.
Framework: langchain (2025-03)
Issue: core: incorrect token count (usage_metadata) in streaming mode
Mechanism cluster: M-cost-observability
Notes: paper:bu with EXACT-SOURCE-LINE-TRACED 11-MONTH-LATER-FIX evidence. OPEN at fetch (fix-shipped-but-issue-not-closed pattern). Filed Mar 22 2025 by @andrePankraz tagged investigate + bug + core. Reporter quotable: "The aggregated token usage is totally wrong and much to high." CONCRETE MECHANISM: streaming with vLLM/Qwen/OpenAI-compatible providers returns usage_metadata PER TOKEN (input_tokens=713 every chunk, output_tokens=1), langchain_core.messages.ai.add_ai_message_chunks calls add_usage to sum across chunks producing input_tokens * count(tokens) inflation. Reporter cites EXACT source-lines: libs/core/langchain_core/messages/ai.py Line 406 + libs/partners/openai/langchain_openai/chat_models/base.py Line 2805 _create_usage_metadata. MINIMAL REPRO POSTED Mar 23 2025: Reporter shows AsyncCallbackHandler.on_llm_end captures inflated final aggregate -- final usage_metadata={input_tokens=455, output_tokens=22, total_tokens=477} when real values are 35/11/46 -- 13x input inflation, 2x output inflation. CONTROLLED-COMPARISON EVIDENCE: @le-codeur-rapide cannot reproduce with vanilla OpenAI gpt-4o (because OpenAI native API only sends usage_metadata once at end, not per token). Reporter Mar 25 explicitly clarifies: "I use VLLM Inference Server with OpenAI API." MAINTAINER ADMIN-WORK-ONLY 5+ MONTHS: @mdrxy Sep 10 2025 title rename, Sep 11 2025 title rename, Nov 14 2025 added core label, Jan 23 2026 added external label -- all admin work NO substantive engagement. NEW SUB-SHAPE: maintainer-admin-work-without-substantive-engagement. FIX-SHIPPED 11.5 MONTHS LATER: @antonio-mello-ai Mar 4 2026 PR #35545 commit ca669b8 + commit e82104a same day. ISSUE STILL OPEN AT FETCH despite fix-commit. NEW closure sub-shape (19th in taxonomy): fix-shipped-but-issue-not-closed (parallels LANG-021 closure-by-merge-not-released but inverse -- code merged, issue not updated). Sub-mechanism within M-cost-observability: streaming-mode-aggregation-arithmetic-inflates-by-token-count -- 5th sub-mechanism for M-cost-observability category.
Framework: langchain (2023-08)
Issue: Regression in structured_chat agent Output parser
Mechanism cluster: M-retry-loop
Notes: paper:bu with GOLD-STANDARD EXACT-INTRODUCING-PR archaeological evidence. AUTO-STALE-CLOSED Nov 27 2023 by @dosubot (3 months filing-to-close). Filed Aug 21 2023 by @Und3rf10w tagged area: agent + bug. STRONGEST ARCHAEOLOGY IN CATALOG (surpasses LANG-027 53ac1eb commit reference and CRAI-011 efe27bd commit reference): Reporter quotable from issue body: "This regression affects Langchain >=0.0.262. This regression was introduced with #8965." Reporter pinpoints THREE archaeological data points: (a) exact version-affected-from (>=0.0.262); (b) exact introducing-PR (#8965); (c) exact mechanism (structured_chat agent output parser fails when tool action_input contains backticks or embedded JSON). Concrete mechanism quote: "If an agent output to input to a tool (e.g. to generate an AgentAction) contains either backticks (such as to represent a code block with triple-tick), or embedded JSON (such as a structured JSON string in the action_input key), then the output parsing will fail." Reporter proposes pattern fix using regex with re.DOTALL flag. DOSUBOT-DEFLECTION-PATTERN (n=4 in catalog): Dosubot bot Aug 21 cited tangentially-related PRs #9460 + #8357 + #8965 as similar-issues, suggested 3 generic workarounds (control-character stripping, json.loads parsing, handle_parsing_errors flag) -- NONE address the introducing-PR regex-greedy-matching root cause. Reporter never engaged after dosubot. Auto-stale-bot closed 3 months later. NEW PATTERN: stale-bot-closes-issues-with-exact-introducing-PR-identified -- even when reporter has done framework archaeology themselves, auto-stale-bot still closes. THREE DISTINCT ARCHAEOLOGICAL RESOLUTION FORMS NOW IN CATALOG: (a) commit-only (CRAI-011 efe27bd); (b) commit-and-PR (LANG-027 53ac1eb + #19420); (c) PR-only (LANG-031 #8965). Each carries different evidentiary weight. Adds to fix-pattern typology silent-regression family.
Framework: langchain (2024-06)
Issue: langchain-openai - max_tokens - 2 confirmed bugs
Mechanism cluster: M-budget-primitive-missing
Notes: paper:bu with TWO-DISTINCT-REGRESSIONS-IN-ONE-ISSUE pattern + DOCS-IMPLEMENTATION-MISMATCH. Filed Jun 7 2024 by @ventz tagged plug:openai + bug. Closed Jun 7 2024 SAME DAY by Contributor @efriis as completed citing @keenborder786 explanation. TWO INDEPENDENT REGRESSIONS DOCUMENTED IN BODY: REGRESSION 1 -- ChatOpenAI max_tokens parameter does not account for prompt tokens. Reporter setup: gpt-4 (8192 context) with max_tokens=8192 fails (HTTP 400 context_length_exceeded "8225 tokens (33 in messages, 8192 in completion)"), but max_tokens=100 works correctly. Reporter quotable analysis: "It seems to me that the proper functionality in this case would be to subtract the user prompt (messages) from the max_tokens, and updating the max_tokens accordingly. Essentially, max_tokens is being invalidated due to the shared context of the user prompt." REGRESSION 2 -- max_tokens=-1 documented as "returns as many tokens as possible given the prompt and the models maximal context size" but ChatOpenAI rejects with HTTP 400. Reporter ATTACHED screenshot of langchain docs at api.python.langchain.com showing -1 documented behavior. MAINTAINER-DEFLECTION-FROM-DOCS-CONTRADICTION: Contributor @keenborder786 Jun 7 2024: "the langchain documentation does not state anywhere that it does allow this" -- directly contradicts reporter screenshot. @keenborder786 partial-explanation: "By default, OpenAI reserves 32 tokens for messages titles... For GPT-4, choose a max token count to be any figure less than (8192-32), i.e., 8160." REPORTER REPRODUCES 8160-ALSO-FAILS COUNTEREVIDENCE: "if I set 8160 with this prompt:... 213 tokens (in messages) + 8160 (completion) = 8373 > 8192 limit." MAINTAINER-CLOSE-IGNORING-COUNTEREVIDENCE: @efriis Jun 7 closed as completed CITING keenborder786 explanation BEFORE seeing reporter 8160-fails counterevidence. Reporter Jun 11 forced-escalation, then filed separate issue #22778. NEW closure sub-shape (20th in taxonomy): closed-citing-explanation-before-counterevidence-posted. DOCS-IMPLEMENTATION-MISMATCH-PATTERN at n=2: PYAI-001 (Pydantic AI ships broken total_tokens_limit=300 in docs example) + LANG-032. Sub-mechanism within M-budget-primitive-missing: max_tokens-does-not-account-for-prompt -- primitive exists but does not compose correctly with prompt-token accounting. Cross-cluster fit also includes docs-contradicts-implementation pattern.
Framework: langchain (2026-01)
Issue: Deprecation warning guides users to langchain.agents breaking stream_mode messages token streaming
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bu with FUNCTIONAL-REGRESSION-VIA-DEPRECATION-REDIRECT pattern (NEW SUB-SHAPE). OPEN at fetch tagged bug + core + groq + langchain. Filed Jan 6 2026 by @AmazyanAyoub. Reporter quotable: "Following the deprecation warning effectively breaks production streaming features, downgrading the agent from real-time token streaming to step-based buffering." CONCRETE MIGRATION-CAUSED-REGRESSION: Reporter uses langgraph.prebuilt.create_react_agent (works with stream_mode=messages token-streaming); IDE Pylance deprecation warning recommends migration to langchain.agents.create_react_agent; migration BREAKS streaming because LangChain legacy create_react_agent ignores stream_mode=messages parameter and buffers entire response. Reporter posted self-contained reproduction showing both behaviors side-by-side: LangGraph variant: "[SUCCESS] Streamed token by token"; LangChain Legacy variant: "[FAIL] Did not stream tokens; yielded full steps." Reporter expected behavior quote: "1. Ideally, the deprecation warning should not guide users to a legacy module (langchain.agents) that lacks feature parity (streaming) with the modern module (langgraph). 2. Alternatively, langchain.agents should support stream_mode=messages to match the behavior of the code it is replacing." Reporter explicit framing: "No crash error. The issue is a functional regression (streaming vs buffering)." MAINTAINER-DEFLECTION-PATTERN: Contributor @keenborder786 Jan 7 2026: "I do not understand your case, migrating from langchain_classic to langchain_v1 is a breaking change so user will be aware about this breaking change as well." -- maintainer dismisses reporter concrete repro showing functional regression with abstract appeal-to-breaking-change-awareness. Reporter Jan 7: "If you want let us contact and I will show you the problem am talking about" -- escape-to-personal-contact-channel pattern (user gives up on issue tracker). @mdrxy Jan 23 2026 added external label without further engagement (deflection-via-labeling). NEW closure sub-shape (21st in taxonomy): maintainer-deflection-via-appeal-to-breaking-change-awareness. NEW REGRESSION SUB-SHAPE: deprecation-warning-points-to-broken-replacement. Cost angle: token-streaming-vs-buffering matters because streaming reduces user-facing latency cost AND allows mid-stream token-based budget enforcement; full-response buffering breaks both capabilities. Cross-cluster fit: M-providerOptions-silently-dropped variant (stream_mode=messages parameter silently ignored).
Framework: langchain (2026-02)
Issue: ChatOpenAI silently drops reasoning_content from OpenAI-compatible providers
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bu with CASCADING-FIX-ATTEMPTS-WITHOUT-MERGED-RESOLUTION pattern. Closed Feb 17 2026 by Member @mdrxy as completed via duplicate-reference: "Closing, please refer to #34328" -- closure-by-duplicate-reference pattern. Filed Feb 7 2026 by @vivekjainmaiet tagged openai + external + bug. Reporter quotable: "When using ChatOpenAI to interact with OpenAI-compatible providers (specifically vLLM and DeepSeek) that return reasoning/thought process data, the reasoning_content field is silently dropped. Even though the raw JSON response from the server includes the reasoning_content key (verified via httpx), LangChain internal parsing in the langchain-openai package does not extract this field into the AIMessage object." Source-line traced to _convert_dict_to_message in langchain_openai base. Reporter explicit user-impact framing: separate Thought Process from Final Answer in UI for create_react_agent with thinking models (DeepSeek-R1, Qwen-Thinking). WORKAROUND-VIA-DIFFERENT-PACKAGE: Reporter explicit: "The issue is resolved if I switch from ChatOpenAI to ChatDeepSeek from the langchain-deepseek package, which proves the server is sending the data correctly." escape-to-other-package pattern at n=4 in catalog. MAINTAINER ATTEMPTED-FIX-SUGGESTION: Contributor @keenborder786 Feb 7: extra_body chat_template_kwargs enable_thinking=True -- reporter Feb 7 confirms STILL FAILS. CASCADING-FIX-ATTEMPTS-WITHOUT-MERGED-RESOLUTION (NEW PATTERN): SIX SEPARATE PRs in 3 weeks NONE confirmed merged: (1) @saakshigupta2002 Feb 8 PR #35065 + PR #35067; (2) @kelvinvelasquez-SDE Feb 12 PR #35180; (3) @dotuananh0712 Feb 21 PR #35374; (4) @jackjin1997 Feb 23 PR #35411; (5) @hnustwjj last month langchainjs#10666; (6) @nstung463 last week PR #37065. 6 PRs across 3+ months NONE confirmed merged at fetch -- bug-attracts-many-fix-attempts-but-none-merged sub-pattern (distinct from LANG-022 stuck-open-with-multiple-PRs n=3 because LANG-034 has 6 PRs and closure-by-duplicate-reference). CROSS-ECOSYSTEM CASCADE at n=4+ systems: @Ahmad-Zaaza Mar 6 ag-ui-protocol/ag-ui#1251; @codex4010 last week microsoft/agent-framework#5538; @siglimumuni yesterday swarmclawai/swarmclaw#67 -- 4-system cross-ecosystem confirmation pattern (parallels PYAI-003 4-system Pydantic-AI/Langfuse/Latitude/MLflow cascade). Sub-mechanism within M-providerOptions-silently-dropped: reasoning_content-silently-dropped-from-vLLM-DeepSeek-compatible-providers. PROMOTIONAL-SPAM-PATTERN at n=4: @veeceey Feb 8 comment "Hidden as spam" by GitHub auto-moderation -- first auto-hidden-spam instance in catalog. NEW closure-pattern (22nd in taxonomy): closed-by-duplicate-reference -- @mdrxy points to #34328 without confirming any of the 6 fix-PRs merged.
Framework: langchain (2024-07)
Issue: Maximum Context Length Exceeded Due to Base64-Encoded Image in Prompt
Mechanism cluster: M-multimodal-cost-amplification
Notes: paper:bu with STRONGEST single concrete-amplification-number in catalog. AUTO-STALE-CLOSED Nov 22 2024 by @dosubot bot as not planned (~4.5 months filing-to-close). Filed Jul 11 2024 by @tusharraskar tagged plug:openai (no bug label applied). GOLD-STANDARD CONCRETE AMPLIFICATION EVIDENCE: Reporter shows full BadRequestError: "Error code: 400 - This model maximum context length is 8192 tokens. However, you requested 254457 tokens (254201 in the messages, 256 in the completion)." -- SINGLE base64-encoded image -> 254,201 input tokens -> 31x context window overflow on Phi-3-vision-128k-instruct configured with max_model_len=8192. Setup: VLLMOpenAI client + microsoft/Phi-3-vision-128k-instruct via vllm.entrypoints.openai.api_server + base64-encoded PNG screenshot of invoice document. Reporter framework-architectural-mechanism quote: "The base64-encoded image is being considered part of the prompt, significantly increasing the token count and leading to the context length issue. Even if the model context length were 128k tokens, including a base64-encoded image in the prompt would always exceed the model limit." Reporter expected-behavior paper-quotable statement: "The model should process the image without considering the base64 string as part of the prompt token count, thereby avoiding the context length issue." MULTI-USER CONFIRMATION: @lizaibeim Aug 16 2024 (1 month post-filing): "same problem there." POST-CLOSURE 7-MONTH-LATER CONFIRMATION + CROSS-FRAMEWORK-ASYMMETRY: @killian-mahe Feb 3 2025: "Hi, I have the same problem. When I use the OpenAI SDK, my request works without a context length error, but when I use the Langchain VLLMOpenAI class, I get a maximum length error too." -- confirms OpenAI-SDK-works-but-LangChain-does-not asymmetry pattern (parallels LANG-034 reasoning_content-silently-dropped and LANG-037 in this batch with same vLLM-asymmetry signature). Three-row pattern at n=3 in catalog: LangChain-only-failure-mode-with-base-SDK-working. Sub-mechanism: M-multimodal-cost-amplification base64-image-treated-as-text-string-tokens. STRONGEST single piece of cost-amplification evidence in catalog (254K input tokens for one image, 31x context overflow). Adds LangChain to M-multimodal-cost-amplification cluster as foundation row. NEW paper category established this batch: M-multimodal-cost-amplification.
Framework: langchain (2026-01)
Issue: Support approximate token counting for image blocks in langchain-core
Mechanism cluster: M-multimodal-cost-amplification
Notes: paper:fr with REPORTER-SHIPPED-FIX-PR-MERGED-FAST pattern. Filed Jan 25 2026 by @subhashyadavon tagged feature + external + core. CLOSED Jan 27 2026 by Collaborator @ccurme via PR #34883 (2 DAYS filing-to-merge -- ties CRAI-011 as fastest in catalog with different mechanism shape). Reporter cost-impact paper-quotable: "If the default approximate counter is used, a single base64-encoded image can appear to consume 25,000+ tokens simply because of its string length, causing trim_messages to aggressively discard virtually all other conversation history." Reporter framework-architectural-mechanism: "Currently, when count_tokens_approximately encounters a message with a list of content blocks (standard for multimodal models), it falls back to len(repr(message.content)). For messages containing base64-encoded images, this results in counting tens of thousands of characters as if they were text tokens, leading to massive overestimation of the token count." Reporter user-impact-on-context-management: "Currently, I have to work around this by either stripping images from my trimming logic or implementing a custom token counter." Source-line: libs/core/langchain_core/messages/utils.py count_tokens_approximately function. Proposed fix: iterate content blocks, identify image_url blocks, apply fixed 85-token penalty (aligned with OpenAI low-res base penalty) instead of counting characters. REPORTER-SHIPPED-FIX-WITH-VERIFICATION-EVIDENCE: "I have already implemented this fix in a local fork, verified it with a new test suite handling 100k+ character base64 strings, and ensured zero regressions by running the existing 141 message utility tests. I am ready to submit the PR as soon as this is reviewed." NEW PATTERN: REPORTER-PREPARED-PR-SUPERSEDED-BY-DRIVE-BY-CONTRIBUTOR. @jiaming2li Jan 26 2026 (1 day post-filing) actually opened PR #34883 -- drive-by contributor shipped fix BEFORE reporter could submit theirs. Reporter prepared-fork was superseded. @ccurme Jan 27 2026 merged. Sub-mechanism: M-multimodal-cost-amplification token-counter-overestimates-via-base64-string-length. DUAL-PATH PROOF for M-multimodal-cost-amplification: LANG-035 (no fix, ~5-month auto-stale-close in 2024) + LANG-036 (2-day fast fix in 2026). 18-MONTH GAP between same-mechanism reports demonstrates mechanism-known-but-unaddressed-until-community-pressure pattern -- the count_tokens_approximately bug existed for entire LANG-035 base64-amplification window (Jul 2024 onward) but only got fixed when (a) reporter framed it as feature-not-bug AND (b) reporter-prepared-PR demonstrated zero-regression on 141 existing tests. Methodology-relevant: framing-and-evidence-quality drives fix outcome, not bug-severity.
Framework: langchain (2025-07)
Issue: LangChain-OpenAI raises error due to null choices when using vLLM OpenAI-compatible API
Mechanism cluster: M-multimodal-cost-amplification
Notes: paper:bu with EXTREME-CASCADING-FIX-ATTEMPTS pattern (12+ PRs over 7 months NONE merged) + CLOSED-AS-NOT-PLANNED-DESPITE-MASSIVE-COMMUNITY-FIX-EFFORT. Filed Jul 26 2025 by @escon1004 tagged bug. CLOSED Mar 2 2026 by Member @mdrxy as not planned (~7 months filing-to-close) deflecting to umbrella issue: "Putting this complexity into ChatOpenAI to handle a non-OpenAI provider is not warranted. See #34328." Reporter shows THREE-WAY COMPARISON: (1) langchain_openai.ChatOpenAI raises TypeError: "Received response with null value for choices."; (2) raw requests library works correctly returning full JSON with choices field populated; (3) official openai Python SDK works correctly. Cross-framework asymmetry: same vLLM endpoint, same model, same image, same multimodal payload -- only LangChain fails. STRONGEST controlled-comparison evidence in catalog. Source-line traced to langchain_openai/chat_models/base.py _create_chat_result line 1201. EXTREME-CASCADING-FIX-ATTEMPTS at n=12 PRs over 7 months NONE merged: @20ns Jul 26 2025 + Jul 29 2025 (3 PRs); @KaparthyReddy Oct 24 2025 PR #33662; @nightcityblade Feb 15 2026 PR #35236; @ademicho123 Feb 21 2026 PR #35388; @ddarji1409 Feb 23 2026 PR #35406; @LakshmiSravyaVedantham Feb 26 2026 PR #35444; @senguttuvan Feb 27 2026 PR #35462; @VedantMadane Feb 28 2026; @he-yufeng Mar 2 2026 PR #35498; @Faisalsaificode 2 weeks before fetch PR #36913. STRONGEST cascading-fix-attempts in catalog (12 PRs surpasses LANG-034 6 PRs). @mdrxy Sep 10 2025 added help wanted + investigate labels then closed Mar 2 2026 deflecting to umbrella issue #34328. POST-CLOSURE PR ATTEMPT: @Faisalsaificode 2 weeks before fetch filed PR #36913 -- fix attempts continue AFTER official closure. PROMOTIONAL-SPAM-PATTERN at n=5: @AkibDa Sep 26 2025 had TWO comments Hidden as spam by GitHub auto-moderation. Sub-mechanism: framework-only-failure-mode-versus-base-SDK-and-raw-HTTP-working. CLASSIFICATION CAVEAT: original triage put this in M-providerOptions-silently-dropped, but mechanism may fit better as M-adapter-parsing-failure (response-parsing not parameter-passing). Cross-references LANG-034 (reasoning_content-silent-drop, same closure-by-duplicate-reference to #34328 with overlapping PR cascade). NEW closure-pattern variant (23rd in taxonomy): closed-as-not-planned-citing-non-OpenAI-provider-justification (mdrxy explicit reasoning: non-OpenAI provider is not warranted). Distinct from closed-by-duplicate-reference because closing rationale is scope-declaration not deferral.