sajjadanwar0/token-budgets-catalog huggingface dataset:
Framework: dspy (2024-12)
Issue: Using MIPROv2 max_bootstrapped_demos > 0 with dspy.Image causes context to blow up
Mechanism cluster: M-context-amplification
Notes: paper:bu; closed-with-promise-without-PR-link pattern -- @okhat closed Jan 3 2025, reopened same day, then closed again Jan 3 with "likely will be fixed by Isaacs latest PR" (no PR number given). Filed Dec 18 2024 by @vbeutner. MAINTAINER DEFLECTION + EXPERIMENTAL-FEATURE-DISCLAIMER PATTERN: @okhat (Collaborator/founder) Dec 19 2024: "Support for dspy.Image is experimental and undocumented, so we cannot guarantee debugging help at the moment. In a month from now, I expect dspy.Image to mature and we will surely catch any typical issues by then." Title amended same day from "vlm" to "dspy.Image" to scope to specific feature. CONCRETE REPRO with full output trace: reporter posted MIPROv2 step-2 output showing the b64-encoded image string [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABqQAAAiYCAIAAAA+NVHk...] getting embedded into the proposed_instruction prompt as task_demos field. REPORTER QUOTABLE: "The optimizer will put the b64encoded image string into the prompt causing the context to blow up beyond the 200k context window available. The compilation eventually errors out due to context window exceeded." Reporter Dec 19 followup: "it works well for me otherwise and is not a blocker for me. I just wanted to put it on your radar" -- low-pressure reporter. Sub-mechanism: M-optimizer-context-amplification at MIPROv2 bootstrap-demo step (raw-base64-image-injection). Distinct from DSPY-001 (batch-summary step) -- different MIPROv2 sub-step amplifies via different mechanism. ARCHITECTURALLY DISTINCTIVE TO DSPY -- only framework with automated optimizer that injects raw training data (including image base64) into proposal prompts. Adds second n=2 within DSPy to optimizer-context-amplification cluster (with DSPY-001). Pattern parallels MAST-001 (Upstash command-count) -- both are cost mechanisms arising from framework distinctive architectural choice (memory-storage with command-counting in MAST-001; optimizer with prompt-injection in DSPY-003). BATCH-31 RETROACTIVE CROSS-CLUSTER PROMOTION: Re-reading body confirms M-multimodal-cost-amplification cluster fit. Reporter @vbeutner explicit: "When using a vlm, optimizing using MIPROv2 & setting max_bootstrap_demos > 0. The optimizer will put the b64encoded image string into the prompt causing the context to blow up beyond the 200k context window available." Dataset shows 626104-byte + 85120-byte + 212204-byte base64 image strings. Sub-mechanism: MIPROv2-bootstrap-demo-multimodal-injection at DSPy compile-time optimizer layer. Cross-cluster fit with primary M-context-amplification (compile-time) cluster. Cross-cluster pair with DSPY-005: two DSPy-specific multimodal sub-mechanisms across two distinct optimizers (MIPROv2 bootstrap-demo + GEPA reflection-minibatch). DSPy contributes n=2 to M-multimodal-cost-amplification cluster via this retroactive promotion + DSPY-005 upgrade.
Framework: dspy (2025-06)
Issue: Token Tracking Issue - Bedrock with Streamified Predict
Mechanism cluster: M-cost-observability
Notes: paper:bf; closed completed Jun 24 2025 by Collaborator @chenmoneygithub via PR #8431 -- 5-day filing-to-merge with reporter-as-fixer pattern. Filed Jun 19 2025 by Contributor @BTripp1986 against DSPy 3.0.0b1 + 2.6.27 (both versions tested). DEEP DEBUG TRAIL: reporter walked LiteLLM code in debugger -- "I can see that all usage fields are being set, but at some point in the chunk iterator it seems that they are getting stripped. The LiteLLM code is way too dense and indirect for me to follow exactly where that is happening." ACTIVE MAINTAINER COLLABORATION: @chenmoneygithub (Collaborator) Jun 19 ruled out OpenAI/Anthropic/Gemini/Databricks via own debugging ("I cannot produce this issue with OpenAI, Anthropic, Gemini or Databricks") + initially suspected upstream LiteLLM/Bedrock chunk-format issue ("I do not have an AWS account for bedrock testing"). Reporter Jun 20 HARD-CONFIRMED framework-side via direct LiteLLM acompletion test: usage data IS in final stream chunk -- proving DSPy chunk-iterator (not LiteLLM) is dropping the data. Reporter Jun 20: "I have found the root of the problem. I will submit a PR." PR #8431 merged Jun 24. REGRESSION CONFIRMATION: @lambda-science Jun 19 "Same issue over here, was working kind of fine in the past" -- suggests recent regression. Sub-mechanism: M-cost-observability streaming-chunk-iterator-drops-usage. DIRECT PARALLEL TO LANG-013 (get_openai_callback streaming returns 0). Two cross-framework rows (LangChain + DSPy) for streaming-token-tracking specifically -- streaming + framework-callback abstractions are recurring weak spot. Reporter-collaborates-then-PRs is a distinct sub-shape from solo reporter-as-fixer (LLMI-005) and PR-race competition (LANG-023).
Framework: dspy (2025-09)
Issue: Token Limit when running GEPA on multimodal problem
Notes: paper:bu; UPGRADED from SKIPPED in v22-batch-31 revision per codebook section 10. Original triage skipped as S6 (no-engagement-thin-thread + reporter-self-closed-same-day) but new cluster-fit analysis recharacterizes as M-multimodal-cost-amplification at DSPy GEPA optimizer layer warranting retention. Filed Sep 27 2025 by @wmpauli (no labels applied). CLOSED Sep 27 2025 SAME DAY by reporter as completed -- fastest filing-to-self-close in catalog. CONCRETE AMPLIFICATION ERROR: "litellm.BadRequestError: Vertex_aiException BadRequestError - The input token count (1051408) exceeds the maximum number of tokens allowed (1048576). status: INVALID_ARGUMENT" -- 1,051,408 input tokens exceeding Vertex AI Gemini 2.5 flash 1,048,576-token limit by 0.27% on iteration 2 of GEPA reflective mutation step. Reporter explicit speculation: "Is it possible that this is due to the images being sent along? In my case they may be 512**2 resolution images." 512x512 image dataset for image-tagger training task. NEW SUB-PATTERN: REPORTER-FOUND-OWN-WORKAROUND-AT-COST: Reporter Sep 27 self-comment "Lowering reflection_minibatch_size to 1 appears to help with the issue" -- workaround reduces batch size from default to 1 (trade optimizer-quality for cost-budget by reducing minibatch to single-example reflection). Reporter closes own issue same-day without engaging maintainer. Sub-mechanism within M-multimodal-cost-amplification at DSPy GEPA optimizer layer: GEPA-reflection-minibatch-multimodal-amplification -- distinct from MIPROv2 sub-mechanism in DSPY-003 (bootstrap-demo-multimodal-injection). Two DSPy-specific multimodal sub-mechanisms across two distinct optimizers. NEW closure sub-shape (26th in taxonomy): closed-by-reporter-via-config-tuning-workaround-not-framework-fix (distinct from CRAI-008 closed-by-reporter-via-notebook-isolation-workaround since this is config-parameter-tuning not isolation pattern; workaround-degrades-feature-functionality variant). Quantitative anchor distinct shape from LANG-035 31x context overflow: this shows just-over-the-limit (0.27% over) failure -- cumulative optimizer-batched context approaching upper edge of largest available context windows. ESTABLISHES DSPy AS 4TH CONFIRMED FRAMEWORK in M-multimodal-cost-amplification cluster (joins LangChain x3 + AutoGen x1 + Mastra x3). With DSPY-003 retroactive cross-cluster note: DSPy contributes n=2 to multimodal cluster. M-multimodal-cost-amplification cluster reaches n=8 across 4 frameworks confirmed (n=9 with DSPY-003 cross-cluster) with 6 distinct sub-mechanisms total. REVISION TRANSPARENCY NOTE: original S6 skip was correct on narrow reading (zero-engagement); updated retention reflects cluster-fit-coherent evidence per codebook section 10 revision policy.
Framework: langchain (2023-02)
Issue: Disable Retry on Timeout
Mechanism cluster: M-retry-loop
Notes: paper:bu; auto-stale-closed Sep 18 2023 by dosubot as not-planned. Filed Feb 16 2023 by @totalhack -- earliest cost-framed retry issue in LangChain catalog (foundation row for M-retry-loop cluster). REPORTER COST FRAMING: "The OpenAI API has been quite unreliable this week and I am getting a bunch of timeouts. I do not want langchain to retry as I am pretty sure OpenAI is still counting these requests toward billing, so I would like to set max_retries=None or 0 but that does not seem to work." FOUNDER-WORKS-FOR-ME PATTERN: @hwchase17 (Contributor, LangChain founder) Feb 19 2023: "setting max_retries=0 seems to work for me - what issue are you seeing?" -- minimal investigation, no follow-up to debugging. Reporter Feb 19: "That did not work for me. I will have to test again though." Multiple-users-confirm-broken: @asmithar Feb 22 2023 reproduced with concrete code: "LLMChain(llm=OpenAI(temperature=0, model_name=text-davinci-003, max_tokens=-1, max_retries=0), prompt=prompt)." Auto-stale-marked Sep 9 2023 by dosubot. Pattern: founder-confirms-works-for-me + multiple-users-confirm-doesnt-work + no-resolution + auto-stale-close. Sub-mechanism: max_retries-zero-not-respected (different from LANG-026 fake-parameter and LANG-027 commented-decorator -- here the parameter IS wired through, but underlying urllib3/httpx retry layer bypasses it). dosubot Nov 24 2023 cross-referenced #13816 (URL Lib error). FIRST ROW IN M-RETRY-LOOP CLUSTER. Establishes pattern as existing from project earliest days (Feb 2023, ~2 weeks after public launch). NEW closure sub-shape: founder-works-for-me (distinct from auto-stale-without-engagement, by-design-declined, deflection-to-upstream, experimental-disclaimer, closed-by-promise-without-delivery, stuck-open-with-multiple-PRs).
Framework: langchain (2023-10)
Issue: Retry limit is not respected
Mechanism cluster: M-retry-loop
Reported user dollar loss: 120
Notes: paper:bu with populated user_dollar_loss field; auto-stale-closed Feb 7 2024 as not-planned. CRITICAL CATALOG MILESTONE: FIRST ROW with concrete user_dollar_loss figure populated by reporter -- $120 in 10 minutes. Filed Oct 7 2023 by @yutongp on LangChain 0.0305. REPORTER QUOTABLE COST FRAMING: "langchain kept retrying on this error definetly more than the default 6 times, I had 14 retry log from langchain in my terminal. I think this is a quite critical bug, my OpenAI usage limit blew up the $120 limit under 10mins because of this. retry limit should kick it to short circuit this situation or there should be some other method to prevent unexpected length of document being passed to openAI." Reporter loaded a 13524539-character PDF document via WebBaseLoader; OpenAI returned rate_limit_exceeded errors which LangChain retry logic ignored, billing 14 attempts. Tagged area:models + bug. dosubot Oct 7 provided LLM response focused on chunking-the-document (preventing the trigger), NOT addressing the retry-limit-not-respected mechanism itself -- bot misdiagnosis pattern. dosubot response noted "The default retry limit in LangChain is set to 3" but reporter observed 14, suggesting the actual default-and-respect-of-limit are both broken. ZERO HUMAN MAINTAINER ENGAGEMENT throughout 4-month thread despite explicit dollar damage and Harrison Chase + Ankush Gola tagged. dosubot mentioned in #11657 (chat model answering incorrectly) Oct 11 2023. Auto-stale-marked Jan 31 2024. Sub-mechanism: rate-limit-induced-retry-loop with each-retry-billed (rate_limit_exceeded triggers retry; retry hits same rate limit; loop continues until external billing limit; each attempt counts toward OpenAI tokens-per-minute quota, amplifying cost without progress). Distinct from LANG-024 (max_retries=0-not-respected): here the issue is retries-exceed-stated-default. EVIDENCE-QUALITY MILESTONE: First row in catalog where reporter explicitly quantifies dollar damage. Pattern: critical-cost-bug-with-concrete-dollar-loss-ignored + auto-stale-close. New sub-shape: dosubot-misdiagnoses-by-treating-trigger-not-mechanism.
Framework: langchain (2025-02)
Issue: ChatFireworks does not retry on API errors (max_retries parameter is fake)
Mechanism cluster: M-retry-loop
Notes: paper:bu; auto-stale-closed May 26 2025 as not-planned without ChatFireworks fix. Filed Feb 17 2025 by @aimestereo. SOURCE-LINE-LEVEL TRACE through wrapping layers: ChatFireworks accepts max_retries parameter (per documented example in body) -> passes to Fireworks(BaseFireworks) and AsyncFireworks(BaseFireworks) client wrappers -> client wrappers DO NOT pass it to actual FireworksClient/ImageInference(FireworksClient) clients -> FireworksClient has ZERO retry logic at all. REPORTER QUOTABLE: "Class ChatFireworks has this parameter, but it is not used... FireworksClient do not have/receive this parameter, but it also does not have any retry specific logic." MULTIPLE ERROR TYPES NOT RETRIED in body stack traces: httpx.ReadError, httpx.ConnectError, BadGatewayError, RateLimitError -- all surface to caller without any retry attempt. Reporter Feb 17 same-day proposed routing-around-the-bug rather than fixing it: "langchain fireworks client can be marked as deprecated? (considering lack functionality) and client for OpenAI can be used instead." Reporter migrated to ChatOpenAI with Fireworks OpenAI-compatible base_url -- "works like a charm." @keenborder786 Feb 18 mentioned PR #29856 "OpenAI client for ChatFireworks" -- response was to add OpenAI-client compatibility, not to fix ChatFireworks retry logic. dosubot tagged bug. @langcarl auto-tagged investigate. Auto-stale-marked May 19 2025. Pattern: bug-filed -> reporter-finds-own-workaround -> deprecation-suggestion -> no-fix-to-original -> auto-stale-close. Sub-mechanism: max_retries-fake-parameter (accepted-by-API-but-never-wired-through-wrapper-layers). Distinct from LANG-024, LANG-025, LANG-027. PROVIDER-ASYMMETRIC PATTERN: ChatFireworks documented behavior diverges from ChatMistralAI (LANG-027), ChatAnthropic, ChatOpenAI -- some adapters wire retry, some do not, all expose the same max_retries API surface. Cross-row evidence with LANG-027 establishes provider-adapter-implements-retry-config-asymmetrically pattern.
Framework: langchain (2025-03)
Issue: max_retries Parameter in ChatMistralAI Class Is Ineffective Due to Commented Code
Mechanism cluster: M-retry-loop
Notes: paper:bf with strongest fix-shipped-immediately-regressed evidence in catalog at source-line + commit-reference level. Filed Mar 19 2025 by @amarion35; closed Mar 27 2025 by Collaborator @ccurme via PR #30448 (commit 63673b7) -- 8-day filing-to-merge with reporter+contributor handoff. SOURCE-LINE-LEVEL EVIDENCE: langchain/libs/partners/mistralai/langchain_mistralai/chat_models.py:460 shows "# retry_decorator = _create_retry_decorator(self, run_manager=run_manager)" -- commented-out retry decorator. REGRESSION PROVENANCE EXACT: reporter identified "The lines were commented a year ago in this commit 53ac1eb in this PR #19420." YEAR-LONG SHIPPED REGRESSION: ~12 months between Mar 2024 (PR #19420 commenting out the decorator) and Mar 2025 (issue filed) where all ChatMistralAI users had broken retries shipped without anyone noticing. CONCRETE REPRO: "mistral = ChatMistralAI(timeout=1, max_retries=10)... mistral.invoke(test_input)... Timeout after 1.0581834316253662 seconds" -- proves only 1 attempt happens despite max_retries=10 + timeout=1s configuration. CONTRIBUTOR-FIX-PATTERN: @andrasfe (Contributor) Mar 23 opened PR #30448 quotable: "as per above, I un-commented code, validated that the retries work as expected, added unit test that validates the ChatMistralAI retry functionality, and added an integration test for the retries. I do not know why this was commented out a year ago - I could not find a reason for it. Will leave it to the maintainer but I do not see immediate harm in re-enabling it." Tested in unit + integration. @langcarl auto-tagged investigate. dosubot tagged bug. Sub-mechanism: retry-decorator-commented-out (regression at exact source-line level, ~12-month shipped persistence). STRONGEST FIX-SHIPPED-IMMEDIATELY-REGRESSED EVIDENCE IN CATALOG: better than MAST-009/012 (Mastra) and LANG-021 (LangGraph v1.0) because (a) exact line + exact commit + exact PR reference, (b) year-long persistence demonstrates regression went unnoticed by maintainers despite affecting all ChatMistralAI users, (c) contributor explicitly could not find the original reason for commenting -- speaks to maintainer-knowledge-loss between PR #19420 author and current team. New sub-mechanism within fix-shipped-regressed: maintainer-knowledge-loss-across-team-transitions enables silent-major-feature-regression in patch-version-merge.
Framework: mastra (2025-07)
Issue: Does new workflows have same retry logic as legacy workflows?
Mechanism cluster: M-retry-loop
Notes: paper:bf; closed Jul 9 2025 by @taofeeq-deru via PR #5749 -- 2-day filing-to-merge. Filed Jul 7 2025 by Member @abhiaiyer91 (Mastra Member/founder). MAINTAINER-FILES-OWN-FRAMEWORK-BUG PATTERN: this is the SECOND case where Mastra Member @abhiaiyer91 himself files an issue about Mastra own implementation gap (first was MAST-002 Token / Cost Usage Tracking?). Reporter inspected code FIRST then filed: "Browsing through the code, it does seem like the retries property is still respected. However, I cannot find any usage of the delay. Is this intentional?" -- maintainer-as-debugger. Discord-converted issue tagged Workflows + status:needs-triage, auto-assigned to @rase- via github-actions. @taofeeq-deru (Contributor) Jul 8 same-day: "thanks for reporting We have fixed it on this #5749, it will be released today." @taofeeq-deru Jul 9: "this is now fixed on @mastra/core@0.10.11." Sub-mechanism: retry-config-delay-not-honored-in-workflow-refactor (legacy workflows had retryConfig with attempts+delay; new workflows kept retries but lost delay parameter during refactor). Parallels MAST-012 (system prompt caching regression in refactor). MAINTAINER-FILES-OWN PATTERN AT n=5 across 4 frameworks: ATGN-006 by @ekzhu, ATGN-009 by @fniedtner (Microsoft AutoGen), GPTE-003 by @AntonOsika (gpt-engineer), MAST-002 by @abhiaiyer91 (Mastra cost-tracking), MAST-016 by @abhiaiyer91 (Mastra retry-config). Two-from-same-author shows the founder is actively-monitoring-own-framework-for-regressions -- distinct evidence from external-user-pattern. M-RETRY-LOOP CLUSTER: adds Mastra at n=1 (MAST-016) to LangChain n=4 (LANG-024/025/026/027). Cluster reaches n=5 across 2 frameworks with 5 distinct sub-mechanisms (max_retries-zero-not-respected, rate-limit-induced-loop-with-billing, max_retries-fake-parameter, retry-decorator-commented-out, retry-config-delay-lost-in-refactor). Cost-framing here is implicit (delay-loss = back-to-back retries hammering provider, but no concrete dollar figure) -- weaker than LANG-025 ($120) but mechanism-coherent.
Framework: smolagents (2025-02)
Issue: Getting Token Usage as part of the Final Answer
Mechanism cluster: M-cost-observability
Notes: paper:fr; closed Mar 19 2025 completed by Member @albertvillanova: "You can have that information from the agent.monitor." Filed Feb 8 2025 by @mattmarcin tagged enhancement. CRITICAL TWO-PART REQUEST: reporter asked for "the used input and output token count or even actual API usage expense" -- part 1 (token counts) addressed by existing Monitor class which closes as completed; part 2 (dollar cost / actual API usage expense) silently unaddressed. Reporter explicit cost framing: "Being able to get usage/cost data from API calls would help from a business perspective to track cost of execution and aid in iterating on agents to lower costs." @jonha892 Feb 26 2025 ADDITIONAL GAP DOCUMENTATION: "Providers like openai, gemini, and anthropic use caching to automatically optimize costs. This is not tracked in this library." -- second Monitor gap (cached token tracking) noted but not in closure scope. Sub-mechanism: token-vs-dollar-asymmetry within M-cost-observability cluster. DIRECT PARALLEL to MAST-002 (Mastra), ATGN-006 + ATGN-009 + ATGN-010 (Microsoft AutoGen), GPTE-003 (gpt-engineer), DSPY-002 (DSPy). STRENGTHENS cross-framework cluster to n=6 across 6 frameworks. FIRST SMOLAGENTS ENTRY in catalog. Closed-as-resolved-by-pointing-to-partial-feature pattern: borderline S5 retained on cluster-fit consistency (parallels MAST-002 retain decision on same closure shape).
Framework: smolagents (2025-03)
Issue: Agent memory/history consolidation after a number of interactions
Mechanism cluster: M-context-amplification
Notes: paper:fr; OPEN at fetch (13+ months) despite multiple PRs in flight and active community engagement. Filed Mar 6 2025 by @nikos-kal tagged enhancement. EXPLICIT COST FRAMING: "The management of memory, thus context size, is important because it translates limitations in terms of cost, speed, model choice etc. Other agentic LLM applications like Claude Code deal with this by summarizing the context every few steps or remembering only a limited history." Reporter cited Claude Code architecture as reference. Reporter Mar 10 self-discovered PARTIAL WORKAROUND via step_callbacks (existing feature can do post-step cleanup but not first-class memory_size config). MULTIPLE COMMUNITY PROPOSALS: @Symbolk (Contributor) Mar 17 2025 suggested mem0 integration; @nikos-kal Mar 18 countered: "We still need an easy memory_size configuration in smolagents though because even if you switch to mem0, smolagents would still internally keep their own memory." @ChristianWeyer Mar 26 2025: "So, pluggable memory is on the way @nikos-kal?" -- waiting on resolution. FORK-BASED IMPLEMENTATION PATTERN: @joaopauloschuler (Contributor) Mar 11 2026 implemented in own fork: "In my own fork, I implemented it" -- fork-based-implementation-not-merged-upstream signal-class (NEW closure sub-shape, distinct from LANG-022 stuck-open-with-multiple-PRs because here contributor does not attempt upstream merge). RECENT PRs: @agent-morrow Mar 29 2026 cross-referenced #2129; @saslifat-gif Apr 3 2026 PR #2153 "feat: add max_context_chars parameter for automatic memory truncation." Sub-mechanism: M-context-amplification at agent-step-loop runtime level. DISTINCT FROM DSPY-001/DSPY-003 (optimizer-compile-time amplification at MIPROv2 batch-summary and bootstrap-demo steps respectively) -- here it is runtime-iteration-amplification at MultiStepAgent.run() level. ESTABLISHES M-CONTEXT-AMPLIFICATION AT TWO ARCHITECTURAL LEVELS: compile-time (DSPy optimizers) AND runtime (smolagents agent steps). Cluster expands from dspy-architecture-specific (n=2) to architecture-class with 2 sub-classes (n=3 across 2 frameworks).
Framework: smolagents (2026-01)
Issue: ENH: Handling max output token limit
Notes: SKIPPED for paper after URL verification (S6 no-engagement-thin-thread). Filed Jan 1 2026 by @OscarGarciaF tagged enhancement OPEN at fetch. Brief 4-line body asking architectural question: "Does smolagents keep all the history from previous iterations as-is (contributing to ever increasing token usage? Is there an standard approach on how to handle runs that are approaching a the token limit?" ZERO comments from anyone -- solo reporter, no community responses, no maintainer engagement at fetch. Mechanism IS framework-side and parallels SMAG-002 (M-context-amplification at agent-step-loop level), but evidence-quality bar requires more than thin solo report. Consistent with DSPY-005 skip (also S6, also same architectural pattern as a stronger row in same batch). SMAG-002 already covers M-context-amplification at smolagents runtime level with extensive engagement and PRs in flight -- adding SMAG-003 would inflate cluster size with marginal evidence. Could revisit if SMAG-003 gains engagement post-fetch.
Framework: smolagents (2025-04)
Issue: Is there a step timeout feature?
Mechanism cluster: M-retry-loop
Notes: paper:fr; OPEN at fetch tagged bug. Filed Apr 2 2025 by @forest-snow with concrete benchmark context: Haiku 3.5 (code agent) on smolagents benchmark MATH subset using LiteLLM and Bedrock, "often take much time because the model may write a brute-force code snippet and it takes a while to execute a step (>500 seconds)." REPORTER EXPLICIT GAP-IN-EXISTING-CONTROLS: "While there is a MAX OPERATIONS for the local python executor, it did not seem to help fix this issue." -- MAX_OPERATIONS counts AST operations not wall-clock time, leaving wall-clock-runaway uncontrolled. ACTIVE MAINTAINER ENGAGEMENT: @aymeric-roucher (Collaborator) Apr 4 2025 asked for minimal repro isolated to python executor: "Could you provide a code snippet that involves, not the full agent and LLM, but only the python executor with the faulty code?" COMMUNITY CONFIRMATION 4 months later: @heckfy88 Aug 22 2025 confirmed mechanism with concrete repro suggestion: "it is the default LocalPythonExecutor, it does not have a timeout arg, so every tool call is limited by, I think, 300s, after that is timeout exception. To reproduce it just call a tool that has time.sleep(1000)." @coderabbitai 3 weeks before fetch cross-referenced langflow PR #12713. Sub-mechanism: per-step-wall-clock-runaway during code execution. CLASSIFICATION NOTE: candidate cluster fit is contested. Original triage assigned to M-retry-loop as wall-clock variant. Alternative classification: this fits with MAST-014 (OM observer-LLM blocking 148s, ephemeral-token-counting) as wall-clock-amplification-inside-agent-loop -- distinct mechanism class from HTTP-retry-on-error (LANG-024/025/026/027 + MAST-007/016 are all HTTP-retry mechanisms; SMAG-004 is Python-executor wall-clock). Candidate new cluster M-execution-time-runaway with MAST-014 (n=2 across 2 frameworks, below codebook section 10 n=3 threshold to elevate to own M-category in v1.1). For v23 retention: classify as wall-clock-amplification-candidate, do not absorb into M-retry-loop. [v23 IRR adjudication: retagged bu->fr per second-rater agreement; original tag bu by the first author, codebook v1.1 will tighten bu/fr boundary on interrogative-feature-gap titles.]
Framework: smolagents (2025-03)
Issue: List of Free Local LLM Models - Up and Running
Mechanism cluster: M-retry-loop
Notes: paper:bu; closed Mar 18 2025 by Member @albertvillanova as completed -- BUT closed via docs-PR (huggingface/agents-course#327) NOT framework code fix. Filed Mar 13 2025 by @psymbio tagged enhancement. CONCRETE FRAMEWORK COST-IMPACT: body quotes "You have exceeded your monthly included credits for Inference Providers. Subscribe to PRO to get 20x more monthly included credits. [Step 17: Duration 0.30 seconds | Input tokens: 53,107 | Output tokens: 4,313]" -- agent ran 17 steps consuming ~57K tokens before HF Pro credits exhausted, no whole-run budget primitive caps spend. CLOSED-BY-DOCS-PR PATTERN: @HKanoje Mar 14 2025 provided LiteLLMModel+Ollama workaround code; @aymeric-roucher (Collaborator) Mar 15 endorsed: "The solution described by @HKanoje is good! If you think it is worth it, please open a PR"; @HKanoje Mar 16 opened PR #327 against agents-course; @albertvillanova Mar 18 closed citing the docs PR (NOT a framework code change). POST-CLOSURE MULTI-USER PUSHBACK demonstrates workaround insufficiency: @solarslurpi Apr 21 2025: "Why are you saying it works with Ollama? Have you tried all the exercises with it? The more I tried, it did not work. My hypothesis is models for smolagents need to be >= 20B parameters... I also disagree that Ollama just works." @al-swaiti Jun 9 2025 (3 months after closure): same 402 Payment Required error against Qwen/Qwen2.5-Coder-32B-Instruct from HF Inference API -- "IS there free model to test spaces!!!" with full traceback. Sub-mechanism: missing-whole-run-budget-primitive (framework has max_steps but no max_tokens or max_cost cap that would prevent credit-exhaustion across long-running agent runs). NEW SUB-MECHANISM CATEGORY: M-budget-primitive-missing. Distinct from M-context-amplification (per-step memory growth) and M-retry-loop (retry-on-error) -- this is the structural absence of cost-aware stopping criteria for normal agent loop execution. CRUCIAL FOR v23 SECTION 3 framing: this is the cluster that Budget type FULLY solves (vs partial mitigation for M-context-amplification and M-retry-loop). Candidate to test cross-framework reach: AGPT-001/AGPT-002, NANO-001 are feature-requests-for-budget-primitives (adjacent but those are requests not cost-incidents-from-absence). NEW closure sub-shape: closed-by-docs-PR-without-framework-fix (10th sub-shape, distinct from workaround-only-no-fix LANG-016/LANG-022).
Framework: autogen (2023-12)
Issue: Cost calculation for the whole chat session
Mechanism cluster: M-context-amplification
Notes: paper:fr; closed Jan 17 2024 via PR #1269 ("Add usage summary for agents") by Contributor @sonichi -- closure addresses observability portion BUT architectural cost amplification explicitly admitted unsolvable without rewriting GroupChat. RENUMBERED FROM ORIGINAL PASTE (was "ATGN-010" in user paste; ID collision with existing ATGN-010 #4835 prompted renumber to next free ATGN-018). Filed Dec 27 2023 by @stephanedebove tagged enhancement. POWERFUL MAINTAINER-CONFIRMED ARCHITECTURAL COST AMPLIFICATION: Member @afourney Dec 28 2023 quotable: "Group chat is tricky because many messages are duplicated (and thus only need to be counted once), and because there is a hidden OAI call for orchestration that does not show up as a message. At present the entire history is passed to all agents, and is also passed to OAI when doing orchestration. The costs are super-linear -- long chats are much much more costly than short chats. You can mitigate this by using the round_robin... but then the orchestration is basically a simple loop with no llm calls at all. There is no way to mitigate this short of rewriting GroupChat." THIS SINGLE QUOTE DOCUMENTS THREE DISTINCT MECHANISMS: (1) hidden OAI orchestration call cost-not-surfaced; (2) entire history broadcast to all agents = M-context-amplification at delegation layer; (3) super-linear-cost-vs-length-of-chat. Reporter explicit user-impact framing: "is no way to know the cost for a whole CustomGroupChat() session?" + "ways to mitigate the OpenAI costs during a group chat session?" + architectural question "do agents receive the whole history of conversation when they are called?" PR #1269 addressed observability (token tracking summary) but NOT the architectural amplification. Pattern: closed-as-completed-but-architectural-issue-remains. NEW closure sub-shape (11th in taxonomy): closed-by-fixing-adjacent-issue-while-architectural-issue-remains-acknowledged-unsolvable. FOUNDATION ROW FOR M-DELEGATION-FANOUT cluster: maintainer-confirmed architectural cost amplification at GroupChat layer. Tag note: bu chosen over fr because the substantive content is the maintainer admission of unfixed mechanism, not the closed-completed observability portion. Filing-to-close: 21 days for observability fix; architectural cost-amp explicitly unsolved at fetch. [v23 IRR adjudication: retagged bu->fr per second-rater agreement; original tag bu by the first author, codebook v1.1 will tighten bu/fr boundary on interrogative-feature-gap titles.]
Framework: autogen (2024-01)
Issue: GroupChat keeps calling all the available agent and does not stop even after output from one agent is generated
Mechanism cluster: M-delegation-fanout
Notes: paper:bu; AUTO-STALE-CLOSED at 9 months by Collaborator @rysweet Oct 19 2024: "closing as stale - missing info from OP. please reopen I you want to update." RENUMBERED FROM ORIGINAL PASTE (was "ATGN-011"; ID collision with existing ATGN-011 #5620 prompted renumber to next free ATGN-019). Filed Jan 30 2024 by @saurabhgssingh tagged 0.2 + group chat/teams. CONCRETE FRAMEWORK MECHANISM: Reporter shows GroupChat with 4 agents (user_proxy, coding_assistant, simulator_worker_agent, text_2_sql_agent) and max_round=24. Even when question routes cleanly to one agent (e.g. text-to-sql query), GroupChatManager invokes ALL configured agents through cycles -- up to ~24 * 4 = 96 unnecessary LLM calls per session. CONTRIBUTOR ENGAGEMENT BEFORE STALE-CLOSE: @ekzhu (Contributor) Jan 30 2024 same day requested llm_config + pointed at architectural workaround: "One thing we are working on is graph-based group chat, which allows user to add constraints to the speaker transition: PR #857". Reporter Jan 31 provided full Azure config. Contributor @sonichi Feb 12 2024 asked: "How are the text2sql and rag agent defined?" -- never followed up. Then 8.5 MONTHS SILENCE before stale-close. Sub-mechanism: M-delegation-fanout fanout-no-stop -- GroupChatManager has no first-class routing constraint to skip irrelevant agents. Linked architectural fix PR #857 (graph-based speaker transition) -- speaker-transition-constraints not first-class in 0.2 codebase. NEW closure sub-shape (12th in taxonomy): early-engagement-then-silence-then-stale-close (distinct from auto-stale-without-engagement LANG-008/017/019/020 where there was no human engagement at all).
Framework: mastra (2025-12)
Issue: Propagate AbortSignal through Agent Networks to sub-agents
Mechanism cluster: M-delegation-fanout
Notes: paper:bf with NEW FIX-PATTERN-SHAPE: fix-attempted-then-abandoned-then-re-engaged-and-shipped. Filed Dec 4 2025 by @bergr7 tagged enhancement. CLOSED Mar 3 2026 via PR #13491 by @graysonhicks (Member) -- 3 months filing-to-merge. Direct M-delegation-fanout cancellation-not-propagated mechanism: "When using agent.network() for multi-agent collaboration, the abortSignal option is not available and abort signals do not propagate to sub-agents. If you abort the supervisor agent, the subagents continue executing to completion and the results from sub-agents still land in memory, even though the parent stream was aborted." SUB-AGENTS BURN TOKENS AFTER PARENT CANCELLED -- direct cost-amplification on cancellation. FIX-ATTEMPTED-AND-ABANDONED FIRST: @abhiaiyer91 (Member) made commit de07375 SAME DAY as filing (Dec 4 2025): "feat: propagate abortSignal through agent networks to sub-agents" + opened PR #10904 same day. PR #10904 WAS CLOSED UNMERGED (decision reversed silently). 2.5 MONTHS SILENCE then external prompt from Contributor @raulpineda Feb 13 2026 with explicit cost-framing: "@abhiaiyer91 did you decide against this when closing #10904? Not being able to cancel network generations feels like a potentially expensive limitation." 12 days later @graysonhicks (Member) self-assigned Feb 25 2026, made commit 9966321 same day + opened PR #13491. Merged Mar 3 2026. Additional follow-up commit by @wardpeet Mar 9 2026. NEW FIX-PATTERN-SHAPE distinct from existing taxonomy: fix-attempted-then-abandoned-then-re-engaged-and-shipped. Three sub-shapes now in fix-shipped-immediately-regressed/closure typology: (a) fix-shipped-immediately-regressed (LANG-027 ~12 month silent regression, MAST-009/012), (b) fix-shipped-not-released (LANG-021 11+ weeks user-facing gap), (c) NEW: fix-attempted-then-abandoned-then-re-engaged-and-shipped (MAST-017). Sub-mechanism within M-delegation-fanout: cancellation-not-propagated -- distinct from ATGN-018 (super-linear-history-broadcast), ATGN-019 (fanout-no-stop), ATGN-004 (official-example-context-overflow). Adds Mastra at n=2 to M-delegation-fanout cluster (with MAST-014 OM observer-LLM blocking, also delegation-related). Maintainer-files-own pattern instance n=6: abhiaiyer91 attempted day-one fix; PR closed unmerged means failure-not-success of pattern initially; graysonhicks Member shipped final fix. Honest reframing: maintainer-files-own at n=6 across 5 people, 4 frameworks, with caveat that maintainer-files-own does NOT guarantee merge -- pattern is "maintainer notices and tries to fix" not "maintainer notices and successfully fixes." Comment-spam-noise observation: @chorghemaruti64-creator Feb 22 2026 left lengthy semi-promotional comment about external Agenium DNS service -- documents promotional-spam pattern in long-running issues, methodological texture not paper-quotable directly.
Framework: crewai (2024-07)
Issue: Token Usage Metrics for Each Task
Mechanism cluster: M-cost-observability
Notes: paper:fr; AUTO-STALE-CLOSED Dec 17 2024 as not planned by github-actions stale-bot (5 months filing-to-close). RENUMBERED FROM ORIGINAL PASTE (was "CRAI-001" but ID collision with existing CRAI-001 #6; renumbered to next free CRAI-006). Filed Jul 14 2024 by @lirockyzhang tagged enhancement + no-issue-activity. DIRECT M-COST-OBSERVABILITY GRANULARITY-GAP: Reporter quotable: "Is it possible to get a usage metrics for each task after execution instead of a total usage metrics after the whole crew finished running? It would be easier to target tasks that use excessive tokens and reduce their token usage." Per-task vs per-crew cost-aggregation gap. MULTI-USER CONFIRMATION: @ChaseRichardsonGit Oct 23 2024: "+1 - Would love this feature as well. What I am getting out of crew usage metrics and Langtrace are not adding up and being able to solve it out via individual task metrics would be ideal." -- explicit cross-tool-discrepancy framing (Langtrace numbers do not match crew metrics, suggesting framework-level cost-tracking ambiguity beyond just granularity). POST-CLOSURE USER CONFIRMS UNRESOLVED: @Hisan-007 Jan 2 2025: "i am still looking for a solution, did anyone find anything, or a work around perhaps?" Sub-mechanism: per-task-vs-per-crew-cost-aggregation-gap. Parallels MAST-002 (Mastra cost-aggregation gap), ATGN-018 (AutoGen GroupChat costs hidden orchestration call), SMAG-001 (smolagents Monitor only at agent level). FIRST CrewAI cost-observability ENTRY in catalog -- previously CrewAI rows were retry-loop and by-design clusters only. M-cost-observability cluster reaches large cross-framework reach.
Framework: crewai (2024-11)
Issue: crew result token usage always 0
Mechanism cluster: M-cost-observability
Notes: paper:bu; AUTO-STALE-CLOSED Jan 9 2025 as not planned by github-actions stale-bot (8 weeks filing-to-close). RENUMBERED FROM ORIGINAL PASTE (was "CRAI-002" but ID collision; renumbered to CRAI-007). Filed Nov 14 2024 by @gergirod tagged bug + no-issue-activity. CONCRETE EVIDENCE: Reporter shows full Crew with 2 agents (researcher + info-gatherer) on gpt-4o-mini + crewAI 0.80.0 producing total_tokens=0 prompt_tokens=0 cached_prompt_tokens=0 completion_tokens=0 successful_requests=0 despite output being correct. STRONG MAINTAINER ENGAGEMENT BEFORE STALE-CLOSE: @thiagomoretto (Contributor) self-assigned Nov 26 2024. @thiagomoretto Nov 28 cannot reproduce with similar code via uv run crewai run -- gets correct total_tokens=12548 prompt_tokens=10657 cached_prompt_tokens=2816 completion_tokens=1891 successful_requests=6. Investigation continued through Dec 4 2024. SOURCE-LINE-TRACED ROOT-CAUSE-HYPOTHESIS: @thiagomoretto Dec 4 2024 cited specific code path src/crewai/crew.py L566-L571 with quote: "It recreates the object, so if you have an old reference, you might end up with zero usage metrics... I believe there is a small improvement in the lib like instead of replacing the object entirely, just resetting the metrics to ensure the values are zeroed before summing all, but I am not sure this has other side effects." Reporter Dec 4 confirms still zero with suggested pattern (kickoff-first-then-access) AND latest version 0.83.0 -- maintainer hypothesis disconfirmed by reporter testing. Maintainer asks if model-specific issue with LiteLLM callbacks. 30 DAYS SILENCE then auto-stale-bot closes despite mid-investigation status. POST-CLOSURE 6-MONTH-LATER CONFIRMATION: @mohsin2596 Jul 10 2025: "where did you end up with this issue? I am facing the same issue currently." Sub-mechanism: cost-tracking-returns-silent-zeros (silent-failure-not-error). Parallels DSPY-002 (Langfuse zero values), ATGN-008 et al. -- cross-framework instance of silent-zero-cost-counts pattern. NEW CLOSURE SUB-SHAPE (13th in taxonomy): stale-closed-during-active-investigation -- auto-bot does not distinguish active troubleshooting from inactivity. Distinct from typical auto-stale-close where issue genuinely went silent. NEW evidence sub-shape: maintainer-hypothesis-tested-and-disconfirmed-by-reporter -- mechanism diagnosis was wrong, validated by reporter testing.
Framework: crewai (2025-05)
Issue: Manager agent repeatedly performs tasks
Mechanism cluster: M-delegation-fanout
Notes: paper:bu; CLOSED BY REPORTER Jun 4 2025 as completed VIA WORKAROUND-NOT-FRAMEWORK-FIX. RENUMBERED FROM ORIGINAL PASTE (was "CRAI-003" but ID collision; renumbered to CRAI-008). Filed May 15 2025 by @joshylhs tagged bug. Reporter quotable: "My manager agent will take over and perform all the tasks themselves, if not, almost all tasks." Hierarchical Process + gpt-4o + temperature=0.2 + manager backstory explicitly forbidding self-execution. Tested with both allow_delegation=True and False. N-AGENT-SCALING-FAILURE-MODE: Reporter context: "was not happening very much before when I had 2-3 agents. Currently, anything more than that will trigger the problem." Reporter setup: 6 agents (manager + classification + extraction + query + response + diagnosis + guardrail). CONCRETE FRAMEWORK ERROR LOG: Delegate work to coworker tool fails with pydantic validation: dict-passed-instead-of-string in tool args. MULTI-USER CONFIRMATION: @Saadkhh May 22 2025: "I am also facing this issue." CROSS-REFERENCE TO PRE-EXISTING INSTANCE: @joshylhs May 22 referenced #1823 (older instance of same bug). COLLABORATOR-PROMISE-WITHOUT-FOLLOWUP: @lorenzejay (Collaborator) May 23 2025: "I will take a look at this!" -- never followed up. REPORTER-SELF-RESOLUTION VIA WORKAROUND: @joshylhs May 29 2025: notebook-cell-isolation workaround NOT a framework fix. NEW SUB-MECHANISM within M-delegation-fanout: manager-bypass-pattern. PAIRS WITH CRAI-010 (#4783) for 10-MONTH-SPANNING evidence May 2025 + Mar 2026 + earlier #1823 reference. NEW CLOSURE PATTERN (14th in taxonomy): closed-by-reporter-via-workaround-not-framework-fix -- distinct from closed-by-docs-PR (SMAG-005) because workaround is notebook-environment artifact rather than provider-switch.
Framework: crewai (2025-06)
Issue: Crew Agent Experiencing Issues with Basic Task Execution
Mechanism cluster: M-delegation-fanout
Notes: paper:bu; AUTO-STALE-CLOSED Jul 22 2025 as not planned by github-actions stale-bot (7 weeks filing-to-close). RENUMBERED FROM ORIGINAL PASTE (was "CRAI-004" but ID collision; renumbered to CRAI-009). Filed Jun 3 2025 by @KGoutha9 tagged bug + no-issue-activity. DIRECT FANOUT-LOOP WITH CONCRETE TIMING: Reporter explicit: "agent is invoking all the un necessary tools and going on in a loop for nearly 10 min." Setup: 1 employee-info-agent + 4 tools + hierarchical Process + max_rpm=100 + manager_llm gpt-4o on Azure. CONCRETE LOG EVIDENCE: First two tasks succeed; third task collapses into Crew Manager calling Ask question to coworker 4+ times in succession asking same question despite already knowing the answer. Tool counter shows: Used Ask question to coworker (3) + Used Ask question to coworker (4) -- explicit framework counter showing repeated identical calls. SOURCE-LINE-TRACED FRAMEWORK LIMITATION: @Vidit-Ostwal (Contributor) Jun 3 2025 traced root cause to crewAI/src/crewai/llm.py L810-832 _handle_tool_call function with quote: "I can see this particular line of code: tool_call = tool_calls[0]" -- only first tool call processed regardless of how many returned by LLM. MULTI-AGENT-SCALING-FAILURE CONFIRMED: Reporter tested with 2 tools -- works as expected. With 4+ tools -- breaks. Reporter scaling argument: "in practical scenarios, we might need at least 10 tools for each agent." @Vidit-Ostwal Jun 3 confirms scalability limitation: "Agreed, workflows wont be scalable unless we have a good amount of tools with every agent." -- maintainer admits framework limitation. CROSS-FRAMEWORK COMPARISON: Reporter Jun 3 explicit: "when working with react agent in langgraph this is working as expected" -- LangGraph React agent does not show this bug, narrowing root cause to crewAI specifically. Sub-mechanism within M-delegation-fanout: fanout-loop-without-progress + framework-tool-call-extraction-limitation. PAIRS with ATGN-019 (GroupChat keeps calling all agents) + SMAG-004 (per-step wall-clock-runaway) -- same fanout-no-stop pattern in third framework. M-delegation-fanout cluster cross-framework reach now includes CrewAI (3rd ecosystem).
Framework: crewai (2026-03)
Issue: Hierarchical process delegation fails - manager agents cannot delegate to worker agents
Mechanism cluster: M-delegation-fanout
Notes: paper:bu; AUTO-STALE-CLOSED ~mid-April 2026 as not planned by github-actions stale-bot DESPITE TWO PRS IN FLIGHT -- MOST AGGRESSIVE AUTO-STALE-CLOSE INSTANCE IN CATALOG. RENUMBERED FROM ORIGINAL PASTE (was "CRAI-005" but ID collision; renumbered to CRAI-010). Filed Mar 9 2026 by @mojinfu tagged bug + no-issue-activity. Reporter quotable: "In CrewAI hierarchical process, manager agents are unable to delegate tasks to worker agents, even when allow_delegation=True is set. This breaks the core functionality of hierarchical task coordination... Manager agents only execute tasks using their own tools and never delegate to other agents, effectively making the hierarchical process behave like a sequential process." REPORTER PROVIDES PROPOSED FIX with concrete code: _update_manager_tools should inject delegation tools to other agents (excluding self). TWO PRS IN FLIGHT AT CLOSE-TIME: (1) @devin-ai-integration (AI BOT) Mar 9 2026 commit 9ba1ec0 + PR #4784; (2) @NIK-TIGER-BILL Mar 24 2026 PR #5037 + commit e47abd1 Mar 27. NEW PATTERN: AI-bot-PR-attempt via devin-ai-integration -- first instance of AI-bot contribution observed in catalog. DETAILED TECHNICAL ANALYSIS: @laniakea001 Mar 12 2026 identified 3 possible root causes. POST-CLOSURE EXTERNAL TECHNICAL CONFIRMATION: @srinivasycplf8 2 weeks before fetch: "Read your repro and the _update_manager_tools fix -- the manager silently falls through to its own tools and the hierarchical process collapses into a sequential one framing is the cleanest description of that failure mode I have seen written down. The part that always gets me about this class of bug is that the run looks syntactically fine in the logs -- no delegation, no error, just a quietly wrong process shape, which is the worst kind of wrong for an agent graph." -- external technical confirmation paper-quotable. EXTERNAL VALIDATION FROM PUBLISHED ARTICLES: Towards Data Science article Nov 2025 "Why CrewAIs Manager-Worker Architecture Fails" + azguards.com Mar 2026 "The Delegation Ping-Pong: Breaking Infinite Handoff Loops in CrewAI Hierarchical Topologies" both independently document this bug with token-burn evidence and Delegation Ping-Pong framing. Sub-mechanism within M-delegation-fanout: hierarchical-process-collapses-to-sequential + manager-silently-falls-through-to-own-tools. PAIRS WITH CRAI-008 (#2838) FOR 10-MONTH-SPANNING EVIDENCE: same hierarchical-process-broken bug documented at May 2025 (CRAI-008), Mar 2026 (CRAI-010), with earlier instance referenced via #1823. NEW CLOSURE-PATTERN (15th in taxonomy): stale-closed-despite-active-PRs-in-flight -- most aggressive auto-stale-close documented to date. NEW ACTOR-CLASS: AI-bot-as-issue-fixer (devin-ai-integration). NEW EVIDENCE-CLASS: external-third-party-confirmation via published technical articles (Towards Data Science, azguards.com).