sajjadanwar0/token-budgets-catalog huggingface dataset:
Framework: claude-code (2025-07)
Issue: Terminal Rendering Glitch on Warp with Long Conversation Context
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class; also S2). Filed Jul 18 2025 by @zubenelakrab; closed Aug 20 2025 as duplicate of #5224. Mouse-event/terminal rendering glitch on Warp terminal with long conversation context. UI bug, not cost mechanism.
Framework: claude-code (2025-07)
Issue: request (voice support for visually impaired users)
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class; also S2). Filed Jul 26 2025 by @Aaronminer1; closed Aug 15 2025 as duplicate of #2189. Voice-support accessibility feature request. Not cost-related.
Framework: litellm (2025-02)
Issue: automated nesting when using litellm sdk within langfuse observe() decorated function
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). Filed Feb 9 2025 by @marcklingen (Langfuse contributor); closed Aug 6 2025 as not planned. Feature request to auto-detect langfuse_context and set existing_trace_id/parent_observation_id automatically when LiteLLM SDK is called from inside @observe()-decorated function. Observability/DX integration, not cost mechanism.
Framework: litellm (2024-11)
Issue: Number of golden records in output is not the same as requested in input during synthesis
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). Filed Nov 18 2024 by @nalin-programmer in wrong repo (issue is about deepeval golden-records synthesis, not LiteLLM). Closed same-day by @ishaan-jaff: "you are on the wrong repo." Not relevant to any cost mechanism.
Framework: litellm (2024-12)
Issue: (UI QA) - stop making expensive UI queries when 1M+ spendLogs in DB
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). PR merged Dec 15 2024 by @ishaan-jaff. UI performance fix for spendLogs DB queries (avoid expensive queries when >1M rows). Internal UI optimization, not cost mechanism in the framework-budget-overrun sense.
Framework: litellm (2024-09)
Issue: Add a (strict) lint check either before or during runtime for litellm config.yaml
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). Filed Sep 25 2024 by @ishaan-jaff (maintainer); auto-closed May 7 2025 as not planned. Feature request for config.yaml lint check on startup. Config validation, not cost mechanism.
Framework: litellm (2025-04)
Issue: Suspected logic bug in get_optional_params
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). PR by @peteski22 Apr 24 2025; auto-closed Aug 26 2025 as stale. Refactors get_optional_params function-call/tools handling logic and adjusts ollama special-case. Refs issue #10244. Internal refactor, not cost mechanism.
Framework: litellm (2024-10)
Issue: (prod stable branch) Oct 28th
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). Production stable branch merge PR by @ishaan-jaff Oct 30 2024; closed by author same day after consolidation. 1201 line additions across 74 files; not a single-issue cost mechanism.
Framework: litellm (2025-03)
Issue: (UI) Fixes to session token handling
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). PR by @ishaan-jaff Mar 4 2025; closed by author same day. Cookie/session-token UI handling fixes (stop storing userID in URL, use cookie utils). Auth/UI bug, not cost mechanism.
Framework: litellm (2025-01)
Issue: heroku additions - initial test
Notes: SKIPPED for paper after URL verification (S1 wrong-failure-class). PR by @hillarysanders Jan 11 2025; closed Jan 17 2025 by author (deleted head repository). Initial WIP for Heroku LLM provider integration. Provider integration, not cost mechanism.
Framework: mastra (2025-07)
Issue: [BUG] Upstash for storage is getting ridiculously expensive
Mechanism cluster: M-storage-amplification
Reported user dollar loss: 100
Notes: paper:bu; closed Aug 9 2025 via PR #6672 (docs warning, NOT code fix). Reporter @MaryOfLA documents +$100 USD spent in development phase ("Just 20 or 30 conversations testing the agent setup") with claude-sonnet-4-20250514 + Upstash Redis storage; three-word agent question generated huge entry. Second user @leo-paz (Jul 25): "same crazy charge of 80$ for me in local dev, the upstash storage type seems unusable." Maintainer @TylerBarnes engaged Upstash co-founder @enesakar directly: "I would love to get your input on how we can improve our Upstash storage adapter to make it less costly." Resolution was docs warning about pay-as-you-go pricing rather than adapter optimization; @enesakar recommended fixed plans starting from $10/month. Underlying mechanism (Mastra Memory adapter issuing many Redis commands per agent turn) was not changed. Tagged p:high, Bug issue type. First Mastra entry in catalog. NOVELTY: first storage-layer cost case in the catalog (all other dollar-loss rows -- AIDR-001 $13.55, AIDR-003 $83, AIDR-005 $4.99, OCLW-001 $4.85, NANO-001 $2.70, CCDE-001 $235, DNSW-001 ~$2150 -- are LLM cost). Sub-mechanism within M-cost-observability: storage-layer cost amplification (provider command counting times framework storage adapter inefficiency). n=1; needs n=3 per codebook section 10 to become its own M-storage-amplification category.
Framework: mastra (2025-07)
Issue: Token / Cost Usage Tracking?
Mechanism cluster: M-cost-observability
Notes: paper:fr; closed completed Jul 30 2025 by Member @wardpeet citing https://mastra.ai/blog/nested-streaming-support and forthcoming docs. Filed by Member @abhiaiyer91 himself on behalf of Discord user. Quotable: "Pre Mastra, I built a basic cost logger that tracked the token usage results from OpenAI calls. Would be great to have something like that embedded into Mastra... Ideally, it would be possible to attribute token usage to some kind of (user) identifier." Tagged Agents + p:high. Critical follow-up by @KeyCode17 (Aug 1 2025): the fix exposes completionTokens/promptTokens/totalTokens but NOT cost-in-dollars. Cited OpenRouter response format which exposes a cost field directly with cost/cost_details/upstream_inference_cost. Quotable: "I think cost is more readable to human because it is straight into currency right?" Issue stayed closed despite incomplete-fix critique. Pattern matches ATGN-010 RequestUsage-missing-cost-field (also OPEN, also tokens-but-no-dollars distinction). First Mastra entry in M-cost-observability cluster. Strengthens cross-framework maintainer-files-own-request pattern (now 4 cases): ATGN-006 by @ekzhu, ATGN-009 by @fniedtner (Microsoft), GPTE-003 by @AntonOsika, MAST-002 by @abhiaiyer91 (Mastra Member). Token-vs-cost gap is structurally consistent: tokens are easy to expose at SDK level; cost requires pricing tables that go stale (cf LANG-005 standardize_model_name on o3, ATGN-008 _MODEL_TOKEN_LIMITS KeyError on llama3.3:latest).
Framework: mastra (2025-10)
Issue: [FEATURE] Track token usage across agent network iterations
Mechanism cluster: M-cost-observability
Notes: paper:fr; closed completed Nov 3 2025 by @taofeeq-deru via PR #9649 (commit 1521d71). Filed by Contributor @rase- on Oct 20 2025. Body: "Currently, the individual steps in an agent network have their usages reported through nested events. However, there is no overarching usage tracking. Add usage data into existing agent network events, most likely these should live in the step finish events." Tagged enhancement + p:high + trio-tnt. Self-assigned then reassigned to @taofeeq-deru on Oct 24 2025; PR merged ~10 days later. Distinct M-cost-observability sub-mechanism from MAST-002: agent-network-iteration-level rollup (nested events not summed) vs framework-level token tracking (entirely missing). Parallels AutoGen sub-mechanism stratification: ATGN-006 TaskResult-level, ATGN-008 model-table-level, ATGN-009 v0.4-regression-overall, ATGN-010 RequestUsage-missing-cost-field -- different layers of the same observability stack. Mastra is now the third framework with multiple cost-observability sub-mechanism rows: LangChain has LANG-006/007/008/010 (4 rows), AutoGen has ATGN-006/008/009/010 (4 rows), Mastra has MAST-002/003 (2 rows). Strengthens cross-framework architectural-pattern claim that cost-observability gaps are not framework-specific.
Framework: mastra (2025-06)
Issue: [BUG] memory processor only run the 1st time agent is called
Mechanism cluster: M-context-amplification
Notes: paper:mf; closed by Member @abhiaiyer91 Jul 11 2025: "Closing as is by design for now!" -- direct by-design closure quotable. Filed by @vikyw89 Jun 26 2025 documenting TokenLimiter(127000) memory processor only running at memory-retrieval step, not per-LLM-call inside agentic loop. Quotable user concern: "Some model only has 60k context window (deepseek), which there is no way to get low level and manage the token limit using mastra." Maintainer @TylerBarnes (Jun 26) recharacterized label from bug to enhancement and gave full mf framing: "this is by design right now unfortunately. Memory processors only process messages retrieved from memory, not new input messages or response messages from the LLM. We are planning to take the memory processors concept and expand it into a new agent message processors feature. When we do that this will work the way you are expecting." Workaround pointed at AI SDK middleware (doStream wrap) -- not framework-native. NOTE on later state: subsequent release notes (post-PR #14170) document "Fixed exponential token growth during multi-step agent workflows by implementing processInputStep on TokenLimiter" -- eventual fix shipped in different code path months later. BATCH-26 EVIDENCE EXPANSION: maintainer-label-change pattern documented (TylerBarnes Jun 26 SAME DAY removed bug label and added enhancement label -- maintainer-reclassification-from-bug-to-fr signal). Maintainer-explicit-missing-feature-admission with promised-future-feature-without-delivery-date. Workaround-via-lower-level-API: escape-to-AI-SDK-middleware bypassing Mastra abstraction (parallels MAST-006 escape-to-Vercel-AI-SDK pattern). User has to drop one abstraction layer to solve the problem. POST-CLOSURE CROSS-REFERENCE: @bkidy Jul 23 2025 mentioned issue #6169 "Cannot import TokenLimiter and ToolCallFilter from @mastra/memory/processors" -- separate primitive-instability bug suggesting TokenLimiter primitive itself may have packaging/import issues beyond runtime semantics. NEW CLOSURE PATTERN: closed-as-wontfix-with-promised-future-feature (16th sub-shape in closure typology). Mechanism: M-context-amplification with documented-mitigation-not-firing-per-step pattern. CROSS-CLUSTER FIT (added v22-batch-26): also fits M-budget-primitive-missing as primitive-exists-but-only-runs-on-memory-retrieval-not-loop-iterations sub-mechanism. Distinct from SMAG-005 (primitive-doesnt-exist), MAST-005 (primitive-via-callback-only), CRAI-011 (primitive-exists-but-broken-via-silent-regression). Tag stays mf because maintainer explicitly framed gap as by-design with future-feature promise -- textbook mf shape per codebook. Strengthens by-design cluster (n=7 with this row, becoming 8 cross-framework cases).
Framework: mastra (2025-09)
Issue: [BUG] Langfuse exporter not capturing output tokens from LLM calls
Mechanism cluster: M-cost-observability
Notes: paper:bf; closed completed Oct 14 2025 by Member @epinzur via PR #8790 (preceded by reporter-filed PR #8417). Filed Sep 25 2025 by @rmarquet21 against @mastra/langfuse 0.0.8 + @ai-sdk/anthropic 2.0.17 with thinking.budgetTokens 12000 enabled. Tagged bug + p:high + Observability (AI Telemetry). Documented gap: "LLM trace in Langfuse shows: Output: null, Completion tokens: 0, Total tokens only counting input tokens (e.g. 10,528 -> 0 (Sum 10,528))." Reporter explicitly noted: "However, the agent span correctly shows output data, indicating the LLM is producing output that is not being captured in the trace." Linked to umbrella tracking issue #8149. 19-day resolution. Mastra first M-cost-observability bug-with-fix (vs MAST-002/003 which are feature requests). Sub-mechanism: observability-export-layer mapping gap, distinct from MAST-002 framework-level token tracking and MAST-003 network-iteration-level rollup. Direct parallel to LANG-006/007/008/010 cluster. NOTABLE asymmetry: 19-day fix here vs LANG-008 4-month auto-stale-close on same bug class -- strengthens codebook asymmetric-attention claim. Mastra now has 3 M-cost-observability rows (MAST-002 framework-level fr, MAST-003 network-iteration fr, MAST-006 observability-layer bf) -- matches AutoGen 4-row stratification (ATGN-006/008/009/010) and LangChain 4-row pattern. Cross-framework architectural-pattern claim now well-supported by 3 ecosystems.
Framework: mastra (2025-10)
Issue: Mastra Server not handling rate limit error for anthropic provider
Mechanism cluster: M-retry-loop
Notes: paper:bf; closed completed Oct 7 2025 (6-day resolution) via PR #8567 by @LekoArts (Member). Filed Oct 1 2025 by Contributor @wardpeet on behalf of Discord user @meghdip-fenrir. Documented mechanism: Anthropic API returns 429 with "This request would exceed the rate limit for your organization of 30,000 input tokens per minute"; Mastra Server fails to catch the AI_APICallError exception, resulting in an unhandled exception in the agent stream that returns HTTP 200 OK with crashed body. Stack trace shows error originating at AnthropicMessagesLanguageModel.doStream then propagating through wrapStream and createStream. Maintainer @roaminro quotable acknowledgment: "We do not currently handle rate limit errors automatically. But, yes, I do not think it should return a 200 :)" Reporter critical follow-up @meghdip-fenrir: "The automatic retry also fails because of the response being 200. It would be nice if there was a way to automatically wait for the retry after and make the request again when the rate limit is hit for providers" -- articulates that the framework-level retry-on-429 feature is also missing. Tagged bug + p:high + Mastra Server. Direct parallel to LANG-016 (Anthropic rate-limit silent-hang) but distinct sub-mechanism: 200-OK-on-429 breaks client-side retry logic (vs LANG-016 server-hangs-indefinitely). M-rate-limit-amplification cluster now spans 3 frameworks: LangChain (LANG-012, LANG-016), Aider (AIDR-003), Mastra (MAST-007). Reporter follow-up about missing auto-retry-after-header echoes the unfilled feature gap that LANG-014, LANG-016, AIDR-003 all surfaced.
Framework: mastra (2026-01)
Issue: [FEATURE] Expose provider response headers in agent.generate() and agent.stream()
Notes: SKIPPED for paper after URL verification (S5 user-found-existing-feature). Filed Jan 16 2026 by @tomaskupka requesting response.headers be added to agent.generate() and agent.stream() return values to enable rate-limit-aware batch processing. Tagged enhancement + impact:high + effort:medium + Agents. Body has strong cost-observability framing including "Cost optimization: Access to headers like anthropic-ratelimit-tokens-reset helps optimize batch processing" and "Token usage tracking - Estimate rate limits by accumulating usage.inputTokens across requests" alternatives-considered section. Member @graysonhicks responded same day with code pointers showing the feature already exists and noted: "I do think the docs for generate are kind of hiding this under the result type though." Reporter @tomaskupka tested 3 days later (Jan 19): "Just tested agent.generate(my prompt).response?.headers?.[anthropic-ratelimit-input-tokens-remaining], and it works!" Closed Jan 21 2026 in PR #12115 -- DOCS-ONLY fix. Same shape as MAST-005, CRAI-005, AGPT-004, LLMI-001. Codebook S5 cleanly applies. Note: this case adds qualitative signal that even when frameworks DO expose cost-optimization primitives, docs hiding them defeats the purpose -- worth a sentence in paper Section 6 if the discoverability theme gets developed.
Framework: mastra (2025-10)
Issue: [FEATURE] Add support for providerOptions (e.g. cache control) when defining tools
Mechanism cluster: M-context-amplification
Notes: paper:fr; closed completed Nov 29 2025 (~5 week resolution) via PR #10649 by Member @abhiaiyer91 (commit d6ce671). Filed Oct 21 2025 by @diegoarda. Tagged enhancement + Tools. Cost-optimization framing throughout: "Having this flexibility is useful to optimize latency, cost, and correctness depending on the tool purpose" and proposed Mastra-internal forwarding of providerOptions including {anthropic: {cacheControl: {type: ephemeral}}} from tool definitions to underlying model calls. CRITICAL post-merge regression report by @Prince-Mendiratta (Jan 23 2026): "this does not seem to be working? I am on v1" with full weatherAgent config showing both thinking budget_tokens: 32000 and cacheControl: {type: ephemeral} in defaultOptions.providerOptions.anthropic block. No maintainer reply at fetch time. Pattern: cost-control feature shipped, then breaks during major version bump (v1) -- parallels ATGN-009 (v0.2 -> v0.4 cost-observability regression). M-cost-feature-gap cluster expansion: Mastra second fr-shaped cost-control row (MAST-002 cost-tracking, MAST-009 cache-control), parallel to AIDR-009 (/compact manual cost-control). Cross-framework cost-feature-gap cluster: AGPT-001/002, AIDR-009, NANO-001, MAST-002, MAST-009 -- n=5 paper-tagged cases across 5 frameworks/years.
Framework: mastra (2025-07)
Issue: [BUG] Query performance reduces as memory grows larger
Mechanism cluster: M-storage-amplification
Notes: paper:bf; closed completed Sep 16 2025 (~2 month resolution) by Member @NikAiyer via PR #7757 (commit 60c9cec). Filed Jul 13 2025 by @alexbaileyuk against @mastra/pg 0.12.2. Tagged Bug issue type + p:high + Storage & Databases. Reporter explicitly bridged to MAST-001: "Could possibly relate to Upstash for storage is getting ridiculously expensive." Documented mechanism: mastra_threads PostgreSQL table only had primary-key index; getThreadsByResourceId generated SELECT...WHERE resourceId = $1 ORDER BY createdAt DESC executing as full table scan. Community contributor @yeouchien expanded scope with code-line references to 4 affected query paths: getMessages, getMessagesPaginated, getTracesPaginated, getEvals -- all full-scanning their respective tables on resourceId/thread_id/agent_name/name. Resolution: PR #7757 added index management to PostgreSQL storage adapter plus automatic indexes on initialization. NOVELTY: n=2 case for storage-amplification sub-mechanism within Mastra (MAST-001 Upstash command-count + MAST-010 PostgreSQL full-scan); per codebook section 10, n=3 minimum to elevate to its own M-storage-amplification category. Follow-up Discord issue #8044 by @wardpeet extending request to MongoDB; follow-up users @kosukeoya and @atomaisc explicitly asking about MongoDB support. Suggests storage-cost amplification recurs across backend choices and is not Postgres-specific. Distinct from LLM-cost-amplification cluster: framework-induced storage-backend cost from inefficient query patterns rather than retry-storm or context-amplification. ACTION ITEM for batch 18+: search PostgreSQL/Redis/Mongo storage cost issues across LangChain, AutoGen, LlamaIndex repos to test whether storage-amplification reaches n=3 cross-framework, which would justify M-storage-amplification as its own paper category.
Framework: mastra (2025-12)
Issue: [BUG] Memory class does not support AI SDK v5 providerOptions for embedding dimensions
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bf; closed completed Dec 23 2025 (NEXT-DAY resolution) via PR #11362 by Member @abhiaiyer91 (commit 7bcbf10). Filed Dec 22 2025 by @marto against @mastra/memory 0.15.13 + @mastra/core 0.24.8 + ai 5.0.115 + @ai-sdk/openai 2.0.88. Documented mechanism with source line reference: @mastra/memory/dist/index.js line 522 calls embedMany() without providerOptions, blocking OpenAI Matryoshka Representation Learning dimension-shortening for text-embedding-3-large/small. Concrete amplification factor cited in body: "Storage costs: Users must use full 3072-dimension vectors instead of optimized 256-dimension vectors (12x more storage)." Body cites OpenAI MTEB benchmark: "text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536." Reporter explicitly stated "No workaround: There is no way to configure this through Memory current API" then in same body provided a working withOpenAIDimensions wrapper using doEmbed() override. Body cited related closed issues: #7117, #3236. Retag pattern: @abhiaiyer91 retagged from bug to enhancement+effort:low+impact:high+For Release before fixing -- same maintainer-perspective signal as MAST-004 but unlike MAST-004 mf closure, this one shipped within 24 hours. CLUSTER MILESTONE: n=3 case for storage-amplification within Mastra alone (MAST-001 Upstash command-count + MAST-010 PostgreSQL full-scan + MAST-011 embedding-dimension-not-passed) -- per codebook section 10 (n=3 minimum for new category), this clears the threshold to elevate storage-amplification from sub-mechanism to its own M-storage-amplification category in v1.1 codebook revision. Three distinct sub-mechanisms: (a) storage-backend-command-count, (b) framework-side-missing-indexes, (c) embedding-dimension-not-passed-through. ACTION ITEM v1.1: codebook revision should add M-storage-amplification with these 3 cases as exemplars.
Framework: mastra (2025-09)
Issue: Anthropic System Prompt Caching Regression
Mechanism cluster: M-storage-amplification
Notes: paper:bf; closed completed Sep 19 2025 (2-day resolution) via PR #7987 by Member @abhiaiyer91 (commit cf34503). Filed Sep 17 2025 by Contributor @wardpeet on behalf of Discord user. Tagged p:urgent. Reporter documented Mastra Agent class wrapping logic dropping providerOptions for system role messages despite passing them in streamVNext message array. Reporter provided MINIMAL REPRO PROVING MECHANISM: identical code structure with @ai-sdk/anthropic streamText direct call DOES cache the system prompt; same code through Mastra Agent class wrapping omits the providerOptions. Reporter quote: "it seems that Mastra has some proprietary logic wrapped over the AI-SDK that omits the added providerOptions." Maintainer @abhiaiyer91 quotable resolution: "Hi we just added the ability to pass SystemMessage in instructions rather than just a string! This should solve your issue natively!" CRITICAL POST-MERGE REGRESSION: @razasaad reported Sep 25 2025 (6 days post-close) with @mastra/core 0.18.0 using exactly the pattern @abhiaiyer91 said would work: "I am still encountering the issue with caching request not being propagated." No maintainer reply at fetch time. Same fix-shipped-immediately-regressed pattern as MAST-009 (Prince-Mendiratta v1 regression). providerOptions-silently-dropped recurring sub-mechanism cluster within Mastra now n=3: MAST-006 (Langfuse output mapping), MAST-009 (tool-level cacheControl), MAST-012 (system-message cacheControl).
Framework: mastra (2026-03)
Issue: Anthropic prompt caching with agent.stream() and Memory -- how to cache conversation history?
Mechanism cluster: M-storage-amplification
Notes: paper:bu; closed Mar 6 2026 (next day) by Member @roaminro pointing to mastracode promptCacheMiddleware as recommended approach. Filed Mar 5 2026 by Contributor @daneatmastra from Discord (user brandrsn) against @mastra/core 1.9.0 + claude-haiku-4-5-20251001 + agent.stream() with maxSteps: 5 + Memory with lastMessages: 20. INITIAL COST DATA: "The Anthropic dashboard confirms ~50% cache hit rate -- system prompt + tools are cached. The remaining 50% is conversation history loaded from Memory, which has no cache_control markers. As conversations grow, the hit rate drops." Reporter described attempted moving-cache-breakpoint pattern: "We tried adding providerOptions to messages passed to agent.stream(), but it gets stripped for UIMessages (from useChat) during the adapter chain." CRITICAL POST-CLOSE FOLLOW-UP by @brandrsn (Mar 10): "we were not able to use the processInputStep processor approach you recommended. In our case, conversations include tool calls, and we found that aiV5UIMessagesToAIV5ModelMessages() maps providerOptions by array index between UIMessages and ModelMessages. When assistant messages with tool calls get split into multiple ModelMessages, the indices diverge and the providerOptions ends up on the wrong message (or is lost entirely)." Reporter achieved ~92% cache hit rate using AI SDK middleware workaround vs 50% baseline -- quantifiable cost mechanism. Reporter explicit unanswered question: "is the middleware approach the right long-term solution, or is the processInputStep index mapping something you would consider a bug to fix?" No maintainer answer at fetch time. Cited related issue #13848 pairs with MAST-006 Langfuse exporter cluster. ACTIVE FOLLOW-UPS post-close: #15474 filed 2 weeks before fetch; PR #16133 filed 4 hours before fetch by @intojhanurag -- index-mapping bug remains active topic. NEW sub-mechanism: providerOptions-index-mapping-breaks-on-tool-call-splits (distinct from MAST-006/009/012 whole-providerOptions-dropped pattern). 50% -> 92% cache hit rate jump = 42 percentage points unmitigated Anthropic cache cost.
Framework: mastra (2026-01)
Issue: [FEATURE] Force fix JSON retry the call on JSON failures
Notes: SKIPPED for paper after URL verification (S3 weak-evidence-stub for cost cluster). Filed Jan 30 2026 by Contributor @TiagoGouvea against @mastra/core 1.1.0 proposing 5 structured-output recovery features: maxRetries, retryOn, tokenExpansionMultiplier (2x on retry), attemptJsonRepair, fallbackStrategy. Closed SAME DAY via PR #12526 by Member @graysonhicks (commit 47eba72) -- partial fix in agent-network code path only. Maintainer quotable: "Yes, this can definitely be a problem. I know we already handle it in some places, and I just did a PR yesterday to address in another place. If it continues to be a big problem, I think more advanced control as you propose could be a good idea." Reporter post-close evidence Jan 31 + Feb 2 documents recurring failures across kimi-k2, qwen-turbo, gemini-2.5-flash, llama-3.1-8b-instant, Qwen3-235B. Reporter built and published json-llm-repair package as external workaround. tokenExpansionMultiplier proposal is a token-burning pattern that would parallel AIDR-006 BUT honest classification: underlying mechanism is OUTPUT-VALIDATION-RECOVERY not BUDGET-OVERRUN. Adding to M-cost-feature-gap cluster would dilute with output-correctness-retry cases. Same-day partial-fix + maintainer acknowledgment of broader proposal + post-close evidence not engaged puts this in S3 territory. CONDITIONAL: if a future batch surfaces explicit token-amplification incidents from validation retries (e.g., user reporting 5x retry costs from JSON repair loop), this could pair with that case as M-output-recovery-amplification at n=2.
Framework: langchain (2023-08)
Issue: ParentDocumentRetriever is slow with FAISS because add_documents uses embed_query without batching
Mechanism cluster: M-storage-amplification
Notes: paper:bu; auto-stale-closed by dosubot Feb 20 2024 as not-planned with NO HUMAN-MAINTAINER ENGAGEMENT throughout 6-month thread. Filed Aug 29 2023 by Contributor @oegedijk. Documented mechanism with source line: vectorstores/faiss.py:166 contains embeddings = [self.embedding_function(text) for text in texts] -- single-call list comprehension where batched embed_documents() would be ~1000x more efficient (OpenAI default batch=1000). Reporter posted working monkeypatch using OpenAIEmbeddings.embed_documents() directly bypassing FAISS.add_documents. COMMUNITY VALIDATION: @jamesw6811 (Sep 20 2023): "I think this patch would make a good PR"; @yiouyou (Oct 20 2023) + @weissenbacherpwc (Nov 7 2023) reused the patch asking only setup questions; @jsxyhelu (Apr 8 2024, post-stale-close) provided full working integration with HuggingFaceBgeEmbeddings adapter. dosubot tagged auto:improvement, encouraged PR, but never escalated to human maintainer review. Same sub-mechanism as MAST-011 (embedding-dimension-not-passed-through): framework storage adapter does not pass available efficiency primitives through to embedding API. Same sub-mechanism as LANG-019 (Azure Search batch-API-not-used) -- confirms pattern is adapter-class architectural, not adapter-specific. CROSS-FRAMEWORK SIGNAL: combined with MAST-001/010/011 (Mastra n=3), LANG-017/019 (LangChain), LLMI-005/006 (LlamaIndex), establishes M-storage-amplification as cross-framework architectural class. Adds first LangChain entry to M-storage-amplification cluster.
Framework: llamaindex (2024-06)
Issue: LanceDBVectorStore database size blows up on creation
Mechanism cluster: M-storage-amplification
Notes: paper:bf; closed completed Jul 8 2024 via PR #13993 by Member @logan-markewich (commit c828e11) -- 1-day from filing to merge. Filed Jun 7 2024 by Contributor @marrrcin against llama-index 0.10.43 on macOS 14.4. CONCRETE REPRO with MockEmbedding(768) proves framework adapter is culprit not embedding model: "storing 500 docs takes 1.2GB on disk" vs "same data in e.g. Chroma takes ~5MB on disk" = ~240x storage amplification. Reporter quotable mechanism diagnosis: "It is a bug in LanceDBVectorStore.add logic which results in quadratic storage increase. I am working on a fix right now." Body shows add() source code constructing append_data dict per node with vector embedded inline plus full metadata, suggesting per-node duplication on each table.add() call. Reporter-as-fixer pattern: filed bug + fix PR same day, merged next day. dosubot offered generic optimization advice (create_index, compression, scalar/vector index) that missed the actual quadratic-add-logic mechanism -- contrast with @marrrcin correct root-cause diagnosis demonstrates value of human reporter expertise over LLM bot triage. Sub-mechanism within M-storage-amplification: vector-store-write-amplification (distinct from MAST-001 storage-command-count which is read-amplification, distinct from LANG-017/019 embedding-batching-not-used which is embedding-side). Adds first LlamaIndex entry to M-storage-amplification cluster. Pattern parallels MAST-001 at write-time -- both are framework-side-write-pattern-induced storage cost, different storage backends.
Framework: langchain (2023-07)
Issue: Loading vectorstore from disk still calls to OpenAI embeddings API
Notes: SKIPPED for paper after URL verification (S3 weak-evidence-stub). Filed Jul 27 2023 by @xxt07d; auto-stale-closed by dosubot Nov 2 2023 as not-planned. dosubot tagged auto:question (not auto:improvement) -- bot classification as user error rather than framework bug. Reporter evidence ambiguous on second-look: code shows if os.path.exists(path) branch loads via Chroma + embedding_function=OpenAIEmbeddings() and else branch creates via VectorstoreIndexCreator. Reporter reports "2 requests to text-embedding-ada-002-v2 since there are 2 text files" verified via OpenAI billing dashboard -- this is EXPECTED first-run behavior (creating embeddings for 2 fresh files), NOT the load-path re-embedding the reporter framing implies. Reporter framed as "still calls" suggesting a second-run observation, but reproduction steps describe first-run only. ZERO community confirmations, ZERO human-maintainer engagement, dosubot user-error classification. Pattern of re-embedding-on-load IS legitimate (LLMI-006 has stronger evidence for it via Disiok maintainer ack), but this specific case does not clear the bar. Skipping per S3 keeps M-storage-amplification cluster mechanism-coherent.
Framework: llamaindex (2023-05)
Issue: How to store embedding vector when using a custom embedding model?
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bu; closed completed Jun 11 2023 by Collaborator @Disiok with EXPLICIT ACKNOWLEDGMENT: "Looks like the embedding not saving to disk is a bug, going to fix this!" -- but no clear evidence of fix landing in identifiable PR. Filed May 17 2023 by @DZ9 against llama-index 0.6.7 + langchain 0.0.169 with HuggingFaceEmbeddings (text2vec-large-chinese model) and ChatGLM custom LLM. Documented mechanism: GPTListIndex with custom embedder, index.storage_context.persist() writes empty vector_store.json, "the embedding vector will regenerate every time I run the code." Direct cost-amplification mechanism: persistent storage primitive does not actually persist embeddings, forcing full re-embedding cost on every load. @Disiok partial workaround pointer: GPTListIndex stores embeddings in index_store.json not vector_store.json -- recommended switching to VectorStoreIndex. BUT @Disiok explicitly committed to fix the underlying bug separately from the workaround. CRITICAL POST-CLOSURE FOLLOW-UP by @fishfree (Jan 16 2024, ~7 months later): "@Disiok Sorry to disturb you. You marked this issue as completed, then where is your commit, please?" -- pointed question suggesting promised fix never shipped in identifiable PR. No maintainer reply at fetch time. Pattern: closed-as-completed-by-promise-without-delivery (similar to ATGN-010 "we will address after 0.4.0" pattern). Sub-mechanism within M-storage-amplification: re-embedding-on-load (distinct from LLMI-005 vector-store-write-amplification, distinct from LANG-017/019 embedding-batching). Pattern parallels MAST-011 (Memory class does not pass providerOptions for dimensions) -- both are framework persistence layer not preserving cost-relevant config. Adds second LlamaIndex entry to M-storage-amplification cluster.
Framework: langchain (2023-10)
Issue: Azure Cognitive Search vector DB store performs slow embedding as it does not utilize the batch embedding functionality
Mechanism cluster: M-storage-amplification
Notes: paper:bu; auto-stale-closed by dosubot Feb 12 2024 as not-planned despite multiple community confirmations + apparent fix attempts. Filed Oct 3 2023 by @butlfrazp (Author noted "I am going to work on this issue"). Documented mechanism: AzureSearch.add_documents processes documents individually, not utilizing batch embedding capability that Milvus implementation supports. COMMUNITY CONFIRMATIONS: @pietz (Oct 27 2023): "I am facing the same issue. Currently adding documents will request an embedding for each doc sequentially until it has 1000 embeddings"; @pablocastilla (Nov 6 2023): "When we split a document in 180 chunks it takes one minute to upload to azure." REFERENCED FIXES BEFORE STALE-CLOSE: @msetbar PR #15594 (Jan 5 2024) "community: Fixing a performance issue with AzureSearch to perform batch embedding"; @hwchase17 (LangChain founder/Member) commit eb76f9c (Jan 12 2024) referencing this issue. CRITICAL POST-CLOSURE @pietz (Feb 27 2024, 15 days after stale-close): "Nothing has happened here, so let me post my dirty workaround" with concrete upload_documents() function delivering "10x faster in my initial test" -- demonstrates user-facing performance gap persisted at fetch despite apparent partial-fix activity. pietz workaround processes 1000 docs per batch via embedder.embed_documents() then uploads via vector_store.client.upload_documents(docs_batch). Pattern: fix-attempts-without-clear-user-facing-resolution + auto-stale-close. Same sub-mechanism as LANG-017 (batch-API-not-used) but for different framework adapter (Azure Search vs FAISS) -- confirms pattern is adapter-class architectural not adapter-specific. CROSS-FRAMEWORK MILESTONE: with LANG-017/019 + MAST-001/010/011 + LLMI-005/006 (now n=7 across 3 frameworks), M-storage-amplification cluster definitively clears codebook section 10 threshold for own paper category. Sub-mechanisms documented across cluster: (a) embedding-batching-not-used (LANG-017 FAISS, LANG-019 Azure Search), (b) embedding-dimension-not-passed (MAST-011), (c) re-embedding-on-load (LLMI-006), (d) vector-store-write-amplification (LLMI-005), (e) storage-backend-command-count (MAST-001 Upstash), (f) framework-side-missing-indexes (MAST-010 PostgreSQL). Six distinct sub-mechanisms across three frameworks confirms architectural-class rather than framework-quirk.
Framework: langchain (2024-09)
Issue: Anthropic prompt caching in langchain does not work with ChatPromptTemplate
Mechanism cluster: M-storage-amplification
Notes: paper:bf; closed completed Sep 10 2025 by Member @mdrxy via PR #30967 (Collaborator @ccurme Apr 23 2025: "This should be closed by #30967 upon release"). Filed Sep 20 2024 by @raajChit. 12-month thread with deep source-line-level mechanism documentation. KEY DIAGNOSIS by Contributor @Nilesh3105 (Mar 3 2025): langchain_core/prompts/chat.py lines 523-538 (_StringImageMessagePromptTemplate.from_template) -- "when it receives a list of dict with text key, it converts it to a StringPromptTemplate and drops all the additional properties/kwargs"; lines 647-658 (format method) -- "creates a new dict but it cannot and does not push additional properties/kwargs present in the original message." MULTIPLE COST CONFIRMATIONS: @drorm Dec 24 2024 "Without caching it is just too slow and too expensive"; @drorm Mar 4 2025 with concrete token-usage logs cache_creation_input_tokens: 1694, cache_read_input_tokens: 2521 demonstrating "I have a partial solution. It reduced my cost by 2/3"; @0ca Mar 2 2025 cross-referenced separate work "saving 90% on cache hits"; @kornicameister Jan 24 2025 confirmed same issue with 80k token prompt. MAINTAINER JOURNEY: @ccurme assigned @baskaryan Sep 21 2024 -> auto-stale-marked Dec 20 -> @drorm protest revoked stale Dec 24 -> reassigned to @ccurme Feb 13 2025 -> @Nilesh3105 PR #30537 Mar 28 -> @ccurme final PR #30967 Apr 23 -> closed Sep 10 2025. POST-CLOSURE @mathew3 Sep 26 2025 clarified part of user-perceived issue was Anthropic minimum-cacheable-prompt-length floor (1024 tokens for Sonnet/Opus, 2048 for Haiku) -- informational addendum not framework bug. Sub-mechanism: template-formatter-drops-additional_kwargs. Direct parallel to MAST-009 (providerOptions/cacheControl on tools, dropped via formatter). Adds first LangChain row to providerOptions-silently-dropped cross-framework cluster.
Framework: langchain (2025-10)
Issue: [LangGraph v1.0] Support Bedrock Anthropic prompt caching in create_agent system prompt
Mechanism cluster: M-storage-amplification
Notes: paper:bf with strong fix-merged-not-released delay signal. Filed Oct 22 2025 by @Haknt; closed completed Nov 8 2025 by Collaborator @sydney-runkle "Fixed by #33640, will be released soon!" (self-assigned Nov 7, closed Nov 8 -- 17-day filing-to-merge). v1.0 API REGRESSION MECHANISM: pre-v1.0 create_react_agent(prompt=cached_system_message) with structured SystemMessage(content=[{cache_control: ...}]) worked; v1.0 create_agent(system_prompt=INFO_GATHERER_PROMPT) accepts only str, breaking structured-message approach with cache_control. DIRECT FIX-SHIPPED-IMMEDIATELY-REGRESSED PARALLEL to MAST-009/012, but at major-version-migration boundary not patch-version regression -- distinct sub-shape of same architectural failure mode. Reporter explicit cost framing: "essential to reduce costs and latency when reusing long, static system prompts or tool definitions." Reporter alternative considered: "using a before_model middleware to rewrite the system message with a cache_control block" -- middleware-as-workaround signal. FIX-MERGED-NOT-RELEASED GAP: despite Nov 8 closure with "released soon," users continued reporting blockers through January 2026: @jvjfranca-auramind Nov 12 "We really need this fix, any idea when it will be released?"; @AlexDugheria Nov 15 "This is blocking our transition to langchain v1"; @zechenzhangAGI Nov 28 "I have same question as well, and for coding agent, it would also be great to cache the middleware in between tool calls." Closed-as-fixed-but-shipped-late pattern adds 11+ weeks user-facing impact gap. @elonf Jan 24 2026 mentioned PR #34867 -- ongoing related work. Sub-mechanism: API-regression-drops-structured-support at major-version-boundary. Adds second LangChain row to providerOptions-silently-dropped cluster with distinct fix-shipped-immediately-regressed sub-shape evidence.
Framework: langchain (2025-10)
Issue: AnthropicPromptCachingMiddleware breaks model fallback with cache_control param
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bu; OPEN at fetch despite 4+ months engagement and 3+ user-filed PRs. Filed Oct 29 2025 by Contributor @bart0401 with concrete minimal repro + complete root-cause analysis + execution-flow diagram. SOPHISTICATED MECHANISM: AnthropicPromptCachingMiddleware.before_model() adds cache_control -> Anthropic call fails (peer closed connection) -> ModelFallbackMiddleware switches to ChatOpenAI -> reuses message objects with cache_control still attached -> OpenAI rejects with TypeError: AsyncCompletions.create() got an unexpected keyword argument cache_control. "COMPLETE SERVICE FAILURE" FRAMING: "forcing production systems to choose between prompt caching optimization (AnthropicPromptCachingMiddleware) or reliability (ModelFallbackMiddleware), but not both." Reporter pre-flagged that unsupported_model_behavior=ignore only controls warnings, not message-state cleanup -- documented why the existing framework knob does not solve the problem. MULTIPLE PR ATTEMPTS WITHOUT CONVERGENCE: reporter PR #33716 Oct 29 (commit af5d71c); @ivancastanop PR #33850 Nov 6; @yashwantbezawada PR #33894 Nov 8 (commit db638fa). NONE MERGED at fetch. MAINTAINER ENGAGEMENT EXTENSIVE BUT NO RESOLUTION: @sydney-runkle (Collaborator) self-assigned Dec 12. @sydney-runkle Jan 5 2026: "It is possible we would want to clear the model settings to account for this, but that would be breaking. We could also add more hooks and solely clear the cache_control param" -- design indecision not implementation gap. @ccurme (Collaborator) Feb 12 2026 posted brittle workaround: swap middleware order. Workaround-only-no-fix pattern. MULTIPLE USER CONFIRMATIONS: @josiahcoad Dec 20; @dylan-clark-sh Dec 29; @srtab Dec 2 cross-referenced own bug daiv#681; @joy7758 Mar 11 2026 substantive cross-reference: "If middleware-mutated messages are reused across fallback model calls without re-running the middleware pipeline, provider-specific state can leak into the next model attempt." Sub-mechanism: middleware-mutated-state-leaks-across-fallback. Distinct from LANG-020 (template-side dropping) -- this is the inverse: framework wrongly-keeps-config when it should strip on transition between providers. INVERSE-PAIR to LANG-023: same middleware (AnthropicPromptCachingMiddleware), one bug stuck OPEN (LANG-022), one bug fast-fixed (LANG-023) -- internal asymmetry in attention to same component.
Framework: langchain (2025-12)
Issue: AnthropicPromptCachingMiddleware applies cache_control to tool_use/tool_result blocks called by code_execution causing API errors
Mechanism cluster: providerOptions-silently-dropped
Notes: paper:bf; closed completed in PR #34579 by Collaborator @ccurme Jan 6 2026 -- 7-day filing-to-merge. Released in langchain-anthropic 1.3.1 confirmed by @ccurme Jan 6: "Released in langchain-anthropic 1.3.1." Filed Dec 30 2025 by @pawel-kotowski with concrete minimal repro using claude-sonnet-4-5-20250929 + advanced-tool-use-2025-11-20 beta + code_execution server tool + suggested fix (skip blocks where block.get(caller, {}).get(type, ) contains code_execution). MECHANISM: AnthropicPromptCachingMiddleware applies cache_control to last content block in conversation without checking if that block is a tool_use/tool_result called by code_execution. Anthropic API rejects: "Tool use result for toolu_013QtuxLfX1Dm5JN97F9ZkWL called by code execution cannot have cache_control. These blocks are not rendered in Claude context." Reporter quotes Anthropic API spec: tool_use blocks called by code_execution have caller={type: code_execution_20250825, tool_id: srvtoolu_...} field that must be exempted from cache_control. PR-COMPETITION-WITH-FAST-RESOLUTION: three contributors filed competing PRs same week -- @monish-instinct PR #34543 Dec 30 (commit 1002bd5), @Ramsairohit PR #34562 Jan 1, @lwtaiyty PR #34579 Jan 2 (selected by @ccurme). FOR POSTERITY: @ccurme Jan 6 posted full ~70-line offending payload showing exact placement of erroneous cache_control on code-execution-called tool_result block, alongside correct surrounding messages -- detailed mechanism documentation for future reference. Sub-mechanism: middleware-applies-where-it-should-not (subblock-targeting bug). INVERSE-PAIR to LANG-022: same middleware (AnthropicPromptCachingMiddleware), one bug fast-fixed via PR-race (LANG-023, 7 days), one bug stuck OPEN with multiple PRs (LANG-022, 4+ months) -- internal asymmetry in attention to same component. The asymmetry tracks complexity-of-fix: LANG-023 is local subblock-skip logic (clear suggested fix); LANG-022 requires architectural decision about middleware re-running on fallback (Sydney noted breaking trade-off Jan 5).
Framework: dspy (2024-06)
Issue: How to set a max input length?
Mechanism cluster: M-retry-loop
Notes: paper:bu; closed Jun 22 2024 by @okhat without merged fix despite documented framework-side mechanism + maintainer acknowledgment. Filed Jun 12 2024 by @hellwigt-eq using HuggingFace TGI. STACK-TRACE-LEVEL MECHANISM: mipro_optimizer.py:459 compile() calls _generate_first_N_candidates which calls _observe_data at :247 with trainset[0:upper_lim].__repr__() -- batch summary step embeds repr of training set into LM call, which can exceed TGI max-input-tokens=8192 validation. Concrete error: "inputs tokens + max_new_tokens must be <= 8192. Given: 13800 inputs tokens and 75 max_new_tokens" = ~70% over context limit. PROMISED-PR-PATTERN: @arnavsinghvi11 (Collaborator) Jun 18: "there is not a clean way to set a max length on the final prompt sent to the LLM, but for this use case in TGI, you can add a validation check before the request is sent. this could be added via a PR with exploration on keeping flags for rejection, truncation+retry, etc." -- maintainer endorsed PR direction, never landed. POST-CLOSURE REPORTER WORKAROUND: @hellwigt-eq Jun 27: "I found that MIPRO() has an optional parameter called view_data_batch_size which defaults to 10. In my case, I was able to avoid these length errors by setting that to 3 instead." -- existing-knob-discovered-by-reporter pattern (default value too high for typical TGI configurations). @skunkwerk Oct 22 2024 cross-referenced in #1627. Sub-mechanism: M-optimizer-context-amplification at MIPROv2 batch-summary step. Distinct from DSPY-003 (different MIPROv2 sub-step). FIRST DSPY ENTRY in catalog. Establishes M-optimizer-context-amplification cluster which is DSPY-ARCHITECTURE-DISTINCTIVE -- only framework with automated optimizer that injects representations of training data into LM prompts at compile-time. Pattern parallels LANG-019 (Azure Search batch embedding) -- "this could be added via a PR" + closed without merge.
Framework: dspy (2024-08)
Issue: Integration of Langfuse with DSPy Framework - Zero Values for Cost and Token Usage
Mechanism cluster: M-cost-observability
Notes: paper:bu; closed Sep 19 2024 by reporter with ambiguous closure ("thanks for the info!") without explicit confirmation of cost/tokens-fix. Filed Aug 6 2024 by @mahitha-29. Reporter integrated Langfuse via documented @observe decorator -- traces visible in Langfuse dashboard but cost+tokens always zero. MAINTAINER DEFLECTION PATTERN: @okhat (Collaborator/founder) Aug 6: "Can you tag the langfuse team, in particular whoever made the contribution to DSPy? You can find this from the git commit history." -- bounced to upstream rather than diagnosing. @arnavsinghvi11 (Collaborator) Aug 16: pointed to merged Langfuse PR #1186. RELEASE-DELAY GAP: @xucailiang (Contributor, original Langfuse PR author) Aug 28: "The integration of dspy for langfuse appeared after dspy==2.4.13, so if you want to use pip install, you may need to wait for the next official version of dspy to be updated." Reporter Aug 22 hit this: "As per the latest DSPy v2.4.13 release, which was done three weeks ago, it does not cover the integration of Langfuse with DSPy." Reporter self-closed Sep 19 with "thanks for the info!" -- closure-without-explicit-confirmation-of-cost-tokens-fix pattern (tested-on-main not pip-release at fetch time, no follow-up confirming cost/tokens-now-shown). Honest evidence-base note: closure ambiguity weakens bu evidence-bar slightly relative to DSPY-001/004; row note transparently flags ambiguity. Sub-mechanism: M-cost-observability obs-bridge -- DSPy <-> Langfuse bridge does not propagate cost/token values. DIRECT PARALLEL TO MAST-006 (Mastra <-> Langfuse exporter output tokens=0). Two cross-framework rows (Mastra + DSPy) for Langfuse-bridge specifically -- structurally suggests Langfuse integration is a recurring weak spot across multiple frameworks. Maintainer-deflection-to-upstream is a distinct closure sub-shape from auto-stale, by-design, or experimental-feature-disclaimer.