| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="UTF-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> |
| <title>LLM Tool-Calling Models: Cost & Context Analysis</title> |
| <link rel="stylesheet" href="style.css"> |
| <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js"></script> |
| <script src="https://cdn.jsdelivr.net/npm/chartjs-plugin-annotation@3.0.1/dist/chartjs-plugin-annotation.min.js"></script> |
| <script src="https://cdn.jsdelivr.net/npm/papaparse@5.4.1/papaparse.min.js"></script> |
| </head> |
| <body> |
| <header> |
| <h1>Agentic LLM Price Comparison</h1> |
| <p class="subtitle">Cost-effectiveness analysis of 218 tool-calling models from OpenRouter</p> |
| <p class="date">Data snapshot: November 8, 2025</p> |
| <nav class="header-nav"> |
| <a href="output-input-analysis.html">🔢 Output/Input Price Analysis →</a> |
| </nav> |
| </header> |
|
|
| <main> |
| <section class="intro"> |
| <h2>Overview</h2> |
| <p>This analysis examines LLM models with tool-calling capabilities - essential for agentic AI systems, MCP (Model Context Protocol) implementations, and multi-tool orchestration. Models are evaluated across two key dimensions:</p> |
| <ul> |
| <li><strong>Cost per token</strong> - Economic efficiency for production use</li> |
| <li><strong>Context window size</strong> - Capability for complex, long-form tasks</li> |
| </ul> |
|
|
| <div class="attribution"> |
| <h3>Methodology</h3> |
| <p><strong>Quadrant Classification:</strong> Models are categorized into four quadrants based on median values calculated from the dataset:</p> |
| <ul> |
| <li><strong>Cost Division:</strong> Median average cost (mean of input + output pricing) = <span id="median-cost-display">calculated from data</span></li> |
| <li><strong>Context Division:</strong> Median context window size = <span id="median-context-display">calculated from data</span></li> |
| </ul> |
| <p>This creates four categories: Low Cost/High Context, High Cost/High Context, Low Cost/Low Context, and High Cost/Low Context.</p> |
|
|
| <h3>Attribution & Data Notes</h3> |
| <ul> |
| <li><strong>Data Collection:</strong> <a href="https://danielrosehill.com" target="_blank">Daniel Rosehill</a></li> |
| <li><strong>Collection Date:</strong> November 8, 2025</li> |
| <li><strong>Source:</strong> Prices derived from API calls to <a href="https://openrouter.ai/" target="_blank">OpenRouter</a></li> |
| <li><strong>Pricing Variability:</strong> OpenRouter pricing can fluctuate slightly according to the end inference provider. The same models may have slightly different pricing even at the time of capture.</li> |
| <li><strong>Data Exclusions:</strong> Free models (cost = $0) were excluded from this analysis</li> |
| </ul> |
| </div> |
| </section> |
|
|
| <section class="stats-grid"> |
| <div class="stat-card"> |
| <div class="stat-number" id="total-models">218</div> |
| <div class="stat-label">Models Analyzed</div> |
| </div> |
| <div class="stat-card"> |
| <div class="stat-number" id="vendors">-</div> |
| <div class="stat-label">Vendors</div> |
| </div> |
| <div class="stat-card"> |
| <div class="stat-number" id="median-input">-</div> |
| <div class="stat-label">Median Input ($/M)</div> |
| </div> |
| <div class="stat-card"> |
| <div class="stat-number" id="median-output">-</div> |
| <div class="stat-label">Median Output ($/M)</div> |
| </div> |
| <div class="stat-card"> |
| <div class="stat-number" id="median-multiple">-</div> |
| <div class="stat-label">Median Out/In Multiple</div> |
| </div> |
| <div class="stat-card"> |
| <div class="stat-number" id="context-range">-</div> |
| <div class="stat-label">Context Range</div> |
| </div> |
| </section> |
|
|
| <div class="tabs"> |
| <button class="tab-button active" data-tab="models">All Models</button> |
| <button class="tab-button" data-tab="quadrants">Quadrant Analysis</button> |
| <button class="tab-button" data-tab="analysis">Visual Analysis</button> |
| </div> |
|
|
| <div class="tab-content active" id="tab-models"> |
| <section class="data-notes"> |
| <h3>Data Notes</h3> |
| <ul> |
| <li><strong>Cost Calculation:</strong> Input and output pricing shown separately (per million tokens)</li> |
| <li><strong>Data Exclusions:</strong> Free models (cost = $0) were excluded from this analysis</li> |
| <li><strong>Color Coding:</strong> Cells are color-coded by tier - see legend below for ranges</li> |
| </ul> |
| </section> |
|
|
| <section class="data-table-section"> |
| <h2>All Models - Comprehensive View</h2> |
| <p>Complete dataset with color-coded cost and context tiers. Click column headers to sort.</p> |
|
|
| <div class="tier-legends"> |
| <div class="legend-group"> |
| <h4>Cost Tiers ($/M tokens)</h4> |
| <div class="legend-items"> |
| <div class="legend-item"><span class="tier-indicator cost-very-low"></span> Very Low (< $0.10)</div> |
| <div class="legend-item"><span class="tier-indicator cost-low"></span> Low ($0.10 - $0.50)</div> |
| <div class="legend-item"><span class="tier-indicator cost-medium"></span> Medium ($0.50 - $2.00)</div> |
| <div class="legend-item"><span class="tier-indicator cost-high"></span> High ($2.00 - $5.00)</div> |
| <div class="legend-item"><span class="tier-indicator cost-very-high"></span> Very High (> $5.00)</div> |
| </div> |
| </div> |
| <div class="legend-group"> |
| <h4>Context Window Tiers</h4> |
| <div class="legend-items"> |
| <div class="legend-item"><span class="tier-indicator context-small"></span> Small (< 32K)</div> |
| <div class="legend-item"><span class="tier-indicator context-medium"></span> Medium (32K - 128K)</div> |
| <div class="legend-item"><span class="tier-indicator context-large"></span> Large (128K - 256K)</div> |
| <div class="legend-item"><span class="tier-indicator context-very-large"></span> Very Large (256K - 1M)</div> |
| <div class="legend-item"><span class="tier-indicator context-ultra"></span> Ultra (> 1M)</div> |
| </div> |
| </div> |
| </div> |
| <div class="table-controls"> |
| <input type="text" id="search" placeholder="Search models..." /> |
| <select id="quadrant-filter"> |
| <option value="">All Quadrants</option> |
| <option value="Low Cost / High Context">Low Cost / High Context</option> |
| <option value="High Cost / High Context">High Cost / High Context</option> |
| <option value="Low Cost / Low Context">Low Cost / Low Context</option> |
| <option value="High Cost / Low Context">High Cost / Low Context</option> |
| </select> |
| </div> |
| <div class="table-wrapper"> |
| <table id="models-table"> |
| <thead> |
| <tr> |
| <th class="sortable" data-sort="model_name">Model Name <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="displayVendor">Vendor <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="context_length">Context <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="input_price_usd_per_m">Input ($/M) <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="output_price_usd_per_m">Output ($/M) <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="output_input_multiple">Out/In Multiple <span class="sort-indicator"></span></th> |
| <th class="sortable" data-sort="quadrant">Quadrant <span class="sort-indicator"></span></th> |
| </tr> |
| </thead> |
| <tbody id="table-body"> |
| |
| </tbody> |
| </table> |
| </div> |
| </section> |
| </div> |
|
|
| <div class="tab-content" id="tab-quadrants"> |
| <section class="data-notes"> |
| <h3>Quadrant Analysis</h3> |
| <ul> |
| <li><strong>Overview:</strong> Models divided into four quadrants based on median cost and context values</li> |
| <li><strong>Green (Low Cost / High Context):</strong> Best value - low prices with large context windows</li> |
| <li><strong>Blue (High Cost / High Context):</strong> Premium models with extensive context capabilities</li> |
| <li><strong>Orange (Low Cost / Low Context):</strong> Budget-friendly for simple tasks</li> |
| <li><strong>Red (High Cost / Low Context):</strong> Specialized models with limited context</li> |
| </ul> |
| </section> |
|
|
| <section class="chart-section"> |
| <h2>All Quadrants Overview</h2> |
| <p>Complete view showing all models across quadrants with median division lines.</p> |
| <div class="chart-container"> |
| <canvas id="allQuadrantsChart"></canvas> |
| </div> |
| </section> |
|
|
| <section class="quadrant-details"> |
| <h2>Individual Quadrant Deep Dive</h2> |
| <p>Detailed view of each quadrant's models for easier comparison within each category.</p> |
|
|
| <div class="quadrant-grid"> |
| <div class="quadrant-detail"> |
| <h3 style="color: #10b981;">Low Cost / High Context</h3> |
| <div class="chart-container-small"> |
| <canvas id="quadrant-lchc"></canvas> |
| </div> |
| </div> |
|
|
| <div class="quadrant-detail"> |
| <h3 style="color: #2563eb;">High Cost / High Context</h3> |
| <div class="chart-container-small"> |
| <canvas id="quadrant-hchc"></canvas> |
| </div> |
| </div> |
|
|
| <div class="quadrant-detail"> |
| <h3 style="color: #f59e0b;">Low Cost / Low Context</h3> |
| <div class="chart-container-small"> |
| <canvas id="quadrant-lclc"></canvas> |
| </div> |
| </div> |
|
|
| <div class="quadrant-detail"> |
| <h3 style="color: #ef4444;">High Cost / Low Context</h3> |
| <div class="chart-container-small"> |
| <canvas id="quadrant-hclc"></canvas> |
| </div> |
| </div> |
| </div> |
| </section> |
| </div> |
|
|
| <div class="tab-content" id="tab-analysis"> |
| <section class="data-notes"> |
| <h3>Analysis Notes</h3> |
| <ul> |
| <li><strong>Quadrant Divisions:</strong> Based on median values for cost and context window size</li> |
| <li><strong>Logarithmic Scale:</strong> Cost axis uses log scale to better visualize the wide price range</li> |
| <li><strong>Median Lines:</strong> Dashed lines show median cost and context values across all models</li> |
| </ul> |
| </section> |
|
|
| <section class="chart-section"> |
| <h2>Cost vs Context Quadrant Analysis</h2> |
| <p>Interactive scatter plot showing the relationship between pricing and context window size. Hover over points for detailed model information.</p> |
| <div class="chart-container"> |
| <canvas id="quadrantChart"></canvas> |
| </div> |
| </section> |
| </div> |
| </main> |
|
|
| <footer> |
| <p>Data source: <a href="https://openrouter.ai/" target="_blank">OpenRouter API</a> | Analysis: <a href="https://github.com/danielrosehill/OR-Models-With-Tools-0811" target="_blank">GitHub Repository</a></p> |
| <p class="disclaimer">Prices are subject to change. For current pricing, always refer to OpenRouter's live API.</p> |
| </footer> |
|
|
| <script src="app.js"></script> |
| </body> |
| </html> |
|
|
