crm-results/hf_leaderboard_latency_cost.csv

Model Name,Cost and Speed: Flavor,Version,Platform,Mean Output Tokens,Mean Cost per 1K Requests,Cost Band,Response Time (Sec)
AI21 Jamba-Instruct,Long,,AI21,232.9,1.6,Medium,4.0
AI21 Jamba-Instruct,Short,,AI21,243.9,0.5,Low,4.0
Claude 3 Haiku,Long,,Bedrock,236.9,1.0,Medium,2.8
Claude 3 Haiku,Short,,Bedrock,245.4,0.4,Low,2.2
Claude 3 Opus,Long,,Bedrock,242.7,61.1,High,12.2
Claude 3 Opus,Short,,Bedrock,243.2,25.4,High,8.4
Cohere Command R+,Long,,Bedrock,245.7,11.7,High,7.7
Cohere Command R+,Short,,Bedrock,249.9,5.1,High,7.1
Cohere Command Text,Long,,Bedrock,238.7,4.3,High,12.9
Cohere Command Text,Short,,Bedrock,245.6,1.1,Medium,9.6
Gemini Pro 1.5,Long,,Google,245.7,11.0,High,5.5
Gemini Pro 1.5,Short,,Google,247.5,3.3,Medium,5.4
Gemini Pro 1,Long,,Google,228.9,1.7,Medium,6.0
Gemini Pro 1,Short,,Google,247.4,0.6,Low,4.4
GPT 3.5 Turbo,Long,,OpenAI,249.9,1.6,Medium,4.5
GPT 3.5 Turbo,Short,,OpenAI,238.3,0.6,Low,4.2
GPT 4 Turbo,Long,,OpenAI,247.6,32.0,High,12.3
GPT 4 Turbo,Short,,OpenAI,250.0,11.7,High,12.3
GPT4-o,Long,,OpenAI,248.4,15.9,High,5.1
GPT4-o,Short,,OpenAI,250.0,5.8,High,5.0
Mistral 7B,Long,Mistral-7B-Instruct-v0.2,Self-host (g5.48xlarge),242.0,16.5,Low,3.74
Mistral 7B,Short,Mistral-7B-Instruct-v0.2,Self-host (g5.48xlarge),247.0,15.5,Low,3.68
LLaMA 3 8B,Long,Meta-Llama-3-8B-Instruct,Self-host (g5.48xlarge),251.5,7.0,Medium,3.42
LLaMA 3 8B,Short,Meta-Llama-3-8B-Instruct,Self-host (g5.48xlarge),243.6,6.0,Low,3.33
LLaMA 3 70B,Long,llama-3-70b-instruct,Self-host (p4d.24xlarge),243.9,67.7,High,6.78
LLaMA 3 70B,Short,llama-3-70b-instruct,Self-host (p4d.24xlarge),251.2,99.0,Medium,6.25
Mixtral 8x7B,Long,mixtral-8x7b-instruct,Self-host (p4d.24xlarge),248.5,8.22,Medium,4.90
Mixtral 8x7B,Short,mixtral-8x7b-instruct,Self-host (p4d.24xlarge),250.0,8.11,Low,4.54
SF-TextBase 7B,Long,CRM-TextBase-7b-22k-g5 (endpoint),Self-host (g5.48xlarge),248.5,16.80,Low,3.50
SF-TextBase 7B,Short,CRM-TextBase-7b-22k-g5 (endpoint),Self-host (g5.48xlarge),248.7,15.50,Low,3.45
SF-TextBase 70B,Long,TextBase-70B-8K,Self-host (p4de.24xlarge),253.7,28.17,High,7.76
SF-TextBase 70B,Short,TextBase-70B-8K,Self-host (p4de.24xlarge),249.7,26.96,Medium,7.48
SF-TextSum,Long,CRM-TSUM-7b-22k-g5 (endpoint),Self-host (g5.48xlarge),244.0,16.55,Low,3.43
SF-TextSum,Short,CRM-TSUM-7b-22k-g5 (endpoint),Self-host (g5.48xlarge),250.4,15.60,Low,3.38
XGen 2,Long,EinsteinXgen2E4DSStreaming (endpoint),Self-host (p4de.24xlarge),250.0,16.03,Medium,5.04
XGen 2,Short,EinsteinXgen2E4DSStreaming (endpoint),Self-host (p4de.24xlarge),250.0,11.40,Medium,4.66