references / epoch-llm-price-trends
LLM inference prices have fallen rapidly but unequally across tasks
epoch.ai (live) · exact snapshot · saved 2026-08-19 · sha256 5fcfff254a52…
The highlighted passages are the ones this site cites, shown in the context of the archived copy.
LLM inference prices have fallen rapidly but unequally across tasks | Epoch AI Latest Our work Products Trends & statistics Data explorers Capabilities & benchmarking Publications Papers & reports Data Insights Newsletter Podcast All publications Data explorers Capabilities & benchmarking Models Data centers Chip owners Companies Polling on AI use All data explorers Our benchmarks Epoch Capabilities Index MirrorCode FrontierMath: Tiers 1-4 FrontierMath: Open problems Topics AI progress Scaling Software progress Open models Capabilities & benchmarks Math Industry Leading companies Finances Geopolitics Infrastructure Chips Data centers Energy Impacts Adoption & use Economic impact Future of AI All topics About Who we are About Epoch AI Team Transparency Careers Engage with us Donate Consultations For press Contact Latest Our work Products Trends & statistics Headline numbers on how fast AI is advancing Data explorers Interactive databases on models, chips, and more Capabilities & benchmarking Evaluating the performance of AI models Publications Papers & reports Data Insights Newsletter Podcast All publications Data explorers Capabilities & benchmarking Models Data centers Chip owners Companies All data explorers Our benchmarks Epoch Capabilities Index MirrorCode FrontierMath: Tiers 1-4 FrontierMath: Open problems Navigate by topic AI progress Scaling Software progress Open models Capabilities & benchmarks Math Industry Leading companies Finances Geopolitics Infrastructure Chips Data centers Energy Impacts Adoption & use Economic impact Future of AI About Who we are About Epoch AI Team Transparency Careers Engage with us Donate Consultations For press Contact Data Insights LLM inference prices have fallen rapidly but unequally across tasks Data Insight Mar. 12, 2025 LLM inference prices have fallen rapidly but unequally across tasks Cite By Ben Cottier , Ben Snodin, David Owen , and Tom Adamczewski The inference price of LLMs has fallen dramatically in recent years. We looked at the results of state-of-the-art models on six benchmarks over the past three years, and then measured how quickly the price to achieve those milestones has fallen. For instance, the price to achieve GPT-4's performance on a set of PhD-level science questions fell by 40x per year. The rate of decline varies dramatically depending on the performance milestone, ranging from 9x to 900x per year. The fastest price drops in that range have occurred in the past year, so it's less clear that those will persist. View All benchmarks MMLU (General knowledge) GPQA Diamond (Ph.D. level science questions) MATH-500 (Math) MATH 5 (Advanced math) HumanEval (Coding) LMSys Chatbot Arena ELO (Chatbot competition) Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. Table Lowest inference prices at fixed performance Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Inference prices of other models Lowest inference prices at fixed performance Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens Predicted log price Benchmark score MMLU GPT-3 [43.9, 100] GPT-3 2021-11-20 $60.00 60.19 43.9 MMLU GPT-3 [43.9, 100] GPT-3 2022-09-01 $20.00 8.78 43.9 MMLU GPT-3 [43.9, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.50 68.0 MMLU GPT-3 [43.9, 100] Llama-2-Chat-13B 2023-07-18 $0.56 1.01 45.0 MMLU GPT-3 [43.9, 100] Llama 2-7B 2023-12-01 $0.20 0.40 45.3 MMLU GPT-3 [43.9, 100] Llama 2-7B 2024-04-14 $0.13 0.16 45.3 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.08 71.0 MMLU GPT-3 [43.9, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.05 64.0 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.05 75.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5 2022-11-30 $20.00 10.01 64.8 MMLU GPT-3.5 [64.8, 100] GPT-3.5 Turbo 2023-03-06 $2.00 4.84 68.0 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.76 68.0 MMLU GPT-3.5 [64.8, 100] Claude-3-Haiku 2024-03-04 $0.50 0.31 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 79.0 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 71.0 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 75.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5 Turbo 2023-03-06 $2.00 2.59 68.0 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.58 68.0 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.28 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 79.0 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 71.0 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.08 75.0 MMLU GPT-4-0314 [86.0, 100] GPT-4-0314 2023-03-14 $37.50 66.21 86.0 MMLU GPT-4-0314 [86.0, 100] GPT-4 Turbo 2023-11-06 $15.00 11.00 87.0 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.63 87.0 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 2.44 86.0 MMLU GPT-4-0314 [86.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4 Turbo 2023-11-06 $15.00 30.75 87.0 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.82 87.0 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.32 88.0 MMLU GPT-4 Turbo [87.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.40 87.0 MMLU GPT-4 Turbo [87.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.35 88.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0314 2023-03-14 $37.50 53.03 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4 Turbo 2023-11-06 $15.00 4.68 50.0 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 1.49 36.0 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Haiku 2024-03-04 $0.50 1.38 33.0 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.70 39.0 GPQA Diamond GPT-4-0314 [33.0, 100] Phi 4 2024-12-13 $0.12 0.08 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4 Turbo 2023-11-06 $15.00 34.11 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 4.69 53.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 3.14 56.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 2.22 50.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.15 61.0 GPQA Diamond GPT-4 Turbo [50.0, 100] Phi 4 2024-12-13 $0.12 0.49 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 11.09 53.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 5.50 56.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 0.93 61.0 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Phi 4 2024-12-13 $0.12 0.21 53.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 7.19 56.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.68 61.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] DeepSeek-V3 2024-12-26 $0.48 0.41 57.0 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 0.22 62.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.66 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.84 44.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.19 55.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.2-Instruct-3B 2024-09-24 $0.08 0.06 50.0 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.06 70.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4 Turbo 2023-11-06 $15.00 21.37 74.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.49 79.0 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.59 79.0 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 0.27 74.0 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 83.0 MATH-500 GPT-4 Turbo [74.0, 100] Phi 4 2024-12-13 $0.12 0.07 81.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 79.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 83.0 MATH-500 GPT-4o-2024-05 [79.0, 100] Phi 4 2024-12-13 $0.12 0.06 81.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0613 2023-06-13 $37.50 37.05 23.0 MATH 5 GPT-4-0613 [23.0, 100] GPT-4 Turbo 2023-11-06 $15.00 6.67 36.0 MATH 5 GPT-4-0613 [23.0, 100] Mistral-8x22 2024-04-17 $1.20 0.98 23.0 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.75 23.0 MATH 5 GPT-4-0613 [23.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4 Turbo 2023-11-06 $15.00 23.91 36.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 1.73 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.50 41.0 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.69 48.0 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.27 58.0 MATH 5 GPT-4 Turbo [36.0, 100] Phi 4 2024-12-13 $0.12 0.09 65.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 2.68 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-mini 2024-07-18 $0.26 0.82 48.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.24 58.0 MATH 5 GPT-4o-2024-05 [48.0, 100] Phi 4 2024-12-13 $0.12 0.06 65.0 HumanEval GPT-4-0314 [67.0, 100] GPT-4-0314 2023-03-14 $37.50 15.72 67.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 6.40 71.0 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 1.51 71.0 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.47 77.0 HumanEval GPT-4-0314 [67.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.15 76.0 HumanEval GPT-4-0314 [67.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.12 71.0 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.12 67.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 2.42 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.99 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Haiku 2024-03-04 $0.50 0.48 77.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemma-2-27B 2024-06-24 $0.26 0.24 76.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-NeMo 2024-07-18 $0.13 0.21 71.0 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Phi 4 2024-12-13 $0.12 0.09 87.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5 Turbo 2023-03-06 $2.00 2.53 1,106.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 0.55 1,107.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Haiku 2024-03-04 $0.50 0.27 1,179.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-8B 2024-04-18 $0.15 0.20 1,152.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.18 1,227.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-8B 2024-07-23 $0.10 0.11 1,172.0 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.07 1,211.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0314 2023-03-14 $37.50 87.40 1,186.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4 Turbo 2023-11-06 $15.00 5.28 1,256.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 1.29 1,201.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 0.76 1,206.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 0.58 1,227.0 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-8B 2024-10-03 $0.07 0.10 1,211.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4 Turbo 2023-11-06 $15.00 28.13 1,256.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 1.63 1,285.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 1.40 1,260.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-mini 2024-07-18 $0.26 0.60 1,273.0 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 0.22 1,271.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 10.85 1,285.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 3.25 1,337.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 1.62 1,301.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] DeepSeek-V3 2024-12-26 $0.48 0.43 1,318.0 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 0.24 1,358.0 Inference prices of other models Benchmark Threshold model Performance range Model Name Release Date USD per 1M Tokens MMLU GPT-3 [43.9, 100] GPT-3.5 2022-11-30 $20.00 MMLU GPT-3 [43.9, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2023-11 2023-11-06 $0.75 MMLU GPT-3 [43.9, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3 [43.9, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3 [43.9, 100] Llama 2-13B 2023-12-08 $0.35 MMLU GPT-3 [43.9, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3 [43.9, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3 [43.9, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3 [43.9, 100] Claude-3-Haiku 2024-03-04 $0.50 MMLU GPT-3 [43.9, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3 [43.9, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3 [43.9, 100] Llama 2-13B 2024-04-14 $0.22 MMLU GPT-3 [43.9, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3 [43.9, 100] Llama-3-Instruct-8B 2024-04-18 $0.15 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-05 2024-05-10 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3 [43.9, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3 [43.9, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3 [43.9, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3 [43.9, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3 [43.9, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3 [43.9, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3 [43.9, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3 [43.9, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3 [43.9, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3 [43.9, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3 [43.9, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3 [43.9, 100] Mistral 7B 2024-09-14 $0.25 MMLU GPT-3 [43.9, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3 [43.9, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3 [43.9, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3 [43.9, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3 [43.9, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3 [43.9, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3 [43.9, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3 [43.9, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3 [43.9, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3 [43.9, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 [64.8, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 [64.8, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 [64.8, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 [64.8, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 [64.8, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 [64.8, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 [64.8, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 [64.8, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 [64.8, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 [64.8, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 [64.8, 100] Mistral-NeMo 2024-07-18 $0.13 MMLU GPT-3.5 [64.8, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 [64.8, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 [64.8, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 [64.8, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 [64.8, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 [64.8, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 [64.8, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 [64.8, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 [64.8, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 [64.8, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 [64.8, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 [64.8, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 [64.8, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 [64.8, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 [64.8, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4-0314 2023-03-14 $37.50 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MMLU GPT-3.5 Turbo [68.0, 100] Llama 2-70B Chat 2023-12-01 $0.76 MMLU GPT-3.5 Turbo [68.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3-Opus 2024-03-04 $30.00 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-8x22 2024-04-17 $1.20 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-9B 2024-06-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Gemma-2-27B 2024-06-24 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-mini 2024-07-18 $0.26 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MMLU GPT-3.5 Turbo [68.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Claude 2 2024-08-12 $12.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-08-13 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MMLU GPT-3.5 Turbo [68.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MMLU GPT-3.5 Turbo [68.0, 100] Command R+ 2024-09-13 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Reka Core 2024-09-14 $4.00 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-3.5 Turbo [68.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-3.5 Turbo [68.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MMLU GPT-3.5 Turbo [68.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MMLU GPT-3.5 Turbo [68.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-3.5 Turbo [68.0, 100] Phi 4 2024-12-13 $0.12 MMLU GPT-3.5 Turbo [68.0, 100] Mistral Small 3 2025-01-30 $0.48 MMLU GPT-3.5 Turbo [68.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MMLU GPT-4-0314 [86.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4-0314 [86.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MMLU GPT-4-0314 [86.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MMLU GPT-4-0314 [86.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MMLU GPT-4 Turbo [87.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MMLU GPT-4 Turbo [87.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4-0125 2024-01-25 $15.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-8x22 2024-04-17 $1.20 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemma-2-27B 2024-06-24 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-NeMo 2024-07-18 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-mini 2024-07-18 $0.26 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 GPQA Diamond GPT-4-0314 [33.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 1.0 Pro 2024-08-12 $0.75 GPQA Diamond GPT-4-0314 [33.0, 100] Claude 2.1 2024-08-12 $12.00 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-08-13 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 GPQA Diamond GPT-4-0314 [33.0, 100] Command R+ 2024-09-13 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4-0314 [33.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 GPQA Diamond GPT-4-0314 [33.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4-0314 [33.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Mistral Small 3 2025-01-30 $0.48 GPQA Diamond GPT-4-0314 [33.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3-Opus 2024-03-04 $30.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4 Turbo [50.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4 Turbo [50.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4 Turbo [50.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 GPQA Diamond GPT-4o-2024-05 [53.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 GPQA Diamond GPT-4o-2024-05 [53.0, 100] DeepSeek-V3 2024-12-26 $0.48 GPQA Diamond GPT-4o-2024-05 [53.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 GPQA Diamond Claude-3.5-Sonnet-2024-06 [56.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4 Turbo 2023-11-06 $15.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-8x22 2024-04-17 $1.20 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3-Instruct-70B 2024-04-18 $0.89 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-9B 2024-06-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-NeMo 2024-07-18 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Jamba 1.5-Large 2024-09-12 $3.50 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Reka Core 2024-09-14 $4.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Phi 4 2024-12-13 $0.12 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-3.5-Turbo-2023-06 [44.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4 Turbo [74.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH-500 GPT-4 Turbo [74.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH-500 GPT-4 Turbo [74.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH-500 GPT-4 Turbo [74.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH-500 GPT-4o-2024-05 [79.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH-500 GPT-4o-2024-05 [79.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH-500 GPT-4o-2024-05 [79.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4-0613 [23.0, 100] GPT-4-0125 2024-01-25 $15.00 MATH 5 GPT-4-0613 [23.0, 100] Claude-3-Opus 2024-03-04 $30.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4-0613 [23.0, 100] Gemma-2-27B 2024-06-24 $0.26 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-mini 2024-07-18 $0.26 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4-0613 [23.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4-0613 [23.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4-0613 [23.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 MATH 5 GPT-4-0613 [23.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4-0613 [23.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4-0613 [23.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4-0613 [23.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 MATH 5 GPT-4 Turbo [36.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 MATH 5 GPT-4 Turbo [36.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4 Turbo [36.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4 Turbo [36.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4 Turbo [36.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Mistral Small 3 2025-01-30 $0.48 MATH 5 GPT-4 Turbo [36.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 MATH 5 GPT-4o-2024-05 [48.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 MATH 5 GPT-4o-2024-05 [48.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 MATH 5 GPT-4o-2024-05 [48.0, 100] DeepSeek-V3 2024-12-26 $0.48 MATH 5 GPT-4o-2024-05 [48.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-4-0314 [67.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-4-0314 [67.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2024-02 2024-02-26 $6.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-4-0314 [67.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-4-0314 [67.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-4-0314 [67.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-4-0314 [67.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-4-0314 [67.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-4-0314 [67.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-4-0314 [67.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-4-0314 [67.0, 100] Phi 4 2024-12-13 $0.12 HumanEval GPT-4-0314 [67.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-4-0314 [67.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-4-0314 [67.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4 Turbo 2023-11-06 $15.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Small-2024-02 2024-02-26 $1.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Sonnet 2024-03-04 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3-Opus 2024-03-04 $30.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-8x22 2024-04-17 $1.20 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-05 2024-05-13 $7.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-mini 2024-07-18 $0.26 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-70B 2024-07-23 $0.72 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Llama-3.1-Instruct-405B 2024-07-23 $3.50 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-07 2024-07-24 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-08 2024-08-06 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Reka Core 2024-09-14 $4.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Claude-3.5-Haiku 2024-10-22 $1.60 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral-Large-2-2024-11 2024-11-18 $3.00 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] GPT-4o-2024-11 2024-11-20 $4.38 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] DeepSeek-V3 2024-12-26 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Mistral Small 3 2025-01-30 $0.48 HumanEval GPT-3.5-Turbo-2023-06 [71.0, 100] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0314 2023-03-14 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0613 2023-06-13 $37.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2023-06 2023-06-13 $3.25 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4 Turbo 2023-11-06 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-3.5-Turbo-2024-01 2024-01-25 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2024-02 2024-02-26 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Sonnet 2024-03-04 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-8x22 2024-04-17 $1.20 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3-Instruct-70B 2024-04-18 $0.89 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 1.0 Pro 2024-08-12 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R 2024-08-13 $0.75 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-Coder-V2 236B 2024-09-11 $0.18 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-3.5 Turbo [1106.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4-0125 2024-01-25 $15.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3-Opus 2024-03-04 $30.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-05 2024-05-13 $7.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-05 2024-05-23 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-9B 2024-06-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemma-2-27B 2024-06-24 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-mini 2024-07-18 $0.26 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-70B 2024-07-23 $0.72 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-07 2024-07-24 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-08-13 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Jamba 1.5-Large 2024-09-12 $3.50 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Command R+ 2024-09-13 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Reka Core 2024-09-14 $4.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini-1.5-Flash-2024-09 2024-09-24 $0.13 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Claude-3.5-Haiku 2024-10-22 $1.60 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral-Large-2-2024-11 2024-11-18 $3.00 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Phi 4 2024-12-13 $0.12 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Mistral Small 3 2025-01-30 $0.48 LMSys Chatbot Arena ELO GPT-4-0314 [1186.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-06 2024-06-20 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Llama-3.1-Instruct-405B 2024-07-23 $3.50 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-08 2024-08-06 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini-1.5-Pro-2024-09 2024-09-24 $2.19 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Claude-3.5-Sonnet-2024-10 2024-10-22 $6.00 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] DeepSeek-V3 2024-12-26 $0.48 LMSys Chatbot Arena ELO GPT-4 Turbo [1256.0, inf] Gemini 2.0 Flash 2025-02-05 $0.18 LMSys Chatbot Arena ELO GPT-4o-2024-05 [1285.0, inf] GPT-4o-2024-11 2024-11-20 $4.38 Enable JavaScript to see an interactive visualization. There are many potential reasons for these price drops. Some are well-known, such as models becoming smaller and hardware becoming more cost-effective , while other important factors might be difficult to determine from public information. Epoch's work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons BY license. Learn more about this graph The dataset for this insight combines data on large language model (LLM) API prices and benchmark scores from Artificial Analysis and Epoch AI. We used this dataset to identify the lowest-priced LLMs that match or exceed a given score on a benchmark. We then fit a log-linear regression model to the prices of these LLMs over time, to measure the rate of decrease in price. We applied the same method to several benchmarks (e.g. MMLU, HumanEval) and performance thresholds (e.g. GPT-3.5 level, GPT-4o level) to determine the variation across performance metrics. Note that while the data insight provides some commentary on what factors drive these price drops, we did not explicitly model these factors. Reduced profit margins may explain some of the drops in price, but we didn't find clear evidence for this. Code to reproduce our analysis can be found here . Data The data on LLM API pricing comes from both Artificial Analysis and our own database of API prices. Our data contributed prices for GPT-3, GPT-3.5 and Llama 2 models, because prices for those models were not publicly available from Artificial Analysis. These prices made up 9 of the 36 unique observations shown in this data insight. If the price of an LLM changed after its initial release, then we recorded that as a unique observation with the same model name, but a new price and "release" date. Where prices differed by their maximum allowed context length, we only included the prices for the shortest context. We aggregated prices over different LLM API providers and token types according to Artificial Analysis' methodology, which is as follows. All prices were a 3:1 weighted average of input and output token prices. If the LLM had a first-party API, (e.g. OpenAI for o1), then we used the prices from that API. If a first-party API was not available (e.g. Meta's Llama models), then we used the median of prices across providers. We excluded reasoning models from our analysis of per-token prices. Reasoning models tend to generate a much larger number of tokens than other models, making these models cost more in total to evaluate on a benchmark. This makes it misleading to compare reasoning models to other models on price per token, at a given performance level. The benchmark data covers 6 benchmarks: GPQA Diamond (PhD-level science questions), MMLU (general knowledge), MATH-500 (math problems), MATH level 5 (advanced math problems), HumanEval (coding), and Chatbot Arena Elo (head-to-head comparisons of chatbots, with human judges). Like the price data, the benchmark data is sourced from both Artificial Analysis and our benchmark database . In the combined dataset, most of the benchmark scores are as reported by Artificial Analysis. The methodology to obtain those scores is described by Artificial Analysis (except we use MMLU rather than MMLU-Pro). We used data from our own evaluations for MATH level 5 (a benchmark not reported by Artificial Analysis), and for cases where Artificial Analysis did not have scores for GPQA Diamond. For other benchmarks where Artificial Analysis had missing scores, we used scores from Papers with Code or scores reported by the developer of the model. Our methodology for MATH level 5 and GPQA Diamond evaluations is described here . For the GPT-4-0314 model, direct evaluation on GPQA Diamond was unavailable. However, we know that GPT-4-0613 performed similarly to GPT-4-0314 across many other benchmarks (see here , p.5). Based on that, we assumed that GPT-4-0314 had the same GPQA Diamond score as GPT-4-0613. Analysis To analyze the decline in LLM prices over time, we focused on the most cost-effective LLMs above a certain performance threshold at each point in time. To identify these models, we iterated through models sorted by release date. In each iteration, we added a model to the set of cheapest models if it had a lower price than all previous models that scored at or above the threshold. After identifying the cheapest models, we fit a log-linear regression to price vs. release date. We used a log-linear model because it had a much lower Bayesian Information Criterion than a linear model. The slope of the log-linear model corresponds to a price reduction over time, e.g. 40x per year. We set a threshold of at least 4 data points to include a regression result. We applied the above method to all of the benchmarks and performance thresholds. For each benchmark, we identified instances where a new model achieved a better performance than all previous models, and used these instances as our performance thresholds. Across all of these benchmarks and performance thresholds, we found prices declining between 9x per year and 900x per year, with a median of 50x per year. Note that the "mid-range" value in the figure is close to the median of the trends that are shown, but is not always the exact median, for the sake of visual clarity. In addition to measuring the rates that prices have fallen, we analyzed how the rate relates to the time period it occurred in. We found that the fastest trends (e.g. 900x per year) start after January 2024. When we removed all model data before January 2024 (after selecting the lowest-price models), the rates also increased overall—even at performance levels that were achieved before 2024. In particular, the median rate increased from 50x per year to 200x per year. This suggests that the fastest trends are caused by a recent acceleration in price drops across performance levels and benchmarks. Assumptions There is the potential for significant variation in the number of tokens that models generate to achieve their benchmark score, which could mean that trends in price per token differ from trends in evaluation cost. However, our investigations found that evaluation costs have declined similarly to prices per token. We used Epoch AI benchmark evaluation results for this analysis, as these report the number of tokens processed for each evaluation. For the six cost trends we estimated, the largest difference was 200x per year for evaluation cost vs. 400x per year for price per token, for Claude-3.5-Sonnet-2024-06 level performance on GPQA Diamond. We relied on benchmarks to measure an AI model's performance, which leads to several limitations. Firstly, benchmarks do not comprehensively measure a model's usefulness: there are important capabilities besides knowledge and reasoning, and other important features like output speed. Another limitation of benchmark scores is that AI models can be overfit to these benchmarks. AI developers may accidentally train models on benchmark data, or use knowledge about the benchmark to inform training (which seems to have become more common since 2024). These practices may drive up benchmark scores without generalizing. Benchmark evaluations can also differ due to randomness in model outputs and variations in prompting. There were 33 evaluations on GPQA Diamond that overlapped between Artificial Analysis and our benchmark database. The mean absolute difference between scores from the two evaluations was 2.9 percentage points, with a standard deviation of 1.7 and an average signed difference of 0.1. This is within expectations, since the standard error on our GPQA Diamond scores is generally around 3 percentage points, and variance is further increased by subtle differences in prompting and scoring. We also relied on our dataset having good enough model coverage to accurately measure trends in prices. There may be models not in our dataset that were more cost-effective than the "cheapest models" we selected. However, it's unlikely that such models would greatly alter the observed trends, since the cheapest models at a given performance level are more likely to be well-known, and therefore captured in our dataset. Download this data Lowest inference prices at fixed performance CSV , Updated Nov. 20, 2025 Inference prices of other models CSV , Updated Nov. 20, 2025 Explore this data AI Capabilities Benchmark results featuring the performance of leading AI models on challenging tasks. Related topics Finances Scaling Cite Epoch AI's work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons Attribution license . Citation Ben Cottier et al. (2025), LLM inference prices have fallen rapidly but unequally across tasks . Published online at epoch.ai. Retrieved from 'https://epoch.ai/data-insights/llm-inference-price-trends' [online resource]. Accessed 19 Aug 2026. BibTeX Citation @misc{epoch2025llminferencepricetrends, title={LLM inference prices have fallen rapidly but unequally across tasks}, author={Ben Cottier and Ben Snodin and David Owen and Tom Adamczewski}, year={2025}, url={https://epoch.ai/data-insights/llm-inference-price-trends}, note={Accessed: 2026-08-19}} Investigating the trajectory of AI for the benefit of society Get the latest from Epoch AI in your inbox Subscribe Our Work Latest AI Trends & Statistics Publications Data Insights Newsletter: Gradient Updates Podcast Data on AI Models AI Data Centers Hardware Companies Chip Sales Polling Benchmarking AI Capabilities FrontierMath Company About Team Consultations Transparency Donate Careers © 2026 Epoch AI Privacy Policy Cookie Policy Designed by And—Now Feedback Feedback Have a question? Noticed something wrong? Let us know. Message If you would like a reply, please include your name and email address. Name Email address Cancel Submit LLM inference prices have fallen rapidly but unequally across tasks Epoch AI is a research institute investigating key trends and questions that will shape the trajectory and governance of Artificial Intelligence.