Chatbot Arena

This leaderboard is based on the following three benchmarks.

• Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 1.8M+ user votes to compute Elo ratings.
• MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
• MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best for Model Size Class

Model	▤ Size	Arena Elo	MMLU	Context Window	License
GPT-4o-2024-08-08	72B +	1316	88.7	128K	Proprietary
Llama-3.1-405b-Instruct	72B +	1266	88.6	128K	Llama 3.1 Community
Llama-3.1-70b-Instruct	40B - 72B	1249	86	128K	Llama 3.1 Community
Gemma-2-27b-it	24B - 40B	1218	75.2	8K	Gemma license
Llama-3.1-8b-Instruct	4B - 24B	1168	73	128K	Llama 3.1 Community
Phi-3-Mini-4k-Instruct-June-24	1B - 4B	1071	70.9	4K	MIT

Full Leaderboard

Model	🏆 Arena Elo	Coding Elo	MT-bench	MMLU	Votes	Organization	License
🥇 GPT-4o-2024-08-08	1316	1322			31148	OpenAI	Proprietary
🥈 Gemini-1.5-Pro-Exp-0827	1300	1291			22844	Google	Proprietary
🥈 Grok-2-08-13	1294	1284			16215	xAI	Proprietary
🥉 GPT-4o-mini-2024-07-18	1274	1281		82	26088	OpenAI	Proprietary
🥉 Claude 3.5 Sonnet	1270	1297		88.7	56674	Anthropic	Proprietary
🥉 Gemini-1.5-Flash-Exp-0827	1268	1253			16780	Google	Proprietary
🥉 Grok-2-Mini-08-13	1267	1258			16731	xAI	Proprietary
🥉 Llama-3.1-405b-Instruct	1266	1278		88.6	27397	Meta	Llama 3.1 Community
Gemini-1.5-Pro-0514	1259	1263		85.9	76952	Google	Proprietary
GPT-4-Turbo-2024-04-09	1257	1264			90149	OpenAI	Proprietary
Gemini-1.5-Pro-0409	1257	1231		81.9	55597	Google	Proprietary
Mistral-Large-2407	1251	1271			20734	Mistral	Mistral Research
GPT-4-1106-preview	1251	1255	9.32		95438	OpenAI	Proprietary
Llama-3.1-70b-Instruct	1249	1251		86	22161	Meta	Llama 3.1 Community
Athene-70b	1249	1249			17254	NexusFlow	CC-BY-NC-4.0
Claude 3 Opus	1248	1250		86.8	161782	Anthropic	Proprietary
GPT-4-0125-preview	1245	1245			88711	OpenAI	Proprietary
Yi-Large-preview	1240	1245			51715	01 AI	Proprietary
Reka-Core-20240722	1231	1208			12638	Reka AI	Proprietary
Gemini-1.5-Flash-001	1227	1232		78.9	62135	Google	Proprietary
Deepseek-v2-API-0628	1219	1241			19438	DeepSeek AI	DeepSeek
Jamba-1.5-Large	1218	1223		81.2	7629	AI21 Labs	Jamba Open
Gemma-2-27b-it	1218	1206			33751	Google	Gemma license
Gemma-2-9b-it-SimPO	1214	1194			8862	Princeton	MIT
Deepseek-Coder-v2-0724	1213	1264			11396	DeepSeek	Proprietary
command-r-plus-08-2024	1213	1169			1843	Cohere	Proprietary
Yi-Large	1212	1219			16672	01 AI	Proprietary
Nemotron-4-340B-Instruct	1209	1198			20653	Nvidia	NVIDIA Open Model
Gemini App (2024-01-24)	1209	1172			11817	Google	Proprietary
GLM-4-0520	1207	1216			10232	Zhipu AI	Proprietary
Llama-3-70b-Instruct	1206	1200		82	163952	Meta	Llama 3 Community
Gemini-1.5-Flash-8b-Exp-0827	1203	1187			17382	Google	Proprietary
Claude 3 Sonnet	1201	1213		79	113053	Anthropic	Proprietary
Reka-Flash-20240722	1200	1187			13014	Reka AI	Proprietary
Reka-Core-20240501	1199	1190		83.2	62642	Reka AI	Proprietary
Command R+	1190	1165			80888	Cohere	CC-BY-NC-4.0
Gemma-2-9b-it	1188	1165			28281	Google	Gemma license
Qwen2-72B-Instruct	1187	1187	9.12	84.2	37463	Alibaba	Qianwen LICENSE
GPT-4-0314	1186	1196	8.96	86.4	55963	OpenAI	Proprietary
GLM-4-0116	1183	1191			7582	Zhipu AI	Proprietary
Qwen-Max-0428	1183	1190			25734	Alibaba	Proprietary
Claude 3 Haiku	1179	1189		75.2	112921	Anthropic	Proprietary
DeepSeek-Coder-V2-Instruct	1178	1239			15794	DeepSeek AI	DeepSeek License
Jamba-1.5-Mini	1174	1178		69.7	7731	AI21 Labs	Jamba Open
command-r-08-2024	1172	1140			1885	Cohere	Proprietary
Llama-3.1-8b-Instruct	1168	1175		73	20642	Meta	Llama 3.1 Community
Reka-Flash-Preview-20240611	1165	1155			20461	Reka AI	Proprietary
GPT-4-0613	1163	1167	9.18		91662	OpenAI	Proprietary
Qwen1.5-110B-Chat	1161	1175	8.88	80.4	27457	Alibaba	Qianwen LICENSE
Mistral-Large-2402	1157	1170		81.2	64930	Mistral	Proprietary
Yi-1.5-34B-Chat	1157	1162		76.8	25161	01 AI	Apache-2.0
Reka-Flash-21B-online	1156	1147			16027	Reka AI	Proprietary
Llama-3-8b-Instruct	1152	1146		68.4	109362	Meta	Llama 3 Community
Claude-1	1149	1136	7.9	77	21163	Anthropic	Proprietary
Command R	1149	1123			56363	Cohere	CC-BY-NC-4.0
Mistral Medium	1148	1153	8.61	75.3	35545	Mistral	Proprietary
Reka-Flash-21B	1148	1141		73.5	25795	Reka AI	Proprietary
Qwen1.5-72B-Chat	1147	1160	8.61	77.5	40628	Alibaba	Qianwen LICENSE
Mixtral-8x22b-Instruct-v0.1	1147	1153		77.8	53121	Mistral	Apache 2.0
Claude-2.0	1132	1135	8.06	78.5	12766	Anthropic	Proprietary
Gemini-1.0-Pro-001	1132	1103		71.8	18781	Google	Proprietary
Gemma-2-2b-it	1132	1084		51.3	16857	Google	Gemma license
Zephyr-ORPO-141b-A35b-v0.1	1127	1124			4858	HuggingFace	Apache 2.0
Qwen1.5-32B-Chat	1125	1149	8.3	73.4	22751	Alibaba	Qianwen LICENSE
Mistral-Next	1125	1133			12379	Mistral	Proprietary
Phi-3-Medium-4k-Instruct	1123	1124		78	25396	Microsoft	MIT
Starling-LM-7B-beta	1119	1129	8.12		16653	Nexusflow	Apache-2.0
Claude-2.1	1118	1132	8.18		37685	Anthropic	Proprietary
GPT-3.5-Turbo-0613	1117	1135	8.39		38938	OpenAI	Proprietary
Mixtral-8x7b-Instruct-v0.1	1114	1114	8.3	70.6	76109	Mistral	Apache 2.0
Claude-Instant-1	1111	1109	7.85	73.4	20620	Anthropic	Proprietary
Yi-34B-Chat	1111	1106		73.5	15933	01 AI	Yi License
Gemini Pro	1111	1092		71.8	6560	Google	Proprietary
Qwen1.5-14B-Chat	1109	1126	7.91	67.6	18665	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-0314	1107	1116	7.94	70	5651	OpenAI	Proprietary
GPT-3.5-Turbo-0125	1106	1124			68869	OpenAI	Proprietary
WizardLM-70B-v1.0	1106	1071	7.71	63.7	8389	Microsoft	Llama 2 Community
DBRX-Instruct-Preview	1103	1118		73.7	33725	Databricks	DBRX LICENSE
Phi-3-Small-8k-Instruct	1102	1107		75.7	18493	Microsoft	MIT
Tulu-2-DPO-70B	1099	1093	7.89		6658	AllenAI/UW	AI2 ImpACT Low-risk
Llama-2-70b-chat	1093	1072	6.86	63	39619	Meta	Llama 2 Community
OpenChat-3.5-0106	1092	1102	7.8	65.8	12979	OpenChat	Apache-2.0
Vicuna-33B	1091	1067	7.12	59.2	22930	LMSYS	Non-commercial
Snowflake Arctic Instruct	1090	1077		67.3	34194	Snowflake	Apache 2.0
Starling-LM-7B-alpha	1088	1080	8.09	63.9	10417	UC Berkeley	CC-BY-NC-4.0
Nous-Hermes-2-Mixtral-8x7B-DPO	1085	1080			3841	NousResearch	Apache-2.0
Gemma-1.1-7b-it	1084	1085		64.3	25070	Google	Gemma license
NV-Llama2-70B-SteerLM-Chat	1081	1023	7.54	68.5	3634	Nvidia	Llama 2 Community
pplx-70b-online	1078	1028			6890	Perplexity AI	Proprietary
DeepSeek-LLM-67B-Chat	1077	1080		71.3	4980	DeepSeek AI	DeepSeek License
OpenChat-3.5	1076	1054	7.81	64.3	8112	OpenChat	Apache-2.0
OpenHermes-2.5-Mistral-7b	1075	1059			5089	NousResearch	Apache-2.0
Mistral-7B-Instruct-v0.2	1072	1074	7.6		20062	Mistral	Apache-2.0
Phi-3-Mini-4k-Instruct-June-24	1071	1081		70.9	12885	Microsoft	MIT
Qwen1.5-7B-Chat	1070	1089	7.6	61	4865	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-1106	1068	1095	8.32		17019	OpenAI	Proprietary
Phi-3-Mini-4k-Instruct	1066	1086		68.8	21108	Microsoft	MIT
Llama-2-13b-chat	1063	1051	6.65	53.6	19726	Meta	Llama 2 Community
Dolphin-2.2.1-Mistral-7B	1063	1025			1714	Cognitive Computations	Apache-2.0
SOLAR-10.7B-Instruct-v1.0	1062	1047	7.58	66.2	4290	Upstage AI	CC-BY-NC-4.0
WizardLM-13b-v1.2	1059	1027	7.2	52.7	7186	Microsoft	Llama 2 Community
Zephyr-7b-beta	1053	1030	7.34	61.4	11318	HuggingFace	MIT
MPT-30B-chat	1046	1031	6.39	50.4	2649	MosaicML	CC-BY-NC-SA-4.0
pplx-7b-online	1045	1015			6336	Perplexity AI	Proprietary
CodeLlama-70B-instruct	1043	1048			1192	Meta	Llama 2 Community
CodeLlama-34B-instruct	1043	1042		53.7	7509	Meta	Llama 2 Community
Vicuna-13B	1042	1032	6.57	55.8	19789	LMSYS	Llama 2 Community
Zephyr-7b-alpha	1041	1034	6.88		1815	HuggingFace	MIT
Gemma-7b-it	1038	1047		64.3	9176	Google	Gemma license
Phi-3-Mini-128k-Instruct	1037	1029		68.1	21602	Microsoft	MIT
Llama-2-7b-chat	1037	1003	6.27	45.8	14550	Meta	Llama 2 Community
Qwen-14B-Chat	1035	1056	6.96	66.5	5067	Alibaba	Qianwen LICENSE
falcon-180b-chat	1035	1017		68	1325	TII	Falcon-180B TII License
Guanaco-33B	1033	966	6.53	57.6	2999	UW	Non-commercial
Gemma-1.1-2b-it	1021	1036		64.3	11359	Google	Gemma license
StripedHyena-Nous-7B	1017	1000			5266	Together AI	Apache 2.0
OLMo-7B-instruct	1015	1017			6494	Allen AI	Apache-2.0
Mistral-7B-Instruct-v0.1	1008	1008	6.84	55.4	9138	Mistral	Apache 2.0
Vicuna-7B	1005	982	6.17	49.8	7017	LMSYS	Llama 2 Community
PaLM-Chat-Bison-001	1004	990	6.4		8742	Google	Proprietary
Gemma-2b-it	989	1000		42.3	4919	Google	Gemma license
Qwen1.5-4B-Chat	989	991		56.1	7803	Alibaba	Qianwen LICENSE
Koala-13B	965	938	5.35	44.7	7036	UC Berkeley	Non-commercial
ChatGLM3-6B	955	953			4761	Tsinghua	Apache-2.0
GPT4All-13B-Snoozy	932	910	5.41	43	1786	Nomic AI	Non-commercial
MPT-7B-Chat	928	900	5.42	32	4016	MosaicML	CC-BY-NC-SA-4.0
ChatGLM2-6B	924	892	4.96	45.5	2709	Tsinghua	Apache-2.0
RWKV-4-Raven-14B	922	897	3.98	25.6	4936	RWKV	Apache 2.0
Alpaca-13B	902	790	4.53	48.1	5872	Stanford	Non-commercial
OpenAssistant-Pythia-12B	894	873	4.32	27	6380	OpenAssistant	Apache 2.0
ChatGLM-6B	879	884	4.5	36.1	4993	Tsinghua	Non-commercial
FastChat-T5-3B	868	760	3.04	47.7	4303	LMSYS	Apache 2.0
StableLM-Tuned-Alpha-7B	840	858	2.75	24.4	3336	Stability AI	CC-BY-NC-SA-4.0
Dolly-V2-12B	822	746	3.28	25.7	3485	Databricks	MIT
LLaMA-13B	799	669	2.61	47	2443	Meta	Non-commercial

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of R_A and player B a rating of R_B, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.