Chatbot Arena

This leaderboard is based on the following benchmarks.

• Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 2.5M+ user votes to compute Elo ratings.
• MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
• MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.
• Arena-Hard-Auto - an automatic evaluation tool for instruction-tuned LLMs.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best Open LM

Model	Arena Elo	MMLU	License
Llama-3.3-70B-Instruct	1256	86	Llama 3.3
Qwen2.5-72B-Instruct	1257	86.8	Qwen
Deepseek-v2.5	1258	80.4	DeepSeek

Full Leaderboard

Model	Arena Elo	Coding Elo	Arena Hard	MT-bench	MMLU	Votes	Organization	License
🥇 Gemini-Exp-1206	1372	1369				13175	Google	Proprietary
🥇 Gemini-2.0-Flash-Thinking-Exp	1369	1343				5499	Google	Proprietary
🥇 ChatGPT-4o-latest (2024-11-20)	1364	1351				26458	OpenAI	Proprietary
🥇 Gemini-2.0-Flash-Exp	1355	1343				12190	Google	Proprietary
🥇 o1-preview	1335	1355	90.4			32685	OpenAI	Proprietary
🥈 o1-mini	1306	1360	92			41393	OpenAI	Proprietary
🥈 Gemini-1.5-Pro-002	1301	1289				37673	Google	Proprietary
🥈 Grok-2-08-13	1288	1284				59987	xAI	Proprietary
🥈 Yi-Lightning	1287	1303	81.5			29185	01 AI	Proprietary
🥈 Claude 3.5 Sonnet (20241022)	1283	1323	85.2		88.7	39879	Anthropic	Proprietary
🥉 Athene-v2-Chat-72B	1277	1295	85			13497	NexusFlow	NexusFlow
🥉 GLM-4-Plus	1274	1283				27995	Zhipu AI	Proprietary
🥉 GPT-4o-mini-2024-07-18	1273	1284	74.94		82	56867	OpenAI	Proprietary
🥉 Gemini-1.5-Flash-002	1271	1252				30754	Google	Proprietary
🥉 Llama-3.1-Nemotron-70B-Instruct	1269	1271	84.9			7669	Nvidia	Llama 3.1
🥉 Llama-3.1-405B-Instruct	1267	1277	69.3		88.6	62431	Meta	Llama 3.1
🥉 Grok-2-Mini-08-13	1266	1262				51515	xAI	Proprietary
🥉 Yi-Lightning-lite	1264	1267				17190	01 AI	Proprietary
🥉 Deepseek-v2.5	1258	1288				26510	DeepSeek	DeepSeek
🥉 Qwen2.5-72B-Instruct	1257	1282	78			36199	Alibaba	Qwen
🥉 GPT-4-Turbo-2024-04-09	1256	1263	82.63			102236	OpenAI	Proprietary
🥉 Llama-3.3-70B-Instruct	1256	1258				8089	Meta	Llama-3.3
Mistral-Large-2407	1251	1269	70.42			48375	Mistral	Mistral Research
Athene-70B	1250	1254	77.6			20644	NexusFlow	CC-BY-NC-4.0
GPT-4-1106-preview	1250	1253		9.32		103822	OpenAI	Proprietary
Mistral-Large-2411	1249	1266				4386	Mistral	MRL
Llama-3.1-70B-Instruct	1248	1251	55.73		86	58389	Meta	Llama 3.1
Claude 3 Opus	1248	1250	60.36		86.8	197260	Anthropic	Proprietary
GPT-4-0125-preview	1245	1244	77.96			97155	OpenAI	Proprietary
Amazon Nova Pro 1.0	1244	1260				6655	Amazon	Proprietary
Yi-Large-preview	1240	1245	71.48			51704	01 AI	Proprietary
Claude 3.5 Haiku (20241022)	1238	1265				4535	Anthropic	Propretary
Reka-Core-20240904	1235	1222				7973	Reka AI	Proprietary
Reka-Core-20240722	1231	1208				13311	Reka AI	Proprietary
Qwen-Plus-0828	1227	1245				14694	Alibaba	Proprietary
Gemini-1.5-Flash-001	1227	1232	49.61		78.9	65724	Google	Proprietary
Jamba-1.5-Large	1221	1228			81.2	9143	AI21 Labs	Jamba Open
Deepseek-v2-API-0628	1220	1242				19563	DeepSeek AI	DeepSeek
Gemma-2-27B-it	1220	1211	57.51			68050	Google	Gemma license
Amazon Nova Lite 1.0	1219	1235				6731	Amazon	Proprietary
Qwen2.5-Coder-32B-Instruct	1217	1260				5745	Alibaba	Apache 2.0
Gemma-2-9B-it-SimPO	1216	1197				10570	Princeton	MIT
Command R+ (08-2024)	1215	1181				10577	Cohere	CC-BY-NC-4.0
Deepseek-Coder-v2-0724	1214	1266	62.3			11744	DeepSeek	Proprietary
Yi-Large	1212	1220	63.7			16670	01 AI	Proprietary
Llama-3.1-Nemotron-51B-Instruct	1212	1211				3926	Nvidia	Llama 3.1
Gemini-1.5-Flash-8B-001	1212	1206				32411	Google	Proprietary
Nemotron-4-340B-Instruct	1209	1198				20648	Nvidia	NVIDIA Open Model
Aya-Expanse-32B	1208	1191				23033	Cohere	CC-BY-NC-4.0
Gemini App (2024-01-24)	1208	1171				11833	Google	Proprietary
GLM-4-0520	1207	1216	63.84			10227	Zhipu AI	Proprietary
Llama-3-70B-Instruct	1206	1200	46.57		82	163892	Meta	Llama 3
Reka-Flash-20240904	1205	1191				8158	Reka AI	Proprietary
Gemini-1.5-Flash-8B-Exp-0827	1205	1189				25413	Google	Proprietary
Claude 3 Sonnet	1201	1213	46.8		79	113102	Anthropic	Proprietary
Reka-Flash-20240722	1201	1187				13747	Reka AI	Proprietary
Reka-Core-20240501	1200	1190			83.2	62615	Reka AI	Proprietary
Amazon Nova Micro 1.0	1197	1218				6763	Amazon	Proprietary
Gemma-2-9B-it	1191	1171				46390	Google	Gemma license
Command R+ (04-2024)	1190	1164	33.07			80916	Cohere	CC-BY-NC-4.0
Hunyuan-Standard-256K	1189	1226				2928	Tencent	Proprietary
Qwen2-72B-Instruct	1187	1187	46.86	9.12	84.2	38956	Alibaba	Qianwen LICENSE
GPT-4-0314	1186	1196	50	8.96	86.4	55978	OpenAI	Proprietary
GLM-4-0116	1183	1191	55.72			7583	Zhipu AI	Proprietary
Qwen-Max-0428	1183	1190				25711	Alibaba	Proprietary
Ministral-8B-2410	1182	1201				5144	Mistral	MRL
Aya-Expanse-8B	1180	1168				4485	Cohere	CC-BY-NC-4.0
Claude 3 Haiku	1179	1190	41.47		75.2	122415	Anthropic	Proprietary
Command R (08-2024)	1179	1162				10874	Cohere	CC-BY-NC-4.0
DeepSeek-Coder-V2-Instruct	1178	1239				15782	DeepSeek AI	DeepSeek License
Llama-3.1-8B-Instruct	1176	1186	21.34		73	52288	Meta	Llama 3.1
Jamba-1.5-Mini	1176	1181			69.7	9279	AI21 Labs	Jamba Open
Reka-Flash-Preview-20240611	1165	1155				20453	Reka AI	Proprietary
GPT-4-0613	1163	1167	37.9	9.18		91645	OpenAI	Proprietary
Qwen1.5-110B-Chat	1161	1175		8.88	80.4	27470	Alibaba	Qianwen LICENSE
Mistral-Large-2402	1157	1170	37.71		81.2	64955	Mistral	Proprietary
Yi-1.5-34B-Chat	1157	1162			76.8	25157	01 AI	Apache-2.0
Reka-Flash-21B-online	1156	1147				16036	Reka AI	Proprietary
Llama-3-8B-Instruct	1152	1146	20.56		68.4	109274	Meta	Llama 3
QwQ-32B-Preview	1150	1145				2823	Alibaba	Apache 2.0
InternLM2.5-20B-chat	1149	1158				10674	InternLM	Other
Claude-1	1149	1136		7.9	77	21169	Anthropic	Proprietary
Command R (04-2024)	1149	1123	17.02			56400	Cohere	CC-BY-NC-4.0
Mistral Medium	1148	1153	31.9	8.61	75.3	35547	Mistral	Proprietary
Mixtral-8x22b-Instruct-v0.1	1148	1153	36.36		77.8	53816	Mistral	Apache 2.0
Reka-Flash-21B	1148	1141			73.5	25820	Reka AI	Proprietary
Qwen1.5-72B-Chat	1147	1160	36.12	8.61	77.5	40651	Alibaba	Qianwen LICENSE
Gemma-2-2b-it	1142	1105			51.3	37951	Google	Gemma license
Claude-2.0	1132	1135	23.99	8.06	78.5	12763	Anthropic	Proprietary
Gemini-1.0-Pro-001	1131	1103			71.8	18795	Google	Proprietary
Zephyr-ORPO-141b-A35b-v0.1	1127	1124				4861	HuggingFace	Apache 2.0
Qwen1.5-32B-Chat	1125	1149		8.3	73.4	22764	Alibaba	Qianwen LICENSE
Mistral-Next	1124	1132	27.37			12378	Mistral	Proprietary
Phi-3-Medium-4k-Instruct	1123	1125	33.37		78	26132	Microsoft	MIT
Starling-LM-7B-beta	1119	1129	23.01	8.12		16674	Nexusflow	Apache-2.0
Claude-2.1	1118	1132	22.77	8.18		37704	Anthropic	Proprietary
GPT-3.5-Turbo-0613	1117	1135	24.82	8.39		38965	OpenAI	Proprietary
Mixtral-8x7B-Instruct-v0.1	1114	1114	23.4	8.3	70.6	76148	Mistral	Apache 2.0
Claude-Instant-1	1111	1109		7.85	73.4	20627	Anthropic	Proprietary
Yi-34B-Chat	1111	1106	23.15		73.5	15930	01 AI	Yi License
Yi-34B-Chat	1111	1106	23.15		73.5	15930	01 AI	Yi License
Gemini Pro	1111	1092	17.8		71.8	6560	Google	Proprietary
Qwen1.5-14B-Chat	1109	1126		7.91	67.6	18680	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-0314	1107	1115	18.05	7.94	70	5647	OpenAI	Proprietary
GPT-3.5-Turbo-0125	1106	1124	23.34			68902	OpenAI	Proprietary
WizardLM-70B-v1.0	1106	1071		7.71	63.7	8382	Microsoft	Llama 2
DBRX-Instruct-Preview	1103	1118	24.63		73.7	33735	Databricks	DBRX LICENSE
Llama-3.2-3B-Instruct	1103	1080				8426	Meta	Llama 3.2
Phi-3-Small-8k-Instruct	1102	1107	29.77		75.7	18502	Microsoft	MIT
Tulu-2-DPO-70B	1099	1093	14.99	7.89		6662	AllenAI/UW	AI2 ImpACT Low-risk
Granite-3.0-8B-Instruct	1094	1097				7066	IBM	Apache 2.0
Llama-2-70B-chat	1093	1072	11.55	6.86	63	39645	Meta	Llama 2
OpenChat-3.5-0106	1091	1102		7.8	65.8	12986	OpenChat	Apache-2.0
Vicuna-33B	1091	1067	8.63	7.12	59.2	22954	LMSYS	Non-commercial
Snowflake Arctic Instruct	1090	1077	17.61		67.3	34183	Snowflake	Apache 2.0
Starling-LM-7B-alpha	1089	1080	12.8	8.09	63.9	10414	UC Berkeley	CC-BY-NC-4.0
Gemma-1.1-7B-it	1084	1084	12.09		64.3	25079	Google	Gemma license
Nous-Hermes-2-Mixtral-8x7B-DPO	1084	1079				3835	NousResearch	Apache-2.0
NV-Llama2-70B-SteerLM-Chat	1081	1023		7.54	68.5	3637	Nvidia	Llama 2
pplx-70B-online	1078	1028				6891	Perplexity AI	Proprietary
DeepSeek-LLM-67B-Chat	1077	1079			71.3	4986	DeepSeek AI	DeepSeek License
OpenChat-3.5	1077	1054		7.81	64.3	8110	OpenChat	Apache-2.0
Granite-3.0-2B-Instruct	1074	1088				7238	IBM	Apache 2.0
OpenHermes-2.5-Mistral-7B	1074	1058				5089	NousResearch	Apache-2.0
Mistral-7B-Instruct-v0.2	1072	1074	12.57	7.6		20067	Mistral	Apache-2.0
Phi-3-Mini-4K-Instruct-June-24	1071	1082			70.9	12856	Microsoft	MIT
Qwen1.5-7B-Chat	1070	1089		7.6	61	4869	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-1106	1068	1095	18.87	8.32		17031	OpenAI	Proprietary
Phi-3-Mini-4k-Instruct	1066	1086			68.8	21099	Microsoft	MIT
Llama-2-13b-chat	1063	1051		6.65	53.6	19739	Meta	Llama 2
Dolphin-2.2.1-Mistral-7B	1063	1026				1713	Cognitive Computations	Apache-2.0
SOLAR-10.7B-Instruct-v1.0	1062	1047		7.58	66.2	4288	Upstage AI	CC-BY-NC-4.0
WizardLM-13b-v1.2	1059	1026		7.2	52.7	7182	Microsoft	Llama 2
Llama-3.2-1B-Instruct	1054	1047				8547	Meta	Llama 3.2
Zephyr-7B-beta	1053	1030		7.34	61.4	11332	HuggingFace	MIT
MPT-30B-chat	1045	1031		6.39	50.4	2649	MosaicML	CC-BY-NC-SA-4.0
pplx-7B-online	1044	1015				6336	Perplexity AI	Proprietary
CodeLlama-34B-instruct	1043	1042			53.7	7512	Meta	Llama 2
CodeLlama-70B-instruct	1042	1048				1194	Meta	Llama 2
Zephyr-7B-alpha	1042	1034		6.88		1814	HuggingFace	MIT
Vicuna-13B	1042	1032		6.57	55.8	19794	LMSYS	Llama 2
Gemma-7B-it	1037	1047	7.47		64.3	9181	Google	Gemma license
Phi-3-Mini-128k-Instruct	1037	1029	15.43		68.1	21634	Microsoft	MIT
Llama-2-7B-chat	1037	1002		6.27	45.8	14549	Meta	Llama 2
Qwen-14B-Chat	1035	1056		6.96	66.5	5070	Alibaba	Qianwen LICENSE
falcon-180b-chat	1034	1018			68	1327	TII	Falcon-180B TII License
Guanaco-33B	1032	966		6.53	57.6	3000	UW	Non-commercial
Gemma-1.1-2b-it	1021	1036	3.37		64.3	11354	Google	Gemma license
StripedHyena-Nous-7B	1018	999				5270	Together AI	Apache 2.0
OLMo-7B-instruct	1016	1017				6505	Allen AI	Apache-2.0
Mistral-7B-Instruct-v0.1	1008	1008		6.84	55.4	9144	Mistral	Apache 2.0
Vicuna-7B	1005	981		6.17	49.8	7014	LMSYS	Llama 2
PaLM-Chat-Bison-001	1004	990		6.4		8744	Google	Proprietary
Gemma-2B-it	989	1000			42.3	4924	Google	Gemma license
Qwen1.5-4B-Chat	988	990			56.1	7814	Alibaba	Qianwen LICENSE
Koala-13B	964	937		5.35	44.7	7036	UC Berkeley	Non-commercial
ChatGLM3-6B	955	953				4765	Tsinghua	Apache-2.0
GPT4All-13B-Snoozy	932	910		5.41	43	1787	Nomic AI	Non-commercial
MPT-7B-Chat	928	900		5.42	32	4013	MosaicML	CC-BY-NC-SA-4.0
ChatGLM2-6B	924	892		4.96	45.5	2710	Tsinghua	Apache-2.0
RWKV-4-Raven-14B	922	897		3.98	25.6	4934	RWKV	Apache 2.0
Alpaca-13B	902	789		4.53	48.1	5875	Stanford	Non-commercial
OpenAssistant-Pythia-12B	893	873		4.32	27	6379	OpenAssistant	Apache 2.0
ChatGLM-6B	879	884		4.5	36.1	4992	Tsinghua	Non-commercial
FastChat-T5-3B	868	759		3.04	47.7	4303	LMSYS	Apache 2.0
StableLM-Tuned-Alpha-7B	840	858		2.75	24.4	3340	Stability AI	CC-BY-NC-SA-4.0
Dolly-V2-12B	822	746		3.28	25.7	3485	Databricks	MIT
LLaMA-13B	800	669		2.61	47	2445	Meta	Non-commercial

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of R_A and player B a rating of R_B, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.