Chatbot Arena

This leaderboard is based on the following three benchmarks.

• Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 2.2M+ user votes to compute Elo ratings.
• MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
• MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best for Model Size Class

Model	▤ Size	Arena Elo	MMLU	License
ChatGPT-4o-latest	72B +	1340	88.7	Proprietary
o1-preview	72B +	1333	92.3	Proprietary
Llama-3.1-Nemotron-70B-Instruct	32B - 72B	1269	86	Llama 3.1
Qwen2.5-32B-Instruct	16B - 32B		83.9	Qwen
Qwen2.5-14B-Instruct	8B - 16B		80	Qwen
Llama-3.1-8B-Instruct	4B - 8B	1175	73	Llama 3.1
Llama-3.2-3B-Instruct	1B - 4B	1103	63.4	Llama 3.2

Full Leaderboard

Model	🏆 Arena Elo	Coding Elo	MT-bench	MMLU	Votes	Organization	License
🥇 Gemini-Exp-1114	1344	1329			6446	Google	Proprietary
🥇 ChatGPT-4o-latest (2024-09-03)	1340	1339			42225	OpenAI	Proprietary
🥇 o1-preview	1333	1354			26268	OpenAI	Proprietary
🥈 o1-mini	1308	1364			28953	OpenAI	Proprietary
🥈 Gemini-1.5-Pro-002	1301	1290			23856	Google	Proprietary
🥈 Grok-2-08-13	1290	1287			47908	xAI	Proprietary
🥈 Yi-Lightning	1287	1303			27114	01 AI	Proprietary
🥈 Claude 3.5 Sonnet (2024-10-22)	1283	1321		88.7	26047	Anthropic	Proprietary
🥉 GLM-4-Plus	1275	1285			25601	Zhipu AI	Proprietary
🥉 GPT-4o-mini-2024-07-18	1272	1284		82	48407	OpenAI	Proprietary
🥉 Gemini-1.5-Flash-002	1272	1254			18112	Google	Proprietary
🥉 Llama-3.1-Nemotron-70B-Instruct	1269	1273			7263	Nvidia	Llama 3.1
🥉 Llama-3.1-405B-Instruct	1267	1278		88.6	48804	Meta	Llama 3.1
🥉 Grok-2-Mini-08-13	1267	1262			39214	xAI	Proprietary
🥉 Yi-Lightning-lite	1264	1267			17284	01 AI	Proprietary
🥉 Qwen-Max-0919	1263	1280			17642	Alibaba	Proprietary
🥉 Qwen2.5-72B-Instruct	1259	1283			23112	Alibaba	Qwen
🥉 Deepseek-v2.5	1258	1288			22918	DeepSeek	DeepSeek
🥉 GPT-4-Turbo-2024-04-09	1256	1263			101865	OpenAI	Proprietary
Mistral-Large-2407	1251	1269			45046	Mistral	Mistral Research
Athene-70B	1250	1254			20697	NexusFlow	CC-BY-NC-4.0
GPT-4-1106-preview	1250	1253	9.32		103870	OpenAI	Proprietary
Claude 3 Opus	1248	1250		86.8	183545	Anthropic	Proprietary
Llama-3.1-70B-Instruct	1247	1251		86	44687	Meta	Llama 3.1
GPT-4-0125-preview	1245	1244			97202	OpenAI	Proprietary
Yi-Large-preview	1240	1245			51701	01 AI	Proprietary
Reka-Core-20240722	1230	1209			13326	Reka AI	Proprietary
Qwen-Plus-0828	1227	1246			14739	Alibaba	Proprietary
Gemini-1.5-Flash-001	1227	1232		78.9	65794	Google	Proprietary
Jamba-1.5-Large	1221	1228		81.2	9186	AI21 Labs	Jamba Open
Deepseek-v2-API-0628	1219	1242			19605	DeepSeek AI	DeepSeek
Gemma-2-27B-it	1219	1209			54961	Google	Gemma license
Gemma-2-9B-it-SimPO	1216	1197			10613	Princeton	MIT
Command R+ (08-2024)	1215	1182			10608	Cohere	CC-BY-NC-4.0
Deepseek-Coder-v2-0724	1214	1267			11772	DeepSeek	Proprietary
Yi-Large	1213	1220			16672	01 AI	Proprietary
Llama-3.1-Nemotron-51B-Instruct	1212	1211			3943	Nvidia	Llama 3.1
Gemini-1.5-Flash-8B-001	1211	1204			19267	Google	Proprietary
Nemotron-4-340B-Instruct	1209	1198			20641	Nvidia	NVIDIA Open Model
Gemini App (2024-01-24)	1208	1171			11820	Google	Proprietary
GLM-4-0520	1206	1216			10228	Zhipu AI	Proprietary
Llama-3-70B-Instruct	1206	1200		82	163858	Meta	Llama 3
Gemini-1.5-Flash-8B-Exp-0827	1205	1190			25471	Google	Proprietary
Claude 3 Sonnet	1201	1213		79	113045	Anthropic	Proprietary
Reka-Flash-20240722	1201	1188			13774	Reka AI	Proprietary
Reka-Core-20240501	1200	1190		83.2	62609	Reka AI	Proprietary
Gemma-2-9B-it	1190	1170			39510	Google	Gemma license
Command R+ (04-2024)	1190	1164			80879	Cohere	CC-BY-NC-4.0
Hunyuan-Standard-256K	1188	1226			2860	Tencent	Proprietary
Qwen2-72B-Instruct	1187	1187	9.12	84.2	38980	Alibaba	Qianwen LICENSE
GPT-4-0314	1186	1195	8.96	86.4	55966	OpenAI	Proprietary
GLM-4-0116	1183	1191			7582	Zhipu AI	Proprietary
Qwen-Max-0428	1183	1190			25703	Alibaba	Proprietary
Command R (08-2024)	1180	1162			10924	Cohere	CC-BY-NC-4.0
Ministral-8B-2410	1179	1190			2333	Mistral	MRL
Claude 3 Haiku	1179	1189		75.2	122422	Anthropic	Proprietary
DeepSeek-Coder-V2-Instruct	1178	1239			15796	DeepSeek AI	DeepSeek License
Jamba-1.5-Mini	1176	1182		69.7	9296	AI21 Labs	Jamba Open
Llama-3.1-8B-Instruct	1175	1185		73	42983	Meta	Llama 3.1
Reka-Flash-Preview-20240611	1165	1155			20466	Reka AI	Proprietary
GPT-4-0613	1163	1167	9.18		91646	OpenAI	Proprietary
Qwen1.5-110B-Chat	1161	1175	8.88	80.4	27472	Alibaba	Qianwen LICENSE
Mistral-Large-2402	1157	1170		81.2	64909	Mistral	Proprietary
Yi-1.5-34B-Chat	1157	1162		76.8	25150	01 AI	Apache-2.0
Reka-Flash-21B-online	1156	1147			16023	Reka AI	Proprietary
Llama-3-8B-Instruct	1152	1146		68.4	109312	Meta	Llama 3
InternLM2.5-20B-chat	1149	1159			10709	InternLM	Other
Claude-1	1149	1136	7.9	77	21149	Anthropic	Proprietary
Command R (04-2024)	1149	1123			56400	Cohere	CC-BY-NC-4.0
Mixtral-8x22b-Instruct-v0.1	1148	1153		77.8	53814	Mistral	Apache 2.0
Mistral Medium	1148	1152	8.61	75.3	35537	Mistral	Proprietary
Reka-Flash-21B	1148	1141		73.5	25802	Reka AI	Proprietary
Qwen1.5-72B-Chat	1147	1160	8.61	77.5	40636	Alibaba	Qianwen LICENSE
Gemma-2-2b-it	1140	1102		51.3	31041	Google	Gemma license
Claude-2.0	1132	1135	8.06	78.5	12761	Anthropic	Proprietary
Gemini-1.0-Pro-001	1131	1103		71.8	18787	Google	Proprietary
Zephyr-ORPO-141b-A35b-v0.1	1127	1124			4849	HuggingFace	Apache 2.0
Qwen1.5-32B-Chat	1126	1149	8.3	73.4	22753	Alibaba	Qianwen LICENSE
Mistral-Next	1124	1132			12376	Mistral	Proprietary
Phi-3-Medium-4k-Instruct	1123	1125		78	26143	Microsoft	MIT
Starling-LM-7B-beta	1119	1129	8.12		16665	Nexusflow	Apache-2.0
Claude-2.1	1118	1132	8.18		37683	Anthropic	Proprietary
GPT-3.5-Turbo-0613	1117	1135	8.39		38947	OpenAI	Proprietary
Mixtral-8x7B-Instruct-v0.1	1114	1114	8.3	70.6	76142	Mistral	Apache 2.0
Claude-Instant-1	1111	1109	7.85	73.4	20617	Anthropic	Proprietary
Yi-34B-Chat	1111	1106		73.5	15919	01 AI	Yi License
Gemini Pro	1111	1091		71.8	6561	Google	Proprietary
Qwen1.5-14B-Chat	1109	1126	7.91	67.6	18666	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-0125	1106	1124			68876	OpenAI	Proprietary
GPT-3.5-Turbo-0314	1106	1115	7.94	70	5647	OpenAI	Proprietary
WizardLM-70B-v1.0	1106	1071	7.71	63.7	8382	Microsoft	Llama 2
DBRX-Instruct-Preview	1103	1118		73.7	33718	Databricks	DBRX LICENSE
Llama-3.2-3B-Instruct	1103	1081			8454	Meta	Llama 3.2
Phi-3-Small-8k-Instruct	1102	1107		75.7	18499	Microsoft	MIT
Tulu-2-DPO-70B	1099	1093	7.89		6662	AllenAI/UW	AI2 ImpACT Low-risk
Llama-2-70B-chat	1093	1072	6.86	63	39617	Meta	Llama 2
OpenChat-3.5-0106	1092	1102	7.8	65.8	12971	OpenChat	Apache-2.0
Vicuna-33B	1091	1067	7.12	59.2	22941	LMSYS	Non-commercial
Snowflake Arctic Instruct	1090	1077		67.3	34163	Snowflake	Apache 2.0
Starling-LM-7B-alpha	1088	1080	8.09	63.9	10414	UC Berkeley	CC-BY-NC-4.0
Gemma-1.1-7B-it	1084	1084		64.3	25074	Google	Gemma license
Nous-Hermes-2-Mixtral-8x7B-DPO	1084	1079			3837	NousResearch	Apache-2.0
NV-Llama2-70B-SteerLM-Chat	1081	1023	7.54	68.5	3638	Nvidia	Llama 2
pplx-70B-online	1078	1028			6892	Perplexity AI	Proprietary
DeepSeek-LLM-67B-Chat	1077	1079		71.3	4984	DeepSeek AI	DeepSeek License
OpenChat-3.5	1076	1054	7.81	64.3	8110	OpenChat	Apache-2.0
OpenHermes-2.5-Mistral-7B	1074	1058			5090	NousResearch	Apache-2.0
Mistral-7B-Instruct-v0.2	1072	1074	7.6		20060	Mistral	Apache-2.0
Phi-3-Mini-4K-Instruct-June-24	1071	1083		70.9	12874	Microsoft	MIT
Qwen1.5-7B-Chat	1070	1089	7.6	61	4862	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-1106	1068	1095	8.32		17026	OpenAI	Proprietary
Phi-3-Mini-4k-Instruct	1066	1086		68.8	21118	Microsoft	MIT
Llama-2-13b-chat	1063	1051	6.65	53.6	19730	Meta	Llama 2
SOLAR-10.7B-Instruct-v1.0	1062	1046	7.58	66.2	4288	Upstage AI	CC-BY-NC-4.0
Dolphin-2.2.1-Mistral-7B	1062	1024			1713	Cognitive Computations	Apache-2.0
WizardLM-13b-v1.2	1059	1026	7.2	52.7	7183	Microsoft	Llama 2
Llama-3.2-1B-Instruct	1053	1047			8568	Meta	Llama 3.2
Zephyr-7B-beta	1053	1030	7.34	61.4	11325	HuggingFace	MIT
MPT-30B-chat	1046	1031	6.39	50.4	2649	MosaicML	CC-BY-NC-SA-4.0
pplx-7B-online	1045	1015			6334	Perplexity AI	Proprietary
CodeLlama-34B-instruct	1043	1042		53.7	7509	Meta	Llama 2
Vicuna-13B	1042	1032	6.57	55.8	19782	LMSYS	Llama 2
Zephyr-7B-alpha	1041	1034	6.88		1813	HuggingFace	MIT
CodeLlama-70B-instruct	1040	1047			1190	Meta	Llama 2
Gemma-7B-it	1037	1047		64.3	9177	Google	Gemma license
Phi-3-Mini-128k-Instruct	1037	1029		68.1	21616	Microsoft	MIT
Llama-2-7B-chat	1037	1002	6.27	45.8	14551	Meta	Llama 2
Qwen-14B-Chat	1035	1056	6.96	66.5	5069	Alibaba	Qianwen LICENSE
falcon-180b-chat	1034	1017		68	1326	TII	Falcon-180B TII License
Guanaco-33B	1033	965	6.53	57.6	2997	UW	Non-commercial
Gemma-1.1-2b-it	1021	1036		64.3	11349	Google	Gemma license
StripedHyena-Nous-7B	1018	999			5272	Together AI	Apache 2.0
OLMo-7B-instruct	1016	1017			6495	Allen AI	Apache-2.0
Mistral-7B-Instruct-v0.1	1008	1008	6.84	55.4	9139	Mistral	Apache 2.0
Vicuna-7B	1005	981	6.17	49.8	7017	LMSYS	Llama 2
PaLM-Chat-Bison-001	1004	990	6.4		8743	Google	Proprietary
Gemma-2B-it	990	1000		42.3	4913	Google	Gemma license
Qwen1.5-4B-Chat	988	990		56.1	7812	Alibaba	Qianwen LICENSE
Koala-13B	964	937	5.35	44.7	7033	UC Berkeley	Non-commercial
ChatGLM3-6B	955	953			4765	Tsinghua	Apache-2.0
GPT4All-13B-Snoozy	932	910	5.41	43	1786	Nomic AI	Non-commercial
MPT-7B-Chat	928	900	5.42	32	4012	MosaicML	CC-BY-NC-SA-4.0
ChatGLM2-6B	924	892	4.96	45.5	2708	Tsinghua	Apache-2.0
RWKV-4-Raven-14B	922	896	3.98	25.6	4935	RWKV	Apache 2.0
Alpaca-13B	902	789	4.53	48.1	5872	Stanford	Non-commercial
OpenAssistant-Pythia-12B	893	873	4.32	27	6382	OpenAssistant	Apache 2.0
ChatGLM-6B	879	884	4.5	36.1	4993	Tsinghua	Non-commercial
FastChat-T5-3B	868	759	3.04	47.7	4300	LMSYS	Apache 2.0
StableLM-Tuned-Alpha-7B	840	858	2.75	24.4	3338	Stability AI	CC-BY-NC-SA-4.0
Dolly-V2-12B	822	746	3.28	25.7	3485	Databricks	MIT
LLaMA-13B	799	669	2.61	47	2445	Meta	Non-commercial

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of R_A and player B a rating of R_B, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.