Chatbot Arena

This leaderboard is based on the following three benchmarks.

• Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 1.5M+ user votes to compute Elo ratings.
• MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
• MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best for Model Size Class

Model	▤ Size	Arena Elo	MMLU	Context Window	License
GPT-4o-2024-05-13	72B +	1287	88.7	128K	Proprietary
Llama-3.1-405b-Instruct	72B +		88.6	128K	Llama 3 Community
Llama-3.1-70b-Instruct	40B - 72B	1207	86	128K	Llama 3 Community
Gemma-2-27B-it	24B - 40B	1217	75.2	8K	Gemma license
Llama-3.1-8b-Instruct	4B - 24B	1152	73	128K	Llama 3 Community
Phi-3-Mini-4k-Instruct	1B - 4B	1066	68.8	4K	MIT

Full Leaderboard

Model	🏆 Arena Elo	Coding Elo	MT-bench	MMLU	Votes	Organization	License
GPT-4o-2024-05-13	1287	1297		88.7	64700	OpenAI	Proprietary
GPT-4o-mini-2024-07-18	1280	1276		82	4449	OpenAI	Proprietary
Claude 3.5 Sonnet	1272	1301		88.7	34591	Anthropic	Proprietary
Gemini-Advanced-0514	1267	1256			48001	Google	Proprietary
Gemini-1.5-Pro-API-0514	1261	1264		85.9	57448	Google	Proprietary
GPT-4-Turbo-2024-04-09	1257	1264			76597	OpenAI	Proprietary
Gemini-1.5-Pro-API-0409	1257	1232		81.9	55681	Google	Proprietary
GPT-4-1106-preview	1251	1255	9.32		88475	OpenAI	Proprietary
Claude 3 Opus	1248	1251		86.8	147947	Anthropic	Proprietary
GPT-4-0125-preview	1245	1245			81807	OpenAI	Proprietary
Yi-Large-preview	1240	1247			50499	01 AI	Proprietary
Gemini-1.5-Flash-API-0514	1228	1233		78.9	48048	Google	Proprietary
Deepseek-v2-API-0628	1222	1245			8089	DeepSeek AI	Proprietary
Gemma-2-27B-it	1217	1207			19624	Google	Gemma license
Yi-Large	1214	1221			15276	01 AI	Proprietary
Nemotron-4-340B-Instruct	1209	1199			20685	Nvidia	NVIDIA Open Model
Bard (Gemini Pro)	1209	1172			11830	Google	Proprietary
GLM-4-0520	1207	1217			10253	Zhipu AI	Proprietary
Llama-3-70b-Instruct	1207	1200		82	154914	Meta	Llama 3 Community
Claude 3 Sonnet	1201	1214		79	112955	Anthropic	Proprietary
Reka-Core-20240501	1200	1191		83.2	61757	Reka AI	Proprietary
Command R+	1190	1165			79583	Cohere	CC-BY-NC-4.0
Gemma-2-9B-it	1188	1167			19774	Google	Gemma license
Qwen2-72B-Instruct	1187	1187	9.12	84.2	30310	Alibaba	Qianwen LICENSE
GPT-4-0314	1186	1196	8.96	86.4	55984	OpenAI	Proprietary
GLM-4-0116	1183	1192			7589	Zhipu AI	Proprietary
Qwen-Max-0428	1183	1191			25775	Alibaba	Proprietary
DeepSeek-Coder-V2-Instruct	1179	1241			15644	DeepSeek AI	DeepSeek License
Claude 3 Haiku	1179	1190		75.2	104266	Anthropic	Proprietary
Reka-Flash-Preview-20240611	1165	1156			19489	Reka AI	Proprietary
Qwen1.5-110B-Chat	1162	1176	8.88	80.4	27497	Alibaba	Qianwen LICENSE
GPT-4-0613	1161	1166	9.18		85431	OpenAI	Proprietary
Mistral-Large-2402	1157	1170		81.2	64444	Mistral	Proprietary
Yi-1.5-34B-Chat	1157	1163		76.8	25171	01 AI	Apache-2.0
Reka-Flash-21B-online	1156	1148			16041	Reka AI	Proprietary
Llama-3-8b-Instruct	1152	1147		68.4	102317	Meta	Llama 3 Community
Claude-1	1149	1136	7.9	77	21179	Anthropic	Proprietary
Command R	1149	1124			55724	Cohere	CC-BY-NC-4.0
Mistral Medium	1148	1153	8.61	75.3	35568	Mistral	Proprietary
Qwen1.5-72B-Chat	1147	1161	8.61	77.5	40662	Alibaba	Qianwen LICENSE
Reka-Flash-21B	1147	1142		73.5	25801	Reka AI	Proprietary
Mixtral-8x22b-Instruct-v0.1	1146	1152		77.8	45772	Mistral	Apache 2.0
Claude-2.0	1131	1135	8.06	78.5	12783	Anthropic	Proprietary
Gemini Pro (Dev API)	1131	1103		71.8	18794	Google	Proprietary
Zephyr-ORPO-141b-A35b-v0.1	1127	1126			4865	HuggingFace	Apache 2.0
Qwen1.5-32B-Chat	1125	1150	8.3	73.4	22770	Alibaba	Qianwen LICENSE
Mistral-Next	1124	1133			12383	Mistral	Proprietary
Phi-3-Medium-4k-Instruct	1122	1126		78	18222	Microsoft	MIT
Starling-LM-7B-beta	1119	1130	8.12		16663	Nexusflow	Apache-2.0
Claude-2.1	1118	1133	8.18		37713	Anthropic	Proprietary
GPT-3.5-Turbo-0613	1117	1136	8.39		38965	OpenAI	Proprietary
Mixtral-8x7b-Instruct-v0.1	1114	1114	8.3	70.6	73266	Mistral	Apache 2.0
Claude-Instant-1	1111	1110	7.85	73.4	20631	Anthropic	Proprietary
Yi-34B-Chat	1111	1107		73.5	15947	01 AI	Yi License
Gemini Pro	1111	1093		71.8	6568	Google	Proprietary
Qwen1.5-14B-Chat	1109	1127	7.91	67.6	18673	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-0314	1106	1115	7.94	70	5663	OpenAI	Proprietary
WizardLM-70B-v1.0	1106	1072	7.71	63.7	8392	Microsoft	Llama 2 Community
GPT-3.5-Turbo-0125	1105	1124			67470	OpenAI	Proprietary
DBRX-Instruct-Preview	1103	1119		73.7	33738	Databricks	DBRX LICENSE
Phi-3-Small-8k-Instruct	1102	1108		75.7	18517	Microsoft	MIT
Tulu-2-DPO-70B	1099	1094	7.89		6669	AllenAI/UW	AI2 ImpACT Low-risk
Llama-2-70b-chat	1093	1073	6.86	63	39635	Meta	Llama 2 Community
OpenChat-3.5-0106	1091	1103	7.8	65.8	12985	OpenChat	Apache-2.0
Vicuna-33B	1091	1068	7.12	59.2	22954	LMSYS	Non-commercial
Snowflake Arctic Instruct	1090	1078		67.3	34214	Snowflake	Apache 2.0
Starling-LM-7B-alpha	1088	1081	8.09	63.9	10424	UC Berkeley	CC-BY-NC-4.0
Gemma-1.1-7B-it	1084	1086		64.3	25112	Google	Gemma license
Nous-Hermes-2-Mixtral-8x7B-DPO	1084	1080			3843	NousResearch	Apache-2.0
NV-Llama2-70B-SteerLM-Chat	1080	1024	7.54	68.5	3636	Nvidia	Llama 2 Community
pplx-70b-online	1078	1029			6893	Perplexity AI	Proprietary
DeepSeek-LLM-67B-Chat	1077	1080		71.3	4984	DeepSeek AI	DeepSeek License
OpenChat-3.5	1076	1054	7.81	64.3	8121	OpenChat	Apache-2.0
OpenHermes-2.5-Mistral-7b	1074	1060			5090	NousResearch	Apache-2.0
Mistral-7B-Instruct-v0.2	1072	1074	7.6		20068	Mistral	Apache-2.0
Qwen1.5-7B-Chat	1070	1089	7.6	61	4869	Alibaba	Qianwen LICENSE
GPT-3.5-Turbo-1106	1068	1096	8.32		17028	OpenAI	Proprietary
Phi-3-Mini-4k-Instruct	1066	1087		68.8	21159	Microsoft	MIT
Llama-2-13b-chat	1063	1052	6.65	53.6	19757	Meta	Llama 2 Community
Phi-3-Mini-4k-Instruct-June-24	1062	1071		70.9	6268	Microsoft	MIT
SOLAR-10.7B-Instruct-v1.0	1062	1048	7.58	66.2	4293	Upstage AI	CC-BY-NC-4.0
Dolphin-2.2.1-Mistral-7B	1062	1026			1714	Cognitive Computations	Apache-2.0
WizardLM-13b-v1.2	1058	1027	7.2	52.7	7195	Microsoft	Llama 2 Community
Zephyr-7b-beta	1053	1031	7.34	61.4	11334	HuggingFace	MIT
MPT-30B-chat	1045	1031	6.39	50.4	2649	MosaicML	CC-BY-NC-SA-4.0
pplx-7b-online	1045	1016			6338	Perplexity AI	Proprietary
CodeLlama-70B-instruct	1042	1049			1191	Meta	Llama 2 Community
CodeLlama-34B-instruct	1042	1043		53.7	7514	Meta	Llama 2 Community
Vicuna-13B	1042	1033	6.57	55.8	19798	LMSYS	Llama 2 Community
Zephyr-7b-alpha	1041	1035	6.88		1817	HuggingFace	MIT
Gemma-7B-it	1037	1048		64.3	9177	Google	Gemma license
Phi-3-Mini-128k-Instruct	1037	1030		68.1	21620	Microsoft	MIT
Llama-2-7b-chat	1037	1003	6.27	45.8	14568	Meta	Llama 2 Community
Qwen-14B-Chat	1035	1056	6.96	66.5	5071	Alibaba	Qianwen LICENSE
falcon-180b-chat	1034	1018		68	1326	TII	Falcon-180B TII License
Guanaco-33B	1032	966	6.53	57.6	3003	UW	Non-commercial
Gemma-1.1-2B-it	1021	1037		64.3	11375	Google	Gemma license
StripedHyena-Nous-7B	1017	1001			5266	Together AI	Apache 2.0
OLMo-7B-instruct	1015	1017			6500	Allen AI	Apache-2.0
Mistral-7B-Instruct-v0.1	1008	1009	6.84	55.4	9151	Mistral	Apache 2.0
Vicuna-7B	1005	983	6.17	49.8	7023	LMSYS	Llama 2 Community
PaLM-Chat-Bison-001	1003	991	6.4		8747	Google	Proprietary
Gemma-2B-it	989	1001		42.3	4919	Google	Gemma license
Qwen1.5-4B-Chat	988	991		56.1	7811	Alibaba	Qianwen LICENSE
Koala-13B	964	938	5.35	44.7	7040	UC Berkeley	Non-commercial
ChatGLM3-6B	955	954			4763	Tsinghua	Apache-2.0
GPT4All-13B-Snoozy	931	910	5.41	43	1787	Nomic AI	Non-commercial
MPT-7B-Chat	927	900	5.42	32	4019	MosaicML	CC-BY-NC-SA-4.0
ChatGLM2-6B	924	893	4.96	45.5	2708	Tsinghua	Apache-2.0
RWKV-4-Raven-14B	921	897	3.98	25.6	4942	RWKV	Apache 2.0
Alpaca-13B	901	791	4.53	48.1	5874	Stanford	Non-commercial
OpenAssistant-Pythia-12B	893	874	4.32	27	6385	OpenAssistant	Apache 2.0
ChatGLM-6B	879	884	4.5	36.1	4999	Tsinghua	Non-commercial
FastChat-T5-3B	869	760	3.04	47.7	4309	LMSYS	Apache 2.0
StableLM-Tuned-Alpha-7B	840	859	2.75	24.4	3337	Stability AI	CC-BY-NC-SA-4.0
Dolly-V2-12B	822	747	3.28	25.7	3488	Databricks	MIT
LLaMA-13B	799	669	2.61	47	2444	Meta	Non-commercial

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of R_A and player B a rating of R_B, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.