Chatbot Arena

This leaderboard is based on the following three benchmarks.

• Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 900K+ user votes to compute Elo ratings.
• MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
• MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best for Model Size Class

⬍ Model	▤ Size	Arena Elo	MMLU	License
GPT-4-Turbo-2024-04-09	█ > 72B	1259	86.7	Proprietary
Llama-3-70b-Instruct	▆ 32.1B - 72B	1210	82	Llama 3 Community
Claude 3 Haiku	▄ 8.1B - 32B	1181	75.2	Proprietary
Llama-3-8b-Instruct	▂ 4.1B - 8B	1153	68.4	Llama 3 Community
Phi-3-Mini-128k-Instruct	▁ ≤ 4B	1050	68.1	MIT

Full Leaderboard

⬍ Model	🏆 Arena Elo	MT-bench	MMLU	Votes	Organization	License
GPT-4-Turbo-2024-04-09	1259			35931	OpenAI	Proprietary
GPT-4-1106-preview	1253	9.32		73547	OpenAI	Proprietary
Claude 3 Opus	1251		86.8	80997	Anthropic	Proprietary
Gemini 1.5 Pro API-0409-Preview	1250		81.9	39482	Google	Proprietary
GPT-4-0125-preview	1247			67354	OpenAI	Proprietary
Llama-3-70b-Instruct	1210		82	53404	Meta	Llama 3 Community
Bard (Gemini Pro)	1209			12387	Google	Proprietary
Claude 3 Sonnet	1201		79	78956	Anthropic	Proprietary
Command R+	1191			44988	Cohere	CC-BY-NC-4.0
GPT-4-0314	1190	8.96	86.4	52079	OpenAI	Proprietary
Claude 3 Haiku	1181		75.2	69660	Anthropic	Proprietary
GPT-4-0613	1165	9.18		70726	OpenAI	Proprietary
Mistral-Large-2402	1157		81.2	47755	Mistral	Proprietary
Llama-3-8b-Instruct	1153		68.4	52702	Meta	Llama 3 Community
Reka-Flash-21B-online	1153			12307	Reka AI	Proprietary
Qwen1.5-72B-Chat	1152	8.61	77.5	34812	Alibaba	Qianwen LICENSE
Claude-1	1151	7.9	77	21768	Anthropic	Proprietary
Mistral Medium	1148	8.61	75.3	33625	Mistral	Proprietary
Command R	1148			39140	Cohere	CC-BY-NC-4.0
Reka-Flash-21B	1147		73.5	16999	Reka AI	Proprietary
Mixtral-8x22b-Instruct-v0.1	1146		77.8	25341	Mistral	Apache 2.0
Gemini Pro (Dev API)	1136		71.8	20086	Google	Proprietary
Qwen1.5-32B-Chat	1134	8.3	73.4	17407	Alibaba	Qianwen LICENSE
Claude-2.0	1132	8.06	78.5	13413	Anthropic	Proprietary
Zephyr-ORPO-141b-A35b-v0.1	1129			5207	HuggingFace	Apache 2.0
Mistral-Next	1127			13059	Mistral	Proprietary
GPT-3.5-Turbo-0613	1120	8.39		40845	OpenAI	Proprietary
Claude-2.1	1119	8.18		39731	Anthropic	Proprietary
Starling-LM-7B-beta	1119	8.12		18020	Nexusflow	Apache-2.0
Qwen1.5-14B-Chat	1118	7.91	67.6	20605	Alibaba	Qianwen LICENSE
Gemini Pro	1115		71.8	6818	Google	Proprietary
Mixtral-8x7b-Instruct-v0.1	1114	8.3	70.6	64921	Mistral	Apache 2.0
Yi-34B-Chat	1111		73.5	13177	01 AI	Yi License
Claude-Instant-1	1110	7.85	73.4	21559	Anthropic	Proprietary
GPT-3.5-Turbo-0314	1109	7.94	70	5878	OpenAI	Proprietary
WizardLM-70B-v1.0	1109	7.71	63.7	8867	Microsoft	Llama 2 Community
GPT-3.5-Turbo-0125	1107			47220	OpenAI	Proprietary
Tulu-2-DPO-70B	1103	7.89		6935	AllenAI/UW	AI2 ImpACT Low-risk
DBRX-Instruct-Preview	1103		73.7	31953	Databricks	DBRX LICENSE
OpenChat-3.5-0106	1099	7.8	65.8	14159	OpenChat	Apache-2.0
Snowflake Arctic Instruct	1098		67.3	28605	Snowflake	Apache 2.0
Vicuna-33B	1094	7.12	59.2	24375	LMSYS	Non-commercial
Starling-LM-7B-alpha	1092	8.09	63.9	10962	UC Berkeley	CC-BY-NC-4.0
Llama-2-70b-chat	1089	6.86	63	39678	Meta	Llama 2 Community
Nous-Hermes-2-Mixtral-8x7B-DPO	1087			3980	NousResearch	Apache-2.0
Gemma-1.1-7B-it	1083		64.3	9351	Google	Gemma license
NV-Llama2-70B-SteerLM-Chat	1082	7.54	68.5	3759	Nvidia	Llama 2 Community
DeepSeek-LLM-67B-Chat	1080		71.3	5177	DeepSeek AI	DeepSeek License
OpenChat-3.5	1079	7.81	64.3	8440	OpenChat	Apache-2.0
OpenHermes-2.5-Mistral-7b	1079			5275	NousResearch	Apache-2.0
Qwen1.5-7B-Chat	1077	7.6	61	5025	Alibaba	Qianwen LICENSE
pplx-70b-online	1075			7234	Perplexity AI	Proprietary
Mistral-7B-Instruct-v0.2	1074	7.6		20413	Mistral	Apache-2.0
GPT-3.5-Turbo-1106	1073	8.32		17819	OpenAI	Proprietary
Dolphin-2.2.1-Mistral-7B	1066			1775	Cognitive Computations	Apache-2.0
SOLAR-10.7B-Instruct-v1.0	1065	7.58	66.2	4472	Upstage AI	CC-BY-NC-4.0
WizardLM-13b-v1.2	1062	7.2	52.7	7611	Microsoft	Llama 2 Community
Llama-2-13b-chat	1057	6.65	53.6	18530	Meta	Llama 2 Community
Zephyr-7b-beta	1054	7.34	61.4	11924	HuggingFace	MIT
Phi-3-Mini-128k-Instruct	1050		68.1	19804	Microsoft	MIT
Vicuna-13B	1048	6.57	55.8	20695	LMSYS	Llama 2 Community
MPT-30B-chat	1048	6.39	50.4	2791	MosaicML	CC-BY-NC-SA-4.0
CodeLlama-70B-instruct	1048			1321	Meta	Llama 2 Community
CodeLlama-34B-instruct	1047		53.7	8036	Meta	Llama 2 Community
Gemma-7B-it	1044		64.3	9859	Google	Gemma license
Zephyr-7b-alpha	1043	6.88		1901	HuggingFace	MIT
pplx-7b-online	1043			6603	Perplexity AI	Proprietary
Llama-2-7b-chat	1041	6.27	45.8	15427	Meta	Llama 2 Community
Qwen-14B-Chat	1039	6.96	66.5	5295	Alibaba	Qianwen LICENSE
falcon-180b-chat	1037		68	1393	TII	Falcon-180B TII License
Guanaco-33B	1034	6.53	57.6	3181	UW	Non-commercial
StripedHyena-Nous-7B	1023			5505	Together AI	Apache 2.0
OLMo-7B-instruct	1021			7023	Allen AI	Apache-2.0
Gemma-1.1-2B-it	1015		64.3	4693	Google	Gemma license
Mistral-7B-Instruct-v0.1	1012	6.84	55.4	9618	Mistral	Apache 2.0
PaLM-Chat-Bison-001	1010	6.4		9222	Google	Proprietary
Vicuna-7B	1009	6.17	49.8	7373	LMSYS	Llama 2 Community
Qwen1.5-4B-Chat	1003		56.1	8491	Alibaba	Qianwen LICENSE
Gemma-2B-it	1000		42.3	5323	Google	Gemma license
Koala-13B	969	5.35	44.7	7300	UC Berkeley	Non-commercial
ChatGLM3-6B	961			4944	Tsinghua	Apache-2.0
GPT4All-13B-Snoozy	939	5.41	43	1907	Nomic AI	Non-commercial
MPT-7B-Chat	933	5.42	32	4190	MosaicML	CC-BY-NC-SA-4.0
ChatGLM2-6B	933	4.96	45.5	2880	Tsinghua	Apache-2.0
RWKV-4-Raven-14B	928	3.98	25.6	5129	RWKV	Apache 2.0
Alpaca-13B	908	4.53	48.1	6111	Stanford	Non-commercial
OpenAssistant-Pythia-12B	900	4.32	27	6623	OpenAssistant	Apache 2.0
ChatGLM-6B	887	4.5	36.1	5195	Tsinghua	Non-commercial
FastChat-T5-3B	877	3.04	47.7	4521	LMSYS	Apache 2.0
StableLM-Tuned-Alpha-7B	849	2.75	24.4	3461	Stability AI	CC-BY-NC-SA-4.0
Dolly-V2-12B	827	3.28	25.7	3666	Databricks	MIT
LLaMA-13B	804	2.61	47	2538	Meta	Non-commercial
WizardLM-30B		7.01	58.7		Microsoft	Non-commercial
Vicuna-13B-16k		6.92	54.5		LMSYS	Llama 2 Community
WizardLM-13B-v1.1		6.76	50		Microsoft	Non-commercial
Tulu-30B		6.43	58.1		AllenAI/UW	Non-commercial
Guanaco-65B		6.41	62.1		UW	Non-commercial
OpenAssistant-LLaMA-30B		6.41	56		OpenAssistant	Non-commercial
WizardLM-13B-v1.0		6.35	52.3		Microsoft	Non-commercial
Vicuna-7B-16k		6.22	48.5		LMSYS	Llama 2 Community
Baize-v2-13B		5.75	48.9		UCSD	Non-commercial
XGen-7B-8K-Inst		5.55	42.1		Salesforce	Non-commercial
Nous-Hermes-13B		5.51	49.3		NousResearch	Non-commercial
MPT-30B-Instruct		5.22	47.8		MosaicML	CC-BY-SA 3.0
Falcon-40B-Instruct		5.17	54.7		TII	Apache 2.0

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of R_A and player B a rating of R_B, the probability of player A winning is

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

This algorithm has two distinct features:

1. It can be computed asynchronously by players around the world.
2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.