Chatbot Arena

Attribution LMSYS May 20, 2024

This leaderboard is based on the following three benchmarks.

  • Chatbot Arena - a crowdsourced, randomized battle platform for large language models (LLMs). We use 1.1M+ user votes to compute Elo ratings.
  • MT-Bench - a set of challenging multi-turn questions. We use GPT-4 to grade model responses.
  • MMLU (5-shot) - a test to measure a model’s multitask accuracy on 57 tasks.

| Vote | Blog | GitHub | Paper | Dataset | Twitter | Discord |

Best for Model Size Class

Model▤ SizeArena EloMMLUContext WindowLicense
OpenAI GPT-4o-2024-05-1372B +128788.7128KProprietary
Meta Llama-3-70b-Instruct40B - 72B1203828KLlama 3 Community
01.AI Yi-1.5-34B-Chat-16K24B - 40B76.816KApache 2.0
Anthropic Claude 3 Haiku8B - 24B118175.2200KProprietary
Meta Llama-3-8b-Instruct4B - 8B115468.48KLlama 3 Community
Microsoft Phi-3-Mini-128k-Instruct1B - 4B105368.1128KMIT

Full Leaderboard
Model🏆 Arena EloMT-benchMMLUVotesOrganizationLicense
Gemini 1.5 Pro API-0409-Preview124881.962929GoogleProprietary
Claude 3 Opus124686.8121218AnthropicProprietary
Yi-Large-preview12361567101 AIProprietary
Bard (Gemini Pro)120812387GoogleProprietary
Llama-3-70b-Instruct120382129016MetaLlama 3 Community
Claude 3 Sonnet11997997268AnthropicProprietary
Reka-Core-20240501119583.237076Reka AIProprietary
Command R+118862689CohereCC-BY-NC-4.0
Claude 3 Haiku118175.286889AnthropicProprietary
GLM-4-011611756167Zhipu AIProprietary
Qwen1.5-110B-Chat11698.8880.418348AlibabaQianwen LICENSE
Reka-Flash-21B-online115518181Reka AIProprietary
Llama-3-8b-Instruct115468.483255MetaLlama 3 Community
Qwen1.5-72B-Chat11538.6177.541536AlibabaQianwen LICENSE
Command R114948036CohereCC-BY-NC-4.0
Reka-Flash-21B114873.524970Reka AIProprietary
Mistral Medium11468.6175.337912MistralProprietary
Mixtral-8x22b-Instruct-v0.1114477.837010MistralApache 2.0
Gemini Pro (Dev API)113571.820086GoogleProprietary
Qwen1.5-32B-Chat11348.373.422828AlibabaQianwen LICENSE
Zephyr-ORPO-141b-A35b-v0.111295463HuggingFaceApache 2.0
Qwen1.5-14B-Chat11197.9167.621550AlibabaQianwen LICENSE
Yi-34B-Chat111673.51745701 AIYi License
Gemini Pro111571.86818GoogleProprietary
Mixtral-8x7b-Instruct-v0.111148.370.670960MistralApache 2.0
WizardLM-70B-v1.011107.7163.78866MicrosoftLlama 2 Community
Tulu-2-DPO-70B11047.896935AllenAI/UWAI2 ImpACT Low-risk
DBRX-Instruct-Preview110373.732964DatabricksDBRX LICENSE
Snowflake Arctic Instruct109867.333341SnowflakeApache 2.0
Llama-2-70b-chat10956.866342764MetaLlama 2 Community
Starling-LM-7B-alpha10928.0963.910962UC BerkeleyCC-BY-NC-4.0
Gemma-1.1-7B-it109164.315438GoogleGemma license
NV-Llama2-70B-SteerLM-Chat10837.5468.53759NvidiaLlama 2 Community
DeepSeek-LLM-67B-Chat108071.35177DeepSeek AIDeepSeek License
Qwen1.5-7B-Chat10797.6615332AlibabaQianwen LICENSE
pplx-70b-online10777234Perplexity AIProprietary
Llama-2-13b-chat10676.6553.621042MetaLlama 2 Community
SOLAR-10.7B-Instruct-v1.010667.5866.24472Upstage AICC-BY-NC-4.0
Dolphin-2.2.1-Mistral-7B10661775Cognitive ComputationsApache-2.0
WizardLM-13b-v1.210647.252.77611MicrosoftLlama 2 Community
Vicuna-13B10506.5755.820950LMSYSLlama 2 Community
CodeLlama-34B-instruct104953.78035MetaLlama 2 Community
CodeLlama-70B-instruct10471320MetaLlama 2 Community
pplx-7b-online10456603Perplexity AIProprietary
Gemma-7B-it104464.39859GoogleGemma license
Llama-2-7b-chat10436.2745.815690MetaLlama 2 Community
Qwen-14B-Chat10416.9666.55295AlibabaQianwen LICENSE
falcon-180b-chat1039681392TIIFalcon-180B TII License
Gemma-1.1-2B-it103164.39405GoogleGemma license
StripedHyena-Nous-7B10245505Together AIApache 2.0
OLMo-7B-instruct10207023Allen AIApache-2.0
Mistral-7B-Instruct-v0.110156.8455.49618MistralApache 2.0
Vicuna-7B10116.1749.87373LMSYSLlama 2 Community
Qwen1.5-4B-Chat100356.18491AlibabaQianwen LICENSE
Gemma-2B-it100042.35323GoogleGemma license
Koala-13B9715.3544.77300UC BerkeleyNon-commercial
GPT4All-13B-Snoozy9405.41431907Nomic AINon-commercial
RWKV-4-Raven-14B9303.9825.65129RWKVApache 2.0
OpenAssistant-Pythia-12B9024.32276623OpenAssistantApache 2.0
FastChat-T5-3B8793.0447.74520LMSYSApache 2.0
StableLM-Tuned-Alpha-7B8502.7524.43461Stability AICC-BY-NC-SA-4.0
Vicuna-13B-16k6.9254.5LMSYSLlama 2 Community
Vicuna-7B-16k6.2248.5LMSYSLlama 2 Community
MPT-30B-Instruct5.2247.8MosaicMLCC-BY-SA 3.0
Falcon-40B-Instruct5.1754.7TIIApache 2.0

If you want to see more models, please help us add them.

💻 Code: The Arena Elo ratings are computed by this notebook. The MT-bench scores (single-answer grading on a scale of 10) are computed by fastchat.llm_judge. The MMLU scores are computed by InstructEval. Higher values are better for all benchmarks. Empty cells mean not available. The latest and detailed leaderboard is here.

More Statistics for Chatbot Arena

🔗 Arena Statistics

Transition from online Elo rating system to Bradley-Terry model

We adopted the Elo rating system for ranking models since the launch of the Arena. It has been useful to transform pairwise human preference to Elo ratings that serve as a predictor of winrate between models. Specifically, if player A has a rating of RA and player B a rating of RB, the probability of player A winning is

{\displaystyle E_{\mathsf {A}}={\frac {1}{1+10^{(R_{\mathsf {B}}-R_{\mathsf {A}})/400}}}~.}

ELO rating has been used to rank chess players by the international community for over 60 years. Standard Elo rating systems assume a player’s performance changes overtime. So an online algorithm is needed to capture such dynamics, meaning recent games should weigh more than older games. Specifically, after each game, a player’s rating is updated according to the difference between predicted outcome and actual outcome.

{\displaystyle R_{\mathsf {A}}'=R_{\mathsf {A}}+K\cdot (S_{\mathsf {A}}-E_{\mathsf {A}})~.}

This algorithm has two distinct features:

  1. It can be computed asynchronously by players around the world.
  2. It allows for players performance to change dynamically – it does not assume a fixed unknown value for the players rating.

This ability to adapt is determined by the parameter K which controls the magnitude of rating changes that can affect the overall result. A larger K essentially put more weight on the recent games, which may make sense for new players whose performance improves quickly. However as players become more senior and their performance “converges” then a smaller value of K is more appropriate. As a result, USCF adopted K based on the number of games and tournaments completed by the player (reference). That is, the Elo rating of a senior player changes slower than a new player.

When we launched the Arena, we noticed considerable variability in the ratings using the classic online algorithm. We tried to tune the K to be sufficiently stable while also allowing new models to move up quickly in the leaderboard. We ultimately decided to adopt a bootstrap-like technique to shuffle the data and sample Elo scores from 1000 permutations of the online plays. You can find the details in this notebook. This provided consistent stable scores and allowed us to incorporate new models quickly. This is also observed in a recent work by Cohere. However, we used the same samples to estimate confidence intervals which were therefore too wide (effectively CI’s for the original online Elo estimates).

In the context of LLM ranking, there are two important differences from the classic Elo chess ranking system. First, we have access to the entire history of all games for all models and so we don’t need a decentralized algorithm. Second, most models are static (we have access to the weights) and so we don’t expect their performance to change. However, it is worth noting that the hosted proprietary models may not be static and their behavior can change without notice. We try our best to pin specific model API versions if possible.

To improve the quality of our rankings and their confidence estimates, we are adopting another widely used rating system called the Bradley–Terry (BT) model. This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate. Similar to Elo rating, BT model is also based on pairwise comparison to derive ratings of players to estimate win rate between each other. The core difference between BT model vs the online Elo system is the assumption that player’s performance does not change (i.e., game order does not matter) and the computation takes place in a centralized fashion.

MT-Bench Effectively Distinguishes Among Chatbots

We observe a clear distinction among chatbots of varying abilities, with scores showing a high correlation with the Chatbot Arena Elo rating. In particular, MT-Bench reveals noticeable performance gaps between GPT-4 and GPT-3.5, and between open and proprietary models.

To delve deeper into the distinguishing factors among chatbots, we select a few representative chatbots and break down their performance per category. GPT-4 shows superior performance in Coding and Reasoning compared to GPT-3.5.

Figure 5: The comparison of 6 representative LLMs regarding their abilities in 8 categories: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities