Human Leaderboard: 30 day

  • Ranking: The position of the model in the leaderboard as ordered by Overall Score
  • Organization: The group responsible for the model or forecasts
  • Model: The LLM model & prompt info or the human group and forecast aggregation method
    • zero shot: used a zero-shot prompt
    • scratchpad: used a scratchpad prompt with instructions that outline a procedure the model should use to reason about the question
    • with freeze values: means that, for questions from market sources, the prompt was supplemented with the aggregate human forecast from the relevant platform on the day the question set was generated
    • with news: means that the prompt was supplemented with relevant news summaries obtained through an automated process
  • Dataset Score: The average Brier score across all questions sourced from datasets
  • Market Score (resolved): The average Brier score across all resolved questions sourced from prediction markets and forecast aggregation platforms
  • Market Score (unresolved): The average Brier score across all unresolved questions sourced from prediction markets and forecast aggregation platforms
  • Market Score (overall): The average Brier score across all questions sourced from prediction markets and forecast aggregation platforms
  • Overall Resolved Score: The average of the Dataset Score and the Market Score (resolved) columns
  • Overall Score: The average of the Dataset Score and the Market Score (overall) columns
  • Overall Score 95% CI: The 95% confidence interval for the Overall Score
  • Pairwise p-value comparing to No. 1 (bootstrapped): The p-value calculated by bootstrapping the differences in overall score between each model and the best forecaster (the group with rank 1) under the null hypothesis that there's no difference.
  • Pct. more accurate than No. 1: The percent of questions where this forecaster had a better overall score than the best forecaster (with rank 1)
  • Pct. imputed: The percent of questions for which this forecaster did not provide a forecast and hence had a forecast value imputed (0.5 for dataset questions and the aggregate human forecast on the forecast due date for questions sourced from prediction markets or forecast aggregation platforms)
Ranking Organization Model Dataset Score (N=106) Market Score (resolved) (N=14) Market Score (unresolved) (N=63) Market Score (overall) (N=77) Overall Resolved Score (N=120) Overall Score (N=183) Overall Score 95% CI Pairwise p-value comparing to No. 1 (bootstrapped) Pct. more accurate than No. 1 Pct. Imputed
1 ForecastBench Superforecaster median forecast 0.138 0.119 0.036 0.051 0.128 0.094 [0.072, 0.116] 0% 0%
2 ForecastBench Public median forecast 0.161 0.165 0.022 0.048 0.163 0.105 [0.084, 0.125] 0.109 31% 0%
3 Anthropic Claude-3-5-Sonnet-20240620 (scratchpad with freeze values) 0.151 0.196 0.039 0.067 0.174 0.109 [0.087, 0.132] 0.086 36% 0%
4 Anthropic Claude-3-5-Sonnet-20240620 (scratchpad with news with freeze values) 0.159 0.215 0.038 0.070 0.187 0.115 [0.093, 0.136] 0.039 32% 0%
5 Anthropic Claude-3-5-Sonnet-20240620 (zero shot with freeze values) 0.158 0.288 0.026 0.074 0.223 0.116 [0.089, 0.142] 0.048 37% 0%
6 OpenAI GPT-4o (scratchpad with news with freeze values) 0.176 0.119 0.045 0.059 0.147 0.117 [0.097, 0.138] 0.01 27% 0%
7 OpenAI GPT-4-Turbo-2024-04-09 (zero shot with freeze values) 0.181 0.184 0.026 0.055 0.183 0.118 [0.097, 0.139] 0.006 38% 0%
8 OpenAI GPT-4 (zero shot with freeze values) 0.181 0.184 0.028 0.056 0.183 0.118 [0.096, 0.141] 0.007 36% 0%
9 OpenAI GPT-4o (scratchpad with freeze values) 0.182 0.170 0.034 0.059 0.176 0.120 [0.099, 0.141] 0.008 31% 0%
10 Mistral AI Mistral-Large-Latest (scratchpad with freeze values) 0.162 0.158 0.064 0.081 0.160 0.121 [0.105, 0.138] 0.006 25% 0%
11 Anthropic Claude-3-5-Sonnet-20240620 (scratchpad) 0.151 0.252 0.058 0.093 0.202 0.122 [0.099, 0.145] 0.019 32% 0%
12 Anthropic Claude-3-5-Sonnet-20240620 (superforecaster with news 1) 0.157 0.228 0.058 0.089 0.193 0.123 [0.101, 0.145] 0.012 30% 0%
13 Anthropic Claude-3-5-Sonnet-20240620 (superforecaster with news 3) 0.169 0.186 0.058 0.082 0.178 0.125 [0.105, 0.146] 0.008 29% 4%
14 Meta Llama-3-70b-Chat-Hf (zero shot with freeze values) 0.180 0.177 0.053 0.076 0.179 0.128 [0.105, 0.151] 0.003 30% 0%
15 Anthropic Claude-3-5-Sonnet-20240620 (scratchpad with news) 0.159 0.222 0.069 0.097 0.191 0.128 [0.105, 0.151] 0.006 27% 0%
16 Anthropic Claude-3-Opus-20240229 (zero shot with freeze values) 0.185 0.206 0.041 0.071 0.196 0.128 [0.104, 0.152] 0.003 31% 0%
17 Mistral AI Mistral-Large-Latest (zero shot with freeze values) 0.185 0.196 0.047 0.074 0.191 0.130 [0.105, 0.155] 0.001 27% 0%
18 OpenAI GPT-4o (scratchpad with news) 0.176 0.195 0.065 0.089 0.185 0.132 [0.109, 0.156] 0.001 24% 0%
19 Google Gemini-1.5-Pro (zero shot with freeze values) 0.189 0.201 0.048 0.076 0.195 0.132 [0.106, 0.158] 0.003 32% 0%
20 Anthropic Claude-2.1 (scratchpad with freeze values) 0.210 0.061 0.054 0.055 0.135 0.132 [0.116, 0.149] <0.001 31% 22%
21 OpenAI GPT-4 (scratchpad with freeze values) 0.196 0.173 0.046 0.069 0.185 0.132 [0.115, 0.15] 0.001 23% 0%
22 OpenAI GPT-4o (scratchpad) 0.182 0.199 0.059 0.084 0.191 0.133 [0.112, 0.154] <0.001 26% 0%
23 Qwen Qwen1.5-110B-Chat (zero shot with freeze values) 0.202 0.170 0.042 0.065 0.186 0.134 [0.114, 0.154] <0.001 27% 0%
24 OpenAI GPT-4o (zero shot with freeze values) 0.205 0.185 0.036 0.063 0.195 0.134 [0.109, 0.158] 0.001 36% 0%
25 Meta Llama-3-70b-Chat-Hf (scratchpad with freeze values) 0.196 0.149 0.059 0.076 0.172 0.136 [0.118, 0.153] <0.001 25% 0%
26 Google Gemini-1.5-Pro (scratchpad with news with freeze values) 0.175 0.176 0.079 0.096 0.176 0.136 [0.114, 0.157] 0.001 26% 0%
27 Anthropic Claude-3-5-Sonnet-20240620 (zero shot) 0.158 0.308 0.071 0.114 0.233 0.136 [0.109, 0.163] 0.006 29% 0%
28 Anthropic Claude-3-Opus-20240229 (scratchpad with freeze values) 0.180 0.173 0.074 0.092 0.177 0.136 [0.115, 0.157] <0.001 25% 0%
29 Google Gemini-1.5-Flash (scratchpad with freeze values) 0.199 0.164 0.058 0.077 0.182 0.138 [0.116, 0.161] <0.001 25% 0%
30 Google Gemini-1.5-Flash (zero shot with freeze values) 0.189 0.205 0.063 0.089 0.197 0.139 [0.11, 0.168] 0.001 30% 0%
31 OpenAI GPT-4-Turbo-2024-04-09 (zero shot) 0.181 0.235 0.067 0.098 0.208 0.139 [0.118, 0.161] <0.001 24% 0%
32 OpenAI GPT-4-Turbo-2024-04-09 (scratchpad with freeze values) 0.195 0.152 0.070 0.085 0.173 0.140 [0.115, 0.165] <0.001 26% 0%
33 Anthropic Claude-3-Opus-20240229 (scratchpad) 0.180 0.189 0.080 0.100 0.184 0.140 [0.12, 0.16] <0.001 24% 0%
34 Anthropic Claude-2.1 (scratchpad) 0.210 0.111 0.061 0.070 0.160 0.140 [0.12, 0.159] <0.001 29% 23%
35 Google Gemini-1.5-Pro (scratchpad with freeze values) 0.183 0.222 0.072 0.099 0.203 0.141 [0.121, 0.162] <0.001 26% 0%
36 Mistral AI Mixtral-8x22B-Instruct-V0.1 (scratchpad with freeze values) 0.198 0.187 0.063 0.086 0.192 0.142 [0.122, 0.161] <0.001 25% 0%
37 Mistral AI Mixtral-8x22B-Instruct-V0.1 (zero shot with freeze values) 0.206 0.183 0.056 0.079 0.194 0.142 [0.116, 0.169] <0.001 32% 0%
38 Mistral AI Mistral-Large-Latest (scratchpad) 0.162 0.230 0.099 0.123 0.196 0.142 [0.123, 0.162] <0.001 25% 0%
39 Google Gemini-1.5-Pro (scratchpad with news) 0.175 0.174 0.098 0.112 0.174 0.143 [0.122, 0.165] <0.001 27% 0%
40 ForecastBench Imputed Forecaster 0.250 0.107 0.021 0.037 0.179 0.143 [0.129, 0.158] <0.001 33% 100%
41 Anthropic Claude-2.1 (zero shot with freeze values) 0.222 0.184 0.041 0.067 0.203 0.145 [0.124, 0.165] <0.001 31% 0%
42 OpenAI GPT-4-Turbo-2024-04-09 (scratchpad) 0.195 0.206 0.071 0.095 0.200 0.145 [0.125, 0.165] <0.001 22% 0%
43 Google Gemini-1.5-Pro (scratchpad) 0.183 0.233 0.081 0.109 0.208 0.146 [0.125, 0.167] <0.001 24% 0%
44 OpenAI GPT-4 (scratchpad) 0.196 0.176 0.079 0.096 0.186 0.146 [0.129, 0.163] <0.001 19% 0%
45 Anthropic Claude-3-Opus-20240229 (zero shot) 0.185 0.244 0.083 0.112 0.215 0.149 [0.123, 0.174] <0.001 23% 0%
46 OpenAI GPT-4-Turbo-2024-04-09 (scratchpad with news with freeze values) 0.208 0.157 0.076 0.090 0.183 0.149 [0.124, 0.175] <0.001 30% 0%
47 Anthropic Claude-3-Opus-20240229 (superforecaster with news 1) 0.168 0.272 0.101 0.132 0.220 0.150 [0.127, 0.173] <0.001 25% 0%
48 OpenAI GPT-4 (zero shot) 0.181 0.181 0.105 0.119 0.181 0.150 [0.13, 0.17] <0.001 22% 0%
49 Meta Llama-3-70b-Chat-Hf (zero shot) 0.180 0.233 0.095 0.120 0.206 0.150 [0.128, 0.172] <0.001 24% 0%
50 Qwen Qwen1.5-110B-Chat (scratchpad with news with freeze values) 0.198 0.182 0.085 0.103 0.190 0.150 [0.13, 0.171] <0.001 25% 0%
51 OpenAI GPT-4o (zero shot) 0.205 0.232 0.067 0.097 0.218 0.151 [0.128, 0.174] <0.001 25% 0%
52 Meta Llama-3-8b-Chat-Hf (zero shot with freeze values) 0.202 0.246 0.069 0.101 0.224 0.152 [0.125, 0.179] <0.001 27% 0%
53 Mistral AI Mixtral-8x7B-Instruct-V0.1 (zero shot with freeze values) 0.214 0.202 0.065 0.090 0.208 0.152 [0.118, 0.186] <0.001 36% 0%
54 OpenAI GPT-4o (superforecaster with news 3) 0.213 0.174 0.076 0.094 0.194 0.153 [0.133, 0.174] <0.001 24% 9%
55 Google Gemini-1.5-Pro (superforecaster with news 3) 0.193 0.171 0.101 0.114 0.182 0.153 [0.132, 0.175] <0.001 25% 0%
56 Google Gemini-1.5-Flash (scratchpad) 0.199 0.202 0.086 0.108 0.201 0.153 [0.131, 0.176] <0.001 21% 0%
57 Google Gemini-1.5-Pro (zero shot) 0.189 0.297 0.079 0.119 0.243 0.154 [0.126, 0.181] <0.001 24% 0%
58 Qwen Qwen1.5-110B-Chat (scratchpad with freeze values) 0.210 0.205 0.074 0.098 0.208 0.154 [0.134, 0.174] <0.001 20% 0%
59 Mistral AI Mixtral-8x22B-Instruct-V0.1 (scratchpad with news with freeze values) 0.204 0.181 0.088 0.105 0.193 0.154 [0.132, 0.177] <0.001 23% 0%
60 Anthropic Claude-3-5-Sonnet-20240620 (superforecaster with news 2) 0.198 0.220 0.090 0.114 0.209 0.156 [0.13, 0.182] <0.001 27% 0%
61 Anthropic Claude-2.1 (scratchpad with news) 0.216 0.190 0.075 0.096 0.203 0.156 [0.135, 0.177] <0.001 25% 8%
62 Qwen Qwen1.5-110B-Chat (scratchpad with news) 0.198 0.188 0.099 0.115 0.193 0.156 [0.135, 0.177] <0.001 25% 0%
63 Mistral AI Mixtral-8x22B-Instruct-V0.1 (scratchpad) 0.198 0.229 0.090 0.116 0.213 0.157 [0.136, 0.177] <0.001 23% 0%
64 Google Gemini-1.5-Flash (scratchpad with news with freeze values) 0.207 0.232 0.080 0.108 0.220 0.158 [0.131, 0.184] <0.001 25% 0%
65 Mistral AI Mixtral-8x7B-Instruct-V0.1 (scratchpad) 0.206 0.131 0.104 0.109 0.169 0.158 [0.136, 0.18] <0.001 27% 14%
66 Mistral AI Mistral-Large-Latest (zero shot) 0.185 0.195 0.118 0.132 0.190 0.159 [0.132, 0.185] <0.001 21% 0%
67 Mistral AI Mixtral-8x22B-Instruct-V0.1 (scratchpad with news) 0.204 0.209 0.092 0.114 0.207 0.159 [0.137, 0.18] <0.001 22% 0%
68 OpenAI GPT-4-Turbo-2024-04-09 (superforecaster with news 3) 0.218 0.161 0.087 0.101 0.189 0.159 [0.137, 0.181] <0.001 27% 17%
69 Qwen Qwen1.5-110B-Chat (zero shot) 0.202 0.226 0.092 0.116 0.214 0.159 [0.138, 0.18] <0.001 18% 1%
70 OpenAI GPT-4o (superforecaster with news 1) 0.202 0.255 0.086 0.116 0.228 0.159 [0.131, 0.188] <0.001 27% 0%
71 OpenAI GPT-4-Turbo-2024-04-09 (scratchpad with news) 0.208 0.203 0.090 0.110 0.206 0.159 [0.135, 0.184] <0.001 26% 0%
72 Anthropic Claude-3-Opus-20240229 (superforecaster with news 3) 0.192 0.129 0.126 0.127 0.161 0.159 [0.137, 0.182] <0.001 24% 10%
73 Anthropic Claude-2.1 (scratchpad with news with freeze values) 0.216 0.142 0.094 0.103 0.179 0.160 [0.138, 0.181] <0.001 24% 3%
74 OpenAI GPT-4-Turbo-2024-04-09 (superforecaster with news 1) 0.200 0.272 0.087 0.121 0.236 0.161 [0.136, 0.185] <0.001 21% 0%
75 Meta Llama-3-8b-Chat-Hf (zero shot) 0.202 0.329 0.075 0.121 0.266 0.162 [0.133, 0.191] <0.001 26% 0%
76 Mistral AI Mistral-Large-Latest (scratchpad with news with freeze values) 0.208 0.200 0.097 0.116 0.204 0.162 [0.138, 0.186] <0.001 25% 0%
77 Qwen Qwen1.5-110B-Chat (scratchpad) 0.210 0.239 0.087 0.115 0.225 0.162 [0.142, 0.183] <0.001 20% 0%
78 Meta Llama-3-8b-Chat-Hf (scratchpad with freeze values) 0.230 0.170 0.078 0.095 0.200 0.163 [0.145, 0.18] <0.001 22% 0%
79 Google Gemini-1.5-Flash (zero shot) 0.189 0.228 0.115 0.136 0.209 0.163 [0.135, 0.19] <0.001 22% 0%
80 Qwen Qwen1.5-110B-Chat (superforecaster with news 1) 0.208 0.303 0.077 0.118 0.255 0.163 [0.137, 0.189] <0.001 22% 0%
81 Anthropic Claude-3-Opus-20240229 (scratchpad with news with freeze values) 0.205 0.212 0.103 0.123 0.208 0.164 [0.14, 0.188] <0.001 25% 0%
82 Anthropic Claude-3-Opus-20240229 (scratchpad with news) 0.205 0.201 0.106 0.123 0.203 0.164 [0.141, 0.188] <0.001 26% 0%
83 Google Gemini-1.5-Pro (superforecaster with news 1) 0.212 0.267 0.086 0.119 0.239 0.165 [0.14, 0.191] <0.001 26% 0%
84 Mistral AI Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 1) 0.206 0.228 0.103 0.126 0.217 0.166 [0.14, 0.192] <0.001 23% 0%
85 Meta Llama-3-70b-Chat-Hf (scratchpad) 0.196 0.224 0.116 0.136 0.210 0.166 [0.147, 0.185] <0.001 22% 0%
86 Google Gemini-1.5-Flash (scratchpad with news) 0.207 0.203 0.108 0.125 0.205 0.166 [0.14, 0.192] <0.001 24% 0%
87 Mistral AI Mistral-Large-Latest (scratchpad with news) 0.208 0.164 0.117 0.126 0.186 0.167 [0.144, 0.19] <0.001 24% 0%
88 Anthropic Claude-3-Opus-20240229 (superforecaster with news 2) 0.193 0.243 0.119 0.141 0.218 0.167 [0.141, 0.193] <0.001 26% 0%
89 ForecastBench LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news 0.253 0.197 0.063 0.087 0.225 0.170 [0.155, 0.185] <0.001 19% 56%
90 ForecastBench LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news 0.253 0.206 0.062 0.088 0.229 0.170 [0.155, 0.185] <0.001 18% 56%
91 ForecastBench LLM Crowd (gpt-4o, claude-3.5-sonnet, gemini-1.5-pro) with news 0.253 0.203 0.063 0.089 0.228 0.171 [0.156, 0.186] <0.001 18% 56%
92 Mistral AI Mixtral-8x7B-Instruct-V0.1 (zero shot) 0.214 0.233 0.105 0.128 0.223 0.171 [0.137, 0.205] <0.001 25% 0%
93 Mistral AI Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 3) 0.229 0.168 0.101 0.113 0.199 0.171 [0.152, 0.191] <0.001 17% 24%
94 Mistral AI Mixtral-8x22B-Instruct-V0.1 (zero shot) 0.206 0.284 0.105 0.137 0.245 0.171 [0.142, 0.2] <0.001 23% 0%
95 Mistral AI Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 1) 0.240 0.207 0.080 0.103 0.223 0.172 [0.145, 0.198] <0.001 30% 15%
96 Anthropic Claude-2.1 (zero shot) 0.222 0.227 0.104 0.127 0.225 0.174 [0.155, 0.194] <0.001 19% 0%
97 OpenAI GPT-4o (superforecaster with news 2) 0.243 0.202 0.086 0.107 0.223 0.175 [0.146, 0.204] <0.001 25% 1%
98 Mistral AI Mistral-Large-Latest (superforecaster with news 1) 0.211 0.297 0.106 0.141 0.254 0.176 [0.148, 0.205] <0.001 24% 0%
99 Qwen Qwen1.5-110B-Chat (superforecaster with news 3) 0.233 0.207 0.101 0.121 0.220 0.177 [0.156, 0.198] <0.001 21% 8%
100 Mistral AI Mixtral-8x7B-Instruct-V0.1 (scratchpad with freeze values) 0.206 0.171 0.147 0.151 0.189 0.179 [0.15, 0.207] <0.001 27% 11%
101 Anthropic Claude-2.1 (superforecaster with news 3) 0.224 0.146 0.132 0.135 0.185 0.179 [0.156, 0.203] <0.001 24% 6%
102 Mistral AI Mistral-Large-Latest (superforecaster with news 3) 0.232 0.191 0.116 0.129 0.212 0.181 [0.159, 0.203] <0.001 22% 10%
103 Mistral AI Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 3) 0.239 0.166 0.113 0.123 0.202 0.181 [0.158, 0.203] <0.001 28% 14%
104 Mistral AI Mixtral-8x22B-Instruct-V0.1 (superforecaster with news 2) 0.238 0.188 0.110 0.124 0.213 0.181 [0.158, 0.205] <0.001 22% 2%
105 Mistral AI Mistral-Large-Latest (superforecaster with news 2) 0.214 0.235 0.130 0.149 0.225 0.182 [0.154, 0.209] <0.001 23% 1%
106 Meta Llama-3-8b-Chat-Hf (scratchpad) 0.230 0.229 0.114 0.135 0.230 0.183 [0.163, 0.202] <0.001 21% 0%
107 Meta Llama-2-70b-Chat-Hf (scratchpad with freeze values) 0.226 0.212 0.125 0.141 0.219 0.183 [0.165, 0.201] <0.001 22% 0%
108 Google Gemini-1.5-Flash (superforecaster with news 2) 0.217 0.247 0.128 0.150 0.232 0.183 [0.155, 0.212] <0.001 22% 0%
109 Mistral AI Mixtral-8x7B-Instruct-V0.1 (superforecaster with news 2) 0.256 0.185 0.098 0.114 0.220 0.185 [0.154, 0.216] <0.001 32% 22%
110 Qwen Qwen1.5-110B-Chat (superforecaster with news 2) 0.226 0.266 0.117 0.144 0.246 0.185 [0.162, 0.208] <0.001 24% 3%
111 Meta Llama-2-70b-Chat-Hf (zero shot with freeze values) 0.239 0.160 0.129 0.135 0.200 0.187 [0.158, 0.216] <0.001 24% 0%
112 Google Gemini-1.5-Flash (superforecaster with news 1) 0.229 0.298 0.116 0.149 0.263 0.189 [0.157, 0.221] <0.001 23% 0%
113 OpenAI GPT-4-Turbo-2024-04-09 (superforecaster with news 2) 0.242 0.186 0.126 0.137 0.214 0.190 [0.161, 0.218] <0.001 26% 2%
114 Google Gemini-1.5-Flash (superforecaster with news 3) 0.242 0.196 0.126 0.139 0.219 0.190 [0.166, 0.215] <0.001 20% 14%
115 Anthropic Claude-3-Haiku-20240307 (scratchpad with freeze values) 0.244 0.174 0.131 0.139 0.209 0.191 [0.17, 0.212] <0.001 22% 0%
116 Anthropic Claude-2.1 (superforecaster with news 2) 0.243 0.201 0.129 0.142 0.222 0.192 [0.165, 0.22] <0.001 27% 12%
117 Anthropic Claude-3-Haiku-20240307 (zero shot with freeze values) 0.296 0.152 0.076 0.090 0.224 0.193 [0.17, 0.215] <0.001 25% 1%
118 Google Gemini-1.5-Pro (superforecaster with news 2) 0.239 0.280 0.119 0.148 0.260 0.194 [0.161, 0.227] <0.001 22% 0%
119 Anthropic Claude-3-Haiku-20240307 (superforecaster with news 2) 0.238 0.187 0.142 0.150 0.213 0.194 [0.172, 0.217] <0.001 21% 0%
120 Meta Llama-2-70b-Chat-Hf (scratchpad) 0.226 0.248 0.149 0.167 0.237 0.197 [0.176, 0.217] <0.001 20% 0%
121 Anthropic Claude-3-Haiku-20240307 (scratchpad) 0.244 0.214 0.135 0.150 0.229 0.197 [0.175, 0.218] <0.001 24% 0%
122 Mistral AI Mixtral-8x7B-Instruct-V0.1 (scratchpad with news with freeze values) 0.300 0.155 0.083 0.096 0.227 0.198 [0.17, 0.225] <0.001 25% 13%
123 OpenAI GPT-3.5-Turbo-0125 (scratchpad with freeze values) 0.250 0.274 0.124 0.151 0.262 0.200 [0.179, 0.222] <0.001 22% 0%
124 Anthropic Claude-3-Haiku-20240307 (scratchpad with news with freeze values) 0.270 0.173 0.127 0.135 0.221 0.202 [0.182, 0.223] <0.001 22% 0%
125 Anthropic Claude-3-Haiku-20240307 (scratchpad with news) 0.270 0.202 0.127 0.141 0.236 0.205 [0.184, 0.226] <0.001 21% 0%
126 OpenAI GPT-3.5-Turbo-0125 (scratchpad) 0.250 0.274 0.140 0.165 0.262 0.207 [0.185, 0.229] <0.001 21% 0%
127 Mistral AI Mixtral-8x7B-Instruct-V0.1 (scratchpad with news) 0.300 0.197 0.097 0.115 0.248 0.207 [0.178, 0.237] <0.001 24% 13%
128 ForecastBench Always 0.5 0.250 0.250 0.146 0.165 0.250 0.208 [0.198, 0.218] <0.001 16% 0%
129 Anthropic Claude-2.1 (superforecaster with news 1) 0.269 0.258 0.124 0.148 0.264 0.209 [0.183, 0.234] <0.001 24% 4%
130 Anthropic Claude-3-Haiku-20240307 (zero shot) 0.296 0.174 0.116 0.127 0.235 0.211 [0.188, 0.235] <0.001 23% 1%
131 Meta Llama-2-70b-Chat-Hf (zero shot) 0.239 0.221 0.195 0.200 0.230 0.220 [0.189, 0.25] <0.001 22% 1%
132 Anthropic Claude-3-Haiku-20240307 (superforecaster with news 3) 0.261 0.224 0.168 0.178 0.243 0.220 [0.2, 0.24] <0.001 20% 33%
133 Anthropic Claude-3-Haiku-20240307 (superforecaster with news 1) 0.287 0.243 0.182 0.193 0.265 0.240 [0.211, 0.269] <0.001 21% 0%
134 OpenAI GPT-3.5-Turbo-0125 (zero shot with freeze values) 0.411 0.184 0.067 0.088 0.297 0.250 [0.216, 0.284] <0.001 38% 0%
135 ForecastBench Random Uniform 0.356 0.318 0.207 0.227 0.337 0.291 [0.249, 0.334] <0.001 24% 0%
136 ForecastBench Always 0 0.340 0.500 0.187 0.244 0.420 0.292 [0.23, 0.353] <0.001 32% 0%
137 OpenAI GPT-3.5-Turbo-0125 (zero shot) 0.411 0.218 0.163 0.173 0.314 0.292 [0.257, 0.327] <0.001 25% 0%
138 ForecastBench Always 1 0.660 0.500 0.606 0.587 0.580 0.624 [0.559, 0.688] <0.001 25% 0%