{"id":812,"date":"2025-09-02T12:00:00","date_gmt":"2025-09-02T12:00:00","guid":{"rendered":"https:\/\/forecastingresearch.org\/?post_type=research&p=812"},"modified":"2026-05-04T17:42:48","modified_gmt":"2026-05-04T17:42:48","slug":"near-term-xpt-accuracy","status":"publish","type":"research","link":"https:\/\/forecastingresearch.org\/research\/near-term-xpt-accuracy","title":{"rendered":"Assessing Near-Term Accuracy in the Existential Risk Persuasion Tournament"},"content":{"rendered":"\n

Abstract<\/h2>\n\n\n\n

In June\u2013October 2022, we convened 169 people to participate in the \u201cExistential Risk Persuasion Tournament\u201d (XPT<\/a>). The XPT participants included both superforecasters with proven forecasting track records and domain experts with subject-matter expertise. The tournament incentivized accurate forecasting and persuasive argumentation about long-term risks humanity may face, including risks from artificial intelligence (AI), climate change, nuclear war, and pandemics. This report analyzes respondents\u2019 forecasting accuracy on 38 near-term questions that resolved by mid-2025. Key findings include: (a) there was overall performance parity between superforecasters and domain experts, with both groups underestimating AI progress and overestimating improvements in climate technology; (b) both superforecasters and domain experts substantially outperformed a baseline of educated members of the general public; (c) at the individual level, the median superforecaster and median domain expert performed statistically indistinguishably from simple extrapolation algorithms; (d) at the aggregate level, superforecasters and domain experts showed improved accuracy and some evidence of outperforming simple extrapolation algorithms; (e) there was no statistically significant correlation between near-term accuracy and long-term existential risk forecasts.<\/p>\n\n\n\n

\n
View the full PDF report \n \n<\/svg>\n\n \n<\/svg><\/a><\/div>\n<\/div>\n\n\n\n
\n
Acknowledgments<\/summary>\n

This research would not have been possible without the support of the\nMusk Foundation, Open Philanthropy, and the Long-Term Future Fund. We\ngreatly appreciate the assistance and input of Sam Glover, Rory Svarc,\nand Bridget Williams throughout the project.<\/p>\n<\/details>\n\n\n\n

Disclaimers<\/summary>\n

The views expressed in this paper do not necessarily reflect those of\nthe Federal Reserve Bank of Chicago or the Federal Reserve System.<\/p>\n<\/details>\n<\/div>\n\n\n\n

Executive Summary<\/h2>\n\n\n\n

This report evaluates the accuracy of near-term forecasts<\/strong> made by domain experts and superforecasters in the Existential Risk Persuasion Tournament (XPT<\/a>).1<\/a><\/sup><\/p>\n\n\n\n

Background<\/h3>\n\n\n\n

The XPT tournament took place in June\u2013October 2022. The tournament\nconvened 169 participants to generate probabilistic forecasts about\nhumanity\u2019s long-term future and potential global risks such as climate\nchange, nuclear war, pandemics, and artificial intelligence (AI). Of\nthese participants, 89 were superforecasters with track records of high\naccuracy on near-term questions, while 80 were domain experts. In\naddition, we sampled hundreds of public participants for comparison. The\nXPT represents the largest existential risk forecasting tournament to\ndate, uniquely combining superforecasters and domain experts to predict\nhumanity\u2019s long-term risks.<\/p>\n\n\n\n

The tournament included 59 forecasting questions set to resolve at\ndates ranging from mid-2024 to as late as 2100. These questions broke\ndown into 172 subquestions over multiple forecasting horizons and, in\nsome cases, across different countries. Out of these 172 subquestions,\n38 have known outcomes (i.e., are \u201cresolved\u201d) as of mid-2025. We note\nthat the XPT tournament concluded prior to the public release of ChatGPT\nin November 2022.<\/p>\n\n\n\n

Key Findings on Accuracy<\/h3>\n\n\n\n

Performance parity between superforecasters and domain\nexperts.<\/strong> The near-term questions revealed no meaningful\naccuracy differences between superforecasters and experts forecasting on\nquestions within their domain of expertise. Both groups achieved nearly\nidentical accuracy scores. The performance gap between the most- and\nleast-accurate XPT participant groups spanned just 0.18 standard\ndeviations, comparable to the difference between median and slightly\nabove-median performance. These small differences were not statistically\nsignificant, indicating that neither a proven forecasting track record\nnor domain expertise provided a consistent edge for these near-term\npredictions.<\/p>\n\n\n\n

Individual forecasters outperformed public participants but\nnot simple algorithms.<\/strong> Both superforecasters and domain experts\nstrongly outperformed a sample of educated public participants, who\nscored 1.82 standard deviations below the median XPT participant.\nHowever, individual forecasters\u2019 performance was not statistically\ndistinguishable from two simple algorithms: a \u201cno-change\u201d forecast and\ntrend extrapolation. These simple algorithms performed well partly\nbecause many questions involved low-probability events (which did not\noccur) or slow-moving variables (where trends persisted).<\/p>\n\n\n\n

Aggregate forecasts demonstrated the wisdom of\ncrowds.<\/strong> Median aggregation of XPT participants\u2019 forecasts\nachieved a substantial improvement over individual performance,\nincreasing accuracy by roughly 1 standard deviation. These aggregated\npredictions showed weak but positive evidence of outperforming the\n\u201cno-change\u201d forecast, though not trend extrapolation. This finding\nreinforces the well-established principle that combining multiple\nforecasts improves accuracy.<\/p>\n\n\n\n

Main Insights across Subject\nAreas<\/h3>\n\n\n\n

Despite the strong overall performance of aggregate forecasts, XPT\nparticipants systematically misjudged progress in specific domains.<\/p>\n\n\n\n

Respondents underestimated AI progress, especially\nsuperforecasters.<\/strong> XPT participants significantly underestimated\nthe pace of AI advancement across multiple benchmarks. For three\nstandard AI benchmarks\u2014MATH, MMLU, and QuALITY\u2014domain experts assigned\nprobabilities of 21.4%, 25.0%, and 43.5% respectively to the outcomes\nachieved by the end of 2024. Superforecasters were even more\npessimistic, assigning only 9.3%, 7.2%, and 20.1% respectively. The\nInternational Mathematical Olympiad results proved particularly\nsurprising: AI systems achieved gold-level performance in July 2025, an\noutcome to which domain experts assigned only an 8.6% probability and\nsuperforecasters a mere 2.3% probability. Overall, superforecasters\nassigned an average probability of just 9.7% to the observed outcomes\nacross these four AI benchmarks, compared to 24.6% from domain\nexperts.<\/p>\n\n\n\n

Climate technology progress was overestimated.<\/strong> In\ncontrast to AI, forecasters were overly optimistic about the development\nof green technology. In 2024, the cost of hydrogen produced using\nrenewable electricity remained higher than anticipated at $7.50 USD\/kg\n(median forecasts of $4.50 by superforecasters and $3.50 USD\/kg by\ndomain experts), and direct air CO\u2082 capture technology captured only\n0.01 MtCO\u2082\/year (median forecasts of 0.32 by superforecasters and 0.60\nMtCO\u2082\/year by domain experts).<\/p>\n\n\n\n

Implications for Long-Term\nRisks<\/h3>\n\n\n\n

No correlation between near-term accuracy and long-term\nexistential risk forecasts.<\/strong> There was no statistically\nsignificant correlation between forecasters\u2019 near-term accuracy and\ntheir forecasts of long-term risks. Ideally, we would use near-term\nforecasting ability to assess the reliability of forecasts about\nhumanity\u2019s long-term future. Unfortunately, in our XPT data, near-term\nforecasting accuracy did not consistently align with any particular\nposition on long-term risks. Overall, near-term forecasting accuracy\nprovides limited evidence at present for identifying who makes the most\ncredible long-term risk forecasts.<\/p>\n\n\n\n

Next Steps<\/h3>\n\n\n\n

Given the faster-than-expected progress on AI capabilities, it is\nmore important than ever to understand the likely future trajectory and\nimpact of AI. In our current and future work, we aim to shed more light\non these questions. Our current projects on this front include a\nlongitudinal panel of AI experts and a survey of economists on the\nexpected economic impacts of AI. Through these systematic efforts to\ngather expert perspectives, we will provide empirically grounded\ninsights that can inform policy and decision-making.<\/p>\n\n\n\n

1. Introduction<\/h2>\n\n\n\n

The Existential Risk Persuasion Tournament (XPT)2<\/a><\/sup> convened 169 participants from June to October 2022 to forecast questions about humanity\u2019s long-term future and the impact of global risks such as climate change, nuclear war, pandemics, and artificial intelligence (AI). Of these 169 participants, 89 were experienced forecasters with a track record of high accuracy on near-term questions (\u201csuperforecasters\u201d), and the other 80 were specialists working in domains related to global risks and humanity\u2019s future (\u201cexperts\u201d). Additionally, hundreds of public participants provided their answers to the same forecasting questions in 2023 and 2024.<\/p>\n\n\n\n

We recruited superforecasters with assistance from Good Judgment, Inc<\/u><\/a>. To find\nexperts, we contacted organizations, academic departments, and research\nlabs working on existential-risk-related issues; we also made several\nposts via social media and websites such as the Effective Altruism\nForum. We received hundreds of expressions of interest and offered slots\nto the most qualified among the interested applicants. The final expert\nsample included 32 AI experts, 12 biorisk experts, 12 nuclear experts, 9\nclimate experts, and 15 \u201cgeneral\u201d experts who study existential risks\nmore broadly (referred to as \u201cx-risk generalists\u201d). Many in the expert\npool were affiliated with the Effective Altruism (EA) community; 42% of\nexperts participating in the XPT reported having attended an EA meetup\nin the past.<\/p>\n\n\n\n

The median expert in the XPT forecast a 20% probability of global\ncatastrophe\u2014defined as a loss of at least 10% of the global\npopulation\u2014and a 6% probability of human extinction by 2100.\nSuperforecasters viewed the world as less risky, forecasting a 9% and 1%\nprobability of global catastrophe and human extinction by 2100,\nrespectively. This held across domains, though not uniformly:\nsuperforecasters and experts were much further apart on risk related to\nAI than on the risk of nuclear war.<\/p>\n\n\n\n

Participants in the tournament forecast on questions set to resolve\nat various dates ranging from as early as mid-2024 to as late as 2100.\nThe 59 forecasting questions in the XPT broke down into 172\nsubquestions. Of these, 32 questions (38 subquestions) have resolved as\nof the writing of this report. The resolved questions provide us with a\nunique opportunity to evaluate forecasting accuracy across different\nexpertise groups, identify key surprises, and explore the relationship\nbetween near-term forecasting accuracy and predictions of long-term\nexistential risks.<\/p>\n\n\n\n

While we analyze all resolved questions in our dataset, our confidence in resolutions varies across questions (see Table A1.2<\/a> in the Appendix). Of the 32 resolved forecasting questions, 47% (15\/32) have been definitively resolved based on authoritative data sources, while 53% (17\/32) have been provisionally resolved based on available evidence or expert consultation. These provisional resolutions reflect two constraints. First, some questions require expert panels for adjudication (particularly biorisk questions lacking clear ground truth). Second, others await authoritative data publications like International Energy Agency (IEA) reports or labor statistics from the Organisation for Economic Co-operation and Development (OECD).<\/p>\n\n\n\n

2. Forecasting Performance<\/h2>\n\n\n\n

2.1 Accuracy Metrics<\/h3>\n\n\n\n

We measure forecasting performance using two main accuracy\nmetrics.<\/p>\n\n\n\n

Our primary accuracy metric is the Accuracy Score<\/strong>.\nAccuracy Score<\/em> evaluates forecasting performance using the\noriginal XPT scoring rules (log score for binary questions; S score for\ncontinuous questions). Accuracy Score<\/em> is standardized to\nmeasure performance relative to the median XPT participant (i.e., the\nmedian across experts and superforecasters). For example, an\nAccuracy Score<\/em> of 0.25 means a forecaster was 0.25 standard\ndeviations more accurate than the median XPT participant. An\nAccuracy Score<\/em> of 0.25 would place them roughly in the top 40%\nof all forecasters. Higher Accuracy Score<\/em> values indicate\nbetter accuracy.<\/p>\n\n\n\n

Our secondary accuracy metric is Standardized Absolute\nForecast Error<\/strong> (SAFE<\/em>). SAFE<\/em> measures how\n\u201csurprising\u201d the actual outcome was relative to forecasters\u2019\nexpectations. For example, a SAFE<\/em> of 1.0 means the outcome was\n1 standard deviation from the forecast\u2014corresponding to a moderate but\nnot extreme surprise. Lower SAFE<\/em> values indicate better\naccuracy.<\/p>\n\n\n\n

Table 2.1 provides a summary and interpretation of our main accuracy metrics. Technical details are provided in the Appendix<\/a>.<\/p>\n\n\n\n

Metric<\/strong><\/td>Description<\/strong><\/td>Interpretation<\/strong><\/td>Use case<\/strong><\/td><\/tr>
Accuracy Score<\/strong>\n(primary)<\/em><\/td>

Average standardized score across all questions.<\/p>

Higher values are better.<\/p><\/td>

Accuracy Score<\/em> = 0.25 means a forecaster is 0.25 standard deviations more accurate than the median XPT participant, placing them roughly in the top 40% of all forecasters.<\/td>Used to measure relative performance.<\/td><\/tr>
Standardized Absolute Forecast\nError, SAFE<\/em><\/strong> (secondary)<\/em><\/td>

Average absolute forecast error in units of predictive standard deviations.<\/p>

Lower values are better.<\/p><\/td>

SAFE<\/em> = 1.0 means outcomes are on average one standard deviation from forecasters\u2019 expectations (corresponding to 16th<\/sup>\/84th<\/sup> percentile realizations).<\/td>Used to measure absolute performance; how\n\u201csurprising\u201d questions were to forecasters.<\/td><\/tr><\/tbody><\/table><\/div>
Table 2.1:<\/strong> Primary and secondary accuracy metrics used to evaluate forecasting accuracy.<\/figcaption><\/figure>\n\n\n\n

To provide an apples-to-apples comparison between the different\ngroups of XPT participants, we calculate accuracy metrics at the\nindividual forecaster level.<\/strong><\/p>\n\n\n\n

\n

Box 1: Individual\nversus aggregate forecasts<\/h4>\n\n\n\n

When analyzing forecasting performance, it is important to\ndistinguish between individual and aggregate forecasts. Individual\nforecasts<\/em> represent each forecaster\u2019s predictions, while\naggregate forecasts<\/em> combine predictions from multiple\nforecasters within a group.<\/p>\n\n\n\n

For individual-level accuracy, we calculate metrics for each\nforecaster separately, and then take the median across all individuals\nin a group. For aggregate-level accuracy, we first combine forecasts via\nmedian aggregation, and then calculate the accuracy of that combined\nforecast.<\/p>\n\n\n\n

To compare accuracy between different groups of XPT participants, we\nuse individual-level metrics. This is important to ensure fair\ncomparisons. The median subquestion in the XPT has 32 superforecaster\npredictions versus only 4 domain-expert predictions. Since aggregating\nmore forecasters improves accuracy via a wisdom-of-the-crowd\neffect<\/u><\/a>, comparing group aggregates would unfairly advantage\nsuperforecasters due to their greater sample size.<\/p>\n\n\n\n

Outside of group comparisons, however, we primarily analyze aggregate forecasts. In particular, we use aggregate forecasts when examining substantive questions\u2014such as whether forecasters correctly anticipated AI progress or developments in climate technology (see Section 3<\/a>). The reason is that aggregation yields more accurate predictions. As a result, the aggregate forecasts produce the most reliable measure of the XPT participants\u2019 collective judgment.<\/p>\n<\/div><\/div>\n\n\n\n

2.2 Relative Accuracy<\/h3>\n\n\n\n

2.2.1 Main Results<\/h4>\n\n\n\n

Figure 2.1 summarizes the overall forecasting performance.<\/p>\n\n\n\n

The graph provides the Accuracy Score<\/em> of the median XPT\nparticipant (i.e., individual-level accuracy) by subgroup:<\/p>\n\n\n\n