{"id":1680,"date":"2024-03-11T12:00:00","date_gmt":"2024-03-11T12:00:00","guid":{"rendered":"https:\/\/forecastingresearch.org\/?post_type=research&p=1680"},"modified":"2026-05-05T14:35:56","modified_gmt":"2026-05-05T14:35:56","slug":"roots-of-disagreement-on-ai-risk","status":"publish","type":"research","link":"https:\/\/forecastingresearch.org\/research\/roots-of-disagreement-on-ai-risk","title":{"rendered":"Roots of Disagreement on AI Risk"},"content":{"rendered":"\n

Abstract<\/h2>\n\n\n\n

We brought together generalist forecasters and domain experts (n=22) who disagreed about the risk AI poses to humanity in the next century. The \u201cconcerned\u201d participants (all of whom were domain experts) predicted a 20% chance of an AI-caused existential catastrophe by 2100, while the \u201cskeptical\u201d group (mainly \u201csuperforecasters\u201d) predicted a 0.12% chance. Participants worked together to find the strongest near-term cruxes: forecasting questions resolving by 2030 that would lead to the largest change in their beliefs (in expectation) about the risk of existential catastrophe by 2100. Neither the concerned nor the skeptics substantially updated toward the other\u2019s views during our study, though one of the top short-term cruxes we identified is expected to close the gap in beliefs about AI existential catastrophe by about 5%: approximately 1 percentage point out of the roughly 20 percentage point gap in existential catastrophe forecasts. We find greater agreement about a broader set of risks from AI over the next thousand years: the two groups gave median forecasts of 30% (skeptics) and 40% (concerned) that AI will have severe negative effects on humanity by causing major declines in population, very low self-reported well-being, or extinction.<\/p>\n\n\n\n

\n
View the full PDF report \n \n<\/svg>\n\n \n<\/svg><\/a><\/div>\n<\/div>\n\n\n\n
\n
Acknowledgments<\/summary>\n

This research would not have been possible without the support of Open Philanthropy. <\/p>\n\n\n\n

We thank the research participants for their invaluable contributions. <\/p>\n\n\n\n

We greatly appreciate the assistance of Page Hedley for data analysis and editing on the report, Taylor Smith and Bridget Williams as adversarial collaboration moderators, and Kayla Gamin, Coralie Consigny, and Harrison Durland for their careful editing. <\/p>\n\n\n\n

We thank Elie Hassenfeld, Eli Lifland, Nick Beckstead, Bob Sawyer, Kjirste Morrell, Adam Jarvis, Dan Mayland, Jeremiah Stanghini, Jonathan Hosgood, Dwight Smith, Ted Sanders, Scott Eastman, John Croxton, Raimondas Lencevicius, Alexandru Marcoci, Kevin Dorst, Jaime Sevilla, Rose Hadshar, Holden Karnofsky, Benjamin Tereick, Isabel Juniewicz, Walter Frick, Alex Lawsen, Matt Clancy, Tegan McCaslin, and Lyle Ungar for comments on the report.<\/p>\n<\/details>\n<\/div><\/div>\n\n\n\n

Executive summary<\/h2>\n\n\n\n

In the summer of 2022, researchers affiliated with the Forecasting Research Institute (FRI) (a<\/u><\/a>)1<\/a><\/sup> ran the Existential Risk Persuasion Tournament<\/u><\/a> (XPT) (a<\/u><\/a>), which identified large disagreements between domain experts and generalist forecasters about key risks to humanity (Karger et al. 2023). This new project\u2014a structured adversarial collaboration run in April and May 2023\u2014is a follow-up to the XPT focused on better understanding the drivers of disagreement about AI risk.<\/p>\n\n\n\n

Methods<\/h3>\n\n\n\n

We recruited participants to join \u201cAI skeptic\u201d (n=11) and \u201cAI concerned\u201d (n=11) groups that disagree strongly about the probability that AI will cause an existential catastrophe by 2100.2<\/a><\/sup> The skeptic group included nine superforecasters and two domain experts. The concerned group consisted of domain experts referred to us by staff members at Open Philanthropy (the funder of this project) and the broader Effective Altruism community.<\/p>\n\n\n\n

Participants spent 8 weeks (skeptic median: 80 hours of work on the\nproject; concerned median: 31 hours) reading background materials,\ndeveloping forecasts, and engaging in online discussion and video calls.\nWe asked participants to work toward a better understanding of their\nsources of agreement and disagreement, and to propose and investigate\n\u201ccruxes\u201d: short-term indicators, usually resolving by 2030, that would\ncause the largest updates in expectation to each group\u2019s view on the\nprobability of existential catastrophe due to AI by 2100.<\/p>\n\n\n\n

Results:\nWhat drives (and doesn\u2019t drive) disagreement over AI risk<\/h3>\n\n\n\n

At the beginning of the project, the median \u201cskeptic\u201d forecasted a 0.10% chance of existential catastrophe due to AI by 2100, and the median \u201cconcerned\u201d participant forecasted a 25% chance. By the end, these numbers were 0.12% and 20% respectively, though many participants did not attribute their updates to arguments made during the project.3<\/a><\/sup><\/p>\n\n\n\n

We organize our findings as responses to four hypotheses about what\ndrives disagreement:<\/p>\n\n\n\n

Hypothesis #1 – Disagreements about AI risk persist due to lack of engagement among participants, low quality of participants, or because the skeptic and concerned groups did not understand each other’s arguments4<\/a><\/sup><\/h4>\n\n\n\n

We found moderate evidence against these possibilities. Participants\nengaged for 25-100 hours each (skeptic median: 80 hours; concerned\nmedian: 31 hours), this project included a selective group of\nsuperforecasters and domain experts, and the groups were able to\nsummarize each other’s arguments well during the project and in\nfollow-up surveys. (More<\/u><\/a>)<\/p>\n\n\n\n

Hypothesis\n#2 – Disagreements about AI risk are explained by different short-term\nexpectations (e.g. about AI capabilities, AI policy, or other factors\nthat could be observed by 2030)<\/h4>\n\n\n\n

Most of the disagreement about AI risk by 2100 is not explained by indicators resolving by 2030 that we examined in this project. According to our metrics of crux quality, one of the top cruxes we identified is expected to close the gap in beliefs about AI existential catastrophe by about 5% (approximately 1.2 percentage points out of the 22.7 percentage point gap in forecasts for the median pair) when it resolves in 2030.5<\/a><\/sup> For at least half of participants in each group, there was a question that was at least 5-10% as informative as being told by an oracle whether AI in fact caused an existential catastrophe or not.6<\/a><\/sup> It is difficult to contextualize the size of these effects because this is the first project applying question metrics to AI forecasting questions that we are aware of.<\/p>\n\n\n\n

However, near-term cruxes shed light on what the groups believe, where they disagree, and why:<\/p>\n\n\n\n