{"id":981,"date":"2021-10-31T12:00:00","date_gmt":"2021-10-31T12:00:00","guid":{"rendered":"https:\/\/forecastingresearch.org\/?post_type=research&#038;p=981"},"modified":"2026-04-17T17:44:50","modified_gmt":"2026-04-17T17:44:50","slug":"reciprocal-scoring-forecasting-unanswerable-questions","status":"publish","type":"research","link":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions","title":{"rendered":"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Abstract<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We propose an elicitation method, Reciprocal Scoring (RS), that challenges forecasters to predict the forecasts of other forecasters. Two studies show how RS can generate accurate forecasts of otherwise unanswerable questions. Study 1 establishes the epistemic credibility of RS: forecasters randomly assigned to use RS were as accurate as forecasters predicting objectively resolvable outcomes using a proper scoring rule\u2014and both groups were more accurate than a control group that felt accountable to neither intersubjective RS metrics nor objective metrics. Study 2 establishes the practical value of RS. We ask highly accurate forecasters to predict each other\u2019s forecasts of the effect of government policies on COVID-19 mortality, yielding a real-time ranking of the expected effectiveness of pandemic-containment policies. As in Study 1, RS forecasters converged but in this case on policy recommendations that stand up to scrutiny, even with the benefit of hindsight. The core contribution of RS is its power to create accountability for accuracy in policy debates that have long been stalemated by the absence of accountability.<\/p>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"btn orange\" href=\"https:\/\/dx.doi.org\/10.2139\/ssrn.3954498\" target=\"_blank\" rel=\"noreferrer noopener\">Read the full paper in SSRN <svg width=\"7\" height=\"9\" viewBox=\"0 0 7 9\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\">\n  <path d=\"M0.000156283 8.60806L4.22416 4.33606V4.24006L0.000156283 6.10352e-05H1.80816L6.06416 4.28806L1.80816 8.60806H0.000156283Z\" fill=\"#102B23\"\/>\n<\/svg>\n<svg width=\"8\" height=\"10\" viewBox=\"0 0 8 10\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\">\n  <path d=\"M0.601719 8.85794L4.82572 4.58594V4.48994L0.601719 0.249939H2.40972L6.66572 4.53794L2.40972 8.85794H0.601719Z\" fill=\"#102B23\"\/>\n<\/svg><\/a><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"Reciprocal scoring challenges forecasters to predict the forecasts of other forecasters, creating accountability for accuracy in policy debates.","protected":false},"featured_media":984,"template":"","meta":{"footnotes":""},"research_type":[5],"class_list":["post-981","research","type-research","status-publish","has-post-thumbnail","hentry","research_type-academic-article"],"acf":[],"yoast_head":"<title>Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute\" \/>\n<meta property=\"og:description\" content=\"Reciprocal scoring challenges forecasters to predict the forecasts of other forecasters, creating accountability for accuracy in policy debates.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions\" \/>\n<meta property=\"og:site_name\" content=\"Forecasting Research Institute\" \/>\n<meta property=\"article:modified_time\" content=\"2026-04-17T17:44:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2021\/11\/illustration_Midjourney_reciprocal-scoring.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1607\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions\",\"name\":\"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2021\\\/11\\\/illustration_Midjourney_reciprocal-scoring.jpg\",\"datePublished\":\"2021-10-31T12:00:00+00:00\",\"dateModified\":\"2026-04-17T17:44:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2021\\\/11\\\/illustration_Midjourney_reciprocal-scoring.jpg\",\"contentUrl\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2021\\\/11\\\/illustration_Midjourney_reciprocal-scoring.jpg\",\"width\":2560,\"height\":1607},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/reciprocal-scoring-forecasting-unanswerable-questions#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/forecastingresearch.org\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/#website\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/\",\"name\":\"Forecasting Research Institute\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/forecastingresearch.org\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>","yoast_head_json":{"title":"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions","og_locale":"en_US","og_type":"article","og_title":"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute","og_description":"Reciprocal scoring challenges forecasters to predict the forecasts of other forecasters, creating accountability for accuracy in policy debates.","og_url":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions","og_site_name":"Forecasting Research Institute","article_modified_time":"2026-04-17T17:44:50+00:00","og_image":[{"width":2560,"height":1607,"url":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2021\/11\/illustration_Midjourney_reciprocal-scoring.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions","url":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions","name":"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions &#8211; Forecasting Research Institute","isPartOf":{"@id":"https:\/\/forecastingresearch.org\/#website"},"primaryImageOfPage":{"@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage"},"image":{"@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage"},"thumbnailUrl":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2021\/11\/illustration_Midjourney_reciprocal-scoring.jpg","datePublished":"2021-10-31T12:00:00+00:00","dateModified":"2026-04-17T17:44:50+00:00","breadcrumb":{"@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions#primaryimage","url":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2021\/11\/illustration_Midjourney_reciprocal-scoring.jpg","contentUrl":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2021\/11\/illustration_Midjourney_reciprocal-scoring.jpg","width":2560,"height":1607},{"@type":"BreadcrumbList","@id":"https:\/\/forecastingresearch.org\/research\/reciprocal-scoring-forecasting-unanswerable-questions#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/forecastingresearch.org\/"},{"@type":"ListItem","position":2,"name":"Reciprocal Scoring: A Method for Forecasting Unanswerable Questions"}]},{"@type":"WebSite","@id":"https:\/\/forecastingresearch.org\/#website","url":"https:\/\/forecastingresearch.org\/","name":"Forecasting Research Institute","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/forecastingresearch.org\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/981","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research"}],"about":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/types\/research"}],"version-history":[{"count":7,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/981\/revisions"}],"predecessor-version":[{"id":1807,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/981\/revisions\/1807"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/media\/984"}],"wp:attachment":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/media?parent=981"}],"wp:term":[{"taxonomy":"research_type","embeddable":true,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research_type?post=981"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}