{"id":2236,"date":"2026-06-01T12:00:00","date_gmt":"2026-06-01T12:00:00","guid":{"rendered":"https:\/\/forecastingresearch.org\/?post_type=research&#038;p=2236"},"modified":"2026-07-02T22:46:45","modified_gmt":"2026-07-02T22:46:45","slug":"measuring-judgment-quality-forecasting-tournaments","status":"publish","type":"research","link":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments","title":{"rendered":"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\">Abstract<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Decision-makers routinely rely on expert judgments accompanied by written explanations, yet explanation quality is difficult to measure at scale. Forecasting tournaments offer a natural testing ground: probabilistic judgments are paired with natural-language rationales and scored against realized outcomes. We introduce Explanation Quality Markers (EQMs), a set of sixty theory-guided reasoning patterns scored by large language models (LLMs). In a pre-registered analysis of over 55,000 forecast-rationale pairs from a multiyear forecasting tournament, EQMs predict accuracy at both the forecast and forecaster levels, consistently outperforming pre-LLM text-analysis methods. More than 90% of statistically significant pattern-level EQM-accuracy correlations match our directional hypotheses. The signal is asymmetric: EQMs identify likely underperformers more reliably than they distinguish the very best forecasters. Benchmarked against traditional indicators of forecasting skill, EQMs are the strongest predictor at the forecast level and competitive at the forecaster level, though weaker than prior accuracy. Human ratings of rationale quality are less consistently correlated with accuracy and place disproportionate weight on rationale length. Results transfer to an independent forecasting study. EQMs provide a scalable, interpretable method for extracting judgment-relevant information from written explanations.<\/p>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<details class=\"wp-block-details is-layout-flow wp-block-details-is-layout-flow\"><summary>Acknowledgments<\/summary>\n<p class=\"wp-block-paragraph\">This research was funded by a grant from Open Philanthropy (now known as Coefficient Giving).<\/p>\n<\/details>\n\n\n\n<details class=\"wp-block-details is-layout-flow wp-block-details-is-layout-flow\"><summary>Disclaimer<\/summary>\n<p class=\"wp-block-paragraph\">The views expressed in this paper do not necessarily reflect the views of the Federal Reserve Bank of Chicago or the Federal Reserve System.<\/p>\n<\/details>\n<\/div><\/div>\n\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"btn orange\" href=\"https:\/\/doi.org\/10.48550\/arXiv.2606.30987\">Available on arXiv <svg width=\"7\" height=\"9\" viewBox=\"0 0 7 9\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\">\n  <path d=\"M0.000156283 8.60806L4.22416 4.33606V4.24006L0.000156283 6.10352e-05H1.80816L6.06416 4.28806L1.80816 8.60806H0.000156283Z\" fill=\"#102B23\"\/>\n<\/svg>\n<svg width=\"8\" height=\"10\" viewBox=\"0 0 8 10\" fill=\"none\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\">\n  <path d=\"M0.601719 8.85794L4.82572 4.58594V4.48994L0.601719 0.249939H2.40972L6.66572 4.53794L2.40972 8.85794H0.601719Z\" fill=\"#102B23\"\/>\n<\/svg><\/a><\/div>\n<\/div>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"excerpt":{"rendered":"We introduce EQMs, a scalable, interpretable method for extracting judgment-relevant information from written rationales.","protected":false},"featured_media":2262,"template":"","meta":{"footnotes":""},"research_type":[5],"class_list":["post-2236","research","type-research","status-publish","has-post-thumbnail","hentry","research_type-academic-article"],"acf":[],"yoast_head":"<title>Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute\" \/>\n<meta property=\"og:description\" content=\"We introduce EQMs, a scalable, interpretable method for extracting judgment-relevant information from written rationales.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments\" \/>\n<meta property=\"og:site_name\" content=\"Forecasting Research Institute\" \/>\n<meta property=\"article:modified_time\" content=\"2026-07-02T22:46:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2026\/05\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1607\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments\",\"name\":\"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg\",\"datePublished\":\"2026-06-01T12:00:00+00:00\",\"dateModified\":\"2026-07-02T22:46:45+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments#primaryimage\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg\",\"contentUrl\":\"https:\\\/\\\/forecastingresearch.org\\\/wp-content\\\/uploads\\\/2026\\\/05\\\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg\",\"width\":2560,\"height\":1607},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/research\\\/measuring-judgment-quality-forecasting-tournaments#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/forecastingresearch.org\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/forecastingresearch.org\\\/#website\",\"url\":\"https:\\\/\\\/forecastingresearch.org\\\/\",\"name\":\"Forecasting Research Institute\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/forecastingresearch.org\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"}]}<\/script>","yoast_head_json":{"title":"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments","og_locale":"en_US","og_type":"article","og_title":"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute","og_description":"We introduce EQMs, a scalable, interpretable method for extracting judgment-relevant information from written rationales.","og_url":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments","og_site_name":"Forecasting Research Institute","article_modified_time":"2026-07-02T22:46:45+00:00","og_image":[{"width":2560,"height":1607,"url":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2026\/05\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments","url":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments","name":"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments &#8211; Forecasting Research Institute","isPartOf":{"@id":"https:\/\/forecastingresearch.org\/#website"},"primaryImageOfPage":{"@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments#primaryimage"},"image":{"@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments#primaryimage"},"thumbnailUrl":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2026\/05\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg","datePublished":"2026-06-01T12:00:00+00:00","dateModified":"2026-07-02T22:46:45+00:00","breadcrumb":{"@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments#primaryimage","url":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2026\/05\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg","contentUrl":"https:\/\/forecastingresearch.org\/wp-content\/uploads\/2026\/05\/illustration_Midjourney_eqm-measuring-judgment-quality.jpg","width":2560,"height":1607},{"@type":"BreadcrumbList","@id":"https:\/\/forecastingresearch.org\/research\/measuring-judgment-quality-forecasting-tournaments#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/forecastingresearch.org\/"},{"@type":"ListItem","position":2,"name":"Measuring Judgment Quality in Natural-Language Explanations: Evidence from Forecasting Tournaments"}]},{"@type":"WebSite","@id":"https:\/\/forecastingresearch.org\/#website","url":"https:\/\/forecastingresearch.org\/","name":"Forecasting Research Institute","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/forecastingresearch.org\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"}]}},"_links":{"self":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/2236","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research"}],"about":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/types\/research"}],"version-history":[{"count":27,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/2236\/revisions"}],"predecessor-version":[{"id":2436,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research\/2236\/revisions\/2436"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/media\/2262"}],"wp:attachment":[{"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/media?parent=2236"}],"wp:term":[{"taxonomy":"research_type","embeddable":true,"href":"https:\/\/forecastingresearch.org\/api\/wp\/v2\/research_type?post=2236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}