Senior LLM Researcher

About FRI

Forecasting Research Institute (FRI) advances the science of forecasting to improve decision-making on high-stakes issues. Building on the work of Chief Scientist Philip Tetlock, FRI develops practical forecasting tools and applies them to society’s most critical decisions.

Our work includes:

• Creating panels of experts and skilled forecasters to make predictions about complex, long-term challenges facing society, including AI progress, bioweapons, and nuclear risk.

• Methodological innovations for predicting hard-to-forecast outcomes, such as applying adversarial collaboration to debates about the future of AI and improving forecasts of low-probability events.

• Using AI tools to enhance forecasting research. For example, we maintain a benchmark, ForecastBench, for evaluating the accuracy of AI-produced forecasts.

Our work has been featured in major news outlets, including The Economist, TIME, and Vox. Our team has presented our findings at side events at the United Nations, to Los Alamos National Laboratory, and to leading frontier AI companies, among other top decision-makers.

The role

LLMs have the potential to dramatically change the field of forecasting research. Our benchmark on LLM forecasting capabilities, ForecastBench, documents that LLMs already outperform the median public forecaster, and they are becoming more capable over time. 

The Forecasting Research Institute (FRI) seeks a Senior LLM Researcher to help us apply LLMs to speed up and improve the process of creating actionable forecasting reports for policymakers (e.g., see our work on forecasting nuclear risk and other global risks) and to advance the science of forecasting as applied to LLMs (e.g. creating benchmarks like ForecastBench and using LLMs to assess rationale quality).

Projects may include:

• Creating practically useful LLM tools to support all aspects of the forecasting research process, such as:

  • Using LLMs to quickly identify the highest-quality forecasting rationales within a dataset, and then quickly synthesizing why forecasters believe what they believe in a way that decisionmakers can easily engage with.

  • Using LLMs to augment adversarial collaborations: quickly generating large numbers of forecasting questions that get at the “crux” of why people disagree about topics such as the magnitude of AI risk or nuclear risk, and using LLMs to forecast and explain different perspectives on these topics in a way that enables faster assessment of which perspectives are more accurate.

• Doing scientific studies to rigorously establish where LLMs are more and less effective relative to humans (or in combination with them). For example, this could include:

  • Comparing the “value of information” provided by LLM-generated forecasting questions vs. questions written by top humans

  • Assessing how LLMs can best help humans improve their forecasting.

  • Expanding on our work on ForecastBench to assess LLMs’ proficiency relative to humans at doing conditional forecasting.

  • Using LLMs to rigorously assess the features of forecasting rationales most associated with forecasting accuracy

In this role you would manage one or more research projects. This may involve:

• Coordinating a team of research analysts and research assistants, delegating tasks as appropriate, and ensuring that projects stay on track to meet deadlines

• Contributing to the design of research projects that will help advance the science of forecasting and/or make forecasting more useful to policymakers

• Planning data collection and ensuring smooth implementation of this process

• Planning and overseeing data analysis, working closely with data analysts

• Interpreting the results of research, and identifying key findings and implications for policy and philanthropic priorities

• Coordinating the writing of project reports and (where appropriate) academic papers

• Planning and executing evaluation of research projects

Qualifications

The ideal candidate would have the following characteristics:

• Passionate about FRI’s mission of improving decision-making on high-stakes issues

• Strong coding skills (Python, LLM APIs, portfolio of projects showing technical expertise)

• Strong generalist research and writing skills

• Excellent project manager who drives work ahead independently

• Clear, explicit communicator, both in writing and verbally

• Familiarity with, and interest in, quantitative forecasting

• Ability to work productively in a self-directed (remote) environment

• Experience managing people is preferred but not required

• Experience managing the full research project cycle—from design to implementation to delivering results and evaluating—is preferred but not required

• Familiarity with some of the data skills listed in our data analyst job description, though not required, is preferred

We highly encourage you to apply, even if you don't feel completely qualified. Some of the best people we’ve worked with in our careers felt underqualified when they first applied, and we’re glad they did so anyway.