Senior LLM Researcher
LLMs have the potential to dramatically change the field of forecasting research. Our benchmark on LLM forecasting capabilities, ForecastBench, documents that LLMs already outperform the median public forecaster, and they are becoming more capable over time.
The Forecasting Research Institute (FRI) seeks a Senior LLM Researcher to help us apply LLMs to speed up and improve the process of creating actionable forecasting reports for policymakers (e.g., see our work on forecasting nuclear risk and other global risks) and to advance the science of forecasting as applied to LLMs (e.g. creating benchmarks like ForecastBench and using LLMs to assess rationale quality).
Projects may include:
Creating practically useful LLM tools to support all aspects of the forecasting research process, such as:
Using LLMs to quickly identify the highest-quality forecasting rationales within a dataset, and then quickly synthesizing why forecasters believe what they believe in a way that decisionmakers can easily engage with.
Using LLMs to augment adversarial collaborations: quickly generating large numbers of forecasting questions that get at the “crux” of why people disagree about topics such as the magnitude of AI risk or nuclear risk, and using LLMs to forecast and explain different perspectives on these topics in a way that enables faster assessment of which perspectives are more accurate.
Doing scientific studies to rigorously establish where LLMs are more and less effective relative to humans (or in combination with them). For example, this could include:
Comparing the “value of information” provided by LLM-generated forecasting questions vs. questions written by top humans
Assessing how LLMs can best help humans improve their forecasting.
Expanding on our work on ForecastBench to assess LLMs’ proficiency relative to humans at doing conditional forecasting.
Using LLMs to rigorously assess the features of forecasting rationales most associated with forecasting accuracy
The role
In this role you would manage one or more research projects. This may involve:
Coordinating a team of research analysts and research assistants, delegating tasks as appropriate, and ensuring that projects stay on track to meet deadlines
Contributing to the design of research projects that will help advance the science of forecasting and/or make forecasting more useful to policymakers
Planning data collection and ensuring smooth implementation of this process
Planning and overseeing data analysis, working closely with data analysts
Interpreting the results of research, and identifying key findings and implications for policy and philanthropic priorities
Coordinating the writing of project reports and (where appropriate) academic papers
Planning and executing evaluation of research projects
Qualifications
The ideal candidate would have the following characteristics:
Passionate about FRI’s mission of improving decision-making on high-stakes issues
Strong coding skills (Python, LLM APIs, portfolio of projects showing technical expertise)
Strong generalist research and writing skills
Excellent project manager who drives work ahead independently
Clear, explicit communicator, both in writing and verbally
Familiarity with, and interest in, quantitative forecasting
Ability to work productively in a self-directed (remote) environment
Experience managing people is preferred but not required
Experience managing the full research project cycle—from design to implementation to delivering results and evaluating—is preferred but not required
Familiarity with some of the data skills listed in our data analyst job description, though not required, is preferred
We highly encourage you to apply, even if you don't feel completely qualified. Some of the best people we’ve worked with in our careers felt underqualified when they first applied, and we’re glad they did so anyway.
The fine print
Role details
We prefer full-time but are open to part-time roles for the right candidate.
This role is entirely remote, and we are able to hire people in most countries. Applicants can work whichever hours of the day work for them but must be consistently available between our core hours of 11am and 3pm ET.
Salary is commensurate with experience.
We provide 80% contribution towards fully funded health insurance for staff, 30 days per year of paid time off (including holidays), and up to 10 days of paid sick leave, among other benefits.
Application process
We are actively reviewing applications on a rolling basis and will take down this job posting when the role is filled.
The application process will vary by candidate but for successful candidates will typically involve:
An initial application, including a 1-hour work test
A paid 10-hour work test
Three virtual interviews with members of our team