Forecasting AI Futures Resource Hub

Predictions, analyses, organizations, blogs... and everything else

Like a collector, I gather resources and information. Storing it and categorizing it. I do this for any project I undertake. But for Forecasting AI Futures, I realized the collection might be quite useful for others.

So, I set to work on restructuring it and make it look nice!

Here is the result: Forecasting AI Futures Resource Hub

The resource hub is still in its infancy but will grow to become more comprehensive over time. If you have ideas for how to improve it—helping it mature into its adult form—they are very welcome.

There are two major parts. The Prediction Database:

And the Forecasting AI Ecosystem:

I’ve also included some of the resources in this post.

Key Links

• Forecasting ecosystem: 🗺️Map of the Prediction Market & Forecasting Ecosystem​

• Introduction to Prediction Markets: Prediction markets: can betting be good for the world?

• New to Forecasting? Forecasting & Prediction - LessWrong

Some remarks

• AI trends: Analyzing trends in AI development and investments is crucial for forecasting AI. There is an organization doing exactly this: Epoch AI. See their collection of Machine Learning Trends.

• Prediction accuracy and calibration: Blindly deferring to a prediction without verifying its reliability is unwise. Check calibration and accuracy (e.g. Brier score). See Calibration City for calibration and accuracy data for popular prediction markets and Metaculus. Credible forecasters may publish their track records, like this track record for the Samotsvety forecasting group.

• Forecasting applications: Forecasting is a powerful decision-making tool. For instance, Convergence Analysis combines AI scenario forecasting with governance research and recommendations (see their Theory of Change).

• Forecasting bots: One AI capability of particular interest to forecasters is the ability to generate accurate predictions. AIs are, as far as I know, still lacking in forecasting skills compared to professional human forecasters. I have included some information, predictions, and resources regarding AI forecasting agents in the resource hub.

The prediction database

• The current space of predictions

  The prediction database highlights where forecasting communities are focusing their attention, and what they are missing. It is a tool for discovering questions that have yet to be asked—and then asking them.

• Why a hierarchical structure?

  A spreadsheet would have been more flexible than the current hierarchical structure of the database—each prediction could have several tags. It would, however, be harder to include important information in the database, such as links to benchmark leaderboards and other resources. Additionally, I find a hierarchical structure more intuitive—it mirrors my way of thinking about information and forecasting.

• Selected predictions

  Not every AI-related prediction is included in the database. The ones included reflect what I find meaningful and interesting. I have, however, surely introduced some pointless or vague predictions as well—I have not been extremely prudent with the selection since the database was originally created for personal use. I plan to refine the selections over time.