Decomposing the Effects of Crowd-Wisdom Aggregators: The Bias-Information-Noise (BIN) Model

Aggregating predictions from multiple judges often yields more accurate predictions than relying on a single judge, which is known as the wisdom-of-the-crowd effect. However, a wide range of aggregation methods are available, which range from one-size-fits-all techniques, such as simple averaging, prediction markets, and Bayesian aggregators, to customized (supervised) techniques that require past performance data, such as weighted averaging. In this study, the authors applied a wide range of aggregation methods to subjective probability estimates from geopolitical forecasting tournaments. The authors used the bias–information–noise (BIN) model to disentangle three mechanisms that allow aggregators to improve the accuracy of predictions: reducing bias and noise, and extracting valid information across forecasters. Simple averaging operates almost entirely by reducing noise, whereas more complex techniques such as prediction markets and Bayesian aggregators exploit all three pathways to allow better signal extraction as well as greater noise and bias reduction. Finally, the authors explored the utility of a BIN approach for the modular construction of aggregators.