Enhancing the Wisdom of AI-Assisted Crowds: Theory and Experimental Evidence

Accurate forecasts are critical for managerial decision making. Such forecasts may be generated by human experts or by artificial intelligence (AI) technologies. A decision maker can benefit from the distinct advantages that each source may offer by providing AI assistance to the experts, allowing them to augment the information contained in the AI forecast by incorporating their own knowledge about the variable of interest. When multiple experts are available, accuracy can be further improved by utilizing the wisdom of crowds, forming a consensus by averaging each of their AI-assisted forecasts. However, the potential accuracy of a crowd of AI-assisted forecasters may be limited by two structural features. First, because the AI assistance is valuable to each expert at an individual level, the opinion of the AI can end up being overrepresented in the crowd’s consensus. Second, the experts may fail to appropriately utilize the AI assistance when forming their forecasts, either under- or overemphasizing the information it provides. Using a stylized Bayesian model of information aggregation, the authors develop a procedure that can recover the most accurate consensus forecast given all information collectively observed by the AI technology and every expert in the crowd. This procedure works by pivoting the average AI-assisted forecast either toward or away from the crowd’s average initial forecast. The authors test the performance of the proposed aggregation method in three laboratory experiments and find that it is at least as accurate as, and in many cases more accurate than, the AI-assisted crowd, AI technology on its own, and the unassisted crowd of forecasters.