What makes a successful forecaster?

Without successful forecasting there is little chance of ever beating the market. We consider three levels of decision making relevant to investors.

Mar 26, 2017 | Scott E. Wolle

In brief

Without successful forecasting there is no chance of ever beating the market. But forecasting isn't easy – though investors have an advantage over some other groups whose views of the future are an important part of their work: investors can succeed with even a small majority of correct views, and can hold multiple, unrelated views at once. We consider three levels of decision making relevant to investors. These are: the quality of inputs, the combination of inputs to form a decision and the combination of decisions to create a portfolio. In this way, good judgement is absolutely possible – but it does not come naturally. It is the product of processes and team structures that explicitly pursue it.

Economic forecasting has a reputation for always getting it wrong. Yet it can't be denied that everybody eagerly awaits the most recent estimates of future growth, interest rates and stock market returns. We give an overview of common forecasting pitfalls, and show what can be done methodologically and conceptually to improve forecast quality.

The only function of economic forecasting is to make astrology look respectable. John Kenneth Galbraith

2016 may be remembered for many things – but excellence in forecasting will most assuredly not be one of them. Voters’ choices in favour of “Brexit” in the UK and of Donald Trump for President in the US stand as remarkably egregious misses by the experts. Yet, our surprise at these failures must itself be viewed as surprising, given the well-documented inaccuracy of expert forecasters1. Philip Tetlock, perhaps the most prominent researcher of expert judgement, famously stated that, “the average expert was roughly as accurate as a dart-throwing chimpanzee.”2 Ironically, it seems that the uncertainty associated with last year’s forecast errors has actually served to increase demand for forecasts.

Fortunately, the research on forecasting and judgement does contain some hopeful news. First, the fact that the average forecaster performs poorly allows for some forecasters to do well. Second, the forecasters who do perform well tend to exhibit behaviours that overcome the weaknesses that make forecasting so difficult. Investors have an advantage over some other groups whose views of the future are an important part of their work. We can succeed with even a small majority of correct views, and can hold multiple, unrelated views at once. This article considers three levels of decision making relevant to investors: the quality of inputs, the combination of inputs to form a decision and the combination of decisions to create a portfolio.

Inputs

Quantopian, a website and self-described crowdsourced quantitative investment firm, provides a wealth of tools to aspiring quantitative investors, including tutorials, fourteen years of data on stocks and the opportunity to license successful strategies to the firm. Membership has doubled in each of the past few years, and has now reached over 100,000. These members have developed over 300,000 investing algorithms.

Where is all the knowledge we have lost in information?  Thomas Stearns Eliot

Quantopian reflects both the opportunity and vulnerability of vast increases in data availability. Expanded information can help fundamental investors narrow their investment universe through the use of screens, as well as reduce the likelihood of certain cognitive errors [see Jones3 for a good summary]. The benefit for quantitative investors, of course, is more direct.

Yet, information has a cost in terms of false positives. Consider the 300,000 algorithms mentioned above: even if none of them had any information whatsoever, approximately 15,000 would pass standard statistical tests of significance, simply by random chance4.

To continue reading this article, click Download PDF

 

1 Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: a meta-analysis. Psychological assessment, 12(1), 19.
2 Tetlock, Philip E., and Dan Gardner. 2015. Superforecasting: the art and science of prediction.
3 Making Better (Investment) Decisions The Journal of Portfolio Management, vol. 40, no. 2. (January 2014), pp. 128-143, doi:10.3905/jpm.2014.40.2.128 by Robert C. Jones
4 A common test of significance is for a p value below 0.05 (Z score 1.645). In a normal distribution, 5% of observations will meet this criterion. 5% x 300,000 = 15,000. This assumes that the 300,000 were the total number of algorithms tested which may be a low estimate given that unsuccessful algorithms may not have been saved.