False factors: Be aware

There’s a lot of analysis of investment data and factors that show impressive results, but are any of them too good to be true? Invesco Quantitative Strategies explains how investors can spot poor data analysis.

Jul 11, 2019 | Xavier Gerard


Investment data is now easier to access and the tools used to analyse it have progressed dramatically. This, combined with the growing financial incentives behind factor investing, means that investors are increasingly presented with results that display impressive in-sample performance.

However, investors should develop a healthy degree of scepticism and be aware that without careful statistical adjustments, appropriate cross-validation and the imposition of realistic investment constraints many of these results may just be too good to be true.

Xavier Gerard, senior research analyst, recently presented Invesco Quantitative Strategies’ (IQS) stand on the issue at the latest Invesco Asia Investment Lecture Series on Factor Investing. The IQS team’s concerns echo those from prominent academics in the field such as Campbell Harvey and others.1  

Here are some ways in which the team thinks that they can help investors spot poor data analysis:
 

  • Question multiple testing: With more access to investment data, better analytical tools, and financial interests driving the need for breakthroughs in factor-investing research, the problem of data mining is becoming more acute today. As a result, countless factors are being tested, with many false factors that appear to work getting undue attention. It is therefore more important than ever for investors to scrutinize the basis for each factor. They should also look for a healthy and robust research environment, question what is it that incentivized the analysts to produce the research and keep track of how many factors are being tested.
     
  • Be sceptical of over-complexity: This is because slight tweaks to a model can alter results. To avoid this trap, investors should scrutinize the rationale more and ask how robust the results are to different signal constructions.
     
  • Don’t be fooled by overfitting: With multiple testing, this is perhaps the biggest problem facing investors today. In particular, investors should look out for combinations of vaguely-related and at best barely significant factors that are combined together into what looks like a strong predictor of future returns. The open secret about these models is that the weights on the factors are often chosen with some hindsight bias. Investors should avert this by scrutinizing the rationale for bundling weak signals together, ask for the full set of considered factors, question why plausible candidates are omitted, demand to see the individual statistical significance of each factor in the model and that the significance of the combined factors is tested under a multivariate t-test (e.g. Hotelling’s T-squared).
     
  • Beware the limits of out-sample testing: Though out-sample validation is the easiest way to rule out spurious factors, a true out-sample test is hard to find. Experienced researchers know too well how the historical environment impacts the performance of their factors. Moreover, a typical problem is when a factor is researched in a country with a view of testing its robustness in a separate market, assuming that this provides a valid out-sample. Unfortunately, there are often significant cross-country correlations in factor performance. To cut down the chances that a false factor data-mined in-sample is validated out-sample, investors should ask whether factor dynamics are different enough between markets so that there is more information on the signal’s predictive power.2  
     
  • Research environment vs. reality: An additional difficulty that investors must overcome in order to form realistic expectations from factor investing is the use of often unrealistic implementation constraints. Investors should keep an eye out for good results from research that does not take seriously into account liquidity restrictions, transaction costs or turnover constraints.
     

To conclude, the IQS team argues that factor research has significantly improved our understanding of how the market works and that factor investing constitutes an important pillar of modern investing. But they warn that the growing financial incentives behind factor investing are such that investors should display a healthy degree of scepticism when assessing research results. The team’s view is that factor investing is here to stay, but it must be done in the right way and investors must form realistic expectations.

Xavier Gerard is Senior Research Analyst at Invesco Quantitative Strategies.

^1 Arnott, Robert D. and Harvey, Campbell R. and Markowitz, Harry “A Backtesting Protocol in the Era of Machine Learning” The Journal of Financial Data Science, vol. 1, no. 1 (Winter): 64-74.  
^2 See Xiaomeng Lu, Robert F. Stambaugh, and Yu Yuan “Anomalies Abroad: Beyond Data Mining” NBER Working Paper No. 23809 (September 2017) available at https://www.nber.org/papers/w23809.pdf

 

Factor investing at Invesco

 

Related articles

Important information

The opinions referenced above are those of the author as of July 10, 2019. These comments should not be construed as recommendations, but as an illustration of broader themes. Forward-looking statements are not guarantees of future results. They involve risks, uncertainties and assumptions; there can be no assurance that actual results will not differ materially from expectations.

An investment in emerging market countries carries greater risks compared to more developed economies.

Fixed-income investments are subject to credit risk of the issuer and the effects of changing interest rates. Interest rate risk refers to the risk that bond prices generally fall as interest rates rise and vice versa. An issuer may be unable to meet interest and/or principal payments, thereby causing its instruments to decrease in value and lowering the issuer’s credit rating.

This document has been prepared only for those persons to whom Invesco has provided it for informational purposes only. This document is not an offering of a financial product and is not intended for and should not be distributed to retail clients who are resident in jurisdiction where its distribution is not authorized or is unlawful. Circulation, disclosure, or dissemination of all or any part of this document to any person without the consent of Invesco is prohibited.

This document may contain statements that are not purely historical in nature but are "forward-looking statements", which are based on certain assumptions of future events. Forward-looking statements are based on information available on the date hereof, and Invesco does not assume any duty to update any forward-looking statement. Actual events may differ from those assumed. There can be no assurance that forward-looking statements, including any projected returns, will materialize or that actual market conditions and/or performance results will not be materially different or worse than those presented.

The information in this document has been prepared without taking into account any investor’s investment objectives, financial situation or particular needs. Before acting on the information the investor should consider its appropriateness having regard to their investment objectives, financial situation and needs.

You should note that this information:
•    may contain references to amounts which are not in local currencies;
•    may contain financial information which is not prepared in accordance with the laws or practices of your country of residence;
•    may not address risks associated with investment in foreign currency denominated investments; and
•    does not address local tax issues.

All material presented is compiled from sources believed to be reliable and current, but accuracy cannot be guaranteed. Investment involves risk. Please review all financial material carefully before investing. The opinions expressed are based on current market conditions and are subject to change without notice. These opinions may differ from those of other Invesco investment professionals.

The distribution and offering of this document in certain jurisdictions may be restricted by law. Persons into whose possession this marketing material may come are required to inform themselves about and to comply with any relevant restrictions. This does not constitute an offer or solicitation by anyone in any jurisdiction in which such an offer is not authorised or to any person to whom it is unlawful to make such an offer or solicitation.