How to Measure Forecast Accuracy: How to Compare Professional Forecasters alongside Statistical Models
Abstract
The forecasting data is essential during the process of constructing investment strategies. It is critical to measure their accuracy ex post in order to evaluate and analyze the portfolio performance. In addition, measuring the sensitivity of our actions to the forecasts ex ante is also important for appropriate adjustments. We will be focusing on the ex post part of that in this article. We test the accuracy of forecasts using two methods: the distribution method and odds ratio measure. We also explore the use of the Information Coefficient (IC) to the forecasts data. The results show that the mean of forecasting data is the same as the empirical data, but the variance has different performance.
Table of Contents
1 Introduction
The forecasting asset returns, volatility and correlations play an important role in the investment strategy. They are used in the portfolio optimization process. In this case, the accuracy of forecasting data is essential. In this report, we are trying to use three methods to measure the forecasts accuracy.
- Comparing the distribution of forecasts and empirical data
- Using odds ratio to measure the accuracy
- Using Information Coefficient (IC) to measure the accuracy
This section references several forecasts from JPM throughout the years to use their asset price forecasts
Notes:
JPM Long Term Capital Market Assumptions (LTCMA) Release Dates
- 2016 - published 29 Oct 2015
- 2017 - published 26 Oct 2016
- 2018 - published 27 Oct 2017
- 2019 - published 29 Oct 2018
The price of ETFs data set is from Tiingo [https://www.tiingo.com/].
The time of data is from 2 Jan 2013 to 10 May 2019.
2 Related Research
2.1 The Distribution Method
By comparing the empirical distribution of ETFs return and the simualted distribution of the forecasting data, we can see the similar and different features of those two data sets.
2.2 The Odds Ratio Measure
Odds ratio is a statistic that quantifies the strength of the association of two events, A and B. This statistic is defined as the ratio of the odds of A in the presence of B and the odds of A in the absence of B. Two events are independent if and only if the odds ratio equals to 1.
In the Baysian Model, odds ratio is used to test the difference of two models for the selection purpose. In our report, we used this statistic to explore whether the assumption of forcasting data is different from the empirical data.
2.3 The IC Measure
The Information Coefficient (IC) is a measure used to evaluate the performance of an active strategy, which shows how closely the analysts' financial forecasts match the result from the empirical market. In this report, we implement IC to the forecasts data and empirical data.