Imagine we’re traders in a market where the cross-section of returns is related to variables:
In the equation above, denotes the mean return, and each captures the relationship between returns and the th right-hand-side variable. Some notation: I’m going to be using a “star” to denote true parameter values and a “hat” to denote in-sample estimates of these values. e.g., denotes an in-sample estimate of . To make things simple, let’s assume that , , , and .
Notice that learning about the most likely parameter values,
So, finding is homework question from Econometrics 101. Nothing could be simpler.
But, what if we’re interested in predicting returns,
rather than learning about the most likely parameter values? It might seem like this is the same exact problem. And, if we’re only considering right-hand-side variable, then it is the same exact problem. When the best predictions come from using . But, it turns out that when there’s or more right-hand-side variables (and an unknown mean), this is no longer true. When we can make better predictions with less likely parameters. When there’s a tension between the learning and predicting.
Why? That’s the topic of today’s post.
2. Maximum Likelihood
Finding the most likely (ML) parameter values is equivalent to minimizing the negative log likelihood. So, because we’re assume that , this is just
where the “” at the end denotes a bunch of terms that don’t include any of the parameters that we’re optimization over. Optimizing each parameter value then gives the first-order conditions below:
And, solving this system of equations and unknowns gives the most likely parameter values:
Clearly, the most likely parameter values are just the coefficients from a cross-sectional regression.
Now, for the sake of argument, let’s imagine there’s an oracle who knows the true parameter values, . With access to this oracle, we can compute our mean squared error when using the maximum-likelihood estimates to predict returns given any choice of true parameter values:
The first term, , captures the unavoidable error. i.e., even if we knew the true parameter values, we still wouldn’t be able to predict . And, the second and third terms, and , capture the error that comes from using the most likely parameter values rather than the true parameter values.
3. A Better Predictor
With this benchmark in mind, let’s take a look at a variant of the James-Stein (JS) estimator:
In the definition above, denotes a bias factor that shrinks the maximum-likelihood estimates towards zero whenever . So, with access to an oracle, we can compute our mean squared error when using the James-Stein estimates to predict returns just like before:
Now the third term is more complicated. If we use a more biased estimator, and , then using the most likely parameter values rather than the true parameter values to predict returns is going to cause less damage. But, bias is going to generate really bad predictions whenever the true parameter value is large . The and terms capture these two opposing forces.
Comparing the maximum-likelihood and James-Stein prediction errors reveals that we should prefer the James-Stein estimator to the maximum-likelihood estimator if there’s a such that:
But, here’s the thing: if we have access to an oracle, then there’s always going to be some that satisfies this inequality. This is easier to see if we rearrange things a bit:
So, there’s always a sufficiently small such that this inequality holds. Thus, for any , there’s some such that the James-Stein estimates gives better predictions than the most likely parameter values.
4. Abandoning the Oracle
Perhaps this isn’t a useful comparison? In the real world, we can’t see the true parameter values when deciding which estimator to use. So, we can’t know ahead of time whether or not we’ve picked a small enough . It turns out that having to estimate changes things, but only when there’s just right-hand-side variable. When there are variables, James-Stein with an estimated still gives better predictions.
To see where this distinction comes from, let’s first solve for the optimal choice of when we still have access to the oracle. This will tell us what we have to estimate when we abandon the oracle. The optimal choice of solves:
So, if we differentiate, then we can solve the first-order condition for :
If the maximum-likelihood estimates are really noisy relative to the size of the true parameter values (i.e., is close to ), then using the most likely parameter values rather than the true parameter values is going to increase our prediction error a lot. So, we should use more biased parameter estimates.
But, notice what this formula is telling us. To estimate the right amount of bias, all we have to do is estimate the variance of the true parameter values. We don’t have to estimate every single one. So, we can estimate the variance of the true parameter values as follows,
where the factor of on the left-hand side is a degrees-of-freedom correction. To see why we need this correction, recall that
so not all maximum-likelihood estimates can move independently. There is an adding-up constraint.
Thus, if we have to estimate the right amount of bias to use, then we should choose:
Notice that, when we have to estimate the right amount of bias, only if . If , then the denominator is infinite and . After all, if there’s only right-hand-side variable, then the equation to estimate the variance of the true parameter values has the same first-order condition as the equation to estimate . With this estimated amount of bias, our prediction error becomes
which is always less than the maximum-likelihood prediction error whenever .
5. What This Means
My last post looked at one reason why it’s harder to predict the cross-section of returns when : Bayesian variable selection doesn’t scale. If we’re not sure which subset of variables actually predict returns, then finding the subset of variables that’s the most likely to predict returns means solving a non-convex optimization problem. It turns out that solving this optimization problem means doing an exhaustive search over the powerset containing all possible subsets of variables. And, this just isn’t feasible when .
But, this scaling problem isn’t the only reason why it’s harder to predict the cross-section of returns when . And, this post points out another one of these reasons: even if you could solve this non-convex optimization problem and find the most likely parameter values, these parameter values wouldn’t give the best predictions. When , there’s a fundamental tension between making good predictions and learning about the most likely parameter values in the data-generating process for returns. So, when traders are going to solve the prediction problem and live with the resulting biased beliefs about the underlying parameter values. What’s more, the with the best out-of-sample fit in the data is going to quantify how much the desire to make good predictions distorts traders’ beliefs.