1. Motivating Question
Imagine you’ve just seen Apple’s most recent return, , which is Apple’s long-run expected return, , plus some random noise, :
You want to use this realized return, , to estimate Apple’s long-run expected return, . The LASSO is a popular way to solve this problem. The LASSO estimates Apple’s long-run expected return, , by choosing a that’s as close as possible to the realized while taking into account an absolute-value penalty,
where is the strength of this penalty. If you use the LASSO, then you’ll estimate:
Suppose that you chose . If Apple’s most recent stock return was , then the LASSO will pick . And, if Apple’s most recent stock return was , then the LASSO will still pick . But, if Apple’s most recent stock return was , then the LASSO will give an estimate of .
The LASSO seems like it’s throwing away lots of information. In the example above, you didn’t adjust your estimate of Apple’s long-run expected return at all when you saw returns of and . So, it’s surprising that, if Apple’s long-run expected return, , was drawn from a Laplace distribution,
then using the LASSO to estimate would be the Bayesian thing to do (Park and Casella, 2008). If , then it’s correct to just ignore any return smaller than when estimating .
Why is this? If you cross your eyes and squint, you can sort of see why the Laplace distribution might be linked to the LASSO. Both use the Greek-letter and involve . But, lot’s of distributions use the absolute-value operator (e.g., the Wishart distribution). And, there are lots of Greek letters. That’s how letters work. I could just as easily have called the scale parameter in the Laplace distribution , , or instead of . So, what’s special about the Laplace distribution? What is it about the Laplace distribution that makes using the LASSO correct? How can it ever be Bayesian to throw information away?
2. Simpler Problem
To answer these questions, let’s start by looking at a simpler inference problem. Suppose that Apple’s long-run expected return is drawn from a Normal distribution, :
If is drawn from a Normal distribution, then you definitely don’t want to use the LASSO.
Bayes’ rule tells you that:
is the posterior likelihood that Apple’s long-run expected return is given that you’ve just seen a realized return of . is the probability that Apple realizes a return of if its long-run expected return is . And, is the probability that Apple’s long-run expected return is in the first place.
You want to choose the that maximizes this posterior likelihood , or equivalently, that minimizes the negative of the log of this posterior likelihood:
When Apple’s long-run expected return is drawn from a Normal distribution, you want to choose a that’s as close as possible to while taking into account a quadratic penalty not an absolute-value penalty. When is drawn from a Normal distribution, you’re never going to ignore small realized returns.
On one hand, you could pick a that’s really close to Apple’s recent return to make small. On the other hand, you could pick a close to to make small. Your priors determine what you do:
If you don’t have very strong priors about Apple’s long-run expected return (), then you’re going to pick since . By contrast, if you have very strong priors (), then you’re going to pick since . To illustrate, suppose that you’re really sure that Apple’s long-run expected return is close to with . Then, if you see Apple realize a return of , you’re going to think that this realization was probably due to a positive random shock, , and only pick .
3. Mixture Model
Now, let’s tweak the setup slightly. Suppose that, instead of being constant, the standard deviation of Apple’s long-run expected return can be either high or low,
with the high value much larger than and the low value much smaller than . Each case equally likely: . It turns out that you’re going to behave a lot like someone using the LASSO when you estimate Apple’s long-run expected return in this mixture model.
Regardless of the model, if you want to estimate Apple’s long-run expected return, then you have to use Bayes’ rule. And, just like before, Bayes’ rule tells you that:
But, now there’s an extra layer to the problem. The standard deviation of Apple’s long-run expected return can either be high or low,
You don’t know which it is. But, if you knew that , then you’d pick . Whereas, if you knew that , then you’d pick . Your estimate when is going to really different from your estimate when .
Let’s flesh out what this means. You want to estimate Apple’s long-run expected return, , by choosing the that maximizes the posterior likelihood ,
It’s hard to solve for analytically when and can take on arbitrary values, but the assumption that simplifies things nicely. And, the resulting analysis reveals why you’re going to do something LASSO-esque when learning about Apple’s long-run expected return in this mixture model.
There are cases. First, consider the case where Apple realizes a really big return, . This really big return would be really unlikely if because is really small. So, you can safely assume that and just solve the optimization problem from Section 2:
But, as we saw in Section 2 that, if your priors are really weak (), then you should ignore them since . So, you’re going to set whenever , just like someone using the LASSO.
Now, consider the other case where Apple realizes a really small return, . Again, this really small return would be really unlikely if because is really big. So, you can assume that and just solve the optimization problem:
But, now the opposite logic holds. If your priors are really strong (), then you should ignore since . So, you’re going to set whenever . This is the LASSO’s dead zone!
The figure below shows that, as the high and low standard deviations get more extreme, you’re going to behave more and more like someone using the LASSO when learning about Apple’s long-run expected return in this mixture model. But, the insight is more general than that. You’re going to behave like someone using the LASSO any time a small realized return, , tells you that you should be using stronger priors about Apple’s long-run expected return, .
4. Laplace Distribution
If Apple’s long-run expected return is drawn from a Laplace distribution, then you face an estimation problem just like the one in the mixture model above. Andrews and Mallows (1974) shows that a Laplace distribution can be re-written as the weighted average of Normal distributions with different standard deviations,
where the weights follow an Exponential distribution. The Exponential distribution has a really fat tail. If the standard deviation of Apple’s long-run expected return is distributed , then these standard deviations could be either really large or really small. We just saw that this is exactly what needs to happen for a LASSO-like estimation strategy to be optimal. There are lots of distributions for that have this property—we just saw another one above. But, if you use , then the probabilities of realizing large and small values of line up in such a way that it’s precisely optimal to use the LASSO.
In the original paper, there are a ton of extra hyper-parameters. For example, is a random variable. This clearly isn’t necessary. You just need the standard deviation of Apple’s long-run expected return to fluctuate wildly around . You can get a situation where the LASSO is really close to being optimal with just .
Also, in the original paper, there’s a lengthy discussion about properly “conditioning on .” The authors include this bizarre example of how the posterior distribution of might not be unimodal if you don’t condition on that, for me anyways, always seems to come out of left field. And, textbooks typically brush this point under the rug, calling it a technical conditions. But, the analysis above shows that it’s not just a technical condition. It’s actually really important!
To see why, consider estimating Apple’s long-run expected return in a mixture model with
The only difference from before is that instead of . If isn’t sufficiently large relative to , then you’re never going to ignore the Apple’s realized return when is small. With these new numbers, rather than . When choosing a distribution for , you’ve got to make sure that the high standard-deviation outcomes are big enough and the low standard-deviation outcomes are small enough relative to . Otherwise, a LASSO-like estimation strategy can’t be optimal.
FYI: Here’s the code to create the figures.