The Amelioration of Uncertainty

## You’re Calculating Volatility Wrong

You’re a student of volatility.  You wrote the book on volatility.  You probably invented volatility and would rather find out you’re adopted than be accused of unlawful calculation.  Nevertheless, you’re getting it wrong.

To calculate the volatility of end-of-day stock prices the usual rule is the standard deviation of returns.  This works if stocks are traded every day, but not for real data since markets are closed almost a third of the year.  The standard deviation overestimates the true volatility because it treats returns from Friday to Monday (72 hours) the same as from Monday to Tuesday (24 hours).

The good news is there’s a volatility estimator with these properties:

1. It is as simple to calculate as the standard deviation of returns.
2. It has a solid theoretical derivation.
3. It seems to correct the well known discrepancy between implied volatilities in option prices and realized volatilities.
4. It has an amazing “invariance under missing data property” that allows it accurately estimate the true volatility with 70-90% of the days missing.
5. It cures baldness.

So here is the estimator in detail

Where is the price at time (for more particulars see my statistics thesis).

The invariance under missing data property has to be seen to be believed.  Here are pictures of it for simulated and real end-of-day stock data.

So where does this “invariance under missing data” property come from?

It’s clear from the equation how it works.  The formula estimates the missing data and then proceeds as if it had a full set.  This is a common trick for handling missing data but here it falls out of the formula automatically.

Theoretically, the estimator is derived from the Brownian motion transition probabilities and the Maximum Likelihood Principle.  Both are well understood, but neither gives any hint where this property comes from.   To see its origin requires a deeper understanding of Brownian Motion.

Brownian motion has a “path integral” interpretation which deals with probability distributions on paths .  In fact, Brownian motion is a key example, playing a role entirely similar to the Normal Distribution on the real line.  Path integrals are familiar to Physicists from Quantum Field Theory but aren’t well known to Statisticians.  Looking at it from a statistician’s viewpoint, one sees why it works: the unknown prices are treated as nuisance parameters that are integrated out exactly the way a Bayesian would want.  The result is precisely the robustness against the unknown values that a Bayesian would expect.

One consequence is that any estimator similarly derived will have the same property.  Such path integral methods are ubiquitous in option pricing because of physicists-turned-quants and may yet be important for Economics proper.  Much more about that later topic another day.

June 8, 2011
• December 15, 2011steven

Have you actually compared returns over weekends and weekdays? I did a quick compare using log returns of SPY (compare weekend return with the next weekday’s return) and found no statistically significant differences. I applied a battery of tests: KS test for distributions, tests of mean, Levene’s test, and an eyeball test of the scatter plot. Nothing. Now this is a (weighted) basket of 500 stocks, and perhaps your results are for individual stocks (The title ‘S&P 100′ on your graph is somewhat ambiguous). But shouldn’t I see such an effect?

• December 16, 2011Joseph

Steven,

I believe there is academic research showing a volatility difference on non-trading days. On the other hand, I’ve seen results like yours before showing no difference.

I only looked at it one time. The idea was to use deviations from the “invariance under missing data property” as a gauge of the constancy of the volatility. I didn’t go real deep into it, but the result was that if there is some difference in volatility between when the market is open verses closed, it isn’t huge.

The efficient market folks would probably explain that result by saying “news” is generated at a similar rate during weekends or even overnight as it is during regular work hours.

• December 17, 2011steven

Joseph;

I am a little confused because it seems like your are modeling a property which may not exist in the market. Do you have any references on the academic research showing this effect exists?

• December 18, 2011Joseph

Just the opposite. Everything above assumes the property in question doesn’t exist.

This question has almost certainly been looked at thousands of times. The academic research I’m most familiar with is the French and Roll paper “Stock Return Variances: the arrival of Information and the Reaction of Traders”.

This seems to show a big effect, but I believe they are asking a slightly different question.

• December 22, 2011steven

Joseph;

Thanks for clearing that up. I’ll check out that paper!